Glue stopped working after updating to version 4

Platform
1

Heya,

I recently cleared out my docker images and just deleted the localstack one knowing it’d just pull when I restarted.

I have a Glue job which is just a simple ETL from dynamodb and was working literally 10 minutes beforehand so I haven’t touched the code.

Now when I run it with aws glue start-job-run --job-name example-glue-job I get:

ERROR: '['pip', 'show', 'aws-glue-libs']': exit code 1; output: b'WARNING: Package(s) not found: aws-glue-libs\n'
ERROR: '['/var/lib/localstack/lib/spark/3.1.1/bin/spark-submit', '--files', '/var/lib/localstack/lib/spark-drivers/spark-common/glue-override.conf', '/tmp/script-177bbbf5.py', '--JOB_NAME', 'example-glue-job']': exit code 1; output: b'ERROR: JAVA_HOME None does not exist.\n/var/lib/localstack/lib/spark/3.1.1/bin/spark-class: line 71: None/bin/java: No such file or directory\n/var/lib/localstack/lib/spark/3.1.1/bin/spark-class: line 96: CMD: bad array subscript\n'
2024-11-30T21:37:19.359  INFO --- [-functhread9] l.p.c.s.glue.job_executor  : Glue job execution failed: Command '['/var/lib/localstack/lib/spark/3.1.1/bin/spark-submit', '--files', '/var/lib/localstack/lib/spark-drivers/spark-common/glue-override.conf', '/tmp/script-177bbbf5.py', '--JOB_NAME', 'example-glue-job']' returned non-zero exit status 1.

Any ideas?

Thanks

Marc

ps. loving localstack am currently creating a demo for my employer :slight_smile:

2

Also to add a more readable log from cloudwatch

2024-11-30 22:01:05
[1d489959]
ERROR: JAVA_HOME None does not exist.
2024-11-30 22:01:05
[1d489959]
/var/lib/localstack/lib/spark/3.1.1/bin/spark-class: line 71: None/bin/java: No such file or directory
2024-11-30 22:01:05
[1d489959]
/var/lib/localstack/lib/spark/3.1.1/bin/spark-class: line 96: CMD: bad array subscript
2024-11-30 22:01:05
[1d489959]
None

and the glue job(which deploys fine)

resource "aws_glue_job" "example" {
    name        = "example-glue-job"
    role_arn    = aws_iam_role.glue_role.arn
    command {
        name            = "glueetl"
        script_location = "s3://test-bucket/transform.py"
        python_version  = "3"
    }

    glue_version = "3.0"
    number_of_workers = 10
    worker_type = "G.1X"    
}

I’ve also tested to confirm, the same code works on the 3.8 image fine!

3

Hi @marc, Please delete your volume folder and allow it to re-download all the required third-party tools. The latest release introduces changes that should be fixed by re-creating the volume folder.

4

Heya, yeah thanks for this - I should mention that I was initially using the localstack cli before switching to docker compose, I can replicate the issue on the cli when upgrading with a non pip install to v4 - I’d imagine it’s the same issue - thanks for your help!

All fixed!

1 Like