Hello! I’m having an issue with S3 service run with docker-compose container.
We are using localstack for local development, we have dynamodb tables and s3 bucket.
How the issue looks like - we have setup the scripts months ago. They were not touched in recent times. They were working just fine for months. But recently I’ve been having issue - after creating of local resources, I connect to them using nestjs application. It works fine, without issues. Application sees both dynamodb tables, as well as the s3 bucket. No issues. Then, somewhere, out of the blue, I’m getting following errors:
ERROR [ExceptionsHandler] getaddrinfo ENOTFOUND my-bucket-name.s3.localhost.localstack.cloud
Error: getaddrinfo ENOTFOUND my-bucket-name.s3.localhost.localstack.cloud
at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:108:26)
at GetAddrInfoReqWrap.callbackTrampoline (node:internal/async_hooks:130:17)
Literally nothing changed, the container was setup as first thing in the morning, it was working completely correctly for hours, and sometimes it just starts throwing ENOTFOUND error. It makes no sense. What makes even less sense, is the fact that there is no reliable way of fixing this issue. I can restart the container, recreate the whole infrastructure, restart the application - it does not help. The error resolves itself(stops being thrown) on it’s own… I did not observe any regularity here. Everything works - it stops, and then, in some random moment it just stops throwing errors and works again.
It’s been very frustrating as this error prevents me from work, and I did not find any reliable fix or workaround.
How the configuration looks like:
docker-compose.yml file:
>
> version: '3.8'
> services:
> localstack:
> container_name: 'my_container_name'
> image: localstack/localstack:latest
> ports:
> - '4566:4566'
> environment:
> - SERVICES=s3,dynamodb
> - DEBUG=1
> - DEFAULT_REGION=eu-central-1
> - AWS_ACCESS_KEY_ID=test
> - AWS_SECRET_ACCESS_KEY=test
> volumes:
> - './volume:/var/lib/localstack'
> - '/var/run/docker.sock:/var/run/docker.sock'
> - './create-bucket.sh:/etc/localstack/init/ready.d/create-bucket.sh'
> - './create-table.sh:/etc/localstack/init/ready.d/create-table.sh'
create bucket script:
#!/bin/bash
awslocal s3api create-bucket --bucket my-bucket-name --region eu-central-1 --create-bucket-configuration LocationConstraint=eu-central-1
awslocal s3api put-bucket-versioning --bucket my-bucket-name --versioning-configuration MFADelete=Disabled,Status=Enabled
awslocal s3api put-bucket-cors --bucket my-bucket-name --cors-configuration '{
"CORSRules": [
{
"AllowedHeaders": ["*"],
"AllowedMethods": ["GET"],
"AllowedOrigins": ["http://localhost"]
}
]
}'
Application configuration:
S3_BUCKET_NAME=my-bucket-name
AWS_REGION=eu-central-1
S3_ENDPOINT=http://s3.localhost.localstack.cloud:4566/
DYNAMO_DB_ENDPOINT=http://localhost:4566
What I tried so far - I updated the docker image to the newest one - did not help. I used forcePathStyle set to true for s3 client - did not help(this one actually makes the error 100% times reproducible). I changed the port of localstack container - did not help. I found some thread where somebody suggested increasing the container resources. I tried setting some high limit for the container - did not help.
I connected to the container and tried to fetch the bucket using command
awslocal s3api list-objects --bucket my-bucket-name
The bucket is there and is available. However, at the same time application can still throw ENOTFOUND errors.
This issue seems to be similar to this one: question: Does the s3 service work with multipart uploads · Issue #5016 · localstack/localstack · GitHub
however I already applied both of the suggested fixes. They did not help…
At the same time as I’m getting those s3 errors, the dynamodb works just fine from the same container(!). I’ve been checking container logs, and the ones related to dynamodb are returning correct results, while at the time of issue occurrence, the ones directed to s3 are not even logged. It seems like they are not reaching the endpoint related to s3(?).
Any idea what could be wrong?
