Hey!
Thanks for your answer!
We have a complex setup but I will try to explain it as clearly as possible.
We are using Traefik as a reverse-proxy to handle all container interactions.
Traefik does not use the docker-compose ports or expose properties nor the EXPOSE within the Dockerfile. Instead it uses it’s own labels to setup the networking.
There are many benefits but the one important thing here is that it handles HTTPS with it’s own certificate store (that we are generating on our end) and it also obfuscates ports so multiple containers can be ran on the same port. (Traefik realizes a hidden mapping to make this work)
Right now, when calling LocalStack from the host machine, things work as intended. It is answering through HTTPS and the mapping 443:4566 is behaving nicely.
Problems start to appear when trying to reach LocalStack from within a container.
Because we are not using LocalStack networking, we cannot rely on the domains you are providing (*.localstack.cloud and so on). Instead, we must use Docker’s internal networking through a network that we call traefik_proxy.
This means when we want to call LocalStack from within one of our containers, we need to call it by it’s container name (for example localstack-container) instead of a domain name. When doing this, a lot of libraries (such as aws-sdk for Node.JS or boto3 for Python) break because LocalStack is trying to send them a certificate, but this certificate does not match since we are using the container name.
I am aware that we can give LocalStack our certificate, however this will not work because a container name does not have a TLD which will lead to an invalid certificate. We tried this solution and the certificate got rejected every time.
We tweaked the code for aws-sdk and boto3 to allow the use of pure HTTP instead of HTTPS. This somewhat improved things but we are still having issues.
I need to detail the workflow of what breaks for you to be able to perceive why it does not work.
We have a Python container that will call LocalStack using boto3 through HTTP to generate a pre-signed URL. This works as intended.
This Python container will then spawn a child process that is actually calling the ffmpeg binary and passes it the presigned URL. At this point, LocalStack starts acting up. As long as the pre-signed URL contains any parameter, it will reject the connection. So we basically need to skip the pre-signed URL for our local environment for this to work and instead give it the URL without the 3 pre-signed parameters.
This is a workaround that unfortunately led us to yet another problem.
FFMPEG is a software that can do very long processes and we obviously need it to go through to validate our code. The issue here, is that this call is asynchronous. Meaning several FFMPEG processes can be spawned simultaneously. This is when things fall apart and we haven’t been able to figure out a solution yet. When this happens, LocalStack will reject any other connection. It will accept the first one but you cannot have more than one connection at the same time. It is important to note that FFMPEG is using TCP to call LocalStack. It is directly calling tcp://<URL>.
That shouldn’t be an issue since TCP can obviously handle multiple connections on the same port, but it seems to be the root of the issue.
I know this is a complex setup with a lot to digest so let me know if anything is unclear so I can give you more information.
I will be a bit insisting on this, but if we could simply run the S3 container instead of the whole LocalStack container stack, it would solve our issue since we would be able to handle all this traffic through Traefik and Docker without having to work through the exposed LocalStack router (or proxy, not sure how it is built).
Thanks a lot for your time!