Is it possible to avoid using Docker in Docker and simply export all internal containers?

Hey!

So I’ve been banging my head against a wall for over a week on LocalStack to create our local development environment.
We are using Traefik as a reverse-proxy to use domains for all of our containers which works perfectly fine.
LocalStack however is being very temperamental about this and to be honest it is starting to be quite the nightmare to setup.
Since apparently the localstack/localstack image is actually running several containers, would it be possible to have the images of these containers separately?
I feel like LocalStack is trying to do too many things at the same time and it is creating more problems than it solves in it’s current state of bundling. It works fine for basic Docker infrastructures but for more advanced setups it really creates impossible situations.

I know this can sound a bit hard. My intention is not to say LocalStack is bad, it’s actually wonderful. I just think it is trying way too hard to solve an issue that is completely outside it’s scope. I think networking should be left to users or at least we should have an option to do so. If there is a way to do this, it would solve most of our issues and make our lives (mine mainly) way better.

Thanks in advance for helping me and thanks a lot for your work!

Hi @SmashingQuasar,

We would like to help you.
Could you please provide further details on what you would like to do and how LocalStack is currently blocking it?
We strive to accommodate everyone’s network configuration and make it as easy as possible.
Could you please provide us with a brief description of where our documentation is lacking?

Links:
Network troubleshooting | Docs (localstack.cloud)
Transparent Endpoint Injection | Docs (localstack.cloud)
Configuration | Docs (localstack.cloud)
Elastic Kubernetes Service (EKS) | Docs (localstack.cloud)

Hey!

Thanks for your answer!

We have a complex setup but I will try to explain it as clearly as possible.

We are using Traefik as a reverse-proxy to handle all container interactions.
Traefik does not use the docker-compose ports or expose properties nor the EXPOSE within the Dockerfile. Instead it uses it’s own labels to setup the networking.
There are many benefits but the one important thing here is that it handles HTTPS with it’s own certificate store (that we are generating on our end) and it also obfuscates ports so multiple containers can be ran on the same port. (Traefik realizes a hidden mapping to make this work)

Right now, when calling LocalStack from the host machine, things work as intended. It is answering through HTTPS and the mapping 443:4566 is behaving nicely.
Problems start to appear when trying to reach LocalStack from within a container.
Because we are not using LocalStack networking, we cannot rely on the domains you are providing (*.localstack.cloud and so on). Instead, we must use Docker’s internal networking through a network that we call traefik_proxy.

This means when we want to call LocalStack from within one of our containers, we need to call it by it’s container name (for example localstack-container) instead of a domain name. When doing this, a lot of libraries (such as aws-sdk for Node.JS or boto3 for Python) break because LocalStack is trying to send them a certificate, but this certificate does not match since we are using the container name.
I am aware that we can give LocalStack our certificate, however this will not work because a container name does not have a TLD which will lead to an invalid certificate. We tried this solution and the certificate got rejected every time.
We tweaked the code for aws-sdk and boto3 to allow the use of pure HTTP instead of HTTPS. This somewhat improved things but we are still having issues.

I need to detail the workflow of what breaks for you to be able to perceive why it does not work.

We have a Python container that will call LocalStack using boto3 through HTTP to generate a pre-signed URL. This works as intended.
This Python container will then spawn a child process that is actually calling the ffmpeg binary and passes it the presigned URL. At this point, LocalStack starts acting up. As long as the pre-signed URL contains any parameter, it will reject the connection. So we basically need to skip the pre-signed URL for our local environment for this to work and instead give it the URL without the 3 pre-signed parameters.
This is a workaround that unfortunately led us to yet another problem.

FFMPEG is a software that can do very long processes and we obviously need it to go through to validate our code. The issue here, is that this call is asynchronous. Meaning several FFMPEG processes can be spawned simultaneously. This is when things fall apart and we haven’t been able to figure out a solution yet. When this happens, LocalStack will reject any other connection. It will accept the first one but you cannot have more than one connection at the same time. It is important to note that FFMPEG is using TCP to call LocalStack. It is directly calling tcp://<URL>.
That shouldn’t be an issue since TCP can obviously handle multiple connections on the same port, but it seems to be the root of the issue.

I know this is a complex setup with a lot to digest so let me know if anything is unclear so I can give you more information.

I will be a bit insisting on this, but if we could simply run the S3 container instead of the whole LocalStack container stack, it would solve our issue since we would be able to handle all this traffic through Traefik and Docker without having to work through the exposed LocalStack router (or proxy, not sure how it is built).

Thanks a lot for your time!

Hi @SmashingQuasar and thanks for the details!
I cannot answer so much about the networking part, but I can maybe help for your S3 issues.

Regarding the generation of the pre-signed URL, it is purely a client side operation, meaning using boto3 to generate a pre-sign URL will not call LocalStack, but will use your configured client credentials and endpoint to generate a pre-signed URL locally, with no network call.
About LocalStack rejecting the connection, this seems extremely bizarre that query string parameters would make it reject it, and by default LocalStack does not reject invalid pre-signed URL, it would need to be activated by a feature flag. We might have an issue there, and it would be useful to have LocalStack logs when this happens. If it is possible, could you open a GitHub issue or if there’s not much to share, just reply in this post what is happening on LocalStack side when it receives the request?

For the FFMPEG TCP call, would it be possible to have a minimal reproducible sample? This seems like an issue, feel free to open a GitHub issue too, we’d be glad to have a look and try to solve it!

About running the S3 container, S3 actually runs fully inside the LocalStack container as Python code, we have our gateway and handler chain to be able to properly route requests to the right service and operation, you can read more about it here. Most of our services are running inside the LocalStack container, and only some of them (lambda, etc…) will spawn additional containers on the host machine, but not inside the LocalStack container. I hope that makes it a bit more clear?

Thanks a lot for taking the time to give more details about your use case, and we’re very glad that you’re liking LocalStack so far, and I would like to assist you in making the setup easier for your local development environment. Sorry to not be able to answer about the networking part, somebody else from the team will!

Hi @SmashingQuasar , thanks for raising this issue. I will try to answer the networking part of your use case and get back to you with a configuration that may be suitable. I want to improve our documentation around reverse proxies and the like, and your approach is not uncommon. As part of an ongoing initiative, we are trying to make connecting to LocalStack more straightforward. You are right that we have good documentation around the simple use cases, and may be lacking in more advanced setups, but we need to work with people like yourself to understand these use cases.