EKS Cluster in PODMAN/Localstack

Local Stack currentely implements EKS feature in K38 technology based on Docker only.

Need the same feature in PODMAN!

Hello,
Any one working on this feature?

Thanks
Matt

Hey Matt! Thanks for the feature request. Currently we don’t have this on our roadmap, but maybe you could share a bit more about your setup and what it is you want to test exactly.

Its that I am trying to introduce Loalstack to one of my Fortune 500 client as part of their Techstack!

My Setup/requirement is simple! Ability to create EKS cluster in localstack running in PODMAN! Very simple :slight_smile:

Thanks
Matt

PS: Considering the fact that most Fortune 100/500 companies including Toyota Motors North America, HCSC, WABTec etc moving away from Docker desk top to PODMAN and lots of them have AWS I am really surprised/litte diappointed that PODMAN comatibilty is not on your roadmap! :frowning:

Hey Matt,

the support for EKS with Podman depends mostly on K3D, which is the technology LocalStack uses for EKS. There is a document for K3D Podman support which involves a few setup steps. Using Podman instead of Docker - k3d

We can prepare a small sample, but in the meantime, could you give the setup steps a try?

Hello @thrau ,
I tried the instructions and still no luck! Any help will be greatly appreciated.

Thanks
Matt

Ready.
2023-06-12T20:37:58.628 WARN — [ asgi_gw_0] localstack.aws.accounts : Ignoring production AWS credentials provided to LocalStack. Falling back to default account ID.
2023-06-12T20:37:59.342 INFO — [ asgi_gw_0] localstack.utils.bootstrap : Execution of “_load_service_plugin” took 702.84ms
2023-06-12T20:37:59.342 INFO — [ asgi_gw_0] localstack.utils.bootstrap : Execution of “require” took 703.55ms
2023-06-12T20:37:59.344 INFO — [ asgi_gw_0] l.u.container_networking : Unable to get network name of main container “localstack_main”, falling back to “bridge”: Docker not available
2023-06-12T20:37:59.345 INFO — [ asgi_gw_0] l.u.container_networking : Determined main container network: bridge
2023-06-12T20:37:59.346 INFO — [ asgi_gw_0] l.u.container_networking : Unable to get IP address of main Docker container “localstack_main”: Docker not available
2023-06-12T20:37:59.364 WARN — [functhread12] l.utils.docker_utils : Unexpected error when attempting to determine container port status: Docker not available
2023-06-12T20:37:59.533 INFO — [ asgi_gw_0] localstack.request.aws : AWS eks.CreateCluster => 200
2023-06-12T20:38:59.371 INFO — [functhread12] l.services.eks.k8s_utils : Error creating cluster cluster1 - deleting and retrying: Timeout while waiting for EKS startup…
2023-06-12T20:39:00.712 WARN — [functhread12] l.utils.docker_utils : Unexpected error when attempting to determine container port status: Docker not available
2023-06-12T20:40:00.718 INFO — [functhread12] l.services.eks.k8s_utils : Error creating cluster cluster1 - deleting and retrying: Timeout while waiting for EKS startup…
2023-06-12T20:40:02.012 WARN — [functhread12] l.utils.docker_utils : Unexpected error when attempting to determine container port status: Docker not available

Hello,
@thrau et all!

Any feedback please?

Thanks
Matt

Hi Matt,

Thanks for sharing the logs. We have created an internal feature request and we will start looking into this! :pray:

We would appreciate if you can share us an end-to-end sample (similar to this Elastic Kubernetes Service (EKS) | Docs) so that we can test our implementation against it.

We are also looking to further improve our Podman support. Please let us know if you run into any issues with LocalStack on Podman!

Hello any feedback please?
Any reply will be much appreciated!

Thanks
Matt

Hello Matt,
we have a fix in the pipeline which is currently being reviewed by the team. We should have it shipped with the latest docker image by the end of the week.

Hello Matt!
We have a fix in place now. Please note that you have to start LocalStack with the new configuration flag: EKS_ENABLE_DNS_FIX=0 to allow podman to start.

With this flag, and the latest localstack/localstack-pro image (please do not forget to pull!), EKS should startup fine.

Hello @dfangl
Thank you very much for the reply! still no luck! Can you please share your docker-cmpose.yml and any other steps in detial?

Ready.
2023-06-28T15:16:09.858  WARN --- [   asgi_gw_0] localstack.aws.accounts    : Ignoring production AWS credentials provided to LocalStack. Falling back to default account ID.
2023-06-28T15:16:10.549  INFO --- [   asgi_gw_0] localstack.utils.bootstrap : Execution of "_load_service_plugin" took 681.55ms
2023-06-28T15:16:10.550  INFO --- [   asgi_gw_0] localstack.utils.bootstrap : Execution of "require" took 682.24ms
2023-06-28T15:16:10.551  INFO --- [   asgi_gw_0] l.u.container_networking   : Unable to get network name of main container "localstack_main", falling back to "bridge": Docker not available
2023-06-28T15:16:10.551  INFO --- [   asgi_gw_0] l.u.container_networking   : Determined main container network: bridge
2023-06-28T15:16:10.553  INFO --- [   asgi_gw_0] l.u.container_networking   : Unable to get main container IP address: Docker not available
2023-06-28T15:16:10.573  WARN --- [functhread12] l.utils.docker_utils       : Unexpected error when attempting to determine container port status: Docker not available
2023-06-28T15:16:10.724  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS eks.CreateCluster => 200


I mount the podman socket as docker socket into the container, and let it do the rest:

version: "3.8"

services:
  localstack:
    container_name: "${LOCALSTACK_DOCKER_NAME-localstack_main}"
    image: docker.io/localstack/localstack-pro  # required for Pro
    ports:
      - "127.0.0.1:4566:4566"            # LocalStack Gateway
      - "127.0.0.1:4510-4559:4510-4559"  # external services port range
      - "127.0.0.1:53:53"                # DNS config (required for Pro)
      - "127.0.0.1:53:53/udp"            # DNS config (required for Pro)
      - "127.0.0.1:443:443"              # LocalStack HTTPS Gateway (required for Pro)
    environment:
      - DEBUG=1
      - PERSISTENCE=${PERSISTENCE-}
      - LOCALSTACK_API_KEY=${LOCALSTACK_API_KEY-}  # required for Pro
      - DOCKER_HOST=unix:///var/run/docker.sock
      - EKS_ENABLE_DNS_FIX=0
    volumes:
      - "${LOCALSTACK_VOLUME_DIR:-./volume}:/var/lib/localstack"
      - "/run/podman/podman.sock:/var/run/docker.sock"

Hi @dfangl ,
I tried and still I am getting errors. The loalstack started OK this time. But I am getting the following error while creating EKS Cluster. Its still looking for Docker!

Traceback (most recent call last):
  File "/opt/code/localstack/.venv/lib/python3.10/site-packages/localstack/utils/container_utils/docker_sdk_client.py", line 74, in _create_client
    return docker.from_env(timeout=DOCKER_SDK_DEFAULT_TIMEOUT_SECONDS)
  File "/opt/code/localstack/.venv/lib/python3.10/site-packages/docker/client.py", line 96, in from_env
    return cls(
  File "/opt/code/localstack/.venv/lib/python3.10/site-packages/docker/client.py", line 45, in __init__
    self.api = APIClient(*args, **kwargs)
  File "/opt/code/localstack/.venv/lib/python3.10/site-packages/docker/api/client.py", line 197, in __init__
    self._version = self._retrieve_server_version()
  File "/opt/code/localstack/.venv/lib/python3.10/site-packages/docker/api/client.py", line 221, in _retrieve_server_version
    raise DockerException(
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', ConnectionRefusedError(111, 'Connection refused'))
2023-06-28T19:04:52.846  WARN --- [functhread27] l.utils.docker_utils       : Unexpected error when attempting to determine container port status: Docker not available
2023-06-28T19:04:52.846 DEBUG --- [functhread34] localstack.utils.run       : Executing command: ['/var/lib/localstack/lib/k3d/v5.4.9/k3d-linux-amd64', 'cluster', 'create', 'cluster1', '--image', 'rancher/k3s:v1.22.6-k3s1', '-p', '8081:80@loadbalancer']

Hello dfangl,
Any feedback please?

This issue occurs in Windows 10.

Thanks
Matt

Hell @dfangl ,
Is there any feedback on this issue?

I tried this on both Mac OSX (M2) and on winodws 10 with WSL2.

I am getting the same issue!!

If its working for you, it means I do not have the correct image! Could you please double check on this and get back to me?

I have been stuck on this for a while! Any help will be appreciated!!

Thanks
Matt

2023-06-30T17:14:02.869  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS eks.CreateCluster => 200
2023-06-30T17:15:02.665  INFO --- [functhread22] l.services.eks.k8s_utils   : Error creating cluster cluster1 - deleting and retrying: Timeout while waiting for EKS startup...
2023-06-30T17:15:03.766  WARN --- [functhread22] l.utils.docker_utils       : Unexpected error when attempting to determine container port status: Docker not available
2023-06-30T17:16:03.772  INFO --- [functhread22] l.services.eks.k8s_utils   : Error creating cluster cluster1 - deleting and retrying: Timeout while waiting for EKS startup...
2023-06-30T17:16:04.894  WARN --- [functhread22] l.utils.docker_utils       : Unexpected error when attempting to determine container port status: Docker not available

On both Windows and MacOS, you podman will run in a virtual machine. Therefore, you have to ensure to mount the correct socket.

You can find your podman connections by running: podman system connection ls (podman-system-connection-list — Podman documentation).

You also might need to set your machine as rootful.

Unfortunately, I do not have a Windows or MacOS machine with podman installed right now, but the gist is, that you have to make sure to mount the correct socket into the container, or, alternatively, skip mounting a socket and set DOCKER_HOST to connect to the podman socket via tcp.

HI @dfangl ,
On Windows I get

# podman system connection list  

podman-machine-default       ssh://user@localhost:54641/run/user/1000/podman/podman.sock  C:\Users\matt.madhavan\.ssh\podman-machine-default  true         
podman-machine-default-root  ssh://root@localhost:54641/run/podman/podman.sock            C:\Users\matt.madhavan\.ssh\podman-machine-default  false       

And on Mac :

podman-machine-default       ssh://core@127.0.0.1:61324/run/user/503/podman/podman.sock  /Users/matt/.ssh/podman-machine-default  false
podman-machine-default-root  ssh://root@127.0.0.1:61324/run/podman/podman.sock           /Users/matt/.ssh/podman-machine-default  true

Any ideas as how to proceed further?

I appreciate isn advance!

Thanks
Matt

Hi @dfangl ,
For the Mac I tried to work based on the following link!

[type or paste code here](https://onlyutkarsh.medium.com/podman-cannot-connect-to-the-docker-daemon-error-on-macos-f65a6759204)

Still no luck. Any ides please?

Thanks
Matt

I will try to check it out on Linux using a separate podman machine this week, to check out if this will be helpful for your operating systems.