Localstack start command returning 'Enable privileged port mapping' error

PJ_Simpson · October 3, 2023, 7:59am

I have been using community version without issue, however upon receiving my pro key and starting localstack like so:

LOCALSTACK_API_KEY=XXXXXX localstack start

I encounter the following error:

2023-10-03T08:36:04.036 ERROR --- [-functhread2] localstack.utils.bootstrap : Error while starting LocalStack container: Docker process returned with errorcode 125
docker: Error response from daemon: Ports are not available: exposing port TCP 127.0.0.1:443 -> 0.0.0.0:0: not allowed as current user.
You can enable privileged port mapping from Docker -> Settings... -> Advanced -> Enable privileged port mapping.
time="2023-10-03T08:36:04+01:00" level=error msg="error waiting for container: "

Privileged port mapping IS enabled in my Docker Desktop instance.

If I attempt to start the stopped container in the DD UI, it throws the same error, however if I used the re-start button, it appears to run fine:

2023-10-03 08:50:05 
2023-10-03 08:50:05 LocalStack version: 2.3.2.dev20231002131204
2023-10-03 08:50:05 LocalStack Docker container id: 614b760542cf
2023-10-03 08:50:05 LocalStack build date: 2023-10-02
2023-10-03 08:50:05 LocalStack build git hash: 8c4811f
2023-10-03 08:50:05 
2023-10-03 08:50:06 2023-10-03T07:50:06.663  INFO --- [  MainThread] l.bootstrap.licensingv2    : Successfully requested and activated new license XXXXXXXXXXXXXXXXXXXXXXX:Pro 🔑✅
2023-10-03 08:50:08 2023-10-03T07:50:08.598  INFO --- [  MainThread] l.extensions.platform      : loaded 0 extensions
2023-10-03 08:50:08 2023-10-03T07:50:08.651  INFO --- [  MainThread] botocore.credentials       : Found credentials in environment variables.
2023-10-03 08:50:08 2023-10-03T07:50:08.676  INFO --- [-functhread4] hypercorn.error            : Running on https://0.0.0.0:4566 (CTRL + C to quit)
2023-10-03 08:50:08 2023-10-03T07:50:08.676  INFO --- [-functhread4] hypercorn.error            : Running on https://0.0.0.0:4566 (CTRL + C to quit)
2023-10-03 08:50:08 2023-10-03T07:50:08.676  INFO --- [-functhread4] hypercorn.error            : Running on https://0.0.0.0:443 (CTRL + C to quit)
2023-10-03 08:50:08 2023-10-03T07:50:08.676  INFO --- [-functhread4] hypercorn.error            : Running on https://0.0.0.0:443 (CTRL + C to quit)
2023-10-03 08:50:08 Ready.

…but the localstack webapp cannot connect

Could not connect to running LocalStack instance. Make sure LocalStack is running and that its endpoint is accessible from this browser.
Update the endpoint URL in the above Settings if you are running LocalStack on a non-standard port or on a remote host.

Any interactions with the localstack instance fail with the following error:

Could not connect to the endpoint URL: "http://localhost:4566/..."

This includes scripts, apps, etc which reliably work against the community version.

I can switch back to the community version and start it with the CLI and it runs as expected.

Is anybody able to guide me as to where I might be going wrong? I was also wondering if this may be a bug in Docker ?

Will paste full DEBUG output below.

Docker version 24.0.6
Docker Desktop version 4.23.0

PJ_Simpson · October 3, 2023, 8:39am

DEBUG=1 LOCALSTACK_API_KEY=XXXXXXX localstack start


     __                     _______ __             __
    / /   ____  _________ _/ / ___// /_____ ______/ /__
   / /   / __ \/ ___/ __ `/ /\__ \/ __/ __ `/ ___/ //_/
  / /___/ /_/ / /__/ /_/ / /___/ / /_/ /_/ / /__/ ,<
 /_____/\____/\___/\__,_/_//____/\__/\__,_/\___/_/|_|

 💻 LocalStack CLI 2.2.0

[08:59:44] starting LocalStack in Docker mode 🐳                                                                                                                         localstack.py:409
───────────────────────────────────────────────────────────────────── LocalStack Runtime Log (press CTRL-C to quit) ──────────────────────────────────────────────────────────────────────
2023-10-03T08:59:44.091 DEBUG --- [  MainThread] stevedore.extension        : found extension EntryPoint(name='activate_pro_key_on_host', value='localstack_ext.plugins:activate_pro_key_on_host', group='localstack.hooks.prepare_host')
2023-10-03T08:59:44.092 DEBUG --- [  MainThread] stevedore.extension        : found extension EntryPoint(name='configure_extensions_dev_host', value='localstack_ext.extensions.plugins:configure_extensions_dev_host', group='localstack.hooks.prepare_host')
2023-10-03T08:59:44.093 DEBUG --- [  MainThread] stevedore.extension        : found extension EntryPoint(name='patch_community_pro_detection', value='localstack_ext.plugins:patch_community_pro_detection', group='localstack.hooks.prepare_host')
2023-10-03T08:59:44.093 DEBUG --- [  MainThread] stevedore.extension        : found extension EntryPoint(name='start_ec2_daemon', value='localstack_ext.plugins:start_ec2_daemon', group='localstack.hooks.prepare_host')
2023-10-03T08:59:44.093 DEBUG --- [  MainThread] stevedore.extension        : found extension EntryPoint(name='prepare_host_machine_id', value='localstack.utils.analytics.metadata:prepare_host_machine_id', group='localstack.hooks.prepare_host')
2023-10-03T08:59:44.093 DEBUG --- [  MainThread] plugin.manager             : instantiating plugin PluginSpec(localstack.hooks.prepare_host.activate_pro_key_on_host = <function activate_pro_key_on_host at 0x103232320>)
2023-10-03T08:59:44.093 DEBUG --- [  MainThread] plugin.manager             : loading plugin localstack.hooks.prepare_host:activate_pro_key_on_host
2023-10-03T08:59:44.093 DEBUG --- [  MainThread] plugin.manager             : instantiating plugin PluginSpec(localstack.hooks.prepare_host.configure_extensions_dev_host = <function configure_extensions_dev_host at 0x1032331c0>)
2023-10-03T08:59:44.093 DEBUG --- [  MainThread] plugin.manager             : plugin localstack.hooks.prepare_host:configure_extensions_dev_host is disabled
2023-10-03T08:59:44.093 DEBUG --- [  MainThread] plugin.manager             : instantiating plugin PluginSpec(localstack.hooks.prepare_host.patch_community_pro_detection = <function patch_community_pro_detection at 0x103232200>)
2023-10-03T08:59:44.093 DEBUG --- [  MainThread] plugin.manager             : loading plugin localstack.hooks.prepare_host:patch_community_pro_detection
2023-10-03T08:59:44.093 DEBUG --- [  MainThread] plugin.manager             : instantiating plugin PluginSpec(localstack.hooks.prepare_host.start_ec2_daemon = <function start_ec2_daemon at 0x103232560>)
2023-10-03T08:59:44.093 DEBUG --- [  MainThread] plugin.manager             : loading plugin localstack.hooks.prepare_host:start_ec2_daemon
2023-10-03T08:59:44.093 DEBUG --- [  MainThread] plugin.manager             : instantiating plugin PluginSpec(localstack.hooks.prepare_host.prepare_host_machine_id = <function prepare_host_machine_id at 0x101f73be0>)
2023-10-03T08:59:44.093 DEBUG --- [  MainThread] plugin.manager             : loading plugin localstack.hooks.prepare_host:prepare_host_machine_id
2023-10-03T08:59:44.643  INFO --- [  MainThread] l.bootstrap.licensing      : Successfully activated API key
2023-10-03T08:59:44.643 DEBUG --- [  MainThread] localstack_ext.plugins     : pro activation done
2023-10-03T08:59:44.643  INFO --- [  MainThread] localstack.utils.bootstrap : Execution of "prepare_host" took 553.33ms
2023-10-03T08:59:44.643 DEBUG --- [  MainThread] localstack.utils.run       : Executing command: ['docker', 'ps']
2023-10-03T08:59:44.781 DEBUG --- [  MainThread] localstack.utils.run       : Executing command: ['docker', 'ps', '--format', '{{json . }}']
2023-10-03T08:59:44.825 DEBUG --- [  MainThread] stevedore.extension        : found extension EntryPoint(name='configure_extensions_dev_container', value='localstack_ext.extensions.plugins:configure_extensions_dev_container', group='localstack.hooks.configure_localstack_container')
2023-10-03T08:59:44.826 DEBUG --- [  MainThread] stevedore.extension        : found extension EntryPoint(name='configure_pro_container', value='localstack_ext.plugins:configure_pro_container', group='localstack.hooks.configure_localstack_container')
2023-10-03T08:59:44.826 DEBUG --- [  MainThread] stevedore.extension        : found extension EntryPoint(name='_mount_machine_file', value='localstack.utils.analytics.metadata:_mount_machine_file', group='localstack.hooks.configure_localstack_container')
2023-10-03T08:59:44.826 DEBUG --- [  MainThread] stevedore.extension        : found extension EntryPoint(name='configure_edge_port', value='localstack.plugins:configure_edge_port', group='localstack.hooks.configure_localstack_container')
2023-10-03T08:59:44.826 DEBUG --- [  MainThread] plugin.manager             : instantiating plugin PluginSpec(localstack.hooks.configure_localstack_container.configure_extensions_dev_container = <function configure_extensions_dev_container at 0x1032330a0>)
2023-10-03T08:59:44.826 DEBUG --- [  MainThread] plugin.manager             : plugin localstack.hooks.configure_localstack_container:configure_extensions_dev_container is disabled
2023-10-03T08:59:44.826 DEBUG --- [  MainThread] plugin.manager             : instantiating plugin PluginSpec(localstack.hooks.configure_localstack_container.configure_pro_container = <function configure_pro_container at 0x103232680>)
2023-10-03T08:59:44.826 DEBUG --- [  MainThread] plugin.manager             : loading plugin localstack.hooks.configure_localstack_container:configure_pro_container
2023-10-03T08:59:44.826 DEBUG --- [  MainThread] plugin.manager             : instantiating plugin PluginSpec(localstack.hooks.configure_localstack_container._mount_machine_file = <function _mount_machine_file at 0x101f73d00>)
2023-10-03T08:59:44.826 DEBUG --- [  MainThread] plugin.manager             : loading plugin localstack.hooks.configure_localstack_container:_mount_machine_file
2023-10-03T08:59:44.826 DEBUG --- [  MainThread] plugin.manager             : instantiating plugin PluginSpec(localstack.hooks.configure_localstack_container.configure_edge_port = <function configure_edge_port at 0x103233760>)
2023-10-03T08:59:44.826 DEBUG --- [  MainThread] plugin.manager             : loading plugin localstack.hooks.configure_localstack_container:configure_edge_port
2023-10-03T08:59:44.827 DEBUG --- [  MainThread] l.u.c.docker_cmd_client    : Run container with cmd: ['docker', 'run', '--rm', '--entrypoint', 'sh', '-p', '53:53', '-p', '53:53/udp', 'localstack/localstack', '-c', 'echo test123']
2023-10-03T08:59:44.827 DEBUG --- [  MainThread] localstack.utils.run       : Executing command: ['docker', 'run', '--rm', '--entrypoint', 'sh', '-p', '53:53', '-p', '53:53/udp', 'localstack/localstack', '-c', 'echo test123']
2023-10-03T08:59:46.354 DEBUG --- [  MainThread] localstack.plugins         : configuring container with edge ports: [443, 4566]
2023-10-03T08:59:46.359 DEBUG --- [  MainThread] localstack.utils.run       : Executing command: ['which', 'tail']
2023-10-03T08:59:46.372 DEBUG --- [-functhread1] localstack.utils.run       : Executing command: ['tail', '-f', '/var/folders/82/npr22gmd18x9kqvh2zvg17rm0000gn/T/localstack-cli/localstack_main_container.log']
2023-10-03T08:59:46.377 DEBUG --- [-functhread2] localstack.utils.run       : Executing command: ['docker', 'ps', '--format', '{{json . }}']
2023-10-03T08:59:46.449 DEBUG --- [-functhread2] l.u.c.docker_cmd_client    : Run container with cmd: ['docker', 'run', '--rm', '--name', 'localstack_main', '--privileged', '-v', '/Users/petersimpson/.kube/config:/root/.kube/config', '-v', '/Users/petersimpson/Library/Caches/localstack-cli/machine.json:/var/lib/localstack/cache/machine.json', '-v', '/Users/petersimpson/Library/Caches/localstack/volume:/var/lib/localstack', '-v', '/var/run/docker.sock:/var/run/docker.sock', '-p', '127.0.0.1:443:443', '-p', '127.0.0.1:4566:4566', '-p', '127.0.0.1:4510-4559:4510-4559', '-e', 'DEBUG=1', '-e', 'LOCALSTACK_API_KEY=XXXXXXX', '-e', 'ACTIVATE_PRO=1', '-e', 'LOCALSTACK_CLI=1', '-e', 'DOCKER_HOST=unix:///var/run/docker.sock', '-e', 'SET_TERM_HANDLER=1', '-p', '0.0.0.0:53:53', '-p', '0.0.0.0:53:53/udp', 'localstack/localstack-pro']
2023-10-03T08:59:46.450 DEBUG --- [-functhread2] localstack.utils.run       : Executing command: ['docker', 'run', '--rm', '--name', 'localstack_main', '--privileged', '-v', '/Users/petersimpson/.kube/config:/root/.kube/config', '-v', '/Users/petersimpson/Library/Caches/localstack-cli/machine.json:/var/lib/localstack/cache/machine.json', '-v', '/Users/petersimpson/Library/Caches/localstack/volume:/var/lib/localstack', '-v', '/var/run/docker.sock:/var/run/docker.sock', '-p', '127.0.0.1:443:443', '-p', '127.0.0.1:4566:4566', '-p', '127.0.0.1:4510-4559:4510-4559', '-e', 'DEBUG=1', '-e', 'LOCALSTACK_API_KEY=XXXXXX', '-e', 'ACTIVATE_PRO=1', '-e', 'LOCALSTACK_CLI=1', '-e', 'DOCKER_HOST=unix:///var/run/docker.sock', '-e', 'SET_TERM_HANDLER=1', '-p', '0.0.0.0:53:53', '-p', '0.0.0.0:53:53/udp', 'localstack/localstack-pro']
2023-10-03T08:59:46.559 ERROR --- [-functhread2] localstack.utils.bootstrap : Error while starting LocalStack container
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/localstack/utils/container_utils/docker_cmd_client.py", line 680, in _run_async_cmd
    raise subprocess.CalledProcessError(
subprocess.CalledProcessError: Command '['docker', 'run', '--rm', '--name', 'localstack_main', '--privileged', '-v', '/Users/petersimpson/.kube/config:/root/.kube/config', '-v', '/Users/petersimpson/Library/Caches/localstack-cli/machine.json:/var/lib/localstack/cache/machine.json', '-v', '/Users/petersimpson/Library/Caches/localstack/volume:/var/lib/localstack', '-v', '/var/run/docker.sock:/var/run/docker.sock', '-p', '127.0.0.1:443:443', '-p', '127.0.0.1:4566:4566', '-p', '127.0.0.1:4510-4559:4510-4559', '-e', 'DEBUG=1', '-e', 'LOCALSTACK_API_KEY=XXXXXX', '-e', 'ACTIVATE_PRO=1', '-e', 'LOCALSTACK_CLI=1', '-e', 'DOCKER_HOST=unix:///var/run/docker.sock', '-e', 'SET_TERM_HANDLER=1', '-p', '0.0.0.0:53:53', '-p', '0.0.0.0:53:53/udp', 'localstack/localstack-pro']' returned non-zero exit status 125.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/localstack/utils/bootstrap.py", line 416, in run
    return DOCKER_CLIENT.run_container(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/localstack/utils/container_utils/docker_cmd_client.py", line 605, in run_container
    return self._run_async_cmd(cmd, stdin, kwargs.get("name") or "", image_name)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/localstack/utils/container_utils/docker_cmd_client.py", line 696, in _run_async_cmd
    raise ContainerException(
localstack.utils.container_utils.container_client.ContainerException: ('Docker process returned with errorcode 125', None, b'docker: Error response from daemon: Ports are not available: exposing port TCP 127.0.0.1:443 -> 0.0.0.0:0: not allowed as current user.\nYou can enable privileged port mapping from Docker -> Settings... -> Advanced -> Enable privileged port mapping.\ntime="2023-10-03T08:59:46+01:00" level=error msg="error waiting for container: "\n')
Error: Docker process returned with errorcode 125

dfangl · October 3, 2023, 9:49am

Hello!
It seems your docker daemon still thinks privileged port mapping is disabled, can you please double check and also restart Docker Desktop for good measure?

What happens if you run docker run --rm -p 443:443 alpine echo test for example, does this work?

Also, I would suggest updating to CLI version 2.3.1, since we fixed some unnecessary flags there.

As last workaround from our side, you can set GATEWAY_LISTEN=:4566. While this should prevent LS from trying to expose port 443 to the host, it is more a fix for the symptoms, rather than the issue (which seems to be docker denying privileged port forwardings).

PJ_Simpson · October 3, 2023, 10:52am

Hi @dfangl , thanks so much for looking into this so promptly.

I can confirm that workaround is good, and I am able to interact with Pro features, so thanks again for that!

I can get docker to map those ports outside of the LocalStack process:

docker run --rm -p 443:443 alpine echo test
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
579b34f0a95b: Pull complete 
Digest: sha256:eece025e432126ce23f223450a0326fbebde39cdf496a85d8c016293fc851978
Status: Downloaded newer image for alpine:latest
test

Upgrading the CLI hasn’t helped (I also attempted this before and after a restart of Docker Desktop):

 💻 LocalStack CLI 2.3.1

[11:38:19] starting LocalStack in Docker mode 🐳                                                                                                                         localstack.py:495
2023-10-03T11:38:19.173  INFO --- [  MainThread] l.bootstrap.licensingv2    : Successfully activated cached license cf5c1e60-8872-4dfb-9488-b78ccc894a03:Pro from /Users/petersimpson/Library/Caches/localstack-cli/license.json 🔑✅
⠋ Starting LocalStack container2023-10-03T11:38:19.944 ERROR --- [-functhread1] localstack.utils.bootstrap : Error while starting LocalStack container: Docker process returned with errorcode 1
Error response from daemon: Ports are not available: exposing port TCP 127.0.0.1:443 -> 0.0.0.0:0: failed to connect to /var/run/com.docker.vmnetd.sock: is vmnetd running?: dial unix /var/run/com.docker.vmnetd.sock: connect: no such file or directory
Error: failed to start containers: 8725af3b0886e16b704fbca5f57b4f5f4c1d522a848ae1037df26c5157fdc8bf

❌ Error: Docker process returned with errorcode 1

If it helps at all with replication:

>> sw_vers
   
ProductName:	macOS
ProductVersion:	12.0.1

>> uname -a 

Darwin xxxxx-Air 21.1.0 Darwin Kernel Version 21.1.0: Wed Oct 13 17:33:24 PDT 2021; root:xnu-8019.41.5~1/RELEASE_ARM64_T8101 arm64

dfangl · October 3, 2023, 11:31am

This error message now differs from the one above. There was some issue with Docker Desktop < 4.16.2, described here: Error response from daemon: is vmnetd running? · Issue #6677 · docker/for-mac · GitHub Although it seems some people with later versions also experience it.

Can you check your docker desktop version, and also check out the fixes described at the end of the linked issue? (Specifically this one: Error response from daemon: is vmnetd running? · Issue #6677 · docker/for-mac · GitHub)

PJ_Simpson · October 3, 2023, 12:39pm

@dfangl perfect, that’s done the trick!

FWIW: I am on the latest version of DD (4.23.0)

sudo /Applications/Docker.app/Contents/MacOS/install vmnetd

… and after that the command to start LocalStack Pro version succeeds as expected.

Topic		Replies	Views
LocalStack Desktop indicates Endpoint unreachable Platform	4	494	November 21, 2023
Docker compose pro `is vmnetd running?` with Docker Desktop version 4.23.0 {used localstack start for now} Platform	3	1313	October 2, 2023
Run LocalStack as non-root in open shift - Permission denied Platform localstack	1	966	September 28, 2022
Remote docker host. How to? Platform localstack	1	552	May 22, 2023
Separating LocalStack Community and Pro containers Announcement localstack , announcement , docker-image	0	1043	March 14, 2023

Localstack start command returning 'Enable privileged port mapping' error

Related Topics