Execution environment for function failed during startup, check for errors during the startup of your Lambda function

[dvanteru]

Hi There,

Any help is highly appreciated, I have been struggling with the below error for last 3 days and tried all the configurations including zip type and hotreloading of lamda code but of no use. It would be great if you can let me know which configuration is missing. When the function is created it doesn’t have any code in the folder “/var/runtime”. What is the configuration that I have to use? Is there any system variable that is available to override where to look for code files when a function is created.

l.s.l.i.executor_endpoint : Execution environment startup failed: {"errorType":"Runtime.ImportModuleError","errorMessage":"Error: Cannot find module 'index'\nRequire stack:\n- /var/runtime/index.mjs","trace":["Runtime.ImportModuleError: Error: Cannot find module 'index'","Require stack:","- /var/runtime/index.mjs"," at _loadUserApp (file:///var/runtime/index.mjs:1087:17)"," at async UserFunction.js.module.exports.load (file:///var/runtime/index.mjs:1119:21)"," at async start (file:///var/runtime/index.mjs:1282:23)"," at async file:///var/runtime/index.mjs:1288:1"]}

awslocal batch script:

awslocal  lambda create-function \
  --code S3Bucket='hot-reload',S3Key='/usr/lambda' \
  --function-name abc-function \
  --handler app/index.handler \
  --role "arn:aws:iam::000000000000:role/DummyRole" \
  --runtime nodejs20.x \
  --environment "Variables={LOG_LEVEL=debug,UPDATE_REQUEST_QUEUE_URL=${UPDATE_REQUEST_QUEUE_URL},MESSAGE_GROUPS=2}"
log "Created Lambda function"

Docker compose:

version: "3.0"

networks:
  test:

services:
  localstack:
    image: localstack/localstack:3.4.0
    networks:
      - test
    environment:
      - SERVICES=sns,sqs,kms,events,lambda,s3,iam
      - LOCALSTACK_HOST=localstack
      - AWS_ACCESS_KEY_ID=test
      - AWS_SECRET_ACCESS_KEY=test
      - AWS_REGION=eu-west-2
      - AWS_DEFAULT_REGION=eu-west-2
      - DEFAULT_REGION=eu-west-2
      - SKIP_SSL_CERT_DOWNLOAD=1
      - LOCALSTACK_KMS_BACKEND=moto
      - LAMBDA_RUNTIME_ENVIRONMENT_TIMEOUT=2000
      - LAMBDA_KEEPALIVE_MS=6000000
      - LAMBDA_EXECUTOR=docker-reuse
      - DEBUG=0
      - UPDATE_REQUEST_QUEUE_URL=http://localstack:4566/000000000000/test_update_request_queue.fifo
      - UPDATE_REQUEST_SNS_TOPIC_ARN=arn:aws:sns:eu-west-2:000000000000:test_update_request_queue.fifo
      - MESSAGE_GROUPS=2
    volumes:
      - ./test/integration/localstackinit.d/init:/etc/localstack/init/boot.d/
      - ./test/integration/localstackinit.d/ready/:/etc/localstack/init/ready.d/
      - ./sd_refresh_docs.zip://home/aws/functions/sd_refresh_docs.zip
      - ./:/usr/lambda/
      - "/var/run/docker.sock:/var/run/docker.sock"
    healthcheck:
      test: [ "CMD", "curl", "http://localhost.localstack.cloud:4566/_localstack/health" ]
      interval: 10s
      timeout: 5s
      retries: 10
    expose:
      - 4566
      - 4575
      - 9915
    ports:
      - 4566:4566
      - 4575:4575
      - 9915:9915
      - 4571:4572
      - 8090:8080

Below is the structure of lambda code

All the code files are in app folder including index.js

Please suggest if I am missing anything.

Thank you

[HarshCasper]

Hello — Could you please use Discourse Markdown to format the commands, Docker Compose configurations, and other details in code syntax for easier readability? Here is a helpful reference: https://markdown-it.github.io/

Additionally, could you share your Lambda code with us? This will assist in debugging the setup. Alternatively, you may place all relevant files in a public GitHub or GitLab repository for us to review.

[dvanteru]

Formatted the above code and description as suggested

[Marcel]

Hi @dvanteru,

If you are looking to use hot-reload, please be sure to use the correct folder. Please provide the path to the folder on your host machine.

I would suggest you to test it out firstly with the sample provided in Hot Reloading | Docs (localstack.cloud).

[dvanteru]

Hi There,

Code exists in the volume mount that docker container is running from, and the s3 refers to the path below

[Marcel]

Thanks @dvanteru,

As previously stated, you must use the host machine path. Do not use the mount path in the LocalStack container.

[dvanteru]

Thank you Marcel,

if you look at the code above, docker-compose contains a service called localstack and this has the mount path /usr/lambda where the code for lambda exists, for lamda container the host will become the docker container i.e., “localstack_1” etc.

CONTAINER ID   IMAGE                         COMMAND                  CREATED        STATUS                  PORTS                                                                                                                                             NAMES
6cc599514167   localstack/localstack:3.4.0   "docker-entrypoint.sh"   22 hours ago   Up 22 hours (healthy)   0.0.0.0:4566->4566/tcp, 4510-4559/tcp, 0.0.0.0:4575->4575/tcp, 5678/tcp, 0.0.0.0:9915->9915/tcp, 0.0.0.0:4571->4572/tcp, 0.0.0.0:8090->8080/tcp   sd-refresh-docs-localstack-1
27f4f8ec3570   nginx:latest                  "/docker-entrypoint.…"   22 hours ago   Up 22 hours             80/tcp, 0.0.0.0:8082->8082/tcp

Which has the following mount paths as below

[
    {
        "Type": "bind",
        "Source": "/Users/abc.xyz/projects/abc/components/sd-refresh-docs/test/integration/localstackinit.d/ready",
        "Destination": "/etc/localstack/init/ready.d",
        "Mode": "rw",
        "RW": true,
        "Propagation": "rprivate"
    },
    {
        "Type": "bind",
        "Source": "/Users/abc.xyz/projects/abc/components/sd-refresh-docs",
        "Destination": "/usr/lambda",
        "Mode": "rw",
        "RW": true,
        "Propagation": "rprivate"
    },
    {
        "Type": "bind",
        "Source": "/var/run/docker.sock",
        "Destination": "/var/run/docker.sock",
        "Mode": "rw",
        "RW": true,
        "Propagation": "rprivate"
    },
    {
        "Type": "bind",
        "Source": "/Users/abc.xyz/projects/abc/components/sd-refresh-docs/volume",
        "Destination": "/var/lib/localstack",
        "Mode": "rw",
        "RW": true,
        "Propagation": "rprivate"
    },
    {
        "Type": "bind",
        "Source": "/Users/abc.xyz/projects/abc/components/sd-refresh-docs/test/integration/localstackinit.d/init",
        "Destination": "/etc/localstack/init/boot.d",
        "Mode": "rw",
        "RW": true,
        "Propagation": "rprivate"
    }
]

Now the code to create the lambda is as below:

awslocal  lambda create-function \
  --code S3Bucket='hot-reload',S3Key='/usr/lambda/' \
  --function-name sd-refresh-docs-function \
  --handler app/index.handler \
  --role "arn:aws:iam::000000000000:role/DummyRole" \
  --runtime nodejs18.x \
  --environment "Variables={LOG_LEVEL=debug,UPDATE_REQUEST_QUEUE_URL=${UPDATE_REQUEST_QUEUE_URL},MESSAGE_GROUPS=2}"
log "Created Lambda function"

The error that I am receiving when the lamda is triggered and when it tries to create the container is as follows:

2024-05-16 09:05:31 2024-05-16T08:05:31.241  WARN --- [   asgi_gw_1] l.s.l.i.executor_endpoint  : Execution environment startup failed: {"errorType":"Runtime.ImportModuleError","errorMessage":"Error: Cannot find module 'index'\nRequire stack:\n- /var/runtime/index.mjs","trace":["Runtime.ImportModuleError: Error: Cannot find module 'index'","Require stack:","- /var/runtime/index.mjs","    at _loadUserApp (file:///var/runtime/index.mjs:1087:17)","    at async UserFunction.js.module.exports.load (file:///var/runtime/index.mjs:1119:21)","    at async start (file:///var/runtime/index.mjs:1282:23)","    at async file:///var/runtime/index.mjs:1288:1"]}
2024-05-16 09:05:31 2024-05-16T08:05:31.241  INFO --- [   asgi_gw_1] localstack.request.http    : POST /_localstack_lambda/ec18ef4a96c126382ee1160eb4dbf6ab/status/ec18ef4a96c126382ee1160eb4dbf6ab/error => 202
2024-05-16 09:05:31 2024-05-16T08:05:31.241  WARN --- [on:$LATEST_0] l.s.l.i.execution_environm : Failed to start execution environment ec18ef4a96c126382ee1160eb4dbf6ab: Environment startup failed
2024-05-16 09:05:31 2024-05-16T08:05:31.241  WARN --- [on:$LATEST_0] l.s.l.i.execution_environm : Execution environment ec18ef4a96c126382ee1160eb4dbf6ab for function arn:aws:lambda:eu-west-2:000000000000:function:sd-refresh-docs-function:$LATEST failed during startup. Check for errors during the startup of your Lambda function.

My question is how do we intercept the lambda function container to pass the correct location of code instead of referring to “/var/runtime”

Thank you

[Marcel]

Here is an example of the commands from a Makefile
(used the sample from localstack-pro-samples/lambda-hot-reloading/typescript (github.com))

Please be aware that the S3Key path needs to be absolute path from your host machine.

deploy:
		awslocal lambda create-function \
			--function-name hello-world-hot \
			--runtime "nodejs16.x" \
			--role arn:aws:iam::123456789012:role/lambda-ex \
			--code S3Bucket="hot-reload",S3Key="D:\LocalStack_Support\lambda_hot_reload\dist" \
			--handler index.handler

invoke:
		awslocal lambda invoke \
			--function-name hello-world-hot \
			--payload '{"action": "test"}' output.txt

get:
		awslocal lambda get-function --function-name hello-world-hot

[dvanteru]

Hi Marcel,

Thank you, I understand what you say but my context is different.

I am trying to create lamda within the docker container and trying to raise an event so it triggers lambda which is working fine, but after the function is triggered localstack create’s a new container which fails to map to the function code location.

Is there any code sample that demonstrates the following with nodejs.

A docker compose file with localstack with services like sns,lambda,events and then post an event into the queue which is subscribed by lambda. Code for functio app is using nodejs and the index.js is in a folder named “app”.

[dfangl]

Hi @dvanteru !

I understand that you mount your lambda code directory into the localstack container using this line in your docker compose configuration:

      - ./:/usr/lambda/

However, lambda will start a new container for lambda execution, where the code is mapped. Since bind mounts in docker are always pointing to the host (on the host side), and not inside another container, you absolutely have to use the host path instead of /usr/lambda in your S3Key. So, basically, the absolute path of the project in your last screenshot.

If you replace /usr/lambda in the S3Key with the /Users/abc.xyz/projects/abc/components/sd-refresh-docs on the host, it will work. Again, the mount into the LocalStack container in your docker compose does not matter here - bind mounts are always from the host into a container, never from container into container.

You do not need the /usr/lambda mount into the localstack container at all, the LocalStack container itself will never try to read the code for hot-reloading, the path will just be mounted from the host into the lambda container.

I unfortunately do not have an example ready for you with the described stack, but I would advise you to take another look at the pro samples regarding hot-reloading: localstack-pro-samples/lambda-hot-reloading at master · localstack-samples/localstack-pro-samples · GitHub and the hot reloading documentation mentioned in this thread.

Also, the concrete stack does not matter for the hot-reloading, it can be tested in isolation as well, at least if it correctly loads the code (please try to manually invoke it - if the payload is not correct it is still fine, the error message should at least show the code being loaded correctly).