Lambda with Hot Reload functionality is erroring when triggered

YeeP · October 10, 2023, 12:29am

I believe the hot reload functionality is working as localstack attempts to run “something” every time I update the file. My guess right now is the file might need to be zipped, but the example in the docs for hot reloading does show an index.js file being used. Hot Reloading | Docs

When I update the lambda, this seems to be the error.
("b'Mounts denied: \nThe path /lambda-builder/dist is not shared from the host and is not known to Docker.

The volume in question is here:
- "${PWD}/lambda-builder/container-volumes/shared-files/dist:/lambda-builder/dist"

PWD is:
/Users/myusername/Projects/localstack_poc/localstack/poc_function

So as far as I can tell I should be good on the file sources.

This is what the function is registered as inside the container:

# awslocal lambda get-function --function-name lambdaHandler
{
    "Configuration": {
        "FunctionName": "lambdaHandler",
        "FunctionArn": "arn:aws:lambda:us-east-1:000000000000:function:lambdaHandler",
        "Runtime": "nodejs18.x",
        "Role": "arn:aws:iam::000000000000:role/lambda-role",
        "Handler": "index.handler",
        "CodeSize": 0,
        "Description": "",
        "Timeout": 3,
        "MemorySize": 128,
        "LastModified": "2023-10-09T19:44:34.098167+0000",
        "CodeSha256": "hot-reloading-hash-not-available",
        "Version": "$LATEST",
        "TracingConfig": {
            "Mode": "PassThrough"
        },
        "RevisionId": "2da2ba8f-f76b-436c-a8e1-95b628fba638",
        "State": "Active",
        "LastUpdateStatus": "Successful",
        "PackageType": "Zip",
        "Architectures": [
            "x86_64"
        ],
        "EphemeralStorage": {
            "Size": 512
        },
        "SnapStart": {
            "ApplyOn": "None",
            "OptimizationStatus": "Off"
        },
        "RuntimeVersionConfig": {
            "RuntimeVersionArn": "arn:aws:lambda:us-east-1::runtime:8eeff65f6809a3ce81507fe733fe09b835899b99481ba22fd75b5a7338290ec1"
        }
    },
    "Code": {
        "RepositoryType": "S3",
        "Location": "Code location: /lambda-builder/dist"
    }
}
# cd /lambda-builder/dist
# ls -la
total 12
drwxr-xr-x 4 root root  128 Oct  9 19:44 .
drwxr-xr-x 3 root root 4096 Oct  9 14:41 ..
-rw-r--r-- 1 root root  800 Oct  9 19:44 index.js
-rw-r--r-- 1 root root 1055 Oct  9 19:44 index.js.map

If I update the function, I can see the updates in the above files (this is from inside the localstack container).

When the lambda is triggered, this is the entire error:

localstack_main  | 2023-10-09T19:44:35.050 DEBUG --- [   asgi_gw_1] l.services.sqs.provider    : creating queue key=lambdaHandler-64cba9cd7ae85d64c545cc090f5b3500 attributes=None tags=None
localstack_main  | 2023-10-09T19:44:43.851  INFO --- [   asgi_gw_1] localstack.request.aws     : AWS sqs.SendMessage => 200
localstack_main  | 2023-10-09T19:44:44.500 DEBUG --- [   asgi_gw_2] l.services.sqs.models      : de-queued message SqsMessage(id=baf726ac-7dba-48a6-aca1-7ea660ee0bea,group=None) from arn:aws:sqs:us-east-1:000000000000:sqsQueue
localstack_main  | 2023-10-09T19:44:44.504 DEBUG --- [functhread16] l.s.l.e.sqs_event_source_l : Sending event from event source arn:aws:sqs:us-east-1:000000000000:sqsQueue to Lambda arn:aws:lambda:us-east-1:000000000000:function:lambdaHandler
localstack_main  | 2023-10-09T19:44:44.505 DEBUG --- [    Thread-5] l.s.l.i.assignment         : Starting new environment
localstack_main  | 2023-10-09T19:44:44.505 DEBUG --- [    Thread-5] l.s.l.i.docker_runtime_exe : Creating service endpoint for function arn:aws:lambda:us-east-1:000000000000:function:lambdaHandler:$LATEST executor 3e947b2a52a95384927c2249ceb8abc3
localstack_main  | 2023-10-09T19:44:44.506 DEBUG --- [    Thread-5] l.s.l.i.docker_runtime_exe : Finished creating service endpoint for function arn:aws:lambda:us-east-1:000000000000:function:lambdaHandler:$LATEST executor 3e947b2a52a95384927c2249ceb8abc3
localstack_main  | 2023-10-09T19:44:44.506 DEBUG --- [    Thread-5] l.s.l.i.docker_runtime_exe : Assigning container name of localstack-main-lambda-lambdahandler-3e947b2a52a95384927c2249ceb8abc3 to executor 3e947b2a52a95384927c2249ceb8abc3
localstack_main  | 2023-10-09T19:44:44.524 DEBUG --- [    Thread-5] l.u.c.container_client     : Getting networks for container: localstack_main
localstack_main  | 2023-10-09T19:44:44.532  INFO --- [    Thread-5] l.u.container_networking   : Determined main container network: poc_function_default
localstack_main  | 2023-10-09T19:44:44.532 DEBUG --- [    Thread-5] l.u.c.container_client     : Getting ipv4 address for container localstack_main in network poc_function_default.
localstack_main  | 2023-10-09T19:44:44.540  INFO --- [    Thread-5] l.u.container_networking   : Determined main container target IP: 172.18.0.3
localstack_main  | 2023-10-09T19:44:44.543 DEBUG --- [    Thread-5] plugin.manager             : no extensions found in namespace localstack.hooks.lambda_start_docker_executor
localstack_main  | 2023-10-09T19:44:44.543 DEBUG --- [    Thread-5] l.u.c.docker_sdk_client    : Creating container with attributes: {'self': <localstack.utils.container_utils.docker_sdk_client.SdkDockerClient object at 0xffffb6bd1cd0>, 'image_name': 'public.ecr.aws/lambda/nodejs:18', 'name': 'localstack-main-lambda-lambdahandler-3e947b2a52a95384927c2249ceb8abc3', 'entrypoint': '/var/rapid/init', 'remove': False, 'interactive': False, 'tty': False, 'detach': False, 'command': None, 'mount_volumes': [VolumeBind(host_dir='/lambda-builder/dist', container_dir='/var/task', read_only=True)], 'ports': <PortMappings: {}>, 'exposed_ports': [], 'env_vars': {'AWS_DEFAULT_REGION': 'us-east-1', 'AWS_REGION': 'us-east-1', 'AWS_LAMBDA_FUNCTION_NAME': 'lambdaHandler', 'AWS_LAMBDA_FUNCTION_MEMORY_SIZE': 128, 'AWS_LAMBDA_FUNCTION_VERSION': '$LATEST', 'AWS_LAMBDA_INITIALIZATION_TYPE': 'on-demand', 'AWS_LAMBDA_LOG_GROUP_NAME': '/aws/lambda/lambdaHandler', 'AWS_LAMBDA_LOG_STREAM_NAME': '2023/10/09/[$LATEST]3e947b2a52a95384927c2249ceb8abc3', 'AWS_ACCESS_KEY_ID': 'XXXXXXXXXXXXXXX', 'AWS_SECRET_ACCESS_KEY': 'XXXXXXXXXXXXXXXXX', 'AWS_SESSION_TOKEN': 'XXXXXXXXXXXXXXXXX', 'LAMBDA_TASK_ROOT': '/var/task', 'LAMBDA_RUNTIME_DIR': '/var/runtime', 'AWS_XRAY_CONTEXT_MISSING': 'LOG_ERROR', 'AWS_XRAY_DAEMON_ADDRESS': '127.0.0.1:2000', '_AWS_XRAY_DAEMON_PORT': '2000', '_AWS_XRAY_DAEMON_ADDRESS': '127.0.0.1', 'TZ': ':UTC', 'AWS_LAMBDA_FUNCTION_TIMEOUT': 3, 'LOCALSTACK_HOSTNAME': '172.18.0.3', 'EDGE_PORT': '4566', 'AWS_ENDPOINT_URL': 'http://172.18.0.3:4566', 'LOCALSTACK_RUNTIME_ID': '3e947b2a52a95384927c2249ceb8abc3', 'LOCALSTACK_RUNTIME_ENDPOINT': 'http://172.18.0.3:4566/_localstack_lambda/3e947b2a52a95384927c2249ceb8abc3', '_HANDLER': 'index.handler', 'AWS_EXECUTION_ENV': 'Aws_Lambda_nodejs18.x', 'LOCALSTACK_INIT_LOG_LEVEL': 'debug', 'LOCALSTACK_HOT_RELOADING_PATHS': '/var/task'}, 'user': None, 'cap_add': None, 'cap_drop': None, 'security_opt': None, 'network': 'poc_function_default', 'dns': '172.18.0.3', 'additional_flags': '', 'workdir': None, 'privileged': False, 'labels': None, 'platform': 'linux/amd64', 'ulimits': None}
localstack_main  | 2023-10-09T19:44:44.579 DEBUG --- [    Thread-5] localstack.packages.api    : Installation of lambda-runtime skipped (already installed).
localstack_main  | 2023-10-09T19:44:44.579 DEBUG --- [    Thread-5] localstack.packages.api    : Performing runtime setup for already installed package.
localstack_main  | 2023-10-09T19:44:44.580 DEBUG --- [    Thread-5] l.u.c.docker_sdk_client    : Copying file /usr/lib/localstack/lambda-runtime/v0.1.19-pre/arm64/. into localstack-main-lambda-lambdahandler-3e947b2a52a95384927c2249ceb8abc3:/
localstack_main  | 2023-10-09T19:44:44.876 DEBUG --- [    Thread-5] l.u.c.docker_sdk_client    : Starting container localstack-main-lambda-lambdahandler-3e947b2a52a95384927c2249ceb8abc3
localstack_main  | 2023-10-09T19:44:44.883  WARN --- [    Thread-5] l.s.l.i.execution_environm : Failed to start execution environment 3e947b2a52a95384927c2249ceb8abc3: 
localstack_main  | 2023-10-09T19:44:44.883  WARN --- [    Thread-5] l.s.l.i.execution_environm : Execution environment 3e947b2a52a95384927c2249ceb8abc3 for function arn:aws:lambda:us-east-1:000000000000:function:lambdaHandler:$LATEST failed during startup. Check for errors during the startup of your Lambda function.
localstack_main  | 2023-10-09T19:44:44.889 DEBUG --- [    Thread-5] l.s.l.i.execution_environm : Logs from the execution environment 3e947b2a52a95384927c2249ceb8abc3 after startup error:
localstack_main  | [lambda 3e947b2a52a95384927c2249ceb8abc3] 
localstack_main  | 2023-10-09T19:44:44.889 DEBUG --- [    Thread-5] l.u.c.docker_sdk_client    : Stopping container: localstack-main-lambda-lambdahandler-3e947b2a52a95384927c2249ceb8abc3
localstack_main  | 2023-10-09T19:44:44.895 DEBUG --- [    Thread-5] l.u.c.docker_sdk_client    : Removing container: localstack-main-lambda-lambdahandler-3e947b2a52a95384927c2249ceb8abc3
localstack_main  | 2023-10-09T19:44:44.940  INFO --- [    Thread-5] localstack.utils.threads   : Thread run method <function EventSourceAsfAdapter.invoke.<locals>._invoke at 0xffff6f2ea660>(None) failed: Could not start new environment:  Traceback (most recent call last):
localstack_main  |   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/api/client.py", line 268, in _raise_for_status
localstack_main  |     response.raise_for_status()
localstack_main  |   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status
localstack_main  |     raise HTTPError(http_error_msg, response=self)
localstack_main  | requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.43/containers/96a2b6267e8135d2ec4137d98511ef50644a5a9e8fa293e74894bc7bc4a26937/start
localstack_main  | 
localstack_main  | The above exception was the direct cause of the following exception:
localstack_main  | 
localstack_main  | Traceback (most recent call last):
localstack_main  |   File "/opt/code/localstack/localstack/utils/container_utils/docker_sdk_client.py", line 570, in start_container
localstack_main  |     container.start()
localstack_main  |   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/models/containers.py", line 405, in start
localstack_main  |     return self.client.api.start(self.id, **kwargs)
localstack_main  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
localstack_main  |   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/utils/decorators.py", line 19, in wrapped
localstack_main  |     return f(self, resource_id, *args, **kwargs)
localstack_main  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
localstack_main  |   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/api/container.py", line 1126, in start
localstack_main  |     self._raise_for_status(res)
localstack_main  |   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/api/client.py", line 270, in _raise_for_status
localstack_main  |     raise create_api_error_from_http_exception(e) from e
localstack_main  |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
localstack_main  |   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception
localstack_main  |     raise cls(e, response=response, explanation=explanation) from e
localstack_main  | docker.errors.APIError: 500 Server Error for http+docker://localhost/v1.43/containers/96a2b6267e8135d2ec4137d98511ef50644a5a9e8fa293e74894bc7bc4a26937/start: Internal Server Error ("b'Mounts denied: \nThe path /lambda-builder/dist is not shared from the host and is not known to Docker.\nYou can configure shared paths from Docker -> Preferences... -> Resources -> File Sharing.\nSee https://docs.docker.com/desktop/mac for more info.'

Any suggestions would be much appreciated.

Marcel · October 10, 2023, 5:58am

Hi @YeeP,

Please follow the documentation for Hot Reloading | Docs (localstack.cloud),

Deploy via an S3 Bucket. You need to use the magic variable hot-reload as the bucket.

Set the S3 key to the path of the directory your lambda function resides in. The handler is then referenced by the filename of your lambda code and the function in that code that needs to be invoked.

It might be helpful to start with the default sample and work from there.

YeeP · October 10, 2023, 1:22pm

@Marcel I did follow that guide. You can see in the “get-function” response:

"Code": {
        "RepositoryType": "S3",
        "Location": "Code location: /lambda-builder/dist"
    }

But let me be more specific, here is my initialization script:

#!/bin/bash
echo "Create Queue"
aws \
  sqs create-queue \
  --queue-name sqsQueue \
  --endpoint-url http://localhost:4566 \
  --region us-east-1

echo "Create Lambda"
aws \
  lambda create-function \
  --endpoint-url=http://localhost:4566 \
  --function-name lambdaHandler \
  --role arn:aws:iam::000000000000:role/lambda-role \
  --code S3Bucket=hot-reload,S3Key="/lambda-builder/dist" \
  --handler index.handler \
  --runtime nodejs18.x \
  --region us-east-1


echo "Bind Queue to Lambda"
aws \
  lambda create-event-source-mapping \
  --function-name lambdaHandler \
  --batch-size 1 \
  --event-source-arn "arn:aws:sqs:us-east-1:000000000000:sqsQueue" \
  --endpoint-url=http://localhost:4566 \
  --region us-east-1

echo "All resources initialized! 🚀"

You can see that the lambda is setup for hot-reloading:
--code S3Bucket=hot-reload,S3Key="/lambda-builder/dist" \

EDIT: OK I just re-read the guide again. I think the wording is confusing. You referenced the python example, but here is the link to the typescript example:

As you stated, it says in step one says “Deploy via S3 Bucket”. However, I think this is not actually what is happening here. The guides never show an S3 bucket being created. Instead the reference to the S3 bucket where the function would normally be stored is modified to point toward the file system inside the container. If this is incorrect please let me know. As you can see in my init script, I am not creating any S3 buckets.

If it makes it easier to see the whole picture; here is my typescript code I am using for the lambda:

import { Context, APIGatewayProxyResult, APIGatewayEvent } from 'aws-lambda';

export async function index(event: APIGatewayEvent, context: Context): Promise<APIGatewayProxyResult> {
    console.log(`Event: ${JSON.stringify(event, null, 2)}`);
    console.log(`Context: ${JSON.stringify(context, null, 2)}`);
    return {
        statusCode: 200,
        body: JSON.stringify({
            message: 'Hello World!',
        }),
    };
}

Here is the code after esbuild builds it:

var __defProp = Object.defineProperty;
var __getOwnPropDesc = Object.getOwnPropertyDescriptor;
var __getOwnPropNames = Object.getOwnPropertyNames;
var __hasOwnProp = Object.prototype.hasOwnProperty;
var __export = (target, all) => {
  for (var name in all)
    __defProp(target, name, { get: all[name], enumerable: true });
};
var __copyProps = (to, from, except, desc) => {
  if (from && typeof from === "object" || typeof from === "function") {
    for (let key of __getOwnPropNames(from))
      if (!__hasOwnProp.call(to, key) && key !== except)
        __defProp(to, key, { get: () => from[key], enumerable: !(desc = __getOwnPropDesc(from, key)) || desc.enumerable });
  }
  return to;
};
var __toCommonJS = (mod) => __copyProps(__defProp({}, "__esModule", { value: true }), mod);

// container-volumes/shared-files/lambda-source/index.ts
var lambda_source_exports = {};
__export(lambda_source_exports, {
  index: () => index
});
module.exports = __toCommonJS(lambda_source_exports);
async function index(event, context) {
  console.log(`Event: ${JSON.stringify(event, null, 2)}`);
  console.log(`Context: ${JSON.stringify(context, null, 2)}`);
  return {
    statusCode: 200,
    body: JSON.stringify({
      message: "Hello World!"
    })
  };
}
// Annotate the CommonJS export names for ESM import in node:
0 && (module.exports = {
  index
});

Although the localstack error is suggesting that this is a file sharing issue, I think that the code is in the right location on my machine (A sub directory of /Users). Can anyone confirm that the guide is correct that even though the lambda is setup to have "PackageType": "Zip", placing a js file there will run.

YeeP · October 10, 2023, 2:21pm

Ok I think I have proven this is not a file sharing issue with the volume;

I stood up the localstack container with only the creation of an SQS queue in the init script.
I went inside the container and creating a file index.js inside /opt/usr
I registered the lambda like so:

aws \
  lambda create-function \
  --endpoint-url=http://localhost:4566 \
  --function-name lambdaHandler \
  --role arn:aws:iam::000000000000:role/lambda-role \
  --code S3Bucket=hot-reload,S3Key="/opt/usr" \
  --handler index.handler \
  --runtime nodejs18.x \
  --region us-east-1

I bound the lambda to the queue
I triggered the lambda

Just to be clear, there was no use of any methods to externally modify this lambda code, and no shared volume associated with the location of the lambda. Here is the error I received:

localstack_main  |   File "/opt/code/localstack/.venv/lib/python3.11/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception
localstack_main  |     raise cls(e, response=response, explanation=explanation) from e
localstack_main  | docker.errors.APIError: 500 Server Error for http+docker://localhost/v1.43/containers/133f97a7613442af9f634f365fd5b8123869c01002688be9eaba14622b6bd1a1/start: Internal Server Error ("b'Mounts denied: \nThe path /opt/usr is not shared from the host and is not known to Docker.\nYou can configure shared paths from Docker -> Preferences... -> Resources -> File Sharing.\nSee https://docs.docker.com/desktop/mac for more info.'")

I have noticed that the documentation is referencing (PWD) on the location of the lambda code. This line does not work in an init script that is automatically triggered.

/etc/localstack/init/ready.d/init-aws.sh: line 11: PWD: command not found

darasimi · December 22, 2023, 3:19am

@YeeP, Yes, the guide is correct. I followed it and was able to spin up a lambda with hot-reloading enabled.

You are experiencing the file sharing issue because the S3 key(/lambda-builder/dist) in your code snippet is not a subdirectory of the /Users folder or any of the allowed directories that could be bind mounted into Docker containers. Hence docker does not have the permission to bind mount the S3 key path.

Try this instead:

echo "Create Lambda"
aws \
  lambda create-function \
  --endpoint-url=http://localhost:4566 \
  --function-name lambdaHandler \
  --role arn:aws:iam::000000000000:role/lambda-role \
  --code S3Bucket=hot-reload,S3Key="$(PWD)/lambda-builder/dist" \ # included PWD here
  --handler index.handler \
  --runtime nodejs18.x \
  --region us-east-1

YeeP · December 22, 2023, 5:40am

I appreciate the response. The directory you are referring to though is inside the container.‘/lambda-builder/dist’

darasimi · December 22, 2023, 1:00pm

Hmmm, I see where the problem lies. Having the directory inside the container is likely the root cause of the issue as Localstack requires that directory to exists on the host.

Try moving the directory to the host, preferably a /Users subdirectory, and set the new path to be the S3 key.

Marcel · December 27, 2023, 6:02am

Hi @YeeP,

It appears that you are not providing a full path to your local directory. In our example, we start the S3Key path with $(pwd) or at least use an env variable that is pointing to this folder.

Example with full path in S3Key

awslocal lambda create-function --function-name my-cool-local-function \
    --code S3Bucket="hot-reload",S3Key="/tmp/aws-doc-sdk-examples/python/example_code/lambda/boto_client_examples" \
    --handler lambda_handler_basic.lambda_handler \
    --runtime python3.8 \
    --role arn:aws:iam::000000000000:role/lambda-role

Example with $PWD:

awslocal lambda create-function \
    --function-name hello-world \
    --runtime "nodejs16.x" \
    --role arn:aws:iam::123456789012:role/lambda-ex \
    --code S3Bucket="hot-reload",S3Key="$(PWD)/dist" \
    --handler index.handler

Example with the ENV var:

Pass LAMBDA_MOUNT_CWD env var with the path to the built code directory (in our case, to the folder with unzipped FatJar):

LAMBDA_MOUNT_CWD=$(pwd)/build/hot serverless deploy --stage local

Topic		Replies	Views
Running Lambda in localstack inside a k8s strucutre Platform localstack	7	584	January 22, 2024
I can't run Lambda in LocalStack Platform localstack	1	1432	July 19, 2023
Lambda is currently not working on CircleCI for my current docker configurations Platform localstack	4	525	October 16, 2023
Lambda file-system-configs not added to function Platform serverless	1	116	May 28, 2024
Deployment works fine but Lambda invocation fails with error errorType":"Runtime.ImportModuleError" "errorMessage":"Error: Cannot find module Require stack:- /var/task/test-lambdahandler-override.js - /var/runtime/UserFunction.js localstack	1	1210	May 22, 2023

Lambda with Hot Reload functionality is erroring when triggered

Related topics