Caching Docker images pulled for Lambda functions

Platform
1

Hi I’ve noticed that in running integration tests against Lambda functions in a LocalStack container that the Lambda takes ~30 seconds to go from PENDING to ACTIVE. And 30 seconds of PENDING is spent pulling the Docker image for the Lambda runtime.

Everytime I re-run the integration test suite, I have to sit for 30 seconds waiting for Localstack to pull the runtime Docker image (in my case .NET) from the Docker registry. :pensive:

In each integration test run I am spinning up the localstack container, and then disposing it after the test finishes.

Is there a way to cache the Docker images pulled for the Lambda runtimes, between separate Localstack container instances (but from the same Localstack image)?

2

Hi @kyzooghost,

You ought to be able to cache the data by utilizing the options your CI platform provider offers.
Running LocalStack locally on your machine means that all the Docker images are downloaded once and then reused as needed.

3

Hey @Marcel thanks for the response.

I see that the public.ecr.aws/lambda/dotnet is available from running docker images. However the first .NET Lambda deployed in each integration test run still takes ~30 seconds to progress from PENDING to ACTIVE state. Any subsequent .NET Lambda is in ACTIVE state almost immediately.

What is the Localstack container doing in these ~30 seconds, and is there a way to ‘prewarm’ the image to avoid these 30 seconds?

4

You can try using one of the lambda configurations from the Configuration | Docs. (LAMBDA_PREBUILD_IMAGES=1)

LAMBDA_PREBUILD_IMAGES 0 (default) Prebuild images before execution which increases the cold start time but reduces the time until the Lambda function is ACTIVE. (preview state).

Please look at logs from LocalStack by running it with LS_LOG=trace to see all the logged operations on the LocalStack side.

I have tried running the dotnet8 sample lambda function on my machine, but I was unable to see a ~30-second delay.

5

Thanks @Marcel for the suggestions

Unfortunately LAMBDA_PREBUILD_IMAGES=1 doesn’t make an impact for the situation. Below are the relevant logs after invoking IAmazonLambda.CreateFunctionAsync from the .NET AWS SDK

2024-11-06T01:12:20.091  INFO --- [et.reactor-0] localstack.request.aws     : AWS lambda.CreateFunction => 201; 000000000000/ap-southeast-2; CreateFunctionRequest({'FunctionName': 'HelloWorld', 'Runtime': 'dotnet8', 'Role': 'arn:aws:iam::012345678901:role/DummyRole', 'Handler': 'HelloWorldLambda::HelloWorldLambda.Function::FunctionHandler', 'Code': {'ZipFile': 'Bytes(76.536KB)'}}, headers={'x-amz-api-version': '2015-03-31', 'User-Agent': 'aws-sdk-dotnet-coreclr/3.7.405.11 ua/2.0 os/macos#14.7.0 md/ARCH#Arm64 lang/.NET_Core#8.0.8 md/aws-sdk-dotnet-core#3.7.400.38 api/Lambda#3.7.405.11 cfg/retry-mode#legacy md/ClientAsync cfg/init-coll#1', 'amz-sdk-invocation-id': 'e4975491-cad4-48c0-892a-0c8c7bf32885', 'amz-sdk-request': 'attempt=1; max=5', 'Host': 'localhost:4566', 'X-Amz-Date': '20241106T011219Z', 'X-Amz-Content-SHA256': '38f53c118d4637b61f959c20f6b3962677df44d9a168d30514b219931e704cd6', 'Authorization': 'AWS4-HMAC-SHA256 Credential=test/20241106/ap-southeast-2/lambda/aws4_request, SignedHeaders=content-type;host;user-agent;x-amz-api-version;x-amz-content-sha256;x-amz-date, Signature=679544dc9517b5c2cbb84258cb7d50dfdc8b4b6ede8e5be38e361aeb9e24fd7f', 'Content-Length': '102242', 'Content-Type': 'application/json', 'x-moto-account-id': '000000000000'}); FunctionConfiguration({'RevisionId': 'd3465842-e1c6-417f-aca2-5d2317ab6d5a', 'FunctionName': 'HelloWorld', 'FunctionArn': 'arn:aws:lambda:ap-southeast-2:000000000000:function:HelloWorld', 'LastModified': '2024-11-06T01:12:20.090316+0000', 'Version': '$LATEST', 'Description': '', 'Role': 'arn:aws:iam::012345678901:role/DummyRole', 'Timeout': 3, 'Runtime': 'dotnet8', 'Handler': 'HelloWorldLambda::HelloWorldLambda.Function::FunctionHandler', 'MemorySize': 128, 'PackageType': <PackageType.Zip: 'Zip'>, 'TracingConfig': {'Mode': <TracingMode.PassThrough: 'PassThrough'>}, 'EphemeralStorage': {'Size': 512}, 'SnapStart': {'ApplyOn': <SnapStartApplyOn.None_: 'None'>, 'OptimizationStatus': <SnapStartOptimizationStatus.Off: 'Off'>}, 'RuntimeVersionConfig': {'RuntimeVersionArn': 'arn:aws:lambda:ap-southeast-2::runtime:8eeff65f6809a3ce81507fe733fe09b835899b99481ba22fd75b5a7338290ec1'}, 'LoggingConfig': {'LogFormat': <LogFormat.Text: 'Text'>, 'LogGroup': '/aws/lambda/HelloWorld'}, 'State': <State.Pending: 'Pending'>, 'StateReason': 'The function is being created.', 'StateReasonCode': <StateReasonCode.Creating: 'Creating'>, 'Architectures': [<Architecture.x86_64: 'x86_64'>], 'CodeSize': 76536, 'CodeSha256': 'vimn3oLavV0JskSimp/J7YKEtEUGvDYL1vW4NwAt+6Q='}, headers={'Content-Type': 'text/plain; charset=utf-8', 'Content-Length': '1043', 'x-amzn-requestid': '1fbefb84-ed52-4e0a-a6eb-f5ed29a90717', 'x-amz-request-id': '1fbefb84-ed52-4e0a-a6eb-f5ed29a90717'})

2024-11-06T01:12:20.093 DEBUG --- [et.reactor-0] rolo.gateway.wsgi          : HEAD localhost:4566/awslambda-ap-southeast-2-tasks/snapshots/000000000000/HelloWorld-f64c17e8-db3c-4cac-b2e4-239763116306

2024-11-06T01:12:20.093 DEBUG --- [et.reactor-0] l.aws.protocol.serializer  : Determined accept type (None) is not supported by this serializer. Using default of this serializer: application/xml

2024-11-06T01:12:20.093  INFO --- [et.reactor-0] l.request.internal.aws     : AWS s3.HeadObject => 200

2024-11-06T01:12:20.095 DEBUG --- [et.reactor-1] rolo.gateway.wsgi          : GET localhost:4566/awslambda-ap-southeast-2-tasks/snapshots/000000000000/HelloWorld-f64c17e8-db3c-4cac-b2e4-239763116306

2024-11-06T01:12:20.095 DEBUG --- [et.reactor-1] l.aws.protocol.serializer  : Determined accept type (None) is not supported by this serializer. Using default of this serializer: application/xml

2024-11-06T01:12:20.095  INFO --- [et.reactor-1] l.request.internal.aws     : AWS s3.GetObject => 200

2024-11-06T01:12:20.096 DEBUG --- [rvice-task_0] localstack.utils.run       : Executing command: ['unzip', '-o', '-q', '/tmp/tmp7fk9m0yw']

2024-11-06T01:12:20.098 DEBUG --- [rvice-task_0] l.u.c.docker_sdk_client    : Pulling Docker image: public.ecr.aws/lambda/dotnet:8

2024-11-06T01:13:07.239 DEBUG --- [rvice-task_0] l.s.l.i.version_manager    : Version preparation of function arn:aws:lambda:ap-southeast-2:000000000000:function:HelloWorld:$LATEST took 47148.54ms

2024-11-06T01:13:07.239 DEBUG --- [rvice-task_0] l.s.l.i.version_manager    : Changing Lambda arn:aws:lambda:ap-southeast-2:000000000000:function:HelloWorld:$LATEST (id 963d0b1b) to active

In both LAMBDA_PREBUILD_IMAGES=0 and LAMBDA_PREBUILD_IMAGES=1 cases the ‘version preparation of the Lambda’ takes 45-50s

6

If I grep for the log lines reporting on ‘version preparation’, I get the below logs showing that the first Lambda deployment takes ages and the remaining Lambdas deploy quickly:

2024-11-06T01:17:24.266 DEBUG — [rvice-task_0] l.s.l.i.version_manager : Version preparation of function arn:aws:lambda:ap-southeast-2:000000000000:function:HelloWorld:$LATEST took 48381.92ms

2024-11-06T01:17:27.066 DEBUG — [rvice-task_0] l.s.l.i.version_manager : Version preparation of function arn:aws:lambda:ap-southeast-2:000000000000:function:ReportCenter_Finaliser_Lambda:$LATEST took 243.40ms

2024-11-06T01:17:33.024 DEBUG — [rvice-task_0] l.s.l.i.version_manager : Version preparation of function arn:aws:lambda:ap-southeast-2:000000000000:function:ReportCenter_Generator_Lambda:$LATEST took 216.33ms

2024-11-06T01:17:39.623 DEBUG — [rvice-task_0] l.s.l.i.version_manager : Version preparation of function arn:aws:lambda:ap-southeast-2:000000000000:function:ReportCenter_Request_Lambda:$LATEST took 279.30ms

7

Here is the FunctionHandler code for the HelloWorldLambda

using Amazon.Lambda.Core;

// Assembly attribute to enable the Lambda function's JSON input to be converted into a .NET class.
[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]

namespace HelloWorldLambda;

public class Function
{
    
    /// <summary>
    /// A simple function that takes a string and does a ToUpper
    /// </summary>
    /// <param name="input">The event for the Lambda function handler to process.</param>
    /// <param name="context">The ILambdaContext that provides methods for logging and describing the Lambda environment.</param>
    /// <returns></returns>
    public string FunctionHandler(string input, ILambdaContext context)
    {
        return input.ToUpper();
    }
}

The remaining Lambdas are more complicated and have 3rd party dependencies (e.g. Newtonsoft.Json, Npgsql)

8

Thanks for sharing the logs. This is caused by the slow download speed of the docker image. I am sorry, but we cannot affect this. As you mentioned, follow-up runs are quick.

You can take a look at our documentation for some CI providers | Docs and give that a go.

I suggest checking the options your CI provider offers or running the tests on your own machine.

9

Thanks @Marcel for taking the time to read through and debug my issue.

The ~30+ second delay is when I’m running the tests on my own machine (brand new Macbook) unfortunately.

Would love to dig deeper into the source code to see what’s happening here if I had the bandwidth :sweat_smile:

10

@kyzooghost, Are you saying that 30-second delay occurs even after the public.ecr.aws/lambda/dotnet image has been downloaded to your local machine (assuming Docker Desktop)?

How are you starting LocalStack? Do you have any settings that are not the default?
Can you please confirm that you are using ARM-based machine and docker images?

11

@Marcel Yes the 30+ second delay is after the public.ecr.aws/lambda/dotnet has been downloaded to my local. I can confirm that I am using an Apple M3 (ARM-based), and below is my docker images list:

public.ecr.aws/lambda/dotnet                                                                                           8                                                                        d479b820feeb   8 days ago      329MB
bitnami/postgresql                                                                                                     17.0.0                                                                        17368936cb10   2 weeks ago     364MB
localstack/localstack                                                                                                  latest                                                                        519021e612e3   2 weeks ago     1.09GB

So I am using Testcontainers - https://testcontainers.com - to start and dispose of the LocalStack container. The intention is to have the dotnet test command spin up and then dispose of containers for integration testing, all in native C# code and no manual bash scripting.

This is the C# Testcontainers code snippet for configuring the LocalStack container:

            _container = new ContainerBuilder()
                    .WithImage("localstack/localstack:3.8")
                    .WithName("localstack")
                    .WithCleanUp(true)
                    .WithEnvironment("DEFAULT_REGION", "ap-southeast-2")
                    .WithEnvironment("SERVICES", "lambda,stepfunctions,s3,events")
                    .WithEnvironment("DEBUG", "1")
                    // This will save at least 20 seconds on each test run
                    .WithEnvironment("SKIP_SSL_CERT_DOWNLOAD", "1")
                    .WithEnvironment("LS_LOG", "trace")
                    .WithEnvironment("LS_LOG_TO_CONSOLE", "1")
                    // Location of Unix socket that docker daemon (in container) listens to
                    .WithEnvironment("DOCKER_HOST", $"unix:///var/run/docker.sock")
                    // Make sure that DOCKER_HOST = Bind Mount destination
                    .WithBindMount(unixSocketPath, "/var/run/docker.sock")
                    .WithPortBinding(4566, 4566)
                    .WithNetwork(network)
                    .WithNetworkAliases(containerName)
                    .WithLogger(TestUtilities.CreateLogger<LocalStackFixture>())
                    .WithWaitStrategy(Wait.ForUnixContainer().UntilCommandIsCompleted(new[] { "awslocal", "s3api", "create-bucket", "--bucket", bucketName }))
                    .Build();
12

I would like to propose that you test this on your machine to see if the issue with the delay is part of the testcontainers or not.

To start LocalStack outside testcontainers, please follow Getting Started | Docs

#!/bin/bash

# Prerequisites:
# 1. .NET SDK installed (https://dotnet.microsoft.com/download)
# 2. AWS CLI installed and configured (https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html)
# 3. LocalStack installed and running (https://docs.localstack.cloud/getting-started/)
# 4. Amazon.Lambda.Tools installed (dotnet tool install -g Amazon.Lambda.Tools)
# 5. Amazon.Lambda.Templates installed (dotnet new install Amazon.Lambda.Templates)
# Links:
# https://docs.aws.amazon.com/lambda/latest/dg/csharp-package-cli.html
# https://docs.aws.amazon.com/toolkit-for-visual-studio/latest/user-guide/lambda-cli-publish.html
# https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-endpoints.html

# Install Lambda templates and tools
dotnet new install Amazon.Lambda.Templates
dotnet tool install -g Amazon.Lambda.Tools

# Create a new Lambda project
dotnet new lambda.EmptyFunction --name myDotnetFunction

# Change to the project directory
cd myDotnetFunction/src/myDotnetFunction

# Set environment variables for LocalStack
export AWS_ENDPOINT_URL="http://localhost.localstack.cloud:4566"
export AWS_ACCESS_KEY_ID="test"
export AWS_SECRET_ACCESS_KEY="test"
export AWS_DEFAULT_REGION="us-east-1"

# Deploy the Lambda function 
# Modify the function-architecture as needed. (x86_64)
dotnet lambda deploy-function \
    --use-container-for-build true \
    --function-role "arn:aws:iam::000000000000:role/lambda-role" \
    --function-name "myDotnetFunction" \
    --framework "net8.0" \
    --function-architecture "arm64" 


# Invoke the Lambda function
dotnet lambda invoke-function "myDotnetFunction" --payload '"Hello, world!"'

# Return to the previous directory
cd -
13

Hey @Marcel I’m away from my work laptop for the next 2 weeks, will try this out and report back after then :+1:

14

Hey @Marcel I was not able to get this example working on my local

However before using TestContainers we were doing something similar to your example by using LocalStack in a Docker Compose setup. Anecdotally we observed the delay in this prior setup as well - Lambda functions in LocalStack would take 30-60 seconds after creation for the first successful invocation

15

Hi @kyzooghost,

I am unable to reproduce your issue since it seems to be specific to your local configuration and system, which I cannot duplicate in my environment. I suggest deploying the basic Lambda sample that was shared, as it would be the most effective way to proceed.

Would you mind detailing the issues preventing you from deploying and invoking the Lambda sample? We can investigate them further.

Providing a step-by-step description of your process would be very helpful.

16 
$ dotnet new install Amazon.Lambda.Templates

---

The following template packages will be installed:
   Amazon.Lambda.Templates

Amazon.Lambda.Templates (version 7.2.0) is already installed, it will be replaced with latest version.
Amazon.Lambda.Templates::7.2.0 was successfully uninstalled.
Amazon.Lambda.Templates could not be installed, no NuGet feeds are configured or they are invalid.

Unhandled exception: NuGet.Protocol.Core.Types.FatalProtocolException: Unable to load the service index for source https://***/nuget/nuget.
 ---> System.Net.Http.HttpRequestException: The proxy tunnel request to proxy 'http://***/' failed with status code '503'."

I am using LocalStack in a tightly-controlled environment which requires a decent amount of time to get basic stuff working. I cannot download packages from public software registries :upside_down_face: