Caching Docker images pulled for Lambda functions

Platform
1

Hi I’ve noticed that in running integration tests against Lambda functions in a LocalStack container that the Lambda takes ~30 seconds to go from PENDING to ACTIVE. And 30 seconds of PENDING is spent pulling the Docker image for the Lambda runtime.

Everytime I re-run the integration test suite, I have to sit for 30 seconds waiting for Localstack to pull the runtime Docker image (in my case .NET) from the Docker registry. :pensive:

In each integration test run I am spinning up the localstack container, and then disposing it after the test finishes.

Is there a way to cache the Docker images pulled for the Lambda runtimes, between separate Localstack container instances (but from the same Localstack image)?

2

Hi @kyzooghost,

You ought to be able to cache the data by utilizing the options your CI platform provider offers.
Running LocalStack locally on your machine means that all the Docker images are downloaded once and then reused as needed.

3

Hey @Marcel thanks for the response.

I see that the public.ecr.aws/lambda/dotnet is available from running docker images. However the first .NET Lambda deployed in each integration test run still takes ~30 seconds to progress from PENDING to ACTIVE state. Any subsequent .NET Lambda is in ACTIVE state almost immediately.

What is the Localstack container doing in these ~30 seconds, and is there a way to ‘prewarm’ the image to avoid these 30 seconds?

4

You can try using one of the lambda configurations from the Configuration | Docs. (LAMBDA_PREBUILD_IMAGES=1)

LAMBDA_PREBUILD_IMAGES 0 (default) Prebuild images before execution which increases the cold start time but reduces the time until the Lambda function is ACTIVE. (preview state).

Please look at logs from LocalStack by running it with LS_LOG=trace to see all the logged operations on the LocalStack side.

I have tried running the dotnet8 sample lambda function on my machine, but I was unable to see a ~30-second delay.

5

Thanks @Marcel for the suggestions

Unfortunately LAMBDA_PREBUILD_IMAGES=1 doesn’t make an impact for the situation. Below are the relevant logs after invoking IAmazonLambda.CreateFunctionAsync from the .NET AWS SDK

2024-11-06T01:12:20.091  INFO --- [et.reactor-0] localstack.request.aws     : AWS lambda.CreateFunction => 201; 000000000000/ap-southeast-2; CreateFunctionRequest({'FunctionName': 'HelloWorld', 'Runtime': 'dotnet8', 'Role': 'arn:aws:iam::012345678901:role/DummyRole', 'Handler': 'HelloWorldLambda::HelloWorldLambda.Function::FunctionHandler', 'Code': {'ZipFile': 'Bytes(76.536KB)'}}, headers={'x-amz-api-version': '2015-03-31', 'User-Agent': 'aws-sdk-dotnet-coreclr/3.7.405.11 ua/2.0 os/macos#14.7.0 md/ARCH#Arm64 lang/.NET_Core#8.0.8 md/aws-sdk-dotnet-core#3.7.400.38 api/Lambda#3.7.405.11 cfg/retry-mode#legacy md/ClientAsync cfg/init-coll#1', 'amz-sdk-invocation-id': 'e4975491-cad4-48c0-892a-0c8c7bf32885', 'amz-sdk-request': 'attempt=1; max=5', 'Host': 'localhost:4566', 'X-Amz-Date': '20241106T011219Z', 'X-Amz-Content-SHA256': '38f53c118d4637b61f959c20f6b3962677df44d9a168d30514b219931e704cd6', 'Authorization': 'AWS4-HMAC-SHA256 Credential=test/20241106/ap-southeast-2/lambda/aws4_request, SignedHeaders=content-type;host;user-agent;x-amz-api-version;x-amz-content-sha256;x-amz-date, Signature=679544dc9517b5c2cbb84258cb7d50dfdc8b4b6ede8e5be38e361aeb9e24fd7f', 'Content-Length': '102242', 'Content-Type': 'application/json', 'x-moto-account-id': '000000000000'}); FunctionConfiguration({'RevisionId': 'd3465842-e1c6-417f-aca2-5d2317ab6d5a', 'FunctionName': 'HelloWorld', 'FunctionArn': 'arn:aws:lambda:ap-southeast-2:000000000000:function:HelloWorld', 'LastModified': '2024-11-06T01:12:20.090316+0000', 'Version': '$LATEST', 'Description': '', 'Role': 'arn:aws:iam::012345678901:role/DummyRole', 'Timeout': 3, 'Runtime': 'dotnet8', 'Handler': 'HelloWorldLambda::HelloWorldLambda.Function::FunctionHandler', 'MemorySize': 128, 'PackageType': <PackageType.Zip: 'Zip'>, 'TracingConfig': {'Mode': <TracingMode.PassThrough: 'PassThrough'>}, 'EphemeralStorage': {'Size': 512}, 'SnapStart': {'ApplyOn': <SnapStartApplyOn.None_: 'None'>, 'OptimizationStatus': <SnapStartOptimizationStatus.Off: 'Off'>}, 'RuntimeVersionConfig': {'RuntimeVersionArn': 'arn:aws:lambda:ap-southeast-2::runtime:8eeff65f6809a3ce81507fe733fe09b835899b99481ba22fd75b5a7338290ec1'}, 'LoggingConfig': {'LogFormat': <LogFormat.Text: 'Text'>, 'LogGroup': '/aws/lambda/HelloWorld'}, 'State': <State.Pending: 'Pending'>, 'StateReason': 'The function is being created.', 'StateReasonCode': <StateReasonCode.Creating: 'Creating'>, 'Architectures': [<Architecture.x86_64: 'x86_64'>], 'CodeSize': 76536, 'CodeSha256': 'vimn3oLavV0JskSimp/J7YKEtEUGvDYL1vW4NwAt+6Q='}, headers={'Content-Type': 'text/plain; charset=utf-8', 'Content-Length': '1043', 'x-amzn-requestid': '1fbefb84-ed52-4e0a-a6eb-f5ed29a90717', 'x-amz-request-id': '1fbefb84-ed52-4e0a-a6eb-f5ed29a90717'})

2024-11-06T01:12:20.093 DEBUG --- [et.reactor-0] rolo.gateway.wsgi          : HEAD localhost:4566/awslambda-ap-southeast-2-tasks/snapshots/000000000000/HelloWorld-f64c17e8-db3c-4cac-b2e4-239763116306

2024-11-06T01:12:20.093 DEBUG --- [et.reactor-0] l.aws.protocol.serializer  : Determined accept type (None) is not supported by this serializer. Using default of this serializer: application/xml

2024-11-06T01:12:20.093  INFO --- [et.reactor-0] l.request.internal.aws     : AWS s3.HeadObject => 200

2024-11-06T01:12:20.095 DEBUG --- [et.reactor-1] rolo.gateway.wsgi          : GET localhost:4566/awslambda-ap-southeast-2-tasks/snapshots/000000000000/HelloWorld-f64c17e8-db3c-4cac-b2e4-239763116306

2024-11-06T01:12:20.095 DEBUG --- [et.reactor-1] l.aws.protocol.serializer  : Determined accept type (None) is not supported by this serializer. Using default of this serializer: application/xml

2024-11-06T01:12:20.095  INFO --- [et.reactor-1] l.request.internal.aws     : AWS s3.GetObject => 200

2024-11-06T01:12:20.096 DEBUG --- [rvice-task_0] localstack.utils.run       : Executing command: ['unzip', '-o', '-q', '/tmp/tmp7fk9m0yw']

2024-11-06T01:12:20.098 DEBUG --- [rvice-task_0] l.u.c.docker_sdk_client    : Pulling Docker image: public.ecr.aws/lambda/dotnet:8

2024-11-06T01:13:07.239 DEBUG --- [rvice-task_0] l.s.l.i.version_manager    : Version preparation of function arn:aws:lambda:ap-southeast-2:000000000000:function:HelloWorld:$LATEST took 47148.54ms

2024-11-06T01:13:07.239 DEBUG --- [rvice-task_0] l.s.l.i.version_manager    : Changing Lambda arn:aws:lambda:ap-southeast-2:000000000000:function:HelloWorld:$LATEST (id 963d0b1b) to active

In both LAMBDA_PREBUILD_IMAGES=0 and LAMBDA_PREBUILD_IMAGES=1 cases the ‘version preparation of the Lambda’ takes 45-50s

6

If I grep for the log lines reporting on ‘version preparation’, I get the below logs showing that the first Lambda deployment takes ages and the remaining Lambdas deploy quickly:

2024-11-06T01:17:24.266 DEBUG — [rvice-task_0] l.s.l.i.version_manager : Version preparation of function arn:aws:lambda:ap-southeast-2:000000000000:function:HelloWorld:$LATEST took 48381.92ms

2024-11-06T01:17:27.066 DEBUG — [rvice-task_0] l.s.l.i.version_manager : Version preparation of function arn:aws:lambda:ap-southeast-2:000000000000:function:ReportCenter_Finaliser_Lambda:$LATEST took 243.40ms

2024-11-06T01:17:33.024 DEBUG — [rvice-task_0] l.s.l.i.version_manager : Version preparation of function arn:aws:lambda:ap-southeast-2:000000000000:function:ReportCenter_Generator_Lambda:$LATEST took 216.33ms

2024-11-06T01:17:39.623 DEBUG — [rvice-task_0] l.s.l.i.version_manager : Version preparation of function arn:aws:lambda:ap-southeast-2:000000000000:function:ReportCenter_Request_Lambda:$LATEST took 279.30ms

7

Here is the FunctionHandler code for the HelloWorldLambda

using Amazon.Lambda.Core;

// Assembly attribute to enable the Lambda function's JSON input to be converted into a .NET class.
[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]

namespace HelloWorldLambda;

public class Function
{
    
    /// <summary>
    /// A simple function that takes a string and does a ToUpper
    /// </summary>
    /// <param name="input">The event for the Lambda function handler to process.</param>
    /// <param name="context">The ILambdaContext that provides methods for logging and describing the Lambda environment.</param>
    /// <returns></returns>
    public string FunctionHandler(string input, ILambdaContext context)
    {
        return input.ToUpper();
    }
}

The remaining Lambdas are more complicated and have 3rd party dependencies (e.g. Newtonsoft.Json, Npgsql)

8

Thanks for sharing the logs. This is caused by the slow download speed of the docker image. I am sorry, but we cannot affect this. As you mentioned, follow-up runs are quick.

You can take a look at our documentation for some CI providers | Docs and give that a go.

I suggest checking the options your CI provider offers or running the tests on your own machine.

9

Thanks @Marcel for taking the time to read through and debug my issue.

The ~30+ second delay is when I’m running the tests on my own machine (brand new Macbook) unfortunately.

Would love to dig deeper into the source code to see what’s happening here if I had the bandwidth :sweat_smile: