Localstack v3.0.0 in Github action Lambda invocation failing

Platform
1

I have a Github action that looks like this:

name: integration-test
on:
  push:
    branches: ["main", "dev"]
  pull_request:
    branches: ["main", "dev"]
jobs:
  localstack:
    runs-on: ubuntu-latest
    steps:
      # Get code
      - uses: actions/checkout@v3
      # Start LocalStack
      - name: Start LocalStack
        env:
          LOCALSTACK_AUTH_TOKEN: ${{ secrets.LOCALSTACK_AUTH_TOKEN }}
          ACTIVATE_PRO: 1
          DEBUG: 1
        run: |
          pip install localstack awscli-local[ver1] aws-sam-cli-local   # install LocalStack cli, awslocal, and SAM cli
          docker pull localstack/localstack-pro                         # Make sure to pull the latest version of the image
          localstack start -d                                           # Start LocalStack in the background

          echo "Waiting for LocalStack startup..."                      # Wait 30 seconds for the LocalStack container
          localstack wait -t 30                                         # to become ready before timing out
          echo "Startup complete"
      - name: Create S3 Bucket
        run: awslocal s3 mb s3://devtestprodaws
      # Build and deploy SAM template
      - name: Deploy SAM Template
        working-directory: ./Cloudformation
        run: |
          samlocal build --template-file template.yaml
          samlocal deploy --config-file samconfig.toml --capabilities CAPABILITY_NAMED_IAM --no-confirm-changeset
      - name: Get function url
        run: |
          echo FUNCTION_URL=$(awslocal lambda list-function-url-configs --function-name devtestprodaws-main | grep -o '"FunctionUrl": "[^"]*' | awk -F'"' '{print $4}') >> $GITHUB_ENV
      # Run integration tests
      - name: Run integration tests
        working-directory: ./Lambda/__test__/integration
        run: bun test

My integration test simply makes a POST request to the Lambda function URL.

test("POST /color", async () => {
        const res: Response = await fetch(
            `${process.env.FUNCTION_URL}color?id=myId`,
            {
                method: "POST",
                headers: {
                    "Content-type": "application/json"
                },
                body: JSON.stringify({ color })
            }
        );

        console.log("res:", res);
        console.log("body:", await res.text());
}

This is the output I get when running the Github action:

2023-11-21T11:15:38.5229465Z Timeout: test "POST /color" timed out after 10000ms
2023-11-21T11:15:38.5230725Z e[0me[31m(fail)e[0m Integration tests > POST /color [10000.06ms]
2023-11-21T11:15:38.6952215Z res: Response (3.88 KB) {
2023-11-21T11:15:38.6952670Z   ok: false,
2023-11-21T11:15:38.6953901Z   url: "http://2cdb2xqseo3mn7h9hwpx9ktgogm4vzwf.lambda-url.us-east-1.localhost.localstack.cloud:4566/color?id=myId",
2023-11-21T11:15:38.6955014Z   status: 500,
2023-11-21T11:15:38.6955296Z   statusText: "",
2023-11-21T11:15:38.6955600Z   headers: Headers {
2023-11-21T11:15:38.6956034Z     "content-type": "application/json",
2023-11-21T11:15:38.6956504Z     "content-length": "3877",
2023-11-21T11:15:38.6956925Z     "date": "Tue, 21 Nov 2023 11:15:38 GMT",
2023-11-21T11:15:38.6957440Z     "x-amzn-errortype": "InternalError",
2023-11-21T11:15:38.6958065Z     "x-amzn-requestid": "24d7e8d3-0d3e-475a-9122-eaedc7aad7e4",
2023-11-21T11:15:38.6958781Z     "x-amz-request-id": "24d7e8d3-0d3e-475a-9122-eaedc7aad7e4",
2023-11-21T11:15:38.6959355Z     "server": "hypercorn-h11"
2023-11-21T11:15:38.6959695Z   },
2023-11-21T11:15:38.6959946Z   redirected: false,
2023-11-21T11:15:38.6960249Z   bodyUsed: false,
2023-11-21T11:15:38.6960553Z   Blob (3.88 KB)
2023-11-21T11:15:38.6960829Z }
2023-11-21T11:15:38.6983925Z body: {"__type": "InternalError", "message": "exception while calling lambda with unknown operation: Traceback (most recent call last):\n  File \"/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/services/lambda_/invocation/executor_endpoint.py\", line 134, in wait_for_startup\n    self.startup_future.result()\n  File \"/usr/local/lib/python3.11/concurrent/futures/_base.py\", line 454, in result\n    raise CancelledError()\nconcurrent.futures._base.CancelledError\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File \"/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/services/lambda_/invocation/execution_environment.py\", line 185, in start\n    self.runtime_executor.start(self.get_environment_variables())\n  File \"/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/services/lambda_/invocation/docker_runtime_executor.py\", line 364, in start\n    self.executor_endpoint.wait_for_startup()\n  File \"/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/services/lambda_/invocation/executor_endpoint.py\", line 138, in wait_for_startup\n    raise ShutdownDuringStartup(\nlocalstack.services.lambda_.invocation.executor_endpoint.ShutdownDuringStartup: Executor environment shutdown during container startup\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File \"/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/aws/chain.py\", line 90, in handle\n    handler(self, self.context, response)\n  File \"/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/aws/handlers/routes.py\", line 27, in __call__\n    router_response = self.router.dispatch(context.request)\n                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/http/router.py\", line 442, in dispatch\n    return self.dispatcher(request, handler, args)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/http/dispatcher.py\", line 70, in _dispatch\n    result = endpoint(request, **args)\n             ^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/services/lambda_/urlrouter.py\", line 80, in handle_lambda_url_invocation\n    result = self.lambda_service.invoke(\n             ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/services/lambda_/invocation/lambda_service.py\", line 324, in invoke\n    return version_manager.invoke(\n           ^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/services/lambda_/invocation/version_manager.py\", line 200, in invoke\n    with self.assignment_service.get_environment(\n  File \"/usr/local/lib/python3.11/contextlib.py\", line 137, in __enter__\n    return next(self.gen)\n           ^^^^^^^^^^^^^^\n  File \"/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/services/lambda_/invocation/assignment.py\", line 66, in get_environment\n    execution_environment = self.start_environment(function_version)\n                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/services/lambda_/invocation/assignment.py\", line 90, in start_environment\n    execution_environment.start()\n  File \"/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/services/lambda_/invocation/execution_environment.py\", line 199, in start\n    raise EnvironmentStartupTimeoutException(\nlocalstack.services.lambda_.invocation.execution_environment.EnvironmentStartupTimeoutException: Execution environment timed out during startup.\n"}

I’ve tried testing Localstack on my local machine and it works fine. I also tried test running the Github action using act (GitHub - nektos/act: Run your GitHub Actions locally 🚀) which also works fine. So what is causing this issue to occur when running it on the Github action and how can I fix it?

2

Hi @tobyloki,

Thanks for reaching out.

The issue’s root might lie with the workflow in LS. Namely it must pull the runtime docker image first before it’s able to run your lambda while your test waits only 10 secs. This seems a pretty short time period to pull the image, import any layers if it’s necessary, start up the instance and respond to the request.
To resolve the issue I see a few options:

  • you configure your test to wait more before times out, let’s say 30-60 secs,
  • run localstack update docker-images which would update/download the default python 3.9 runtime image,
  • you pull/build the image yourself before hand if it’s something different from the above mentioned default runtime

Let us know if this helps.