Localstack not triggering init hook and erroring when I manually trigger it

YeeP · October 7, 2023, 2:36pm

First I will cover the init script and how it is setup
Keep in mind all logs were generated after ALL images, volumes, and containers were pruned from docker. As well as all local volume files deleted.

I really hope this is something silly I’m missing, but I feel like I have done my best to review as much as I can find in the documentation before reaching out for help. Some of the settings in this docker-compose file are planned for future use with another container. (4th volume listed which is referenced in the init script and the use of a docker network for communication between containers)

Here is my docker-compose file:

version: '3.8'

services:
  localstack:
    container_name: "${LOCALSTACK_DOCKER_NAME-localstack_main}"
    image: localstack/localstack:2.3.0
    ports:
      - "127.0.0.1:4566:4566"            # LocalStack Gateway
      - "127.0.0.1:4510-4559:4510-4559"  # external services port range
    environment:
      - AWS_DEFAULT_REGION=us-east-1
      - AWS_ACCESS_KEY_ID=foo
      - AWS_SECRET_ACCESS_KEY=bar
      - DEBUG=1
      - DOCKER_HOST=unix:///var/run/docker.sock
    volumes:
      - "./init-aws.sh:/etc/localstack/init/ready.d/init-aws.sh"
      - "./lambda-builder/container-volumes/localstack-volume:/var/lib/localstack"
      - "/var/run/docker.sock:/var/run/docker.sock"
      - "./lambda-builder/container-volumes/shared-files/dist:/lambda-builder/dist"
    working_dir: /lambda-builder
    env_file:
      - ./localstack.env
    networks:
      - localstack
networks:
    localstack:
        external: false
        driver: bridge
        name: localstack

I have confirmed that localstack says the config is valid:

 ~/Projects/localstack_poc/localstack/poc_function  localstack config validate 
✔ config valid

As you can see, the init-aws.sh file is in the correct location as far as it is referenced in the docker-compose file, and it is listed as executable:

 ~/Projects/localstack_poc/localstack/poc_function  ls -la
total 24
drwxr-xr-x@ 6 myusername  staff   192 Oct  7 07:55 .
drwxr-xr-x@ 3 myusername  staff    96 Oct  6 17:40 ..
-rw-r--r--@ 1 myusername  staff  1950 Oct  7 07:44 docker-compose.yml
-rwxr-xr-x@ 1 myusername  staff  1251 Oct  7 07:55 init-aws.sh
drwxr-xr-x@ 8 myusername  staff   256 Oct  7 07:41 lambda-builder
-rw-r--r--@ 1 myusername  staff    40 Oct  7 07:38 localstack.env
 ~/Projects/localstack_poc/localstack/poc_function 

Here are the internals of the init script:

#!/bin/bash

echo "Create Queue"
aws \
  sqs create-queue \
  --queue-name sqsQueue \
  --endpoint-url http://localhost:4566

echo "Create Lambda"
aws \
  lambda create-function \
  --endpoint-url=http://localhost:4566 \
  --function-name lambdaHandler \
  --role arn:aws:iam::000000000000:role/lambda-role \
  --code S3Bucket=hot-reload,S3Key="/lambda-builder/dist" \
  --handler index.handler \
  --runtime nodejs18.x \

echo "Bind Queue to Lambda"
aws \
  lambda create-event-source-mapping \
  --function-name lambdaHandler \
  --batch-size 1 \
  --event-source-arn "arn:aws:sqs:us-east-1:000000000000:sqsQueue" \
  --endpoint-url=http://localhost:4566

I should interject here that I have also reduced this script to only creating an s3 bucket and minimized the docker-compose file to contain exactly what is shown in the docs here: Installation | Docs with the addition of the initialization hook volume, and I get the same results.

Results
The Init scripts discovered: {BOOT: [], START: [], READY: ... log never shows up.
I have repeatedly stepped through what is recommended here: Initialization Hooks | Docs with no luck.

Here are the logs when standing the container up:

 ~/Projects/localstack_poc/localstack/poc_function  docker-compose up                               
[+] Building 0.0s (0/0)                                                                                                                                                                                       docker:desktop-linux
[+] Running 1/0
 ✔ Container localstack_main  Created                                                                                                                                                                                         0.0s 
Attaching to localstack_main
localstack_main  | LocalStack supervisor: starting
localstack_main  | LocalStack supervisor: localstack process (PID 15) starting
localstack_main  | 
localstack_main  | LocalStack version: 2.3.0
localstack_main  | LocalStack Docker container id: 602ba73bc407
localstack_main  | LocalStack build date: 2023-09-28
localstack_main  | LocalStack build git hash: d53f3a49
localstack_main  | 
localstack_main  | 2023-10-07T14:22:49.289 DEBUG --- [  MainThread] stevedore._cache           : reading /root/.cache/python-entrypoints/f1cf862cc3589e48f44b8987620df99ac8015595fb5b101c80bca52ba3cadf40
localstack_main  | 2023-10-07T14:22:49.289 DEBUG --- [  MainThread] stevedore.extension        : found extension EntryPoint(name='register_pickle_patches_runtime', value='localstack_persistence.pickling.__plugins__:register_pickle_patches_runtime', group='localstack.hooks.on_infra_start')
localstack_main  | 2023-10-07T14:22:49.289 DEBUG --- [  MainThread] stevedore.extension        : found extension EntryPoint(name='register_public_cloudpods_endpoints', value='localstack_persistence.pods.__plugins__:register_public_cloudpods_endpoints', group='localstack.hooks.on_infra_start')
localstack_main  | 2023-10-07T14:22:49.290 DEBUG --- [  MainThread] stevedore.extension        : found extension EntryPoint(name='register_reset_state_resource', value='localstack_persistence.reset.__plugins__:register_reset_state_resource', group='localstack.hooks.on_infra_start')
localstack_main  | 2023-10-07T14:22:49.290 DEBUG --- [  MainThread] stevedore.extension        : found extension EntryPoint(name='register_state_load_strategy', value='localstack_persistence.snapshot.__plugins__:register_state_load_strategy', group='localstack.hooks.on_infra_start')
localstack_main  | 2023-10-07T14:22:49.381  WARN --- [  MainThread] l.services.internal        : Enabling diagnose endpoint, please be aware that this can expose sensitive information via your network.
localstack_main  | 2023-10-07T14:22:49.382 DEBUG --- [  MainThread] stevedore.extension        : found extension EntryPoint(name='register_state_resource', value='localstack_persistence.snapshot.__plugins__:register_state_resource', group='localstack.hooks.on_infra_start')
localstack_main  | 2023-10-07T14:22:49.382 DEBUG --- [  MainThread] stevedore.extension        : found extension EntryPoint(name='register_state_save_strategy', value='localstack_persistence.snapshot.__plugins__:register_state_save_strategy', group='localstack.hooks.on_infra_start')
localstack_main  | 2023-10-07T14:22:49.382 DEBUG --- [  MainThread] plugin.manager             : instantiating plugin PluginSpec(localstack.hooks.on_infra_start.register_pickle_patches_runtime = <function register_pickle_patches_runtime at 0xffff8c9b0180>)
localstack_main  | 2023-10-07T14:22:49.382 DEBUG --- [  MainThread] plugin.manager             : loading plugin localstack.hooks.on_infra_start:register_pickle_patches_runtime
localstack_main  | 2023-10-07T14:22:49.382 DEBUG --- [  MainThread] plugin.manager             : instantiating plugin PluginSpec(localstack.hooks.on_infra_start.register_public_cloudpods_endpoints = <function register_public_cloudpods_endpoints at 0xffff8c9b04a0>)
localstack_main  | 2023-10-07T14:22:49.382 DEBUG --- [  MainThread] plugin.manager             : loading plugin localstack.hooks.on_infra_start:register_public_cloudpods_endpoints
localstack_main  | 2023-10-07T14:22:49.382 DEBUG --- [  MainThread] plugin.manager             : instantiating plugin PluginSpec(localstack.hooks.on_infra_start.register_reset_state_resource = <function register_reset_state_resource at 0xffff8c9b14e0>)
localstack_main  | 2023-10-07T14:22:49.382 DEBUG --- [  MainThread] plugin.manager             : plugin localstack.hooks.on_infra_start:register_reset_state_resource is disabled
localstack_main  | 2023-10-07T14:22:49.382 DEBUG --- [  MainThread] plugin.manager             : instantiating plugin PluginSpec(localstack.hooks.on_infra_start.register_state_load_strategy = <function register_state_load_strategy at 0xffff8bfb2b60>)
localstack_main  | 2023-10-07T14:22:49.382 DEBUG --- [  MainThread] plugin.manager             : plugin localstack.hooks.on_infra_start:register_state_load_strategy is disabled
localstack_main  | 2023-10-07T14:22:49.382 DEBUG --- [  MainThread] plugin.manager             : instantiating plugin PluginSpec(localstack.hooks.on_infra_start.register_state_resource = <function register_state_resource at 0xffff8bfb2a20>)
localstack_main  | 2023-10-07T14:22:49.382 DEBUG --- [  MainThread] plugin.manager             : plugin localstack.hooks.on_infra_start:register_state_resource is disabled
localstack_main  | 2023-10-07T14:22:49.382 DEBUG --- [  MainThread] plugin.manager             : instantiating plugin PluginSpec(localstack.hooks.on_infra_start.register_state_save_strategy = <function register_state_save_strategy at 0xffff8bfb2e80>)
localstack_main  | 2023-10-07T14:22:49.382 DEBUG --- [  MainThread] plugin.manager             : plugin localstack.hooks.on_infra_start:register_state_save_strategy is disabled
localstack_main  | 2023-10-07T14:22:49.430 DEBUG --- [  MainThread] localstack.utils.threads   : start_thread called without providing a custom name
localstack_main  | 2023-10-07T14:22:49.431 DEBUG --- [-functhread1] localstack.utils.run       : Executing command: whoami
localstack_main  | 2023-10-07T14:22:49.436 DEBUG --- [-functhread1] localstack.utils.ssl       : Using cached SSL certificate (less than 6hrs since last update).
localstack_main  | 2023-10-07T14:22:49.441  INFO --- [-functhread2] hypercorn.error            : Running on https://0.0.0.0:4566 (CTRL + C to quit)
localstack_main  | 2023-10-07T14:22:49.441  INFO --- [-functhread2] hypercorn.error            : Running on https://0.0.0.0:4566 (CTRL + C to quit)
localstack_main  | 2023-10-07T14:22:49.735 DEBUG --- [  MainThread] plugin.manager             : no extensions found in namespace localstack.aws.provider
localstack_main  | Ready.
localstack_main  | 2023-10-07T14:22:49.739 DEBUG --- [  MainThread] stevedore.extension        : found extension EntryPoint(name='do_run_state_load_all', value='localstack_persistence.snapshot.__plugins__:do_run_state_load_all', group='localstack.hooks.on_infra_ready')
localstack_main  | 2023-10-07T14:22:49.740 DEBUG --- [  MainThread] plugin.manager             : instantiating plugin PluginSpec(localstack.hooks.on_infra_ready.do_run_state_load_all = <function do_run_state_load_all at 0xffff8bfb2ca0>)
localstack_main  | 2023-10-07T14:22:49.740 DEBUG --- [  MainThread] plugin.manager             : plugin localstack.hooks.on_infra_ready:do_run_state_load_all is disabled

This is what I see when running an exec (inside container)

# cd /etc/localstack/init/ready.d
# ls -la
total 12
drwxr-xr-x 2 root root 4096 Oct  7 13:51 .
drwxr-xr-x 1 root root 4096 Oct  7 13:51 ..
-rwxr-xr-x 1 root root 1251 Oct  7 13:55 init-aws.sh
#

Errors when manually triggering script
I found out that the status endpoint will actually trigger the script, I am sure that is already known by everyone here, but I will use it to trigger the script so show the data it returns, and the logs from inside the container.

Response from status endpoint:

{
  "completed": {
    "BOOT": false,
    "START": false,
    "READY": false,
    "SHUTDOWN": false
  },
  "scripts": [
    {
      "stage": "READY",
      "name": "init-aws.sh",
      "state": "UNKNOWN"
    }
  ]
}

Logs when script is triggered by status endpoint:

localstack_main  | 2023-10-07T14:27:01.445 DEBUG --- [   asgi_gw_0] l.a.p.service_router       : loading service catalog index cache file /var/lib/localstack/cache/service-catalog-2_3_0-1_31_56.pickle
localstack_main  | 2023-10-07T14:27:01.465 DEBUG --- [   asgi_gw_0] localstack.runtime.init    : Init scripts discovered: {BOOT: [], START: [], READY: [Script(path='/etc/localstack/init/ready.d/init-aws.sh', stage=READY, state=UNKNOWN)], SHUTDOWN: []}
localstack_main  | 2023-10-07T14:27:01.466  INFO --- [   asgi_gw_0] localstack.request.http    : GET /_localstack/init => 200
localstack_main  | 2023-10-07T14:27:01.580 ERROR --- [   asgi_gw_0] l.aws.handlers.logging     : exception during call chain
localstack_main  | Traceback (most recent call last):
localstack_main  |   File "/opt/code/localstack/localstack/aws/chain.py", line 90, in handle
localstack_main  |     handler(self, self.context, response)
localstack_main  |   File "/opt/code/localstack/localstack/aws/handlers/service_plugin.py", line 36, in __call__
localstack_main  |     return self.require_service(chain, context, response)
localstack_main  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
localstack_main  |   File "/opt/code/localstack/localstack/aws/handlers/service_plugin.py", line 44, in require_service
localstack_main  |     raise NotImplementedError
localstack_main  | NotImplementedError
localstack_main  | 2023-10-07T14:27:01.582  INFO --- [   asgi_gw_0] l.aws.handlers.service     : API for service 's3' not yet implemented or pro feature - please check https://docs.localstack.cloud/references/coverage/ for further information
localstack_main  | 2023-10-07T14:27:01.584  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS s3.ListObjects => 501 (InternalFailure)

I can produce more logs if I trigger the script directly in the container:

# cd /etc/localstack/init/ready.d
# ls
init-aws.sh
# bash init-aws.sh
Create Queue

An error occurred (InternalFailure) when calling the CreateQueue operation: API for service 'sqs' not yet implemented or pro feature - please check https://docs.localstack.cloud/references/coverage/ for further information
Create Lambda

An error occurred (InternalFailure) when calling the CreateFunction operation: API for service 'lambda' not yet implemented or pro feature - please check https://docs.localstack.cloud/references/coverage/ for further information
Bind Queue to Lambda

An error occurred (InternalFailure) when calling the CreateEventSourceMapping operation: API for service 'lambda' not yet implemented or pro feature - please check https://docs.localstack.cloud/references/coverage/ for further information

I followed the reference to the coverage docs and I honestly do not think I am trying to create anything that is not covered in the community edition. Even if I was trying to do something that was not covered, I would expect the error to show up when mounting the container into docker.

Any help would be much appreciated, I hope I provided enough information to paint the full picture.

Thanks

HarshCasper · October 7, 2023, 8:45pm

Hi,

Thanks for sharing your issue!

Here is a simple docker-compose.yml configuration:

version: "3.8"

services:
  localstack:
    container_name: "${LOCALSTACK_DOCKER_NAME-localstack_main}"
    image: localstack/localstack
    ports:
      - "127.0.0.1:4566:4566"
    environment:
      - DEBUG=1
      - DOCKER_HOST=unix:///var/run/docker.sock
    volumes:
      - "${PWD}/init-aws.sh:/etc/localstack/init/ready.d/init-aws.sh"  # ready hook
      - "${LOCALSTACK_VOLUME_DIR:-./volume}:/var/lib/localstack"
      - "/var/run/docker.sock:/var/run/docker.sock"

Ensure that your init-aws.sh file is in the same directory as the Compose configuration.

Create a init-aws.sh file with the following content:

#!/bin/bash

echo "Create Queue"
aws \
  sqs create-queue \
  --queue-name sqsQueue \
  --endpoint-url http://localhost:4566 \
  --region us-east-1

echo "Create Lambda"
aws \
  lambda create-function \
  --endpoint-url=http://localhost:4566 \
  --function-name lambdaHandler \
  --role arn:aws:iam::000000000000:role/lambda-role \
  --code S3Bucket=hot-reload,S3Key="/lambda-builder/dist" \
  --handler index.handler \
  --runtime nodejs18.x \
  --region us-east-1

echo "Bind Queue to Lambda"
aws \
  lambda create-event-source-mapping \
  --function-name lambdaHandler \
  --batch-size 1 \
  --event-source-arn "arn:aws:sqs:us-east-1:000000000000:sqsQueue" \
  --endpoint-url=http://localhost:4566 \
  --region us-east-1

Note: I have added a region parameter and configured it to us-east-1. You can alternatively use awslocal or configure your region to an alternative one.

Make the script executable: chmod +x init-aws.sh.

Start the Docker Compose: docker-compose up.

You will find the following in your logs:

localstack_main  | Create Queue
localstack_main  | 2023-10-07T20:43:40.189  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS sqs.CreateQueue => 200
localstack_main  | {
localstack_main  |     "QueueUrl": "http://localhost:4566/000000000000/sqsQueue"
localstack_main  | }
localstack_main  | Create Lambda
localstack_main  | 2023-10-07T20:43:41.057  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS lambda.CreateFunction => 201
localstack_main  | {
localstack_main  |     "FunctionName": "lambdaHandler",
localstack_main  |     "FunctionArn": "arn:aws:lambda:us-east-1:000000000000:function:lambdaHandler",
localstack_main  |     "Runtime": "nodejs18.x",
localstack_main  |     "Role": "arn:aws:iam::000000000000:role/lambda-role",
localstack_main  |     "Handler": "index.handler",
localstack_main  |     "CodeSize": 0,
localstack_main  |     "Description": "",
localstack_main  |     "Timeout": 3,
localstack_main  |     "MemorySize": 128,
localstack_main  |     "LastModified": "2023-10-07T20:43:41.055842+0000",
localstack_main  |     "CodeSha256": "hot-reloading-hash-not-available",
localstack_main  |     "Version": "$LATEST",
localstack_main  |     "TracingConfig": {
localstack_main  |         "Mode": "PassThrough"
localstack_main  |     },
localstack_main  |     "RevisionId": "a686f3be-ebaf-4382-97a1-22407aba4031",
localstack_main  |     "State": "Pending",
localstack_main  |     "StateReason": "The function is being created.",
localstack_main  |     "StateReasonCode": "Creating",
localstack_main  |     "PackageType": "Zip",
localstack_main  |     "Architectures": [
localstack_main  |         "x86_64"
localstack_main  |     ],
localstack_main  |     "EphemeralStorage": {
localstack_main  |         "Size": 512
localstack_main  |     },
localstack_main  |     "SnapStart": {
localstack_main  |         "ApplyOn": "None",
localstack_main  |         "OptimizationStatus": "Off"
localstack_main  |     },
localstack_main  |     "RuntimeVersionConfig": {
localstack_main  |         "RuntimeVersionArn": "arn:aws:lambda:us-east-1::runtime:8eeff65f6809a3ce81507fe733fe09b835899b99481ba22fd75b5a7338290ec1"
localstack_main  |     }
localstack_main  | }
localstack_main  | Bind Queue to Lambda
localstack_main  | 2023-10-07T20:43:41.429  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS lambda.CreateEventSourceMapping => 202
localstack_main  | {
localstack_main  |     "UUID": "ce42136c-3513-4153-b58a-af62d2ed2c6a",
localstack_main  |     "BatchSize": 1,
localstack_main  |     "MaximumBatchingWindowInSeconds": 0,
localstack_main  |     "EventSourceArn": "arn:aws:sqs:us-east-1:000000000000:sqsQueue",
localstack_main  |     "FunctionArn": "arn:aws:lambda:us-east-1:000000000000:function:lambdaHandler",
localstack_main  |     "LastModified": 1696711421.42465,
localstack_main  |     "State": "Creating",
localstack_main  |     "StateTransitionReason": "USER_INITIATED",
localstack_main  |     "FunctionResponseTypes": []
localstack_main  | }

Check your SQS queues and Lambda functions:

$ awslocal lambda list-functions
{
    "Functions": [
        {
            "FunctionName": "lambdaHandler",
            "FunctionArn": "arn:aws:lambda:us-east-1:000000000000:function:lambdaHandler",
            "Runtime": "nodejs18.x",
            "Role": "arn:aws:iam::000000000000:role/lambda-role",
            "Handler": "index.handler",
            "CodeSize": 0,
            "Description": "",
            "Timeout": 3,
            "MemorySize": 128,
            "LastModified": "2023-10-07T20:35:27.239789+0000",
            "CodeSha256": "hot-reloading-hash-not-available",
            "Version": "$LATEST",
            "TracingConfig": {
                "Mode": "PassThrough"
            },
            "RevisionId": "d9809112-9b6a-49fa-9091-9abec3d2e357",
            "PackageType": "Zip",
            "Architectures": [
                "x86_64"
            ],
            "EphemeralStorage": {
                "Size": 512
            },
            "SnapStart": {
                "ApplyOn": "None",
                "OptimizationStatus": "Off"
            }
        }
    ]
}

$ awslocal sqs list-queues
{
    "QueueUrls": [
        "http://localhost:4566/000000000000/sqsQueue"
    ]
}

YeeP · October 9, 2023, 1:49pm

@HarshCasper - I will give this a try, but can you explain why the env variables are needed in the init script volume and the “local” localstack volume? I just would like to understand what I did wrong that is fixed by your suggestion. Ill go ahead and change my code now and give it a try. Thank you for the response.

YeeP · October 9, 2023, 1:58pm

@HarshCasper So I have taken your recommendation and have been experimenting. The cause of my issue looks to be the declaration of a working_dir. Any clues as to why this is an issue? I should be able to make everything work without that, so I do not think it will be a problem.

Thank you for your response, I appreciate it.

Topic		Replies	Views
Initialization hook not working? Platform	3	878	July 10, 2023
Localstack initialization hooks with from volumes Platform localstack	2	291	July 20, 2023
.NET Lambda on LocalStack throws "exception while calling lambda with unknown operation:" Platform localstack	2	35	September 17, 2024
Error appearing during execution of init script Platform localstack	1	516	October 27, 2023
Localstack.cloud never resolves in browser/ping Platform localstack	5	218	May 27, 2024

Localstack not triggering init hook and erroring when I manually trigger it

Related topics