Support Zappa with LocalStack

Feature Requests
, ,
1

Introduction

Zappa allows developers to build and deploy server-less, event-driven Python applications (including, but not limited to, WSGI web apps) on AWS Lambda + API Gateway, in a manner similar to AWS Chalice. I would like to use Zappa with LocalStack to deploy Django and Flask apps locally, before pushing them to the production.

There has been legitimate community interest around using Zappa with LocalStack, as documented here:

Current setup

To test out the integration with LocalStack, I built a simplistic “Hello World” app to be deployed with Zappa. The directory structure looks like this:

├── requirements.txt
├── test.py
└── zappa_settings.json

The contents are:

$ cat requirements.txt
zappa
flask

$ cat test.py
from flask import Flask

app = Flask(__name__)

@app.route('/')
def index(event=None, context=None):
    return 'Hello World!'

if __name__ == '__main__':
    app.run(debug=True)

$ cat zappa_settings.json
{
    "dev": {
        "app_function": "test.app",
        "aws_region": "us-east-1",
        "profile_name": "default",
        "project_name": "zappa",
        "runtime": "python3.9",
        "s3_bucket": "zappa-onvrhr3wr",
        "aws_endpoint_urls": {
            "s3": "http://localhost:4566",
            "lambda": "http://localhost:4566",
            "apigateway": "http://localhost:4566",
            "iam": "http://localhost:4566",
            "cloudwatch": "http://localhost:4566",
            "acm": "http://localhost:4566",
            "logs": "http://localhost:4566",
            "route53": "http://localhost:4566",
            "sns": "http://localhost:4566",
            "sqs": "http://localhost:4566",
            "cloudformation": "http://localhost:4566",
            "dynamodb": "http://localhost:4566",
            "events": "http://localhost:4566"
        }
    }
}

While deploying the above setup using zappa deploy dev, I encounter the following issue:

Packaging project as zip.
Uploading zappa-dev-1697342100.zip (6.8MiB)..
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 7.09M/7.09M [00:00<00:00, 10.8MB/s]
Waiting for lambda function [zappa-dev] to become active...
Scheduling..
Scheduled zappa-dev-zappa-keep-warm-handler.keep_warm_callback with expression rate(4 minutes)!
Uploading zappa-dev-template-1697342116.json (1.6KiB)..
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 1.59k/1.59k [00:00<00:00, 67.8kB/s]
Waiting for stack zappa-dev to create (this can take a bit)..
  0%|                                                                                                                   | 0/4 [00:03<?, ?res/s]
Deploying API Gateway..
Waiting for lambda function [zappa-dev] to be updated...
Oh no! An error occurred! :(

==============

Traceback (most recent call last):
  File "/Users/harshcasper/code/localstack/zappa/.venv/lib/python3.9/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/Users/harshcasper/code/localstack/zappa/.venv/lib/python3.9/site-packages/urllib3/util/connection.py", line 72, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/Users/harshcasper/.pyenv/versions/3.9.15/lib/python3.9/socket.py", line 954, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno 8] nodename nor servname provided, or not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/harshcasper/code/localstack/zappa/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 714, in urlopen
    httplib_response = self._make_request(
  File "/Users/harshcasper/code/localstack/zappa/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 403, in _make_request
    self._validate_conn(conn)
  File "/Users/harshcasper/code/localstack/zappa/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1053, in _validate_conn
    conn.connect()
  File "/Users/harshcasper/code/localstack/zappa/.venv/lib/python3.9/site-packages/urllib3/connection.py", line 363, in connect
    self.sock = conn = self._new_conn()
  File "/Users/harshcasper/code/localstack/zappa/.venv/lib/python3.9/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x10868e370>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/harshcasper/code/localstack/zappa/.venv/lib/python3.9/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
  File "/Users/harshcasper/code/localstack/zappa/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 798, in urlopen
    retries = retries.increment(
  File "/Users/harshcasper/code/localstack/zappa/.venv/lib/python3.9/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='c2dhqjdf1v.execute-api.us-east-1.amazonaws.com', port=443): Max retries exceeded with url: /dev/ (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x10868e370>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/harshcasper/code/localstack/zappa/.venv/lib/python3.9/site-packages/zappa/cli.py", line 3046, in handle
    sys.exit(cli.handle())
  File "/Users/harshcasper/code/localstack/zappa/.venv/lib/python3.9/site-packages/zappa/cli.py", line 520, in handle
    self.dispatch_command(self.command, stage)
  File "/Users/harshcasper/code/localstack/zappa/.venv/lib/python3.9/site-packages/zappa/cli.py", line 562, in dispatch_command
    self.deploy(self.vargs["zip"], self.vargs["docker_image_uri"])
  File "/Users/harshcasper/code/localstack/zappa/.venv/lib/python3.9/site-packages/zappa/cli.py", line 900, in deploy
    self.touch_endpoint(endpoint_url)
  File "/Users/harshcasper/code/localstack/zappa/.venv/lib/python3.9/site-packages/zappa/cli.py", line 2964, in touch_endpoint
    req = requests.get(endpoint_url + touch_path)
  File "/Users/harshcasper/code/localstack/zappa/.venv/lib/python3.9/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "/Users/harshcasper/code/localstack/zappa/.venv/lib/python3.9/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Users/harshcasper/code/localstack/zappa/.venv/lib/python3.9/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/harshcasper/code/localstack/zappa/.venv/lib/python3.9/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/Users/harshcasper/code/localstack/zappa/.venv/lib/python3.9/site-packages/requests/adapters.py", line 519, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='c2dhqjdf1v.execute-api.us-east-1.amazonaws.com', port=443): Max retries exceeded with url: /dev/ (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x10868e370>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known'))

==============

The LocalStack logs are:

2023-10-15T03:54:59.961  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS iam.GetRole => 404 (NoSuchEntity)
2023-10-15T03:54:59.986  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS iam.CreateRole => 200
2023-10-15T03:55:00.012  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS iam.GetRolePolicy => 404 (NoSuchEntity)
2023-10-15T03:55:00.050  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS iam.PutRolePolicy => 200
2023-10-15T03:55:00.944  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS lambda.ListVersionsByFunction => 404 (ResourceNotFoundException)
2023-10-15T03:55:03.457  INFO --- [   asgi_gw_1] localstack.request.aws     : AWS s3.HeadBucket => 404 (NoSuchBucket)
2023-10-15T03:55:03.479  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS s3.CreateBucket => 200
2023-10-15T03:55:04.147  INFO --- [   asgi_gw_1] localstack.request.aws     : AWS s3.PutObject => 200
2023-10-15T03:55:04.166  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS lambda.GetFunction => 404 (ResourceNotFoundException)
2023-10-15T03:55:04.711  INFO --- [   asgi_gw_1] localstack.request.aws     : AWS lambda.CreateFunction => 201
2023-10-15T03:55:04.773  INFO --- [   asgi_gw_1] localstack.request.aws     : AWS lambda.GetFunctionConfiguration => 200
2023-10-15T03:55:09.859  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS lambda.GetFunctionConfiguration => 200
2023-10-15T03:55:14.915  INFO --- [   asgi_gw_2] localstack.request.aws     : AWS lambda.GetFunctionConfiguration => 200
2023-10-15T03:55:14.937  INFO --- [   asgi_gw_1] localstack.request.aws     : AWS lambda.GetFunction => 200
2023-10-15T03:55:14.971  INFO --- [   asgi_gw_2] localstack.request.aws     : AWS lambda.GetPolicy => 404 (ResourceNotFoundException)
2023-10-15T03:55:15.042  INFO --- [   asgi_gw_1] localstack.request.aws     : AWS events.ListRuleNamesByTarget => 200
2023-10-15T03:55:15.118  INFO --- [   asgi_gw_2] localstack.request.aws     : AWS events.PutRule => 200
2023-10-15T03:55:16.458  INFO --- [   asgi_gw_1] localstack.request.aws     : AWS lambda.AddPermission => 201
2023-10-15T03:55:16.486  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS events.PutTargets => 200
2023-10-15T03:55:16.522  INFO --- [   asgi_gw_1] localstack.request.aws     : AWS s3.HeadBucket => 200
2023-10-15T03:55:16.540  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS s3.PutObject => 200
2023-10-15T03:55:17.859  INFO --- [   asgi_gw_1] localstack.utils.bootstrap : Execution of "_load_service_plugin" took 1290.91ms
2023-10-15T03:55:17.860  INFO --- [   asgi_gw_1] localstack.utils.bootstrap : Execution of "require" took 1291.35ms
2023-10-15T03:55:17.928  INFO --- [   asgi_gw_1] localstack.request.aws     : AWS cloudformation.DescribeStacks => 400 (ValidationError)
2023-10-15T03:55:17.950  INFO --- [   asgi_gw_1] localstack.request.aws     : AWS s3.GetObject => 200
2023-10-15T03:55:17.955  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS cloudformation.CreateStack => 200
2023-10-15T03:55:20.997  INFO --- [   asgi_gw_1] localstack.request.aws     : AWS cloudformation.DescribeStacks => 200
2023-10-15T03:55:21.027  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS s3.HeadBucket => 200
2023-10-15T03:55:21.040  INFO --- [   asgi_gw_1] localstack.request.aws     : AWS s3.DeleteObject => 204
2023-10-15T03:55:21.050  WARN --- [   asgi_gw_0] l.aws.protocol.serializer  : Response object StackResourceDetail contains a member which is not specified: Timestamp
2023-10-15T03:55:21.050  WARN --- [   asgi_gw_0] l.aws.protocol.serializer  : Response object StackResourceDetail contains a member which is not specified: PreviousResourceStatus
2023-10-15T03:55:21.051  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS cloudformation.DescribeStackResource => 200
2023-10-15T03:55:21.060  INFO --- [   asgi_gw_1] localstack.request.aws     : AWS apigateway.GetRestApi => 200
2023-10-15T03:55:21.070  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS apigateway.UpdateRestApi => 200
2023-10-15T03:55:21.078  INFO --- [   asgi_gw_1] localstack.request.aws     : AWS apigateway.UpdateRestApi => 200
2023-10-15T03:55:21.089  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS apigateway.CreateDeployment => 201
2023-10-15T03:55:21.113  INFO --- [   asgi_gw_1] localstack.request.aws     : AWS apigateway.UpdateStage => 200
2023-10-15T03:55:21.123  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS lambda.GetFunctionConfiguration => 200
2023-10-15T03:55:21.163  INFO --- [   asgi_gw_1] localstack.request.aws     : AWS s3.HeadBucket => 200
2023-10-15T03:55:21.181  INFO --- [   asgi_gw_0] localstack.request.aws     : AWS s3.DeleteObject => 204
2023-10-15T04:03:15.694  WARN --- [uncthread430] l.services.events.provider : InputTransformer is currently not supported for scheduled rules
2023-10-15T04:03:15.852  INFO --- [ev:$LATEST_0] l.u.container_networking   : Determined main container network: bridge
2023-10-15T04:03:15.880  INFO --- [ev:$LATEST_0] l.u.container_networking   : Determined main container target IP: 172.17.0.2
2023-10-15T04:03:22.648  INFO --- [   asgi_gw_2] localstack.request.http    : POST /_localstack_lambda/ec5dcc2d16bd0115ca01dc2ebee9f6be/status/ec5dcc2d16bd0115ca01dc2ebee9f6be/ready => 202
2023-10-15T04:03:24.469  INFO --- [   asgi_gw_0] localstack.request.http    : POST /_localstack_lambda/ec5dcc2d16bd0115ca01dc2ebee9f6be/invocations/f6a3f9c4-3258-4ebd-8b0b-0480438e22c3/logs => 202
2023-10-15T04:03:24.474  INFO --- [   asgi_gw_1] localstack.request.http    : POST /_localstack_lambda/ec5dcc2d16bd0115ca01dc2ebee9f6be/invocations/f6a3f9c4-3258-4ebd-8b0b-0480438e22c3/response => 202
2023-10-15T04:07:15.341  WARN --- [uncthread961] l.services.events.provider : InputTransformer is currently not supported for scheduled rules
2023-10-15T04:07:15.486  INFO --- [   asgi_gw_0] localstack.request.http    : POST /_localstack_lambda/ec5dcc2d16bd0115ca01dc2ebee9f6be/invocations/ce1119f0-15af-488e-a10f-ea553f5c79f9/logs => 202
2023-10-15T04:07:15.490  INFO --- [   asgi_gw_3] localstack.request.http    : POST /_localstack_lambda/ec5dcc2d16bd0115ca01dc2ebee9f6be/invocations/ce1119f0-15af-488e-a10f-ea553f5c79f9/response => 202
2023-10-15T04:11:14.978  WARN --- [ncthread1490] l.services.events.provider : InputTransformer is currently not supported for scheduled rules
2023-10-15T04:11:15.107  INFO --- [   asgi_gw_1] localstack.request.http    : POST /_localstack_lambda/ec5dcc2d16bd0115ca01dc2ebee9f6be/invocations/adcbf978-0a6a-4250-9433-1f5f75a2689e/logs => 202
2023-10-15T04:11:15.112  INFO --- [   asgi_gw_2] localstack.request.http    : POST /_localstack_lambda/ec5dcc2d16bd0115ca01dc2ebee9f6be/invocations/adcbf978-0a6a-4250-9433-1f5f75a2689e/response => 202
2023-10-15T04:15:14.612  WARN --- [ncthread2023] l.services.events.provider : InputTransformer is currently not supported for scheduled rules
2023-10-15T04:15:14.749  INFO --- [   asgi_gw_2] localstack.request.http    : POST /_localstack_lambda/ec5dcc2d16bd0115ca01dc2ebee9f6be/invocations/fd36514d-10eb-4ea5-8eae-af968ed96cd8/logs => 202
2023-10-15T04:15:14.754  INFO --- [   asgi_gw_4] localstack.request.http    : POST /_localstack_lambda/ec5dcc2d16bd0115ca01dc2ebee9f6be/invocations/fd36514d-10eb-4ea5-8eae-af968ed96cd8/response => 202

Possible options

To better support Zappa with LocalStack it would be great to have a wrapper-script, similar to chalice-local, or tflocal. It would make it easy to spin up a new Zappa deployment without maintaining external settings or running into specific caveats.

1 Like