SNS Topic with multiple Lambda subscribers causes Docker Desktop to crash
The problem that I am encountering is related to SNS topics that have multiple subscribers. Listed below are the steps and the outcome.
1. Create LocalStack Docker container (docker-compose up -d)
version: "3.11"
services:
localstack:
container_name: "${LOCALSTACK_DOCKER_NAME-localstack_main}"
image: localstack/localstack:2.2
ports:
- "127.0.0.1:4566:4566" # LocalStack Gateway
- "127.0.0.1:4510-4559:4510-4559" # external services port range
- "8055:8080"
environment:
- DEBUG=1
- LOCALSTACK_HOSTNAME=localstack
- TEST_AWS_ACCOUNT_ID=000000000000
- AWS_ACCESS_KEY_ID=dummyaccess
- AWS_SECRET_ACCESS_KEY=dummysecret
- AWS_DEFAULT_REGION=us-east-1
- DOCKER_HOST=unix:///var/run/docker.sock
- LAMBDA_DOCKER_NETWORK=docker_default
volumes:
- "${LOCALSTACK_VOLUME_DIR:-./volume}:/var/lib/localstack"
- "/var/run/docker.sock:/var/run/docker.sock"
2. Apply Terraform to LocalStack (This is just a subset my Terraform that I have but this is related to the issue)
###SNS
#user_login_completed_topic topic definition
resource "aws_sns_topic" "user_login_completed_topic" {
display_name = "Allows the Api to send a message to the other Apis when user logs in"
content_based_deduplication = false
fifo_topic = false
name = "user-login-completed-topic-int"
}
#user_login_completed_topic topic subscrition definitions
resource "aws_sns_topic_subscription" "user_login_completed_news_api_subscription" {
endpoint = aws_lambda_function.news_lambda_function.function_name
protocol = "lambda"
topic_arn = aws_sns_topic.user_login_completed_topic.arn
}
resource "aws_sns_topic_subscription" "user_login_completed_activities_api_subscription" {
endpoint = aws_lambda_function.activities_lambda_function.function_name
protocol = "lambda"
topic_arn = aws_sns_topic.user_login_completed_topic.arn
}
##########################################################################
###Lambda
resource "aws_s3_bucket" "news_lambda_bucket" {
provider = aws
bucket = "news-lambda-int"
}
data "archive_file" "news_lamda_py_zip" {
type = "zip"
source_dir = "./lambda/lambda-proxy"
output_path = "news-lambda-proxy.zip"
}
resource "aws_s3_bucket_object" "news_lambda_proxy" {
bucket = aws_s3_bucket.news_lambda_bucket.id
key = "news-lambda-proxy.zip"
source = data.archive_file.news_lamda_py_zip.output_path
etag = filemd5(data.archive_file.news_lamda_py_zip.output_path)
}
resource "aws_lambda_function" "news_lambda_function" {
function_name = "news-proxy"
runtime = "python3.7"
s3_bucket = aws_s3_bucket.news_lambda_bucket.id
s3_key = aws_s3_bucket_object.news_lambda_proxy.key
s3_object_version = aws_s3_bucket_object.news_lambda_proxy.version_id
handler = "lambda_function.lambda_handler"
role = "arn:aws:iam::000000000000:role/lambda-role"
environment {
variables = {
dns_name = "host.docker.internal:"
api_port = "49301"
api_name = "news"
}
}
}
resource "aws_lambda_permission" "news_lambda_permission" {
action = "lambda:InvokeFunction"
function_name = aws_lambda_function.news_lambda_function.function_name
principal = "sns.amazonaws.com"
source_arn = aws_sns_topic.user_login_completed_topic.arn
}
resource "aws_s3_bucket" "activities_lambda_bucket" {
provider = aws
bucket = "activities-lambda-int"
}
data "archive_file" "activities_lamda_py_zip" {
type = "zip"
source_dir = "./lambda/lambda-proxy"
output_path = "activities-lambda-proxy.zip"
}
resource "aws_s3_bucket_object" "activities_lambda_proxy" {
bucket = aws_s3_bucket.activities_lambda_bucket.id
key = "activities-lambda-proxy.zip"
source = data.archive_file.activities_lamda_py_zip.output_path
etag = filemd5(data.archive_file.activities_lamda_py_zip.output_path)
}
resource "aws_lambda_function" "activities_lambda_function" {
function_name = "rma-activities-proxy"
runtime = "python3.7"
s3_bucket = aws_s3_bucket.activities_lambda_bucket.id
s3_key = aws_s3_bucket_object.activities_lambda_proxy.key
s3_object_version = aws_s3_bucket_object.activities_lambda_proxy.version_id
handler = "lambda_function.lambda_handler"
role = "arn:aws:iam::000000000000:role/lambda-role"
environment {
variables = {
dns_name = "host.docker.internal:"
api_port = "49201"
api_name = "activities"
}
}
}
resource "aws_lambda_permission" "activities_lambda_permission_10" {
action = "lambda:InvokeFunction"
function_name = aws_lambda_function.activities_lambda_function.function_name
principal = "sns.amazonaws.com"
source_arn = aws_sns_topic.user_login_completed_topic.arn
}
##Lambda Funtion
import os
import json
import requests
def lambda_handler(event, context):
url = "https://" + os.getenv("url_env_prefix") + os.getenv("dns_name") + "/services/" + os.getenv("api_name") + "/sns/handlesnsmessage"
sns_message_payload = event["Records"][0]["Sns"]
sns_message_headers = {
"x-amz-sns-message-id": sns_message_payload['MessageId'],
"x-amz-sns-message-type": sns_message_payload["Type"],
"x-amz-sns-subscription-arn" : event["Records"][0]["EventSubscriptionArn"],
"x-amz-sns-topic-arn" : sns_message_payload["TopicArn"]
}
try:
r = requests.post(url = url, data = json.dumps(sns_message_payload), headers = sns_message_headers)
except Exceptions as e:
print(e)
return {
'statusCode': r.status_code
}
3. Start my APIs and set a breakpoint that will get hit if the message is received.
4. Post message to AWS Topic
awslocal sns publish --topic-arn arn:aws:sns:us-east-1:000000000000:user-login-completed-topic-int --message "Test Message"
5. Results
After posting the above message, the output is the message id.
Next, the lambda containers get created in Docker Desktop, but they are in a Status of Created.
My APIs never receive the message.
Docker Desktop is basically not working. I must restart the service.
If I repeat step 3 and 4 but with posting a message to a SNS topic with a single subscriber, everything works. There are no issues with Docker and my API receives the message.
I am not sure if this a container warmup, concurrency, or shared resource issue. I have not been able to pinpoint the exact cause or any LocalStack setting to address it.
Has anyone encountered this issue and what was done to resolve it?