Missing SNS topic error during chalice-local deploy

Platform
1

I am trying to do a chalice deploy to a localstack inside a docker compose on MacOS. It is failing on a missing SNS topic. But the topic exists. I have tried localstack 2.0.2 and 2.1.

From the logs, it appears the KeyError ‘arn:aws:sns:eu-west-1:000000000000:TranscribeComplete’ is the source of the problem. This is odd because the topic exists based on the awslocal sns list-topics command. See below, appreciate any help with what I’m doing wrong or if I’ve hit a bug.

localstack log:

backend-chalice-test-localstack-1 | 2023-07-16T22:29:11.292 ERROR — [ asgi_gw_3] l.aws.handlers.logging : exception during call chain
backend-chalice-test-localstack-1 | Traceback (most recent call last):
backend-chalice-test-localstack-1 | File “/opt/code/localstack/localstack/aws/chain.py”, line 90, in handle
backend-chalice-test-localstack-1 | handler(self, self.context, response)
backend-chalice-test-localstack-1 | File “/opt/code/localstack/localstack/aws/handlers/service.py”, line 123, in call
backend-chalice-test-localstack-1 | handler(chain, context, response)
backend-chalice-test-localstack-1 | File “/opt/code/localstack/localstack/aws/handlers/service.py”, line 93, in call
backend-chalice-test-localstack-1 | skeleton_response = self.skeleton.invoke(context)
backend-chalice-test-localstack-1 | File “/opt/code/localstack/localstack/aws/skeleton.py”, line 154, in invoke
backend-chalice-test-localstack-1 | return self.dispatch_request(context, instance)
backend-chalice-test-localstack-1 | File “/opt/code/localstack/localstack/aws/skeleton.py”, line 166, in dispatch_request
backend-chalice-test-localstack-1 | result = handler(context, instance) or {}
backend-chalice-test-localstack-1 | File “/opt/code/localstack/localstack/aws/forwarder.py”, line 60, in _call
backend-chalice-test-localstack-1 | return handler(context, req)
backend-chalice-test-localstack-1 | File “/opt/code/localstack/localstack/aws/skeleton.py”, line 118, in call
backend-chalice-test-localstack-1 | return self.fn(*args, **kwargs)
backend-chalice-test-localstack-1 | File “/opt/code/localstack/localstack/services/sns/provider.py”, line 733, in subscribe
backend-chalice-test-localstack-1 | for existing_topic_subscription in store.topic_subscriptions[topic_arn]:
> backend-chalice-test-localstack-1 | KeyError: ‘arn:aws:sns:eu-west-1:000000000000:TranscribeComplete’
backend-chalice-test-localstack-1 | 2023-07-16T22:29:11.296 INFO — [ asgi_gw_3] localstack.request.aws : AWS sns.Subscribe => 500 (InternalError)

command line of localstack docker:


# awslocal sns list-topics
{
    "Topics": [
        {
            "TopicArn": "arn:aws:sns:eu-west-1:000000000000:TranscribeComplete"
        }
    ]
}

Relevant code in app.py:

SNS_TOPIC = 'arn:aws:sns:eu-west-1:000000000000:TranscribeComplete'

@app.on_sns_message(topic=SNS_TOPIC)
def process_transcript(event):
	.
	.
	.
2

Hi @miiwiv

Could you confirm that the access key ID used by chalice deployment is the same as the one used by awslocal sns list-topics?

The missing topic error might be caused due to inadvertent use of multi-accounts.

3

Hi Viren,

Thanks for your note. I assume you mean AWS access key. I don’t know how to determine what access key Chalice is using. Perhaps it pulls the one set in my environment.

If someone can advise on that, great. https://github.com/localstack/chalice-local/blob/main/README.md

Or if I can set the AWS access key in the localstack container at the command line, or another way, I could try that.

4

Could you try explicitly setting AWS_ACCESS_KEY_ID=000000000000, AWS_SECRET_ACCESS_KEY=test and AWS_DEFAULT_REGION=eu-west-1 before running aws-local or chalice-local? This should override any other credentials reads.

5

I tried this but no change:

$ export AWS_ACCESS_KEY_ID=000000000000
$ export AWS_SECRET_ACCESS_KEY=test
$ export AWS_DEFAULT_REGION=eu-west-1
$
$ chalice-local deploy
Creating deployment package.
Reusing existing deployment package.
Creating lambda function: ananda-media-search-dev-process_transcript
Subscribing ananda-media-search-dev-process_transcript to SNS topic arn:aws:sns:eu-west-1:000000000000:TranscribeComplete
Traceback (most recent call last):
  File "/Users/Michael/.pyenv/versions/3.9.12/envs/ananda-media-search/lib/python3.9/site-packages/chalice/deploy/deployer.py", line 376, in deploy
    return self._deploy(config, chalice_stage_name)
  File "/Users/Michael/.pyenv/versions/3.9.12/envs/ananda-media-search/lib/python3.9/site-packages/chalice/deploy/deployer.py", line 392, in _deploy
    self._executor.execute(plan)
  File "/Users/Michael/.pyenv/versions/3.9.12/envs/ananda-media-search/lib/python3.9/site-packages/chalice/deploy/executor.py", line 42, in execute
    getattr(self, '_do_%s' % instruction.__class__.__name__.lower(),
  File "/Users/Michael/.pyenv/versions/3.9.12/envs/ananda-media-search/lib/python3.9/site-packages/chalice/deploy/executor.py", line 55, in _do_apicall
    result = method(**final_kwargs)
  File "/Users/Michael/.pyenv/versions/3.9.12/envs/ananda-media-search/lib/python3.9/site-packages/chalice/awsclient.py", line 1336, in subscribe_function_to_topic
    response = sns_client.subscribe(
  File "/Users/Michael/.pyenv/versions/3.9.12/envs/ananda-media-search/lib/python3.9/site-packages/botocore/client.py", line 534, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Users/Michael/.pyenv/versions/3.9.12/envs/ananda-media-search/lib/python3.9/site-packages/botocore/client.py", line 976, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.NotFoundException: An error occurred (NotFound) when calling the Subscribe operation: Topic does not exist

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/Michael/.pyenv/versions/3.9.12/envs/ananda-media-search/lib/python3.9/site-packages/chalice/cli/__init__.py", line 637, in main
    return cli(obj={})
  File "/Users/Michael/.pyenv/versions/3.9.12/envs/ananda-media-search/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/Users/Michael/.pyenv/versions/3.9.12/envs/ananda-media-search/lib/python3.9/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/Users/Michael/.pyenv/versions/3.9.12/envs/ananda-media-search/lib/python3.9/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/Michael/.pyenv/versions/3.9.12/envs/ananda-media-search/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/Michael/.pyenv/versions/3.9.12/envs/ananda-media-search/lib/python3.9/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/Users/Michael/.pyenv/versions/3.9.12/envs/ananda-media-search/lib/python3.9/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/Michael/.pyenv/versions/3.9.12/envs/ananda-media-search/lib/python3.9/site-packages/chalice/cli/__init__.py", line 190, in deploy
    deployed_values = d.deploy(config, chalice_stage_name=stage)
  File "/Users/Michael/.pyenv/versions/3.9.12/envs/ananda-media-search/lib/python3.9/site-packages/chalice/deploy/deployer.py", line 378, in deploy
    raise ChaliceDeploymentError(e)
chalice.deploy.deployer.ChaliceDeploymentError: ERROR - While deploying your chalice application, received the following error:

 An error occurred (NotFound) when calling the Subscribe operation: Topic does 
 not exist
6

Checking back for the next thing I can try or your other thoughts.