Localstack Kinesis + AWS lambda event source mapping?

Platform
1

Hello.

I’m trying to understand if Localstack is able to replace AWS Kinesis + event source mapping for real non Localstack Lambda in our dev environments?

The scenario is the following: we have a Lambda & we configure event source mappings which allows Lambda to get events from Kinesis.
The source mapping requires to configure Kinesis arn.
https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lambda_event_source_mapping

Which arn would I need to configure for Kinesis in that event source mapping & how would I configure it to use localstack Kinesis?

According to this AWS doc an env var
AWS_ENDPOINT_URL_KINESIS
can be used to configure a custom Kinesis endpoint which could be localstack’s & it will probably be used by boto3 we’re using in Lambdas (I haven’t tried it yet, just guessing here). But not sure if it helps the event source mapping.

Also I can pass
AWS_ACCESS_KEY_ID=012345678912
along with

provider "aws" {
 ...
  endpoints {
    kinesis = "http://localstack.example.com:4566"
  }
}

to Terragrunt to deploy Kinesis stream with an arn matching our dev account in order not to rewrite the terraform modules themselves.

So is there any way to use Localstack only for Kinesis in terms of the need to configure event source mapping for AWS non Localstack Lambda?

Thanks.

2

Hi @vainkop :wave:

Do I understand you correctly that you wish to run a hybrid setup as follows:
Kinesis (LocalStack) => Lambda Event Source Mapping (LocalStack) => Lambda function (AWS)

You want to run Kinesis locally, let a local Lambda Event Source Mapping poll from the Kinesis stream, and finally invoke a Lambda function that lives in a remote AWS account, correct?

Could you share your use case to help us understand why you wish to use such a hybrid setup?
It helps to improve the developer experience of LocalStack :pray:

We currently don’t have a blueprint or tests for this exact scenario. However, it could work using our experimental AWS Replicator Extension . The replicator extension can be configured to forward the relevant Lambda lambda:InvokeFunction call using an ARN filter for your specific Lambda function. The “proxy” Lambda function in LocalStack needs to have the same ARN as the one in your target AWS account.

The reverse case (local Lambda & remote source) is discussed in Local Lambda against remote SQS? and @HarshCasper from our DevRel team has created resources related to this use case:

Looking forward to hearing your feedback :ear:

3

Hi @joe4dev

You got it right.

The use case is to save costs on AWS Kinesis & use Localstack Kinesis in K8s instead. We’re using many Kinesis stream & many Lambdas have event source mapping configured for them.

Configuring an AWS event source mapping instead of Localstack would be easier but I’m not sure if it’s possible since AWS event source mapping probably isn’t aware of anything not in AWS since you can use only AWS arn in it’s config.

I will look into the proposed AWS replicator extension solution later.

Thank you.

4

Thanks for the sharing :+1:

Is there a particular reason why the Lambdas need to live in the AWS account and cannot be emulated by LocalStack as well?

PS: We are working on improving Lambda Event Source Mapping. Stay tuned for updates in the coming weeks :chart_with_upwards_trend: