DynamoDBStreams and Partial Batch Failure Handling

Feature Requests
1

Hi folks
I am trying to use partial batch failure handling on a lambda in conjunction with a dynamodbstream. For testing purposes, I am trying to set up a lambda that reports a itemFailure for every third record received via the dynamodbstream (batch size 100). I am struggling to make this work. The lambda does return an object: {“batchItemFailures”: [{itemIdentifier: “03f893b2”}]} (example).

I have configured the eventSource mapping with FunctionResponseTypes: [“ReportBatchItemFailures”] and a DestinationConfig: { OnFailure: { Destination: dlqArn}}

I would like to ask if anybody has got this to work? I am wondering if this is not yet implemented or I have made a mistake in the configuration. I did find a PR for SQS PartialBatchItemFailures, but I do not know if this also includes dynamodbstreams.

Cheers

2

I am seeing the same behavior with Kinesis. Here’s my event config within the serverless function:

Events:
  KinesisEvent:
    Type: Kinesis
    Properties:
      Stream: 
        Fn::Sub: arn:aws:kinesis:${AWS::Region}:${AWS::AccountId}:stream/my-stream
      StartingPosition: TRIM_HORIZON
      BatchSize: 100
      MaximumBatchingWindowInSeconds: 10
      MaximumRetryAttempts: 0
      BisectBatchOnFunctionError: false
      FunctionResponseTypes: 
          - ReportBatchItemFailures
      DestinationConfig: 
        OnFailure: 
          Destination: 
            Fn::GetAtt:
              - MyDeadLetterQueue
              - Arn 
      Enabled: true
3

Hi @butterfire and @eporta :wave:

Thank you for reporting this issue. We are aware of several issues related to Lambda Event Source Mappings and I can confirm that we are lacking test coverage for your specific scenario.

We are working on a complete rewrite of the Lambda Event Source Mapping with better AWS parity, performance, and test coverage. Most tests are already passing and partial batch failure handling has been carefully considered during the re-design. You will soon be able to opt into the new implementation (ETA: in the coming weeks). Stay tuned :chart_with_upwards_trend: :rocket:

PS: Partial batch failure handling is supported for EventBridge Pipes

Best,
Joel

4

Awesome, thanks! I’ll be looking forward to try the new implementation.

5

@eporta @butterfire

Batch failure handling (i.e., ReportBatchItemFailures) is one of the big improvements with our new Lambda event source mapping implementation. We encourage you to give it another try and are looking forward to hearing your feedback :ear: .

Check out our new Lambda event source mapping (ESM) implementation in LocalStack v4 offering better functional and non-functional support for ESM :rocket: . The behavioral coverage documentation provides more details for your use case Lambda | Docs

Feel free to create a GitHub issue or share your feedback if you experience any challenges👂