Effective consumer strategies for handling Amazon Kinesis Data Stream anomalies

Amazon Kinesis Data Streams → AWS Lambda

How to handle consumer (AWS Lambda) failure when reading the records from the Amazon Kinesis data stream.

The purpose of this pattern is to deploy the infrastructure to showcase how handle consumer (AWS Lambda) failure when reading the records from the Amazon Kinesis data stream.
If the handling measures fail, Lambda discards the records and continues processing batches from the stream. With the default settings, this means that a bad record can block processing on the affected shard for up to one week. To avoid this, in this project we are going to configure function's event source mapping with a reasonable number of retries and a maximum record age.
To retain a record of discarded batches, we are going to configure a failed-event destination. Lambda sends the record to the destination - Amazon SQS.

< Back to all patterns

GitHub icon Download this pattern (.zip)

GitHub icon View this pattern on GitHub


Clone repo

git clone https://github.com/aws-samples/serverless-patterns/cd serverless-patterns/kinesis-lambda-error-handling

Deploy

sam deploy --guided


Testing

See the README in the GitHub repo for detailed testing instructions.

Cleanup

Delete the stack: sam delete.

Created by:

Manjunath Arakere

Manjunath Arakere

Senior Solutions Architect @ AWS. Serverless enthusiast.