Amazon KDS as Lambda ESM - Error handling and retries

When Amazon Kinesis is configured as a Lambda ESM, the event source mapping invokes the Lambda function synchronously and retries on errors. The retry mechanism is a varies slightly depending on the type of error.

For an Invocation error (Lambda throttling, request rejection before function receiving it), Lambda retries until all the records expire or exceed the maximum age that is configured on the event source mapping.

For a Function error (runtime errors, error returned by code), Lambda retries the records until they expire or exceed the maximum age or reach the maximum configured retry quota.

If the records still error out after the error handling mechanisms are configured, Lambda discards the records and continues to process other batches from the stream. In order to retain the discarded records, a failed-event destination can be configured to which Lambda sends the discarded records.

With default settings, a bad record can potentially block the processing of the impacted KDS shard for up to one week. To avoid this, it is recommended to configure event source mapping with number of retries and maximum record age that suits your needs. There are also multiple strategies that can be considered to improve the efficiency of the workflow based on your requirements for how you want the retries to be handled.

Reporting batch item failures

When KDS records are consumed and processed in batches, Lambda checkpoints to the highest sequence number of the batch only when the batch is a complete success. If it is not a complete success, Lambda treats this as failure and retries the entire batch. In order to improve this behavior, you can consider partial batch response by enabling ReportBatchItemFailures to allow for partially successful batches. This can reduce the number of retries on a record.

To turn on the ReportBatchItemFailures when KDS is configured as Lambda ESM, return the response with a list of sequence number of KDS records in StreamEventResponse object structure given below.

{
  "batchItemFailures": [
        {
            "itemIdentifier": "<id>"
        }
    ]
}

Note: If the batchItemFailures array contains multiple items, Lambda uses the lowest sequence number to checkpoint and retries all the records from that checkpoint.

Bisecting a batch on function error

In order to further improve the retry mechanism, you can consider enabling BisectBatchOnFunctionError to allow for bisecting a batch during failures irrespective of ReportBatchItemFailures configuration. When enabled, Lambda breaks the impacted batch of records into two and retries them separately. This allows you to isolate erroneous records from the rest of the batch and process other records successfully.

When a batch is partially successful and both BisectBatchOnFunctionError and ReportBatchItemFailures are enabled, the batch is bisected on the returned sequence number and Lambda retries only the remaining records.

In addition to the above mechanisms, it is recommended to configure event sources with Maximum Record Age, Maximum Retry Attempts and Destination on Failure options to efficiently handle failures when processing Kinesis data streams.