[{"data":1,"prerenderedAt":156},["ShallowReactive",2],{"content-svc-lambda-guides-serverless-stream-processing-with-lambda-4-3-KDS-lambda-error-handling":3},{"markdown":4,"frontMatterAttributes":5,"bodyRaw":12,"menu":13,"menuType":149,"isCollapseableMenu":16,"nextDocItem":133,"previousDocItem":114,"contributorPaths":150,"contentName":155,"slug":132,"readingTime":129},"\u003Ch4>Amazon KDS as Lambda ESM - Error handling and retries\u003C\u002Fh4>\n\u003Cp>When Amazon Kinesis is configured as a Lambda ESM, the event source mapping invokes the Lambda function synchronously and retries on errors. The retry mechanism is a varies slightly depending on the type of error.\u003C\u002Fp>\n\u003Cp>For an \u003Ccode>Invocation error\u003C\u002Fcode> (Lambda throttling, request rejection before function receiving it), Lambda retries until all the records expire or exceed the maximum age that is configured on the event source mapping.\u003C\u002Fp>\n\u003Cp>For a \u003Ccode>Function error\u003C\u002Fcode> (runtime errors, error returned by code), Lambda retries the records until they expire or exceed the maximum age or reach the maximum configured retry quota.\u003C\u002Fp>\n\u003Cp>If the records still error out after the error handling mechanisms are configured, Lambda discards the records and continues to process other batches from the stream. In order to retain the discarded records, a failed-event destination can be configured to which Lambda sends the discarded records.\u003C\u002Fp>\n\u003Cp>With default settings, a bad record can potentially block the processing of the impacted KDS shard for up to one week. To avoid this, it is recommended to configure event source mapping with number of retries and maximum record age that suits your needs. There are also multiple strategies that can be considered to improve the efficiency of the workflow based on your requirements for how you want the retries to be handled.\u003C\u002Fp>\n\u003Ch4>Reporting batch item failures\u003C\u002Fh4>\n\u003Cp>When KDS records are consumed and processed in batches, Lambda checkpoints to the highest sequence number of the batch only when the batch is a complete success. If it is not a complete success, Lambda treats this as failure and retries the entire batch. In order to improve this behavior, you can consider partial batch response by enabling \u003Ccode>ReportBatchItemFailures\u003C\u002Fcode> to allow for partially successful batches. This can reduce the number of retries on a record.\u003C\u002Fp>\n\u003Cp>To turn on the \u003Ccode>ReportBatchItemFailures\u003C\u002Fcode> when KDS is configured as Lambda ESM, return the response with a list of sequence number of KDS records in \u003Ccode>StreamEventResponse\u003C\u002Fcode> object structure given below.\u003C\u002Fp>\n\u003Cpre>\u003Ccode>{\n  "batchItemFailures": [\n        {\n            "itemIdentifier": "<id>"\n        }\n    ]\n}\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cblockquote>\n\u003Cp>\u003Cstrong>Note:\u003C\u002Fstrong> If the \u003Ccode>batchItemFailures\u003C\u002Fcode> array contains multiple items, Lambda uses the lowest sequence number to checkpoint and retries all the records from that checkpoint.\u003C\u002Fp>\n\u003C\u002Fblockquote>\n\u003Ch6>Bisecting a batch on function error\u003C\u002Fh6>\n\u003Cp>In order to further improve the retry mechanism, you can consider enabling \u003Ccode>BisectBatchOnFunctionError\u003C\u002Fcode> to allow for bisecting a batch during failures irrespective of \u003Ccode>ReportBatchItemFailures\u003C\u002Fcode> configuration. When enabled, Lambda breaks the impacted batch of records into two and retries them separately. This allows you to isolate erroneous records from the rest of the batch and process other records successfully.\u003C\u002Fp>\n\u003Cp>When a batch is partially successful and both \u003Ccode>BisectBatchOnFunctionError\u003C\u002Fcode> and \u003Ccode>ReportBatchItemFailures\u003C\u002Fcode> are enabled, the batch is bisected on the returned sequence number and Lambda retries only the remaining records.\u003C\u002Fp>\n\u003Cp>In addition to the above mechanisms, it is recommended to configure event sources with \u003Ccode>Maximum Record Age\u003C\u002Fcode>, \u003Ccode>Maximum Retry Attempts\u003C\u002Fcode> and \u003Ccode>Destination on Failure\u003C\u002Fcode> options to efficiently handle failures when processing Kinesis data streams.\u003C\u002Fp>\n",{"title":6,"weight":7,"learningGoals":8,"resources":10},"Amazon KDS - Error handling with Lambda ESM",64,[9],"How to handle errors when configuring Amazon KDS as Lambda ESM",[11],"https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Fwith-kinesis.html","#### Amazon KDS as Lambda ESM - Error handling and retries\n\nWhen Amazon Kinesis is configured as a Lambda ESM, the event source mapping invokes the Lambda function synchronously and retries on errors. The retry mechanism is a varies slightly depending on the type of error.\n\nFor an `Invocation error` (Lambda throttling, request rejection before function receiving it), Lambda retries until all the records expire or exceed the maximum age that is configured on the event source mapping.\n\nFor a `Function error` (runtime errors, error returned by code), Lambda retries the records until they expire or exceed the maximum age or reach the maximum configured retry quota.\n\nIf the records still error out after the error handling mechanisms are configured, Lambda discards the records and continues to process other batches from the stream. In order to retain the discarded records, a failed-event destination can be configured to which Lambda sends the discarded records.\n\nWith default settings, a bad record can potentially block the processing of the impacted KDS shard for up to one week. To avoid this, it is recommended to configure event source mapping with number of retries and maximum record age that suits your needs. There are also multiple strategies that can be considered to improve the efficiency of the workflow based on your requirements for how you want the retries to be handled.\n\n#### Reporting batch item failures\n\nWhen KDS records are consumed and processed in batches, Lambda checkpoints to the highest sequence number of the batch only when the batch is a complete success. If it is not a complete success, Lambda treats this as failure and retries the entire batch. In order to improve this behavior, you can consider partial batch response by enabling `ReportBatchItemFailures` to allow for partially successful batches. This can reduce the number of retries on a record.\n\nTo turn on the `ReportBatchItemFailures` when KDS is configured as Lambda ESM, return the response with a list of sequence number of KDS records in `StreamEventResponse` object structure given below.\n\n```\n{\n  \"batchItemFailures\": [\n        {\n            \"itemIdentifier\": \"\u003Cid>\"\n        }\n    ]\n}\n```\n\n> **Note:** If the `batchItemFailures` array contains multiple items, Lambda uses the lowest sequence number to checkpoint and retries all the records from that checkpoint.\n\n###### Bisecting a batch on function error\n\nIn order to further improve the retry mechanism, you can consider enabling `BisectBatchOnFunctionError` to allow for bisecting a batch during failures irrespective of `ReportBatchItemFailures` configuration. When enabled, Lambda breaks the impacted batch of records into two and retries them separately. This allows you to isolate erroneous records from the rest of the batch and process other records successfully.\n\nWhen a batch is partially successful and both `BisectBatchOnFunctionError` and `ReportBatchItemFailures` are enabled, the batch is bisected on the returned sequence number and Lambda retries only the remaining records.\n\nIn addition to the above mechanisms, it is recommended to configure event sources with `Maximum Record Age`, `Maximum Retry Attempts` and `Destination on Failure` options to efficiently handle failures when processing Kinesis data streams.\n",[14,45,102],{"title":15,"collapsible":16,"isCollapsed":17,"content":18},"Introduction",false,true,[19,25,35],{"title":15,"order":20,"time":21,"path":22,"id":23,"link":24},1,"2 min","1-introduction","1-introduction.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F1-introduction",{"title":26,"order":27,"callout":28,"time":21,"path":32,"id":33,"link":34},"Getting started with streaming data",2,{"title":29,"description":30,"link":31},"Watch the video","This video will help you get started with Amazon Kinesis Data Streams, a massively scalable, durable, and low-cost streaming data service that enables you to continuously capture gigabytes of data per second from hundreds of thousands of sources, such as website clickstreams, database event streams, financial transactions, social media feeds, IT logs, and location-tracking events.","https:\u002F\u002Fwww.youtube.com\u002Fwatch?v=1I1DcJvmd4w","2-getting-started","2-getting-started.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F2-getting-started",{"title":36,"order":37,"callout":38,"time":21,"path":42,"id":43,"link":44},"Event source mapping for Lambda",3,{"title":39,"description":40,"link":41},"Read the docs","Refer to the AWS Lambda documentation for detailed explanations of Lambda event source mappings","https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Finvocation-eventsourcemapping.html","3-event-source-mapping","3-event-source-mapping.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F3-event-source-mapping",{"title":46,"collapsible":16,"isCollapsed":17,"content":47},"Streaming processing with MSK",[48,67,75,82,89,95],{"title":49,"order":50,"youtubeVideoID":51,"resources":52,"callout":60,"time":21,"path":64,"id":65,"link":66},"MSK as event source for Lambda",51,"JKvIypfEiok",[53,57],{"link":54,"type":55,"text":56},"https:\u002F\u002Faws.amazon.com\u002Flambda\u002F","Text","AWS Lambda",{"link":58,"type":55,"text":59},"https:\u002F\u002Faws.amazon.com\u002Fserverless\u002F","Serverless on AWS",{"title":61,"description":62,"link":63},"Follow the Kafka with Lambda learning guide","Refer to the deep dive learning guide for using Kafka with Lambda","https:\u002F\u002Fserverlessland.com\u002Fcontent\u002Fguides\u002Flambda-kafka\u002Fintroduction","msk-stream-processing","msk-stream-processing.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-stream-processing",{"title":68,"order":69,"callout":70,"time":71,"path":72,"id":73,"link":74},"Deep dive into Lambda ESM for MSK",50,{"title":61,"description":62,"link":63},"1 min","msk-esm-deep-dive","msk-esm-deep-dive.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-esm-deep-dive",{"title":76,"order":77,"callout":78,"time":21,"path":79,"id":80,"link":81},"Ordered processing of messages",52,{"title":61,"description":62,"link":63},"msk-ordered-processing","msk-ordered-processing.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-ordered-processing",{"title":83,"order":84,"callout":85,"time":71,"path":86,"id":87,"link":88},"Scaling of consumers",54,{"title":61,"description":62,"link":63},"msk-scaling-consumers","msk-scaling-consumers.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-scaling-consumers",{"title":90,"order":91,"time":71,"path":92,"id":93,"link":94},"Authentication options for Lambda consumers",56,"msk-lambda-auth-options","msk-lambda-auth-options.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-lambda-auth-options",{"title":96,"order":97,"callout":98,"time":71,"path":99,"id":100,"link":101},"Error handling in Lambda functions",58,{"title":61,"description":62,"link":63},"msk-lambda-error-handling","msk-lambda-error-handling.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-lambda-error-handling",{"title":103,"collapsible":16,"isCollapsed":16,"content":104},"Streaming processing with KDS",[105,114,126,133,141],{"title":106,"weight":107,"learningGoals":108,"time":110,"path":111,"id":112,"link":113},"Kinesis Data Streams (KDS) - Key Concepts and Terminologies",60,[109],"Learn key concepts and terminologies of Amazon KDS","4 min","4-1-KDS-key-concepts-terms","4-1-KDS-key-concepts-terms.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-1-KDS-key-concepts-terms",{"title":115,"weight":116,"learningGoals":117,"resources":119,"time":122,"path":123,"id":124,"link":125},"Amazon Kinesis Data Streams - Integration with AWS Lambda",62,[118],"Amazon KDS integration with AWS Lambda for Serverless stream processing",[120,121],"https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Finvocation-eventsourcemapping.html#invocation-eventsourcemapping-batching","https:\u002F\u002Faws.amazon.com\u002Fblogs\u002Fcompute\u002Fnew-aws-lambda-scaling-controls-for-kinesis-and-dynamodb-event-sources\u002F","5 min","4-2-KDS-lambda-intg","4-2-KDS-lambda-intg.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-2-KDS-lambda-intg",{"title":6,"weight":7,"learningGoals":127,"resources":128,"time":129,"path":130,"id":131,"link":132},[9],[11],"3 min","4-3-KDS-lambda-error-handling","4-3-KDS-lambda-error-handling.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-3-KDS-lambda-error-handling",{"title":134,"weight":135,"learningGoals":136,"time":122,"path":138,"id":139,"link":140},"Kinesis Data Streams Scaling- Key Concepts and Terminologies",66,[137],"Learn key concepts and terminologies of Amazon KDS Scaling","4-4-KDS-scaling","4-4-KDS-scaling.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-4-KDS-scaling",{"title":142,"weight":143,"learningGoals":144,"time":122,"path":146,"id":147,"link":148},"Kinesis Data Streams Monitoring",68,[145],"Learn Amazon KDS & Lambda monitoring methodologies","4-5-KDS-monitoring","4-5-KDS-monitoring.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-5-KDS-monitoring","LIST",[151,152,153,154],"content\u002Fcontributors\u002Fmithun-mallick.json","content\u002Fcontributors\u002Fkevin-schwarz.json","content\u002Fcontributors\u002Fnaren-sithambaram.json","content\u002Fcontributors\u002Fprasanna-s-krishnan.json","Serverless Stream Processing with Lambda",1792578874528]