[{"data":1,"prerenderedAt":155},["ShallowReactive",2],{"content-svc-lambda-guides-serverless-stream-processing-with-lambda-msk-lambda-error-handling":3},{"markdown":4,"frontMatterAttributes":5,"bodyRaw":12,"menu":13,"menuType":148,"isCollapseableMenu":16,"nextDocItem":100,"previousDocItem":86,"contributorPaths":149,"contentName":154,"slug":96,"readingTime":68},"\u003Cp>Lambda Event Source Mapping (ESM) triggers the designated Lambda function in a synchronous manner. This means that the ESM's invoking process waits for the function to respond before moving on to the next set of message batches. If an exception is raised within the function's code, the invoker process within ESM will retry sending the same message batch to the Lambda function. Currently, the messages will be retried until they expire, and there is no option to configure a retry threshold or to move malfunctioning messages (often called "poison pill" messages) to a dead letter queue.\u003C\u002Fp>\n\u003Cp>To handle this situation effectively, the Lambda function needs to skillfully catch the exception using its application logic. Afterward, it can choose to either log the exception in CloudWatch or send it to a queue like SQS FIFO, providing an elegant way to manage errors and exceptions.\u003C\u002Fp>\n\u003Cp>\u003Cimg src=\"\u002Fassets\u002Fexternal\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk\u002Fassets\u002Fmsk-lambda-error-handling.png\" alt=\"lambda-msk-error\">\u003C\u002Fp>\n\u003Cp>For a more deep dive learning journey follow the Kafka Lambda learning \u003Ca href=\"https:\u002F\u002Fserverlessland.com\u002Fcontent\u002Fguides\u002Flambda-kafka\u002Fintroduction\">guide\u003C\u002Fa>.\u003C\u002Fp>\n",{"title":6,"order":7,"callout":8},"Error handling in Lambda functions",58,{"title":9,"description":10,"link":11},"Follow the Kafka with Lambda learning guide","Refer to the deep dive learning guide for using Kafka with Lambda","https:\u002F\u002Fserverlessland.com\u002Fcontent\u002Fguides\u002Flambda-kafka\u002Fintroduction","Lambda Event Source Mapping (ESM) triggers the designated Lambda function in a synchronous manner. This means that the ESM's invoking process waits for the function to respond before moving on to the next set of message batches. If an exception is raised within the function's code, the invoker process within ESM will retry sending the same message batch to the Lambda function. Currently, the messages will be retried until they expire, and there is no option to configure a retry threshold or to move malfunctioning messages (often called \"poison pill\" messages) to a dead letter queue.\n\nTo handle this situation effectively, the Lambda function needs to skillfully catch the exception using its application logic. Afterward, it can choose to either log the exception in CloudWatch or send it to a queue like SQS FIFO, providing an elegant way to manage errors and exceptions.\n\n![lambda-msk-error](\u002Fassets\u002Fexternal\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk\u002Fassets\u002Fmsk-lambda-error-handling.png)\n\nFor a more deep dive learning journey follow the Kafka Lambda learning [guide](https:\u002F\u002Fserverlessland.com\u002Fcontent\u002Fguides\u002Flambda-kafka\u002Fintroduction).\n",[14,45,97],{"title":15,"collapsible":16,"isCollapsed":17,"content":18},"Introduction",false,true,[19,25,35],{"title":15,"order":20,"time":21,"path":22,"id":23,"link":24},1,"2 min","1-introduction","1-introduction.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F1-introduction",{"title":26,"order":27,"callout":28,"time":21,"path":32,"id":33,"link":34},"Getting started with streaming data",2,{"title":29,"description":30,"link":31},"Watch the video","This video will help you get started with Amazon Kinesis Data Streams, a massively scalable, durable, and low-cost streaming data service that enables you to continuously capture gigabytes of data per second from hundreds of thousands of sources, such as website clickstreams, database event streams, financial transactions, social media feeds, IT logs, and location-tracking events.","https:\u002F\u002Fwww.youtube.com\u002Fwatch?v=1I1DcJvmd4w","2-getting-started","2-getting-started.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F2-getting-started",{"title":36,"order":37,"callout":38,"time":21,"path":42,"id":43,"link":44},"Event source mapping for Lambda",3,{"title":39,"description":40,"link":41},"Read the docs","Refer to the AWS Lambda documentation for detailed explanations of Lambda event source mappings","https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Finvocation-eventsourcemapping.html","3-event-source-mapping","3-event-source-mapping.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F3-event-source-mapping",{"title":46,"collapsible":16,"isCollapsed":16,"content":47},"Streaming processing with MSK",[48,64,72,79,86,92],{"title":49,"order":50,"youtubeVideoID":51,"resources":52,"callout":60,"time":21,"path":61,"id":62,"link":63},"MSK as event source for Lambda",51,"JKvIypfEiok",[53,57],{"link":54,"type":55,"text":56},"https:\u002F\u002Faws.amazon.com\u002Flambda\u002F","Text","AWS Lambda",{"link":58,"type":55,"text":59},"https:\u002F\u002Faws.amazon.com\u002Fserverless\u002F","Serverless on AWS",{"title":9,"description":10,"link":11},"msk-stream-processing","msk-stream-processing.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-stream-processing",{"title":65,"order":66,"callout":67,"time":68,"path":69,"id":70,"link":71},"Deep dive into Lambda ESM for MSK",50,{"title":9,"description":10,"link":11},"1 min","msk-esm-deep-dive","msk-esm-deep-dive.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-esm-deep-dive",{"title":73,"order":74,"callout":75,"time":21,"path":76,"id":77,"link":78},"Ordered processing of messages",52,{"title":9,"description":10,"link":11},"msk-ordered-processing","msk-ordered-processing.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-ordered-processing",{"title":80,"order":81,"callout":82,"time":68,"path":83,"id":84,"link":85},"Scaling of consumers",54,{"title":9,"description":10,"link":11},"msk-scaling-consumers","msk-scaling-consumers.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-scaling-consumers",{"title":87,"order":88,"time":68,"path":89,"id":90,"link":91},"Authentication options for Lambda consumers",56,"msk-lambda-auth-options","msk-lambda-auth-options.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-lambda-auth-options",{"title":6,"order":7,"callout":93,"time":68,"path":94,"id":95,"link":96},{"title":9,"description":10,"link":11},"msk-lambda-error-handling","msk-lambda-error-handling.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-lambda-error-handling",{"title":98,"collapsible":16,"isCollapsed":17,"content":99},"Streaming processing with KDS",[100,109,121,132,140],{"title":101,"weight":102,"learningGoals":103,"time":105,"path":106,"id":107,"link":108},"Kinesis Data Streams (KDS) - Key Concepts and Terminologies",60,[104],"Learn key concepts and terminologies of Amazon KDS","4 min","4-1-KDS-key-concepts-terms","4-1-KDS-key-concepts-terms.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-1-KDS-key-concepts-terms",{"title":110,"weight":111,"learningGoals":112,"resources":114,"time":117,"path":118,"id":119,"link":120},"Amazon Kinesis Data Streams - Integration with AWS Lambda",62,[113],"Amazon KDS integration with AWS Lambda for Serverless stream processing",[115,116],"https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Finvocation-eventsourcemapping.html#invocation-eventsourcemapping-batching","https:\u002F\u002Faws.amazon.com\u002Fblogs\u002Fcompute\u002Fnew-aws-lambda-scaling-controls-for-kinesis-and-dynamodb-event-sources\u002F","5 min","4-2-KDS-lambda-intg","4-2-KDS-lambda-intg.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-2-KDS-lambda-intg",{"title":122,"weight":123,"learningGoals":124,"resources":126,"time":128,"path":129,"id":130,"link":131},"Amazon KDS - Error handling with Lambda ESM",64,[125],"How to handle errors when configuring Amazon KDS as Lambda ESM",[127],"https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Fwith-kinesis.html","3 min","4-3-KDS-lambda-error-handling","4-3-KDS-lambda-error-handling.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-3-KDS-lambda-error-handling",{"title":133,"weight":134,"learningGoals":135,"time":117,"path":137,"id":138,"link":139},"Kinesis Data Streams Scaling- Key Concepts and Terminologies",66,[136],"Learn key concepts and terminologies of Amazon KDS Scaling","4-4-KDS-scaling","4-4-KDS-scaling.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-4-KDS-scaling",{"title":141,"weight":142,"learningGoals":143,"time":117,"path":145,"id":146,"link":147},"Kinesis Data Streams Monitoring",68,[144],"Learn Amazon KDS & Lambda monitoring methodologies","4-5-KDS-monitoring","4-5-KDS-monitoring.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-5-KDS-monitoring","LIST",[150,151,152,153],"content\u002Fcontributors\u002Fmithun-mallick.json","content\u002Fcontributors\u002Fkevin-schwarz.json","content\u002Fcontributors\u002Fnaren-sithambaram.json","content\u002Fcontributors\u002Fprasanna-s-krishnan.json","Serverless Stream Processing with Lambda",1792924478999]