[{"data":1,"prerenderedAt":155},["ShallowReactive",2],{"content-svc-lambda-guides-serverless-stream-processing-with-lambda-3-event-source-mapping":3},{"markdown":4,"frontMatterAttributes":5,"bodyRaw":12,"menu":13,"menuType":148,"isCollapseableMenu":16,"nextDocItem":43,"previousDocItem":24,"contributorPaths":149,"contentName":154,"slug":38,"readingTime":20},"\u003Cp>AWS Lambda event source mappings (ESM) are Lambda resources that simplify processing stream data. An ESM polls streaming data sources like MSK, KDS, SQS and Amazon MQ for new messages, then invokes a Lambda function to process messages in batches. As a developer, you can focus your time writing business logic in your function code and let ESM take care of your streaming data client integration.\u003C\u002Fp>\n\u003Cp>With the Lambda ESM, developers can quickly build new streaming applications while Lambda takes care of the undifferentiated heavy lifting with built-in managed pollers that are highly available, secure and scale automatically to meet business needs. Instead of operating servers or worrying about infrastructure management tasks like capacity provisioning and patching, customers can now focus on accelerating innovation. With the pay for value pricing model (only paying for the compute required to process data, no idle) and 1ms billing, Serverless allows customers to save overall costs.\u003C\u002Fp>\n\u003Cp>The high-level design of ESM is shown in the diagram below:\u003C\u002Fp>\n\u003Cp>\u003Cimg src=\"\u002Fassets\u002Fexternal\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fassets\u002Fesm-design.png\" alt=\"ESM Design\">\u003C\u002Fp>\n\u003Cp>The Lambda ESM is easy to configure and provides flexibility and control over stream processing behavior.\u003C\u002Fp>\n\u003Cul>\n\u003Cli>MSK and KDS both support configurable batch sizes and starting positions. In the case of KDS a tumbling window duration is also configurable.\u003C\u002Fli>\n\u003Cli>Failures can be propagated to dead letter queues (SQS) or topics (SNS) for Kinesis data stream event source.\u003C\u002Fli>\n\u003Cli>Authentication managed with IAM, and in the case of MSK, SASL\u002FSCRAM and mTLS are also supported.\u003C\u002Fli>\n\u003Cli>Lambda triggers for KDS and MSK also support event filtering based on the JSON payload, using the same syntax as Amazon EventBridge rules.\u003C\u002Fli>\n\u003Cli>Both KDS and MSK publish standard CloudWatch metrics to monitor and alarm on processing success and performance.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>You can find more information on setting up MSK and KDS as event sources for Lambda with the below resources and in the next sections.\u003C\u002Fp>\n\u003Col>\n\u003Cli>\u003Ca href=\"https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Finvocation-eventsourcemapping.html\">Lambda event source mappings\u003C\u002Fa>\u003C\u002Fli>\n\u003Cli>\u003Ca href=\"https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Fwith-kinesis.html\">Using AWS Lambda with Amazon Kinesis\u003C\u002Fa>\u003C\u002Fli>\n\u003Cli>\u003Ca href=\"https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Fwith-msk.html\">Using Lambda with MSK\u003C\u002Fa>\u003C\u002Fli>\n\u003Cli>\u003Ca href=\"https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Finvocation-eventfiltering.html\">Lambda event filtering\u003C\u002Fa>\u003C\u002Fli>\n\u003Cli>\u003Ca href=\"https:\u002F\u002Faws.amazon.com\u002Fblogs\u002Fcompute\u002Fusing-amazon-msk-as-an-event-source-for-aws-lambda\u002F\">Using MSK as event source for Lambda\u003C\u002Fa>\u003C\u002Fli>\n\u003C\u002Fol>\n",{"title":6,"order":7,"callout":8},"Event source mapping for Lambda",3,{"title":9,"description":10,"link":11},"Read the docs","Refer to the AWS Lambda documentation for detailed explanations of Lambda event source mappings","https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Finvocation-eventsourcemapping.html","AWS Lambda event source mappings (ESM) are Lambda resources that simplify processing stream data. An ESM polls streaming data sources like MSK, KDS, SQS and Amazon MQ for new messages, then invokes a Lambda function to process messages in batches. As a developer, you can focus your time writing business logic in your function code and let ESM take care of your streaming data client integration.\n\nWith the Lambda ESM, developers can quickly build new streaming applications while Lambda takes care of the undifferentiated heavy lifting with built-in managed pollers that are highly available, secure and scale automatically to meet business needs. Instead of operating servers or worrying about infrastructure management tasks like capacity provisioning and patching, customers can now focus on accelerating innovation. With the pay for value pricing model (only paying for the compute required to process data, no idle) and 1ms billing, Serverless allows customers to save overall costs.\n\nThe high-level design of ESM is shown in the diagram below:\n\n![ESM Design](\u002Fassets\u002Fexternal\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fassets\u002Fesm-design.png)\n\nThe Lambda ESM is easy to configure and provides flexibility and control over stream processing behavior.\n\n- MSK and KDS both support configurable batch sizes and starting positions. In the case of KDS a tumbling window duration is also configurable.\n- Failures can be propagated to dead letter queues (SQS) or topics (SNS) for Kinesis data stream event source.\n- Authentication managed with IAM, and in the case of MSK, SASL\u002FSCRAM and mTLS are also supported.\n- Lambda triggers for KDS and MSK also support event filtering based on the JSON payload, using the same syntax as Amazon EventBridge rules.\n- Both KDS and MSK publish standard CloudWatch metrics to monitor and alarm on processing success and performance.\n\nYou can find more information on setting up MSK and KDS as event sources for Lambda with the below resources and in the next sections.\n\n1. [Lambda event source mappings](https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Finvocation-eventsourcemapping.html)\n1. [Using AWS Lambda with Amazon Kinesis](https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Fwith-kinesis.html)\n1. [Using Lambda with MSK](https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Fwith-msk.html)\n1. [Lambda event filtering](https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Finvocation-eventfiltering.html)\n1. [Using MSK as event source for Lambda](https:\u002F\u002Faws.amazon.com\u002Fblogs\u002Fcompute\u002Fusing-amazon-msk-as-an-event-source-for-aws-lambda\u002F)\n",[14,39,97],{"title":15,"collapsible":16,"isCollapsed":16,"content":17},"Introduction",false,[18,24,34],{"title":15,"order":19,"time":20,"path":21,"id":22,"link":23},1,"2 min","1-introduction","1-introduction.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F1-introduction",{"title":25,"order":26,"callout":27,"time":20,"path":31,"id":32,"link":33},"Getting started with streaming data",2,{"title":28,"description":29,"link":30},"Watch the video","This video will help you get started with Amazon Kinesis Data Streams, a massively scalable, durable, and low-cost streaming data service that enables you to continuously capture gigabytes of data per second from hundreds of thousands of sources, such as website clickstreams, database event streams, financial transactions, social media feeds, IT logs, and location-tracking events.","https:\u002F\u002Fwww.youtube.com\u002Fwatch?v=1I1DcJvmd4w","2-getting-started","2-getting-started.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F2-getting-started",{"title":6,"order":7,"callout":35,"time":20,"path":36,"id":37,"link":38},{"title":9,"description":10,"link":11},"3-event-source-mapping","3-event-source-mapping.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F3-event-source-mapping",{"title":40,"collapsible":16,"isCollapsed":41,"content":42},"Streaming processing with MSK",true,[43,62,70,77,84,90],{"title":44,"order":45,"youtubeVideoID":46,"resources":47,"callout":55,"time":20,"path":59,"id":60,"link":61},"MSK as event source for Lambda",51,"JKvIypfEiok",[48,52],{"link":49,"type":50,"text":51},"https:\u002F\u002Faws.amazon.com\u002Flambda\u002F","Text","AWS Lambda",{"link":53,"type":50,"text":54},"https:\u002F\u002Faws.amazon.com\u002Fserverless\u002F","Serverless on AWS",{"title":56,"description":57,"link":58},"Follow the Kafka with Lambda learning guide","Refer to the deep dive learning guide for using Kafka with Lambda","https:\u002F\u002Fserverlessland.com\u002Fcontent\u002Fguides\u002Flambda-kafka\u002Fintroduction","msk-stream-processing","msk-stream-processing.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-stream-processing",{"title":63,"order":64,"callout":65,"time":66,"path":67,"id":68,"link":69},"Deep dive into Lambda ESM for MSK",50,{"title":56,"description":57,"link":58},"1 min","msk-esm-deep-dive","msk-esm-deep-dive.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-esm-deep-dive",{"title":71,"order":72,"callout":73,"time":20,"path":74,"id":75,"link":76},"Ordered processing of messages",52,{"title":56,"description":57,"link":58},"msk-ordered-processing","msk-ordered-processing.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-ordered-processing",{"title":78,"order":79,"callout":80,"time":66,"path":81,"id":82,"link":83},"Scaling of consumers",54,{"title":56,"description":57,"link":58},"msk-scaling-consumers","msk-scaling-consumers.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-scaling-consumers",{"title":85,"order":86,"time":66,"path":87,"id":88,"link":89},"Authentication options for Lambda consumers",56,"msk-lambda-auth-options","msk-lambda-auth-options.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-lambda-auth-options",{"title":91,"order":92,"callout":93,"time":66,"path":94,"id":95,"link":96},"Error handling in Lambda functions",58,{"title":56,"description":57,"link":58},"msk-lambda-error-handling","msk-lambda-error-handling.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-lambda-error-handling",{"title":98,"collapsible":16,"isCollapsed":41,"content":99},"Streaming processing with KDS",[100,109,121,132,140],{"title":101,"weight":102,"learningGoals":103,"time":105,"path":106,"id":107,"link":108},"Kinesis Data Streams (KDS) - Key Concepts and Terminologies",60,[104],"Learn key concepts and terminologies of Amazon KDS","4 min","4-1-KDS-key-concepts-terms","4-1-KDS-key-concepts-terms.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-1-KDS-key-concepts-terms",{"title":110,"weight":111,"learningGoals":112,"resources":114,"time":117,"path":118,"id":119,"link":120},"Amazon Kinesis Data Streams - Integration with AWS Lambda",62,[113],"Amazon KDS integration with AWS Lambda for Serverless stream processing",[115,116],"https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Finvocation-eventsourcemapping.html#invocation-eventsourcemapping-batching","https:\u002F\u002Faws.amazon.com\u002Fblogs\u002Fcompute\u002Fnew-aws-lambda-scaling-controls-for-kinesis-and-dynamodb-event-sources\u002F","5 min","4-2-KDS-lambda-intg","4-2-KDS-lambda-intg.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-2-KDS-lambda-intg",{"title":122,"weight":123,"learningGoals":124,"resources":126,"time":128,"path":129,"id":130,"link":131},"Amazon KDS - Error handling with Lambda ESM",64,[125],"How to handle errors when configuring Amazon KDS as Lambda ESM",[127],"https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Fwith-kinesis.html","3 min","4-3-KDS-lambda-error-handling","4-3-KDS-lambda-error-handling.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-3-KDS-lambda-error-handling",{"title":133,"weight":134,"learningGoals":135,"time":117,"path":137,"id":138,"link":139},"Kinesis Data Streams Scaling- Key Concepts and Terminologies",66,[136],"Learn key concepts and terminologies of Amazon KDS Scaling","4-4-KDS-scaling","4-4-KDS-scaling.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-4-KDS-scaling",{"title":141,"weight":142,"learningGoals":143,"time":117,"path":145,"id":146,"link":147},"Kinesis Data Streams Monitoring",68,[144],"Learn Amazon KDS & Lambda monitoring methodologies","4-5-KDS-monitoring","4-5-KDS-monitoring.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-5-KDS-monitoring","LIST",[150,151,152,153],"content\u002Fcontributors\u002Fmithun-mallick.json","content\u002Fcontributors\u002Fkevin-schwarz.json","content\u002Fcontributors\u002Fnaren-sithambaram.json","content\u002Fcontributors\u002Fprasanna-s-krishnan.json","Serverless Stream Processing with Lambda",1792578874452]