[{"data":1,"prerenderedAt":155},["ShallowReactive",2],{"content-svc-lambda-guides-serverless-stream-processing-with-lambda-msk-esm-deep-dive":3},{"markdown":4,"frontMatterAttributes":5,"bodyRaw":12,"menu":13,"menuType":148,"isCollapseableMenu":16,"nextDocItem":70,"previousDocItem":48,"contributorPaths":149,"contentName":154,"slug":69,"readingTime":66},"\u003Cp>Kafka Event Source Mapping (ESM) is a Lambda \u003Ca href=\"https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Finvocation-eventsourcemapping.html\">resource\u003C\u002Fa> that read messages from a Kafka topic and invokes a target Lambda function. It has three main components. The first is the poller process that uses Kafka client libraries to poll the Kafka topic. The ESM resource initializes with a single poller process for reading from all the partitions. As messages are read from the topic, they are passed on to the next component that does batching based on the batch size and interval. Once the batch is ready, it is passed on to a invoker which calls the target Lambda function. Each poller is associated with one instance of a Lambda function and the invocations are synchronous. This pattern allows processing messages in the order in which they are stored within a partition.\u003C\u002Fp>\n\u003Cp>\u003Cimg src=\"\u002Fassets\u002Fexternal\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk\u002Fassets\u002Fmsk-lambda-esm.png\" alt=\"msk-lambda-esm\">\u003C\u002Fp>\n\u003Cp>For a more deep dive learning journey follow the Kafka Lambda learning \u003Ca href=\"https:\u002F\u002Fserverlessland.com\u002Fcontent\u002Fguides\u002Flambda-kafka\u002Fintroduction\">guide\u003C\u002Fa>.\u003C\u002Fp>\n",{"title":6,"order":7,"callout":8},"Deep dive into Lambda ESM for MSK",50,{"title":9,"description":10,"link":11},"Follow the Kafka with Lambda learning guide","Refer to the deep dive learning guide for using Kafka with Lambda","https:\u002F\u002Fserverlessland.com\u002Fcontent\u002Fguides\u002Flambda-kafka\u002Fintroduction","Kafka Event Source Mapping (ESM) is a Lambda [resource](https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Finvocation-eventsourcemapping.html) that read messages from a Kafka topic and invokes a target Lambda function. It has three main components. The first is the poller process that uses Kafka client libraries to poll the Kafka topic. The ESM resource initializes with a single poller process for reading from all the partitions. As messages are read from the topic, they are passed on to the next component that does batching based on the batch size and interval. Once the batch is ready, it is passed on to a invoker which calls the target Lambda function. Each poller is associated with one instance of a Lambda function and the invocations are synchronous. This pattern allows processing messages in the order in which they are stored within a partition.\n\n![msk-lambda-esm](\u002Fassets\u002Fexternal\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk\u002Fassets\u002Fmsk-lambda-esm.png)\n\nFor a more deep dive learning journey follow the Kafka Lambda learning [guide](https:\u002F\u002Fserverlessland.com\u002Fcontent\u002Fguides\u002Flambda-kafka\u002Fintroduction).\n",[14,45,97],{"title":15,"collapsible":16,"isCollapsed":17,"content":18},"Introduction",false,true,[19,25,35],{"title":15,"order":20,"time":21,"path":22,"id":23,"link":24},1,"2 min","1-introduction","1-introduction.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F1-introduction",{"title":26,"order":27,"callout":28,"time":21,"path":32,"id":33,"link":34},"Getting started with streaming data",2,{"title":29,"description":30,"link":31},"Watch the video","This video will help you get started with Amazon Kinesis Data Streams, a massively scalable, durable, and low-cost streaming data service that enables you to continuously capture gigabytes of data per second from hundreds of thousands of sources, such as website clickstreams, database event streams, financial transactions, social media feeds, IT logs, and location-tracking events.","https:\u002F\u002Fwww.youtube.com\u002Fwatch?v=1I1DcJvmd4w","2-getting-started","2-getting-started.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F2-getting-started",{"title":36,"order":37,"callout":38,"time":21,"path":42,"id":43,"link":44},"Event source mapping for Lambda",3,{"title":39,"description":40,"link":41},"Read the docs","Refer to the AWS Lambda documentation for detailed explanations of Lambda event source mappings","https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Finvocation-eventsourcemapping.html","3-event-source-mapping","3-event-source-mapping.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F3-event-source-mapping",{"title":46,"collapsible":16,"isCollapsed":16,"content":47},"Streaming processing with MSK",[48,64,70,77,84,90],{"title":49,"order":50,"youtubeVideoID":51,"resources":52,"callout":60,"time":21,"path":61,"id":62,"link":63},"MSK as event source for Lambda",51,"JKvIypfEiok",[53,57],{"link":54,"type":55,"text":56},"https:\u002F\u002Faws.amazon.com\u002Flambda\u002F","Text","AWS Lambda",{"link":58,"type":55,"text":59},"https:\u002F\u002Faws.amazon.com\u002Fserverless\u002F","Serverless on AWS",{"title":9,"description":10,"link":11},"msk-stream-processing","msk-stream-processing.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-stream-processing",{"title":6,"order":7,"callout":65,"time":66,"path":67,"id":68,"link":69},{"title":9,"description":10,"link":11},"1 min","msk-esm-deep-dive","msk-esm-deep-dive.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-esm-deep-dive",{"title":71,"order":72,"callout":73,"time":21,"path":74,"id":75,"link":76},"Ordered processing of messages",52,{"title":9,"description":10,"link":11},"msk-ordered-processing","msk-ordered-processing.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-ordered-processing",{"title":78,"order":79,"callout":80,"time":66,"path":81,"id":82,"link":83},"Scaling of consumers",54,{"title":9,"description":10,"link":11},"msk-scaling-consumers","msk-scaling-consumers.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-scaling-consumers",{"title":85,"order":86,"time":66,"path":87,"id":88,"link":89},"Authentication options for Lambda consumers",56,"msk-lambda-auth-options","msk-lambda-auth-options.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-lambda-auth-options",{"title":91,"order":92,"callout":93,"time":66,"path":94,"id":95,"link":96},"Error handling in Lambda functions",58,{"title":9,"description":10,"link":11},"msk-lambda-error-handling","msk-lambda-error-handling.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-lambda-error-handling",{"title":98,"collapsible":16,"isCollapsed":17,"content":99},"Streaming processing with KDS",[100,109,121,132,140],{"title":101,"weight":102,"learningGoals":103,"time":105,"path":106,"id":107,"link":108},"Kinesis Data Streams (KDS) - Key Concepts and Terminologies",60,[104],"Learn key concepts and terminologies of Amazon KDS","4 min","4-1-KDS-key-concepts-terms","4-1-KDS-key-concepts-terms.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-1-KDS-key-concepts-terms",{"title":110,"weight":111,"learningGoals":112,"resources":114,"time":117,"path":118,"id":119,"link":120},"Amazon Kinesis Data Streams - Integration with AWS Lambda",62,[113],"Amazon KDS integration with AWS Lambda for Serverless stream processing",[115,116],"https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Finvocation-eventsourcemapping.html#invocation-eventsourcemapping-batching","https:\u002F\u002Faws.amazon.com\u002Fblogs\u002Fcompute\u002Fnew-aws-lambda-scaling-controls-for-kinesis-and-dynamodb-event-sources\u002F","5 min","4-2-KDS-lambda-intg","4-2-KDS-lambda-intg.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-2-KDS-lambda-intg",{"title":122,"weight":123,"learningGoals":124,"resources":126,"time":128,"path":129,"id":130,"link":131},"Amazon KDS - Error handling with Lambda ESM",64,[125],"How to handle errors when configuring Amazon KDS as Lambda ESM",[127],"https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Fwith-kinesis.html","3 min","4-3-KDS-lambda-error-handling","4-3-KDS-lambda-error-handling.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-3-KDS-lambda-error-handling",{"title":133,"weight":134,"learningGoals":135,"time":117,"path":137,"id":138,"link":139},"Kinesis Data Streams Scaling- Key Concepts and Terminologies",66,[136],"Learn key concepts and terminologies of Amazon KDS Scaling","4-4-KDS-scaling","4-4-KDS-scaling.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-4-KDS-scaling",{"title":141,"weight":142,"learningGoals":143,"time":117,"path":145,"id":146,"link":147},"Kinesis Data Streams Monitoring",68,[144],"Learn Amazon KDS & Lambda monitoring methodologies","4-5-KDS-monitoring","4-5-KDS-monitoring.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-5-KDS-monitoring","LIST",[150,151,152,153],"content\u002Fcontributors\u002Fmithun-mallick.json","content\u002Fcontributors\u002Fkevin-schwarz.json","content\u002Fcontributors\u002Fnaren-sithambaram.json","content\u002Fcontributors\u002Fprasanna-s-krishnan.json","Serverless Stream Processing with Lambda",1792665273697]