[{"data":1,"prerenderedAt":155},["ShallowReactive",2],{"content-svc-lambda-guides-serverless-stream-processing-with-lambda-msk-ordered-processing":3},{"markdown":4,"frontMatterAttributes":5,"bodyRaw":12,"menu":13,"menuType":148,"isCollapseableMenu":16,"nextDocItem":77,"previousDocItem":64,"contributorPaths":149,"contentName":154,"slug":76,"readingTime":21},"\u003Cp>When a new event is published to a topic, it is appended to one of the topic's partitions. Events with the same event key (e.g., a customer or vehicle ID) are written to the same partition, and Kafka guarantees that any consumer of a given topic-partition will always read that partition's events in exactly the same order as they were written.\u003C\u002Fp>\n\u003Cp>You can find additional insights into Kafka's architecture within the Kafka \u003Ca href=\"https:\u002F\u002Fkafka.apache.org\u002Fdocumentation\u002F#introduction\">documentation\u003C\u002Fa>. AWS Lambda ensures the sequential handling of messages sourced from Kafka topics by establishing message consumers within a shared consumer group. This consumer group is orchestrated by the Event Source Mapping resource. Users have the option to define the consumer group ID, or the Lambda service will generate one randomly if not specified. Every message consumer within this group is assigned to one or multiple partitions. The service then forwards these consumed messages to the designated Lambda function while preserving their original order. Although, considering varying consumer counts, individual function instances might receive messages from diverse partitions, the inherent order within each partition remains intact. This consumer count can expand up to the number of partitions, guided by the \u003Ca href=\"(https:\u002F\u002Faws.amazon.com\u002Fblogs\u002Fcompute\u002Foffset-lag-metric-for-amazon-msk-as-an-event-source-for-lambda\u002F)\">offset lag\u003C\u002Fa>. Once the polling mechanisms reach peak capacity, each respective Lambda function processes messages from an allocated partition.\u003C\u002Fp>\n\u003Cp>Lambda ESM also supports filtering of messages based on JSON filter expressions. Batching of messages is done before the batch is created. The ordering is still maintained within the partition.\u003C\u002Fp>\n\u003Cp>In case of exceptions in processing of messages, Lambda function needs to either log it or write it to another service such as SQS FIFO to avoid reprocessing of messages and out of order scenarios. Currently, the Kafka-Lambda integration does not have any dead letter queue support or selective redelivery of messages based on exception.\u003C\u002Fp>\n\u003Cp>For a more deep dive learning journey follow the Kafka Lambda learning \u003Ca href=\"https:\u002F\u002Fserverlessland.com\u002Fcontent\u002Fguides\u002Flambda-kafka\u002Fintroduction\">guide\u003C\u002Fa>.\u003C\u002Fp>\n",{"title":6,"order":7,"callout":8},"Ordered processing of messages",52,{"title":9,"description":10,"link":11},"Follow the Kafka with Lambda learning guide","Refer to the deep dive learning guide for using Kafka with Lambda","https:\u002F\u002Fserverlessland.com\u002Fcontent\u002Fguides\u002Flambda-kafka\u002Fintroduction","When a new event is published to a topic, it is appended to one of the topic's partitions. Events with the same event key (e.g., a customer or vehicle ID) are written to the same partition, and Kafka guarantees that any consumer of a given topic-partition will always read that partition's events in exactly the same order as they were written.\n\nYou can find additional insights into Kafka's architecture within the Kafka [documentation](https:\u002F\u002Fkafka.apache.org\u002Fdocumentation\u002F#introduction). AWS Lambda ensures the sequential handling of messages sourced from Kafka topics by establishing message consumers within a shared consumer group. This consumer group is orchestrated by the Event Source Mapping resource. Users have the option to define the consumer group ID, or the Lambda service will generate one randomly if not specified. Every message consumer within this group is assigned to one or multiple partitions. The service then forwards these consumed messages to the designated Lambda function while preserving their original order. Although, considering varying consumer counts, individual function instances might receive messages from diverse partitions, the inherent order within each partition remains intact. This consumer count can expand up to the number of partitions, guided by the [offset lag](\u003C(https:\u002F\u002Faws.amazon.com\u002Fblogs\u002Fcompute\u002Foffset-lag-metric-for-amazon-msk-as-an-event-source-for-lambda\u002F)>). Once the polling mechanisms reach peak capacity, each respective Lambda function processes messages from an allocated partition.\n\nLambda ESM also supports filtering of messages based on JSON filter expressions. Batching of messages is done before the batch is created. The ordering is still maintained within the partition.\n\nIn case of exceptions in processing of messages, Lambda function needs to either log it or write it to another service such as SQS FIFO to avoid reprocessing of messages and out of order scenarios. Currently, the Kafka-Lambda integration does not have any dead letter queue support or selective redelivery of messages based on exception.\n\nFor a more deep dive learning journey follow the Kafka Lambda learning [guide](https:\u002F\u002Fserverlessland.com\u002Fcontent\u002Fguides\u002Flambda-kafka\u002Fintroduction).\n",[14,45,97],{"title":15,"collapsible":16,"isCollapsed":17,"content":18},"Introduction",false,true,[19,25,35],{"title":15,"order":20,"time":21,"path":22,"id":23,"link":24},1,"2 min","1-introduction","1-introduction.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F1-introduction",{"title":26,"order":27,"callout":28,"time":21,"path":32,"id":33,"link":34},"Getting started with streaming data",2,{"title":29,"description":30,"link":31},"Watch the video","This video will help you get started with Amazon Kinesis Data Streams, a massively scalable, durable, and low-cost streaming data service that enables you to continuously capture gigabytes of data per second from hundreds of thousands of sources, such as website clickstreams, database event streams, financial transactions, social media feeds, IT logs, and location-tracking events.","https:\u002F\u002Fwww.youtube.com\u002Fwatch?v=1I1DcJvmd4w","2-getting-started","2-getting-started.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F2-getting-started",{"title":36,"order":37,"callout":38,"time":21,"path":42,"id":43,"link":44},"Event source mapping for Lambda",3,{"title":39,"description":40,"link":41},"Read the docs","Refer to the AWS Lambda documentation for detailed explanations of Lambda event source mappings","https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Finvocation-eventsourcemapping.html","3-event-source-mapping","3-event-source-mapping.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F3-event-source-mapping",{"title":46,"collapsible":16,"isCollapsed":16,"content":47},"Streaming processing with MSK",[48,64,72,77,84,90],{"title":49,"order":50,"youtubeVideoID":51,"resources":52,"callout":60,"time":21,"path":61,"id":62,"link":63},"MSK as event source for Lambda",51,"JKvIypfEiok",[53,57],{"link":54,"type":55,"text":56},"https:\u002F\u002Faws.amazon.com\u002Flambda\u002F","Text","AWS Lambda",{"link":58,"type":55,"text":59},"https:\u002F\u002Faws.amazon.com\u002Fserverless\u002F","Serverless on AWS",{"title":9,"description":10,"link":11},"msk-stream-processing","msk-stream-processing.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-stream-processing",{"title":65,"order":66,"callout":67,"time":68,"path":69,"id":70,"link":71},"Deep dive into Lambda ESM for MSK",50,{"title":9,"description":10,"link":11},"1 min","msk-esm-deep-dive","msk-esm-deep-dive.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-esm-deep-dive",{"title":6,"order":7,"callout":73,"time":21,"path":74,"id":75,"link":76},{"title":9,"description":10,"link":11},"msk-ordered-processing","msk-ordered-processing.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-ordered-processing",{"title":78,"order":79,"callout":80,"time":68,"path":81,"id":82,"link":83},"Scaling of consumers",54,{"title":9,"description":10,"link":11},"msk-scaling-consumers","msk-scaling-consumers.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-scaling-consumers",{"title":85,"order":86,"time":68,"path":87,"id":88,"link":89},"Authentication options for Lambda consumers",56,"msk-lambda-auth-options","msk-lambda-auth-options.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-lambda-auth-options",{"title":91,"order":92,"callout":93,"time":68,"path":94,"id":95,"link":96},"Error handling in Lambda functions",58,{"title":9,"description":10,"link":11},"msk-lambda-error-handling","msk-lambda-error-handling.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-lambda-error-handling",{"title":98,"collapsible":16,"isCollapsed":17,"content":99},"Streaming processing with KDS",[100,109,121,132,140],{"title":101,"weight":102,"learningGoals":103,"time":105,"path":106,"id":107,"link":108},"Kinesis Data Streams (KDS) - Key Concepts and Terminologies",60,[104],"Learn key concepts and terminologies of Amazon KDS","4 min","4-1-KDS-key-concepts-terms","4-1-KDS-key-concepts-terms.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-1-KDS-key-concepts-terms",{"title":110,"weight":111,"learningGoals":112,"resources":114,"time":117,"path":118,"id":119,"link":120},"Amazon Kinesis Data Streams - Integration with AWS Lambda",62,[113],"Amazon KDS integration with AWS Lambda for Serverless stream processing",[115,116],"https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Finvocation-eventsourcemapping.html#invocation-eventsourcemapping-batching","https:\u002F\u002Faws.amazon.com\u002Fblogs\u002Fcompute\u002Fnew-aws-lambda-scaling-controls-for-kinesis-and-dynamodb-event-sources\u002F","5 min","4-2-KDS-lambda-intg","4-2-KDS-lambda-intg.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-2-KDS-lambda-intg",{"title":122,"weight":123,"learningGoals":124,"resources":126,"time":128,"path":129,"id":130,"link":131},"Amazon KDS - Error handling with Lambda ESM",64,[125],"How to handle errors when configuring Amazon KDS as Lambda ESM",[127],"https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Fwith-kinesis.html","3 min","4-3-KDS-lambda-error-handling","4-3-KDS-lambda-error-handling.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-3-KDS-lambda-error-handling",{"title":133,"weight":134,"learningGoals":135,"time":117,"path":137,"id":138,"link":139},"Kinesis Data Streams Scaling- Key Concepts and Terminologies",66,[136],"Learn key concepts and terminologies of Amazon KDS Scaling","4-4-KDS-scaling","4-4-KDS-scaling.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-4-KDS-scaling",{"title":141,"weight":142,"learningGoals":143,"time":117,"path":145,"id":146,"link":147},"Kinesis Data Streams Monitoring",68,[144],"Learn Amazon KDS & Lambda monitoring methodologies","4-5-KDS-monitoring","4-5-KDS-monitoring.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-5-KDS-monitoring","LIST",[150,151,152,153],"content\u002Fcontributors\u002Fmithun-mallick.json","content\u002Fcontributors\u002Fkevin-schwarz.json","content\u002Fcontributors\u002Fnaren-sithambaram.json","content\u002Fcontributors\u002Fprasanna-s-krishnan.json","Serverless Stream Processing with Lambda",1792578874643]