[{"data":1,"prerenderedAt":158},["ShallowReactive",2],{"content-svc-lambda-guides-serverless-stream-processing-with-lambda-msk-stream-processing":3},{"markdown":4,"frontMatterAttributes":5,"bodyRaw":21,"menu":22,"menuType":151,"isCollapseableMenu":25,"nextDocItem":65,"previousDocItem":44,"contributorPaths":152,"contentName":157,"slug":64,"readingTime":30},"\u003Cp>Amazon Managed Streaming for Kafka (MSK) is the managed service for Apache Kafka. Customers can migrate their self-managed Apache Kafka clusters to MSK and take advantage of the managed service features such as patch management, instance scaling, security and monitoring that are built in. MSK integration with Lambda provides customers with a serverless way to process data in their Kafka topics. Lambda service manages the pollers that read from the Kafka topic, perform batching and invoke the target Lambda function. Customers can focus on the core business needed to process the messages.\u003C\u002Fp>\n\u003Cp>Lambda also supports self-managed Apache Kafka. Lambda processes messages the same way for self-managed Kafka as it does for MSK. Please refer to the \u003Ca href=\"https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Fwith-kafka.html\">documentation\u003C\u002Fa> for setting up self-managed Kafka with Lambda.\u003C\u002Fp>\n\u003Cp>Lambda works with Confluent managed Kafka as well. There are two ways in which Lambda can be invoked from Confluent Kafka. Refer to the following blogs around setting up Lambda with Confluent Kafka\u003C\u002Fp>\n\u003Cul>\n\u003Cli>\u003Ca href=\"https:\u002F\u002Faws.amazon.com\u002Fblogs\u002Fapn\u002Fnetwork-access-patterns-of-aws-lambda-for-confluent-cloud\u002F\">Network access patterns of AWS Lambda for Confluent Cloud\u003C\u002Fa>\u003C\u002Fli>\n\u003Cli>\u003Ca href=\"https:\u002F\u002Fwww.confluent.io\u002Fblog\u002Ftwo-ways-to-build-event-driven-serverless-applications\u002F\">Two ways to build event driven architectures\u003C\u002Fa>\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>\u003Cstrong>Topics\u003C\u002Fstrong>\u003C\u002Fp>\n\u003Cul>\n\u003Cli>\u003Ca href=\"msk-esm-deep-dive\">Diving deep into Kafka Event source mapping\u003C\u002Fa>\u003C\u002Fli>\n\u003Cli>\u003Ca href=\"msk-ordered-processing\">Maintaining ordering of messages\u003C\u002Fa>\u003C\u002Fli>\n\u003Cli>\u003Ca href=\"msk-scaling-consumers\">Scaling of consumers\u003C\u002Fa>\u003C\u002Fli>\n\u003Cli>\u003Ca href=\"networking-vpc\">Setting up Lambda with different authentication options\u003C\u002Fa>\u003C\u002Fli>\n\u003Cli>\u003Ca href=\"error-handling\">Error handling\u003C\u002Fa>\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>For a more deep dive learning journey follow the Kafka Lambda learning \u003Ca href=\"https:\u002F\u002Fserverlessland.com\u002Fcontent\u002Fguides\u002Flambda-kafka\u002Fintroduction\">guide\u003C\u002Fa>.\u003C\u002Fp>\n",{"title":6,"order":7,"youtubeVideoID":8,"resources":9,"callout":17},"MSK as event source for Lambda",51,"JKvIypfEiok",[10,14],{"link":11,"type":12,"text":13},"https:\u002F\u002Faws.amazon.com\u002Flambda\u002F","Text","AWS Lambda",{"link":15,"type":12,"text":16},"https:\u002F\u002Faws.amazon.com\u002Fserverless\u002F","Serverless on AWS",{"title":18,"description":19,"link":20},"Follow the Kafka with Lambda learning guide","Refer to the deep dive learning guide for using Kafka with Lambda","https:\u002F\u002Fserverlessland.com\u002Fcontent\u002Fguides\u002Flambda-kafka\u002Fintroduction","Amazon Managed Streaming for Kafka (MSK) is the managed service for Apache Kafka. Customers can migrate their self-managed Apache Kafka clusters to MSK and take advantage of the managed service features such as patch management, instance scaling, security and monitoring that are built in. MSK integration with Lambda provides customers with a serverless way to process data in their Kafka topics. Lambda service manages the pollers that read from the Kafka topic, perform batching and invoke the target Lambda function. Customers can focus on the core business needed to process the messages.\n\nLambda also supports self-managed Apache Kafka. Lambda processes messages the same way for self-managed Kafka as it does for MSK. Please refer to the [documentation](https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Fwith-kafka.html) for setting up self-managed Kafka with Lambda.\n\nLambda works with Confluent managed Kafka as well. There are two ways in which Lambda can be invoked from Confluent Kafka. Refer to the following blogs around setting up Lambda with Confluent Kafka\n\n- [Network access patterns of AWS Lambda for Confluent Cloud](https:\u002F\u002Faws.amazon.com\u002Fblogs\u002Fapn\u002Fnetwork-access-patterns-of-aws-lambda-for-confluent-cloud\u002F)\n- [Two ways to build event driven architectures](https:\u002F\u002Fwww.confluent.io\u002Fblog\u002Ftwo-ways-to-build-event-driven-serverless-applications\u002F)\n\n**Topics**\n\n- [Diving deep into Kafka Event source mapping](msk-esm-deep-dive)\n- [Maintaining ordering of messages](msk-ordered-processing)\n- [Scaling of consumers](msk-scaling-consumers)\n- [Setting up Lambda with different authentication options](networking-vpc)\n- [Error handling](error-handling)\n\nFor a more deep dive learning journey follow the Kafka Lambda learning [guide](https:\u002F\u002Fserverlessland.com\u002Fcontent\u002Fguides\u002Flambda-kafka\u002Fintroduction).\n",[23,54,100],{"title":24,"collapsible":25,"isCollapsed":26,"content":27},"Introduction",false,true,[28,34,44],{"title":24,"order":29,"time":30,"path":31,"id":32,"link":33},1,"2 min","1-introduction","1-introduction.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F1-introduction",{"title":35,"order":36,"callout":37,"time":30,"path":41,"id":42,"link":43},"Getting started with streaming data",2,{"title":38,"description":39,"link":40},"Watch the video","This video will help you get started with Amazon Kinesis Data Streams, a massively scalable, durable, and low-cost streaming data service that enables you to continuously capture gigabytes of data per second from hundreds of thousands of sources, such as website clickstreams, database event streams, financial transactions, social media feeds, IT logs, and location-tracking events.","https:\u002F\u002Fwww.youtube.com\u002Fwatch?v=1I1DcJvmd4w","2-getting-started","2-getting-started.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F2-getting-started",{"title":45,"order":46,"callout":47,"time":30,"path":51,"id":52,"link":53},"Event source mapping for Lambda",3,{"title":48,"description":49,"link":50},"Read the docs","Refer to the AWS Lambda documentation for detailed explanations of Lambda event source mappings","https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Finvocation-eventsourcemapping.html","3-event-source-mapping","3-event-source-mapping.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F3-event-source-mapping",{"title":55,"collapsible":25,"isCollapsed":25,"content":56},"Streaming processing with MSK",[57,65,73,80,87,93],{"title":6,"order":7,"youtubeVideoID":8,"resources":58,"callout":61,"time":30,"path":62,"id":63,"link":64},[59,60],{"link":11,"type":12,"text":13},{"link":15,"type":12,"text":16},{"title":18,"description":19,"link":20},"msk-stream-processing","msk-stream-processing.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-stream-processing",{"title":66,"order":67,"callout":68,"time":69,"path":70,"id":71,"link":72},"Deep dive into Lambda ESM for MSK",50,{"title":18,"description":19,"link":20},"1 min","msk-esm-deep-dive","msk-esm-deep-dive.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-esm-deep-dive",{"title":74,"order":75,"callout":76,"time":30,"path":77,"id":78,"link":79},"Ordered processing of messages",52,{"title":18,"description":19,"link":20},"msk-ordered-processing","msk-ordered-processing.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-ordered-processing",{"title":81,"order":82,"callout":83,"time":69,"path":84,"id":85,"link":86},"Scaling of consumers",54,{"title":18,"description":19,"link":20},"msk-scaling-consumers","msk-scaling-consumers.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-scaling-consumers",{"title":88,"order":89,"time":69,"path":90,"id":91,"link":92},"Authentication options for Lambda consumers",56,"msk-lambda-auth-options","msk-lambda-auth-options.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-lambda-auth-options",{"title":94,"order":95,"callout":96,"time":69,"path":97,"id":98,"link":99},"Error handling in Lambda functions",58,{"title":18,"description":19,"link":20},"msk-lambda-error-handling","msk-lambda-error-handling.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-lambda-error-handling",{"title":101,"collapsible":25,"isCollapsed":26,"content":102},"Streaming processing with KDS",[103,112,124,135,143],{"title":104,"weight":105,"learningGoals":106,"time":108,"path":109,"id":110,"link":111},"Kinesis Data Streams (KDS) - Key Concepts and Terminologies",60,[107],"Learn key concepts and terminologies of Amazon KDS","4 min","4-1-KDS-key-concepts-terms","4-1-KDS-key-concepts-terms.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-1-KDS-key-concepts-terms",{"title":113,"weight":114,"learningGoals":115,"resources":117,"time":120,"path":121,"id":122,"link":123},"Amazon Kinesis Data Streams - Integration with AWS Lambda",62,[116],"Amazon KDS integration with AWS Lambda for Serverless stream processing",[118,119],"https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Finvocation-eventsourcemapping.html#invocation-eventsourcemapping-batching","https:\u002F\u002Faws.amazon.com\u002Fblogs\u002Fcompute\u002Fnew-aws-lambda-scaling-controls-for-kinesis-and-dynamodb-event-sources\u002F","5 min","4-2-KDS-lambda-intg","4-2-KDS-lambda-intg.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-2-KDS-lambda-intg",{"title":125,"weight":126,"learningGoals":127,"resources":129,"time":131,"path":132,"id":133,"link":134},"Amazon KDS - Error handling with Lambda ESM",64,[128],"How to handle errors when configuring Amazon KDS as Lambda ESM",[130],"https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Fwith-kinesis.html","3 min","4-3-KDS-lambda-error-handling","4-3-KDS-lambda-error-handling.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-3-KDS-lambda-error-handling",{"title":136,"weight":137,"learningGoals":138,"time":120,"path":140,"id":141,"link":142},"Kinesis Data Streams Scaling- Key Concepts and Terminologies",66,[139],"Learn key concepts and terminologies of Amazon KDS Scaling","4-4-KDS-scaling","4-4-KDS-scaling.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-4-KDS-scaling",{"title":144,"weight":145,"learningGoals":146,"time":120,"path":148,"id":149,"link":150},"Kinesis Data Streams Monitoring",68,[147],"Learn Amazon KDS & Lambda monitoring methodologies","4-5-KDS-monitoring","4-5-KDS-monitoring.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-5-KDS-monitoring","LIST",[153,154,155,156],"content\u002Fcontributors\u002Fmithun-mallick.json","content\u002Fcontributors\u002Fkevin-schwarz.json","content\u002Fcontributors\u002Fnaren-sithambaram.json","content\u002Fcontributors\u002Fprasanna-s-krishnan.json","Serverless Stream Processing with Lambda",1792578874679]