[{"data":1,"prerenderedAt":160},["ShallowReactive",2],{"content-svc-lambda-guides-serverless-stream-processing-with-lambda-4-KDS-integration-with-lambda":3},{"markdown":4,"frontMatterAttributes":5,"bodyRaw":10,"menu":11,"menuType":151,"isCollapseableMenu":14,"nextDocItem":17,"previousDocItem":152,"contributorPaths":153,"contentName":158,"slug":159,"readingTime":69},"\u003Cp>\u003Ca href=\"https:\u002F\u002Faws.amazon.com\u002Fkinesis\u002Fdata-streams\u002F\">Amazon Kinesis Data Streams\u003C\u002Fa> (KDS) is a serverless data streaming service that makes it easy to capture, process, and store data streams at scale. With KDS, you can collect and process large streams of data in real time. KDS are useful for rapidly moving data off producers, continuously process the data and emitting the data to the consumer application. \u003C\u002Fp>\n\u003Cp>There are no servers to manage with Amazon Kinesis Data Streams. It is also durable, highly available and provides dedicated throughput per consumer making it a good fit for low latency applications. It enables you to continuously capture gigabytes of data per second from hundreds of thousands of sources, such as website click streams, database event streams, financial transactions, social media feeds, IT logs, and location-tracking events.\u003C\u002Fp>\n\u003Cp>The following are the typical use case for using Kinesis Data Streams.\u003C\u002Fp>\n\u003Cul>\n\u003Cli>Accelerated log and data intake \u003C\u002Fli>\n\u003Cli>Real-time metrics and reporting\u003C\u002Fli>\n\u003Cli>Real-time data analytics\u003C\u002Fli>\n\u003Cli>Log\u002Fevent data collection\u003C\u002Fli>\n\u003Cli>Build event driven applications\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>Before you get started with integration with AWS Lambda, it is important to understand the high level architecture, key concepts and terminologies of Kinesis Data Streams.\u003C\u002Fp>\n",{"title":6,"order":7,"learningGoals":8},"Kinesis Data Stream (KDS) integration with Lambda",4,[9],"Introduction to Kinesis Data Streams","[Amazon Kinesis Data Streams](https:\u002F\u002Faws.amazon.com\u002Fkinesis\u002Fdata-streams\u002F) (KDS) is a serverless data streaming service that makes it easy to capture, process, and store data streams at scale. With KDS, you can collect and process large streams of data in real time. KDS are useful for rapidly moving data off producers, continuously process the data and emitting the data to the consumer application. \n\nThere are no servers to manage with Amazon Kinesis Data Streams. It is also durable, highly available and provides dedicated throughput per consumer making it a good fit for low latency applications. It enables you to continuously capture gigabytes of data per second from hundreds of thousands of sources, such as website click streams, database event streams, financial transactions, social media feeds, IT logs, and location-tracking events.\n\nThe following are the typical use case for using Kinesis Data Streams.\n* Accelerated log and data intake \n* Real-time metrics and reporting\n* Real-time data analytics\n* Log\u002Fevent data collection\n* Build event driven applications\n\nBefore you get started with integration with AWS Lambda, it is important to understand the high level architecture, key concepts and terminologies of Kinesis Data Streams.\n",[12,43,100],{"title":13,"collapsible":14,"isCollapsed":15,"content":16},"Introduction",false,true,[17,23,33],{"title":13,"order":18,"time":19,"path":20,"id":21,"link":22},1,"2 min","1-introduction","1-introduction.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F1-introduction",{"title":24,"order":25,"callout":26,"time":19,"path":30,"id":31,"link":32},"Getting started with streaming data",2,{"title":27,"description":28,"link":29},"Watch the video","This video will help you get started with Amazon Kinesis Data Streams, a massively scalable, durable, and low-cost streaming data service that enables you to continuously capture gigabytes of data per second from hundreds of thousands of sources, such as website clickstreams, database event streams, financial transactions, social media feeds, IT logs, and location-tracking events.","https:\u002F\u002Fwww.youtube.com\u002Fwatch?v=1I1DcJvmd4w","2-getting-started","2-getting-started.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F2-getting-started",{"title":34,"order":35,"callout":36,"time":19,"path":40,"id":41,"link":42},"Event source mapping for Lambda",3,{"title":37,"description":38,"link":39},"Read the docs","Refer to the AWS Lambda documentation for detailed explanations of Lambda event source mappings","https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Finvocation-eventsourcemapping.html","3-event-source-mapping","3-event-source-mapping.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F3-event-source-mapping",{"title":44,"collapsible":14,"isCollapsed":15,"content":45},"Streaming processing with MSK",[46,65,73,80,87,93],{"title":47,"order":48,"youtubeVideoID":49,"resources":50,"callout":58,"time":19,"path":62,"id":63,"link":64},"MSK as event source for Lambda",51,"JKvIypfEiok",[51,55],{"link":52,"type":53,"text":54},"https:\u002F\u002Faws.amazon.com\u002Flambda\u002F","Text","AWS Lambda",{"link":56,"type":53,"text":57},"https:\u002F\u002Faws.amazon.com\u002Fserverless\u002F","Serverless on AWS",{"title":59,"description":60,"link":61},"Follow the Kafka with Lambda learning guide","Refer to the deep dive learning guide for using Kafka with Lambda","https:\u002F\u002Fserverlessland.com\u002Fcontent\u002Fguides\u002Flambda-kafka\u002Fintroduction","msk-stream-processing","msk-stream-processing.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-stream-processing",{"title":66,"order":67,"callout":68,"time":69,"path":70,"id":71,"link":72},"Deep dive into Lambda ESM for MSK",50,{"title":59,"description":60,"link":61},"1 min","msk-esm-deep-dive","msk-esm-deep-dive.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-esm-deep-dive",{"title":74,"order":75,"callout":76,"time":19,"path":77,"id":78,"link":79},"Ordered processing of messages",52,{"title":59,"description":60,"link":61},"msk-ordered-processing","msk-ordered-processing.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-ordered-processing",{"title":81,"order":82,"callout":83,"time":69,"path":84,"id":85,"link":86},"Scaling of consumers",54,{"title":59,"description":60,"link":61},"msk-scaling-consumers","msk-scaling-consumers.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-scaling-consumers",{"title":88,"order":89,"time":69,"path":90,"id":91,"link":92},"Authentication options for Lambda consumers",56,"msk-lambda-auth-options","msk-lambda-auth-options.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-lambda-auth-options",{"title":94,"order":95,"callout":96,"time":69,"path":97,"id":98,"link":99},"Error handling in Lambda functions",58,{"title":59,"description":60,"link":61},"msk-lambda-error-handling","msk-lambda-error-handling.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-lambda-error-handling",{"title":101,"collapsible":14,"isCollapsed":15,"content":102},"Streaming processing with KDS",[103,112,124,135,143],{"title":104,"weight":105,"learningGoals":106,"time":108,"path":109,"id":110,"link":111},"Kinesis Data Streams (KDS) - Key Concepts and Terminologies",60,[107],"Learn key concepts and terminologies of Amazon KDS","4 min","4-1-KDS-key-concepts-terms","4-1-KDS-key-concepts-terms.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-1-KDS-key-concepts-terms",{"title":113,"weight":114,"learningGoals":115,"resources":117,"time":120,"path":121,"id":122,"link":123},"Amazon Kinesis Data Streams - Integration with AWS Lambda",62,[116],"Amazon KDS integration with AWS Lambda for Serverless stream processing",[118,119],"https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Finvocation-eventsourcemapping.html#invocation-eventsourcemapping-batching","https:\u002F\u002Faws.amazon.com\u002Fblogs\u002Fcompute\u002Fnew-aws-lambda-scaling-controls-for-kinesis-and-dynamodb-event-sources\u002F","5 min","4-2-KDS-lambda-intg","4-2-KDS-lambda-intg.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-2-KDS-lambda-intg",{"title":125,"weight":126,"learningGoals":127,"resources":129,"time":131,"path":132,"id":133,"link":134},"Amazon KDS - Error handling with Lambda ESM",64,[128],"How to handle errors when configuring Amazon KDS as Lambda ESM",[130],"https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Fwith-kinesis.html","3 min","4-3-KDS-lambda-error-handling","4-3-KDS-lambda-error-handling.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-3-KDS-lambda-error-handling",{"title":136,"weight":137,"learningGoals":138,"time":120,"path":140,"id":141,"link":142},"Kinesis Data Streams Scaling- Key Concepts and Terminologies",66,[139],"Learn key concepts and terminologies of Amazon KDS Scaling","4-4-KDS-scaling","4-4-KDS-scaling.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-4-KDS-scaling",{"title":144,"weight":145,"learningGoals":146,"time":120,"path":148,"id":149,"link":150},"Kinesis Data Streams Monitoring",68,[147],"Learn Amazon KDS & Lambda monitoring methodologies","4-5-KDS-monitoring","4-5-KDS-monitoring.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-5-KDS-monitoring","LIST",null,[154,155,156,157],"content\u002Fcontributors\u002Fmithun-mallick.json","content\u002Fcontributors\u002Fkevin-schwarz.json","content\u002Fcontributors\u002Fnaren-sithambaram.json","content\u002Fcontributors\u002Fprasanna-s-krishnan.json","Serverless Stream Processing with Lambda","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-KDS-integration-with-lambda",1792665273576]