[{"data":1,"prerenderedAt":155},["ShallowReactive",2],{"content-svc-lambda-guides-serverless-stream-processing-with-lambda-1-introduction":3},{"markdown":4,"frontMatterAttributes":5,"bodyRaw":8,"menu":9,"menuType":147,"isCollapseableMenu":11,"nextDocItem":18,"previousDocItem":148,"contributorPaths":149,"contentName":154,"slug":17,"readingTime":14},"\u003Cp>Real-time data streaming has rapidly expanded over the past few years. This is due to digital and traditional businesses needing access to real time data sources to meet customer demands. Streaming data sources are constantly evolving and include internet of things (IoT) sensors that monitor equipment to improve performance and uptime, online gaming companies that use player-game interactions to offer incentives and dynamic experiences, monitoring application logs, mobile apps, click streams, and social media interactions. Capturing and reasoning over real-time data has become a key differentiator for innovators across all types of businesses.\u003C\u002Fp>\n\u003Cp>\u003Cimg src=\"\u002Fassets\u002Fexternal\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fassets\u002Fstreaming-data-sources.png\" alt=\"Streaming data sources\">\u003C\u002Fp>\n\u003Cp>\u003Cstrong>What is streaming data anyway?\u003C\u002Fstrong>\u003C\u002Fp>\n\u003Cp>Streaming data is emitted in a continuous, incremental manner, often at incredibly high volumes and needing to be processed immediately. Real time applications collect streaming data at scale for real-time data movement, real-time analytics, and event stream processing. This data needs to be processed sequentially and incrementally on a record-by-record basis or over sliding time windows, and used for a wide variety of analytics including correlations, aggregations, filtering, and sampling. Stream-based processing can offer real-time actionable insights compared to batch oriented-processes, which often only generates insights after hours, days, or even weeks have passed.\u003C\u002Fp>\n\u003Cp>Amazon Managed Streaming for Apache Kafka (MSK) and Kinesis Data Streams (KDS) are the AWS data streaming services that customers can choose to send streaming data, process, and consume in applications with ultra-low latency. MSK and KDS allow customers to build event driven applications that rely on identifying the trends from their business data to trigger business events. Both of these services preserve the order in which messages are received allowing customers to process them either on the basis of processing time or event time.\u003C\u002Fp>\n\u003Cp>Watch the following video for a better understanding of Kinesis and MSK streaming services.\u003C\u002Fp>\n\u003Ciframe\n    width=\"640\"\n    height=\"480\"\n    src=\"https:\u002F\u002Fwww.youtube.com\u002Fembed\u002FabaVSFFdU-c\"\n    frameborder=\"0\"\n    allow=\"autoplay; encrypted-media\"\n    allowfullscreen\n>\n\u003C\u002Fiframe>\n",{"title":6,"order":7},"Introduction",1,"Real-time data streaming has rapidly expanded over the past few years. This is due to digital and traditional businesses needing access to real time data sources to meet customer demands. Streaming data sources are constantly evolving and include internet of things (IoT) sensors that monitor equipment to improve performance and uptime, online gaming companies that use player-game interactions to offer incentives and dynamic experiences, monitoring application logs, mobile apps, click streams, and social media interactions. Capturing and reasoning over real-time data has become a key differentiator for innovators across all types of businesses.\n\n![Streaming data sources](\u002Fassets\u002Fexternal\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fassets\u002Fstreaming-data-sources.png)\n\n**What is streaming data anyway?**\n\nStreaming data is emitted in a continuous, incremental manner, often at incredibly high volumes and needing to be processed immediately. Real time applications collect streaming data at scale for real-time data movement, real-time analytics, and event stream processing. This data needs to be processed sequentially and incrementally on a record-by-record basis or over sliding time windows, and used for a wide variety of analytics including correlations, aggregations, filtering, and sampling. Stream-based processing can offer real-time actionable insights compared to batch oriented-processes, which often only generates insights after hours, days, or even weeks have passed.\n\nAmazon Managed Streaming for Apache Kafka (MSK) and Kinesis Data Streams (KDS) are the AWS data streaming services that customers can choose to send streaming data, process, and consume in applications with ultra-low latency. MSK and KDS allow customers to build event driven applications that rely on identifying the trends from their business data to trigger business events. Both of these services preserve the order in which messages are received allowing customers to process them either on the basis of processing time or event time.\n\nWatch the following video for a better understanding of Kinesis and MSK streaming services.\n\n\u003Ciframe\n    width=\"640\"\n    height=\"480\"\n    src=\"https:\u002F\u002Fwww.youtube.com\u002Fembed\u002FabaVSFFdU-c\"\n    frameborder=\"0\"\n    allow=\"autoplay; encrypted-media\"\n    allowfullscreen\n>\n\u003C\u002Fiframe>\n",[10,38,96],{"title":6,"collapsible":11,"isCollapsed":11,"content":12},false,[13,18,28],{"title":6,"order":7,"time":14,"path":15,"id":16,"link":17},"2 min","1-introduction","1-introduction.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F1-introduction",{"title":19,"order":20,"callout":21,"time":14,"path":25,"id":26,"link":27},"Getting started with streaming data",2,{"title":22,"description":23,"link":24},"Watch the video","This video will help you get started with Amazon Kinesis Data Streams, a massively scalable, durable, and low-cost streaming data service that enables you to continuously capture gigabytes of data per second from hundreds of thousands of sources, such as website clickstreams, database event streams, financial transactions, social media feeds, IT logs, and location-tracking events.","https:\u002F\u002Fwww.youtube.com\u002Fwatch?v=1I1DcJvmd4w","2-getting-started","2-getting-started.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F2-getting-started",{"title":29,"order":30,"callout":31,"time":14,"path":35,"id":36,"link":37},"Event source mapping for Lambda",3,{"title":32,"description":33,"link":34},"Read the docs","Refer to the AWS Lambda documentation for detailed explanations of Lambda event source mappings","https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Finvocation-eventsourcemapping.html","3-event-source-mapping","3-event-source-mapping.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F3-event-source-mapping",{"title":39,"collapsible":11,"isCollapsed":40,"content":41},"Streaming processing with MSK",true,[42,61,69,76,83,89],{"title":43,"order":44,"youtubeVideoID":45,"resources":46,"callout":54,"time":14,"path":58,"id":59,"link":60},"MSK as event source for Lambda",51,"JKvIypfEiok",[47,51],{"link":48,"type":49,"text":50},"https:\u002F\u002Faws.amazon.com\u002Flambda\u002F","Text","AWS Lambda",{"link":52,"type":49,"text":53},"https:\u002F\u002Faws.amazon.com\u002Fserverless\u002F","Serverless on AWS",{"title":55,"description":56,"link":57},"Follow the Kafka with Lambda learning guide","Refer to the deep dive learning guide for using Kafka with Lambda","https:\u002F\u002Fserverlessland.com\u002Fcontent\u002Fguides\u002Flambda-kafka\u002Fintroduction","msk-stream-processing","msk-stream-processing.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-stream-processing",{"title":62,"order":63,"callout":64,"time":65,"path":66,"id":67,"link":68},"Deep dive into Lambda ESM for MSK",50,{"title":55,"description":56,"link":57},"1 min","msk-esm-deep-dive","msk-esm-deep-dive.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-esm-deep-dive",{"title":70,"order":71,"callout":72,"time":14,"path":73,"id":74,"link":75},"Ordered processing of messages",52,{"title":55,"description":56,"link":57},"msk-ordered-processing","msk-ordered-processing.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-ordered-processing",{"title":77,"order":78,"callout":79,"time":65,"path":80,"id":81,"link":82},"Scaling of consumers",54,{"title":55,"description":56,"link":57},"msk-scaling-consumers","msk-scaling-consumers.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-scaling-consumers",{"title":84,"order":85,"time":65,"path":86,"id":87,"link":88},"Authentication options for Lambda consumers",56,"msk-lambda-auth-options","msk-lambda-auth-options.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-lambda-auth-options",{"title":90,"order":91,"callout":92,"time":65,"path":93,"id":94,"link":95},"Error handling in Lambda functions",58,{"title":55,"description":56,"link":57},"msk-lambda-error-handling","msk-lambda-error-handling.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-lambda-error-handling",{"title":97,"collapsible":11,"isCollapsed":40,"content":98},"Streaming processing with KDS",[99,108,120,131,139],{"title":100,"weight":101,"learningGoals":102,"time":104,"path":105,"id":106,"link":107},"Kinesis Data Streams (KDS) - Key Concepts and Terminologies",60,[103],"Learn key concepts and terminologies of Amazon KDS","4 min","4-1-KDS-key-concepts-terms","4-1-KDS-key-concepts-terms.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-1-KDS-key-concepts-terms",{"title":109,"weight":110,"learningGoals":111,"resources":113,"time":116,"path":117,"id":118,"link":119},"Amazon Kinesis Data Streams - Integration with AWS Lambda",62,[112],"Amazon KDS integration with AWS Lambda for Serverless stream processing",[114,115],"https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Finvocation-eventsourcemapping.html#invocation-eventsourcemapping-batching","https:\u002F\u002Faws.amazon.com\u002Fblogs\u002Fcompute\u002Fnew-aws-lambda-scaling-controls-for-kinesis-and-dynamodb-event-sources\u002F","5 min","4-2-KDS-lambda-intg","4-2-KDS-lambda-intg.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-2-KDS-lambda-intg",{"title":121,"weight":122,"learningGoals":123,"resources":125,"time":127,"path":128,"id":129,"link":130},"Amazon KDS - Error handling with Lambda ESM",64,[124],"How to handle errors when configuring Amazon KDS as Lambda ESM",[126],"https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Fwith-kinesis.html","3 min","4-3-KDS-lambda-error-handling","4-3-KDS-lambda-error-handling.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-3-KDS-lambda-error-handling",{"title":132,"weight":133,"learningGoals":134,"time":116,"path":136,"id":137,"link":138},"Kinesis Data Streams Scaling- Key Concepts and Terminologies",66,[135],"Learn key concepts and terminologies of Amazon KDS Scaling","4-4-KDS-scaling","4-4-KDS-scaling.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-4-KDS-scaling",{"title":140,"weight":141,"learningGoals":142,"time":116,"path":144,"id":145,"link":146},"Kinesis Data Streams Monitoring",68,[143],"Learn Amazon KDS & Lambda monitoring methodologies","4-5-KDS-monitoring","4-5-KDS-monitoring.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-5-KDS-monitoring","LIST",null,[150,151,152,153],"content\u002Fcontributors\u002Fmithun-mallick.json","content\u002Fcontributors\u002Fkevin-schwarz.json","content\u002Fcontributors\u002Fnaren-sithambaram.json","content\u002Fcontributors\u002Fprasanna-s-krishnan.json","Serverless Stream Processing with Lambda",1790764516299]