[{"data":1,"prerenderedAt":155},["ShallowReactive",2],{"content-svc-lambda-guides-serverless-stream-processing-with-lambda-2-getting-started":3},{"markdown":4,"frontMatterAttributes":5,"bodyRaw":12,"menu":13,"menuType":148,"isCollapseableMenu":16,"nextDocItem":29,"previousDocItem":18,"contributorPaths":149,"contentName":154,"slug":28,"readingTime":20},"\u003Cp>Getting started processing Amazon Managed Streaming for Apache Kafka (MSK) and Kinesis Data Streams (KDS) serverless streams with Lambda is easy. Once you've made a decision on which AWS streaming service to use, and you've deployed either KDS or MSK streaming infrastructure to receive data from your source, then simply configure KDS or MSK as the trigger for AWS Lambda. You can use Lambda to process your streaming data and land data to your destination.\u003C\u002Fp>\n\u003Cp>\u003Cimg src=\"\u002Fassets\u002Fexternal\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fassets\u002Fstreaming_steps.png\" alt=\"Streaming data steps\">\u003C\u002Fp>\n\u003Cp>Both KDS and MSK are great solutions for streaming data in the cloud, but there are a few key differences to consider when you choose a solution. KDS is a completely managed regional (multi-AZ) cloud native streaming service with on demand pay as you go pricing. MSK is a fully managed and highly available fully open-source compatible Apache Kafka.\u003C\u002Fp>\n\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>\u003C\u002Fth>\n\u003Cth>KDS\u003C\u002Fth>\n\u003Cth>MSK\u003C\u002Fth>\n\u003C\u002Ftr>\n\u003C\u002Fthead>\n\u003Ctbody>\u003Ctr>\n\u003Ctd>Latency\u003C\u002Ftd>\n\u003Ctd>Readable immediately after written. Propagation delay equivalent to polling interval. Enhanced Fan out available.\u003C\u002Ftd>\n\u003Ctd>Readable immediately after written.\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>Ordering\u003C\u002Ftd>\n\u003Ctd>Ordered per shard\u003C\u002Ftd>\n\u003Ctd>Ordered per topic\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>Cost\u003C\u002Ftd>\n\u003Ctd>On demand billing based per stream\u002Fhr, per GB ingested, per GB retrieved, and data stored.\u003C\u002Ftd>\n\u003Ctd>Billing by instance type per hour per instance, broker storage, and VPC connectivity options.\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>Retention\u003C\u002Ftd>\n\u003Ctd>Default 24 hours extendable up to 365 days.\u003C\u002Ftd>\n\u003Ctd>Configurable retention based on storage available on a broker.\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003C\u002Ftbody>\u003C\u002Ftable>\n\u003Cp>AWS Lambda integrates natively with Amazon Kinesis and Kafka (AWS MSK Provisioned, MSK serverless, self-managed Kafka clusters, and Confluent Cloud) as a consumer to process ingested data. Customers can invoke Lambda functions from messages in Kafka topics or Kinesis data streams to integrate into downstream serverless workflows. Lambda service manages the provisioning of the polling infrastructure, provides settings for batching, error handling and scaling. In both cases, customers are guaranteed ordered processing of messages.\u003C\u002Fp>\n\u003Cp>The resources below will help you think about how to setup your streaming infrastructure and the next topic on Lambda Event Source Mapping will help you understand how Lambda helps you process your streaming data.\u003C\u002Fp>\n\u003Col>\n\u003Cli>\u003Ca href=\"https:\u002F\u002Faws.amazon.com\u002Fblogs\u002Farchitecture\u002Fserverless-stream-based-processing-for-real-time-insights\u002F\">Serverless Stream based processing for real time insights\u003C\u002Fa>\u003C\u002Fli>\n\u003Cli>\u003Ca href=\"https:\u002F\u002Fdocs.aws.amazon.com\u002Fwellarchitected\u002Flatest\u002Fserverless-applications-lens\u002Fstreaming-processing.html\">AWS Well Architected serverless lens for stream processing\u003C\u002Fa>\u003C\u002Fli>\n\u003Cli>\u003Ca href=\"https:\u002F\u002Fwww.youtube.com\u002Fwatch?v=TJS19EuzH2k\">Should My Startup use Kinesis or MSK?\u003C\u002Fa>\u003C\u002Fli>\n\u003Cli>\u003Ca href=\"https:\u002F\u002Fcatalog.us-east-1.prod.workshops.aws\u002Fworkshops\u002F76d4b4eb-bff7-40c6-a925-7f101ad3bd43\u002Fen-US\">Workshop: Serverless Data Processing on AWS\u003C\u002Fa>\u003C\u002Fli>\n\u003C\u002Fol>\n",{"title":6,"order":7,"callout":8},"Getting started with streaming data",2,{"title":9,"description":10,"link":11},"Watch the video","This video will help you get started with Amazon Kinesis Data Streams, a massively scalable, durable, and low-cost streaming data service that enables you to continuously capture gigabytes of data per second from hundreds of thousands of sources, such as website clickstreams, database event streams, financial transactions, social media feeds, IT logs, and location-tracking events.","https:\u002F\u002Fwww.youtube.com\u002Fwatch?v=1I1DcJvmd4w","Getting started processing Amazon Managed Streaming for Apache Kafka (MSK) and Kinesis Data Streams (KDS) serverless streams with Lambda is easy. Once you've made a decision on which AWS streaming service to use, and you've deployed either KDS or MSK streaming infrastructure to receive data from your source, then simply configure KDS or MSK as the trigger for AWS Lambda. You can use Lambda to process your streaming data and land data to your destination.\n\n![Streaming data steps](\u002Fassets\u002Fexternal\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fassets\u002Fstreaming_steps.png)\n\nBoth KDS and MSK are great solutions for streaming data in the cloud, but there are a few key differences to consider when you choose a solution. KDS is a completely managed regional (multi-AZ) cloud native streaming service with on demand pay as you go pricing. MSK is a fully managed and highly available fully open-source compatible Apache Kafka.\n\n|           | KDS                                                                                                               | MSK                                                                                           |\n| --------- | ----------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- |\n| Latency   | Readable immediately after written. Propagation delay equivalent to polling interval. Enhanced Fan out available. | Readable immediately after written.                                                          |\n| Ordering  | Ordered per shard                                                                                                 | Ordered per topic                                                                             |\n| Cost      | On demand billing based per stream\u002Fhr, per GB ingested, per GB retrieved, and data stored.                        | Billing by instance type per hour per instance, broker storage, and VPC connectivity options. |\n| Retention | Default 24 hours extendable up to 365 days.                                                                       | Configurable retention based on storage available on a broker.                                |\n\nAWS Lambda integrates natively with Amazon Kinesis and Kafka (AWS MSK Provisioned, MSK serverless, self-managed Kafka clusters, and Confluent Cloud) as a consumer to process ingested data. Customers can invoke Lambda functions from messages in Kafka topics or Kinesis data streams to integrate into downstream serverless workflows. Lambda service manages the provisioning of the polling infrastructure, provides settings for batching, error handling and scaling. In both cases, customers are guaranteed ordered processing of messages.\n\nThe resources below will help you think about how to setup your streaming infrastructure and the next topic on Lambda Event Source Mapping will help you understand how Lambda helps you process your streaming data.\n\n1. [Serverless Stream based processing for real time insights](https:\u002F\u002Faws.amazon.com\u002Fblogs\u002Farchitecture\u002Fserverless-stream-based-processing-for-real-time-insights\u002F)\n1. [AWS Well Architected serverless lens for stream processing](https:\u002F\u002Fdocs.aws.amazon.com\u002Fwellarchitected\u002Flatest\u002Fserverless-applications-lens\u002Fstreaming-processing.html)\n1. [Should My Startup use Kinesis or MSK?](https:\u002F\u002Fwww.youtube.com\u002Fwatch?v=TJS19EuzH2k)\n1. [Workshop: Serverless Data Processing on AWS](https:\u002F\u002Fcatalog.us-east-1.prod.workshops.aws\u002Fworkshops\u002F76d4b4eb-bff7-40c6-a925-7f101ad3bd43\u002Fen-US)\n",[14,39,97],{"title":15,"collapsible":16,"isCollapsed":16,"content":17},"Introduction",false,[18,24,29],{"title":15,"order":19,"time":20,"path":21,"id":22,"link":23},1,"2 min","1-introduction","1-introduction.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F1-introduction",{"title":6,"order":7,"callout":25,"time":20,"path":26,"id":27,"link":28},{"title":9,"description":10,"link":11},"2-getting-started","2-getting-started.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F2-getting-started",{"title":30,"order":31,"callout":32,"time":20,"path":36,"id":37,"link":38},"Event source mapping for Lambda",3,{"title":33,"description":34,"link":35},"Read the docs","Refer to the AWS Lambda documentation for detailed explanations of Lambda event source mappings","https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Finvocation-eventsourcemapping.html","3-event-source-mapping","3-event-source-mapping.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F3-event-source-mapping",{"title":40,"collapsible":16,"isCollapsed":41,"content":42},"Streaming processing with MSK",true,[43,62,70,77,84,90],{"title":44,"order":45,"youtubeVideoID":46,"resources":47,"callout":55,"time":20,"path":59,"id":60,"link":61},"MSK as event source for Lambda",51,"JKvIypfEiok",[48,52],{"link":49,"type":50,"text":51},"https:\u002F\u002Faws.amazon.com\u002Flambda\u002F","Text","AWS Lambda",{"link":53,"type":50,"text":54},"https:\u002F\u002Faws.amazon.com\u002Fserverless\u002F","Serverless on AWS",{"title":56,"description":57,"link":58},"Follow the Kafka with Lambda learning guide","Refer to the deep dive learning guide for using Kafka with Lambda","https:\u002F\u002Fserverlessland.com\u002Fcontent\u002Fguides\u002Flambda-kafka\u002Fintroduction","msk-stream-processing","msk-stream-processing.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-stream-processing",{"title":63,"order":64,"callout":65,"time":66,"path":67,"id":68,"link":69},"Deep dive into Lambda ESM for MSK",50,{"title":56,"description":57,"link":58},"1 min","msk-esm-deep-dive","msk-esm-deep-dive.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-esm-deep-dive",{"title":71,"order":72,"callout":73,"time":20,"path":74,"id":75,"link":76},"Ordered processing of messages",52,{"title":56,"description":57,"link":58},"msk-ordered-processing","msk-ordered-processing.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-ordered-processing",{"title":78,"order":79,"callout":80,"time":66,"path":81,"id":82,"link":83},"Scaling of consumers",54,{"title":56,"description":57,"link":58},"msk-scaling-consumers","msk-scaling-consumers.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-scaling-consumers",{"title":85,"order":86,"time":66,"path":87,"id":88,"link":89},"Authentication options for Lambda consumers",56,"msk-lambda-auth-options","msk-lambda-auth-options.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-lambda-auth-options",{"title":91,"order":92,"callout":93,"time":66,"path":94,"id":95,"link":96},"Error handling in Lambda functions",58,{"title":56,"description":57,"link":58},"msk-lambda-error-handling","msk-lambda-error-handling.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-lambda-error-handling",{"title":98,"collapsible":16,"isCollapsed":41,"content":99},"Streaming processing with KDS",[100,109,121,132,140],{"title":101,"weight":102,"learningGoals":103,"time":105,"path":106,"id":107,"link":108},"Kinesis Data Streams (KDS) - Key Concepts and Terminologies",60,[104],"Learn key concepts and terminologies of Amazon KDS","4 min","4-1-KDS-key-concepts-terms","4-1-KDS-key-concepts-terms.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-1-KDS-key-concepts-terms",{"title":110,"weight":111,"learningGoals":112,"resources":114,"time":117,"path":118,"id":119,"link":120},"Amazon Kinesis Data Streams - Integration with AWS Lambda",62,[113],"Amazon KDS integration with AWS Lambda for Serverless stream processing",[115,116],"https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Finvocation-eventsourcemapping.html#invocation-eventsourcemapping-batching","https:\u002F\u002Faws.amazon.com\u002Fblogs\u002Fcompute\u002Fnew-aws-lambda-scaling-controls-for-kinesis-and-dynamodb-event-sources\u002F","5 min","4-2-KDS-lambda-intg","4-2-KDS-lambda-intg.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-2-KDS-lambda-intg",{"title":122,"weight":123,"learningGoals":124,"resources":126,"time":128,"path":129,"id":130,"link":131},"Amazon KDS - Error handling with Lambda ESM",64,[125],"How to handle errors when configuring Amazon KDS as Lambda ESM",[127],"https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Fwith-kinesis.html","3 min","4-3-KDS-lambda-error-handling","4-3-KDS-lambda-error-handling.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-3-KDS-lambda-error-handling",{"title":133,"weight":134,"learningGoals":135,"time":117,"path":137,"id":138,"link":139},"Kinesis Data Streams Scaling- Key Concepts and Terminologies",66,[136],"Learn key concepts and terminologies of Amazon KDS Scaling","4-4-KDS-scaling","4-4-KDS-scaling.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-4-KDS-scaling",{"title":141,"weight":142,"learningGoals":143,"time":117,"path":145,"id":146,"link":147},"Kinesis Data Streams Monitoring",68,[144],"Learn Amazon KDS & Lambda monitoring methodologies","4-5-KDS-monitoring","4-5-KDS-monitoring.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-5-KDS-monitoring","LIST",[150,151,152,153],"content\u002Fcontributors\u002Fmithun-mallick.json","content\u002Fcontributors\u002Fkevin-schwarz.json","content\u002Fcontributors\u002Fnaren-sithambaram.json","content\u002Fcontributors\u002Fprasanna-s-krishnan.json","Serverless Stream Processing with Lambda",1792578874433]