[{"data":1,"prerenderedAt":92},["ShallowReactive",2],{"content-guides-lambda-kafka-introduction":3},{"markdown":4,"frontMatterAttributes":5,"bodyRaw":8,"contributorPaths":9,"menuType":11,"contentName":12,"slug":13,"readingTime":14,"menu":15,"nextDocItem":24,"previousDocItem":91},"\u003Ch3>Understanding streaming\u003C\u002Fh3>\n\u003Cp>Real-time data is critical to inform decision-making and for delivering enhanced customer experiences. Digital businesses have access to thousands of data sources that are constantly changing, including Internet of Things (IoT) sensors, application logs, mobile apps, clickstreams, and social media. Making sense and using real-time data has become a key differentiator for innovation across many industries.\u003C\u002Fp>\n\u003Cimg src=\"\u002Fassets\u002Fexternal\u002Fguides\u002Flambda-kafka\u002Fassets\u002Fimages\u002Fkafka-lambda-data-sources.png\">\n\n\u003Ch3>What is streaming data?\u003C\u002Fh3>\n\u003Cp>Streaming data is data that is generated in a nearly continuous, incremental manner, often at high volumes. It must be processed quickly in order, on a record-by-record basis or over sliding time windows to get the most value from it.\u003C\u002Fp>\n\u003Cp>Stream-based processing can offer real-time actionable analytical insights compared to batch oriented-processes, which often only generate insights after hours, days, or even weeks have passed. To learn more about streaming data, see \u003Ca href=\"https:\u002F\u002Faws.amazon.com\u002Fstreaming-data\u002F\">What is streaming data?\u003C\u002Fa>\u003C\u002Fp>\n\u003Cp>Characteristics of streaming data include:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>\u003Cstrong>High volume:\u003C\u002Fstrong> Organizations that collect clickstream data about user behavior can often reach up to \u003Cem>2-3 TB\u003C\u002Fem> every day.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Nearly continuous rather than discrete\u003C\u002Fstrong> : IoT devices like sensors collect data continuously and at regular intervals.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Ordered:\u003C\u002Fstrong> A chat application makes little sense with messages out of order.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Requires low latency processing to be valuable:\u003C\u002Fstrong> Fraud detection must be done as quickly as possible to prevent loss.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Ch3>Examples of streaming data\u003C\u002Fh3>\n\u003Cul>\n\u003Cli>Sensors in transportation vehicles, industrial equipment, and farm machinery send data to a streaming application. The application monitors performance defects any potential defects in advance, and places a spare part order automatically preventing equipment down time.\u003C\u002Fli>\n\u003Cli>A financial institution tracks changes in the stock market in real time, computes value-at-risk, and automatically rebalances portfolios based on stock price movements.\u003C\u002Fli>\n\u003Cli>A real-estate website tracks a subset of data from consumers' mobile devices and makes real-time property recommendations of properties to visit based on their geo-location.\u003C\u002Fli>\n\u003Cli>A solar power company has to maintain power throughput for its customers, or pay penalties. It implemented a streaming data application that monitors all panels in the field, and schedules service in real time, thereby minimizing the periods of low throughput from each panel and the associated penalty payouts.\u003C\u002Fli>\n\u003Cli>A media publisher streams billions of clickstream records from its online properties, aggregates and enriches the data with demographic information about users, and optimizes content placement on its site, delivering relevancy and a better experience to its audience.\u003C\u002Fli>\n\u003Cli>An online gaming company collects streaming data about player-game interactions, and feeds the data into its gaming platform. It then analyzes the data in real-time, offers incentives and dynamic experiences to engage its players.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Ch3>Comparing batch processing and stream processing\u003C\u002Fh3>\n\u003Cp>Batch processing is used to compute queries over different sets of data. It usually computes results that are derived from all the data it encompasses and enables deep analysis of large datasets. In contrast, stream processing requires ingesting a sequence of data, and incrementally updating metrics, reports, and summary statistics in response to each arriving data record. It is better suited for real-time monitoring and response functions.\u003C\u002Fp>\n\u003Cp>Many organizations build a hybrid model by combining the two approaches, and maintain a real-time layer and a batch layer. Data is first processed by a streaming data platform to extract real-time insights, and then persisted into a store like \u003Ca href=\"https:\u002F\u002Faws.amazon.com\u002Fs3\u002F\">Amazon S3\u003C\u002Fa>, where it can be transformed and loaded for a variety of batch processing use cases.\u003C\u002Fp>\n\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Batch processing\u003C\u002Fth>\n\u003Cth>Stream processing\u003C\u002Fth>\n\u003C\u002Ftr>\n\u003C\u002Fthead>\n\u003Ctbody>\u003Ctr>\n\u003Ctd>Data scope\u003C\u002Ftd>\n\u003Ctd>Queries or processing over all or most of the data in the dataset.\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>Data size\u003C\u002Ftd>\n\u003Ctd>Large batches of data.\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>Performance\u003C\u002Ftd>\n\u003Ctd>Latencies in minutes to hours.\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>Analysis\u003C\u002Ftd>\n\u003Ctd>Complex analytics.\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003C\u002Ftbody>\u003C\u002Ftable>\n\u003Cp>For more information, see \u003Ca href=\"https:\u002F\u002Fserverlessland.com\u002Fevent-driven-architecture\u002Fvisuals\u002Fbatching-vs-event-streams\">Batch Processing vs. Event Streaming\u003C\u002Fa>.\u003C\u002Fp>\n",{"title":6,"order":7},"Introducing streaming",1,"### Understanding streaming\n\nReal-time data is critical to inform decision-making and for delivering enhanced customer experiences. Digital businesses have access to thousands of data sources that are constantly changing, including Internet of Things (IoT) sensors, application logs, mobile apps, clickstreams, and social media. Making sense and using real-time data has become a key differentiator for innovation across many industries.\n\n\u003Cimg src=\"\u002Fassets\u002Fexternal\u002Fguides\u002Flambda-kafka\u002Fassets\u002Fimages\u002Fkafka-lambda-data-sources.png\">\n\n### What is streaming data?\n\nStreaming data is data that is generated in a nearly continuous, incremental manner, often at high volumes. It must be processed quickly in order, on a record-by-record basis or over sliding time windows to get the most value from it.\n\nStream-based processing can offer real-time actionable analytical insights compared to batch oriented-processes, which often only generate insights after hours, days, or even weeks have passed. To learn more about streaming data, see [What is streaming data?](https:\u002F\u002Faws.amazon.com\u002Fstreaming-data\u002F)\n\nCharacteristics of streaming data include:\n\n- **High volume:** Organizations that collect clickstream data about user behavior can often reach up to _2-3 TB_ every day.\n- **Nearly continuous rather than discrete** : IoT devices like sensors collect data continuously and at regular intervals.\n- **Ordered:** A chat application makes little sense with messages out of order.\n- **Requires low latency processing to be valuable:** Fraud detection must be done as quickly as possible to prevent loss.\n\n### Examples of streaming data\n\n- Sensors in transportation vehicles, industrial equipment, and farm machinery send data to a streaming application. The application monitors performance defects any potential defects in advance, and places a spare part order automatically preventing equipment down time.\n- A financial institution tracks changes in the stock market in real time, computes value-at-risk, and automatically rebalances portfolios based on stock price movements.\n- A real-estate website tracks a subset of data from consumers' mobile devices and makes real-time property recommendations of properties to visit based on their geo-location.\n- A solar power company has to maintain power throughput for its customers, or pay penalties. It implemented a streaming data application that monitors all panels in the field, and schedules service in real time, thereby minimizing the periods of low throughput from each panel and the associated penalty payouts.\n- A media publisher streams billions of clickstream records from its online properties, aggregates and enriches the data with demographic information about users, and optimizes content placement on its site, delivering relevancy and a better experience to its audience.\n- An online gaming company collects streaming data about player-game interactions, and feeds the data into its gaming platform. It then analyzes the data in real-time, offers incentives and dynamic experiences to engage its players.\n\n### Comparing batch processing and stream processing\n\nBatch processing is used to compute queries over different sets of data. It usually computes results that are derived from all the data it encompasses and enables deep analysis of large datasets. In contrast, stream processing requires ingesting a sequence of data, and incrementally updating metrics, reports, and summary statistics in response to each arriving data record. It is better suited for real-time monitoring and response functions.\n\nMany organizations build a hybrid model by combining the two approaches, and maintain a real-time layer and a batch layer. Data is first processed by a streaming data platform to extract real-time insights, and then persisted into a store like [Amazon S3](https:\u002F\u002Faws.amazon.com\u002Fs3\u002F), where it can be transformed and loaded for a variety of batch processing use cases.\n\n|Batch processing |Stream processing |\n|--------|-------|\n| Data scope | Queries or processing over all or most of the data in the dataset. | Queries or processing over data within a rolling time window, or on just the most recent data record. |\n| Data size | Large batches of data. | Individual records or micro batches consisting of a few records. |\n| Performance | Latencies in minutes to hours. | Requires latency in the order of seconds or milliseconds. |\n| Analysis | Complex analytics. | Simple response functions, aggregates, and rolling metrics. |\n\nFor more information, see [Batch Processing vs. Event Streaming](https:\u002F\u002Fserverlessland.com\u002Fevent-driven-architecture\u002Fvisuals\u002Fbatching-vs-event-streams).",[10],"content\u002Fcontributors\u002Fjulian-wood.json","LESSON","Using AWS Lambda to process Apache Kafka streams","\u002Fcontent\u002Fguides\u002Flambda-kafka\u002Fintroduction","4 min",[16,21,30,38,47,55,64,73,82],{"title":6,"content":17},[18],{"title":6,"order":7,"time":14,"path":19,"id":20,"link":13},"introduction","introduction.md",{"title":22,"content":23},"Using Kafka to build your streaming application",[24],{"title":22,"order":25,"time":26,"path":27,"id":28,"link":29},2,"6 min","using-kafka","using-kafka.md","\u002Fcontent\u002Fguides\u002Flambda-kafka\u002Fusing-kafka",{"title":31,"content":32},"Using Lambda to consume records from Kafka",[33],{"title":31,"order":34,"time":14,"path":35,"id":36,"link":37},3,"using-lambda","using-lambda.md","\u002Fcontent\u002Fguides\u002Flambda-kafka\u002Fusing-lambda",{"title":39,"content":40},"Configuring Kafka and Lambda",[41],{"title":39,"order":42,"time":43,"path":44,"id":45,"link":46},4,"9 min","configuring-kafka-lambda","configuring-kafka-lambda.md","\u002Fcontent\u002Fguides\u002Flambda-kafka\u002Fconfiguring-kafka-lambda",{"title":48,"content":49},"Processing Kafka streams using the Lambda ESM",[50],{"title":48,"order":51,"time":43,"path":52,"id":53,"link":54},5,"processing-esm","processing-esm.md","\u002Fcontent\u002Fguides\u002Flambda-kafka\u002Fprocessing-esm",{"title":56,"content":57},"Scaling and throughput",[58],{"title":56,"order":59,"time":60,"path":61,"id":62,"link":63},6,"5 min","scaling-throughput","scaling-throughput.md","\u002Fcontent\u002Fguides\u002Flambda-kafka\u002Fscaling-throughput",{"title":65,"content":66},"Monitoring and observability",[67],{"title":65,"order":68,"time":69,"path":70,"id":71,"link":72},7,"2 min","monitoring-observability","monitoring-observability.md","\u002Fcontent\u002Fguides\u002Flambda-kafka\u002Fmonitoring-observability",{"title":74,"content":75},"Troubleshooting",[76],{"title":74,"order":77,"time":78,"path":79,"id":80,"link":81},8,"3 min","troubleshooting","troubleshooting.md","\u002Fcontent\u002Fguides\u002Flambda-kafka\u002Ftroubleshooting",{"title":83,"content":84},"Resources",[85],{"title":83,"order":86,"time":87,"path":88,"id":89,"link":90},9,"1 min","resources","resources.md","\u002Fcontent\u002Fguides\u002Flambda-kafka\u002Fresources",null,1791888669261]