[{"data":1,"prerenderedAt":155},["ShallowReactive",2],{"content-svc-lambda-guides-serverless-stream-processing-with-lambda-4-1-KDS-key-concepts-terms":3},{"markdown":4,"frontMatterAttributes":5,"bodyRaw":10,"menu":11,"menuType":148,"isCollapseableMenu":14,"nextDocItem":109,"previousDocItem":93,"contributorPaths":149,"contentName":154,"slug":108,"readingTime":105},"\u003Ch4>Amazon KDS High-Level Architecture AA\u003C\u002Fh4>\n\u003Cp>The following diagram illustrates how Kinesis Data Streams work at a high-level. The producers push the data continuously to a Kinesis Data Stream and the consumer applications consume and process the data in real time. Consumers could be applications hosted on Amazon EC2 instances, Amazon Data Firehose, or AWS Lambda.\u003C\u002Fp>\n\u003Cp>\u003Cimg src=\"\u002Fassets\u002Fexternal\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fkds\u002Fassets\u002Fkds-4-1-architecture.png\" alt=\"KDS high-level architecture\">\u003C\u002Fp>\n\u003Ch4>Terminology\u003C\u002Fh4>\n\u003Ch5>Data Record\u003C\u002Fh5>\n\u003Cp>A \u003Cem>Data Record\u003C\u002Fem> is the unit of data stored in Kinesis Data Streams. They are composed of a sequence number, a \u003Cem>Partition Key\u003C\u002Fem> and the data which forms an immutable sequence of bytes.\u003C\u002Fp>\n\u003Ch5>Shard\u003C\u002Fh5>\n\u003Cp>A Kinesis Data Stream is a set of \u003Cem>Shards\u003C\u002Fem>. A \u003Cem>Shard\u003C\u002Fem> is a sequence of \u003Cem>data records\u003C\u002Fem> that can be uniquely identified. A stream is composed of one or more \u003Cem>shards\u003C\u002Fem>, each of which provide a fixed unit of capacity. The total data capacity of a stream is the sum of the capacities of its shards. Each shard can support up to 5 TPS for reads, up to a maximum total data read at the rate of 2MB per second, up to a 1000 records per second for writes and up to total data write rate of up to 1MB per second.\u003C\u002Fp>\n\u003Ch5>Partition Key\u003C\u002Fh5>\n\u003Cp>A \u003Cem>Partition Key\u003C\u002Fem> groups the data by shard within a data stream. Kinesis Data Stream uses the \u003Cem>Partition Key\u003C\u002Fem> of each data record to organize it to a specific shard. Each data record pushed into the data stream must have an associated \u003Cem>Partition Key\u003C\u002Fem>.\u003C\u002Fp>\n\u003Ch5>Capacity Mode\u003C\u002Fh5>\n\u003Cp>The \u003Cem>Capacity Mode\u003C\u002Fem> of a data stream determines how the capacity is managed and how you are charged for its usage. Currently, there are two \u003Cem>capacity modes\u003C\u002Fem> you can choose from \u003Cstrong>on-demand\u003C\u002Fstrong> mode and \u003Cstrong>provisioned\u003C\u002Fstrong> mode for the streams.\u003C\u002Fp>\n\u003Cp>With the on-demand mode, the capacity is automatically managed by Kinesis Data Streams in order to provide the necessary throughput. You are only charged for the actual throughput that you use and Kinesis Data Streams ramp up or down as needed based on your workload.\u003C\u002Fp>\n\u003Cp>With the provisioned mode, you must specify the number of shards for the data stream. The total capacity is the sum of the capacities of the shards. You can increase or decrease the number of shards as needed. The charge is an hourly rate for the number of shards.\u003C\u002Fp>\n\u003Ch5>Retention Period\u003C\u002Fh5>\n\u003Cp>The \u003Cem>Retention Period\u003C\u002Fem> is the period of time for which the data records are accessible after they are added to the stream. By default, the \u003Cem>Retention Period\u003C\u002Fem> of a stream is set to 24 hours after creation. The \u003Cem>Retention Period\u003C\u002Fem> can be increased up to a maximum of 365 days (8760 hours) and decreased to a minimum of 24 hours. Additional charges apply for streams with \u003Cem>Retention Period\u003C\u002Fem> set for more than 24 hours.\u003C\u002Fp>\n\u003Cp>Please refer to \u003Ca href=\"https:\u002F\u002Fdocs.aws.amazon.com\u002Fstreams\u002Flatest\u002Fdev\u002Fkey-concepts.html\">Amazon Kinesis Data Streams Terminology and Concepts\u003C\u002Fa> to learn more.\u003C\u002Fp>\n\u003Cblockquote>\n\u003Cp>\u003Cstrong>Note :\u003C\u002Fstrong> The terms \u003Ccode>Data records\u003C\u002Fcode>, \u003Ccode>events\u003C\u002Fcode>, \u003Ccode>messages\u003C\u002Fcode> are used interchangeably below\u003C\u002Fp>\n\u003C\u002Fblockquote>\n\u003Ch5>Delivery semantics and Message Ordering\u003C\u002Fh5>\n\u003Cp>When designing a streaming application, it is important to make a note of the delivery semantics and message ordering as it varies depending on the service being used. Semantics and order can have a considerable impact on consumers depending whether an event is processed exactly once, at least once, or at most once. These semantics offer tradeoffs between latency and message durability. It is also important to consider the order in which the events are processed by the consumer.\u003C\u002Fp>\n\u003Cp>Kinesis Data Streams uses \u003Cem>shards\u003C\u002Fem> to organize and persist the events that are processed by consumer applications. Each shard has separate throughput capacity which the consumer uses to process the events in parallel. A consumer consumes the events from a shard in the exact same order in which they are stored. The event ordering is only maintained within a specific shard.\u003C\u002Fp>\n",{"title":6,"weight":7,"learningGoals":8},"Kinesis Data Streams (KDS) - Key Concepts and Terminologies",60,[9],"Learn key concepts and terminologies of Amazon KDS","#### Amazon KDS High-Level Architecture AA\n\nThe following diagram illustrates how Kinesis Data Streams work at a high-level. The producers push the data continuously to a Kinesis Data Stream and the consumer applications consume and process the data in real time. Consumers could be applications hosted on Amazon EC2 instances, Amazon Data Firehose, or AWS Lambda.\n\n![KDS high-level architecture](\u002Fassets\u002Fexternal\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fkds\u002Fassets\u002Fkds-4-1-architecture.png)\n\n#### Terminology\n\n##### Data Record\n\nA _Data Record_ is the unit of data stored in Kinesis Data Streams. They are composed of a sequence number, a _Partition Key_ and the data which forms an immutable sequence of bytes.\n\n##### Shard\n\nA Kinesis Data Stream is a set of _Shards_. A _Shard_ is a sequence of _data records_ that can be uniquely identified. A stream is composed of one or more _shards_, each of which provide a fixed unit of capacity. The total data capacity of a stream is the sum of the capacities of its shards. Each shard can support up to 5 TPS for reads, up to a maximum total data read at the rate of 2MB per second, up to a 1000 records per second for writes and up to total data write rate of up to 1MB per second.\n\n##### Partition Key\n\nA _Partition Key_ groups the data by shard within a data stream. Kinesis Data Stream uses the _Partition Key_ of each data record to organize it to a specific shard. Each data record pushed into the data stream must have an associated _Partition Key_.\n\n##### Capacity Mode\n\nThe _Capacity Mode_ of a data stream determines how the capacity is managed and how you are charged for its usage. Currently, there are two _capacity modes_ you can choose from **on-demand** mode and **provisioned** mode for the streams.\n\nWith the on-demand mode, the capacity is automatically managed by Kinesis Data Streams in order to provide the necessary throughput. You are only charged for the actual throughput that you use and Kinesis Data Streams ramp up or down as needed based on your workload.\n\nWith the provisioned mode, you must specify the number of shards for the data stream. The total capacity is the sum of the capacities of the shards. You can increase or decrease the number of shards as needed. The charge is an hourly rate for the number of shards.\n\n##### Retention Period\n\nThe _Retention Period_ is the period of time for which the data records are accessible after they are added to the stream. By default, the _Retention Period_ of a stream is set to 24 hours after creation. The _Retention Period_ can be increased up to a maximum of 365 days (8760 hours) and decreased to a minimum of 24 hours. Additional charges apply for streams with _Retention Period_ set for more than 24 hours.\n\nPlease refer to [Amazon Kinesis Data Streams Terminology and Concepts](https:\u002F\u002Fdocs.aws.amazon.com\u002Fstreams\u002Flatest\u002Fdev\u002Fkey-concepts.html) to learn more.\n\n> **Note :** The terms `Data records`, `events`, `messages` are used interchangeably below\n\n##### Delivery semantics and Message Ordering\n\nWhen designing a streaming application, it is important to make a note of the delivery semantics and message ordering as it varies depending on the service being used. Semantics and order can have a considerable impact on consumers depending whether an event is processed exactly once, at least once, or at most once. These semantics offer tradeoffs between latency and message durability. It is also important to consider the order in which the events are processed by the consumer.\n\nKinesis Data Streams uses _shards_ to organize and persist the events that are processed by consumer applications. Each shard has separate throughput capacity which the consumer uses to process the events in parallel. A consumer consumes the events from a shard in the exact same order in which they are stored. The event ordering is only maintained within a specific shard.\n",[12,43,100],{"title":13,"collapsible":14,"isCollapsed":15,"content":16},"Introduction",false,true,[17,23,33],{"title":13,"order":18,"time":19,"path":20,"id":21,"link":22},1,"2 min","1-introduction","1-introduction.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F1-introduction",{"title":24,"order":25,"callout":26,"time":19,"path":30,"id":31,"link":32},"Getting started with streaming data",2,{"title":27,"description":28,"link":29},"Watch the video","This video will help you get started with Amazon Kinesis Data Streams, a massively scalable, durable, and low-cost streaming data service that enables you to continuously capture gigabytes of data per second from hundreds of thousands of sources, such as website clickstreams, database event streams, financial transactions, social media feeds, IT logs, and location-tracking events.","https:\u002F\u002Fwww.youtube.com\u002Fwatch?v=1I1DcJvmd4w","2-getting-started","2-getting-started.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F2-getting-started",{"title":34,"order":35,"callout":36,"time":19,"path":40,"id":41,"link":42},"Event source mapping for Lambda",3,{"title":37,"description":38,"link":39},"Read the docs","Refer to the AWS Lambda documentation for detailed explanations of Lambda event source mappings","https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Finvocation-eventsourcemapping.html","3-event-source-mapping","3-event-source-mapping.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F3-event-source-mapping",{"title":44,"collapsible":14,"isCollapsed":15,"content":45},"Streaming processing with MSK",[46,65,73,80,87,93],{"title":47,"order":48,"youtubeVideoID":49,"resources":50,"callout":58,"time":19,"path":62,"id":63,"link":64},"MSK as event source for Lambda",51,"JKvIypfEiok",[51,55],{"link":52,"type":53,"text":54},"https:\u002F\u002Faws.amazon.com\u002Flambda\u002F","Text","AWS Lambda",{"link":56,"type":53,"text":57},"https:\u002F\u002Faws.amazon.com\u002Fserverless\u002F","Serverless on AWS",{"title":59,"description":60,"link":61},"Follow the Kafka with Lambda learning guide","Refer to the deep dive learning guide for using Kafka with Lambda","https:\u002F\u002Fserverlessland.com\u002Fcontent\u002Fguides\u002Flambda-kafka\u002Fintroduction","msk-stream-processing","msk-stream-processing.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-stream-processing",{"title":66,"order":67,"callout":68,"time":69,"path":70,"id":71,"link":72},"Deep dive into Lambda ESM for MSK",50,{"title":59,"description":60,"link":61},"1 min","msk-esm-deep-dive","msk-esm-deep-dive.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-esm-deep-dive",{"title":74,"order":75,"callout":76,"time":19,"path":77,"id":78,"link":79},"Ordered processing of messages",52,{"title":59,"description":60,"link":61},"msk-ordered-processing","msk-ordered-processing.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-ordered-processing",{"title":81,"order":82,"callout":83,"time":69,"path":84,"id":85,"link":86},"Scaling of consumers",54,{"title":59,"description":60,"link":61},"msk-scaling-consumers","msk-scaling-consumers.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-scaling-consumers",{"title":88,"order":89,"time":69,"path":90,"id":91,"link":92},"Authentication options for Lambda consumers",56,"msk-lambda-auth-options","msk-lambda-auth-options.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-lambda-auth-options",{"title":94,"order":95,"callout":96,"time":69,"path":97,"id":98,"link":99},"Error handling in Lambda functions",58,{"title":59,"description":60,"link":61},"msk-lambda-error-handling","msk-lambda-error-handling.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-lambda-error-handling",{"title":101,"collapsible":14,"isCollapsed":14,"content":102},"Streaming processing with KDS",[103,109,121,132,140],{"title":6,"weight":7,"learningGoals":104,"time":105,"path":106,"id":107,"link":108},[9],"4 min","4-1-KDS-key-concepts-terms","4-1-KDS-key-concepts-terms.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-1-KDS-key-concepts-terms",{"title":110,"weight":111,"learningGoals":112,"resources":114,"time":117,"path":118,"id":119,"link":120},"Amazon Kinesis Data Streams - Integration with AWS Lambda",62,[113],"Amazon KDS integration with AWS Lambda for Serverless stream processing",[115,116],"https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Finvocation-eventsourcemapping.html#invocation-eventsourcemapping-batching","https:\u002F\u002Faws.amazon.com\u002Fblogs\u002Fcompute\u002Fnew-aws-lambda-scaling-controls-for-kinesis-and-dynamodb-event-sources\u002F","5 min","4-2-KDS-lambda-intg","4-2-KDS-lambda-intg.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-2-KDS-lambda-intg",{"title":122,"weight":123,"learningGoals":124,"resources":126,"time":128,"path":129,"id":130,"link":131},"Amazon KDS - Error handling with Lambda ESM",64,[125],"How to handle errors when configuring Amazon KDS as Lambda ESM",[127],"https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Fwith-kinesis.html","3 min","4-3-KDS-lambda-error-handling","4-3-KDS-lambda-error-handling.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-3-KDS-lambda-error-handling",{"title":133,"weight":134,"learningGoals":135,"time":117,"path":137,"id":138,"link":139},"Kinesis Data Streams Scaling- Key Concepts and Terminologies",66,[136],"Learn key concepts and terminologies of Amazon KDS Scaling","4-4-KDS-scaling","4-4-KDS-scaling.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-4-KDS-scaling",{"title":141,"weight":142,"learningGoals":143,"time":117,"path":145,"id":146,"link":147},"Kinesis Data Streams Monitoring",68,[144],"Learn Amazon KDS & Lambda monitoring methodologies","4-5-KDS-monitoring","4-5-KDS-monitoring.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-5-KDS-monitoring","LIST",[150,151,152,153],"content\u002Fcontributors\u002Fmithun-mallick.json","content\u002Fcontributors\u002Fkevin-schwarz.json","content\u002Fcontributors\u002Fnaren-sithambaram.json","content\u002Fcontributors\u002Fprasanna-s-krishnan.json","Serverless Stream Processing with Lambda",1792665273594]