Amazon KDS High-Level Architecture AA

The following diagram illustrates how Kinesis Data Streams work at a high-level. The producers push the data continuously to a Kinesis Data Stream and the consumer applications consume and process the data in real time. Consumers could be applications hosted on Amazon EC2 instances, Amazon Data Firehose, or AWS Lambda.

Terminology

Data Record

A Data Record is the unit of data stored in Kinesis Data Streams. They are composed of a sequence number, a Partition Key and the data which forms an immutable sequence of bytes.

Shard

A Kinesis Data Stream is a set of Shards. A Shard is a sequence of data records that can be uniquely identified. A stream is composed of one or more shards, each of which provide a fixed unit of capacity. The total data capacity of a stream is the sum of the capacities of its shards. Each shard can support up to 5 TPS for reads, up to a maximum total data read at the rate of 2MB per second, up to a 1000 records per second for writes and up to total data write rate of up to 1MB per second.

Partition Key

A Partition Key groups the data by shard within a data stream. Kinesis Data Stream uses the Partition Key of each data record to organize it to a specific shard. Each data record pushed into the data stream must have an associated Partition Key.

Capacity Mode

The Capacity Mode of a data stream determines how the capacity is managed and how you are charged for its usage. Currently, there are two capacity modes you can choose from on-demand mode and provisioned mode for the streams.

With the on-demand mode, the capacity is automatically managed by Kinesis Data Streams in order to provide the necessary throughput. You are only charged for the actual throughput that you use and Kinesis Data Streams ramp up or down as needed based on your workload.

With the provisioned mode, you must specify the number of shards for the data stream. The total capacity is the sum of the capacities of the shards. You can increase or decrease the number of shards as needed. The charge is an hourly rate for the number of shards.

Retention Period

The Retention Period is the period of time for which the data records are accessible after they are added to the stream. By default, the Retention Period of a stream is set to 24 hours after creation. The Retention Period can be increased up to a maximum of 365 days (8760 hours) and decreased to a minimum of 24 hours. Additional charges apply for streams with Retention Period set for more than 24 hours.

Please refer to Amazon Kinesis Data Streams Terminology and Concepts to learn more.

Note : The terms Data records, events, messages are used interchangeably below

Delivery semantics and Message Ordering

When designing a streaming application, it is important to make a note of the delivery semantics and message ordering as it varies depending on the service being used. Semantics and order can have a considerable impact on consumers depending whether an event is processed exactly once, at least once, or at most once. These semantics offer tradeoffs between latency and message durability. It is also important to consider the order in which the events are processed by the consumer.

Kinesis Data Streams uses shards to organize and persist the events that are processed by consumer applications. Each shard has separate throughput capacity which the consumer uses to process the events in parallel. A consumer consumes the events from a shard in the exact same order in which they are stored. The event ordering is only maintained within a specific shard.