Real-time data streaming has rapidly expanded over the past few years. This is due to digital and traditional businesses needing access to real time data sources to meet customer demands. Streaming data sources are constantly evolving and include internet of things (IoT) sensors that monitor equipment to improve performance and uptime, online gaming companies that use player-game interactions to offer incentives and dynamic experiences, monitoring application logs, mobile apps, click streams, and social media interactions. Capturing and reasoning over real-time data has become a key differentiator for innovators across all types of businesses.

What is streaming data anyway?

Streaming data is emitted in a continuous, incremental manner, often at incredibly high volumes and needing to be processed immediately. Real time applications collect streaming data at scale for real-time data movement, real-time analytics, and event stream processing. This data needs to be processed sequentially and incrementally on a record-by-record basis or over sliding time windows, and used for a wide variety of analytics including correlations, aggregations, filtering, and sampling. Stream-based processing can offer real-time actionable insights compared to batch oriented-processes, which often only generates insights after hours, days, or even weeks have passed.

Amazon Managed Streaming for Apache Kafka (MSK) and Kinesis Data Streams (KDS) are the AWS data streaming services that customers can choose to send streaming data, process, and consume in applications with ultra-low latency. MSK and KDS allow customers to build event driven applications that rely on identifying the trends from their business data to trigger business events. Both of these services preserve the order in which messages are received allowing customers to process them either on the basis of processing time or event time.

Watch the following video for a better understanding of Kinesis and MSK streaming services.