Understanding streaming

Real-time data is critical to inform decision-making and for delivering enhanced customer experiences. Digital businesses have access to thousands of data sources that are constantly changing, including Internet of Things (IoT) sensors, application logs, mobile apps, clickstreams, and social media. Making sense and using real-time data has become a key differentiator for innovation across many industries.

What is streaming data?

Streaming data is data that is generated in a nearly continuous, incremental manner, often at high volumes. It must be processed quickly in order, on a record-by-record basis or over sliding time windows to get the most value from it.

Stream-based processing can offer real-time actionable analytical insights compared to batch oriented-processes, which often only generate insights after hours, days, or even weeks have passed. To learn more about streaming data, see What is streaming data?

Characteristics of streaming data include:

• High volume: Organizations that collect clickstream data about user behavior can often reach up to 2-3 TB every day.
• Nearly continuous rather than discrete : IoT devices like sensors collect data continuously and at regular intervals.
• Ordered: A chat application makes little sense with messages out of order.
• Requires low latency processing to be valuable: Fraud detection must be done as quickly as possible to prevent loss.

Examples of streaming data

• Sensors in transportation vehicles, industrial equipment, and farm machinery send data to a streaming application. The application monitors performance defects any potential defects in advance, and places a spare part order automatically preventing equipment down time.
• A financial institution tracks changes in the stock market in real time, computes value-at-risk, and automatically rebalances portfolios based on stock price movements.
• A real-estate website tracks a subset of data from consumers' mobile devices and makes real-time property recommendations of properties to visit based on their geo-location.
• A solar power company has to maintain power throughput for its customers, or pay penalties. It implemented a streaming data application that monitors all panels in the field, and schedules service in real time, thereby minimizing the periods of low throughput from each panel and the associated penalty payouts.
• A media publisher streams billions of clickstream records from its online properties, aggregates and enriches the data with demographic information about users, and optimizes content placement on its site, delivering relevancy and a better experience to its audience.
• An online gaming company collects streaming data about player-game interactions, and feeds the data into its gaming platform. It then analyzes the data in real-time, offers incentives and dynamic experiences to engage its players.

Comparing batch processing and stream processing

Batch processing is used to compute queries over different sets of data. It usually computes results that are derived from all the data it encompasses and enables deep analysis of large datasets. In contrast, stream processing requires ingesting a sequence of data, and incrementally updating metrics, reports, and summary statistics in response to each arriving data record. It is better suited for real-time monitoring and response functions.

Many organizations build a hybrid model by combining the two approaches, and maintain a real-time layer and a batch layer. Data is first processed by a streaming data platform to extract real-time insights, and then persisted into a store like Amazon S3, where it can be transformed and loaded for a variety of batch processing use cases.

Batch processing	Stream processing
Data scope	Queries or processing over all or most of the data in the dataset.
Data size	Large batches of data.
Performance	Latencies in minutes to hours.
Analysis	Complex analytics.

For more information, see Batch Processing vs. Event Streaming.