Streaming data presents a unique set of design and architectural challenges for developers. By definition, streaming data is not bounded, having no clear beginning or end. It can be generated by millions of separate producers, such as Internet of Things (IoT) devices or mobile applications. Additionally, streaming data applications must frequently process and analyze this data with minimal latency.

There are several general goals of streaming data workloads. The first is to capture the messages durably and persist these in an ordered queue. Next, the system must process and aggregate the records, keeping up with the pace of new messages arriving. Finally, the application loads the results into other downstream systems for analysis or further processing.

Each of these tasks can be major challenges for traditional, server-based architectures. To ingest the constant stream of messages, you must successfully scale clusters of brokers, and transport and store the data. Whether your compute pipeline is running on EC2 instances or physical services, you must scale those fleets depending upon traffic. There are also data and networking management considerations that require specialized knowledge.

This repo shows how you can build a streaming data backend by using a serverless approach. I provide a complete example application that you can deploy to your AWS account, introducing key streaming concepts and how to handle these in a serverless workload