Getting started processing Amazon Managed Streaming for Apache Kafka (MSK) and Kinesis Data Streams (KDS) serverless streams with Lambda is easy. Once you've made a decision on which AWS streaming service to use, and you've deployed either KDS or MSK streaming infrastructure to receive data from your source, then simply configure KDS or MSK as the trigger for AWS Lambda. You can use Lambda to process your streaming data and land data to your destination.
Both KDS and MSK are great solutions for streaming data in the cloud, but there are a few key differences to consider when you choose a solution. KDS is a completely managed regional (multi-AZ) cloud native streaming service with on demand pay as you go pricing. MSK is a fully managed and highly available fully open-source compatible Apache Kafka.
| KDS | MSK | |
|---|---|---|
| Latency | Readable immediately after written. Propagation delay equivalent to polling interval. Enhanced Fan out available. | Readable immediately after written. |
| Ordering | Ordered per shard | Ordered per topic |
| Cost | On demand billing based per stream/hr, per GB ingested, per GB retrieved, and data stored. | Billing by instance type per hour per instance, broker storage, and VPC connectivity options. |
| Retention | Default 24 hours extendable up to 365 days. | Configurable retention based on storage available on a broker. |
AWS Lambda integrates natively with Amazon Kinesis and Kafka (AWS MSK Provisioned, MSK serverless, self-managed Kafka clusters, and Confluent Cloud) as a consumer to process ingested data. Customers can invoke Lambda functions from messages in Kafka topics or Kinesis data streams to integrate into downstream serverless workflows. Lambda service manages the provisioning of the polling infrastructure, provides settings for batching, error handling and scaling. In both cases, customers are guaranteed ordered processing of messages.
The resources below will help you think about how to setup your streaming infrastructure and the next topic on Lambda Event Source Mapping will help you understand how Lambda helps you process your streaming data.
Created by:
