Amazon Managed Streaming for Kafka (MSK) is the managed service for Apache Kafka. Customers can migrate their self-managed Apache Kafka clusters to MSK and take advantage of the managed service features such as patch management, instance scaling, security and monitoring that are built in. MSK integration with Lambda provides customers with a serverless way to process data in their Kafka topics. Lambda service manages the pollers that read from the Kafka topic, perform batching and invoke the target Lambda function. Customers can focus on the core business needed to process the messages.

Lambda also supports self-managed Apache Kafka. Lambda processes messages the same way for self-managed Kafka as it does for MSK. Please refer to the documentation for setting up self-managed Kafka with Lambda.

Lambda works with Confluent managed Kafka as well. There are two ways in which Lambda can be invoked from Confluent Kafka. Refer to the following blogs around setting up Lambda with Confluent Kafka

• Network access patterns of AWS Lambda for Confluent Cloud
• Two ways to build event driven architectures

Topics

• Diving deep into Kafka Event source mapping
• Maintaining ordering of messages
• Scaling of consumers
• Setting up Lambda with different authentication options
• Error handling

For a more deep dive learning journey follow the Kafka Lambda learning guide.