Lambda ESM for self-managed Kafka 4.2+ using Queue mode (KIP-932 Share Groups). Multiple pollers process the same partition concurrently, breaking the partition-count ceiling.
With Queue consumption mode (ConsumptionMode: Queue), Lambda event pollers use Kafka Share Groups (KIP-932) instead of traditional consumer groups. Multiple pollers can read from the same partition concurrently — parallelism is decoupled from partition count.
This pattern deploys: a dedicated VPC with private subnets and VPC endpoints, a self-managed Kafka 4.2+ broker on EC2 with share groups enabled, a Python Lambda worker function with partial batch response, a producer Lambda for testing, and CloudWatch observability.
Deployment is split into four independent stacks so you can skip the network and broker stacks if you have existing infrastructure.
< Back to all patterns
Clone repo
git clone https://github.com/aws-samples/serverless-patterns/cd serverless-patterns/smk-lambda-queue-mode/python/samDeploy
See README.md for full deployment instructions including three paths: full deployment, bring-your-own Kafka, and bring-your-own Kafka and VPC.Testing
See the GitHub repo for detailed testing instructions.
Cleanup
Delete the ESM first:
aws lambda delete-event-source-mapping --uuid <esm-uuid>, then delete stacks in reverse order: kafka-queue-observability, kafka-queue-app, kafka-queue-broker, kafka-queue-network.Created by:
