AWS Lambda event source mappings (ESM) are Lambda resources that simplify processing stream data. An ESM polls streaming data sources like MSK, KDS, SQS and Amazon MQ for new messages, then invokes a Lambda function to process messages in batches. As a developer, you can focus your time writing business logic in your function code and let ESM take care of your streaming data client integration.
With the Lambda ESM, developers can quickly build new streaming applications while Lambda takes care of the undifferentiated heavy lifting with built-in managed pollers that are highly available, secure and scale automatically to meet business needs. Instead of operating servers or worrying about infrastructure management tasks like capacity provisioning and patching, customers can now focus on accelerating innovation. With the pay for value pricing model (only paying for the compute required to process data, no idle) and 1ms billing, Serverless allows customers to save overall costs.
The high-level design of ESM is shown in the diagram below:
The Lambda ESM is easy to configure and provides flexibility and control over stream processing behavior.
- MSK and KDS both support configurable batch sizes and starting positions. In the case of KDS a tumbling window duration is also configurable.
- Failures can be propagated to dead letter queues (SQS) or topics (SNS) for Kinesis data stream event source.
- Authentication managed with IAM, and in the case of MSK, SASL/SCRAM and mTLS are also supported.
- Lambda triggers for KDS and MSK also support event filtering based on the JSON payload, using the same syntax as Amazon EventBridge rules.
- Both KDS and MSK publish standard CloudWatch metrics to monitor and alarm on processing success and performance.
You can find more information on setting up MSK and KDS as event sources for Lambda with the below resources and in the next sections.
Created by:
