When you initially create an Amazon MSK event source, Lambda allocates one consumer or poller to process all partitions in the Kafka topic. Additionally, Lambda automatically scales up or down the number of consumers, based on message rates. To preserve message ordering in each partition, the maximum number of consumers is one consumer per topic partition.

In one-minute intervals, Lambda evaluates the consumer offset lag of all the partitions in the topic. If the lag is too high, the partition is receiving messages faster than Lambda can process them. If necessary, Lambda adds or removes consumers from the topic. The scaling process of adding or removing consumers occurs within three minutes of evaluation.

If your target Lambda function is throttled, Lambda reduces the number of consumers. This action reduces the workload on the function by reducing the number of messages that consumers can retrieve and send to the function. The processing of messages can also be scaled further by selective invocation of the target Lambda function. This can be achieved by using filter expressions.

For a deep dive learning journey follow the Kafka Lambda learning guide.