When a new event is published to a topic, it is appended to one of the topic's partitions. Events with the same event key (e.g., a customer or vehicle ID) are written to the same partition, and Kafka guarantees that any consumer of a given topic-partition will always read that partition's events in exactly the same order as they were written.

You can find additional insights into Kafka's architecture within the Kafka documentation. AWS Lambda ensures the sequential handling of messages sourced from Kafka topics by establishing message consumers within a shared consumer group. This consumer group is orchestrated by the Event Source Mapping resource. Users have the option to define the consumer group ID, or the Lambda service will generate one randomly if not specified. Every message consumer within this group is assigned to one or multiple partitions. The service then forwards these consumed messages to the designated Lambda function while preserving their original order. Although, considering varying consumer counts, individual function instances might receive messages from diverse partitions, the inherent order within each partition remains intact. This consumer count can expand up to the number of partitions, guided by the offset lag. Once the polling mechanisms reach peak capacity, each respective Lambda function processes messages from an allocated partition.

Lambda ESM also supports filtering of messages based on JSON filter expressions. Batching of messages is done before the batch is created. The ordering is still maintained within the partition.

In case of exceptions in processing of messages, Lambda function needs to either log it or write it to another service such as SQS FIFO to avoid reprocessing of messages and out of order scenarios. Currently, the Kafka-Lambda integration does not have any dead letter queue support or selective redelivery of messages based on exception.

For a more deep dive learning journey follow the Kafka Lambda learning guide.