Select your cookie preferences

We use cookies and similar tools to enhance your experience, provide our services, deliver relevant advertising, and make improvements. Approved third parties also use these tools to help us deliver advertising and provide certain site features.

Using AWS Lambda to process Apache Kafka streams Troubleshooting 3 min

Network security issues

If an ESM's networking, authentication, or authorization settings prevent communication with the cluster, then setup fails before it can invoke a function. The trigger shows an error record that can help troubleshoot the root cause.

When the ESM sends a request to the broker endpoints and doesn't receive a response, the ESM considers the request as timed out. When a timeout to the broker endpoint occurs, the trigger displays the following error message:

"PROBLEM: Connection error. Please check your event source connection configuration. If your event source lives in a VPC, try setting up a new Lambda function or EC2 instance with the same VPC, Subnet, and Security Group settings. Connect the new device to the Kafka cluster and consume messages to ensure that the issue is not related to VPC or Endpoint configuration. If the new device is able to consume messages, please contact Lambda customer support for further investigation."

Follow the steps in the error message. Also, check the networking configurations to be sure that your ESM is properly configured.

Initialization, polling, and invocation errors

You receive the following message if your ESM is configured in a VPC but cannot call the STS API or Secrets Manager API (if required). You also receive the message if the ESM can access the Kafka cluster but can't invoke the function using the Lambda API.

"PROBLEM: Connection error. Your VPC must be able to connect to Lambda and STS, as well as Secrets Manager if authentication is required. You can provide access by configuring PrivateLink or a NAT Gateway."

Check your VPC settings that prevent your ESM from reaching other services like AWS STS and AWS Secrets Manager. Follow the steps in Setting up AWS Lambda with an Apache Kafka cluster within a VPC to properly configure your VPC settings.

If calls to the STS API fail or timeout, then your VPC settings prevent your ESM from reaching the Regional Lambda endpoint on port 443. To resolve this issue, see Setting up AWS Lambda with an Apache Kafka cluster within a VPC.

If SourceAccessConfiguration contains a secret, then be sure to retrieve that secret from Secrets Manager.

For additional troubleshooting steps, see Troubleshoot Lambda triggers with MSK and Kafka clusters.

Amazon MSK consumer group rebalancing

If your Amazon MSK consumer group is rebalancing continuously, see the following troubleshooting steps.

SASL/SCRAM authentication

To troubleshoot issues when using Amazon MSK with SASL/SCRAM authentication, see the following troubleshooting steps.

Lambda ESM consumer cannot join existing consumer group

ESM consumers cannot join a consumer group if the existing consumers are using a different consumer strategy than the ESM consumers. Ensure that all consumers in the group are using the same consumer strategy