[{"data":1,"prerenderedAt":91},["ShallowReactive",2],{"content-guides-lambda-kafka-configuring-kafka-lambda":3},{"markdown":4,"frontMatterAttributes":5,"bodyRaw":8,"contributorPaths":9,"menuType":11,"contentName":12,"slug":13,"readingTime":14,"menu":15,"nextDocItem":50,"previousDocItem":37},"\u003Cp>There are a number of guides to setting up Kafka and the consumer Lambda ESM.\u003C\u002Fp>\n\u003Cp>The \u003Ca href=\"https:\u002F\u002Fkafka.apache.org\u002Fdocumentation\u002F\">Kafka\u003C\u002Fa> and \u003Ca href=\"https:\u002F\u002Fdocs.confluent.io\u002Fhome\u002Foverview.html\">Confluent\u003C\u002Fa> documentation includes steps to install Kafka. \u003Ca href=\"https:\u002F\u002Faws.amazon.com\u002Fblogs\u002Fcompute\u002Fusing-self-hosted-apache-kafka-as-an-event-source-for-aws-lambda\u002F\">Using self-hosted Apache Kafka as an event source for AWS Lambda\u003C\u002Fa> describes setting up the Lambda ESM when consuming from self-hosted Kafka.\u003C\u002Fp>\n\u003Cp>This \u003Ca href=\"https:\u002F\u002Fdocs.aws.amazon.com\u002Fmsk\u002Flatest\u002Fdeveloperguide\u002Fgetting-started.html\">Amazon MSK\u003C\u002Fa> tutorial shows an example of how to create an MSK cluster, produce and consume data, and monitor the health of your cluster using metrics. \u003Ca href=\"https:\u002F\u002Faws.amazon.com\u002Fblogs\u002Fcompute\u002Fusing-amazon-msk-as-an-event-source-for-aws-lambda\u002F\">Using Amazon MSK as an event source for AWS Lambda\u003C\u002Fa> describes setting up Lambda ESM to read from Amazon MSK. The  \u003Ca href=\"https:\u002F\u002Fcatalog.us-east-1.prod.workshops.aws\u002Fworkshops\u002Fc2b72b6f-666b-4596-b8bc-bafa5dcca741\u002Fen-US\">Amazon MSK Labs\u003C\u002Fa> workshop provides an install guide.\u003C\u002Fp>\n\u003Cp>There are a number of \u003Ca href=\"https:\u002F\u002Fserverlessland.com\u002Fpatterns?services=msk\">Serverless Patterns\u003C\u002Fa> you can use to deploy and test Amazon MSK integrations. You can deploy an Amazon MSK cluster using the \u003Ca href=\"https:\u002F\u002Fserverlessland.com\u002Fpatterns\u002Fmsk-cfn-sasl-lambda\">SASL authentication\u003C\u002Fa> pattern. For IAM authentication, there are patterns for \u003Ca href=\"https:\u002F\u002Fserverlessland.com\u002Fpatterns\u002Fmsk-lambda-iam-node-sam\">Node.js\u003C\u002Fa> and \u003Ca href=\"https:\u002F\u002Fserverlessland.com\u002Fpatterns\u002Fmsk-lambda-iam-python-sam\">Python\u003C\u002Fa>.\u003C\u002Fp>\n\u003Ch3>Security\u003C\u002Fh3>\n\u003Cp>Lambda supports several methods to authenticate with your self-managed Kafka cluster and Amazon MSK.\u003C\u002Fp>\n\u003Ch5>SASL\u002FSCRAM authentication\u003C\u002Fh5>\n\u003Cp>Lambda supports Simple Authentication and Security Layer\u002FSalted Challenge Response Authentication Mechanism (SASL\u002FSCRAM) authentication with TLS encryption. Lambda sends the encrypted credentials to authenticate with the cluster. Lambda doesn't support SASL\u002FSCRAM with plaintext. Amazon MSK Serverless doesn't support SASL\u002FSCRAM. The \u003Ca href=\"https:\u002F\u002Fserverlessland.com\u002Fpatterns\u002Fmsk-cfn-sasl-lambda\">SASL\u002FSCRAM authentication\u003C\u002Fa> pattern shows how to use this authentication mechanism with Amazon MSK.\u003C\u002Fp>\n\u003Cp>Lambda with self-managed Kafka supports SASL\u002FPLAIN authentication. Because this uses clear text credentials, the connection to the server must use TLS encryption to ensure that the credentials are protected. Amazon MSK doesn't support SASL\u002FPLAIN authentication.\u003C\u002Fp>\n\u003Cp>For SASL authentication, you store the sign-in credentials as a secret in AWS Secrets Manager. For more information about using Secrets Manager, see \u003Ca href=\"https:\u002F\u002Fdocs.aws.amazon.com\u002Fsecretsmanager\u002Flatest\u002Fuserguide\u002Ftutorials.html\">AWS Secrets Manager tutorials\u003C\u002Fa>.\u003C\u002Fp>\n\u003Ch5>Mutual TLS authentication\u003C\u002Fh5>\n\u003Cp>Mutual TLS (mTLS) provides two-way authentication between the client and server. The client sends a certificate to the server for the server to verify the client, and the server sends a certificate to the client for the client to verify the server.\u003C\u002Fp>\n\u003Cp>Using mTLS with Kafka, Lambda acts as the client. You configure a client certificate (as a secret in Secrets Manager) to authenticate Lambda with your Kafka brokers. The client certificate must be signed by a CA in the server's trust store.\u003C\u002Fp>\n\u003Cp>For more information about mTLS, see \u003Ca href=\"http:\u002F\u002Faws.amazon.com\u002Fblogs\u002Fcompute\u002Fintroducing-mutual-tls-authentication-for-amazon-msk-as-an-event-source\">Introducing mutual TLS authentication for Amazon MSK as an event source\u003C\u002Fa>.\u003C\u002Fp>\n\u003Cp>Amazon MSK, in addition to supporting the same authentication as self-hosted Kafka, supports the following options:\u003C\u002Fp>\n\u003Ch5>Unauthenticated access\u003C\u002Fh5>\n\u003Cp>If no clients access the cluster over the internet, you can use unauthenticated access.\u003C\u002Fp>\n\u003Ch5>IAM role-based authentication\u003C\u002Fh5>\n\u003Cp>You can use IAM to authenticate the identity of clients that connect to the MSK cluster. For Lambda, the ESM handles the authentication. If IAM Auth is active on your MSK cluster, and you don't provide a secret for auth, Lambda automatically defaults to using IAM Auth. You can deploy patterns for \u003Ca href=\"https:\u002F\u002Fserverlessland.com\u002Fpatterns\u002Fmsk-lambda-iam-node-sam\">Node.js\u003C\u002Fa> and \u003Ca href=\"https:\u002F\u002Fserverlessland.com\u002Fpatterns\u002Fmsk-lambda-iam-python-sam\">Python\u003C\u002Fa>, which show how the ESM makes it easier to use IAM authentication. When producing Kafka records, there are open-source client helper libraries and code samples for \u003Ca href=\"https:\u002F\u002Faws.amazon.com\u002Fblogs\u002Fbig-data\u002Famazon-msk-iam-authentication-now-supports-all-programming-languages\u002F\">popular languages\u003C\u002Fa>.\u003C\u002Fp>\n\u003Cp>Make sure that you configure the Kafka cluster to use one of these supported authentication methods. For more information about general Kafka security, see the Kafka \u003Ca href=\"http:\u002F\u002Fkafka.apache.org\u002Fdocumentation.html#security\">Security\u003C\u002Fa> documentation. \u003C\u002Fp>\n\u003Cp>For more information on Lambda integration authentication, see the specific \u003Ca href=\"https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Fwith-msk.html#msk-cluster-permissions\">Amazon MSK\u003C\u002Fa> or \u003Ca href=\"https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Fwith-kafka.html#smaa-authentication\">self-managed Kafka\u003C\u002Fa> documentation.\u003C\u002Fp>\n\u003Cp>Lambda and Kafka supported authentication methods:\u003C\u002Fp>\n\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>\u003C\u002Fth>\n\u003Cth>SASL\u002FSCRAM + TLS\u003C\u002Fth>\n\u003Cth>SASL\u002FPLAIN\u003C\u002Fth>\n\u003Cth>IAM\u003C\u002Fth>\n\u003Cth>TLS\u003C\u002Fth>\n\u003Cth>Unauthenticated\u003C\u002Fth>\n\u003C\u002Ftr>\n\u003C\u002Fthead>\n\u003Ctbody>\u003Ctr>\n\u003Ctd>Self-managed Kafka\u003C\u002Ftd>\n\u003Ctd>Yes\u003C\u002Ftd>\n\u003Ctd>Yes\u003C\u002Ftd>\n\u003Ctd>No\u003C\u002Ftd>\n\u003Ctd>Yes\u003C\u002Ftd>\n\u003Ctd>No\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>Amazon MSK\u003C\u002Ftd>\n\u003Ctd>Yes\u003C\u002Ftd>\n\u003Ctd>No\u003C\u002Ftd>\n\u003Ctd>Yes\u003C\u002Ftd>\n\u003Ctd>Yes\u003C\u002Ftd>\n\u003Ctd>Yes\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>Amazon MSK Serverless\u003C\u002Ftd>\n\u003Ctd>Yes\u003C\u002Ftd>\n\u003Ctd>No\u003C\u002Ftd>\n\u003Ctd>Yes\u003C\u002Ftd>\n\u003Ctd>Yes\u003C\u002Ftd>\n\u003Ctd>Yes\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003C\u002Ftbody>\u003C\u002Ftable>\n\u003Ch4>Lambda permissions\u003C\u002Fh4>\n\u003Cp>Lambda can authenticate against Kafka using IAM or SASL.\u003C\u002Fp>\n\u003Cp>The Lambda function target in the event source mapping does not need to be running in a VPC to receive records from Amazon MSK. For more information on Kafka networking requirements for Lambda, see the \u003Cem>Networking\u003C\u002Fem> section below.\u003C\u002Fp>\n\u003Cp>The Lambda function must have permission to describe VPCs and security groups, and manage elastic network interfaces. These execution roles permissions are:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>ec2:CreateNetworkInterface\u003C\u002Fli>\n\u003Cli>ec2:DescribeNetworkInterfaces\u003C\u002Fli>\n\u003Cli>ec2:DescribeVpcs\u003C\u002Fli>\n\u003Cli>ec2:DeleteNetworkInterface\u003C\u002Fli>\n\u003Cli>ec2:DescribeSubnets\u003C\u002Fli>\n\u003Cli>ec2:DescribeSecurityGroups\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>To create and store logs in a log group in Amazon CloudWatch Logs, your Lambda function must have the following permissions in its execution role:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>\u003Ca href=\"https:\u002F\u002Fdocs.aws.amazon.com\u002FAmazonCloudWatchLogs\u002Flatest\u002FAPIReference\u002FAPI_CreateLogGroup.html\">logs:CreateLogGroup\u003C\u002Fa>\u003C\u002Fli>\n\u003Cli>\u003Ca href=\"https:\u002F\u002Fdocs.aws.amazon.com\u002FAmazonCloudWatchLogs\u002Flatest\u002FAPIReference\u002FAPI_CreateLogStream.html\">logs:CreateLogStream\u003C\u002Fa>\u003C\u002Fli>\n\u003Cli>\u003Ca href=\"https:\u002F\u002Fdocs.aws.amazon.com\u002FAmazonCloudWatchLogs\u002Flatest\u002FAPIReference\u002FAPI_PutLogEvents.html\">logs:PutLogEvents\u003C\u002Fa>\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>When using Amazon MSK, for Lambda to access your cluster on your behalf, your function must have the \u003Ca href=\"https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Fwith-msk.html#msk-permissions-iam-policy\">following permissions\u003C\u002Fa> in its execution role:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>\u003Ca href=\"https:\u002F\u002Fdocs.aws.amazon.com\u002Fmsk\u002F1.0\u002Fapireference\u002Fclusters-clusterarn.html#clusters-clusterarnget\">kafka:DescribeCluster\u003C\u002Fa>\u003C\u002Fli>\n\u003Cli>\u003Ca href=\"https:\u002F\u002Fdocs.aws.amazon.com\u002FMSK\u002F2.0\u002FAPIReference\u002Fv2-clusters-clusterarn.html#v2-clusters-clusterarnget\">kafka:DescribeClusterV2\u003C\u002Fa>\u003C\u002Fli>\n\u003Cli>\u003Ca href=\"https:\u002F\u002Fdocs.aws.amazon.com\u002Fmsk\u002F1.0\u002Fapireference\u002Fclusters-clusterarn-bootstrap-brokers.html#clusters-clusterarn-bootstrap-brokersget\">kafka:GetBootstrapBrokers\u003C\u002Fa>\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>Your Lambda function might also need permissions to:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>Describe your Secrets Manager secret.\u003C\u002Fli>\n\u003Cli>Access your AWS Key Management Service (AWS KMS) customer managed key.\u003C\u002Fli>\n\u003Cli>Access your Amazon VPC.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>To access the Amazon MSK data stream, the Lambda function also needs two Kafka permissions: \u003Ca href=\"https:\u002F\u002Fdocs.aws.amazon.com\u002Fmsk\u002F1.0\u002Fapireference\u002Fclusters-clusterarn.html#clusters-clusterarnget\">kafka:DescribeCluster\u003C\u002Fa> and \u003Ca href=\"https:\u002F\u002Fdocs.aws.amazon.com\u002Fmsk\u002F1.0\u002Fapireference\u002Fclusters-clusterarn-bootstrap-brokers.html#clusters-clusterarn-bootstrap-brokersget\">kafka:GetBootstrapBrokers\u003C\u002Fa>. The policy template \u003Ca href=\"https:\u002F\u002Fdocs.aws.amazon.com\u002Faws-managed-policy\u002Flatest\u002Freference\u002FAWSLambdaMSKExecutionRole.html\">\u003Cem>AWSLambdaMSKExecutionRole\u003C\u002Fem>\u003C\u002Fa> includes these permissions.\u003C\u002Fp>\n\u003Cp>In \u003Ca href=\"https:\u002F\u002Faws.amazon.com\u002Fserverless\u002Fsam\u002F\">AWS Serverless Application Model\u003C\u002Fa> (AWS SAM) templates, you can configure a Lambda function with an \u003Ca href=\"https:\u002F\u002Fdocs.aws.amazon.com\u002Fserverless-application-model\u002Flatest\u002Fdeveloperguide\u002Fsam-property-function-msk.html\">Amazon MSK event source\u003C\u002Fa> mapping and the necessary permissions. For example:\u003C\u002Fp>\n\u003Cpre>\u003Ccode class=\"language-yaml\">Resources:\n  ProcessMSKFunction:\n    Type: AWS::Serverless::Function \n    Properties:\n      CodeUri: code\u002F\n      Timeout: 3\n      Handler: app.handler\n      Runtime: nodejs18.x\n      Policies:\n      - AWSLambdaMSKExecutionRole\n      Events:\n        MSKEvent:\n          Type: MSK\n          Properties:\n            StartingPosition: LATEST\n            Stream: arn:aws:kafka:us-east-1:123456789012:cluster\u002Fmy-cluster\u002F12abcd12-1234-1234-1234-1234abcd1234-1\n            Topics:\n            - MyTopic\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Ch3>Networking\u003C\u002Fh3>\n\u003Cp>When using self-hosted Kafka, you manage the EC2 instances that run Kafka. For high availability, you should run Kafka EC2 instances in a minimum of two Availability Zones in your preferred Region.\u003C\u002Fp>\n\u003Cp>Amazon MSK is a highly available service, so it must be configured to run in a minimum of two Availability Zones in your preferred Region.\u003C\u002Fp>\n\u003Cp>To comply with security best practice, you should deploy your self-managed or Amazon MSK brokers in private subnets in each Region.\u003C\u002Fp>\n\u003Cp>The Lambda function that the ESM invokes does not need connectivity to your Kafka VPC to receive records from Kafka. However, the Lambda service does need a network connection to your Kafka VPC to pull records. Lambda-Kafka ESMs don't inherit the VPC network settings of the Lambda function for both MSK triggers and self-managed Kafka triggers. An MSK ESM uses the subnet and security group settings that are configured on the target MSK cluster. A self-managed Kafka trigger has WAN access by default, but can be configured with network access to a VPC in the same account and AWS Region. Because the network configuration is separate, a Lambda function can run code within a network that doesn't have a route to the Kafka cluster.\u003C\u002Fp>\n\u003Ch4>Self-managed Kafka\u003C\u002Fh4>\n\u003Cp>A self-managed Kafka ESM's network configuration is similar to a Lambda function. By default, the ESM has access to the WAN but isn't configured for access within a VPC. You can manually configure specific subnets and security groups to access a Kafka cluster. The ESM can access a cluster that's reachable from a VPC in the same or a different account that contains the Lambda function.\u003C\u002Fp>\n\u003Cp>For the Lambda function to poll the Kafka instances, you must ensure that there is a NAT Gateway running in the public subnet of each Region. It's possible to route the traffic to a single NAT Gateway in one AZ for test and development workloads. For redundancy in production workloads, it's recommended that there is one NAT Gateway available in each Availability Zone.\u003C\u002Fp>\n\u003Cimg src=\"\u002Fassets\u002Fexternal\u002Fguides\u002Flambda-kafka\u002Fassets\u002Fimages\u002Fkafka-lambda-networking-smk.png\">\n\n\u003Cp>To learn more about configuring the private subnet table to use a NAT Gateway, see this \u003Ca href=\"https:\u002F\u002Faws.amazon.com\u002Fpremiumsupport\u002Fknowledge-center\u002Finternet-access-lambda-function\u002F\">AWS Support response\u003C\u002Fa>. For an example of how to configure the infrastructure with \u003Ca href=\"https:\u002F\u002Faws.amazon.com\u002Fcloudformation\u002F\">AWS CloudFormation\u003C\u002Fa>, see \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fawsdocs\u002Faws-lambda-developer-guide\u002Fblob\u002Fmaster\u002Ftemplates\u002Fvpc-privatepublic.yaml\">this template\u003C\u002Fa>.\u003C\u002Fp>\n\u003Cp>Additional Information: \u003Ca href=\"https:\u002F\u002Faws.amazon.com\u002Fblogs\u002Fcompute\u002Fsetting-up-aws-lambda-with-an-apache-kafka-cluster-within-a-vpc\u002F\">Setting up AWS Lambda with an Apache Kafka cluster within a VPC\u003C\u002Fa>.\u003C\u002Fp>\n\u003Cp>It is possible to create a self-managed Kafka trigger that consumes from an MSK cluster in another account. However, unlike an MSK trigger, AWS IAM authentication isn't available for self-managed Kafka triggers. Also, connecting to the MSK cluster over a VPC-peered connection requires specific VPC workarounds. For more information, see \u003Ca href=\"https:\u002F\u002Faws.amazon.com\u002Fblogs\u002Fbig-data\u002Fconnect-kafka-client-applications-securely-to-your-amazon-msk-cluster-from-different-vpcs-and-aws-accounts\u002F\">Connect Kafka client applications securely to your Amazon MSK cluster from different VPCs and AWS accounts\u003C\u002Fa>.\u003C\u002Fp>\n\u003Ch4>Amazon MSK\u003C\u002Fh4>\n\u003Ch5>Lambda to MSK connectivity\u003C\u002Fh5>\n\u003Cp>To communicate with the MSK cluster, an MSK ESM creates a \u003Ca href=\"https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Ffoundation-networking.html#foundation-nw-eni\">hyperplane elastic network interface\u003C\u002Fa> inside \u003Ca href=\"https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Fconfiguration-vpc.html\">each subnet\u003C\u002Fa> that's used by the cluster. This is similar to how a Lambda function operates within a VPC.\u003C\u002Fp>\n\u003Cp>An MSK ESM doesn't use the Lambda function's VPC settings. Instead, the ESM automatically uses the subnet and security group settings that are configured on the target MSK cluster. The MSK ESM then creates a network interface inside each of the subnets that are used by the MSK cluster. These network interfaces use the same security group that's used by the MSK cluster.\u003C\u002Fp>\n\u003Cimg src=\"\u002Fassets\u002Fexternal\u002Fguides\u002Flambda-kafka\u002Fassets\u002Fimages\u002Fkafka-lambda-networking-msk.png\">\n\n\u003Cp>The MSK cluster's \u003Ca href=\"https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Fwith-msk.html#services-msk-vpc-config\">security group must include a rule\u003C\u002Fa> that grants inbound traffic from itself and outbound traffic to itself. The traffic must also be granted over one of the following open authentication ports that's used by the broker:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>9092 for plaintext\u003C\u002Fli>\n\u003Cli>9094 for TLS\u003C\u002Fli>\n\u003Cli>9096 for SASL\u003C\u002Fli>\n\u003Cli>443 for all configurations\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Ch5>VPC outbound connectivity\u003C\u002Fh5>\n\u003Cp>The Provisioned Mode for the Kafka ESM features simplified networking, so you don't need to worry about outbound connectivity. This also has the advantage of reducing your networking charges.\u003C\u002Fp>\n\u003Cp>If you're using the On-demand(default) version of the Lambda Kafka ESM your MSK VPC also requires outbound network connectivity to Lambda and AWS Security Token Service (STS). The ESM contacts STS from your MSK VPC network to assume a role for the target Lambda function and invokes the function. For some authentication options, it also requires access to AWS Secrets Manager.\u003C\u002Fp>\n\u003Cp>You can configure outbound connectivity to AWS services using VPC endpoints or NAT Gateway.\u003C\u002Fp>\n\u003Cp>Create VPC endpoints for Lambda, STS, as well as Secrets Manager, if required, from your private subnets.\u003C\u002Fp>\n\u003Cimg src=\"\u002Fassets\u002Fexternal\u002Fguides\u002Flambda-kafka\u002Fassets\u002Fimages\u002Fkafka-lambda-vpc-endpoints.png\">\n\n\u003Cp>If using NAT Gateway, you must ensure that there is a NAT Gateway running in the public subnet of each Region. It's possible to route the traffic to a single NAT Gateway in one AZ for test and development workloads. For redundancy in production workloads, it's recommended that there is one NAT Gateway available in each Availability Zone.\u003C\u002Fp>\n\u003Cp>\u003Cimg src=\"\u002Fassets\u002Fexternal\u002Fguides\u002Flambda-kafka\u002Fassets\u002Fimages\u002Fkafka-lambda-nat-gateway.png\">o\u003C\u002Fp>\n\u003Ch5>Cross Account Event Source Mappings\u003C\u002Fh5>\n\u003Cp>You can configure cross account event source mappings for MSK clusters running in another account. This is done using \u003Ca href=\"https:\u002F\u002Fdocs.aws.amazon.com\u002Fmsk\u002Flatest\u002Fdeveloperguide\u002Faws-access-mult-vpc.html\">MSK multi-vpc private connectivity\u003C\u002Fa>. Note that this isn't currently supported for MSK Serverless clusters. For more details \u003Ca href=\"https:\u002F\u002Faws.amazon.com\u002Fblogs\u002Fcompute\u002Ftriggering-aws-lambda-function-from-a-cross-account-amazon-managed-streaming-for-apache-kafka\u002F\">this blog details the setup process\u003C\u002Fa>\u003C\u002Fp>\n",{"title":6,"order":7},"Configuring Kafka and Lambda",4,"There are a number of guides to setting up Kafka and the consumer Lambda ESM.\n\nThe [Kafka](https:\u002F\u002Fkafka.apache.org\u002Fdocumentation\u002F) and [Confluent](https:\u002F\u002Fdocs.confluent.io\u002Fhome\u002Foverview.html) documentation includes steps to install Kafka. [Using self-hosted Apache Kafka as an event source for AWS Lambda](https:\u002F\u002Faws.amazon.com\u002Fblogs\u002Fcompute\u002Fusing-self-hosted-apache-kafka-as-an-event-source-for-aws-lambda\u002F) describes setting up the Lambda ESM when consuming from self-hosted Kafka.\n\nThis [Amazon MSK](https:\u002F\u002Fdocs.aws.amazon.com\u002Fmsk\u002Flatest\u002Fdeveloperguide\u002Fgetting-started.html) tutorial shows an example of how to create an MSK cluster, produce and consume data, and monitor the health of your cluster using metrics. [Using Amazon MSK as an event source for AWS Lambda](https:\u002F\u002Faws.amazon.com\u002Fblogs\u002Fcompute\u002Fusing-amazon-msk-as-an-event-source-for-aws-lambda\u002F) describes setting up Lambda ESM to read from Amazon MSK. The  [Amazon MSK Labs](https:\u002F\u002Fcatalog.us-east-1.prod.workshops.aws\u002Fworkshops\u002Fc2b72b6f-666b-4596-b8bc-bafa5dcca741\u002Fen-US) workshop provides an install guide.\n\nThere are a number of [Serverless Patterns](https:\u002F\u002Fserverlessland.com\u002Fpatterns?services=msk) you can use to deploy and test Amazon MSK integrations. You can deploy an Amazon MSK cluster using the [SASL authentication](https:\u002F\u002Fserverlessland.com\u002Fpatterns\u002Fmsk-cfn-sasl-lambda) pattern. For IAM authentication, there are patterns for [Node.js](https:\u002F\u002Fserverlessland.com\u002Fpatterns\u002Fmsk-lambda-iam-node-sam) and [Python](https:\u002F\u002Fserverlessland.com\u002Fpatterns\u002Fmsk-lambda-iam-python-sam).\n\n### Security\n\nLambda supports several methods to authenticate with your self-managed Kafka cluster and Amazon MSK.\n\n##### SASL\u002FSCRAM authentication\n\nLambda supports Simple Authentication and Security Layer\u002FSalted Challenge Response Authentication Mechanism (SASL\u002FSCRAM) authentication with TLS encryption. Lambda sends the encrypted credentials to authenticate with the cluster. Lambda doesn't support SASL\u002FSCRAM with plaintext. Amazon MSK Serverless doesn't support SASL\u002FSCRAM. The [SASL\u002FSCRAM authentication](https:\u002F\u002Fserverlessland.com\u002Fpatterns\u002Fmsk-cfn-sasl-lambda) pattern shows how to use this authentication mechanism with Amazon MSK.\n\nLambda with self-managed Kafka supports SASL\u002FPLAIN authentication. Because this uses clear text credentials, the connection to the server must use TLS encryption to ensure that the credentials are protected. Amazon MSK doesn't support SASL\u002FPLAIN authentication.\n\nFor SASL authentication, you store the sign-in credentials as a secret in AWS Secrets Manager. For more information about using Secrets Manager, see [AWS Secrets Manager tutorials](https:\u002F\u002Fdocs.aws.amazon.com\u002Fsecretsmanager\u002Flatest\u002Fuserguide\u002Ftutorials.html).\n\n##### Mutual TLS authentication\n\nMutual TLS (mTLS) provides two-way authentication between the client and server. The client sends a certificate to the server for the server to verify the client, and the server sends a certificate to the client for the client to verify the server.\n\nUsing mTLS with Kafka, Lambda acts as the client. You configure a client certificate (as a secret in Secrets Manager) to authenticate Lambda with your Kafka brokers. The client certificate must be signed by a CA in the server's trust store.\n\nFor more information about mTLS, see [Introducing mutual TLS authentication for Amazon MSK as an event source](http:\u002F\u002Faws.amazon.com\u002Fblogs\u002Fcompute\u002Fintroducing-mutual-tls-authentication-for-amazon-msk-as-an-event-source).\n\nAmazon MSK, in addition to supporting the same authentication as self-hosted Kafka, supports the following options:\n\n##### Unauthenticated access\n\nIf no clients access the cluster over the internet, you can use unauthenticated access.\n\n##### IAM role-based authentication\n\nYou can use IAM to authenticate the identity of clients that connect to the MSK cluster. For Lambda, the ESM handles the authentication. If IAM Auth is active on your MSK cluster, and you don't provide a secret for auth, Lambda automatically defaults to using IAM Auth. You can deploy patterns for [Node.js](https:\u002F\u002Fserverlessland.com\u002Fpatterns\u002Fmsk-lambda-iam-node-sam) and [Python](https:\u002F\u002Fserverlessland.com\u002Fpatterns\u002Fmsk-lambda-iam-python-sam), which show how the ESM makes it easier to use IAM authentication. When producing Kafka records, there are open-source client helper libraries and code samples for [popular languages](https:\u002F\u002Faws.amazon.com\u002Fblogs\u002Fbig-data\u002Famazon-msk-iam-authentication-now-supports-all-programming-languages\u002F).\n\nMake sure that you configure the Kafka cluster to use one of these supported authentication methods. For more information about general Kafka security, see the Kafka [Security](http:\u002F\u002Fkafka.apache.org\u002Fdocumentation.html#security) documentation. \n\nFor more information on Lambda integration authentication, see the specific [Amazon MSK](https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Fwith-msk.html#msk-cluster-permissions) or [self-managed Kafka](https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Fwith-kafka.html#smaa-authentication) documentation.\n\nLambda and Kafka supported authentication methods:\n\n|  | SASL\u002FSCRAM + TLS | SASL\u002FPLAIN | IAM | TLS | Unauthenticated |\n|--------|-------|--------|-------|--------|--------|\n| Self-managed Kafka | Yes | Yes | No | Yes | No |\n| Amazon MSK | Yes | No | Yes | Yes | Yes |\n| Amazon MSK Serverless | Yes | No | Yes | Yes | Yes |\n \n\n\n#### Lambda permissions\n\nLambda can authenticate against Kafka using IAM or SASL.\n\nThe Lambda function target in the event source mapping does not need to be running in a VPC to receive records from Amazon MSK. For more information on Kafka networking requirements for Lambda, see the _Networking_ section below.\n\nThe Lambda function must have permission to describe VPCs and security groups, and manage elastic network interfaces. These execution roles permissions are:\n\n- ec2:CreateNetworkInterface\n- ec2:DescribeNetworkInterfaces\n- ec2:DescribeVpcs\n- ec2:DeleteNetworkInterface\n- ec2:DescribeSubnets\n- ec2:DescribeSecurityGroups\n\nTo create and store logs in a log group in Amazon CloudWatch Logs, your Lambda function must have the following permissions in its execution role:\n\n- [logs:CreateLogGroup](https:\u002F\u002Fdocs.aws.amazon.com\u002FAmazonCloudWatchLogs\u002Flatest\u002FAPIReference\u002FAPI_CreateLogGroup.html)\n- [logs:CreateLogStream](https:\u002F\u002Fdocs.aws.amazon.com\u002FAmazonCloudWatchLogs\u002Flatest\u002FAPIReference\u002FAPI_CreateLogStream.html)\n- [logs:PutLogEvents](https:\u002F\u002Fdocs.aws.amazon.com\u002FAmazonCloudWatchLogs\u002Flatest\u002FAPIReference\u002FAPI_PutLogEvents.html)\n\nWhen using Amazon MSK, for Lambda to access your cluster on your behalf, your function must have the [following permissions](https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Fwith-msk.html#msk-permissions-iam-policy) in its execution role:\n\n- [kafka:DescribeCluster](https:\u002F\u002Fdocs.aws.amazon.com\u002Fmsk\u002F1.0\u002Fapireference\u002Fclusters-clusterarn.html#clusters-clusterarnget)\n- [kafka:DescribeClusterV2](https:\u002F\u002Fdocs.aws.amazon.com\u002FMSK\u002F2.0\u002FAPIReference\u002Fv2-clusters-clusterarn.html#v2-clusters-clusterarnget)\n- [kafka:GetBootstrapBrokers](https:\u002F\u002Fdocs.aws.amazon.com\u002Fmsk\u002F1.0\u002Fapireference\u002Fclusters-clusterarn-bootstrap-brokers.html#clusters-clusterarn-bootstrap-brokersget)\n\nYour Lambda function might also need permissions to:\n\n- Describe your Secrets Manager secret.\n- Access your AWS Key Management Service (AWS KMS) customer managed key.\n- Access your Amazon VPC.\n\nTo access the Amazon MSK data stream, the Lambda function also needs two Kafka permissions: [kafka:DescribeCluster](https:\u002F\u002Fdocs.aws.amazon.com\u002Fmsk\u002F1.0\u002Fapireference\u002Fclusters-clusterarn.html#clusters-clusterarnget) and [kafka:GetBootstrapBrokers](https:\u002F\u002Fdocs.aws.amazon.com\u002Fmsk\u002F1.0\u002Fapireference\u002Fclusters-clusterarn-bootstrap-brokers.html#clusters-clusterarn-bootstrap-brokersget). The policy template [_AWSLambdaMSKExecutionRole_](https:\u002F\u002Fdocs.aws.amazon.com\u002Faws-managed-policy\u002Flatest\u002Freference\u002FAWSLambdaMSKExecutionRole.html) includes these permissions.\n\nIn [AWS Serverless Application Model](https:\u002F\u002Faws.amazon.com\u002Fserverless\u002Fsam\u002F) (AWS SAM) templates, you can configure a Lambda function with an [Amazon MSK event source](https:\u002F\u002Fdocs.aws.amazon.com\u002Fserverless-application-model\u002Flatest\u002Fdeveloperguide\u002Fsam-property-function-msk.html) mapping and the necessary permissions. For example:\n\n```yaml\nResources:\n  ProcessMSKFunction:\n    Type: AWS::Serverless::Function \n    Properties:\n      CodeUri: code\u002F\n      Timeout: 3\n      Handler: app.handler\n      Runtime: nodejs18.x\n      Policies:\n      - AWSLambdaMSKExecutionRole\n      Events:\n        MSKEvent:\n          Type: MSK\n          Properties:\n            StartingPosition: LATEST\n            Stream: arn:aws:kafka:us-east-1:123456789012:cluster\u002Fmy-cluster\u002F12abcd12-1234-1234-1234-1234abcd1234-1\n            Topics:\n            - MyTopic\n```\n### Networking\n\nWhen using self-hosted Kafka, you manage the EC2 instances that run Kafka. For high availability, you should run Kafka EC2 instances in a minimum of two Availability Zones in your preferred Region.\n\nAmazon MSK is a highly available service, so it must be configured to run in a minimum of two Availability Zones in your preferred Region.\n\nTo comply with security best practice, you should deploy your self-managed or Amazon MSK brokers in private subnets in each Region.\n\nThe Lambda function that the ESM invokes does not need connectivity to your Kafka VPC to receive records from Kafka. However, the Lambda service does need a network connection to your Kafka VPC to pull records. Lambda-Kafka ESMs don't inherit the VPC network settings of the Lambda function for both MSK triggers and self-managed Kafka triggers. An MSK ESM uses the subnet and security group settings that are configured on the target MSK cluster. A self-managed Kafka trigger has WAN access by default, but can be configured with network access to a VPC in the same account and AWS Region. Because the network configuration is separate, a Lambda function can run code within a network that doesn't have a route to the Kafka cluster.\n\n#### Self-managed Kafka\n\nA self-managed Kafka ESM's network configuration is similar to a Lambda function. By default, the ESM has access to the WAN but isn't configured for access within a VPC. You can manually configure specific subnets and security groups to access a Kafka cluster. The ESM can access a cluster that's reachable from a VPC in the same or a different account that contains the Lambda function.\n\nFor the Lambda function to poll the Kafka instances, you must ensure that there is a NAT Gateway running in the public subnet of each Region. It's possible to route the traffic to a single NAT Gateway in one AZ for test and development workloads. For redundancy in production workloads, it's recommended that there is one NAT Gateway available in each Availability Zone.\n\n\u003Cimg src=\"\u002Fassets\u002Fexternal\u002Fguides\u002Flambda-kafka\u002Fassets\u002Fimages\u002Fkafka-lambda-networking-smk.png\">\n\nTo learn more about configuring the private subnet table to use a NAT Gateway, see this [AWS Support response](https:\u002F\u002Faws.amazon.com\u002Fpremiumsupport\u002Fknowledge-center\u002Finternet-access-lambda-function\u002F). For an example of how to configure the infrastructure with [AWS CloudFormation](https:\u002F\u002Faws.amazon.com\u002Fcloudformation\u002F), see [this template](https:\u002F\u002Fgithub.com\u002Fawsdocs\u002Faws-lambda-developer-guide\u002Fblob\u002Fmaster\u002Ftemplates\u002Fvpc-privatepublic.yaml).\n\nAdditional Information: [Setting up AWS Lambda with an Apache Kafka cluster within a VPC](https:\u002F\u002Faws.amazon.com\u002Fblogs\u002Fcompute\u002Fsetting-up-aws-lambda-with-an-apache-kafka-cluster-within-a-vpc\u002F).\n\nIt is possible to create a self-managed Kafka trigger that consumes from an MSK cluster in another account. However, unlike an MSK trigger, AWS IAM authentication isn't available for self-managed Kafka triggers. Also, connecting to the MSK cluster over a VPC-peered connection requires specific VPC workarounds. For more information, see [Connect Kafka client applications securely to your Amazon MSK cluster from different VPCs and AWS accounts](https:\u002F\u002Faws.amazon.com\u002Fblogs\u002Fbig-data\u002Fconnect-kafka-client-applications-securely-to-your-amazon-msk-cluster-from-different-vpcs-and-aws-accounts\u002F).\n\n#### Amazon MSK\n\n##### Lambda to MSK connectivity\n\nTo communicate with the MSK cluster, an MSK ESM creates a [hyperplane elastic network interface](https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Ffoundation-networking.html#foundation-nw-eni) inside [each subnet](https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Fconfiguration-vpc.html) that's used by the cluster. This is similar to how a Lambda function operates within a VPC.\n\nAn MSK ESM doesn't use the Lambda function's VPC settings. Instead, the ESM automatically uses the subnet and security group settings that are configured on the target MSK cluster. The MSK ESM then creates a network interface inside each of the subnets that are used by the MSK cluster. These network interfaces use the same security group that's used by the MSK cluster.\n\n\u003Cimg src=\"\u002Fassets\u002Fexternal\u002Fguides\u002Flambda-kafka\u002Fassets\u002Fimages\u002Fkafka-lambda-networking-msk.png\">\n\nThe MSK cluster's [security group must include a rule](https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Fwith-msk.html#services-msk-vpc-config) that grants inbound traffic from itself and outbound traffic to itself. The traffic must also be granted over one of the following open authentication ports that's used by the broker:\n\n- 9092 for plaintext\n- 9094 for TLS\n- 9096 for SASL\n- 443 for all configurations\n\n##### VPC outbound connectivity\n\nThe Provisioned Mode for the Kafka ESM features simplified networking, so you don't need to worry about outbound connectivity. This also has the advantage of reducing your networking charges.\n\nIf you're using the On-demand(default) version of the Lambda Kafka ESM your MSK VPC also requires outbound network connectivity to Lambda and AWS Security Token Service (STS). The ESM contacts STS from your MSK VPC network to assume a role for the target Lambda function and invokes the function. For some authentication options, it also requires access to AWS Secrets Manager.\n\nYou can configure outbound connectivity to AWS services using VPC endpoints or NAT Gateway.\n\nCreate VPC endpoints for Lambda, STS, as well as Secrets Manager, if required, from your private subnets.\n\n\u003Cimg src=\"\u002Fassets\u002Fexternal\u002Fguides\u002Flambda-kafka\u002Fassets\u002Fimages\u002Fkafka-lambda-vpc-endpoints.png\">\n\nIf using NAT Gateway, you must ensure that there is a NAT Gateway running in the public subnet of each Region. It's possible to route the traffic to a single NAT Gateway in one AZ for test and development workloads. For redundancy in production workloads, it's recommended that there is one NAT Gateway available in each Availability Zone.\n\n\u003Cimg src=\"\u002Fassets\u002Fexternal\u002Fguides\u002Flambda-kafka\u002Fassets\u002Fimages\u002Fkafka-lambda-nat-gateway.png\">o\n\n##### Cross Account Event Source Mappings\n\nYou can configure cross account event source mappings for MSK clusters running in another account. This is done using [MSK multi-vpc private connectivity](https:\u002F\u002Fdocs.aws.amazon.com\u002Fmsk\u002Flatest\u002Fdeveloperguide\u002Faws-access-mult-vpc.html). Note that this isn't currently supported for MSK Serverless clusters. For more details [this blog details the setup process](https:\u002F\u002Faws.amazon.com\u002Fblogs\u002Fcompute\u002Ftriggering-aws-lambda-function-from-a-cross-account-amazon-managed-streaming-for-apache-kafka\u002F)\n",[10],"content\u002Fcontributors\u002Fjulian-wood.json","LESSON","Using AWS Lambda to process Apache Kafka streams","\u002Fcontent\u002Fguides\u002Flambda-kafka\u002Fconfiguring-kafka-lambda","9 min",[16,25,34,42,47,55,64,73,82],{"title":17,"content":18},"Introducing streaming",[19],{"title":17,"order":20,"time":21,"path":22,"id":23,"link":24},1,"4 min","introduction","introduction.md","\u002Fcontent\u002Fguides\u002Flambda-kafka\u002Fintroduction",{"title":26,"content":27},"Using Kafka to build your streaming application",[28],{"title":26,"order":29,"time":30,"path":31,"id":32,"link":33},2,"6 min","using-kafka","using-kafka.md","\u002Fcontent\u002Fguides\u002Flambda-kafka\u002Fusing-kafka",{"title":35,"content":36},"Using Lambda to consume records from Kafka",[37],{"title":35,"order":38,"time":21,"path":39,"id":40,"link":41},3,"using-lambda","using-lambda.md","\u002Fcontent\u002Fguides\u002Flambda-kafka\u002Fusing-lambda",{"title":6,"content":43},[44],{"title":6,"order":7,"time":14,"path":45,"id":46,"link":13},"configuring-kafka-lambda","configuring-kafka-lambda.md",{"title":48,"content":49},"Processing Kafka streams using the Lambda ESM",[50],{"title":48,"order":51,"time":14,"path":52,"id":53,"link":54},5,"processing-esm","processing-esm.md","\u002Fcontent\u002Fguides\u002Flambda-kafka\u002Fprocessing-esm",{"title":56,"content":57},"Scaling and throughput",[58],{"title":56,"order":59,"time":60,"path":61,"id":62,"link":63},6,"5 min","scaling-throughput","scaling-throughput.md","\u002Fcontent\u002Fguides\u002Flambda-kafka\u002Fscaling-throughput",{"title":65,"content":66},"Monitoring and observability",[67],{"title":65,"order":68,"time":69,"path":70,"id":71,"link":72},7,"2 min","monitoring-observability","monitoring-observability.md","\u002Fcontent\u002Fguides\u002Flambda-kafka\u002Fmonitoring-observability",{"title":74,"content":75},"Troubleshooting",[76],{"title":74,"order":77,"time":78,"path":79,"id":80,"link":81},8,"3 min","troubleshooting","troubleshooting.md","\u002Fcontent\u002Fguides\u002Flambda-kafka\u002Ftroubleshooting",{"title":83,"content":84},"Resources",[85],{"title":83,"order":86,"time":87,"path":88,"id":89,"link":90},9,"1 min","resources","resources.md","\u002Fcontent\u002Fguides\u002Flambda-kafka\u002Fresources",1792578865930]