[{"data":1,"prerenderedAt":91},["ShallowReactive",2],{"content-guides-lambda-kafka-using-lambda":3},{"markdown":4,"frontMatterAttributes":5,"bodyRaw":8,"contributorPaths":9,"menuType":11,"contentName":12,"slug":13,"readingTime":14,"menu":15,"nextDocItem":41,"previousDocItem":27},"\u003Cp>Processing streaming data can be complex in traditional, server-based architectures, especially if you must react in real-time. Many organizations spend significant time and cost managing and scaling their streaming platforms. In order to react fast, they must provision for peak capacity, which adds complexity.\u003C\u002Fp>\n\u003Cp>\u003Ca href=\"https:\u002F\u002Faws.amazon.com\u002Flambda\u002F\">AWS Lambda\u003C\u002Fa> is a serverless compute service that removes the undifferentiated heavy lifting when processing Kafka streams. You don't have to manage infrastructure, can reduce operational overhead, lower costs, and scale on-demand. This helps you focus more on building streaming applications. You can write Lambda functions in \u003Ca href=\"https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Flambda-runtimes.html\">a number of programming languages\u003C\u002Fa>, which provide flexibility when processing streaming data.\u003C\u002Fp>\n\u003Cp>For an introduction to Lambda, see \u003Ca href=\"https:\u002F\u002Fserverlessland.com\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Faws-lambda-fundamentals\u002Fwhat-is-aws-lambda\">AWS Lambda Fundamentals\u003C\u002Fa>.\u003C\u002Fp>\n\u003Cp>There are three ways to use Lambda to process streams: Kafka Connect, \u003Ca href=\"https:\u002F\u002Faws.amazon.com\u002Feventbridge\u002Fpipes\u002F\">Amazon EventBridge Pipes\u003C\u002Fa>, and Lambda event source mappings (ESM). This learning guide focuses primarily on the Lambda ESM.\u003C\u002Fp>\n\u003Ch4>Kafka Connect\u003C\u002Fh4>\n\u003Cp>\u003Ca href=\"https:\u002F\u002Fkafka.apache.org\u002Fdocumentation\u002F#connect\">Kafka Connect\u003C\u002Fa> is a framework to integrate Kafka with other systems using Connectors, which are named depending on the data movement direction. Source connectors read data from systems and store them in Kafka topics. Sink connectors deliver data from Kafka topics into other systems, such as Lambda. Confluent has a pre-built Kafka connector for Lambda. The \u003Ca href=\"https:\u002F\u002Fdocs.confluent.io\u002Fkafka-connectors\u002Faws-lambda\u002Fcurrent\u002Foverview.html\">Confluent Lambda sink connector\u003C\u002Fa> pulls records from one or more Kafka topics, batches them, converts them to JSON, and invokes a Lambda function. You can invoke the Lambda function either synchronously or asynchronously.\u003C\u002Fp>\n\u003Cimg src=\"\u002Fassets\u002Fexternal\u002Fguides\u002Flambda-kafka\u002Fassets\u002Fimages\u002Fkafka-lambda-kafka-connect.png\">\n\n\u003Ch4>Lambda event source mapping\u003C\u002Fh4>\n\u003Cp>Lambda can also integrate natively with your Kafka environments as a consumer to process stream data as soon as it's generated. Lambda and serverless architectures are well-suited for stream processing workloads that are event-driven and have burst or variable compute requirements.\u003C\u002Fp>\n\u003Cp>To consume streaming data from Kafka, you configure an event source mapping (ESM) on your Lambda functions. This is a Lambda service managed resource, which is separate from your function. It continually polls records from the topics in the Kafka cluster. The ESM optionally filters them and batches those records into a payload. Then, it calls the Lambda Invoke API to deliver the payload to your Lambda function synchronously for processing. The ESM can scale up automatically to handle additional load. You can write your processing function in any language.\u003C\u002Fp>\n\u003Cp>The Lambda ESM has a Provisioned Mode which has configurable minimum and maximum scaling limits, as well as more responsive auto-scaling. The Provisioned Mode also supports features like Schema Registry support and native processing of Avro and Protobuf formatted records.\u003C\u002Fp>\n\u003Cp>As Lambda managed the pollers, you don't need to manage a fleet of consumers across multiple teams. Each team creates their ESM and Lambda handles the polling.\u003C\u002Fp>\n\u003Cimg src=\"\u002Fassets\u002Fexternal\u002Fguides\u002Flambda-kafka\u002Fassets\u002Fimages\u002Fkafka-lambda-esm.png\">\n\n\u003Cp>The Lambda function's event payload contains an array of records. Each array item contains details of the topic and Kafka partition identifier, together with a timestamp and \u003Ca href=\"https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FBase64\">base64 encoded\u003C\u002Fa> record:\u003C\u002Fp>\n\u003Cpre>\u003Ccode class=\"language-json\">{\n    "eventSource": "aws:kafka",\n    "eventSourceArn": "arn:aws:kafka:us-east-1:123456789012:cluster\u002Fvpc-2priv-2pub\u002F751d2973-a626-431c-9d4e-d7975eb44dd7-2",\n    "records": {\n      "mytopic-0": [\n          {\n            "topic": "mytopic"\n            "partition": "0",\n            "offset": 15,\n            "timestamp": 1545084650987,\n            "timestampType": "CREATE_TIME",\n            "key":"abcDEFghiJKLmnoPQRstuVWXyz1234==",\n            "value": "SGVsbG8sIHRoaXMgaXMgYSB0ZXN0Lg==",\n            "headers":[\n               {\n                   "headerKey":[\n                      104,\n                      101,\n                      101\n                    ]\n                }\n            ]\n          }\n      ]\n    }\n}\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Ch4>Amazon EventBridge Pipes\u003C\u002Fh4>\n\u003Cp>Amazon EventBridge Pipes helps you create point-to-point integrations between event producers and consumers. You can include optional steps for filtering, transformation, and enrichment.\u003C\u002Fp>\n\u003Cp>You can move data from your on-premises or self-hosted Kafka topics to AWS services such as Amazon Data Firehose or Amazon Simple Queue Service (SQS).\u003C\u002Fp>\n\u003Cimg src=\"\u002Fassets\u002Fexternal\u002Fguides\u002Flambda-kafka\u002Fassets\u002Fimages\u002Fkafka-lambda-eventbridge-pipes.png\">\n\n\u003Cp>AWS Lambda's Event Source Mapping (ESM) and Amazon EventBridge Pipes use the same polling infrastructure to select and send events. EventBridge Pipes is ideal if you need a light processing ETL workflow to one of over 14 supported targets. You don't need to manage Lambda code to send records to a target, which is managed by Pipes. Kafka can be a Pipes source but not a target. ESM is ideal if you want to use Lambda with rich processing to send records to any target.\u003C\u002Fp>\n\u003Ch4>Using Lambda to produce Kafka records\u003C\u002Fh4>\n\u003Cp>You can also use Lambda as a producer to write records to a Kafka topic. \u003Ca href=\"https:\u002F\u002Faws.amazon.com\u002Fpt\u002Fblogs\u002Fcompute\u002Fcreating-a-serverless-apache-kafka-publisher-using-aws-lambda\u002F\">Creating a serverless Apache Kafka publisher using AWS Lambda\u003C\u002Fa> is an example that provides an API to write to Kafka.\u003C\u002Fp>\n",{"title":6,"order":7},"Using Lambda to consume records from Kafka",3,"Processing streaming data can be complex in traditional, server-based architectures, especially if you must react in real-time. Many organizations spend significant time and cost managing and scaling their streaming platforms. In order to react fast, they must provision for peak capacity, which adds complexity.\n\n[AWS Lambda](https:\u002F\u002Faws.amazon.com\u002Flambda\u002F) is a serverless compute service that removes the undifferentiated heavy lifting when processing Kafka streams. You don't have to manage infrastructure, can reduce operational overhead, lower costs, and scale on-demand. This helps you focus more on building streaming applications. You can write Lambda functions in [a number of programming languages](https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Flambda-runtimes.html), which provide flexibility when processing streaming data.\n\nFor an introduction to Lambda, see [AWS Lambda Fundamentals](https:\u002F\u002Fserverlessland.com\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Faws-lambda-fundamentals\u002Fwhat-is-aws-lambda).\n\nThere are three ways to use Lambda to process streams: Kafka Connect, [Amazon EventBridge Pipes](https:\u002F\u002Faws.amazon.com\u002Feventbridge\u002Fpipes\u002F), and Lambda event source mappings (ESM). This learning guide focuses primarily on the Lambda ESM.\n\n#### Kafka Connect\n\n[Kafka Connect](https:\u002F\u002Fkafka.apache.org\u002Fdocumentation\u002F#connect) is a framework to integrate Kafka with other systems using Connectors, which are named depending on the data movement direction. Source connectors read data from systems and store them in Kafka topics. Sink connectors deliver data from Kafka topics into other systems, such as Lambda. Confluent has a pre-built Kafka connector for Lambda. The [Confluent Lambda sink connector](https:\u002F\u002Fdocs.confluent.io\u002Fkafka-connectors\u002Faws-lambda\u002Fcurrent\u002Foverview.html) pulls records from one or more Kafka topics, batches them, converts them to JSON, and invokes a Lambda function. You can invoke the Lambda function either synchronously or asynchronously.\n\n\u003Cimg src=\"\u002Fassets\u002Fexternal\u002Fguides\u002Flambda-kafka\u002Fassets\u002Fimages\u002Fkafka-lambda-kafka-connect.png\">\n\n#### Lambda event source mapping\n\nLambda can also integrate natively with your Kafka environments as a consumer to process stream data as soon as it's generated. Lambda and serverless architectures are well-suited for stream processing workloads that are event-driven and have burst or variable compute requirements.\n\nTo consume streaming data from Kafka, you configure an event source mapping (ESM) on your Lambda functions. This is a Lambda service managed resource, which is separate from your function. It continually polls records from the topics in the Kafka cluster. The ESM optionally filters them and batches those records into a payload. Then, it calls the Lambda Invoke API to deliver the payload to your Lambda function synchronously for processing. The ESM can scale up automatically to handle additional load. You can write your processing function in any language.\n\nThe Lambda ESM has a Provisioned Mode which has configurable minimum and maximum scaling limits, as well as more responsive auto-scaling. The Provisioned Mode also supports features like Schema Registry support and native processing of Avro and Protobuf formatted records.\n\nAs Lambda managed the pollers, you don't need to manage a fleet of consumers across multiple teams. Each team creates their ESM and Lambda handles the polling.\n\n\u003Cimg src=\"\u002Fassets\u002Fexternal\u002Fguides\u002Flambda-kafka\u002Fassets\u002Fimages\u002Fkafka-lambda-esm.png\">\n\nThe Lambda function's event payload contains an array of records. Each array item contains details of the topic and Kafka partition identifier, together with a timestamp and [base64 encoded](https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FBase64) record:\n\n```json\n{\n    \"eventSource\": \"aws:kafka\",\n    \"eventSourceArn\": \"arn:aws:kafka:us-east-1:123456789012:cluster\u002Fvpc-2priv-2pub\u002F751d2973-a626-431c-9d4e-d7975eb44dd7-2\",\n    \"records\": {\n      \"mytopic-0\": [\n          {\n            \"topic\": \"mytopic\"\n            \"partition\": \"0\",\n            \"offset\": 15,\n            \"timestamp\": 1545084650987,\n            \"timestampType\": \"CREATE_TIME\",\n            \"key\":\"abcDEFghiJKLmnoPQRstuVWXyz1234==\",\n            \"value\": \"SGVsbG8sIHRoaXMgaXMgYSB0ZXN0Lg==\",\n            \"headers\":[\n               {\n                   \"headerKey\":[\n                      104,\n                      101,\n                      101\n                    ]\n                }\n            ]\n          }\n      ]\n    }\n}\n```\n\n#### Amazon EventBridge Pipes\n\nAmazon EventBridge Pipes helps you create point-to-point integrations between event producers and consumers. You can include optional steps for filtering, transformation, and enrichment.\n\nYou can move data from your on-premises or self-hosted Kafka topics to AWS services such as Amazon Data Firehose or Amazon Simple Queue Service (SQS).\n\n\u003Cimg src=\"\u002Fassets\u002Fexternal\u002Fguides\u002Flambda-kafka\u002Fassets\u002Fimages\u002Fkafka-lambda-eventbridge-pipes.png\">\n\nAWS Lambda's Event Source Mapping (ESM) and Amazon EventBridge Pipes use the same polling infrastructure to select and send events. EventBridge Pipes is ideal if you need a light processing ETL workflow to one of over 14 supported targets. You don't need to manage Lambda code to send records to a target, which is managed by Pipes. Kafka can be a Pipes source but not a target. ESM is ideal if you want to use Lambda with rich processing to send records to any target.\n\n#### Using Lambda to produce Kafka records\n\nYou can also use Lambda as a producer to write records to a Kafka topic. [Creating a serverless Apache Kafka publisher using AWS Lambda](https:\u002F\u002Faws.amazon.com\u002Fpt\u002Fblogs\u002Fcompute\u002Fcreating-a-serverless-apache-kafka-publisher-using-aws-lambda\u002F) is an example that provides an API to write to Kafka.\n",[10],"content\u002Fcontributors\u002Fjulian-wood.json","LESSON","Using AWS Lambda to process Apache Kafka streams","\u002Fcontent\u002Fguides\u002Flambda-kafka\u002Fusing-lambda","4 min",[16,24,33,38,47,55,64,73,82],{"title":17,"content":18},"Introducing streaming",[19],{"title":17,"order":20,"time":14,"path":21,"id":22,"link":23},1,"introduction","introduction.md","\u002Fcontent\u002Fguides\u002Flambda-kafka\u002Fintroduction",{"title":25,"content":26},"Using Kafka to build your streaming application",[27],{"title":25,"order":28,"time":29,"path":30,"id":31,"link":32},2,"6 min","using-kafka","using-kafka.md","\u002Fcontent\u002Fguides\u002Flambda-kafka\u002Fusing-kafka",{"title":6,"content":34},[35],{"title":6,"order":7,"time":14,"path":36,"id":37,"link":13},"using-lambda","using-lambda.md",{"title":39,"content":40},"Configuring Kafka and Lambda",[41],{"title":39,"order":42,"time":43,"path":44,"id":45,"link":46},4,"9 min","configuring-kafka-lambda","configuring-kafka-lambda.md","\u002Fcontent\u002Fguides\u002Flambda-kafka\u002Fconfiguring-kafka-lambda",{"title":48,"content":49},"Processing Kafka streams using the Lambda ESM",[50],{"title":48,"order":51,"time":43,"path":52,"id":53,"link":54},5,"processing-esm","processing-esm.md","\u002Fcontent\u002Fguides\u002Flambda-kafka\u002Fprocessing-esm",{"title":56,"content":57},"Scaling and throughput",[58],{"title":56,"order":59,"time":60,"path":61,"id":62,"link":63},6,"5 min","scaling-throughput","scaling-throughput.md","\u002Fcontent\u002Fguides\u002Flambda-kafka\u002Fscaling-throughput",{"title":65,"content":66},"Monitoring and observability",[67],{"title":65,"order":68,"time":69,"path":70,"id":71,"link":72},7,"2 min","monitoring-observability","monitoring-observability.md","\u002Fcontent\u002Fguides\u002Flambda-kafka\u002Fmonitoring-observability",{"title":74,"content":75},"Troubleshooting",[76],{"title":74,"order":77,"time":78,"path":79,"id":80,"link":81},8,"3 min","troubleshooting","troubleshooting.md","\u002Fcontent\u002Fguides\u002Flambda-kafka\u002Ftroubleshooting",{"title":83,"content":84},"Resources",[85],{"title":83,"order":86,"time":87,"path":88,"id":89,"link":90},9,"1 min","resources","resources.md","\u002Fcontent\u002Fguides\u002Flambda-kafka\u002Fresources",1789122886977]