[{"data":1,"prerenderedAt":155},["ShallowReactive",2],{"content-svc-lambda-guides-serverless-stream-processing-with-lambda-msk-scaling-consumers":3},{"markdown":4,"frontMatterAttributes":5,"bodyRaw":12,"menu":13,"menuType":148,"isCollapseableMenu":16,"nextDocItem":84,"previousDocItem":72,"contributorPaths":149,"contentName":154,"slug":83,"readingTime":68},"\u003Cp>When you initially create an Amazon MSK event source, Lambda allocates one consumer or poller to process all partitions in the Kafka topic. Additionally, Lambda automatically scales up or down the number of consumers, based on message rates. To preserve message ordering in each partition, the maximum number of consumers is one consumer per topic partition.\u003C\u002Fp>\n\u003Cp>In one-minute intervals, Lambda evaluates the consumer offset lag of all the partitions in the topic. If the lag is too high, the partition is receiving messages faster than Lambda can process them. If necessary, Lambda adds or removes consumers from the topic. The scaling process of adding or removing consumers occurs within three minutes of evaluation.\u003C\u002Fp>\n\u003Cp>If your target Lambda function is throttled, Lambda reduces the number of consumers. This action reduces the workload on the function by reducing the number of messages that consumers can retrieve and send to the function. The processing of messages can also be scaled further by selective invocation of the target Lambda function. This can be achieved by using \u003Ca href=\"https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Finvocation-eventfiltering.html\">filter\u003C\u002Fa> expressions.\u003C\u002Fp>\n\u003Cp>For a deep dive learning journey follow the Kafka Lambda learning \u003Ca href=\"https:\u002F\u002Fserverlessland.com\u002Fcontent\u002Fguides\u002Flambda-kafka\u002Fintroduction\">guide\u003C\u002Fa>.\u003C\u002Fp>\n",{"title":6,"order":7,"callout":8},"Scaling of consumers",54,{"title":9,"description":10,"link":11},"Follow the Kafka with Lambda learning guide","Refer to the deep dive learning guide for using Kafka with Lambda","https:\u002F\u002Fserverlessland.com\u002Fcontent\u002Fguides\u002Flambda-kafka\u002Fintroduction","When you initially create an Amazon MSK event source, Lambda allocates one consumer or poller to process all partitions in the Kafka topic. Additionally, Lambda automatically scales up or down the number of consumers, based on message rates. To preserve message ordering in each partition, the maximum number of consumers is one consumer per topic partition.\n\nIn one-minute intervals, Lambda evaluates the consumer offset lag of all the partitions in the topic. If the lag is too high, the partition is receiving messages faster than Lambda can process them. If necessary, Lambda adds or removes consumers from the topic. The scaling process of adding or removing consumers occurs within three minutes of evaluation.\n\nIf your target Lambda function is throttled, Lambda reduces the number of consumers. This action reduces the workload on the function by reducing the number of messages that consumers can retrieve and send to the function. The processing of messages can also be scaled further by selective invocation of the target Lambda function. This can be achieved by using [filter](https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Finvocation-eventfiltering.html) expressions.\n\nFor a deep dive learning journey follow the Kafka Lambda learning [guide](https:\u002F\u002Fserverlessland.com\u002Fcontent\u002Fguides\u002Flambda-kafka\u002Fintroduction).\n",[14,45,97],{"title":15,"collapsible":16,"isCollapsed":17,"content":18},"Introduction",false,true,[19,25,35],{"title":15,"order":20,"time":21,"path":22,"id":23,"link":24},1,"2 min","1-introduction","1-introduction.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F1-introduction",{"title":26,"order":27,"callout":28,"time":21,"path":32,"id":33,"link":34},"Getting started with streaming data",2,{"title":29,"description":30,"link":31},"Watch the video","This video will help you get started with Amazon Kinesis Data Streams, a massively scalable, durable, and low-cost streaming data service that enables you to continuously capture gigabytes of data per second from hundreds of thousands of sources, such as website clickstreams, database event streams, financial transactions, social media feeds, IT logs, and location-tracking events.","https:\u002F\u002Fwww.youtube.com\u002Fwatch?v=1I1DcJvmd4w","2-getting-started","2-getting-started.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F2-getting-started",{"title":36,"order":37,"callout":38,"time":21,"path":42,"id":43,"link":44},"Event source mapping for Lambda",3,{"title":39,"description":40,"link":41},"Read the docs","Refer to the AWS Lambda documentation for detailed explanations of Lambda event source mappings","https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Finvocation-eventsourcemapping.html","3-event-source-mapping","3-event-source-mapping.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F3-event-source-mapping",{"title":46,"collapsible":16,"isCollapsed":16,"content":47},"Streaming processing with MSK",[48,64,72,79,84,90],{"title":49,"order":50,"youtubeVideoID":51,"resources":52,"callout":60,"time":21,"path":61,"id":62,"link":63},"MSK as event source for Lambda",51,"JKvIypfEiok",[53,57],{"link":54,"type":55,"text":56},"https:\u002F\u002Faws.amazon.com\u002Flambda\u002F","Text","AWS Lambda",{"link":58,"type":55,"text":59},"https:\u002F\u002Faws.amazon.com\u002Fserverless\u002F","Serverless on AWS",{"title":9,"description":10,"link":11},"msk-stream-processing","msk-stream-processing.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-stream-processing",{"title":65,"order":66,"callout":67,"time":68,"path":69,"id":70,"link":71},"Deep dive into Lambda ESM for MSK",50,{"title":9,"description":10,"link":11},"1 min","msk-esm-deep-dive","msk-esm-deep-dive.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-esm-deep-dive",{"title":73,"order":74,"callout":75,"time":21,"path":76,"id":77,"link":78},"Ordered processing of messages",52,{"title":9,"description":10,"link":11},"msk-ordered-processing","msk-ordered-processing.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-ordered-processing",{"title":6,"order":7,"callout":80,"time":68,"path":81,"id":82,"link":83},{"title":9,"description":10,"link":11},"msk-scaling-consumers","msk-scaling-consumers.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-scaling-consumers",{"title":85,"order":86,"time":68,"path":87,"id":88,"link":89},"Authentication options for Lambda consumers",56,"msk-lambda-auth-options","msk-lambda-auth-options.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-lambda-auth-options",{"title":91,"order":92,"callout":93,"time":68,"path":94,"id":95,"link":96},"Error handling in Lambda functions",58,{"title":9,"description":10,"link":11},"msk-lambda-error-handling","msk-lambda-error-handling.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002Fmsk-lambda-error-handling",{"title":98,"collapsible":16,"isCollapsed":17,"content":99},"Streaming processing with KDS",[100,109,121,132,140],{"title":101,"weight":102,"learningGoals":103,"time":105,"path":106,"id":107,"link":108},"Kinesis Data Streams (KDS) - Key Concepts and Terminologies",60,[104],"Learn key concepts and terminologies of Amazon KDS","4 min","4-1-KDS-key-concepts-terms","4-1-KDS-key-concepts-terms.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-1-KDS-key-concepts-terms",{"title":110,"weight":111,"learningGoals":112,"resources":114,"time":117,"path":118,"id":119,"link":120},"Amazon Kinesis Data Streams - Integration with AWS Lambda",62,[113],"Amazon KDS integration with AWS Lambda for Serverless stream processing",[115,116],"https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Finvocation-eventsourcemapping.html#invocation-eventsourcemapping-batching","https:\u002F\u002Faws.amazon.com\u002Fblogs\u002Fcompute\u002Fnew-aws-lambda-scaling-controls-for-kinesis-and-dynamodb-event-sources\u002F","5 min","4-2-KDS-lambda-intg","4-2-KDS-lambda-intg.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-2-KDS-lambda-intg",{"title":122,"weight":123,"learningGoals":124,"resources":126,"time":128,"path":129,"id":130,"link":131},"Amazon KDS - Error handling with Lambda ESM",64,[125],"How to handle errors when configuring Amazon KDS as Lambda ESM",[127],"https:\u002F\u002Fdocs.aws.amazon.com\u002Flambda\u002Flatest\u002Fdg\u002Fwith-kinesis.html","3 min","4-3-KDS-lambda-error-handling","4-3-KDS-lambda-error-handling.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-3-KDS-lambda-error-handling",{"title":133,"weight":134,"learningGoals":135,"time":117,"path":137,"id":138,"link":139},"Kinesis Data Streams Scaling- Key Concepts and Terminologies",66,[136],"Learn key concepts and terminologies of Amazon KDS Scaling","4-4-KDS-scaling","4-4-KDS-scaling.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-4-KDS-scaling",{"title":141,"weight":142,"learningGoals":143,"time":117,"path":145,"id":146,"link":147},"Kinesis Data Streams Monitoring",68,[144],"Learn Amazon KDS & Lambda monitoring methodologies","4-5-KDS-monitoring","4-5-KDS-monitoring.md","\u002Fcontent\u002Fservice\u002Flambda\u002Fguides\u002Fserverless-stream-processing-with-lambda\u002F4-5-KDS-monitoring","LIST",[150,151,152,153],"content\u002Fcontributors\u002Fmithun-mallick.json","content\u002Fcontributors\u002Fkevin-schwarz.json","content\u002Fcontributors\u002Fnaren-sithambaram.json","content\u002Fcontributors\u002Fprasanna-s-krishnan.json","Serverless Stream Processing with Lambda",1792578874662]