[{"data":1,"prerenderedAt":126},["ShallowReactive",2],{"content-svc-step-functions-paved-path-distributed-data-processing-with-step-functions-getting-started":3},{"markdown":4,"frontMatterAttributes":5,"bodyRaw":28,"menu":29,"menuType":121,"isCollapseableMenu":32,"nextDocItem":70,"previousDocItem":35,"contributorPaths":122,"contentName":125,"slug":66,"readingTime":49},"\u003Cp>\u003Cstrong>Getting started with distributed data processing with Step Functions\u003C\u002Fstrong>\u003C\u002Fp>\n\u003Cp>Distributed Map is a task state of Step Functions that runs the same process for multiple entries in a data set at a maximum concurrency of 10,000. The process is a child workflow that can be any combination of AWS SDK calls, an AWS Lambda function(s), or other AWS Step Functions integrations. With Distributed Map, You can iterate over millions of objects such as logs, images, or .csv files stored in Amazon Simple Storage Service (Amazon S3).\u003C\u002Fp>\n\u003Cp> Distributed Map is a \u003Ca href=\"https:\u002F\u002Fdocs.aws.amazon.com\u002Fstep-functions\u002Flatest\u002Fdg\u002Famazon-states-language-map-state.html\">Map state\u003C\u002Fa>. Using \u003Ca href=\"https:\u002F\u002Fdocs.aws.amazon.com\u002Fstep-functions\u002Flatest\u002Fdg\u002Fconcepts-amazon-states-language.html\">Amazon States Language (ASL)\u003C\u002Fa>, you define the mode as DISTRIBUTED to tell Step Functions to run the map state in distributed mode. You can choose to run the child workflow as \u003Ccode>STANDARD\u003C\u002Fcode> or \u003Ccode>EXPRESS\u003C\u002Fcode> using \u003Ccode>ExecutionType\u003C\u002Fcode>.\u003C\u002Fp>\n\u003Cpre>\u003Ccode class=\"language-json\">    "Process": {\n      "Type": "Map",\n      "ItemProcessor": {\n        "ProcessorConfig": {\n          "Mode": "DISTRIBUTED",\n          "ExecutionType": "STANDARD"\n          \n        },...\n      }\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>After defining the Distributed Map state as above, you configure where to read the data from. You provide a bucket and key if you are processing data from S3. \u003C\u002Fp>\n\u003Cpre>\u003Ccode class=\"language-json\">      "ItemReader": {\n        "Resource": "arn:aws:states:::s3:getObject",\n        "Parameters": {\n          "Bucket.$": "$.bucket",\n          "Key.$": "$.key"\n        }\n      }\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>You provide bucket and\u002For prefix, if you are processing list of files from S3. \u003C\u002Fp>\n\u003Cpre>\u003Ccode class=\"language-json\">      "ItemReader": {\n        "Resource": "arn:aws:states:::s3:listObjectsV2",\n        "Parameters": {\n          "Bucket": "$.bucket"\n        }\n      }\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>Additionally, You can process data from \u003Ccode>S3 Inventory list\u003C\u002Fcode> and \u003Ccode>JSON array\u003C\u002Fcode> of data from previous step or workflow input. \nYou control how many child workflows are run concurrently using the \u003Ccode>MaxConcurrency\u003C\u002Fcode> attribute.\u003C\u002Fp>\n\u003Cpre>\u003Ccode>      "MaxConcurrency": 1000\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>You can configure the size of the data sent to each child workflow using the \u003Ccode>ItemBatcher\u003C\u002Fcode> attribute. It can be a count of items, size in KB, or both. \u003C\u002Fp>\n\u003Cpre>\u003Ccode class=\"language-json\">      "ItemBatcher": {\n        "MaxItemsPerBatch": 500\n      }\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>Data quality is a big challenge with data processing. With Distributed Map, you can stop the workflow from continuing to process bad data if certain thresholds are reached. This saves cost and time spent processing bad quality data.\u003C\u002Fp>\n\u003Cpre>\u003Ccode class=\"language-json\">      "ToleratedFailurePercentage": 10,\n      "ToleratedFailureCount": 100\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>Once the processing is done by iterating over all the objects, you can choose to export the results to an S3 bucket. Distributed Map aggregates the results of all successful child workflows in one JSON file and all failed child workflows in another JSON file.\u003C\u002Fp>\n\u003Cpre>\u003Ccode class=\"language-json\">    "ResultWriter": {\n      "Resource": "arn:aws:states:::s3:putObject",\n      "Parameters": {\n        "Bucket": "processedOutput",\n        "Prefix": "test-run"\n      }\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>To learn more about the concepts, follow the link \u003Ca href=\"https:\u002F\u002Fdocs.aws.amazon.com\u002Fstep-functions\u002Flatest\u002Fdg\u002Fconcepts-asl-use-map-state-distributed.html\">here\u003C\u002Fa>.\u003C\u002Fp>\n",{"title":6,"order":7,"youtubeVideoID":8,"learningGoals":9,"callout":11,"resources":15},"Step Functions - Distributed Map",2,"SG6_oy72hh4",[10],"Introduction to Distributed Map.",{"title":12,"description":13,"link":14},"Workshop","Hands-on experience on building multiple real-world use cases of distributed map","https:\u002F\u002Fcatalog.us-east-1.prod.workshops.aws\u002Fworkshops\u002F2a22e604-2f2e-4d7b-85a8-33b38c999234\u002Fen-US",[16,20,24],{"link":17,"text":18,"type":19},"https:\u002F\u002Fdocs.aws.amazon.com\u002Fstep-functions\u002Flatest\u002Fdg\u002Fconcepts-asl-use-map-state-distributed.html","Distributed Map concepts","text",{"link":21,"text":22,"type":23},"https:\u002F\u002Fcatalog.workshops.aws\u002Fstepfunctions\u002Fen-US\u002Fuse-cases\u002Fdata-processing\u002Fdistributed-map","Workshop on Distributed Map","workshop",{"link":25,"text":26,"type":27},"https:\u002F\u002Fserverlessland.com\u002Fsearch?search=Distributed+Map","Distributed Map resources on Serverless Land","samples","**Getting started with distributed data processing with Step Functions**\n\nDistributed Map is a task state of Step Functions that runs the same process for multiple entries in a data set at a maximum concurrency of 10,000. The process is a child workflow that can be any combination of AWS SDK calls, an AWS Lambda function(s), or other AWS Step Functions integrations. With Distributed Map, You can iterate over millions of objects such as logs, images, or .csv files stored in Amazon Simple Storage Service (Amazon S3).\n\n Distributed Map is a [Map state](https:\u002F\u002Fdocs.aws.amazon.com\u002Fstep-functions\u002Flatest\u002Fdg\u002Famazon-states-language-map-state.html). Using [Amazon States Language (ASL)](https:\u002F\u002Fdocs.aws.amazon.com\u002Fstep-functions\u002Flatest\u002Fdg\u002Fconcepts-amazon-states-language.html), you define the mode as DISTRIBUTED to tell Step Functions to run the map state in distributed mode. You can choose to run the child workflow as `STANDARD` or `EXPRESS` using `ExecutionType`.\n\n```json\n    \"Process\": {\n      \"Type\": \"Map\",\n      \"ItemProcessor\": {\n        \"ProcessorConfig\": {\n          \"Mode\": \"DISTRIBUTED\",\n          \"ExecutionType\": \"STANDARD\"\n          \n        },...\n      }\n```\nAfter defining the Distributed Map state as above, you configure where to read the data from. You provide a bucket and key if you are processing data from S3. \n\n```json\n      \"ItemReader\": {\n        \"Resource\": \"arn:aws:states:::s3:getObject\",\n        \"Parameters\": {\n          \"Bucket.$\": \"$.bucket\",\n          \"Key.$\": \"$.key\"\n        }\n      }\n```\n\nYou provide bucket and\u002For prefix, if you are processing list of files from S3. \n\n```json\n      \"ItemReader\": {\n        \"Resource\": \"arn:aws:states:::s3:listObjectsV2\",\n        \"Parameters\": {\n          \"Bucket\": \"$.bucket\"\n        }\n      }\n```\n\nAdditionally, You can process data from `S3 Inventory list` and `JSON array` of data from previous step or workflow input. \nYou control how many child workflows are run concurrently using the `MaxConcurrency` attribute.\n\n```\n      \"MaxConcurrency\": 1000\n```\n\nYou can configure the size of the data sent to each child workflow using the `ItemBatcher` attribute. It can be a count of items, size in KB, or both. \n\n```json\n      \"ItemBatcher\": {\n        \"MaxItemsPerBatch\": 500\n      }\n```\n\nData quality is a big challenge with data processing. With Distributed Map, you can stop the workflow from continuing to process bad data if certain thresholds are reached. This saves cost and time spent processing bad quality data.\n\n```json\n      \"ToleratedFailurePercentage\": 10,\n      \"ToleratedFailureCount\": 100\n```   \n\nOnce the processing is done by iterating over all the objects, you can choose to export the results to an S3 bucket. Distributed Map aggregates the results of all successful child workflows in one JSON file and all failed child workflows in another JSON file.\n\n```json\n    \"ResultWriter\": {\n      \"Resource\": \"arn:aws:states:::s3:putObject\",\n      \"Parameters\": {\n        \"Bucket\": \"processedOutput\",\n        \"Prefix\": \"test-run\"\n      }\n\n```\nTo learn more about the concepts, follow the link [here](https:\u002F\u002Fdocs.aws.amazon.com\u002Fstep-functions\u002Flatest\u002Fdg\u002Fconcepts-asl-use-map-state-distributed.html).\n",[30,53,67,86,99],{"title":31,"collapsible":32,"isCollapsed":33,"content":34},"Introduction",false,true,[35],{"title":36,"order":37,"youtubeVideoID":8,"learningGoals":38,"resources":40,"time":49,"path":50,"id":51,"link":52},"Introduction to Distributed Map for Serverless Data Processing",1,[39],"Understand the benefits of distributed data processing with Serverless.",[41,45,47],{"link":42,"text":43,"type":44},"https:\u002F\u002Faws.amazon.com\u002Fstep-functions\u002F","Learn more about AWS Step Functions","Text",{"link":25,"text":26,"type":46},"Samples",{"link":14,"text":48,"type":12},"Distributed Map workshop","3 min","introduction","introduction.md","\u002Fcontent\u002Fservice\u002Fstep-functions\u002Fpaved-path\u002Fdistributed-data-processing-with-step-functions\u002Fintroduction",{"title":54,"collapsible":32,"isCollapsed":32,"content":55},"Getting Started",[56],{"title":6,"order":7,"youtubeVideoID":8,"learningGoals":57,"callout":58,"resources":59,"time":63,"path":64,"id":65,"link":66},[10],{"title":12,"description":13,"link":14},[60,61,62],{"link":17,"text":18,"type":19},{"link":21,"text":22,"type":23},{"link":25,"text":26,"type":27},"2 min","getting-started","getting-started.md","\u002Fcontent\u002Fservice\u002Fstep-functions\u002Fpaved-path\u002Fdistributed-data-processing-with-step-functions\u002Fgetting-started",{"title":68,"collapsible":32,"isCollapsed":33,"content":69},"Patterns and Usecases",[70],{"title":71,"order":72,"callout":73,"learningGoals":74,"resources":76,"time":82,"path":83,"id":84,"link":85},"Patterns and Use cases",3,{"title":12,"description":13,"link":14},[75],"Learn different patterns and explore use cases",[77,78],{"link":14,"text":48,"type":12},{"link":79,"text":80,"type":81},"https:\u002F\u002Fwww.youtube.com\u002Fwatch?v=Fp-F8ehBUFY","Advanced serverless workflow patterns","video","5 min","patterns-usecases","patterns-usecases.md","\u002Fcontent\u002Fservice\u002Fstep-functions\u002Fpaved-path\u002Fdistributed-data-processing-with-step-functions\u002Fpatterns-usecases",{"title":87,"collapsible":32,"isCollapsed":33,"content":88},"Performance",[89],{"title":87,"order":90,"learningGoals":91,"callout":93,"time":82,"path":96,"id":97,"link":98},4,[92],"Learn different performance optimization techniques",{"title":12,"description":94,"link":95},"Learn different optimization techniques hands-on using the workshop","https:\u002F\u002Fcatalog.us-east-1.prod.workshops.aws\u002Fworkshops\u002F2a22e604-2f2e-4d7b-85a8-33b38c999234\u002Fen-US\u002Fadvanced\u002Foptimization","performance","performance.md","\u002Fcontent\u002Fservice\u002Fstep-functions\u002Fpaved-path\u002Fdistributed-data-processing-with-step-functions\u002Fperformance",{"title":100,"collapsible":32,"isCollapsed":33,"content":101},"Observability",[102],{"title":100,"order":103,"learningGoals":104,"prerequisite":106,"callout":108,"resources":111,"time":63,"path":118,"id":119,"link":120},5,[105],"Learn different techniques and configurations to observe workflows",[107],"Understanding AWS Step Functions",{"title":12,"description":109,"link":110},"Step by step instruction on AWS Step Functions observability","https:\u002F\u002Fcatalog.workshops.aws\u002Fstepfunctions\u002Fen-US\u002Foperations\u002Fobservability",[112,115,117],{"link":113,"text":114,"type":19},"https:\u002F\u002Fdocs.aws.amazon.com\u002Fstep-functions\u002Flatest\u002Fdg\u002Fmonitoring-logging.html","Logging and Monitoring for AWS Step Functions",{"link":25,"text":26,"type":116},"sample",{"link":14,"text":48,"type":12},"observability","observability.md","\u002Fcontent\u002Fservice\u002Fstep-functions\u002Fpaved-path\u002Fdistributed-data-processing-with-step-functions\u002Fobservability","LESSON",[123,124],"content\u002Fcontributors\u002Fuma-ramadoss.json","content\u002Fcontributors\u002Fadam-wagner.json","Distributed data processing with Step Functions",1789209314407]