[{"data":1,"prerenderedAt":128},["ShallowReactive",2],{"content-svc-step-functions-paved-path-distributed-data-processing-with-step-functions-observability":3},{"markdown":4,"frontMatterAttributes":5,"bodyRaw":28,"menu":29,"menuType":122,"isCollapseableMenu":32,"nextDocItem":123,"previousDocItem":99,"contributorPaths":124,"contentName":127,"slug":121,"readingTime":73},"\u003Cp>AWS StepFunctions send logs and metrics to Amazon CloudWatch.  Step Functions is also integrated with AWS X-Ray for tracing individual requests to identify performance bottlenecks and troubleshoot errors. Apart from standard metrics and logs, you can use the following techniques to improve the observability of your workflow.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>Callback on Success\u002FFailure\u003C\u002Fstrong>\u003C\u002Fp>\n\u003Cp>Workflows send \u003Ca href=\"https:\u002F\u002Fdocs.aws.amazon.com\u002Fstep-functions\u002Flatest\u002Fdg\u002Fcw-events.html\">status change events to Amazon EventBridge\u003C\u002Fa>. You can use this integration to apply rules and target another Step Functions state machine for failure handling, send the event to a DLQ for further analysis or send a Slack notification. \u003C\u002Fp>\n\u003Cp>\u003Cstrong>Capture Distributed Map results\u003C\u002Fstrong>\u003C\u002Fp>\n\u003Cp>\u003Ca href=\"https:\u002F\u002Fdocs.aws.amazon.com\u002Fstep-functions\u002Flatest\u002Fdg\u002Finput-output-resultwriter.html\">ResultWriter\u003C\u002Fa> is an attribute of Step Functions Distributed Map state that allows you to capture the results of the integrations and write to an Amazon S3 bucket. Result writer automatically aggregates results of map executions by status of the items and writes in an S3 bucket, one file for each status. You need to apply \u003Ca href=\"https:\u002F\u002Fdocs.aws.amazon.com\u002FAmazonS3\u002Flatest\u002Fuserguide\u002Fobject-lifecycle-mgmt.html\">life cycle policies\u003C\u002Fa> on the S3 bucket so that you do not incur cost. Results writer is a great option if you want to process the failed executions later.\u003C\u002Fp>\n\u003Cpre>\u003Ccode class=\"language-"ResultWriter":\">        "Resource": "arn:aws:states:::s3:putObject",\n        "Parameters": {\n          "Bucket": "your-results-buckets",\n          "Prefix": "results"\n        }\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>\u003Cstrong>Understand failure toleration\u003C\u002Fstrong>\u003C\u002Fp>\n\u003Cp>Failures are common in file data processing. When you have millions of records being processed, you don’t want to proceed to run the workflow when you have lots of missing or incorrect data. You can set a \u003Ca href=\"https:\u002F\u002Fdocs.aws.amazon.com\u002Fstep-functions\u002Flatest\u002Fdg\u002Fmaprun-fail-threshold.html\">failure toleration\u003C\u002Fa> percentage or count with your Distributed Map and it fails the execution when it reaches the failure percentage.\u003C\u002Fp>\n\u003Cpre>\u003Ccode>      "ToleratedFailurePercentage": 15,\n      "ToleratedFailureCount": 100\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>With the above setting, Distributed Map fails if either 100 items fail or 15% of the items fail to process.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>Respond to operational issues quicker\u003C\u002Fstrong>\u003C\u002Fp>\n\u003Cp>When you \u003Ca href=\"https:\u002F\u002Fdocs.aws.amazon.com\u002Fstep-functions\u002Flatest\u002Fdg\u002Fcw-logs.html\">enable logging\u003C\u002Fa> for your workflow, Step Functions automatically queries the logs and shows CloudWatch Log insights in the Step Functions console. \u003C\u002Fp>\n\u003Cp>\u003Cimg src=\"\u002Fassets\u002Fexternal\u002Fservice\u002Fstep-functions\u002Fpaved-path\u002Fdistributed-data-processing-with-step-functions\u002Fassets\u002Fcwinsight.png\" alt=\"\">\u003C\u002Fp>\n\u003Cp>For more information on using Distributed Map for serverless data processing, see the \u003Ca href=\"https:\u002F\u002Fcatalog.workshops.aws\u002Fstepfunctions\u002Fen-US\u002Fuse-cases\u002Fdata-processing\u002Fdistributed-map\">Large-Scale Parallelization with Distributed Map Workshop\u003C\u002Fa> or visit \u003Ca href=\"https:\u002F\u002Fserverlessland.com\u002Fsearch?search=Distributed+Map\">Serverless Land for more Distributed Map resources\u003C\u002Fa>."\u003C\u002Fp>\n",{"title":6,"order":7,"learningGoals":8,"prerequisite":10,"callout":12,"resources":16},"Observability",5,[9],"Learn different techniques and configurations to observe workflows",[11],"Understanding AWS Step Functions",{"title":13,"description":14,"link":15},"Workshop","Step by step instruction on AWS Step Functions observability","https:\u002F\u002Fcatalog.workshops.aws\u002Fstepfunctions\u002Fen-US\u002Foperations\u002Fobservability",[17,21,25],{"link":18,"text":19,"type":20},"https:\u002F\u002Fdocs.aws.amazon.com\u002Fstep-functions\u002Flatest\u002Fdg\u002Fmonitoring-logging.html","Logging and Monitoring for AWS Step Functions","text",{"link":22,"text":23,"type":24},"https:\u002F\u002Fserverlessland.com\u002Fsearch?search=Distributed+Map","Distributed Map resources on Serverless Land","sample",{"link":26,"text":27,"type":13},"https:\u002F\u002Fcatalog.us-east-1.prod.workshops.aws\u002Fworkshops\u002F2a22e604-2f2e-4d7b-85a8-33b38c999234\u002Fen-US","Distributed Map workshop","AWS StepFunctions send logs and metrics to Amazon CloudWatch.  Step Functions is also integrated with AWS X-Ray for tracing individual requests to identify performance bottlenecks and troubleshoot errors. Apart from standard metrics and logs, you can use the following techniques to improve the observability of your workflow.\n\n\n**Callback on Success\u002FFailure**\n\nWorkflows send [status change events to Amazon EventBridge](https:\u002F\u002Fdocs.aws.amazon.com\u002Fstep-functions\u002Flatest\u002Fdg\u002Fcw-events.html). You can use this integration to apply rules and target another Step Functions state machine for failure handling, send the event to a DLQ for further analysis or send a Slack notification. \n\n\n**Capture Distributed Map results**\n\n[ResultWriter](https:\u002F\u002Fdocs.aws.amazon.com\u002Fstep-functions\u002Flatest\u002Fdg\u002Finput-output-resultwriter.html) is an attribute of Step Functions Distributed Map state that allows you to capture the results of the integrations and write to an Amazon S3 bucket. Result writer automatically aggregates results of map executions by status of the items and writes in an S3 bucket, one file for each status. You need to apply [life cycle policies](https:\u002F\u002Fdocs.aws.amazon.com\u002FAmazonS3\u002Flatest\u002Fuserguide\u002Fobject-lifecycle-mgmt.html) on the S3 bucket so that you do not incur cost. Results writer is a great option if you want to process the failed executions later.\n\n```    \"ResultWriter\": {\n        \"Resource\": \"arn:aws:states:::s3:putObject\",\n        \"Parameters\": {\n          \"Bucket\": \"your-results-buckets\",\n          \"Prefix\": \"results\"\n        }\n```\n**Understand failure toleration**\n\nFailures are common in file data processing. When you have millions of records being processed, you don’t want to proceed to run the workflow when you have lots of missing or incorrect data. You can set a [failure toleration](https:\u002F\u002Fdocs.aws.amazon.com\u002Fstep-functions\u002Flatest\u002Fdg\u002Fmaprun-fail-threshold.html) percentage or count with your Distributed Map and it fails the execution when it reaches the failure percentage.\n\n```\n      \"ToleratedFailurePercentage\": 15,\n      \"ToleratedFailureCount\": 100\n``` \nWith the above setting, Distributed Map fails if either 100 items fail or 15% of the items fail to process.\n\n**Respond to operational issues quicker**\n\nWhen you [enable logging](https:\u002F\u002Fdocs.aws.amazon.com\u002Fstep-functions\u002Flatest\u002Fdg\u002Fcw-logs.html) for your workflow, Step Functions automatically queries the logs and shows CloudWatch Log insights in the Step Functions console. \n\n![](\u002Fassets\u002Fexternal\u002Fservice\u002Fstep-functions\u002Fpaved-path\u002Fdistributed-data-processing-with-step-functions\u002Fassets\u002Fcwinsight.png)\n\nFor more information on using Distributed Map for serverless data processing, see the [Large-Scale Parallelization with Distributed Map Workshop](https:\u002F\u002Fcatalog.workshops.aws\u002Fstepfunctions\u002Fen-US\u002Fuse-cases\u002Fdata-processing\u002Fdistributed-map) or visit [Serverless Land for more Distributed Map resources](https:\u002F\u002Fserverlessland.com\u002Fsearch?search=Distributed+Map).\"",[30,53,77,96,109],{"title":31,"collapsible":32,"isCollapsed":33,"content":34},"Introduction",false,true,[35],{"title":36,"order":37,"youtubeVideoID":38,"learningGoals":39,"resources":41,"time":49,"path":50,"id":51,"link":52},"Introduction to Distributed Map for Serverless Data Processing",1,"SG6_oy72hh4",[40],"Understand the benefits of distributed data processing with Serverless.",[42,46,48],{"link":43,"text":44,"type":45},"https:\u002F\u002Faws.amazon.com\u002Fstep-functions\u002F","Learn more about AWS Step Functions","Text",{"link":22,"text":23,"type":47},"Samples",{"link":26,"text":27,"type":13},"3 min","introduction","introduction.md","\u002Fcontent\u002Fservice\u002Fstep-functions\u002Fpaved-path\u002Fdistributed-data-processing-with-step-functions\u002Fintroduction",{"title":54,"collapsible":32,"isCollapsed":33,"content":55},"Getting Started",[56],{"title":57,"order":58,"youtubeVideoID":38,"learningGoals":59,"callout":61,"resources":63,"time":73,"path":74,"id":75,"link":76},"Step Functions - Distributed Map",2,[60],"Introduction to Distributed Map.",{"title":13,"description":62,"link":26},"Hands-on experience on building multiple real-world use cases of distributed map",[64,67,71],{"link":65,"text":66,"type":20},"https:\u002F\u002Fdocs.aws.amazon.com\u002Fstep-functions\u002Flatest\u002Fdg\u002Fconcepts-asl-use-map-state-distributed.html","Distributed Map concepts",{"link":68,"text":69,"type":70},"https:\u002F\u002Fcatalog.workshops.aws\u002Fstepfunctions\u002Fen-US\u002Fuse-cases\u002Fdata-processing\u002Fdistributed-map","Workshop on Distributed Map","workshop",{"link":22,"text":23,"type":72},"samples","2 min","getting-started","getting-started.md","\u002Fcontent\u002Fservice\u002Fstep-functions\u002Fpaved-path\u002Fdistributed-data-processing-with-step-functions\u002Fgetting-started",{"title":78,"collapsible":32,"isCollapsed":33,"content":79},"Patterns and Usecases",[80],{"title":81,"order":82,"callout":83,"learningGoals":84,"resources":86,"time":92,"path":93,"id":94,"link":95},"Patterns and Use cases",3,{"title":13,"description":62,"link":26},[85],"Learn different patterns and explore use cases",[87,88],{"link":26,"text":27,"type":13},{"link":89,"text":90,"type":91},"https:\u002F\u002Fwww.youtube.com\u002Fwatch?v=Fp-F8ehBUFY","Advanced serverless workflow patterns","video","5 min","patterns-usecases","patterns-usecases.md","\u002Fcontent\u002Fservice\u002Fstep-functions\u002Fpaved-path\u002Fdistributed-data-processing-with-step-functions\u002Fpatterns-usecases",{"title":97,"collapsible":32,"isCollapsed":33,"content":98},"Performance",[99],{"title":97,"order":100,"learningGoals":101,"callout":103,"time":92,"path":106,"id":107,"link":108},4,[102],"Learn different performance optimization techniques",{"title":13,"description":104,"link":105},"Learn different optimization techniques hands-on using the workshop","https:\u002F\u002Fcatalog.us-east-1.prod.workshops.aws\u002Fworkshops\u002F2a22e604-2f2e-4d7b-85a8-33b38c999234\u002Fen-US\u002Fadvanced\u002Foptimization","performance","performance.md","\u002Fcontent\u002Fservice\u002Fstep-functions\u002Fpaved-path\u002Fdistributed-data-processing-with-step-functions\u002Fperformance",{"title":6,"collapsible":32,"isCollapsed":32,"content":110},[111],{"title":6,"order":7,"learningGoals":112,"prerequisite":113,"callout":114,"resources":115,"time":73,"path":119,"id":120,"link":121},[9],[11],{"title":13,"description":14,"link":15},[116,117,118],{"link":18,"text":19,"type":20},{"link":22,"text":23,"type":24},{"link":26,"text":27,"type":13},"observability","observability.md","\u002Fcontent\u002Fservice\u002Fstep-functions\u002Fpaved-path\u002Fdistributed-data-processing-with-step-functions\u002Fobservability","LESSON",null,[125,126],"content\u002Fcontributors\u002Fuma-ramadoss.json","content\u002Fcontributors\u002Fadam-wagner.json","Distributed data processing with Step Functions",1792578874877]