Performance optimization for data processing with AWS Step Functions Distributed Map

The performance of your Distributed Map can be influenced by a number of different factors. There are a some pre-requisites before getting started. You should have a non-production version of your Step Functions workflow where you can test the performance of your workflow. It's important that any components that your workflow interacts with are scaled to the same degree as a production worklow to get accurate results. You also want test input data that is similar to what you expect to see in production. If there's a noticeable variation in production workloads, either in the size of objects being process or the number of objects being processed, it is worth have multiple testing sets of data representing that range. Now that we have an environment to test in and some test data let's get started.

Within your Distributed Map you have the child workflow, this could be any combination of AWS SDK calls, Lambda functions, or other Step Functions integrations. When looking at performance this is the place to start. How long does it take to do a single iteration of your child workflow?

An easy way to do this is to use the "Limit number of items" option under Additional Configuration to limit it to a small number of items to trigger just one or two iterations. This allows you to use the same test data, but test with different numbers of items.

Child Workflow Optimization

Choosing workflow type

Once we have the time taken to do a single iteration we can start to optimize it. The first step in that process is deciding which Step Functions workflow type to use within the Distributed Map. Whenever possible, use Express workflows for the Distributed Map. Express workflows are generally less expensive and start faster than Standard workflows. It’s important to realize that Express workflows have at-least once semantics, meaning that a given execution could be executed more than once, where as Standard workflows are exactly once. Express workflows also have to finish within five minutes. So if your child workflow takes longer than five minutes to run, then go with Standard workflows.

Batching

Now that we've decided on the workflow type for the child workflow and optimized any Lambda functions we can think through the batch size. By using batching to send multiple items into each child execution you can take advantage of batch APIs, re-use connections in Lambda functions, and generally spread the cost to do various setup tasks across the processing of multiple items. You want to make sure that increasing the batch size doesn't raise the time to do a single iteration above five minutes if you're using express workflows and fifteen minutes if you're using Lambda in your standard workflows. It's worth following all the performance best practices for any Lambda functions contained in the workflow. This blog series is a great place to start: https://aws.amazon.com/blogs/compute/operating-lambda-performance-optimization-part-1/ . You also want to balance the batch size with the desire to do more in parallel.

Concurrency

The degree of parallelism is determined by the Concurrency setting in Distributed Map, which determines the maximum number of parallel child workflows you want to execute at once. A key consideration here is the service quotas of the AWS services called in your child workflow. For example AWS Lambda in most large regions has a default concurrency quota of 1000 and a default burst limit of 3000. Other services such as AWS Rekognition or AWS Textract have much lower default quotas. The other thing to keep in mind is any performance limitations of other systems that your child workflow interacts with. An example here would be an on-prem relational database that Lambda within a child workflow connects to. This database might have a limit to the number of connections it can support, so you would need to limit your concurrency accordingly. Once you identify all of the AWS Service quotas and any additional concurrency limitation you'll want to test various combinations of batch size and concurrency to find the best performance within your concurrency constraints.

Parent Workflow Optimization

Distributed Map ItemReader Optimization

When processing large numbers of objects in S3 with Distributed Map, you have a couple different options for listing those objects: s3:listObjectsV2 and S3 Inventory List. With S3:listObjectsV2, Step Functions is making S3 listObjectsV2 API calls on your behalf to retrieve all of the items needed to run the Distributed Map. Each call to listObjectsV2 can only return a maximum of 1000 S3 objects. This means that if you have 200,000 objects to process, Step Functions has to make at least 200 API calls. This API is fast and it won't take too long, but if you have an S3 Inventory file that has all the objects listed in it that you need to process, we can use that as the input. Using an S3 Inventory file as the input for a Distributed Map where you're processing large numbers of files is faster than s3:listObjectsV2. This is because for S3 Inventory ItemReaders there is a single s3:getObject call to get the manifest file and then one call for each Inventory file. If you know that your Distributed Map is going to run on a set schedule you can schedule the S3 Inventory to be created ahead of time.

      "ItemReader": {
        "Resource": "arn:aws:states:::s3:getObject",
        "ReaderConfig": {
          "InputType": "MANIFEST"
        },
        "Parameters": {
          "Bucket": "your-s3inventroy-databucket",
          "Key": "manifestfile.json"
        }