Invoking Amazon ECS/Amazon EKS task using Run a Job integrations

You can use Amazon ECS RunTask API to run short running tasks in containers. With Step Functions Run a Job integration, you can invoke Amazon ECS RunTask API and wait for the task to complete. Step Functions starts a new task using the specified task definition, waits for the task to complete and then calls the next step in the workflow.

Similar to ECS integration, Step Functions can interact with Kubernetes API and run jobs as part of your application workflows. Step Functions EKS integration supports only Kubernetes API server with public endpoint access.

Sample code using Amazon ECS Runtask API

          "ECS RunTask": {
            "Type": "Task",
            "Resource": "arn:aws:states:::ecs:runTask.sync",
            "Parameters": {
              "LaunchType": "FARGATE",
              "Cluster": "arn:aws:ecs:{region}:{account}:cluster/{cluster}",
              "TaskDefinition": "arn:aws:ecs:{region}:{account}:task-definition/ecsdemo-sfn:2",
              "Overrides": {
                "ContainerOverrides": [
                  {
                    "Name": "ecsdemo-sfn",
                    "Environment": [
                      {
                        "Name": "ITEMS",
                        "Value.$": "States.JsonToString($)"
                      },
                      {
                        "Name": "INPUT_BUCKET",
                        "Value": "bucket1s50db36y1g4h"
                      }
                    ]
                  }
                ]
              }
            },
            "End": true
          }

Notice the Resource attribute in the above code. If .sync is not added in the resource, the request follows Request Response integration pattern which will not wait for the task to complete.

You can pass parameters to ECS task using Environment attribute within ContainerOverrides.

Considerations

• Container overrides JSON has a max size limit of 8192 symbols. If you need to pass more data to the ECS task, you need to consider storing the payload in external storage or activity integration.
• You can not send response back to the Step Functions workflow from the ECS task/EKS job. You need to use mechanisms such as Amazon S3 or DynamoDB to store the data for later use in the Step Functions workflow or use Wait for a Callback or Activity integration.
• In order to determine if the ECS task is complete, Step Functions call the ECS:DescribeTasks API regularly. At high volume, you might observe throttles on the DescribeTasks API. To avoid such throttles, you can remove ECS:DescribeTasks IAM permission from state machine’s execution role. Step Functions will use Amazon ECS events for Amazon EventBridge to determine if the task is complete.