Integrating Step Functions with containerized workloadsActivity Integration2 min

Activities are a Step Functions feature that enables you to process messages in an external worker running in Amazon EKS task, on-premise servers etc. As activities act as queue between the Step Functions workflow and the external system, they offer a highly scalable, resilient and asynchronous mechanism to connect your services running in Amazon ECS tasks, Amazon EKS pods or basically anywhere. You can use Step Functions activity metrics such as ActivitiesScheduled and ActivitiesStarted to scale in/scale out the workers.

To get started with activities, you create an activity in Step Functions console, API, CLI or in a workflow. You then use the activity state in the workflow to send the task to the activities. The data waits in the activity until it is consumed by the worker. Here is a sample code for worker to consume task from Step Functions activity.


client = boto3.client('stepfunctions', region_name=region, config=config)
activity = client.get_activity_task(
    activityArn = activity_arn,
    workerName = worker_name
  )

Worker receives tasktoken and the task input. Once the worker completes the processing, it sends the success or failure back to Step Functions. This mechanism works very similar to Wait for a Callback integration.

image

if success:
  client.send_task_success(
    taskToken = activity['taskToken'],
    output = stringoutput
  )
else:
  client.send_task_failure(
    taskToken = activity['taskToken'],
    cause = strCause,
    error = strError
  )

Advantages of Activity integration

  1. Activities can be used with any compute environment as long as the compute environment can assume an IAM role and call the Step Functions API.
  2. Activties are an advantage when using with Amazon ECS when you want to send more than 8KB of data to ECS tasks. Activity has a max payload size of 256KB.
  3. Activties offer additional layer of resiliency as they can act as buffers between your workflow and the worker. This is very useful when you have a low scaling worker. The task remains in the activity queue and the workflow waits until the task is consumed. If the worker dies for any reason before processing the task, the task will reappear in the activity after the elapesed time.

Created by:

Uma Ramadoss
Uma RamadossSolutions Architect @ AWS.