Select your cookie preferences

We use cookies and similar tools to enhance your experience, provide our services, deliver relevant advertising, and make improvements. Approved third parties also use these tools to help us deliver advertising and provide certain site features.

S3 to SQS to Lambda

Created with SnapAmazon S3SQSAWS Lambda

Create a Lambda function that resizes images uploaded to S3 via SQS.

The SAM template deploys a Lambda function, an SQS queue, 2 S3 buckets and the IAM resources required to run the application.
An SQS queue consumes ObjectCreated events from an Amazon S3 bucket if the file has .jpg extension. The SQS triggers a Lambda function.
The Lambda code checks the uploaded file is an image and creates a thumbnail version of the image in another bucket.

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Image resizing service through SQS (uksb-1tthgi812) (tag:s3-sqs-lambda)

Parameters:
  SourceBucketName:
    Type: String
  QueueName:
    Type: String
  DestinationBucketName:
    Type: String

Resources:
  ## S3 bucket
  SourceBucket:
    Type: AWS::S3::Bucket
    DependsOn:
      - ResizerQueueQueuePolicy
    Properties:
      BucketName: !Ref SourceBucketName
      NotificationConfiguration:
        QueueConfigurations:
          - Event: "s3:ObjectCreated:*"
            Queue: !GetAtt ResizerQueue.Arn
            Filter:
              S3Key:
                Rules:
                  - Name: suffix
                    Value: '.jpg'

  DestinationBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Ref DestinationBucketName

  ## SQS Queue
  ResizerQueue:
    Type: "AWS::SQS::Queue"
    Properties:
      QueueName: !Ref QueueName

  ## Policies
  ResizerQueueQueuePolicy:
    Type: "AWS::SQS::QueuePolicy"
    Properties:
      PolicyDocument:
        Version: "2012-10-17"
        Id: QueuePolicy
        Statement:
          - Sid: Allow-SendMessage-To-Queue-From-S3-Event-Notification
            Effect: Allow
            Principal: 
              Service: "s3.amazonaws.com"
            Action:
              - "sqs:SendMessage"
            Resource: !GetAtt ResizerQueue.Arn
            Condition:
              ArnLike:
                aws:SourceArn: !Join ["",['arn:aws:s3:::',!Ref SourceBucketName]]
              StringEquals:
                aws:SourceAccount: !Ref AWS::AccountId
      Queues:
        - Ref: ResizerQueue

  ## Lambda function
  ResizerFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: src/
      Handler: app.handler
      Runtime: nodejs20.x
      MemorySize: 2048
      Layers:
        - !Sub 'arn:aws:lambda:${AWS::Region}:175033217214:layer:graphicsmagick:2'
      Policies:
        - S3ReadPolicy:
            BucketName: !Ref SourceBucketName
        - S3CrudPolicy:
            BucketName: !Ref DestinationBucketName
      Environment:
        Variables:
          DESTINATION_BUCKETNAME: !Ref DestinationBucketName
      Events:
        MySQSEvent:
          Type: SQS
          Properties:
            Queue: !GetAtt ResizerQueue.Arn

Outputs:
  SourceBucketName:
    Value: !Ref SourceBucketName
    Description: S3 Bucket for object storage
  DestinationBucketName:
    Value: !Ref DestinationBucketName
    Description: S3 destination Bucket for object storage
  QueueName:
    Value: !Ref QueueName
    Description: SQS Queue for queuing the s3 events
  FunctionArn:
    Value: !Ref ResizerFunction
    Description: ResizerFunction function  Arn

< Back to all patterns


GitHub icon Visit the GitHub repo for this pattern.

Launch Stack

Download

git clone https://github.com/aws-samples/serverless-patterns/ cd serverless-patterns/s3-sqs-lambda

Deploy

npm --prefix ./src install ./srcsam buildsam deploy --guided


Testing

1. Run the following S3 CLI command to upload an image to the S3 bucket. Note, you must edit the {SourceBucketName} placeholder with the name of the source S3 Bucket. This is provided in the stack outputs.
aws s3 cp './events/exampleImage.png' s3://{SourceBucketName}
2. Run the following command to check that a new version of the image has been created in the destination bucket.
aws s3 ls s3://{DestinationBucketName}

Cleanup

1. Delete the stack: sam delete --stack-name STACK_NAME.
2. Confirm the stack has been deleted: aws cloudformation list-stacks --query "StackSummaries[?contains(StackName,'STACK_NAME')].StackStatus"

Ahmed Mohamed

Presented by Ahmed Mohamed

I am a Principal Software Engineer, Solutions Architect and Serverless Enthusiast. I spend my time learning new stuff and building large scale clinical solutions in the cloud.