Spark On AWS Lambda (SoAL)

Spark Runtime for AWS Lambda-SoAL framework

The Spark on AWS Lambda(SoAL) GitHub repository contains the spark runtime docker image. SoAL Framework provides a standalone or local-mode Apache Spark installation within a Docker container. Execution is facilitated by AWS Lambda working in tandem with PySpark scripts. Utilizing container-based image cache along with AWS Lambda's warm-instance features, it was noticed an overall reduction of JVM warm-up time to under 30 seconds. The SOAL Framework solution is ideal for batch and streaming workloads. Based on the benchmarking studies conducted by our team, it was observed that the framework performs well with batch payloads up to 400 MB and streaming data from Amazon MSK and Amazon Kinesis. Workload per session costs depend upon the run duration, AWS Lambda memory, and CPU usage.

< Back to all repos

GitHub icon View this repo on GitHub


Clone this repo

git clone https://github.com/aws-samples/spark-on-aws-lambda

Additional resources


Created by:

John Cherian

John Cherian

John Cherian is spark enthusiast and exploring various options to make spark enterprise framework.

Follow on LinkedIn