7. Observability

In this chapter you will learn about adding observability capabilities to your serverless applications using Terraform and IaC. Observability is a vital trait of any cloud application, as it provides insights into system behavior and interactions, aiding in issue detection and performance optimization. It is common to name Logging, Traces, and Metrics as three pillars of observability.

Another crucial aspect of observability is tagging. Tags assist in organizing resources, enable precise cost tracking, and when used with a Configuration Management Database (CMDB), help manage assets and ensure compliance. In a cloud environment, where resources and configurations are dynamic, a structured observability approach, supported by effective tagging, is essential for operational efficiency and cost management.

Lambda and CloudWatch Logs

AWS Lambda natively integrates with CloudWatch Logs, pushing all logs from your code to a CloudWatch Logs group. Make sure that the execution role you assign to your functions have permissions to create log groups and write log events to CloudWatch Logs:

resource "aws_iam_policy" "lambda_cloudwatch_policy" {
  name        = "LambdaCloudWatchPolicy"
  description = "Allows Lambda to write logs to CloudWatch."

  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Action   = ["logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents"],
        Effect   = "Allow",
        Resource = "arn:aws:logs:{region}:{account-id}:log-group:{log-group-name}" 
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "lambda_cloudwatch_attach" {
  policy_arn = aws_iam_policy.lambda_cloudwatch_policy.arn
  role       = aws_iam_role.lambda_execution_role.name
}

Once you grant the permissions, the Lambda service will automatically create and update the Log Groups and streams for you.

API Gateway and CloudWatch Logs

API Gateway supports two types of logging - execution and access. With execution logging, API Gateway will create and manage the log group for you. You will need to configure the API Gateway with an execution role that has the required permissions.

resource "aws_api_gateway_account" "api_gateway_account" {
  cloudwatch_role_arn = aws_iam_role.cloudwatch.arn
}

See aws_api_gateway_account for a full example.

In order to enable access logging for API Gateway, you need to create a CloudWatch Log Group, ensure the IAM role used by the API Gateway has permissions to publish logs, and enable the access logs for the API Gateway:

# Create a CloudWatch Logs Group
resource "aws_cloudwatch_log_group" "api_gateway_log_group" {
  name = "/aws/api-gateway/greeting_api"
}

# Create an access policy for API Gateway to send execution logs to CloudWatch Logs Group
resource "aws_iam_role_policy" "api_gateway_cloudwatch_logs" {
  name   = "ApiGatewayCloudWatchLogs"
  role   = aws_iam_role.api_gateway_execution_role.name
  
  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Action   = ["logs:PutLogEvents", "logs:CreateLogStream"],
        Effect   = "Allow",
        Resource = aws_cloudwatch_log_group.api_gateway_log_group.arn
      }
    ]
  })
}

# Enable access logs in your API Gateway
resource "aws_api_gateway_stage" "api_gateway_stage" {
  # ... (other configurations)

  access_log_settings {
    destination_arn = aws_cloudwatch_log_group.api_gateway_log_group.arn
    format          = "[$context.requestId] ($context.identity.sourceIp) \"$context.httpMethod $context.routeKey $context.protocol\" $context.status $context.responseLength $context.requestTime"
  }
}

Distributed Tracing with AWS X-Ray

AWS X-Ray offers a tracing perspective into the performance of your applications. It pinpoints how well your applications are operating and helps to identify bottlenecks. When following a request journey from Client --> API Gateway --> SQS --> Lambda --> S3, it's crucial to have such visibility. Integrating X-Ray with managed services like API Gateway and Lambda is straightforward.

Enable distributing tracing with X-Ray for your Lambda functions by adding the tracing_config element to the function resource definition:

resource "aws_lambda_function" "greeting_lambda" {
  # ... (other configurations)

  tracing_config {
    mode = "Active"
  }
}

Enable distributing tracing with X-Ray for your API Gateway by adding the xray_tracing_enabled element to the stage resource definition:

resource "aws_api_gateway_stage" "api_gateway_stage" {
  # ... (other configurations)
  
  xray_tracing_enabled = true
}

Make sure that the execution roles have permissions for publishing telemetry to X-Ray, e.g.

resource "aws_iam_policy" "lambda_xray_policy" {
  name        = "LambdaXRayPolicy"
  description = "Allows Lambda to send trace data to X-Ray."

  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Action   = ["xray:PutTraceSegments", "xray:PutTelemetryRecords"],
        Effect   = "Allow",
        Resource = "*"
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "lambda_xray_attach" {
  policy_arn = aws_iam_policy.lambda_xray_policy.arn
  role       = aws_iam_role.lambda_execution_role.name
}

Tagging Resources

Tagging resources in straightforward with Terraform - you simply attach your desired tags to the relevant resources. The advantage is that you can define a Terraform variable at the start of your main.tf file and then reference these values throughout your configurations for consistent tagging. This approach ensures uniformity while keeping your setup clean and organized.

resource "aws_lambda_function" "greeting_lambda" {
  # ... (other configurations)

  tags = {
    Environment = "Production"
    OwnerTeam   = "Engineering"
  }
}

resource "aws_api_gateway_rest_api" "greeting_api" {
  # ... (other configurations)

  tags = {
    Environment = "Production"
    OwnerTeam   = "Platform"
  }
}

Recap

In this chapter you've learned how to:

Set up logging for both Lambda and API Gateway, sending logs to CloudWatch for easy monitoring and debugging.
Integrate AWS X-Ray for both Lambda and API Gateway for in-depth insights into the application's execution and performance.
Institute a tagging strategy across your application resources for streamlined resource management and cost allocation.

Observability is a linchpin for successful and manageable modern infrastructures. Having insights into how components of your applications interact and perform under various conditions is invaluable in timely issue resolution, as well as cost and performance management.

Next Steps

In the previous chapters you've learned how to construct a scalable and comprehensive architecture, however all of the IaC configuration resided within a monolithic main.tf file. In following chapters you will learn how to refactor the monolithic main.tf into a collection of modular, reusable, and configurable Terraform modules.