Observability

AWS Lambda is a serverless compute platform that automatically scales horizontally as the traffic to your function increases. The environments created by Lambda to run your code are secured to allow access only via the invoke API method. This means that if you previously accessed information about your application via an HTTP endpoint, it will not be possible with Lambda. This changes the tooling that we can use to implement observability practices.

Lambda also doesn't allow you to run processes outside of the APIs and controls provided. You can't run a third-party process in the background like you might on a VM or a container. Instead, Lambda provides APIs to access function telemetry and logging data. Third parties such as Datadog, Hashicorp, and many others have written integrations for these APIs so that you don't have to.

Logging

Lambda functions run in execution environments that are ephemeral and can be terminated between invokes. Your application logging should know this and process your logging in a way that means that you are not losing logs if an environment is terminated.

Applications that have previously logged to file systems and then had a separate process to send these logs to a central place won't work in Lambda. Instead, your Lambda function should follow output logs to stdout and the processing and routing of those logs should be handled by Lambda and the resources you've configured.

For example, if your application is using Log4J or Logback with a file appender you should change this instead log to stdout

An example Log4J file appender configuration

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="warn" name="MyApp" packages="">
    <Appenders>
        <File name="MyFile" fileName="logs/app.log">
            <PatternLayout>
                <Pattern>%d %p %c{1.} [%t] %m%n</Pattern>
            </PatternLayout>
        </File>
        <Async name="Async">
            <AppenderRef ref="MyFile"/>
        </Async>
    </Appenders>
    <Loggers>
        <Root level="error">
            <AppenderRef ref="Async"/>
        </Root>
    </Loggers>
</Configuration>

An example Log4J stdout appender configuration

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="warn" name="MyApp" packages="">
    <Appenders>
        <Console name="STDOUT" target="SYSTEM_OUT">
            <PatternLayout>
                <Pattern>%d %p %c{1.} [%t] %m%n</Pattern>
            </PatternLayout>
        </Console>
    </Appenders>
    <Loggers>
        <Root level="error">
            <AppenderRef ref="STDOUT"/>
        </Root>
    </Loggers>
</Configuration>

Centralized Logging

Cloud Native logging with CloudWatch Logs

By default all messages that are written to stdout are saved to an Amazon CloudWatch log stream. Each Lambda function has its own log stream. CloudWatch is billed by the amount of data ingested, and the amount of storage used. The less logging your application produces the less you will be charged. By default log streams are kept indefinitely, this can be configured to a specific duration based on your requirements.

Logging to a third party

If you have a preferred logging solution that you would like to continue to use, you can most likely use an existing integration. Popular service providers such as Splunk, Datadog, Dynatrace, New Relic, Honeycomb, and many others have created integrations for you to use to make your Lambda logs available in their solutions.

Lambda Partners use the Lambda Telemetry API to take the logs produced to stdout by your function and direct them to their own service.

General guidance

You should use CloudWatch Logs if you don't have an existing preferred logging solution. If this doesn't meet all of your requirements, and the solution you've selected has an existing integration, use that. If the solution doesn't have an existing integration, then you will have to create your own. Your integration will have to work within the constraints of the Lambda execution environment.

Logging to multiple files

In the previous sections, you learned about how the Lambda execution model restricts how logging can be implemented. If your application writes separate information to different log files, for example a log with only error messages and a log with all messages. Then your application must change in order to adapt to the Lambda environment.

General guidance

CloudWatch has powerful search functionality called CloudWatch Logs Insights that you can use to filter for specific log messages. Where possible use a single log stream for each Lambda function and search for the data required. This will reduce the overall complexity of your Lambda function.

If you need to have a separate stream for some application data, then you can use the CloudWatch SDK to write specifically to that log stream using the PutLogEvents API call. You can find a Java example in the aws-doc-sdk-examples project on GitHub.