Serverless (AWS Lambda)

Metrics

Lambda metrics (Invocations, Errors, Duration, Throttles, ...) arrive through the CloudWatch Metric Stream and the Firehose endpoint described in cloud.md. Include the AWS/Lambda namespace in the stream.

Traces and logs

Use the OpenTelemetry Lambda layer for your runtime and point it at the gateway:

Environment:
  Variables:
    AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-instrument
    OTEL_EXPORTER_OTLP_ENDPOINT: https://gateway.example.com:4318
    OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf
    OTEL_EXPORTER_OTLP_HEADERS: x-watchtower-key=wti_...
    OTEL_SERVICE_NAME: checkout-fn

Keep the key in Secrets Manager or SSM and inject it at deploy time rather than writing it in the template. Logs go through a CloudWatch Logs subscription into the same Firehose stream.

Step Functions

The agent's aws_stepfunctions module turns finished executions of standard state machines into traces: one root span per execution (status, error and cause of a failed run), a span per state from its StateEntered to its StateExited event, nested under the Parallel or Map state that encloses it, and a client span per task, Lambda or activity call. Pairing follows the history's previousEventId, so concurrent branches and Map iterations nest correctly. The service name is the state machine's name. It reads ListStateMachines, ListExecutions and GetExecutionHistory (without execution input or output) with the standard AWS credential chain, so the role needs states:ListStateMachines, states:ListExecutions and states:GetExecutionHistory. It is off unless enabled: true and a region are set; calls fail (and are logged once) without credentials. Express workflows have no execution history in this API and are skipped.

modules:
  aws_stepfunctions:
    enabled: true
    region: eu-west-1
    interval: 1m              # poll interval (at least 30s); executions that stopped within two intervals are picked up
    max_executions: 50        # per state machine and poll (at most 200)
    # state_machines: ["arn:aws:states:eu-west-1:123456789012:stateMachine:orders"]   # default: all standard ones (at most 100)

Cold starts

The layer sets faas.coldstart=true on the first invocation span of an execution environment. The RED view and the service page take a cold-start filter: GET /v1/projects/{prj}/apm/services/{service}/red?cold_start=true shows only cold invocations, cold_start=false only warm ones. Omit it for both. The same settings as a JSON environment block (for example a Terraform environment map or a SAM parameter file):

{
  "AWS_LAMBDA_EXEC_WRAPPER": "/opt/otel-instrument",
  "OTEL_EXPORTER_OTLP_ENDPOINT": "https://gateway.example.com:4318",
  "OTEL_SERVICE_NAME": "checkout-fn"
}

And as a dotenv file for local runs of the function:

OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_PROPAGATORS=tracecontext,baggage
OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.5