Serverless mode

[obenkenobi]

Overview

Introduces serverless mode for the agent as according to the agent spec.

Enabling Serverless Node

You should only enable serverless mode in an AWS Lambda environment.

To enable serverless mode either NEW_RELIC_SERVERLESS_MODE_ENABLED is set to true or your AWS lambda environment already provides a set AWS_LAMBDA_FUNCTION_NAME variable.

APM mode detection

The lambda detects if APM mode is available if NEW_RELIC_APM_LAMBDA_MODE is set to true

Transaction Naming

Transaction naming is handled in our 2 instrumentation modules: aws-lambda-java-core-1.2.0 and aws-lambda-java-core-events-1.2.0. These target the aws-lambda-java-core library if they are using the RequestHandler interface.

Our aws-lambda-java-core-1.2.0 instrumentation module by default set the function name to be in the format:

${WebTransaction or OtherTransaction}/Function/${functionName}

Our aws-lambda-java-core-events-1.2.0 also sets the function name to be in the above format. However if APM mode is enabled, the format changes to:

${WebTransaction or OtherTransaction}/Function/${eventSourceType} ${functionName}

Web Transactions

If your event source is of the following:

• API Gateway Proxy Request
• API Gateway V2 Request
• ALB Group Request

The lambda will be treated as a web transaction.

Event Source Parsing

Our new aws-lambda-java-core-events-1.2.0 instrumentation parses event source information and extracts information added as agent attributes. Please refer to the agent spec for more details

Distributed tracing

Our distributed tracing has not changed its functionality in serverless mode. However we do accept inbound headers in web transactions.

Adaptive Sampling Changes

The adaptive sampler now begins its cycle lazily when the first transaction is hit.

Metadata

Our aws-lambda-java-core-1.2.0 captures lambda ARNs and function version which is reported into both transaction agent attributes and as agent metadata.

Harvest Cycle and Data Marshalling

The biggest change in serverless mode is its methodology of data marshalling.

Our harvest cycle is altered to only harvest if a transaction is finished.

When harvesting, our payloads are written to /tmp/newrelic-telemetry and to STDOUT so it is picked up by Cloudwatch.

You can refer to the agent spec regarding the payload's format. It is broken down into 2 major parts, metadata and encoded data and is formatted as a json string.

The metadata includes data about the function, agent, and environment. That portion is never compressed.

Here is an example json:

 {
    "arn": "AWS_LAMBDA_FUNCTION_ARN",
    "protocol_version": 16,
    "function_version": "15",
    "execution_environment": "AWS_Lambda_python3.6",
    "agent_version": "4.2.0.100",
    "metadata_version": 2,
    "agent_language": "python"
  }

The encoded data is all other data (i.e. metrics, traces, and events). It is not compressed when sent to cloudwatch but it is when sent to /tmp/newrelic-telemetry. When it is compressed, it is compressed using gzipn and then encoded with baset64.

Here is an example pre-compression payload:

[
  2,
  "NR_LAMBDA_MONITORING",
  { // Not-Compressed
    "arn": "AWS_LAMBDA_FUNCTION_ARN",
    "protocol_version": 16,
    "function_version": "15",
    "execution_environment": "AWS_Lambda_python3.6",
    "agent_version": "4.2.0.100",
    "metadata_version": 2,
    "agent_language": "python"
  },
  { // Compressed Encoded Data
    "metric_data": [
      null,
      1537900783.364288,
      1537900784.365288,
      [
        [{"name": "Python/WSGI/Output/Bytes", "scope": ""}, [1, 2, 2, 2, 2, 4]],
        [{"name": "HttpDispatcher", "scope": "" }, [1, 3288, 3288, 3288, 3288, 10810944]]
      ]
    ],
    "analytic_event_data": [
      null,
      {"reservoir_size": 10000, "events_seen": 8},
      [
        [
          {
            "databaseCallCount": 1,
            "databaseDuration": 2.2172927856445312e-05,
            "duration": 0.1086888313293457,
            "guid": "24d071916e1f00ee",
            "name": "WebTransaction/Function/__main__:index",
            "port": 8001,
            "priority": 1.9781,
            "sampled": true,
            "timestamp": 1537983775490,
            "totalTime": 0.1086888313293457,
            "traceId": "24d071916e1f00ee",
            "type": "Transaction"
          },
          {},
          {
            "request.method": "GET",
            "request.uri": "/",
            "response.headers.contentLength": 11,
            "response.headers.contentType": "text/html; charset=utf-8",
            "response.status": "200",
            "aws.requestId":"ee7219f7-14b7-4812-b2cc-229d670995e0",
            "aws.lambda.arn":"AWS_LAMBDA_FUNCTION_ARN",
            "aws.lambda.coldStart":true
          }
        ]
      ]
    ],
    "custom_event_data": [
      null,
      { "events_seen": 1, "reservoir_size": 1200 },
      [
        [
          {
            "timestamp": 1538515634777,
            "type": "CustomEvent"
          },
          { // only primitives are supported as custom attribute values
            "foo": "bar"
          }
        ]
      ]
    ],
    "error_event_data": [
      null,
      { "events_seen": 1, "reservoir_size": 100 },
      [
        [
          {
            "duration": 0.008540153503417969,
            "error.class": "builtins:ValueError",
            "error.message": "foo",
            "guid": "8599a29ea51695dd",
            "nr.transactionGuid": "8599a29ea51695dd",
            "port": 8001,
            "priority": 1.00056,
            "sampled": true,
            "timestamp": 1538515743243,
            "traceId": "8599a29ea51695dd",
            "transactionName": "WebTransaction/Function/__main__:error",
            "type": "TransactionError"
          },
          {},
          {
            "request.headers.accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
            "request.headers.host": "localhost:8001",
            "request.headers.userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36",
            "request.method": "GET",
            "request.uri": "/error",
            "response.headers.contentLength": 291,
            "response.headers.contentType": "text/html",
            "response.status": "500",
            "wsgi.output.bytes": 291,
            "wsgi.output.calls.yield": 1,
            "wsgi.output.seconds": 0.0005190372467041016
          }
        ]
      ]
    ],
    "span_event_data": [
      null,
      {
        "events_seen": 1,
        "reservoir_size": 1000
      },
      [
        [
          {
            "category": "generic",
            "duration": 0.10616874694824219,
            "guid": "fb57620df8ca8a04",
            "name": "Function/__main__:index",
            "nr.entryPoint": true,
            "priority": 1.91012,
            "sampled": true,
            "timestamp": 1537900783364,
            "traceId": "446cf2064d931f4e",
            "transactionId": "446cf2064d931f4e",
            "type": "Span"
          },
          {},
          {}
        ]
      ]
    ],
    "sql_trace_data": [
      [
        [
          "OtherTransaction/Function/__main__:query",
          "",
          2394762929494546858,
          "SELECT pg_sleep(0.1)",
          "Datastore/operation/Postgres/select",
          1,
          102.43511199951172,
          102.43511199951172,
          102.43511199951172,
          "eJytkk1rAjEQhv9KyElBzK4fIFILpba9iKhUSlEJMTt1g9kkTbKsS+l/76jQ1kMvxcuQZF7meTMzH3Qr5D56IYEOV/T5eDi+kEZhQyQeJJhIpNCaaBFic0hb9FFpIGsqnGu7ek1bRCsDJO21iDLkprBZqeH2R8eWAXxg4g10VgjDxlaWBVYNzNUxt4aLHd6YgcqDVpIJp9jRws7b0mQ8irC/4CTdE6jyaAD8337Sk+y9BF//30xu7T6wTESxFQG4C7W0btf5DeqfQXAAWUa4GirbYiMuQN3kOqBjg6MqgJ/GfjnD8xA5h4OKnNNNi8LBaaEMx2hwRVZ0vnxYvJLZ5G6KabwvIJQ6EtKQuDGjpJ0k7TbGlHhbhVFKKpXFfJQ06WaD5XIU0SG9Tzrz8WA6eRq8zPAnzvrInYg5t56rDAX9XreDie92GFHghqIwxJ2HQD+/AP817WQ="
        ]
      ]
    ],
    "error_data": [
      null,
      [
        [
          1538516776.94827,
          "OtherTransaction/Function/__main__:main",
          "",
          "builtins:ValueError",
          {
            "agentAttributes": {},
            "intrinsics": {
              "guid": "cd3d2ab1f0576710",
              "priority": 1.13426,
              "sampled": true,
              "totalTime": 0.00023293495178222656,
              "traceId": "cd3d2ab1f0576710",
              "trip_id": "cd3d2ab1f0576710"
            },
            "request_uri": null,
            "stack_trace": [
              "Traceback (most recent call last):",
              "File \"app.py\", line 10, in <module>",
              "File \"background_task.py\", line 103, in wrapper",
              "File \"app.py\", line 6, in main"
            ],
            "userAttributes": {}
          }
        ]
      ]
    ],
    "transaction_sample_data": [
      null,
      [
        [
          1538516656918.499,
          0.141143798828125,
          "OtherTransaction/Function/__main__:main",
          null,
          "eJx1T01rhDAQ/S9zDjbZGBu99VLoSSjeREJW43YgRslHaVn8741tL2W7MDDDm/fevOl7JrgUrKpEVTNZlHVNrvtRPS0ooQUrGSv5Yy3lSbKTIPDath18E+4x2vhmfOe1C3qMuLqH5+R+BqUWjU6p5mjZBMzHaFPAd6Om5PXBUQtaiwGaG+N8cSAuWTv8tqxHFz26gGMWXCHPm8IJGpBMjIzPs6Scn+eJAYG4Rm07XMxhTSm9iX1J96RBL5s1eRl9MgQ2j6vH+AkNKziVgmdzr0fz8q98J6AvxsWnmOOdUzRH1AymYPxfbB/yh8MXyNZ6KA==",
          "815c13ff8033bfd1",
          null,
          false,
          null,
          null
        ]
      ]
    ]
  }
]

DIsabled Features

JFR and Jar Collection is currently disabled. This may be re-enabled per feature request.

What is not included?

We do not include a lambda_handler / set_lambda_handler API. This is because our auto-instrumentation handles that work for us. In the future we may want to support such API for slim lambda layers.

In addition serverless mode alone does affect configuration for span event collection, logging, slow sql traces, and collecting transactions traces. This is handled in our lambda layer start scripts.

In our lambda layer start scripts, by default slow sql traces, span event collections, and collecting transaction traces are enabled.

Our lambda layer start scripts will also disable log forwarding. This is because analytic log events are ignored by the backend making it so such events provide unneeded overhead.

Related Github Issue

#2704

[codecov-commenter]

Codecov Report

❌ Patch coverage is 82.55250% with 108 lines in your changes missing coverage. Please review.
✅ Project coverage is 70.65%. Comparing base (bfbb4f5) to head (9f3aa1a).
⚠️ Report is 5 commits behind head on main.

Files with missing lines	Patch %	Lines
...a/com/newrelic/agent/ServerlessHarvestService.java	66.44%	43 Missing and 7 partials ⚠️
...transport/serverless/DataSenderServerlessImpl.java	84.46%	14 Missing and 2 partials ⚠️
...t/src/main/java/com/newrelic/agent/RPMService.java	54.54%	9 Missing and 1 partial ⚠️
...ent/transport/serverless/ServerlessWriterImpl.java	33.33%	8 Missing ⚠️
...wrelic/agent/serverless/ServerlessServiceImpl.java	75.00%	4 Missing ⚠️
.../src/main/java/com/newrelic/agent/Transaction.java	0.00%	2 Missing and 1 partial ⚠️
...ava/com/newrelic/agent/trace/TransactionTrace.java	25.00%	2 Missing and 1 partial ⚠️
...ic/agent/transport/serverless/TelemetryBuffer.java	98.49%	0 Missing and 3 partials ⚠️
...om/newrelic/agent/config/ServerlessConfigImpl.java	88.88%	2 Missing ⚠️
...ent/introspec/internal/IntrospectorRPMService.java	0.00%	1 Missing ⚠️
... and 8 more

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2796      +/-   ##
============================================
+ Coverage     70.32%   70.65%   +0.33%     
- Complexity    10457    10620     +163     
============================================
  Files           873      881       +8     
  Lines         42287    42913     +626     
  Branches       6419     6486      +67     
============================================
+ Hits          29740    30322     +582     
- Misses         9645     9668      +23     
- Partials       2902     2923      +21     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review is ineligible. To be eligible to request a review, you need a paid Copilot license, or your organization must enable Copilot code review.

[jasonjkeller]

LGTM!