Detect exceptions raised outside of the handler by TalUsvyatsky · Pull Request #475 · DataDog/datadog-lambda-python

TalUsvyatsky · 2024-04-17T20:31:15Z

What does this PR do?

This PR adds telemetry for exceptions that occur outside of the handler function.

When an exception occurs outside of the handler function:

aws.lambda.enhanced.errors is emitted
A trace with an error span is created

Motivation

https://datadoghq.atlassian.net/browse/SVLS-4666
Users should have visibility into this category of errors.
When a line of code raises outside of the handler function, then datadog is not able to detect the error. #210

Testing Guidelines

Created a handler file like this:

  import json
 
  0/0 # Purposely raise an exception

  def hello(event, context):
      body = {
          "message": "some message",
          "input": event,
      }
      return {"statusCode": 200, "body": json.dumps(body)}

Deployed my function and invoked it via function URL
Saw traces with an error span from the call (example trace)
Saw aws.lambda.enhanced.errors emitted (graph)
Traces appear in serverless detail page invocation table (link to search for function)
Issues appear in error tracking and are informative

Additional Notes

Types of Changes

Bug fix
New feature
Breaking change
Misc (docs, refactoring, dependency upgrade, etc.)

Check all that apply

This PR's description is comprehensive
This PR contains breaking changes that are documented in the description
This PR introduces new APIs or parameters that are documented and unlikely to change in the foreseeable future
This PR impacts documentation, and it has been updated (or a ticket has been logged)
This PR's changes are covered by the automated tests
This PR collects user input/sensitive content into Datadog
This PR passes the integration tests (ask a Datadog member to run the tests)

https://datadoghq.atlassian.net/browse/SVLS-4666

datadog_lambda/handler.py

tests/test_handler.py

datadog_lambda/handler.py

purple4reina · 2024-04-23T19:34:07Z

FYI, pulling from main should fix the check layer size test failure.

datadog_lambda/wrapper.py

datadog_lambda/tracing.py

duncanista

Amazing work – LGTM.

There's one nit:

datadog-lambda-python/datadog_lambda/tracing.py

Lines 1236 to 1255 in 21012cb

    
           tags = None 
        
           if context: 
        
               function_arn = (context.invoked_function_arn or "").lower() 
        
               tk = function_arn.split(":") 
        
               function_arn = ":".join(tk[0:7]) if len(tk) > 7 else function_arn 
        
               function_version = tk[7] if len(tk) > 7 else "$LATEST" 
        
               tags = { 
        
                   "cold_start": str(is_cold_start).lower(), 
        
                   "function_arn": function_arn, 
        
                   "function_version": function_version, 
        
                   "request_id": context.aws_request_id, 
        
                   "resource_names": context.function_name, 
        
                   "functionname": context.function_name.lower() 
        
                   if context.function_name 
        
                   else None, 
        
                   "datadog_lambda": datadog_lambda_version, 
        
                   "dd_trace": ddtrace_version, 
        
                   "span.name": "aws.lambda", 
        
               } 
        
           tags = tags or {}

Should we care about some of the tags that can be added from this? i.e. functionname, cold_start (which should always be true if we are loading), datadog_lambda_version, ddtrace_version

duncanista

LGTM – left a nit.

datadog_lambda/tags.py

joeyzhao2018

LGTM 👍

…d outside of the handler function

Co-authored-by: jordan gonzález <30836115+duncanista@users.noreply.github.com>

TalUsvyatsky force-pushed the tal.usvyatsky/capture-handler-import-failure branch 8 times, most recently from 80486fd to 76e5cee Compare April 23, 2024 14:17

TalUsvyatsky marked this pull request as ready for review April 23, 2024 14:32

TalUsvyatsky requested a review from a team as a code owner April 23, 2024 14:32

joeyzhao2018 reviewed Apr 23, 2024

View reviewed changes

datadog_lambda/handler.py Outdated Show resolved Hide resolved

duncanista reviewed Apr 23, 2024

View reviewed changes

datadog_lambda/handler.py Show resolved Hide resolved

duncanista reviewed Apr 23, 2024

View reviewed changes

datadog_lambda/handler.py Outdated Show resolved Hide resolved

duncanista reviewed Apr 23, 2024

View reviewed changes

tests/test_handler.py Show resolved Hide resolved

purple4reina reviewed Apr 23, 2024

View reviewed changes

datadog_lambda/handler.py Outdated Show resolved Hide resolved

purple4reina reviewed Apr 23, 2024

View reviewed changes

datadog_lambda/handler.py Outdated Show resolved Hide resolved

TalUsvyatsky force-pushed the tal.usvyatsky/capture-handler-import-failure branch from 85d0d7d to 0bc08fd Compare April 23, 2024 19:00

purple4reina reviewed Apr 23, 2024

View reviewed changes

datadog_lambda/wrapper.py Outdated Show resolved Hide resolved

purple4reina approved these changes Apr 23, 2024

View reviewed changes

TalUsvyatsky force-pushed the tal.usvyatsky/capture-handler-import-failure branch from 39af365 to f0e8c5e Compare April 24, 2024 13:32

joeyzhao2018 reviewed Apr 24, 2024

View reviewed changes

datadog_lambda/tracing.py Outdated Show resolved Hide resolved

TalUsvyatsky requested review from duncanista and joeyzhao2018 April 24, 2024 18:39

duncanista approved these changes Apr 24, 2024

View reviewed changes

TalUsvyatsky force-pushed the tal.usvyatsky/capture-handler-import-failure branch 2 times, most recently from 26ed616 to 47c0dab Compare April 29, 2024 17:16

TalUsvyatsky requested review from duncanista and purple4reina April 29, 2024 17:46

duncanista approved these changes Apr 29, 2024

View reviewed changes

datadog_lambda/tags.py Outdated Show resolved Hide resolved

purple4reina approved these changes Apr 29, 2024

View reviewed changes

joeyzhao2018 approved these changes Apr 29, 2024

View reviewed changes

TalUsvyatsky and others added 7 commits May 1, 2024 09:55

emit enhanced error metric and create span when an exception is raise…

57ce604

…d outside of the handler function

rename handler

885adfe

move fallback handler to wrapper.py

fb6e60c

fix duration

101cfb9

respect DD_TRACE_ENABLED

63f6812

emit telemetry and raise during init

a4e3f09

Update datadog_lambda/tags.py

7c5aa58

Co-authored-by: jordan gonzález <30836115+duncanista@users.noreply.github.com>

TalUsvyatsky force-pushed the tal.usvyatsky/capture-handler-import-failure branch from f6a8496 to 7c5aa58 Compare May 1, 2024 13:55

TalUsvyatsky merged commit a95341e into main May 1, 2024

TalUsvyatsky deleted the tal.usvyatsky/capture-handler-import-failure branch May 1, 2024 14:15

duncanista mentioned this pull request May 2, 2024

When a line of code raises outside of the handler function, then datadog is not able to detect the error. #210

Closed

TalUsvyatsky mentioned this pull request May 7, 2024

[SVLS-4699] Detect exceptions raised outside of the handler DataDog/datadog-lambda-js#533

Merged

11 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Detect exceptions raised outside of the handler#475

Detect exceptions raised outside of the handler#475
TalUsvyatsky merged 7 commits intomainfrom
tal.usvyatsky/capture-handler-import-failure

TalUsvyatsky commented Apr 17, 2024 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

purple4reina commented Apr 23, 2024

Uh oh!

Uh oh!

Uh oh!

duncanista left a comment

Uh oh!

duncanista left a comment

Uh oh!

Uh oh!

joeyzhao2018 left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	tags = None
	if context:
	function_arn = (context.invoked_function_arn or "").lower()
	tk = function_arn.split(":")
	function_arn = ":".join(tk[0:7]) if len(tk) > 7 else function_arn
	function_version = tk[7] if len(tk) > 7 else "$LATEST"
	tags = {
	"cold_start": str(is_cold_start).lower(),
	"function_arn": function_arn,
	"function_version": function_version,
	"request_id": context.aws_request_id,
	"resource_names": context.function_name,
	"functionname": context.function_name.lower()
	if context.function_name
	else None,
	"datadog_lambda": datadog_lambda_version,
	"dd_trace": ddtrace_version,
	"span.name": "aws.lambda",
	}
	tags = tags or {}

Conversation

TalUsvyatsky commented Apr 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

Testing Guidelines

Additional Notes

Types of Changes

Check all that apply

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

purple4reina commented Apr 23, 2024

Uh oh!

Uh oh!

Uh oh!

duncanista left a comment

Choose a reason for hiding this comment

Uh oh!

duncanista left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

joeyzhao2018 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

TalUsvyatsky commented Apr 17, 2024 •

edited

Loading