Summary

Creating subsegments in Python global context (aka startup code) fails with errors:

It is an AWS best practice to "Initialize SDK clients and database connections outside of the function handler" https://docs.aws.amazon.com/lambda/latest/dg/best-practices.html

However, since XRay will not create a subsegment in the global startup code, we have no view into this code with XRay. Without a subsegment, we also can not put_annotation() with XRay.

Setup

• AWS Lambda
• Python 3.7
• XRay SDK for Python 2.4.2

Repo

1. In Python code outside lambda_handler() write:

with xray_recorder.in_subsegment('mysubseg') as subsegment:
    subsegment.put_annotation('myanno1', '1')
    print('my old school print')

2. Continue with any simple AWS Lambda Python function
3. Execute the function

Result

Errors in the AWS Lambda console (if you used that) and/or errors in the Cloudwatch logs.

[WARNING] 2019-11-24T15:41:32.241Z Subsegment mysubseg discarded due to Lambda worker still initializing
my old school print
[WARNING] 2019-11-24T15:41:32.245Z No subsegment to end.

In XRay console, there is no subsegment for mysubseg. There is no annotation myanno1.
Only seen is the Initialization black box.

Expected

Within Initialization is the subsegment mysubseg (and another other subsegments I might make) and on the mysubseg is the annotation myanno1=1.

Workarounds

• Use print()
• Use the "discarded" error messages as an implied trace.
• Don't use global code or the AWS best practice approach. If you need similar functionality, consider putting it in a python class, have class initializer that sets class attributes, and on every call to the lambda handler, check if the class has yet been initialized.

Notes

I think there is a related yet separate bug: XRay fails to show traces for boto calls in global code. I patch(botocore) yet calls to services like boto3.session.Session().client('ssm')... are invisible and not reported in any xray data or cloudwatch log.