Boto / Botocore integration by raphaelgavache · Pull Request #209 · DataDog/dd-trace-py

raphaelgavache · 2017-03-02T16:04:34Z

# Botocore and Boto integrations

Summary

Adds botocore and boto contribs with testing units:
boto : s3/ec2/sqs/lambda
botocore : s3/ec2/sqs/lambda/kinesis
Boto3 relies on botocore so it will also be traced

Compatibility

boto >=2.29.0
botocore >= 1.4.51

TODOS

Miscellanous notes

botocore is really clean, boto is a mess:
- the first has one big class that customises itself in function of what service is called. The methods are created from JSON and the result has a standard format shared by all services.
- boto has one specific class for each AWS and which can inherit from 2 different classes AWSQueryConnection (ec2 connector for exemple) and AWSAuthConnection (s3 connector ...). Both classes make_request() methods are patched but they do not contain the same metadata. For exemple AWSAuthConnection instance does not contain the operation name called on the s3 AWS.
boto both class AWSQueryConnection and AWSAuthConnection have a region attribute. For some it's a string, others have different classes (sqs has his own class...) but there's also a standard Region class -> this is why I added a function get_region_name

inspired from https://github.com/DataDog/botocore/pull/4/files, which has been tested on staging (see https://dd.datad0g.com/trace/view/177373499236743437)

…e-py into raphael/contrib-boto

… into raphael/contrib-boto

talwai · 2017-03-06T16:52:02Z

+
+        # Original func returns a boto.connection.HPPResponse object
+        result = original_func(*args, **kwargs)
+        span.set_tag(http.STATUS_CODE, getattr(result, "status", 500))


let's not have it default to 500. This is a meaningful error code that influences downstream stats collection and error reporting/alerting so we do not want to set it just because there's no better candidate. Instead it's fine to set the STATUS_CODE meta only when result.status is available

talwai · 2017-03-06T16:53:33Z

+        return result
+
+def get_region_name(region):
+    if type(region) == str:


use isinstance

talwai · 2017-03-06T17:09:13Z

+
+def patch():
+    # Checking for the version compatibility before patching
+    if LooseVersion(pkg_resources.get_distribution("botocore").version) >= LooseVersion("1.4.51"):


i don't think we want version matching code at this layer. instead let the patching function fail if a patchable isn't available - and patch_all will respond to this gracefully

talwai · 2017-03-06T17:17:06Z

+
+# s3, lambda
+def patched_auth_request(original_func, instance, args, kwargs):
+    tracer = getattr(instance, 'datadog_tracer', ddtrace.tracer)


instead use a Pin

e.g. https://github.com/DataDog/dd-trace-py/blob/master/ddtrace/contrib/redis/patch.py#L51-L53

talwai · 2017-03-08T19:23:52Z

+
+    with pin.tracer.trace('boto.command', service=DEFAULT_SERVICE, span_type=SPAN_TYPE) as span:
+
+        # Original func returns a boto.connection.HPPResponse object


talwai · 2017-03-08T19:24:36Z

+        # Original func returns a boto.connection.HPPResponse object
+        result = original_func(*args, **kwargs)
+        span.set_tag(http.STATUS_CODE, getattr(result, "status"))
+        span.set_tag(http.METHOD, getattr(result, "_method", "unknown"))


rather than set "unknown", it's ok to leave metadata unset in general when the needed attribute is not available

b1-88er · 2017-03-13T12:32:24Z

+        s3.get_all_buckets()
+        spans = writer.pop()
+        assert spans
+        span = spans[0]


does length of spans matter here?

one span is generated each time the function patched is called. For each request of the S3 client https://github.com/boto/boto/blob/develop/boto/s3/connection.py#L440, patched function gets called only once. On tests on our staging env there's a unique trace per s3 call. Yet in the mocking env I get 3 spans, I'll investigate on that

talwai · 2017-03-16T15:14:53Z

+    operation_name = None
+    for trace in reversed(traceback.extract_stack()):
+        # Going backwards in the traceback till first call outside off ddtrace before make_request
+        if "ddtrace" not in trace[0].split('/') and trace[2] != 'make_request':


this code worries me. if there's no more reliable way to pull an operation_name, we should at least have a length check on the stacktrace before we try to index into it arbitrarily.

talwai · 2017-03-16T15:22:32Z

+    if endpoint_name in BLACKLIST_ENDPOINT:
+        return True
+    else:
+        return False


just return endpoint_name in BLACKLIST_ENDPOINT

talwai · 2017-03-16T15:23:42Z

+    else:
+        return False
+
+def unpacking_args(args, args_name, traced_args_list):


this needs a docstring

…llocator (#17664) ## Description This PR fixes a segmentation fault in the memory allocation profiler that occurs when a hook call races with `memalloc` start/stop operations. The issue arises from concurrent access to the saved allocator struct, which could be partially written while being read, resulting in`NULL` function pointers being dereferenced. The key indicator in that case is that `#1 0x0000000000000000` frame -- we are trying to execute a null function pointer. ```` Error UnixSignal: Process terminated with SEGV_MAPERR (SIGSEGV) #0 0x00007ff3c303a8d4 #1 0x0000000000000000 memalloc_alloc (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/profiling/collector/_memalloc.cpp:68) #2 0x00007ff39dcb3b20 memalloc_alloc (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/profiling/collector/_memalloc.cpp:68) #3 0x00007ff39dcb3b20 memalloc_malloc(void*, unsigned long) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/profiling/collector/_memalloc.cpp:80) #4 0x00007ff3c3087e1b PyUnicode_New #5 0x00007ff3c30889f4 #6 0x00007ff3c3170c84 #7 0x00007ff3c316b931 #8 0x00007ff3c31aaac8 #9 0x00007ff3c31033ac #10 0x00007ff3c310e2a6 PyObject_CallMethodObjArgs #11 0x00007ff3c310e46d #12 0x00007ff3c31a96c2 #13 0x00007ff3c3102fd7 PyObject_Vectorcall #14 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #15 0x00007ff3c323c094 #16 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #17 0x00007ff3c323c094 #18 0x00007ff3c30e997d PyObject_CallOneArg #19 0x00007ff3c306a480 _PyObject_GenericGetAttrWithDict #20 0x00007ff3c30c620d PyObject_GetAttr #21 0x00007ff3c32309e7 _PyEval_EvalFrameDefault #22 0x00007ff3c323c094 #23 0x00007ff3c312880e #24 0x00007ff3c30e917c _PyObject_MakeTpCall #25 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #26 0x00007ff3c323c094 #27 0x00007ff3c312880e #28 0x00007ff3c30e917c _PyObject_MakeTpCall #29 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #30 0x00007ff3c323c094 #31 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #32 0x00007ff3c323c094 #33 0x00007ff3c317d0fd #34 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #35 0x00007ff3c323c094 #36 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #37 0x00007ff3c323c094 #38 0x00007ff3c317d1b5 #39 0x00007ff3c3102fd7 PyObject_Vectorcall #40 0x00007ff3c3232f4a _PyEval_EvalFrameDefault #41 0x00007ff3c3240da5 #42 0x00007ff3c324112d #43 0x00007ff3c3233be1 _PyEval_EvalFrameDefault #44 0x00007ff3c323c094 #45 0x00007ff3c317d1b5 #46 0x00007ff3c3102fd7 PyObject_Vectorcall #47 0x00007ff3c3232f4a _PyEval_EvalFrameDefault #48 0x00007ff3c323c094 #49 0x00007ff3c31033ac #50 0x00007ff3c310358d PyObject_CallFunctionObjArgs #51 0x00007ff3bf7eb91d WraptBoundFunctionWrapper_call (/project/src/wrapt/_wrappers.c:3750) #52 0x00007ff3c3104055 _PyObject_Call #53 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #54 0x00007ff3c323c094 #55 0x00007ff3c317d23c #56 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #57 0x00007ff3c323c094 #58 0x00007ff3c310416f _PyObject_Call #59 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #60 0x00007ff3c3240da5 #61 0x00007ff3c324112d #62 0x00007ff3c3233be1 _PyEval_EvalFrameDefault #63 0x00007ff3c323c094 #64 0x00007ff3c317d1b5 #65 0x00007ff3c3102fd7 PyObject_Vectorcall #66 0x00007ff3c3232f4a _PyEval_EvalFrameDefault #67 0x00007ff3c323c094 #68 0x00007ff3c317d1b5 #69 0x00007ff3c310416f _PyObject_Call #70 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #71 0x00007ff3c323c094 #72 0x00007ff3c317d1b5 #73 0x00007ff3c310416f _PyObject_Call #74 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #75 0x00007ff3c323c094 #76 0x00007ff3c31033ac #77 0x00007ff3c310358d PyObject_CallFunctionObjArgs #78 0x00007ff3bf7eb91d WraptBoundFunctionWrapper_call (/project/src/wrapt/_wrappers.c:3750) #79 0x00007ff3c30e917c _PyObject_MakeTpCall #80 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #81 0x00007ff3c323c094 #82 0x00007ff3c317d518 #83 0x00007ff3c3155963 #84 0x00007ff3c315393d #85 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #86 0x00007ff3c323c094 #87 0x00007ff3c317d0fd #88 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #89 0x00007ff3c323c094 #90 0x00007ff3c317d0fd #91 0x00007ff3c317d518 #92 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #93 0x00007ff3c323c094 #94 0x00007ff3c30e9371 _PyObject_FastCallDictTstate #95 0x00007ff3c30e958d _PyObject_Call_Prepend #96 0x00007ff3c3109150 #97 0x00007ff3c3104055 _PyObject_Call #98 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #99 0x00007ff3c323c094 #100 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #101 0x00007ff3c30e958d _PyObject_Call_Prepend #102 0x00007ff3c3109150 #103 0x00007ff3c30e917c _PyObject_MakeTpCall #104 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #105 0x00007ff3c323c094 #106 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #107 0x00007ff3c30e958d _PyObject_Call_Prepend #108 0x00007ff3c3109150 #109 0x00007ff3c30e917c _PyObject_MakeTpCall #110 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #111 0x00007ff3c323c094 #112 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #113 0x00007ff3c30e958d _PyObject_Call_Prepend #114 0x00007ff3c3109150 #115 0x00007ff3c30e917c _PyObject_MakeTpCall #116 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #117 0x00007ff3c323c094 #118 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #119 0x00007ff3c30e958d _PyObject_Call_Prepend #120 0x00007ff3c3109150 #121 0x00007ff3c30e917c _PyObject_MakeTpCall #122 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #123 0x00007ff3c323c094 #124 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #125 0x00007ff3c30e958d _PyObject_Call_Prepend #126 0x00007ff3c3109150 #127 0x00007ff3c30e917c _PyObject_MakeTpCall #128 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #129 0x00007ff3c323c094 #130 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #131 0x00007ff3c30e958d _PyObject_Call_Prepend #132 0x00007ff3c3109150 #133 0x00007ff3c30e917c _PyObject_MakeTpCall #134 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #135 0x00007ff3c323c094 #136 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #137 0x00007ff3c30e958d _PyObject_Call_Prepend #138 0x00007ff3c3109150 #139 0x00007ff3c30e917c _PyObject_MakeTpCall #140 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #141 0x00007ff3c323c094 #142 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #143 0x00007ff3c30e958d _PyObject_Call_Prepend #144 0x00007ff3c3109150 #145 0x00007ff3c30e917c _PyObject_MakeTpCall #146 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #147 0x00007ff3c323c094 #148 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #149 0x00007ff3c30e958d _PyObject_Call_Prepend #150 0x00007ff3c3109150 #151 0x00007ff3c30e917c _PyObject_MakeTpCall #152 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #153 0x00007ff3c323c094 #154 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #155 0x00007ff3c30e958d _PyObject_Call_Prepend #156 0x00007ff3c3109150 #157 0x00007ff3c30e917c _PyObject_MakeTpCall #158 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #159 0x00007ff3c323c094 #160 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #161 0x00007ff3c30e958d _PyObject_Call_Prepend #162 0x00007ff3c3109150 #163 0x00007ff3c30e917c _PyObject_MakeTpCall #164 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #165 0x00007ff3c323c094 #166 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #167 0x00007ff3c30e958d _PyObject_Call_Prepend #168 0x00007ff3c3109150 #169 0x00007ff3c30e917c _PyObject_MakeTpCall #170 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #171 0x00007ff3c323c094 #172 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #173 0x00007ff3c30e958d _PyObject_Call_Prepend #174 0x00007ff3c3109150 #175 0x00007ff3c30e917c _PyObject_MakeTpCall #176 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #177 0x00007ff3c323c094 #178 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #179 0x00007ff3c30e958d _PyObject_Call_Prepend #180 0x00007ff3c3109150 #181 0x00007ff3c30e917c _PyObject_MakeTpCall #182 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #183 0x00007ff3c323c094 #184 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #185 0x00007ff3c30e958d _PyObject_Call_Prepend #186 0x00007ff3c3109150 #187 0x00007ff3c30e917c _PyObject_MakeTpCall #188 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #189 0x00007ff3c323c094 #190 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #191 0x00007ff3c323c094 #192 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #193 0x00007ff3c30e958d _PyObject_Call_Prepend #194 0x00007ff3c3109150 #195 0x00007ff3c30e917c _PyObject_MakeTpCall #196 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #197 0x00007ff3c323c094 #198 0x00007ff3c317d0fd #199 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #200 0x00007ff3c323c094 #201 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #202 0x00007ff3c323c094 #203 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #204 0x00007ff3c323c094 #205 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #206 0x00007ff3c323c094 #207 0x00007ff3c317d23c #208 0x00007ff3c31a7ec5 #209 0x00007ff3c301ac77 #210 0x00007ff3c357c573 ```` The fix implements two key changes. 1. **Hook functions (`memalloc_alloc`, `memalloc_realloc`)**: Snapshot the allocator struct locally before use and guard indirect function calls with `NULL` checks. This prevents crashes if a partially-written struct is observed during a start/stop race. 2. **Start/stop operations (`memalloc_start`, `memalloc_stop`)**: Use local variables and single assignments when publishing the allocator struct to `global_memalloc_ctx.pymem_allocator_obj`. This ensures concurrent hook calls observe either the old or new struct, never a partially-written intermediate state. The real root cause is that `PyMem_GetAllocator` is not documented as atomic, and the struct could be read field-by-field while being written to concurrently. By using local copies and single assignments, we ensure atomicity at the C level and prevent observation of inconsistent state. Co-authored-by: thomas.kowalski <thomas.kowalski@datadoghq.com>

tmichelet and others added 7 commits October 4, 2016 16:26

[Contrib] Botocore - first pass of a botocore contrib

55c8635

inspired from https://github.com/DataDog/botocore/pull/4/files, which has been tested on staging (see https://dd.datad0g.com/trace/view/177373499236743437)

Merge branch 'tristan/contrib-botocore' of github.com:DataDog/dd-trac…

3cb4df8

…e-py into raphael/contrib-boto

First commit, botocore contrib (tests working on local machine)

3e5fa76

Contrib test taken into account

3749696

fixing contrib patching issue

f243c68

should pass contrib test

b91d6da

Botocore with tests runing on circleCi

14fee39

clutchski mentioned this pull request Mar 2, 2017

[Contrib] Botocore - first pass of a botocore contrib #72

Closed

3 tasks

Raphael Gavache and others added 7 commits March 2, 2017 12:43

adding http status code and number of retry attempts

ed7ccd6

Adding boto integration

56eb079

Merge branch 'master' into raphael/contrib-boto

60b1bfc

added http.STATUS_CODE and http.METHOD

3ffaf72

Merge branch 'raphael/contrib-boto' of github.com:DataDog/dd-trace-py…

330e91f

… into raphael/contrib-boto

sqs/lambda tests on boto (tests on Circleci)`

270712b

adding mocked AWS calls to botocore tests

19ccd45

raphaelgavache requested a review from talwai March 3, 2017 21:57

talwai reviewed Mar 6, 2017

View reviewed changes

talwai changed the title ~~Raphael/contrib boto~~ Boto / Botocore integration Mar 6, 2017

talwai reviewed Mar 6, 2017

View reviewed changes

Raphael Gavache added 7 commits March 6, 2017 16:48

Pin ddagent, adding tests and nps

3e63ace

small fix for tests

6f28c94

adding boto/botocore in monkey default

e3b4894

fixing json.dumps errors

8860926

no more json encoding

32a29a6

corrections on names

ed79d36

Changing test for py34 success

fc5bc14

talwai reviewed Mar 8, 2017

View reviewed changes

Raphael Gavache added 2 commits March 8, 2017 17:13

botocore test fix

e338c5f

adding sts, kms args/kwargs protection

d975019

palazzem mentioned this pull request Mar 9, 2017

boto tracing #205

Closed

palazzem modified the milestone: 0.7.0 Mar 9, 2017

adding boto sts/kms protection

48f1b9f

palazzem added the integrations label Mar 9, 2017

Adding operation names for the rest of boto integrations

8ec2ec2

b1-88er reviewed Mar 13, 2017

View reviewed changes

Raphael Gavache and others added 8 commits March 13, 2017 17:32

changing boto args, protection, way to get the endpoint

cf3d611

protection against py34/py35 tests with diff messages

3fec0f2

upgrading boto operation name research (scalable)

3d22f92

correction on boto getting operation name

4222296

code cleaning and upgrading tests

ecc48ab

ready for test, setting autopatch to false

916aa31

flake8 correction

10769d5

ready for test, unpatched

fab3448

talwai reviewed Mar 16, 2017

View reviewed changes

Raphael Gavache added 4 commits March 16, 2017 11:43

Aaditya's review

55e57fa

removing len(trace_frame), not working py3

5c6e262

changing traceback to inspect

af2c47e

np: readability

f394197

talwai approved these changes Mar 16, 2017

View reviewed changes

raphaelgavache merged commit 38dc43d into master Mar 16, 2017

palazzem deleted the raphael/contrib-boto branch March 16, 2017 21:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Boto / Botocore integration#209

Boto / Botocore integration#209
raphaelgavache merged 38 commits into
masterfrom
raphael/contrib-boto

raphaelgavache commented Mar 2, 2017 •

edited

Loading

Uh oh!

talwai Mar 6, 2017

Uh oh!

talwai Mar 6, 2017

Uh oh!

talwai Mar 6, 2017

Uh oh!

talwai Mar 6, 2017

Uh oh!

talwai Mar 8, 2017

Uh oh!

talwai Mar 8, 2017

Uh oh!

b1-88er Mar 13, 2017 •

edited

Loading

Uh oh!

raphaelgavache Mar 15, 2017

Uh oh!

talwai Mar 16, 2017

Uh oh!

talwai Mar 16, 2017

Uh oh!

talwai Mar 16, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants


		with pin.tracer.trace('boto.command', service=DEFAULT_SERVICE, span_type=SPAN_TYPE) as span:

		# Original func returns a boto.connection.HPPResponse object

Conversation

raphaelgavache commented Mar 2, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Compatibility

TODOS

Miscellanous notes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

b1-88er Mar 13, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

raphaelgavache commented Mar 2, 2017 •

edited

Loading

b1-88er Mar 13, 2017 •

edited

Loading