Patch by clutchski · Pull Request #92 · DataDog/dd-trace-py

clutchski · 2016-11-04T04:51:45Z

A (hopefully) complete vision for automatic tracing (originally based on #75).

Goals

one liner install. from ddtrace import monkey; monkey.patch_all()
easy to specify different clusters of the same technology (e.g. 'replica-db' vs 'master-db')
easy to understand
easy to extend

API

patch everything:

# patch everything
from ddtrace import monkey; monkey.patch_all()

import redis
conn = redis.Redis()
conn.get("abc") # now this is traced

patch everything with a custom service.

# patch everything
from ddtrace import monkey, Pin; monkey.patch_all()

import redis
conn = redis.Redis()

# Pin (aka Patch INfo) stores instance specific trace info.
pin = Pin(service="cache-cluster")
pin.onto(conn)
conn.get("abc") # now this is traced with service "cache-cluster"

Patch some things (pro user mode)

import redis
from ddtrace.contrib.redis as patch

r = redis.Redis()
r.get("abc")  # this isn't traced

patch_instance(r, service="cache-cluster")

r.get("abc")  # this is traced

Implementation

Done with a combination of monkeypatching and proxies. Both have their pros and cons. Written in more detail in #75.

Rollout plan

v0.4
- implement all datastores
- rollout ddtrace.monkey; patch all()
- leave old apis in place either as a no-op or backwards compatible) with a deprecation notice
v0.5
- do web middleware
- remove old apis

To Do

it can then be easily shared between sqlite and postgres.

default to postgres, sqlite, ec.

Pin will store per instance tracing info, like service name and tags. It will be used so that we can distinguish services on monkeypatched connections like so.

clutchski · 2016-11-04T04:55:56Z

 from ddtrace.contrib.autopatch import autopatch

+# allow logging
+logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)


this is just a testing file. probably can be deleted.

LeoCavaille · 2016-11-04T09:34:36Z

+    'sqlite3',
+    'psycopg',
+    'redis',
 ]


nit: a-z sort + upper case

LeoCavaille · 2016-11-04T09:36:47Z

+            if raise_errors:
+                raise
+            else:
+                log.debug("couldn't patch %s" % module, exc_info=True)


use logger interpolation log.debug("%s", module)

LeoCavaille · 2016-11-04T09:39:44Z

+import logging
+
+# 3p
+import wrapt


If we use that in the preferred patching path, should we vendor it in our project (like requests does for other libs) could users be using this with a conflicting version?

yes. will log an issue (it's already released tho)

LeoCavaille · 2016-11-04T09:43:54Z

+        tracer information particular to traced objects.
+
+        >>> db = sqlite.connect(":memory:")
+        >>> Pin(service="my-sqlite-service").onto(db)


in your example this is a keyword arg but it's a positional in your constructor

LeoCavaille · 2016-11-04T09:48:24Z

+
+    def onto(self, obj):
+        """ Patch this pin onto the given object. """
+        return setattr(obj, '_datadog_pin', self)


This can raise? Should we handle gracefully.

yes we should

LotharSee · 2016-11-04T23:10:03Z

Global comments about the design:

We need a clean way to patch only one module, or all except one. Multiple interfaces are possible, here. I'm fine with the one from gevent http://www.gevent.org/gevent.monkey.html#gevent.monkey.patch_all
Any usage/any combination of patching ways have to be idempotent (it currently is okay only when using the top-level patch_all, which puts it in _patched_modules — not sure about the individual patch methods)
We should provide a global way of either returning the orignal module, or to unpatch. (use-case: a user could put the entire patching behind a feature flag). Make people feel safer, simplify tests (our tests, but also user tests if they would like to disable tracing during some unittests).
For the former point, we might want to have a unified way of patching the modules (a convention in the way we keep reference to the original module?)
Have our tests patching at the beginning of the run and not during the imports (and be sure it works)

Opinion about middlewares: I'd suggest them to run the patch_all themselves. So that the user only has to import/use our middleware.

- also remove `contact_points` it's too much. could be hundreds - clean up styling

clutchski · 2016-11-04T23:24:37Z

@LotharSee:

for only patching one module you can do patch_modules("redis").
idempotency: individual patch methods will have to test themselves
otherwise, agreed. don't have a plan for tests yet.

works, but just in case.

this function does c-level checks of the type of the argument so it must be a raw psycopg connection passed in. this adds a hook to transparently downgrade.

Psycopg types

Patch cassandra

…llocator (#17664) ## Description This PR fixes a segmentation fault in the memory allocation profiler that occurs when a hook call races with `memalloc` start/stop operations. The issue arises from concurrent access to the saved allocator struct, which could be partially written while being read, resulting in`NULL` function pointers being dereferenced. The key indicator in that case is that `#1 0x0000000000000000` frame -- we are trying to execute a null function pointer. ```` Error UnixSignal: Process terminated with SEGV_MAPERR (SIGSEGV) #0 0x00007ff3c303a8d4 #1 0x0000000000000000 memalloc_alloc (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/profiling/collector/_memalloc.cpp:68) #2 0x00007ff39dcb3b20 memalloc_alloc (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/profiling/collector/_memalloc.cpp:68) #3 0x00007ff39dcb3b20 memalloc_malloc(void*, unsigned long) (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/ddtrace/profiling/collector/_memalloc.cpp:80) #4 0x00007ff3c3087e1b PyUnicode_New #5 0x00007ff3c30889f4 #6 0x00007ff3c3170c84 #7 0x00007ff3c316b931 #8 0x00007ff3c31aaac8 #9 0x00007ff3c31033ac #10 0x00007ff3c310e2a6 PyObject_CallMethodObjArgs #11 0x00007ff3c310e46d #12 0x00007ff3c31a96c2 #13 0x00007ff3c3102fd7 PyObject_Vectorcall #14 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #15 0x00007ff3c323c094 #16 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #17 0x00007ff3c323c094 #18 0x00007ff3c30e997d PyObject_CallOneArg #19 0x00007ff3c306a480 _PyObject_GenericGetAttrWithDict #20 0x00007ff3c30c620d PyObject_GetAttr #21 0x00007ff3c32309e7 _PyEval_EvalFrameDefault #22 0x00007ff3c323c094 #23 0x00007ff3c312880e #24 0x00007ff3c30e917c _PyObject_MakeTpCall #25 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #26 0x00007ff3c323c094 #27 0x00007ff3c312880e #28 0x00007ff3c30e917c _PyObject_MakeTpCall #29 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #30 0x00007ff3c323c094 #31 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #32 0x00007ff3c323c094 #33 0x00007ff3c317d0fd #34 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #35 0x00007ff3c323c094 #36 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #37 0x00007ff3c323c094 #38 0x00007ff3c317d1b5 #39 0x00007ff3c3102fd7 PyObject_Vectorcall #40 0x00007ff3c3232f4a _PyEval_EvalFrameDefault #41 0x00007ff3c3240da5 #42 0x00007ff3c324112d #43 0x00007ff3c3233be1 _PyEval_EvalFrameDefault #44 0x00007ff3c323c094 #45 0x00007ff3c317d1b5 #46 0x00007ff3c3102fd7 PyObject_Vectorcall #47 0x00007ff3c3232f4a _PyEval_EvalFrameDefault #48 0x00007ff3c323c094 #49 0x00007ff3c31033ac #50 0x00007ff3c310358d PyObject_CallFunctionObjArgs #51 0x00007ff3bf7eb91d WraptBoundFunctionWrapper_call (/project/src/wrapt/_wrappers.c:3750) #52 0x00007ff3c3104055 _PyObject_Call #53 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #54 0x00007ff3c323c094 #55 0x00007ff3c317d23c #56 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #57 0x00007ff3c323c094 #58 0x00007ff3c310416f _PyObject_Call #59 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #60 0x00007ff3c3240da5 #61 0x00007ff3c324112d #62 0x00007ff3c3233be1 _PyEval_EvalFrameDefault #63 0x00007ff3c323c094 #64 0x00007ff3c317d1b5 #65 0x00007ff3c3102fd7 PyObject_Vectorcall #66 0x00007ff3c3232f4a _PyEval_EvalFrameDefault #67 0x00007ff3c323c094 #68 0x00007ff3c317d1b5 #69 0x00007ff3c310416f _PyObject_Call #70 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #71 0x00007ff3c323c094 #72 0x00007ff3c317d1b5 #73 0x00007ff3c310416f _PyObject_Call #74 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #75 0x00007ff3c323c094 #76 0x00007ff3c31033ac #77 0x00007ff3c310358d PyObject_CallFunctionObjArgs #78 0x00007ff3bf7eb91d WraptBoundFunctionWrapper_call (/project/src/wrapt/_wrappers.c:3750) #79 0x00007ff3c30e917c _PyObject_MakeTpCall #80 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #81 0x00007ff3c323c094 #82 0x00007ff3c317d518 #83 0x00007ff3c3155963 #84 0x00007ff3c315393d #85 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #86 0x00007ff3c323c094 #87 0x00007ff3c317d0fd #88 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #89 0x00007ff3c323c094 #90 0x00007ff3c317d0fd #91 0x00007ff3c317d518 #92 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #93 0x00007ff3c323c094 #94 0x00007ff3c30e9371 _PyObject_FastCallDictTstate #95 0x00007ff3c30e958d _PyObject_Call_Prepend #96 0x00007ff3c3109150 #97 0x00007ff3c3104055 _PyObject_Call #98 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #99 0x00007ff3c323c094 #100 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #101 0x00007ff3c30e958d _PyObject_Call_Prepend #102 0x00007ff3c3109150 #103 0x00007ff3c30e917c _PyObject_MakeTpCall #104 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #105 0x00007ff3c323c094 #106 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #107 0x00007ff3c30e958d _PyObject_Call_Prepend #108 0x00007ff3c3109150 #109 0x00007ff3c30e917c _PyObject_MakeTpCall #110 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #111 0x00007ff3c323c094 #112 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #113 0x00007ff3c30e958d _PyObject_Call_Prepend #114 0x00007ff3c3109150 #115 0x00007ff3c30e917c _PyObject_MakeTpCall #116 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #117 0x00007ff3c323c094 #118 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #119 0x00007ff3c30e958d _PyObject_Call_Prepend #120 0x00007ff3c3109150 #121 0x00007ff3c30e917c _PyObject_MakeTpCall #122 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #123 0x00007ff3c323c094 #124 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #125 0x00007ff3c30e958d _PyObject_Call_Prepend #126 0x00007ff3c3109150 #127 0x00007ff3c30e917c _PyObject_MakeTpCall #128 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #129 0x00007ff3c323c094 #130 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #131 0x00007ff3c30e958d _PyObject_Call_Prepend #132 0x00007ff3c3109150 #133 0x00007ff3c30e917c _PyObject_MakeTpCall #134 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #135 0x00007ff3c323c094 #136 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #137 0x00007ff3c30e958d _PyObject_Call_Prepend #138 0x00007ff3c3109150 #139 0x00007ff3c30e917c _PyObject_MakeTpCall #140 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #141 0x00007ff3c323c094 #142 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #143 0x00007ff3c30e958d _PyObject_Call_Prepend #144 0x00007ff3c3109150 #145 0x00007ff3c30e917c _PyObject_MakeTpCall #146 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #147 0x00007ff3c323c094 #148 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #149 0x00007ff3c30e958d _PyObject_Call_Prepend #150 0x00007ff3c3109150 #151 0x00007ff3c30e917c _PyObject_MakeTpCall #152 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #153 0x00007ff3c323c094 #154 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #155 0x00007ff3c30e958d _PyObject_Call_Prepend #156 0x00007ff3c3109150 #157 0x00007ff3c30e917c _PyObject_MakeTpCall #158 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #159 0x00007ff3c323c094 #160 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #161 0x00007ff3c30e958d _PyObject_Call_Prepend #162 0x00007ff3c3109150 #163 0x00007ff3c30e917c _PyObject_MakeTpCall #164 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #165 0x00007ff3c323c094 #166 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #167 0x00007ff3c30e958d _PyObject_Call_Prepend #168 0x00007ff3c3109150 #169 0x00007ff3c30e917c _PyObject_MakeTpCall #170 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #171 0x00007ff3c323c094 #172 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #173 0x00007ff3c30e958d _PyObject_Call_Prepend #174 0x00007ff3c3109150 #175 0x00007ff3c30e917c _PyObject_MakeTpCall #176 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #177 0x00007ff3c323c094 #178 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #179 0x00007ff3c30e958d _PyObject_Call_Prepend #180 0x00007ff3c3109150 #181 0x00007ff3c30e917c _PyObject_MakeTpCall #182 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #183 0x00007ff3c323c094 #184 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #185 0x00007ff3c30e958d _PyObject_Call_Prepend #186 0x00007ff3c3109150 #187 0x00007ff3c30e917c _PyObject_MakeTpCall #188 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #189 0x00007ff3c323c094 #190 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #191 0x00007ff3c323c094 #192 0x00007ff3c30e92f1 _PyObject_FastCallDictTstate #193 0x00007ff3c30e958d _PyObject_Call_Prepend #194 0x00007ff3c3109150 #195 0x00007ff3c30e917c _PyObject_MakeTpCall #196 0x00007ff3c32335a2 _PyEval_EvalFrameDefault #197 0x00007ff3c323c094 #198 0x00007ff3c317d0fd #199 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #200 0x00007ff3c323c094 #201 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #202 0x00007ff3c323c094 #203 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #204 0x00007ff3c323c094 #205 0x00007ff3c3233dd3 _PyEval_EvalFrameDefault #206 0x00007ff3c323c094 #207 0x00007ff3c317d23c #208 0x00007ff3c31a7ec5 #209 0x00007ff3c301ac77 #210 0x00007ff3c357c573 ```` The fix implements two key changes. 1. **Hook functions (`memalloc_alloc`, `memalloc_realloc`)**: Snapshot the allocator struct locally before use and guard indirect function calls with `NULL` checks. This prevents crashes if a partially-written struct is observed during a start/stop race. 2. **Start/stop operations (`memalloc_start`, `memalloc_stop`)**: Use local variables and single assignments when publishing the allocator struct to `global_memalloc_ctx.pymem_allocator_obj`. This ensures concurrent hook calls observe either the old or new struct, never a partially-written intermediate state. The real root cause is that `PyMem_GetAllocator` is not documented as atomic, and the struct could be read field-by-field while being written to concurrently. By using local copies and single assignments, we ensure atomicity at the C level and prevent observation of inconsistent state. Co-authored-by: thomas.kowalski <thomas.kowalski@datadoghq.com>

clutchski added 24 commits November 4, 2016 03:32

autopatch: pyscopg experiments

4448480

autopatch: add dbapi interface patching

83f0e21

it can then be easily shared between sqlite and postgres.

plug psycopg and sqlite into autopatch

ccfb340

experiments with redis autopatching.

ed16ee5

autopatch: make autopatching safer.

7958bfc

psycopg: add tests for patched version of conn

089ae4d

psycopg: fix indentaiton

86dcadc

patch/redis: add tests for monkeypatched command

5c59949

patch/redis: add pipeline tracing

2117cb8

patch/redis: implement pipeline execute immediate.

68955a2

patch: lint fixes

4cfc0a2

config: restore original postgres config

c76bbb6

patch: remove uneeded code

6bac15c

autopatch: remove cruft

7d2c233

sqlite: add conn.execute traced method

c27ae43

patch/sqlite: ensure we patch all connect methods

fc2cae3

redis/patch: fix service info command

3d81d48

patch/dbapi: clean up default service names.

2c9cbd4

default to postgres, sqlite, ec.

dbapi: fix module naming

4fc21ab

patch/psycopg: share the connection wrapper

d483d41

patch: high level interface & lock

066ebbd

rake: added up command to start instances.

54e477f

patch: Add the pin object.

1f60a6c

Pin will store per instance tracing info, like service name and tags. It will be used so that we can distinguish services on monkeypatched connections like so.

pin: be ever safer with enabled.

41eb7f1

clutchski commented Nov 4, 2016

View reviewed changes

LeoCavaille reviewed Nov 4, 2016

View reviewed changes

clutchski added 4 commits November 4, 2016 14:47

pin: handle errors gracefully

b611cc6

patch/redis: rewrite old api with new patcher.

51bdaad

redis: clean up nomenclature.

3eae515

cass: add tox test target

305ec11

clutchski added 3 commits November 4, 2016 21:31

cass: refactor tests to allow multiple suites

8f38489

cassandra patch: implement wrapt version

1bd0585

add cassandra to list of autopatched modules

a4e10be

clutchski added 2 commits November 4, 2016 23:16

cass: autopatching

11735fe

cass: use real co-ordinator host

99e282a

- also remove `contact_points` it's too much. could be hundreds - clean up styling

clutchski mentioned this pull request Nov 4, 2016

Celery integration #87

Closed

clutchski added 14 commits November 5, 2016 03:48

flake8 before wait

f31d706

cass: fix flake8 comment

2aeba0a

tests: make wait for services quiet.

0e27e90

pin: add get from none test case

b6b38c1

works, but just in case.

patch/cass: more conservative

dabdedc

pin: add __repr__

35a1b14

patch/psycopg2: handle extensions.register_type

62486b0

this function does c-level checks of the type of the argument so it must be a raw psycopg connection passed in. this adds a hook to transparently downgrade.

remove empty file

1f9f995

add file we need

3cd2f4f

Merge pull request #96 from DataDog/psycopg-types

f0725d3

Psycopg types

cass: fix code review comments

8f6e132

remove unused import

97bf6de

cass: actually fix the import

3f609cd

Merge pull request #93 from DataDog/patchcass

bf283d9

Patch cassandra

clutchski merged commit bf283d9 into master Nov 9, 2016

clutchski deleted the patch branch November 9, 2016 21:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Patch#92

Patch#92
clutchski merged 47 commits into
masterfrom
patch

clutchski commented Nov 4, 2016 •

edited

Loading

Uh oh!

clutchski Nov 4, 2016

Uh oh!

LeoCavaille Nov 4, 2016

Uh oh!

LeoCavaille Nov 4, 2016

Uh oh!

LeoCavaille Nov 4, 2016

Uh oh!

clutchski Nov 4, 2016

Uh oh!

LeoCavaille Nov 4, 2016

Uh oh!

LeoCavaille Nov 4, 2016

Uh oh!

clutchski Nov 4, 2016

Uh oh!

LotharSee commented Nov 4, 2016

Uh oh!

clutchski commented Nov 4, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

clutchski commented Nov 4, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Goals

API

Implementation

Rollout plan

To Do

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LotharSee commented Nov 4, 2016

Uh oh!

clutchski commented Nov 4, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

clutchski commented Nov 4, 2016 •

edited

Loading