Instrument Prometheus with OpenTracing#2554
Conversation
|
That's a whole lot of dependencies again. Do you know how this comes together? Seems like bad package architecture somewhere in general as I see logging libraries and an Kafka library by Shopify in there. |
|
You missed the conversations on IRC, just consider the first two commits. Tom's working on the rest. |
I'm working on a minimal, in-process tracer implementation (using many fewer dependencies) here: weaveworks-experiments/loki#7 |
|
Thanks for clarifying. Sounds good. |
6eb15d3 to
b68179e
Compare
|
I have removed the zipkin vendoring & updated common (here prometheus/common#79) Whilst as it stands this change isn't particularly useful, it could be merged as is. Or we could wait for the in-process tracer, which I'm going to continue work on now but is probably a week or so away from being ready for review. |
|
I'd propose we review as-is. |
Whoops missed that. Let me backout the embedded tracer stuff I just pushed. |
|
@brian-brazil I think this should be good to go, updated now prometheus/common#79 is in. |
|
👍 @fabxc do you also want to review? |
storage/local/storage.go
Outdated
There was a problem hiding this comment.
Why are we dropping the returned context?
If one is returned, I assume it has been modified/populated with values in some way relevant subsequently?
There was a problem hiding this comment.
Indeed - if we want to add any children to this span, it needs to be done with the returned context. However, I didn't see anything in this function that took a context - the diff shows the original passed in context was ignored. So when something like seriesForLabelMatchers takes a context, we should pass it the returned one.
There was a problem hiding this comment.
Okay, fine with me to leave it out for now then as long as we remember setting it.
There was a problem hiding this comment.
I will try! It'll be pretty easy to spot when spans are not parented properly.
|
Have rebased. |
|
👍 Thanks! |
|
This might be my lack of experience with OpenTracing, but wouldn't we have to implement some kind of a receiver/backend that the data is sent to? |
|
@brancz , AFAIK Zipkin is an existing opensource backend to receive traces |
|
Yes, but as far as I'm aware that would require us to explicitly configure the ZipKin instance the data should be sent to, but I'm not able to find where this configuration is done in this PR, I'm only seeing the instrumenting code. Does the config maybe default to something and further configuration will be a follow-up? |
|
This is using the default NOOP backend. There's future work planned. |
|
I see thanks for clarifying @brian-brazil ! |
As @brian-brazil said, this is setup to use the noop datasource by default, but adding your own is about 3 lines of code + a recompile away. We didn't include it here as it brings it 100k+ loc in dependancies. I'm working on a much more minimal, in-process tracer that should be applicable. You can follow along at weaveworks-experiments/loki#7. Hopefully this will do what we need, leave the door open for integration with "proper" distributed tracing, and not vendor the world. |
2 change sets:
This has beed mooted as a solution to #1315.
/cc @brian-brazil you seemed keen on this - it wouldn't be too hard to extend this for a
/rpczstyle endpoint that doesn't vendor the whole world. WDYT?