Skip to content

Instrument Prometheus with OpenTracing#2554

Merged
juliusv merged 3 commits intoprometheus:masterfrom
tomwilkie:opentracing
May 2, 2017
Merged

Instrument Prometheus with OpenTracing#2554
juliusv merged 3 commits intoprometheus:masterfrom
tomwilkie:opentracing

Conversation

@tomwilkie
Copy link
Member

@tomwilkie tomwilkie commented Apr 1, 2017

screen shot 2017-04-01 at 16 45 07

2 change sets:

This has beed mooted as a solution to #1315.

/cc @brian-brazil you seemed keen on this - it wouldn't be too hard to extend this for a /rpcz style endpoint that doesn't vendor the whole world. WDYT?

@fabxc
Copy link
Contributor

fabxc commented Apr 3, 2017

That's a whole lot of dependencies again. Do you know how this comes together? Seems like bad package architecture somewhere in general as I see logging libraries and an Kafka library by Shopify in there.
A go-metrics meta lib thingy does itch me a bit as well.

@brian-brazil
Copy link
Contributor

You missed the conversations on IRC, just consider the first two commits. Tom's working on the rest.

@tomwilkie
Copy link
Member Author

@fabxc:

I don't expect the last two changes will ever go upstream, as they are too much code IMO.

I'm working on a minimal, in-process tracer implementation (using many fewer dependencies) here: weaveworks-experiments/loki#7

@fabxc
Copy link
Contributor

fabxc commented Apr 3, 2017

Thanks for clarifying. Sounds good.

@tomwilkie
Copy link
Member Author

I have removed the zipkin vendoring & updated common (here prometheus/common#79)

Whilst as it stands this change isn't particularly useful, it could be merged as is. Or we could wait for the in-process tracer, which I'm going to continue work on now but is probably a week or so away from being ready for review.

@brian-brazil
Copy link
Contributor

I'd propose we review as-is.

@tomwilkie
Copy link
Member Author

I'd propose we review as-is.

Whoops missed that. Let me backout the embedded tracer stuff I just pushed.

@tomwilkie tomwilkie changed the title [WIP] Instrument Prometheus with OpenTracing Instrument Prometheus with OpenTracing Apr 8, 2017
@tomwilkie
Copy link
Member Author

@brian-brazil I think this should be good to go, updated now prometheus/common#79 is in.

@brian-brazil
Copy link
Contributor

👍

@fabxc do you also want to review?

Copy link
Member

@grobie grobie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we dropping the returned context?
If one is returned, I assume it has been modified/populated with values in some way relevant subsequently?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed - if we want to add any children to this span, it needs to be done with the returned context. However, I didn't see anything in this function that took a context - the diff shows the original passed in context was ignored. So when something like seriesForLabelMatchers takes a context, we should pass it the returned one.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, fine with me to leave it out for now then as long as we remember setting it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will try! It'll be pretty easy to spot when spans are not parented properly.

@tomwilkie
Copy link
Member Author

Have rebased.

@juliusv
Copy link
Member

juliusv commented May 2, 2017

👍 Thanks!

@juliusv juliusv merged commit 4d9b917 into prometheus:master May 2, 2017
@juliusv juliusv deleted the opentracing branch May 2, 2017 23:49
@brancz
Copy link
Member

brancz commented May 3, 2017

This might be my lack of experience with OpenTracing, but wouldn't we have to implement some kind of a receiver/backend that the data is sent to?

@redbaron
Copy link
Contributor

redbaron commented May 3, 2017

@brancz , AFAIK Zipkin is an existing opensource backend to receive traces

@brancz
Copy link
Member

brancz commented May 3, 2017

Yes, but as far as I'm aware that would require us to explicitly configure the ZipKin instance the data should be sent to, but I'm not able to find where this configuration is done in this PR, I'm only seeing the instrumenting code. Does the config maybe default to something and further configuration will be a follow-up?

@brian-brazil
Copy link
Contributor

This is using the default NOOP backend. There's future work planned.

@brancz
Copy link
Member

brancz commented May 3, 2017

I see thanks for clarifying @brian-brazil !

@tomwilkie
Copy link
Member Author

This might be my lack of experience with OpenTracing, but wouldn't we have to implement some kind of a receiver/backend that the data is sent to?

As @brian-brazil said, this is setup to use the noop datasource by default, but adding your own is about 3 lines of code + a recompile away. We didn't include it here as it brings it 100k+ loc in dependancies.

I'm working on a much more minimal, in-process tracer that should be applicable. You can follow along at weaveworks-experiments/loki#7. Hopefully this will do what we need, leave the door open for integration with "proper" distributed tracing, and not vendor the world.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants