Skip to content

p2p: Per channel metrics (#3666)#3677

Merged
xla merged 3 commits intotendermint:developfrom
brapse:3666-per-channel-metrics
Jun 5, 2019
Merged

p2p: Per channel metrics (#3666)#3677
xla merged 3 commits intotendermint:developfrom
brapse:3666-per-channel-metrics

Conversation

@brapse
Copy link
Contributor

@brapse brapse commented May 24, 2019

Reactors run in parallel exchanging messages with peers across channels. To understand the behaviour of the system as a whole, it would be helpful that have visibility into the per channel (and thus per reactor) message exchange. With per channel metrics, we can observe when a given reactor has stopped sending/receiving and correlate cpu usage with potential thread saturation.

Cf. #3666

  • Updated all relevant documentation in docs
  • Updated all code comments where relevant
  • Wrote tests
  • Updated CHANGELOG_PENDING.md

@brapse brapse changed the base branch from master to develop May 24, 2019 10:14
@brapse brapse marked this pull request as ready for review May 27, 2019 07:11
@brapse brapse requested review from ebuchman, melekes and xla as code owners May 27, 2019 07:11
@codecov-io
Copy link

Codecov Report

Merging #3677 into develop will increase coverage by 0.13%.
The diff coverage is 71.42%.

@@             Coverage Diff             @@
##           develop    #3677      +/-   ##
===========================================
+ Coverage     63.3%   63.44%   +0.13%     
===========================================
  Files          218      218              
  Lines        18208    18168      -40     
===========================================
- Hits         11527    11526       -1     
+ Misses        5717     5674      -43     
- Partials       964      968       +4
Impacted Files Coverage Δ
p2p/metrics.go 95.23% <100%> (ø) ⬆️
p2p/peer.go 60.89% <66.66%> (+0.35%) ⬆️
privval/signer_validator_endpoint.go 75.55% <0%> (-10%) ⬇️
privval/signer_service_endpoint.go 83.63% <0%> (-5.46%) ⬇️
privval/socket_listeners.go 86.2% <0%> (-3.45%) ⬇️
node/node.go 63.88% <0%> (-0.09%) ⬇️
libs/db/remotedb/grpcdb/server.go 0% <0%> (ø) ⬆️
consensus/reactor.go 72.13% <0%> (+0.59%) ⬆️
blockchain/reactor.go 71.49% <0%> (+0.93%) ⬆️
libs/clist/clist.go 68.18% <0%> (+1.51%) ⬆️
... and 1 more

Copy link
Contributor

@liamsi liamsi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

Copy link
Contributor

@melekes melekes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🍶

Copy link
Contributor

@ebuchman ebuchman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome thanks - should this be considered a breaking change? Ie. will anyones current setup break with this change, or does it just add new/more info ?

@brapse
Copy link
Contributor Author

brapse commented May 28, 2019

Awesome thanks - should this be considered a breaking change? Ie. will anyones current setup break with this change, or does it just add new/more info ?

This should be a non breaking changing. The only concern which might be worth noting is this will create additional metrics and increased data for prometheus server pulling the data in. So long as the peer set is bounded and has low churn, This shouldn't be too big a deal.

@melekes
Copy link
Contributor

melekes commented May 28, 2019

So long as the peer set is bounded and has low churn, This shouldn't be too big a deal.

not the case for seed nodes

@brapse
Copy link
Contributor Author

brapse commented May 28, 2019

Would it be something in the region of:
1000s of peers, 10s of channels?
Which I think is ok to pull and could be queried efficiently with a rule

@xla xla changed the title [p2p] per channel metrics (#3666) p2p: Per channel metrics (#3666) Jun 4, 2019
Copy link
Contributor

@xla xla left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Extra dope

👍 :shipit: 💃

The cardinality concern is valid, but we should be safe.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants