Add documentation about generating flame graphs. #12649

eklitzke · 2018-03-08T18:23:05Z

This adds documentation for generating flame graphs (a profiling format I've found to be very useful). It's pretty straightforward: I've added a new file in doc/, updated doc/developer-notes.md, and checked in a patch and example flame graph to contrib.

You can see how it looks when rendered on GitHub here: https://github.com/eklitzke/bitcoin/blob/flamegraphs/doc/flamegraphs.md

The one thing that's kind of weird is that when you look at an actual flame graph file you get hover events and can interact with the graph (for example, here's the same example.svg file I checked in on an actual web page). The way GitHub renders the markdown file (which refers to the example.svg I added to contrib) prevent this from happening, it just gets rendered as a static image. It would be kind of cool if we could host the example file on bitcoincore.org or something.

maflcko · 2018-03-08T19:48:31Z

Concept ACK 🔥, just some notes:

I'd prefer if we didn't check in the svg file (less files, less to maintain). If someone is into flames, they can generate the svg themselfes.
My preference would be to move the patch file into the devtools subfolder, since it seems this is only useful for devs.
Unrelated: You could add an :(exclude) for .patch files in line 19 of ./contrib/devtools/lint-whitespace.sh if you feel like it.

Sjors

Concept ACK, but having trouble getting this to work on macOS. That could be "fixed" with a note saying this doesn't work on macOS, but it would be nice if it did work.

greadelf returns:

 Not an ELF file - it has the wrong magic bytes at the start

file src/bitcoind
src/bitcoind: Mach-O 64-bit executable x86_64

Sjors · 2018-03-08T20:13:30Z

doc/flamegraphs.md

+graph. You can learn more about flame graphs
+[here](http://www.brendangregg.com/flamegraphs.html).
+
+Example flame graph (click to zoom):


When I preview this page on Github, clicking on the image takes me to the image page, rather than zoom. An animated gif might be a better way to illustrate the behavior.

Sjors · 2018-03-08T20:18:18Z

doc/flamegraphs.md

+# Install perf on Fedora.
+sudo dnf install perf
+```
+


# Install on macOS brew install google-perftools

This is a totally different project.

Sjors · 2018-03-08T20:23:30Z

doc/flamegraphs.md

+### Generating Flame Graphs On Linux Using Perf
+
+To generate flame graphs with labeled function names you'll need a build of
+Bitcoin that has debugging symbols enabled (this should be the case by default).


Are you sure? ./configure --enable-debug is off by default.

Sjors · 2018-03-08T20:26:40Z

doc/flamegraphs.md

+
+```bash
+# If this produces output your binary has debugging symbols.
+$ readelf -S src/bitcoind | grep .debug_str


For OSX, use greadelf instead of readelf, which can be installed using brew install binutils

Sjors · 2018-03-08T20:53:00Z

doc/flamegraphs.md

+
+```bash
+# Optional, enable kernel annotations
+sudo sysctl kernel.perf_event_paranoid=-1


On macOS this complains with:

sysctl: unknown oid 'kernel.perf_event_paranoid'

Sjors · 2018-03-08T20:55:54Z

doc/flamegraphs.md

+
+```bash
+# Profile bitcoind for 60 seconds
+perf record -g --call-graph dwarf -F 101 -p $(pidof bitcoind) -- sleep 60


pidof on macOS can be installed with brew install pidof. perf is gperf on macOS, although it doesn't recognize --call-graph.

eklitzke · 2018-03-09T17:34:32Z

Changes in my latest commit:

Generally reword a lot of the flamegraphs.md file for more detail and clarity
Make the Linux-specific stuff more explicit
Remove the checked in example SVG file
Move the patch to devtools

The other change is to link to a copy of the SVG file I am hosting on monad.io (a domain I own). I feel kind of weird about this but it's the only way I could figure out for hosting the example file in a way where people can click it and actually zoom, use mouseover events, etc. I feel like that's a really compelling part of the flamegraph tool, and is important to demonstrate why it's so much cooler than other profiling tools. If someone who has access to bitcoincore.org or another more official resource (maybe the wiki?) could host the file there (it's a small static cacheable file) I'd be happy to change the link.

eklitzke · 2018-03-09T18:00:10Z

I opened this PR on bitcoincore.org to see if I can get the SVG file hosted here.

laanwj · 2018-03-10T12:36:09Z

Thanks for adding documentation on this!

What about adding this to the maintainer tools or docs repository?

This not bound to bitcoin core's release cycle, after all.

sipa · 2018-03-10T17:38:01Z

doc/flamegraphs.md

+
+```bash
+# Install perf on Debian.
+sudo apt-get install linux-perf


Seems on Ubuntu you need linux-tools-generic, linux-cloud-tools-generic, linux-tools-common.

It seems that linux-tools is the old package name, I can update.

eklitzke · 2018-03-11T00:25:27Z

@laanwj I didn't know about those repositories until you pointed them out to me. I'm happy to move the docs if we can still keep a link to them in developer-notes.md. Does that seem like a reasonable compromise?

fanquake · 2018-03-11T10:09:51Z

@eklitzke Yes, that sounds fine.

martinus · 2018-03-13T10:46:25Z

Flamegraphs are awesome, but I have personally switched from using the flamegraph scripts to the KDAB/hotspot profiler, which basically is a flamegraph GUI for perf: https://github.com/KDAB/hotspot

laanwj · 2018-03-19T16:19:33Z

I'm happy to move the docs if we can still keep a link to them in developer-notes.md. Does that seem like a reasonable compromise?

Yes, that'd be the way to do it.

jnewbery · 2018-03-19T19:26:55Z

doc/flamegraphs.md

+
+Flame graphs are a way of visualizing software CPU and heap profiles. Any tool
+that can create stack data in the correct format can be used to generate a flame
+graph. The flame graphs themselves are SVG files that have a small have a small


s/have a small have a small/have a small/

eklitzke · 2018-03-20T08:56:49Z

Apologies for the delay on this, I was asked on IRC if I still plan to move this to the docs repo, and I do intend to. I will update this tomorrow.

@eklitzke

…ith perf 13782b8 docs: add perf section to developer docs (James O'Beirne) 58180b5 tests: add utility to easily profile node performance with perf (James O'Beirne) Pull request description: Adds a context manager to easily (and selectively) profile node performance during functional test execution using `perf`. While writing some tests, I encountered some odd bitcoind slowness. I wrote up a utility (`TestNode.profile_with_perf`) that generates performance diagnostics for a node by running `perf` during the execution of a particular region of test code. `perf` usage is detailed in the excellent (and sadly unmerged) #12649; all due props to @eklitzke. ### Example ```python with node.profile_with_perf("large-msgs"): for i in range(200): node.p2p.send_message(some_large_msg) node.p2p.sync_with_ping() ``` This generates a perf data file in the test node's datadir (`/tmp/testtxmpod0y/node0/node-0-TestName-large-msgs.perf.data`). Running `perf report` generates nice output about where the node spent most of its time while running that part of the test: ```bash $ perf report -i /tmp/testtxmpod0y/node0/node-0-TestName-large-msgs.perf.data --stdio \ | c++filt \ | less # To display the perf.data header info, please use --header/--header-only options. # # # Total Lost Samples: 0 # # Samples: 135 of event 'cycles:pp' # Event count (approx.): 1458205679493582 # # Children Self Command Shared Object Symbol # ........ ........ ............... ................... ........................................................................................................................................................................................................................................................................ # 70.14% 0.00% bitcoin-net bitcoind [.] CNode::ReceiveMsgBytes(char const*, unsigned int, bool&) | ---CNode::ReceiveMsgBytes(char const*, unsigned int, bool&) 70.14% 0.00% bitcoin-net bitcoind [.] CNetMessage::readData(char const*, unsigned int) | ---CNetMessage::readData(char const*, unsigned int) CNode::ReceiveMsgBytes(char const*, unsigned int, bool&) 35.52% 0.00% bitcoin-net bitcoind [.] std::vector<char, zero_after_free_allocator<char> >::_M_fill_insert(__gnu_cxx::__normal_iterator<char*, std::vector<char, zero_after_free_allocator<char> > >, unsigned long, char const&) | ---std::vector<char, zero_after_free_allocator<char> >::_M_fill_insert(__gnu_cxx::__normal_iterator<char*, std::vector<char, zero_after_free_allocator<char> > >, unsigned long, char const&) CNetMessage::readData(char const*, unsigned int) CNode::ReceiveMsgBytes(char const*, unsigned int, bool&) ... ``` Tree-SHA512: 9ac4ceaa88818d5eca00994e8e3c8ad42ae019550d6583972a0a4f7b0c4f61032e3d0c476b4ae58756bc5eb8f8015a19a7fc26c095bd588f31d49a37ed0c6b3e

GChuf · 2019-10-09T12:41:00Z

@eklitzke the links you posted at doc/flamegraphs are dead - can you generate/upload the pic and svg file again?

maflcko · 2019-10-09T13:31:23Z

@GChuf You can generate them yourself. If you are looking for an illustrative example, there are plenty: https://duckduckgo.com/?q=flamegraph&t=ffab&iax=images&ia=images

GChuf · 2019-10-09T14:12:24Z

Just wanted to point out the broken links. Anyone can find examples on the internet - if I had found a bitcoin example i'd already propose to replace the picture.

achow101 · 2019-10-09T14:29:44Z

Here's a flame graph I generated for doing some of my wallet load time stuff: https://achow101.com/bitcoin-master-flame.svg

@eklitzke

…e with perf Summary: docs: add perf section to developer docs (James O'Beirne) add utility to easily profile node performance with perf (James O'Beirne) Pull request description: Adds a context manager to easily (and selectively) profile node performance during functional test execution using `perf`. While writing some tests, I encountered some odd bitcoind slowness. I wrote up a utility (`TestNode.profile_with_perf`) that generates performance diagnostics for a node by running `perf` during the execution of a particular region of test code. `perf` usage is detailed in the excellent (and sadly unmerged) bitcoin/bitcoin#12649; all due props to @eklitzke. ### Example ```python with node.profile_with_perf("large-msgs"): for i in range(200): node.p2p.send_message(some_large_msg) node.p2p.sync_with_ping() ``` This generates a perf data file in the test node's datadir (`/tmp/testtxmpod0y/node0/node-0-TestName-large-msgs.perf.data`). Running `perf report` generates nice output about where the node spent most of its time while running that part of the test: ```bash $ perf report -i /tmp/testtxmpod0y/node0/node-0-TestName-large-msgs.perf.data --stdio \ | c++filt \ | less https://github.com/bitcoin/bitcoin/pull/14519/files --- Backport of Core PR14519 Test Plan: ninja cd test ../../test/functional/abc-cmdline.py --perf --config=config.ini run command to see test report in the output log Reviewers: #bitcoin_abc, Fabien Reviewed By: #bitcoin_abc, Fabien Differential Revision: https://reviews.bitcoinabc.org/D7080

@eklitzke

…mance with perf 13782b8 docs: add perf section to developer docs (James O'Beirne) 58180b5 tests: add utility to easily profile node performance with perf (James O'Beirne) Pull request description: Adds a context manager to easily (and selectively) profile node performance during functional test execution using `perf`. While writing some tests, I encountered some odd bitcoind slowness. I wrote up a utility (`TestNode.profile_with_perf`) that generates performance diagnostics for a node by running `perf` during the execution of a particular region of test code. `perf` usage is detailed in the excellent (and sadly unmerged) bitcoin#12649; all due props to @eklitzke. ### Example ```python with node.profile_with_perf("large-msgs"): for i in range(200): node.p2p.send_message(some_large_msg) node.p2p.sync_with_ping() ``` This generates a perf data file in the test node's datadir (`/tmp/testtxmpod0y/node0/node-0-TestName-large-msgs.perf.data`). Running `perf report` generates nice output about where the node spent most of its time while running that part of the test: ```bash $ perf report -i /tmp/testtxmpod0y/node0/node-0-TestName-large-msgs.perf.data --stdio \ | c++filt \ | less # To display the perf.data header info, please use --header/--header-only options. # # # Total Lost Samples: 0 # # Samples: 135 of event 'cycles:pp' # Event count (approx.): 1458205679493582 # # Children Self Command Shared Object Symbol # ........ ........ ............... ................... ........................................................................................................................................................................................................................................................................ # 70.14% 0.00% bitcoin-net bitcoind [.] CNode::ReceiveMsgBytes(char const*, unsigned int, bool&) | ---CNode::ReceiveMsgBytes(char const*, unsigned int, bool&) 70.14% 0.00% bitcoin-net bitcoind [.] CNetMessage::readData(char const*, unsigned int) | ---CNetMessage::readData(char const*, unsigned int) CNode::ReceiveMsgBytes(char const*, unsigned int, bool&) 35.52% 0.00% bitcoin-net bitcoind [.] std::vector<char, zero_after_free_allocator<char> >::_M_fill_insert(__gnu_cxx::__normal_iterator<char*, std::vector<char, zero_after_free_allocator<char> > >, unsigned long, char const&) | ---std::vector<char, zero_after_free_allocator<char> >::_M_fill_insert(__gnu_cxx::__normal_iterator<char*, std::vector<char, zero_after_free_allocator<char> > >, unsigned long, char const&) CNetMessage::readData(char const*, unsigned int) CNode::ReceiveMsgBytes(char const*, unsigned int, bool&) ... ``` Tree-SHA512: 9ac4ceaa88818d5eca00994e8e3c8ad42ae019550d6583972a0a4f7b0c4f61032e3d0c476b4ae58756bc5eb8f8015a19a7fc26c095bd588f31d49a37ed0c6b3e

Add documentation about generating flame graphs.

9c25c1b

eklitzke force-pushed the flamegraphs branch from 0156031 to 9c25c1b Compare March 8, 2018 18:27

Sjors reviewed Mar 8, 2018

View reviewed changes

meshcollider added the Docs label Mar 9, 2018

Address GitHub review feedback.

517b380

exclude .patch files from whitespace linter

e22e7af

eklitzke mentioned this pull request Mar 9, 2018

Add example flame graph, to be linked from bitcoin docs. bitcoin-core/bitcoincore.org#522

Closed

eklitzke added 2 commits March 9, 2018 10:03

add todo about hosting on bitcoincore.org

3c9ddd3

fix broken link

e3b29ac

sipa reviewed Mar 10, 2018

View reviewed changes

jnewbery reviewed Mar 19, 2018

View reviewed changes

eklitzke closed this Mar 22, 2018

jamesob mentioned this pull request Oct 19, 2018

tests: add utility to easily profile node performance with perf #14519

Merged

bitcoin locked as resolved and limited conversation to collaborators Dec 16, 2021

Add documentation about generating flame graphs. #12649

Add documentation about generating flame graphs. #12649

Uh oh!

Conversation

eklitzke commented Mar 8, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maflcko commented Mar 8, 2018

Uh oh!

Sjors left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eklitzke commented Mar 9, 2018

Uh oh!

eklitzke commented Mar 9, 2018

Uh oh!

laanwj commented Mar 10, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eklitzke commented Mar 11, 2018

Uh oh!

fanquake commented Mar 11, 2018

Uh oh!

martinus commented Mar 13, 2018

Uh oh!

laanwj commented Mar 19, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eklitzke commented Mar 20, 2018

Uh oh!

GChuf commented Oct 9, 2019

Uh oh!

maflcko commented Oct 9, 2019

Uh oh!

GChuf commented Oct 9, 2019

Uh oh!

achow101 commented Oct 9, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

eklitzke commented Mar 8, 2018 •

edited

Loading

laanwj commented Mar 10, 2018 •

edited

Loading