Skip to content

Perf improvements#81

Merged
sevagh merged 5 commits into
sevagh:masterfrom
gnuvince:perf-improvements
Feb 22, 2018
Merged

Perf improvements#81
sevagh merged 5 commits into
sevagh:masterfrom
gnuvince:perf-improvements

Conversation

@gnuvince

@gnuvince gnuvince commented Feb 20, 2018

Copy link
Copy Markdown
Contributor

This PR improves the performance of pq.

  • Two commits move some computations outside of loops. Both commits affect the current API of pq.
  • One commit introduces concurrency to pq. One thread reads from the consumer and decodes the Protobuf; another thread encodes to JSON and writes to stdout. The two threads communicate via an unbounded channel. This also required changing the public API.

In the tests I've performed the performance increase varies between 50% and 100%.

~/src/pq$ ./pq-release [elided] < records.127MiB | pv -blat >/dev/null
 900k 0:00:21 [41.2k/s]

~/src/pq$ ./pq-threaded [elided] < records.127MiB | pv -blat >/dev/null
 900k 0:00:10 [85.9k/s]

Previously, pq would create the fully-qualified msgtype (i.e., by
preprending a period) inside the decode loop. This commit moves that
operation outside the loop.

API change: PqDecoder::new expects its msgtype to be fully qualified.
This commit creates a CustomFormatter once, outside the decoding loop,
and passes a reference to that formatter to PqDecoder::decode_message.

API change: the PqDecoder::decode_message method accepts a
CustomerFormatter as its third argument rather than a bool.
This is in preparation for making pq multi-threaded; the trait `Read`
is not `Send` and so with the previous declaration (read: &'a mut
Read) ByteConsumer was _not_ Send and could not be moved into a
closure, i.e., with thread::spawn.

This commit modifies the API of ByteConsumer by making it take
ownership of a value of type T where T implements the Read trait.
This commit adds concurrency to the decoding part of
CommandRunner::decode_or_convert.  A pair of threads -- a producer and
a consumer -- are spawned; the producer decodes Protobuf values and
sends the decoded Value over a channel to the consumer who encodes the
values to JSON and outputs them to stdout.
@sevagh

sevagh commented Feb 20, 2018

Copy link
Copy Markdown
Owner

Something about adding threading makes me vaguely uneasy... But the numbers don't lie. When I get some time I'll look it over and merge it.

@sevagh

sevagh commented Feb 22, 2018

Copy link
Copy Markdown
Owner

Tests pass so lgtm.

@sevagh sevagh merged commit 80c7791 into sevagh:master Feb 22, 2018
@sevagh sevagh mentioned this pull request Dec 12, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants