NOTE: i'm still working on some tests to confirm improvements

i spun up docker-cluster-query with 4 query pods.
all mt read/write processes and q0 (query node 0) run master
q1 runs master + stats (#1922)
q2 runs master + stats + nudge fix (#1923)
q3 runs master + stats + nudge fix + /getdata pointslicepool (this branch)

I ran mt-fakemetrics backfill --kafka-mdm-addr localhost:9092 --offset 48h --period 10s --speedup 100 --mpo 1000 and then

cat ./even-load.sh
#!/bin/bash

q0() {
echo 'GET http://localhost:6060/render?target=some.id.of.a.metric.123*&from=-48h' | vegeta attack -rate 25 | vegeta report
}
q1() {
echo 'GET http://localhost:6061/render?target=some.id.of.a.metric.123*&from=-48h' | vegeta attack -rate 25 | vegeta report
}
q2() {
echo 'GET http://localhost:6062/render?target=some.id.of.a.metric.123*&from=-48h' | vegeta attack -rate 25 | vegeta report
}
q3() {
echo 'GET http://localhost:6063/render?target=some.id.of.a.metric.123*&from=-48h' | vegeta attack -rate 25 | vegeta report
}

q0 &
q1 &
q2 &
q3

I ran it for more than an hour.
clearly q3 wins in terms of memory usage. and q2 also does better than q1 and q0.
note the candidate hit stats of the pointslicepool lower left, showing the effect of these PR's in action.

dashboard is here

here are the 3 different PSP behaviors
first one suffers from the nudging bug, which the 2nd one fixes. but we only do half the gets on the pool as puts because upon decoding we always allocate fresh.
only for q3 do we pull the same rate in and out of the pool. with the occassional miss which i presume is due to GC

because we use custom code to use the slice pool. perhaps it can be done cleaner with the msgp extension system but i didn't want to spend too much time on it.