This PR adds the following search request oriented dashboards to Zoekt (following the insights we learned from the Every Computer Performance Book):

1. p99/p90/p70 search request duration

   • Should be self-evident for why this is useful. This would have been helpful to have handy on a recent customer call.

2. amount of "in-flight" search requests (both aggregated and per instance)

   • This dashboard serves as a proxy for the "workload" that zoekt-webserver is currently experiencing. Note, that there are multiple search/transaction types (interactive, versus batched), but I think treating all transactions as the same is a good first-pass as a workload meter. See this example of how the # of concurrent search request roughly correlates to the CPU usage:
     • 
   • Here is a similar one with rate-of-growth of search request and CPU usage (notice how positivity/negativity corresponds with the swings in the cpu graph
   • 

3. rate of growth of the amount of "in-flight" search requests (both aggregated and per instance)

   • This is dashboard is just the derivative of the above dashboard. This dashboard helps to give an intuition about how "quickly" zoekt-webserver is grinding through its request load. It should hover pretty close to zero under normal circumstances.

See the highlighted part of the screenshot for an example of these dashboards in action.

Test plan

I created the above screenshot using sg start monitoring while connected to dogfood's prometheus instance.