Enable the oximeter producer garbage collection RPW by jgallagher · Pull Request #5764 · oxidecomputer/omicron

jgallagher · 2024-05-14T17:21:49Z

Closes #5284.

@bnaecker I'm not sure we have any tests that would fail if this starts pruning stuff it shouldn't - do we need to do any kind of manual testing? Or land this and see what happens on dogfood?

Closes #5284

bnaecker · 2024-05-14T17:30:33Z

That's a good question. No, I don't think we have any test that will catch this over-pruning things. This feels like this might be difficult to test, since it's dependent on some specified amount of time passing.

But I could imagine a test that advances time through several GC periods, and checks that all the entries in the producer table are still there. It looks like the DataStore::producer_endpoint_delete() method actually does hard-deletes of the producer records, so I think you could check that the record exists and that the creation time is the same. You could also check that modification times have changed too, since the producers should be re-registering themselves periodically.

Happy to help craft that test if you'd like.

jgallagher · 2024-05-14T17:44:06Z

Hmm. I'm worried a test like that will be one of:

flaky (e.g., if we cut too close on waiting for producer reregistrations; I am regularly surprised at how overloaded CI runners can be at times)
slow (e.g., if we give a lot of grace period for producer reregistrations)
overly-contrived (e.g., if we hack up producers to reregister in a way that is more test-friendly than timeouts)

I guess what I really wish here was a full-scale integration test that's allowed to be slow: deploy the software, wait N minutes, then inspect the state and check for properties like this. For now I've been punting on tests like this and relying on manual checks (madrid/london/dogfood). If you think it's worth adding a test and trying to make it not flaky/slow, I can give it a shot? I'm on the fence.

bnaecker · 2024-05-14T20:23:14Z

I agree with that assessment, and also think this test will be flaky or similar. I'm pretty comfortable with the amount of testing we have otherwise, so I think running some manual checks in a development environment seems perfectly fine.

jgallagher · 2024-05-15T16:08:15Z

I spot-checked this on london, and it looks like what I expect. After waiting a few minutes after RSS, all the services and sled-agents have reregistered themselves (time_modified is newer than time_created):

root@[fd00:1122:3344:103::3]:32221/omicron> select * from metric_producer;
                   id                  |         time_created          |         time_modified         |    kind    |          ip           | port  | interval | base_route |             oximeter_id
---------------------------------------+-------------------------------+-------------------------------+------------+-----------------------+-------+----------+------------+---------------------------------------
  569e7ecd-7198-4733-bd45-91b03a7990e3 | 2024-05-15 15:35:51.919705+00 | 2024-05-15 15:40:52.133596+00 | service    | fd00:1122:3344:101::5 | 65313 |       10 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  590d1289-9a21-462d-8ed0-02eacafc923c | 2024-05-15 15:35:56.787517+00 | 2024-05-15 15:40:57.001052+00 | service    | fd00:1122:3344:103::2 |  8001 |        1 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  6382aa81-14dc-4eae-a352-268fc3b2943a | 2024-05-15 15:37:02.524028+00 | 2024-05-15 15:39:32.623232+00 | service    | fd00:1122:3344:102::1 |  8001 |        1 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  79838fa6-9ca1-4db3-8c16-37db8f5743be | 2024-05-15 15:36:44.249984+00 | 2024-05-15 15:39:14.581225+00 | service    | fd00:1122:3344:103::5 | 63229 |       10 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  8e2c38a9-fba5-4e9a-85da-a5212b426974 | 2024-05-15 15:36:41.67943+00  | 2024-05-15 15:39:11.810341+00 | sled_agent | fd00:1122:3344:103::1 | 48824 |       30 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  9afba6d0-dc25-41ac-b7ad-de4b63833227 | 2024-05-15 15:36:57.128855+00 | 2024-05-15 15:39:27.253713+00 | service    | fd00:1122:3344:102::2 |  4677 |        1 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  abba0f0c-4778-423e-9e8a-968b91054a61 | 2024-05-15 15:37:15.547859+00 | 2024-05-15 15:39:45.64491+00  | sled_agent | fd00:1122:3344:102::1 | 50188 |       30 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  af6b7320-8f79-4eee-9776-5f2cf4309f63 | 2024-05-15 15:36:08.834575+00 | 2024-05-15 15:41:09.048687+00 | service    | fd00:1122:3344:103::1 |  8001 |        1 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  bae48501-9ac3-405b-b207-0886258477a8 | 2024-05-15 15:36:16.479591+00 | 2024-05-15 15:41:16.66569+00  | sled_agent | fd00:1122:3344:101::1 | 40246 |       30 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  c34ea8da-814f-4ca7-bcce-874c3e2c3384 | 2024-05-15 15:36:10.397988+00 | 2024-05-15 15:41:10.604281+00 | service    | fd00:1122:3344:101::1 |  8001 |        1 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  d16ed58d-7252-4601-bdd7-78f1d51faca5 | 2024-05-15 15:37:07.016021+00 | 2024-05-15 15:39:37.134968+00 | service    | fd00:1122:3344:102::2 |  8001 |        1 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  d8cd2f00-9b5f-486e-a06e-3a9f152a277f | 2024-05-15 15:36:25.279401+00 | 2024-05-15 15:38:55.39873+00  | service    | fd00:1122:3344:103::2 |  4677 |        1 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  e7404ac9-963d-47c2-8331-05bc0bb42c15 | 2024-05-15 15:36:44.875175+00 | 2024-05-15 15:39:15.001095+00 | service    | fd00:1122:3344:102::5 | 56422 |       10 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
(13 rows)

I then manually restarted mg-ddm in one of the switch zones; it comes up with a new producer ID every time it starts, so we see a 14th row (the first row in this output):

root@[fd00:1122:3344:103::3]:32221/omicron> select * from metric_producer order by time_created desc;
                   id                  |         time_created          |         time_modified         |    kind    |          ip           | port  | interval | base_route |             oximeter_id
---------------------------------------+-------------------------------+-------------------------------+------------+-----------------------+-------+----------+------------+---------------------------------------
  39601512-3ac1-4a50-8390-d99f86da5870 | 2024-05-15 15:41:26.034456+00 | 2024-05-15 15:41:26.034456+00 | service    | fd00:1122:3344:103::2 |  8001 |        1 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  abba0f0c-4778-423e-9e8a-968b91054a61 | 2024-05-15 15:37:15.547859+00 | 2024-05-15 15:39:45.64491+00  | sled_agent | fd00:1122:3344:102::1 | 50188 |       30 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  d16ed58d-7252-4601-bdd7-78f1d51faca5 | 2024-05-15 15:37:07.016021+00 | 2024-05-15 15:39:37.134968+00 | service    | fd00:1122:3344:102::2 |  8001 |        1 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  6382aa81-14dc-4eae-a352-268fc3b2943a | 2024-05-15 15:37:02.524028+00 | 2024-05-15 15:39:32.623232+00 | service    | fd00:1122:3344:102::1 |  8001 |        1 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  9afba6d0-dc25-41ac-b7ad-de4b63833227 | 2024-05-15 15:36:57.128855+00 | 2024-05-15 15:39:27.253713+00 | service    | fd00:1122:3344:102::2 |  4677 |        1 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  e7404ac9-963d-47c2-8331-05bc0bb42c15 | 2024-05-15 15:36:44.875175+00 | 2024-05-15 15:39:15.001095+00 | service    | fd00:1122:3344:102::5 | 56422 |       10 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  79838fa6-9ca1-4db3-8c16-37db8f5743be | 2024-05-15 15:36:44.249984+00 | 2024-05-15 15:39:14.581225+00 | service    | fd00:1122:3344:103::5 | 63229 |       10 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  8e2c38a9-fba5-4e9a-85da-a5212b426974 | 2024-05-15 15:36:41.67943+00  | 2024-05-15 15:39:11.810341+00 | sled_agent | fd00:1122:3344:103::1 | 48824 |       30 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  d8cd2f00-9b5f-486e-a06e-3a9f152a277f | 2024-05-15 15:36:25.279401+00 | 2024-05-15 15:41:25.493889+00 | service    | fd00:1122:3344:103::2 |  4677 |        1 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  bae48501-9ac3-405b-b207-0886258477a8 | 2024-05-15 15:36:16.479591+00 | 2024-05-15 15:41:16.66569+00  | sled_agent | fd00:1122:3344:101::1 | 40246 |       30 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  c34ea8da-814f-4ca7-bcce-874c3e2c3384 | 2024-05-15 15:36:10.397988+00 | 2024-05-15 15:41:10.604281+00 | service    | fd00:1122:3344:101::1 |  8001 |        1 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  af6b7320-8f79-4eee-9776-5f2cf4309f63 | 2024-05-15 15:36:08.834575+00 | 2024-05-15 15:41:09.048687+00 | service    | fd00:1122:3344:103::1 |  8001 |        1 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  590d1289-9a21-462d-8ed0-02eacafc923c | 2024-05-15 15:35:56.787517+00 | 2024-05-15 15:40:57.001052+00 | service    | fd00:1122:3344:103::2 |  8001 |        1 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  569e7ecd-7198-4733-bd45-91b03a7990e3 | 2024-05-15 15:35:51.919705+00 | 2024-05-15 15:40:52.133596+00 | service    | fd00:1122:3344:101::5 | 65313 |       10 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
(14 rows)

We kept 14 rows for about 10 minutes, but the old mg-ddm stopped getting time_modified bumps as expected (first row in this output):

root@[fd00:1122:3344:103::3]:32221/omicron> select * from metric_producer order by time_modified asc;
                   id                  |         time_created          |         time_modified         |    kind    |          ip           | port  | interval | base_route |             oximeter_id
---------------------------------------+-------------------------------+-------------------------------+------------+-----------------------+-------+----------+------------+---------------------------------------
  590d1289-9a21-462d-8ed0-02eacafc923c | 2024-05-15 15:35:56.787517+00 | 2024-05-15 15:40:57.001052+00 | service    | fd00:1122:3344:103::2 |  8001 |        1 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  d8cd2f00-9b5f-486e-a06e-3a9f152a277f | 2024-05-15 15:36:25.279401+00 | 2024-05-15 15:48:55.767322+00 | service    | fd00:1122:3344:103::2 |  4677 |        1 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  39601512-3ac1-4a50-8390-d99f86da5870 | 2024-05-15 15:41:26.034456+00 | 2024-05-15 15:48:56.308056+00 | service    | fd00:1122:3344:103::2 |  8001 |        1 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  8e2c38a9-fba5-4e9a-85da-a5212b426974 | 2024-05-15 15:36:41.67943+00  | 2024-05-15 15:49:12.207262+00 | sled_agent | fd00:1122:3344:103::1 | 48824 |       30 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  79838fa6-9ca1-4db3-8c16-37db8f5743be | 2024-05-15 15:36:44.249984+00 | 2024-05-15 15:49:15.072756+00 | service    | fd00:1122:3344:103::5 | 63229 |       10 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  e7404ac9-963d-47c2-8331-05bc0bb42c15 | 2024-05-15 15:36:44.875175+00 | 2024-05-15 15:49:15.427638+00 | service    | fd00:1122:3344:102::5 | 56422 |       10 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  9afba6d0-dc25-41ac-b7ad-de4b63833227 | 2024-05-15 15:36:57.128855+00 | 2024-05-15 15:49:27.656794+00 | service    | fd00:1122:3344:102::2 |  4677 |        1 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  6382aa81-14dc-4eae-a352-268fc3b2943a | 2024-05-15 15:37:02.524028+00 | 2024-05-15 15:49:33.019745+00 | service    | fd00:1122:3344:102::1 |  8001 |        1 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  d16ed58d-7252-4601-bdd7-78f1d51faca5 | 2024-05-15 15:37:07.016021+00 | 2024-05-15 15:49:37.500623+00 | service    | fd00:1122:3344:102::2 |  8001 |        1 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  abba0f0c-4778-423e-9e8a-968b91054a61 | 2024-05-15 15:37:15.547859+00 | 2024-05-15 15:49:46.038157+00 | sled_agent | fd00:1122:3344:102::1 | 50188 |       30 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  569e7ecd-7198-4733-bd45-91b03a7990e3 | 2024-05-15 15:35:51.919705+00 | 2024-05-15 15:50:52.505713+00 | service    | fd00:1122:3344:101::5 | 65313 |       10 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  af6b7320-8f79-4eee-9776-5f2cf4309f63 | 2024-05-15 15:36:08.834575+00 | 2024-05-15 15:51:09.410991+00 | service    | fd00:1122:3344:103::1 |  8001 |        1 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  c34ea8da-814f-4ca7-bcce-874c3e2c3384 | 2024-05-15 15:36:10.397988+00 | 2024-05-15 15:51:10.963595+00 | service    | fd00:1122:3344:101::1 |  8001 |        1 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  bae48501-9ac3-405b-b207-0886258477a8 | 2024-05-15 15:36:16.479591+00 | 2024-05-15 15:51:17.04606+00  | sled_agent | fd00:1122:3344:101::1 | 40246 |       30 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
(14 rows)

After the 10 minute grace period passed, the GC task pruned that stale mg-ddm producer:

root@[fd00:1122:3344:103::3]:32221/omicron> select * from metric_producer order by time_modified asc;
                   id                  |         time_created          |         time_modified         |    kind    |          ip           | port  | interval | base_route |             oximeter_id
---------------------------------------+-------------------------------+-------------------------------+------------+-----------------------+-------+----------+------------+---------------------------------------
  af6b7320-8f79-4eee-9776-5f2cf4309f63 | 2024-05-15 15:36:08.834575+00 | 2024-05-15 15:51:09.410991+00 | service    | fd00:1122:3344:103::1 |  8001 |        1 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  c34ea8da-814f-4ca7-bcce-874c3e2c3384 | 2024-05-15 15:36:10.397988+00 | 2024-05-15 15:51:10.963595+00 | service    | fd00:1122:3344:101::1 |  8001 |        1 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  bae48501-9ac3-405b-b207-0886258477a8 | 2024-05-15 15:36:16.479591+00 | 2024-05-15 15:51:17.04606+00  | sled_agent | fd00:1122:3344:101::1 | 40246 |       30 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  d8cd2f00-9b5f-486e-a06e-3a9f152a277f | 2024-05-15 15:36:25.279401+00 | 2024-05-15 15:51:25.858331+00 | service    | fd00:1122:3344:103::2 |  4677 |        1 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  39601512-3ac1-4a50-8390-d99f86da5870 | 2024-05-15 15:41:26.034456+00 | 2024-05-15 15:51:26.397449+00 | service    | fd00:1122:3344:103::2 |  8001 |        1 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  8e2c38a9-fba5-4e9a-85da-a5212b426974 | 2024-05-15 15:36:41.67943+00  | 2024-05-15 15:51:42.3077+00   | sled_agent | fd00:1122:3344:103::1 | 48824 |       30 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  79838fa6-9ca1-4db3-8c16-37db8f5743be | 2024-05-15 15:36:44.249984+00 | 2024-05-15 15:51:45.191341+00 | service    | fd00:1122:3344:103::5 | 63229 |       10 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  e7404ac9-963d-47c2-8331-05bc0bb42c15 | 2024-05-15 15:36:44.875175+00 | 2024-05-15 15:51:45.554128+00 | service    | fd00:1122:3344:102::5 | 56422 |       10 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  9afba6d0-dc25-41ac-b7ad-de4b63833227 | 2024-05-15 15:36:57.128855+00 | 2024-05-15 15:51:57.749377+00 | service    | fd00:1122:3344:102::2 |  4677 |        1 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  6382aa81-14dc-4eae-a352-268fc3b2943a | 2024-05-15 15:37:02.524028+00 | 2024-05-15 15:52:03.109649+00 | service    | fd00:1122:3344:102::1 |  8001 |        1 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  d16ed58d-7252-4601-bdd7-78f1d51faca5 | 2024-05-15 15:37:07.016021+00 | 2024-05-15 15:52:07.592502+00 | service    | fd00:1122:3344:102::2 |  8001 |        1 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  abba0f0c-4778-423e-9e8a-968b91054a61 | 2024-05-15 15:37:15.547859+00 | 2024-05-15 15:52:16.133657+00 | sled_agent | fd00:1122:3344:102::1 | 50188 |       30 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  569e7ecd-7198-4733-bd45-91b03a7990e3 | 2024-05-15 15:35:51.919705+00 | 2024-05-15 15:53:22.597532+00 | service    | fd00:1122:3344:101::5 | 65313 |       10 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
(13 rows)

Finally, I created 10 instances and waited a couple minutes to confirm they also reregistered themselves periodically (note that time_since_reregister is ~2m30s lower than age for all instances):

root@[fd00:1122:3344:103::3]:32221/omicron> select now() - time_created as age, now() - time_modified as time_since_register,* from metric_producer order by time_created asc;
        age       | time_since_register |                  id                  |         time_created          |         time_modified         |    kind    |           ip            | port  | interval | base_route |             oximeter_id
------------------+---------------------+--------------------------------------+-------------------------------+-------------------------------+------------+-------------------------+-------+----------+------------+---------------------------------------
  00:30:58.474624 | 00:00:57.313337     | 569e7ecd-7198-4733-bd45-91b03a7990e3 | 2024-05-15 15:35:51.919705+00 | 2024-05-15 16:05:53.080992+00 | service    | fd00:1122:3344:101::5   | 65313 |       10 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  00:30:41.559754 | 00:00:40.427864     | af6b7320-8f79-4eee-9776-5f2cf4309f63 | 2024-05-15 15:36:08.834575+00 | 2024-05-15 16:06:09.966465+00 | service    | fd00:1122:3344:103::1   |  8001 |        1 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  00:30:39.996341 | 00:00:38.88854      | c34ea8da-814f-4ca7-bcce-874c3e2c3384 | 2024-05-15 15:36:10.397988+00 | 2024-05-15 16:06:11.505789+00 | service    | fd00:1122:3344:101::1   |  8001 |        1 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  00:30:33.914738 | 00:00:32.771058     | bae48501-9ac3-405b-b207-0886258477a8 | 2024-05-15 15:36:16.479591+00 | 2024-05-15 16:06:17.623271+00 | sled_agent | fd00:1122:3344:101::1   | 40246 |       30 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  00:30:25.114928 | 00:00:23.984285     | d8cd2f00-9b5f-486e-a06e-3a9f152a277f | 2024-05-15 15:36:25.279401+00 | 2024-05-15 16:06:26.410044+00 | service    | fd00:1122:3344:103::2   |  4677 |        1 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  00:30:08.714899 | 00:00:05.665786     | 8e2c38a9-fba5-4e9a-85da-a5212b426974 | 2024-05-15 15:36:41.67943+00  | 2024-05-15 16:06:44.728543+00 | sled_agent | fd00:1122:3344:103::1   | 48824 |       30 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  00:30:06.144345 | 00:00:04.587362     | 79838fa6-9ca1-4db3-8c16-37db8f5743be | 2024-05-15 15:36:44.249984+00 | 2024-05-15 16:06:45.806967+00 | service    | fd00:1122:3344:103::5   | 63229 |       10 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  00:30:05.519154 | 00:00:04.196873     | e7404ac9-963d-47c2-8331-05bc0bb42c15 | 2024-05-15 15:36:44.875175+00 | 2024-05-15 16:06:46.197456+00 | service    | fd00:1122:3344:102::5   | 56422 |       10 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  00:29:53.265474 | 00:02:22.177131     | 9afba6d0-dc25-41ac-b7ad-de4b63833227 | 2024-05-15 15:36:57.128855+00 | 2024-05-15 16:04:28.217198+00 | service    | fd00:1122:3344:102::2   |  4677 |        1 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  00:29:47.870301 | 00:02:16.827484     | 6382aa81-14dc-4eae-a352-268fc3b2943a | 2024-05-15 15:37:02.524028+00 | 2024-05-15 16:04:33.566845+00 | service    | fd00:1122:3344:102::1   |  8001 |        1 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  00:29:43.378308 | 00:02:12.344562     | d16ed58d-7252-4601-bdd7-78f1d51faca5 | 2024-05-15 15:37:07.016021+00 | 2024-05-15 16:04:38.049767+00 | service    | fd00:1122:3344:102::2   |  8001 |        1 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  00:29:34.84647  | 00:02:03.379465     | abba0f0c-4778-423e-9e8a-968b91054a61 | 2024-05-15 15:37:15.547859+00 | 2024-05-15 16:04:47.014864+00 | sled_agent | fd00:1122:3344:102::1   | 50188 |       30 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  00:25:24.359873 | 00:00:23.448937     | 39601512-3ac1-4a50-8390-d99f86da5870 | 2024-05-15 15:41:26.034456+00 | 2024-05-15 16:06:26.945392+00 | service    | fd00:1122:3344:103::2   |  8001 |        1 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  00:03:52.9432   | 00:01:22.820344     | fb8c0dd7-14b7-416c-b0c0-95e7e75f68ce | 2024-05-15 16:02:57.451129+00 | 2024-05-15 16:05:27.573985+00 | instance   | fd00:1122:3344:101::1:0 | 57993 |       30 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  00:03:45.892009 | 00:01:15.797199     | 73464a03-4660-4f66-9083-f38bb1909ded | 2024-05-15 16:03:04.50232+00  | 2024-05-15 16:05:34.59713+00  | instance   | fd00:1122:3344:103::1:0 | 38805 |       30 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  00:03:38.264695 | 00:01:08.157459     | 78e971ad-fc5e-4c8b-b353-09b42240fffa | 2024-05-15 16:03:12.129634+00 | 2024-05-15 16:05:42.23687+00  | instance   | fd00:1122:3344:103::1:1 | 63732 |       30 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  00:03:28.987468 | 00:00:58.891029     | 06f9aa98-41cb-4690-bb69-e6d5a4e4aa0e | 2024-05-15 16:03:21.406861+00 | 2024-05-15 16:05:51.5033+00   | instance   | fd00:1122:3344:102::1:0 | 59361 |       30 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  00:03:21.54341  | 00:00:51.44462      | 3ae960ac-5b01-4bc9-8169-ad846a0dbcd5 | 2024-05-15 16:03:28.850919+00 | 2024-05-15 16:05:58.949709+00 | instance   | fd00:1122:3344:101::1:1 | 52071 |       30 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  00:03:12.901233 | 00:00:42.807754     | 60ab8c41-3225-4fee-81c3-d8d29599838a | 2024-05-15 16:03:37.493096+00 | 2024-05-15 16:06:07.586575+00 | instance   | fd00:1122:3344:101::1:2 | 43328 |       30 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  00:03:04.040039 | 00:00:33.913269     | fde2cf65-c182-4212-a731-d055727a767d | 2024-05-15 16:03:46.35429+00  | 2024-05-15 16:06:16.48106+00  | instance   | fd00:1122:3344:103::1:2 | 44100 |       30 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  00:02:55.352813 | 00:00:25.255938     | b8ab32e3-c79e-4e4a-8d03-2717af44a97e | 2024-05-15 16:03:55.041516+00 | 2024-05-15 16:06:25.138391+00 | instance   | fd00:1122:3344:103::1:3 | 56203 |       30 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  00:02:47.192634 | 00:00:17.092704     | ffc9d15b-8102-4c10-a4a2-a3a1a3d57116 | 2024-05-15 16:04:03.201695+00 | 2024-05-15 16:06:33.301625+00 | instance   | fd00:1122:3344:103::1:4 | 55491 |       30 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
  00:02:38.788737 | 00:00:08.692849     | 742cb1f9-900a-434c-bbef-81d21f5775c6 | 2024-05-15 16:04:11.605592+00 | 2024-05-15 16:06:41.70148+00  | instance   | fd00:1122:3344:103::1:5 | 54110 |       30 | /          | 27e53a2b-0679-4b28-b53f-54c405dee582
(23 rows)

Enable the oximeter producer garbage collection RPW

6d2e68b

Closes #5284

jgallagher requested a review from bnaecker May 14, 2024 17:21

omdb: redact timestamps with variable subsecond precision

2a86b3a

bnaecker approved these changes May 14, 2024

View reviewed changes

jgallagher merged commit 26aba45 into main May 15, 2024

jgallagher deleted the john/prune-stale-oximeter-producers branch May 15, 2024 16:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable the oximeter producer garbage collection RPW#5764

Enable the oximeter producer garbage collection RPW#5764
jgallagher merged 2 commits into
mainfrom
john/prune-stale-oximeter-producers

jgallagher commented May 14, 2024

Uh oh!

bnaecker commented May 14, 2024

Uh oh!

jgallagher commented May 14, 2024

Uh oh!

bnaecker commented May 14, 2024

Uh oh!

jgallagher commented May 15, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

jgallagher commented May 14, 2024

Uh oh!

bnaecker commented May 14, 2024

Uh oh!

jgallagher commented May 14, 2024

Uh oh!

bnaecker commented May 14, 2024

Uh oh!

jgallagher commented May 15, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants