Skip to content

schematelemetrycontroller: shift schedules in time, make tenant-ro#88872

Merged
craig[bot] merged 1 commit intocockroachdb:masterfrom
ajwerner:ajwerner/shift-schedules-based-on-cluster-id
Sep 28, 2022
Merged

schematelemetrycontroller: shift schedules in time, make tenant-ro#88872
craig[bot] merged 1 commit intocockroachdb:masterfrom
ajwerner:ajwerner/shift-schedules-based-on-cluster-id

Conversation

@ajwerner
Copy link
Copy Markdown
Contributor

This commit adds logic to rewrite certain special crontab specs to have the same frequency but a different point in that interval. This is useful to even out the burst of logs we'd surely recieve if all clusters logged at the same time. We don't anticipate that folks will set their own schedules, but for tenants, there is no good reason to let them control the schedules.

Usages of @hourly, @daily, or @Weekly will be spread out accordingly.

The algorithm hashes the cluster ID to seed a random number generator which is used to determine the point in the interval for the schedule to occur.

Fixes #85788

Release note: None

@cockroach-teamcity
Copy link
Copy Markdown
Member

This change is Reviewable

This commit adds logic to rewrite certain special crontab specs to have the
same frequency but a different point in that interval. This is useful to
even out the burst of logs we'd surely recieve if all clusters logged at
the same time. We don't anticipate that folks will set their own schedules,
but for tenants, there is no good reason to let them control the schedules.

Usages of @hourly, @daily, or @Weekly will be spread out accordingly.

The algorithm hashes the cluster ID to seed a random number generator
which is used to determine the point in the interval for the schedule
to occur.

Fixes cockroachdb#85788

Release note: None
@ajwerner ajwerner force-pushed the ajwerner/shift-schedules-based-on-cluster-id branch from e04ce55 to 199506a Compare September 28, 2022 16:41
@ajwerner ajwerner marked this pull request as ready for review September 28, 2022 17:39
@ajwerner ajwerner requested review from a team and postamar September 28, 2022 17:39
81cfbe24-3ea8-11ed-b878-0242ac120002
----
@hourly @daily @weekly
440aaea9-06f5-43c1-94d4-fb2b57601ebd 22 * * * * 22 10 * * * 22 10 * * 5
Copy link
Copy Markdown
Contributor

@chengxiong-ruan chengxiong-ruan Sep 28, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for @weekly why do we distribute on month? shouldn't we distribute on day/hour instead?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://crontab.guru/#22_10_*_*_5

I don't think I distribute on month. The last column IIUC, is day of the week.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

heh, I think I looked at some bad oracle doc...

@ajwerner
Copy link
Copy Markdown
Contributor Author

TFTR!

bors r+

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Sep 28, 2022

Build failed (retrying...):

Copy link
Copy Markdown

@postamar postamar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Is any of this worth documenting? I don't think so but I'll ask anyway.

@ajwerner
Copy link
Copy Markdown
Contributor Author

Is any of this worth documenting?

I'm not particularly eager to. I vote we wait on that.

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Sep 28, 2022

Build succeeded:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

schematelemetry: reduce log burstiness

4 participants