Broadside: generate historical jobs server-side via PostgreSQL#4711
Merged
mauriceyap merged 1 commit intomasterfrom Feb 23, 2026
Merged
Broadside: generate historical jobs server-side via PostgreSQL#4711mauriceyap merged 1 commit intomasterfrom
mauriceyap merged 1 commit intomasterfrom
Conversation
Replaces Go-side historical job generation (which looped in Go, built rows, and sent them over the network in small batches) with four server-side INSERT ... SELECT FROM generate_series statements. All data generation now happens inside PostgreSQL, eliminating network transfer of per-row data during setup. For large historical datasets (e.g. 300M jobs) this reduces setup time from many hours to tens of minutes, with the remaining bottleneck being PostgreSQL I/O and index maintenance. Also fixes two runtime bugs found against a real PostgreSQL instance (per-statement parameter binding; explicit ::bytea casts for CASE WHEN expressions), and a shutdown hang caused by pgxpool.Close blocking on internal health-check goroutines after the schema has been torn down. Signed-off-by: Maurice Yap <mauriceyap@hotmail.co.uk>
MustafaI
approved these changes
Feb 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces Go-side historical job generation (which looped in Go, built rows, and sent them over the network in small batches) with four server-side INSERT ... SELECT FROM generate_series statements. All data generation now happens inside PostgreSQL, eliminating network transfer of per-row data during setup.
For large historical datasets (e.g. 300M jobs) this reduces setup time from many hours to tens of minutes, with the remaining bottleneck being PostgreSQL I/O and index maintenance.
Also fixes two runtime bugs found against a real PostgreSQL instance (per-statement parameter binding; explicit ::bytea casts for CASE WHEN expressions), and a shutdown hang caused by pgxpool.Close blocking on internal health-check goroutines after the schema has been torn down.