feat: reduce build chatter for partitioned tables#214
Merged
Conversation
06bb4c6 to
4577186
Compare
60b6d79 to
c03e327
Compare
When CREATE INDEX targets a partitioned table, Postgres calls tp_build() once per partition. Previously this emitted "started", config, and "completed" NOTICEs for every partition -- producing hundreds of lines for tables with many partitions. Add a ProcessUtility hook that detects CREATE INDEX ... USING bm25 and wraps the entire operation in a build-progress tracker. The tracker: - Shows "started" and config NOTICEs only for the first partition - Accumulates doc counts and avg_length across all partitions - Emits a single "completed" summary with "across N partitions" Non-partitioned tables are unaffected (same output as before). Also simplify "completed" messages to not repeat text_config/k1/b parameters that are already shown in the "Using ..." lines above.
9552405 to
4d08c7f
Compare
tjgreen42
added a commit
that referenced
this pull request
Feb 25, 2026
## Summary - Fix a TOCTOU race condition in `tp_leader_process_buffers()` that can silently lose documents during parallel index builds. The leader collected ready buffers *before* checking `workers_done`, allowing a window where a worker marks its final buffer READY and increments `workers_done` between the two reads — causing the leader to exit without processing the final buffer. - Fix a latent deadlock where `shared->nworkers` was set to the *requested* worker count but never updated to the *actually launched* count. If Postgres launched fewer workers, `workers_done` could never reach `nworkers`. - Update expected test output for the build-completed NOTICE format change from #214. ### The race ``` Leader: Worker (finishing): 1. Mark final buffer READY 2. SpinLock → workers_done++ 1. Collect buffers → num_ready=0 (missed the READY buffer) 2. Check all_done → true 3. num_ready(0) > 0 → false → skip 4. while(!all_done) → EXIT *** FINAL BUFFER NEVER PROCESSED *** ``` ### The fix Check `workers_done` *first* (under SpinLock, providing the memory barrier), *then* collect buffers. Workers set buffer READY before the SpinLock acquire around `workers_done++`, so after the leader's SpinLock release, all final buffers are guaranteed visible. The loop now exits only when `all_done && num_ready == 0`. ### Impact On MS-MARCO (8.8M passages), this race non-deterministically lost ~4M documents (~45% of the corpus). One CI run indexed only 4,834,444 documents while the same data loaded correctly in another run (8,841,770 documents). The root cause was discovered while investigating benchmark validation failures in #239. ## Testing Regression tests pass (47/47). The race is non-deterministic so it cannot be reliably reproduced in unit tests — validation should be done via the MS-MARCO benchmark pipeline which exercises parallel builds at scale.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Before (200-partition table, ~800 lines of output):
After (4 lines):
Approach
ProcessUtility_hookinmod.cdetectsCREATE INDEX ... USING bm25and wraps the build with progress tracking (tp_build_progress_begin/end)emit_log_hookinmod.cintercepts "word is too long" NOTICEs during active builds, counting them for a single aggregated warningtp_build()shows "started" and config NOTICEs only for the first partition, accumulates stats across all partitions, and emits one summary at the endTesting
partitioned_many.outreduced from ~848 lines to ~51 lines