Merged
Conversation
Indexing a partitioned table was problematic,
since it could fail and wasn't really idempotent.
This also fixes potential invalid indices when
migrating a non-partitioned table,
and in general should keep all indices tidy.
From the docs:
For partitioned tables, we need to be careful with how we create the indexes.
If the overall function gets interrupted (the process gets killed, etc.),
we can be left with an invalid index.
We can minimize this possibility by doing the following:
- Create an index for each partition, if it does not exist already.
- This is done outside a transaction, since it may happen concurrently.
- Then we open a transaction for the final steps, which are all very fast/metadata-only.
- A transaction here ensures that we only have the 'parent' index in a successful, completed state.
- Create the 'parent' index 'ONLY ON' the parent table.
- This is a very fast operation.
- Attach all the partition indexes to the parent index. This is metadata-only so also very fast.
- At this point, the parent index should be valid.
- These steps mean that, at any point, the process can be interrupted and resumed,
without losing progress:
- The concurrent index creation for the partitions can fail, and result in an invalid index;
but the next call to update the schema will drop invalid indexes for the table.
Note that successfully created, but unattached, indexes for a partition are valid.
- The parent index creation, and attaching partitions to it, are atomic.
c6b0fb5 to
374b567
Compare
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #968 +/- ##
==========================================
- Coverage 97.62% 97.58% -0.04%
==========================================
Files 490 490
Lines 31098 31117 +19
==========================================
+ Hits 30358 30365 +7
- Misses 740 752 +12 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Indexing a partitioned table was problematic,
since it could fail and wasn't really idempotent.
This also fixes potential invalid indices when
migrating a non-partitioned table,
and in general should keep all indices tidy.
From the docs:
For partitioned tables, we need to be careful with how we create the indexes. If the overall function gets interrupted (the process gets killed, etc.), we can be left with an invalid index.
We can minimize this possibility by doing the following: