Add sharding keys to web_hook_logs_daily
We should do this after &16133 (closed):
I think we should do this afterwards to avoid overlapping the work with the daily partitions.
Implementation Plan
Milestone %18.8 (required stop)
- Add sharding key columns && prepare async index (exclusively on .com) -> !215994 (merged)
- Verify the columns have been added on .com
- Verify the indexes have been created on .com
Milestone %18.9
-
Prepare to shard -> !215236 (merged)
- Index the columns
organization_idproject_idandgroup_id - Create foreign key constraints (not valid) on the columns
organization_idproject_idandgroup_idwith reverse lock order - Create a trigger that assigns one of
organization_idproject_idandgroup_idbased on the parent tableweb_hooks - Create a post migration (only for self managed) to backfill old
web_hook_logs_dailywith a sharding key.- We should not backfill this data (on .com), as the table is too big to do this. Since the table is partitioned and old partitions get deleted eventually, we can just wait until partitions without a sharding key set get dropped.
- Create a multi column not null (not valid) constraint on the sharding key column for new partitions only (so old partitions don't need to be backfilled).
- Set the new columns as the sharding keys
- Index the columns
- Verify the foreign key constraints have been applied on .com
- Validate the foreign key constraints
- Verify all rows on .com have a sharding key column set (there are 14 partitions for the last 14 days - older partitions roll over)
-
(for .com only) Fully shard the table
- validate the sharding key column constraint
Milestone %18.11 (required stop)
Milestone %19.0
-
(for self-managed && JiHu) finalize the backfill
- validate the sharding key constraint
- validate foreign key constraints
Edited by Tomasz Skorupa