chore(wal): non-WAL to WAL, WAL to non-WAL table conversions by glasstiger · Pull Request #2947 · questdb/questdb

glasstiger · 2023-01-24T18:21:02Z

non-WAL to WAL table conversion
WAL to non-WAL table conversion

Syntax:

ALTER TABLE <tablename>` SET TYPE WAL;
ALTER TABLE <tablename>` SET TYPE BYPASS WAL;

Alter statement only creates a _convert file in the table directory.
The actual conversion happens on restart when the _convert file is detected.
The file is 1 byte long.

0: convert to non-WAL
1: convert to WAL

If the table is already the requested type no conversion takes place.
_convert file is removed after conversion.

core/src/main/java/io/questdb/cairo/TableNameRegistryFileStore.java

core/src/main/java/io/questdb/cairo/TableConverter.java

core/src/test/java/io/questdb/griffin/wal/WalTableFailureTest.java

core/src/test/java/io/questdb/griffin/AlterTableSetTypeRestartTest.java

core/src/main/java/io/questdb/cairo/TableConverter.java

core/src/main/java/io/questdb/cairo/TableWriter.java

core/src/main/java/io/questdb/cairo/CairoEngine.java

core/src/main/java/io/questdb/cairo/TableNameRegistryFileStore.java

core/src/test/java/io/questdb/griffin/AlterTableSetTypeRestartTest.java

core/src/main/java/io/questdb/cairo/CairoEngine.java

core/src/main/java/io/questdb/cairo/TableWriter.java

ideoma · 2023-01-27T10:16:03Z

Can we add a test which coverts to WAL and back several times before restart and checks that last command wins?
Also that if WAL table is suspended, conversion to non-WAL still works.

glasstiger · 2023-01-27T11:05:18Z

Can we add a test which coverts to WAL and back several times before restart and checks that last command wins?

Have this already.
AlterTableSetTypeTest.testConvertRandom()
Overwrites the _convert file 30 times and checks the content each time.

Also that if WAL table is suspended, conversion to non-WAL still works.

This will not work as it is now.
will try to move the code to create the _convert file from TableWriter to SqlCompiler, as you suggested in another comment.

ideoma · 2023-01-27T17:33:33Z

[PR Coverage check]

😍 pass : 224 / 275 (81.45%)

file detail

	path	covered line	new line	coverage
🔵	io/questdb/cairo/CairoException.java	0	2	00.00%
🔵	io/questdb/ServerMain.java	1	7	14.29%
🔵	io/questdb/cairo/TableNameRegistryFileStore.java	40	52	76.92%
🔵	io/questdb/cairo/wal/seq/TableSequencerAPI.java	8	10	80.00%
🔵	io/questdb/cairo/TableConverter.java	78	96	81.25%
🔵	io/questdb/cairo/CairoEngine.java	13	16	81.25%
🔵	io/questdb/griffin/SqlCompiler.java	26	32	81.25%
🔵	io/questdb/cairo/TableUtils.java	29	31	93.55%
🔵	io/questdb/Bootstrap.java	9	9	100.00%
🔵	io/questdb/cairo/TableNameRegistry.java	2	2	100.00%
🔵	io/questdb/PropServerConfiguration.java	5	5	100.00%
🔵	io/questdb/cairo/DefaultCairoConfiguration.java	1	1	100.00%
🔵	io/questdb/cairo/TableNameRegistryRW.java	1	1	100.00%
🔵	io/questdb/griffin/CompiledQueryImpl.java	2	2	100.00%
🔵	io/questdb/PropertyKey.java	1	1	100.00%
🔵	io/questdb/cairo/TableNameRegistryRO.java	1	1	100.00%
🔵	io/questdb/griffin/SqlKeywords.java	7	7	100.00%

chore(wal): non-WAL to WAL, WAL to non-WAL table conversions

2324cd5

glasstiger marked this pull request as draft January 24, 2023 18:21

Merge branch 'master' into ia_wal_conv

ff183e9

glasstiger mentioned this pull request Jan 24, 2023

chore(wal): non-WAL to WAL, WAL to non-WAL table conversions #2937

Closed

glasstiger added 5 commits January 25, 2023 20:19

chore(wal): non-WAL to WAL, WAL to non-WAL table conversions

30edc65

Merge remote-tracking branch 'origin/master' into ia_wal_conv

e3a6aed

chore(wal): non-WAL to WAL, WAL to non-WAL table conversions

b402bba

chore(wal): non-WAL to WAL, WAL to non-WAL table conversions

bce3518

chore(wal): non-WAL to WAL, WAL to non-WAL table conversions

52ae607

glasstiger marked this pull request as ready for review January 25, 2023 21:07

marregui reviewed Jan 26, 2023

View reviewed changes

small changes to address PR comments

509c5d5

ideoma requested changes Jan 26, 2023

View reviewed changes

glasstiger and others added 6 commits January 26, 2023 14:15

table conversion enable/disable flag

277ba3c

adjust TableNameRegistry with the list of converted tables

4d2ea43

Merge branch 'master' into ia_wal_conv

4be9cfe

use write() instead of append()

cb3c7cb

no type conversion allowed on non-partitioned table

babda40

remove WAL persistence when converted to non-WAL

b905162

ideoma reviewed Jan 27, 2023

View reviewed changes

core/src/main/java/io/questdb/cairo/CairoEngine.java Show resolved Hide resolved

core/src/main/java/io/questdb/cairo/TableWriter.java Outdated Show resolved Hide resolved

ideoma and others added 2 commits January 27, 2023 11:07

few more tests added

b600e76

create _convert file in SqlCompiler

927dc44

ideoma previously approved these changes Jan 27, 2023

View reviewed changes

new test: convert suspended WAL table to non-WAL

b6c6a36

glasstiger dismissed ideoma’s stale review via b6c6a36 January 27, 2023 14:43

do not start ServerMain's apply job in tests

ae2e2ce

ideoma approved these changes Jan 27, 2023

View reviewed changes

ideoma merged commit 618ee1e into master Jan 27, 2023

ideoma deleted the ia_wal_conv branch January 27, 2023 18:06

ideoma mentioned this pull request Jan 27, 2023

feat(wal): Write Ahead Log (WAL) table storage model #2951

Merged

marregui pushed a commit that referenced this pull request Jan 28, 2023

chore(wal): non-WAL to WAL, WAL to non-WAL table conversions (#2947)

bfc0275

ideoma mentioned this pull request Feb 6, 2023

fix(wal): Reset transaction number on non-WAL to WAL table conversion #2968

Merged

Conversation

glasstiger commented Jan 24, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ideoma commented Jan 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

glasstiger commented Jan 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ideoma commented Jan 27, 2023

[PR Coverage check]

file detail

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

glasstiger commented Jan 24, 2023 •

edited

Loading

ideoma commented Jan 27, 2023 •

edited

Loading

glasstiger commented Jan 27, 2023 •

edited

Loading