Overview

QuestDB 7.0.1 is the production-ready implementation of our new storage type, which makes data ingestion faster and more reliable. In this release, we introduce a Write-Ahead-Log (WAL) to ingest data. Newly created tables will now be WAL tables by default. Existing tables will use legacy storage until explicitly migrated to a new storage type.

🚀 WAL Storage new features

• Ingestion throughput is 3x higher, as benchmarked on the Time Series Benchmarking Suite with twelve threads.
• Concurrent writing via all interfaces. Issues such as #1461 are now solved
• Foundational groundwork for High Availability: Replication will work with WAL tables only

Benchmark

Below are our latest benchmark results using the Time Series Benchmarking Suite:

• Benchmark: CPU-only
• Hardware: 32CPUs, 64GB RAM

The ingestion rate is measured in rows per second:

How to migrate from existing legacy tables to WAL-tables

To create a WAL-based table, the following syntax should be used (note WAL at the end):

create table ABC(x int, t timestamp) timestamp(t) partition by day WAL;

You can also convert your existing table to WAL (followed by an instance restart). The conversion touches table configuration (not data!) and is fully reversible.

To convert an existing table to WAL:

• alter table ABC set type WAL;
• restart instance

To convert a WAL table to a non-WAL table:

• alter table ABC set type bypass WAL;
• restart instance

What is new

CORE

• make WAL tables default on fresh install by @ideoma in #2987
• add external configuration for lag calculation window size by @bluestreak01 in #3025

SQL

• allow list of timestamps of higher precision than required in alter table attach/detach/drop partition by @marregui in #3002
• extend create table statement syntax to support "IN VOLUME volume-alias" by @marregui in #2931

HTTP

• JSON endpoint returns index of designated timestamp column, which is being used by the Web Console by @bluestreak01 in #2969

UI

• made column type a subtext in column header by @bluestreak01 in #2973
• column header is a little taller by @bluestreak01 in #2973
• new header look and feel by @bluestreak01 in #2973
• new "drag handles" by @bluestreak01 in #2973
• designated timestamp columns is highlighted in green by @bluestreak01 in #2973
• column resize fixed as in when columns get narrower and by that do not fill entire viewport - the virtual cells are re-rendered and subsequent glitches fixed by @bluestreak01 in #2973
• column width is rendered based on first 1000 rows of data by @bluestreak01 in #2973
• column name is using ellipsis when it does not fit in column by @bluestreak01 in #2973
• column resize uses ghost bar to improve resize performance on complicated grids by @bluestreak01 in #2973

What we fixed

CORE

• Table type conversion SQL was not processed via REST API by @glasstiger in #2963
• Reset transaction number on non-WAL to WAL table conversion by @glasstiger in #2968
• fix wal table suspension on a race condition by @ideoma in #3019
• fix cleaning up after dropped tables by @ideoma in #3015

SQL

• fix DISTINCT combined with a column alias by @jerrinot in #2982
• fix order by removal in nested queries inside set operations by @bziobrowski in #2970
• fix error on select distinct with join by @bziobrowski in #2989
• fix stack overflow exception in WITH clause parsing by @bziobrowski in #2965
• make sure that order by is not ignored by row_number() function by @puzpuzpuz in #2999
• non-keyed LT JOIN could return equal timestamps by @jerrinot in #3001
• improve order by handling in optimizer by @bziobrowski in #2988
• fix sample by first/last keyed by non-symbol column by @bziobrowski in #2972
• fix inconvertible types error when updating string column by @puzpuzpuz in #3005
• fix (i)like with single character pattern by @bziobrowski in #3007
• fix group by performance when selected columns have aliases by @puzpuzpuz in #3013
• fix symbol column initialization in latest by cursors by @bziobrowski in #3027

ILP

• do not modify representation of Long.MIN_VALUE sent over ILP by @eugenels in #3011

Full Changelog: 7.0.0...7.0.1