Skip to content

[dnm] sql,kv,storage: add AS OF SYSTEM TIME ALL#78540

Draft
jordanlewis wants to merge 3 commits intocockroachdb:masterfrom
jordanlewis:as-of-system-time-all
Draft

[dnm] sql,kv,storage: add AS OF SYSTEM TIME ALL#78540
jordanlewis wants to merge 3 commits intocockroachdb:masterfrom
jordanlewis:as-of-system-time-all

Conversation

@jordanlewis
Copy link
Copy Markdown
Member

First two commits are #78539.
Closes #19749 (though this is still a draft, up for discussion)

This commit adds support for an "all versions scan" style from SQL down
to KV. It permits using the `AS OF SYSTEM TIME ALL` modifier to instruct
the system to return all MVCC versions of the scanned data, rather than
just the most recent under the read timestamp.

Release note: add AS OF SYSTEM TIME ALL, a modifier that gives all MVCC
versions on disk for the scanned data.

This is still a work in progress. Here is a checklist of things that we'll need to do before we can consider merging:

  • scanner: support reverse scans
  • scanner: more exhaustive tests of scans, for cases with intents or newer versions we'd like to not see, etc
  • sql: plumb all versions flag into lookup join, index join, zigzag join, inverted joiner
  • sql: teach the optimizer that all versions scans don't uphold uniqueness. Is this possible? It's important - if there's any place where we assume uniqueness, we need to disable that because we could see more than 1 MVCC version for a single key.
  • sql: add more tests, especially for the cases that can come up from the above point about lost uniqueness.

Previously, the argument list to row.Fetcher.Init was too long. Now, it
takes an arg struct to reduce this pain. This is a pure refactor.

Release note: None
Previously, this function had way too many arguments. Replace them with
an argument struct. This is a pure refactor.

Release note: None
@cockroach-teamcity
Copy link
Copy Markdown
Member

This change is Reviewable

This commit adds support for an "all versions scan" style from SQL down
to KV. It permits using the `AS OF SYSTEM TIME ALL` modifier to instruct
the system to return all MVCC versions of the scanned data, rather than
just the most recent under the read timestamp.

Release note: add AS OF SYSTEM TIME ALL, a modifier that gives all MVCC
versions on disk for the scanned data.
@jordanlewis jordanlewis force-pushed the as-of-system-time-all branch from 627fca3 to 2c66296 Compare March 26, 2022 00:00
@jordanlewis
Copy link
Copy Markdown
Member Author

Here's a simple demo:

demo@127.0.0.1:26257/defaultdb> create table a (a primary key, b) as select 1,2;
CREATE TABLE AS
demo@127.0.0.1:26257/defaultdb> update a set b=3 where a= 1;
UPDATE 1
demo@127.0.0.1:26257/defaultdb> select *, crdb_internal_mvcc_timestamp from a AS OF SYSTEM TIME ALL;
  a | b |  crdb_internal_mvcc_timestamp
----+---+---------------------------------
  1 | 3 | 1648252887709306374.0000000000
  1 | 2 | 1648252885046448846.0000000000
(2 rows)

@jordanlewis jordanlewis changed the title sql,kv,storage: add AS OF SYSTEM TIME ALL [dnm] sql,kv,storage: add AS OF SYSTEM TIME ALL Mar 26, 2022
@michae2
Copy link
Copy Markdown
Collaborator

michae2 commented Feb 16, 2024

This is cool!

@drewdeally
Copy link
Copy Markdown

@jordanlewis what is the status of this enhancement? there a few compelling use cases for this feature.

@jordanlewis
Copy link
Copy Markdown
Member Author

jordanlewis commented Feb 17, 2024 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

kv/sql: ability to query all versions of keys/rows

4 participants