Avoid frequent checkpoints triggered by optimistic insertions#20336
Conversation
47e3341 to
e985f28
Compare
e985f28 to
f8af7bd
Compare
|
Let me explain my modifications to {
"description": "Test PEG Parser + transformer in strict mode",
"skip_compiled": "true",
"on_init": "set allow_parser_override_extension=strict_when_supported;",
"statically_loaded_extensions": [
"core_functions",
"autocomplete"
],
...
}As you can see, in this configuration,
static void LoadInternal(ExtensionLoader &loader) {
TableFunction auto_complete_fun("sql_auto_complete", {LogicalType::VARCHAR}, SQLAutoCompleteFunction,
SQLAutoCompleteBind, SQLAutoCompleteInit);
...
// LOAD `autocomplete` will add it to `parser_extensions`.
auto &config = DBConfig::GetConfig(loader.GetDatabaseInstance());
config.parser_extensions.push_back(PEGParserExtension());
}
void Parser::ParseQuery(const string &query) {
...
string parser_override_option = StringUtil::Lower(options.parser_override_setting);
...
{
if (options.extensions) {
for (auto &ext : *options.extensions) {
...
if (StringUtil::CIEquals(parser_override_option, "strict_when_supported")) {
...
bool is_supported = false;
switch (statement->type) {
case StatementType::CALL_STATEMENT:
...
is_supported = true;
break;
...
}
if (is_supported) {
// An extension exists, and parsing the CALL command failed!
ThrowParserOverrideError(result);
}
}
}
...
}
...
}
...
} |
Mytherin
left a comment
There was a problem hiding this comment.
Thanks!
I think this is fine. We used to rely on CHECKPOINT in order to persist optimistically written data - but since #13372 we can write the block pointers to the WAL as well. Estimating to zero does not seem right though. The size of the entries could be estimated to e.g. row_group_count * sizeof(PersistentColumnData) * types.size().
| Connection con(db); | ||
|
|
||
| // Configuration to lower the checkpoint_threshold | ||
| REQUIRE_NO_FAIL(con.Query("PRAGMA disable_checkpoint_on_shutdown;")); |
There was a problem hiding this comment.
This should be able to be rewritten to a SQL logic test - can we do that? We strongly prefer SQL logic tests over C++ tests.
There was a problem hiding this comment.
Thanks for your comment! I have rewritten the test case to a SQL logic test, see test/sql/storage/optimistic_write/optimistic_write_not_trigger_checkpoint.test for details.
By the way, I've added an estimate of the data block pointer size for optimistic insertion:
data_size = row_group_count * (sizeof(PersistentRowGroupData) +
column_count * (sizeof(PersistentColumnData) + sizeof(DataPointer)));f8af7bd to
fa97b62
Compare
TIME_NS Support + Arrow (duckdb/duckdb#20361) No catalog found Nightly test fix (duckdb/duckdb#20409) Avoid frequent checkpoints triggered by optimistic insertions (duckdb/duckdb#20336)
See issue #20335 for details.