add option 'block_size' to test configs by c-herrewijn · Pull Request #18347 · duckdb/duckdb

c-herrewijn · 2025-07-21T14:52:45Z

This PR adds option block_size to TestConfiguration; and removes the default BLOCK_ALLOC_SIZE from the Makefile.

Since duckdb doesn't have to be recompiled to use different default block size values, there is no need for this flag to be kept in the Makefile or CmakeList.txt

Usage

option block_size can be used to set a default block size for all tests ran with this configuration

{
  "description": "Tests with smallest possible block size",
  "block_size": "16384",
  "skip_compiled": "true"
}

Tests

I ran the following tests to see if it works:

all existing tests pass with the test config
- ./build/release/test/unittest --test-config test/configs/small_block_size.json
setting an illegal value (e.g. "block_size": "1234" raises an Error)
I added the following test (not committed); this test would normally fail (since the default block size is 262144). With the demo test config, however, it succeeds:
- ./build/release/test/unittest test/sql/tmp.test --test-config test/configs/small_block_size.json

# name: test/sql/tmp.test
# group: [tmp]

statement ok
ATTACH '__TEST_DIR__/tmp_db.duckdb' AS db1

query I
select block_size from pragma_database_size() where database_name = 'db1';
----
16384

… Makefile

taniabogatsch

Looks good! I like having release tests in Main.yml (if they don't take too long), and relassert in the nightlies. 👍

c-herrewijn · 2025-07-22T14:40:26Z

I added the following configurations, based on this comment: #18236 (comment)

in memory db with block size 16kB:
- test/configs/small_block_size.json
persistent db with block size 16kB:
- test/configs/latest_storage_block_size_16kB.json

Note that for the configuration with a persistent db, I had to exclude a number of tests that assume the default setting (in-memory db with name 'memory'). A next iteration might be to make that more transparant.

test/configs/latest_storage.json

taniabogatsch · 2025-07-22T14:43:38Z

.github/workflows/Main.yml

+    - name: test/configs/default_storage_block_size_16kB.json
+      if: (success() || failure()) && steps.build.conclusion == 'success'
+      shell: bash
+      run: |
+        ./build/release/test/unittest --test-config test/configs/default_storage_block_size_16kB.json


I think we can remove this for now - it is mostly the same as test/configs/small_block_size.json, no?

The difference is in-memory vs persistent db, but maybe not so relevant in this context indeed

taniabogatsch · 2025-07-22T14:45:00Z

test/configs/latest_storage_block_size_16kB.json

@@ -0,0 +1,35 @@
+{
+  "description": "Run everything with the latest storage on databases with a default block size of 16kB.",
+  "on_init": "ATTACH '__TEST_DIR__/{BASE_TEST_NAME}block_size.db' AS __test__config__latest_storage_block_size_16kB (BLOCK_SIZE 16384, STORAGE_VERSION 'latest'); SET storage_compatibility_version='latest';",


I think if we use "block_size": "16384",, then we can shorten this to:

"ATTACH '__TEST_DIR__/{BASE_TEST_NAME}block_size.db' AS __test__config__latest_storage_block_size_16kB (STORAGE_VERSION 'latest');

taniabogatsch · 2025-07-22T14:45:35Z

test/configs/latest_storage_block_size_16kB.json

@@ -0,0 +1,35 @@
+{
+  "description": "Run everything with the latest storage on databases with a default block size of 16kB.",
+  "on_init": "ATTACH '__TEST_DIR__/{BASE_TEST_NAME}block_size.db' AS __test__config__latest_storage_block_size_16kB (BLOCK_SIZE 16384, STORAGE_VERSION 'latest'); SET storage_compatibility_version='latest';",


"ATTACH '__TEST_DIR__/{BASE_TEST_NAME}block_size.db'

Should we give the files themselves a very long identifier, to also avoid file conflicts? 😅

test/configs/latest_storage.json

taniabogatsch · 2025-07-22T14:48:07Z

Note that for the configuration with a persistent db, I had to exclude a number of tests that assume the default setting (in-memory db with name 'memory'). A next iteration might be to make that more transparant

Let's remove the persistent DB one, like I already noted in my comment :) we can re-add it while looking at the tests later.

…stent db

c-herrewijn · 2025-07-24T09:28:24Z

@Mytherin
Can you merge this one?
I just discussed with Tania and Carlo, the errors in NightlyTests are unrelated.

Mytherin · 2025-07-24T09:50:00Z

Thanks!

Mytherin · 2025-07-24T10:04:48Z

Can we perhaps also do this for block verification? See https://github.com/search?q=repo%3Aduckdb%2Fduckdb%20DUCKDB_BLOCK_VERIFICATION&type=code

add option 'block_size' to test configs (duckdb/duckdb#18347) bump httpfs (duckdb/duckdb#18380)

add option 'block_size' to test configs (duckdb/duckdb#18347) bump httpfs (duckdb/duckdb#18380) Co-authored-by: krlmlr <krlmlr@users.noreply.github.com>

add option 'block_size' to test configs; remove BLOCK_ALLOC_SIZE from…

d5166bc

… Makefile

c-herrewijn requested a review from taniabogatsch July 21, 2025 14:52

c-herrewijn marked this pull request as draft July 21, 2025 15:24

move block_alloc_size tests to Main.yml

5a92e45

taniabogatsch approved these changes Jul 22, 2025

View reviewed changes

c-herrewijn added 2 commits July 22, 2025 15:52

added test configs with persistent db with block size of 16kB

42746c4

add block_size attr to skip tests that have 'require block_size 262144'

2276d7c

c-herrewijn marked this pull request as ready for review July 22, 2025 14:31

carlopi reviewed Jul 22, 2025

View reviewed changes

test/configs/latest_storage.json Outdated Show resolved Hide resolved

taniabogatsch reviewed Jul 22, 2025

View reviewed changes

c-herrewijn added 2 commits July 22, 2025 17:04

fix accidental attribute swap

006572e

remove redundant test case

4e01cbb

duckdb-draftbot marked this pull request as draft July 22, 2025 16:18

Mytherin marked this pull request as ready for review July 23, 2025 08:20

skip extension tests requiring in-memory db in test config with persi…

c12597f

…stent db

duckdb-draftbot marked this pull request as draft July 23, 2025 14:13

c-herrewijn marked this pull request as ready for review July 23, 2025 14:33

Mytherin merged commit 61cf5c7 into duckdb:v1.3-ossivalis Jul 24, 2025
66 of 70 checks passed

github-actions bot pushed a commit to duckdb/duckdb-r that referenced this pull request Jul 24, 2025

vendor: Update vendored sources to duckdb/duckdb@61cf5c7

9e154dc

add option 'block_size' to test configs (duckdb/duckdb#18347) bump httpfs (duckdb/duckdb#18380)

github-actions bot mentioned this pull request Jul 24, 2025

vendor: Update vendored sources to duckdb/duckdb@61cf5c71bd6a2aac2e862ceeb5010d3cf396e709 duckdb/duckdb-r#1350

Merged

c-herrewijn deleted the block_alloc_size_unittester branch August 4, 2025 10:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add option 'block_size' to test configs#18347

add option 'block_size' to test configs#18347
Mytherin merged 7 commits intoduckdb:v1.3-ossivalisfrom
c-herrewijn:block_alloc_size_unittester

c-herrewijn commented Jul 21, 2025 •

edited

Loading

Uh oh!

taniabogatsch left a comment

Uh oh!

c-herrewijn commented Jul 22, 2025 •

edited

Loading

Uh oh!

Uh oh!

taniabogatsch Jul 22, 2025 •

edited

Loading

Uh oh!

c-herrewijn Jul 22, 2025

Uh oh!

taniabogatsch Jul 22, 2025

Uh oh!

taniabogatsch Jul 22, 2025

Uh oh!

Uh oh!

taniabogatsch commented Jul 22, 2025

Uh oh!

c-herrewijn commented Jul 24, 2025

Uh oh!

Uh oh!

Mytherin commented Jul 24, 2025

Uh oh!

Mytherin commented Jul 24, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

c-herrewijn commented Jul 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Usage

Tests

Uh oh!

taniabogatsch left a comment

Choose a reason for hiding this comment

Uh oh!

c-herrewijn commented Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

taniabogatsch Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

c-herrewijn Jul 22, 2025

Choose a reason for hiding this comment

Uh oh!

taniabogatsch Jul 22, 2025

Choose a reason for hiding this comment

Uh oh!

taniabogatsch Jul 22, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

taniabogatsch commented Jul 22, 2025

Uh oh!

c-herrewijn commented Jul 24, 2025

Uh oh!

Uh oh!

Mytherin commented Jul 24, 2025

Uh oh!

Mytherin commented Jul 24, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

c-herrewijn commented Jul 21, 2025 •

edited

Loading

c-herrewijn commented Jul 22, 2025 •

edited

Loading

taniabogatsch Jul 22, 2025 •

edited

Loading