OK...so...it's hard to overstate just HOW MUCH BETTER this makes bootc install with composefs.

I need to inject some precise timings but before a fresh install was measured in minutes, now it's measured in seconds. The "writing files" phase takes about 10s here which is SO SO SO much better than before.

I need to spend some more hands-on time with this, but at the eyeball level this looks awesome! Definitely want to A/B bootc installs with/without to compare, probably a job for next week at this point though.

Also: I'm kinda curious about why this is such a performance increase and I wonder if it's being conflated with c3677f1

This is a very good observation; now because the fsync change came in with the same PR as an API break it's a bit more work to try to do a precise apples-to-apples comparison. I'll look at this though.

My main concern is the O_TMPFILE thing here: we've discussed before that it would be nice to avoid streaming files to disk if we already have that data.... do we know how much of the win is attributed to O_TMPFILE and how much is from not blocking on the write?

But I think we also don't want to load potentially large files into memory either. Now remember, writeback is a thing - the O_TMPFILE is still also just writing to memory (page cache) too but it has the advantage that when we do want to persist, it's ready to go into the filesystem and doesn't need another copy.

Currently I would say that when OCI is used in a sane (split-reproducible) way, there's relatively few objects shared across different layers. I think we don't need to optimize for the already-extant object case now - but if we decided to later, it would make sense as a followup; this isn't conflicting.

In particular: the reason that this issue has been sitting around for so long with nobody paying attention on it is that I kinda assumed that with the ability to parallelize a large number of layers (bootc does layer splitting, right?) the need to parallelize within a single layer has seemed less important...

The https://docs.fedoraproject.org/en-US/bootc/building-from-scratch/ flow results in a single large layer by default right now (and is my main motivation for this), but yes there's some work on a new generic rechunker.

Also: I'm kinda curious about why this is such a performance increase and I wonder if it's being conflated with c3677f1

I had an agent work on this overnight, I didn't do a deep verification but smells right to me (and matches your intuition):

Performance Improvement Summary: composefs-rs PR #208

Assisted-by: OpenCode (Opus 4.5)

Benchmark: bootc install Composefs Import Time

Root Cause of Baseline Slowness

The baseline called fdatasync() after writing each object file. While this ran inside spawn_blocking (so it didn't block the async runtime directly), the thousands of per-object fdatasync calls forced the filesystem to create a massive number of journal commits, causing severe I/O overhead.

Optimization Breakdown

1. syncfs optimization (Allison Karlitskaya)

• Commit: c3677f1 ("repository: change our sync story")
• Change: Replace per-object fdatasync() with a single syncfs() before linking the splitstream
• Why it helps: Instead of thousands of individual journal commits, there's now one bulk sync at the end
• Time: ~29.4s → ~20x faster

2. Parallel object storage (Colin Walters)

• Commit: 5f1cce7 ("oci: Parallelize object storage within tar layers")
• Change: Process multiple large files concurrently within each tar layer using channels and spawn_blocking tasks with semaphore-based concurrency limiting
• Why it helps: Previously files were processed sequentially (one at a time). Now multiple files stream to tmpfiles in parallel, with fs-verity digests computed concurrently
• Time: ~16.7s → ~2x additional, ~35x total

Bug Fix Required

The parallel implementation had a race condition where forked processes could inherit writable file descriptors, causing enable_verity to fail with "File is opened for writing".

Fix: Changed finalize_object_tmpfile() to use enable_verity_maybe_copy() instead of enable_verity_with_retry(). This creates a copy of the file if verity can't be enabled on the original, providing robustness against fd inheritance races.

Visual Summary

┌─────────────────────────────────────────────────────────────────┐
│           Composefs Import Performance Improvement              │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  Baseline                        ████████████████████  580s     │
│  (fdatasync per object)                                         │
│                                                                 │
│  + syncfs optimization           █  29.4s  (20x faster)         │
│  (single bulk sync)                                             │
│                                                                 │
│  + parallel file processing      ▌  16.7s  (35x faster)         │
│  (concurrent I/O within layers)                                 │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

Test Environment

• Benchmark command: bcvk to-disk --composefs-backend --disk-size=10G localhost/bootc target/bench-disk.img
• Image: CentOS Stream 10 bootc with composefs-sealeduki-sdboot variant
• Multiple runs averaged for each configuration

Bug Fix Required

The parallel implementation had a race condition where forked processes could inherit writable file descriptors, causing enable_verity to fail with "File is opened for writing".

Fix: Changed finalize_object_tmpfile() to use enable_verity_maybe_copy() instead of enable_verity_with_retry(). This creates a copy of the file if verity can't be enabled on the original, providing robustness against fd inheritance races.

Glad to see the work we did around getting that right (or at least as right as we can without kernel changes) is paying off!