Skip to content

oxfmt: pre-scan for config files makes formatting slow #22250

@marvinhagemeister

Description

@marvinhagemeister

This follow up to #22225 . The merged changes already greatly improve the formatting time.

Mode Time
before #22232 6.1s
after #22232 3.4s

But profiles reveal that formatting is still blocked by config discovery. All formatting threads are idling for ~1s before doing any work.

Image

All format worker threads during the pre-scan phase:

# overview of work of the first "block" before real formatting starts
rayon_core::registry::WorkerThread::wait_until_cold
  → par_bridge::IterParallelProducer::fold_with
    → _pthread_mutex_firstfit_lock_slow
      → _pthread_mutex_firstfit_lock_wait
        → __psynch_mutexwait              ← 96% of samples

par_bridge() holds its internal mutex while blocking on the empty mpsc::Receiver. Since the channel is empty (no files sent yet — pre-scan is still running), all 12 workers pile up waiting for that same mutex. 96% of their time is spent in __psynch_mutexwait.

The root cause is of architectural nature. The current architecture looks like this:

  1. Pre-scan (parallel walk, dir I/O only) → builds scope map
  2. Main walk (parallel) → sends files to channel
  3. Format workers (rayon, par_bridge) → consume channel

It's currently a linear sequence of steps. Phase 2 cannot start until phase 1 is completed and phase 3 cannot start until phase 2 is done. We read the same directories twice in both phase 1 + 2. This is duplicate work that isn't needed in an ideal world.

The real fix architectural: Replace the three phases with a single parallel walk. Since config files cannot affect files higher up in the tree, we know that all matching files in the current directory can be formatted immediately. Something like this:

  1. Thread reads all entries in foo/bar/
  2. Scan entry names for config filenames — no extra syscalls, names come free from readdir
  3. No config found → stream all files in foo/bar/ immediately using parent scope, recurse into subdirs
  4. Config found → load it (one read), use it for all files in foo/bar/, recurse into subdirs with new scope

Metadata

Metadata

Assignees

Labels

Type

Priority

None yet

Effort

None yet

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions