Summary

cron/scheduler.py::tick() currently holds the fcntl/msvcrt LOCK_EX for the full duration of every due job, not just the scheduling decision. A long-running delegation (multi-minute Opus run) starves every 60s ticker for the entire job lifetime, and combined with compute_next_run's grace window this silently drops overdue interval runs instead of catching up. This PR releases the lock as soon as the critical section (select due jobs + bump next_run_at) is done; job execution then runs lock-free.

Fixes #27485.

The bug

# cron/scheduler.py::tick()
fcntl.flock(lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
try:
    due_jobs = get_due_jobs()
    for job in due_jobs:
        advance_next_run(job["id"])  # critical section ends here
    ...
    with ThreadPoolExecutor(...) as pool:
        for job in due_jobs:
            pool.submit(_process_job, job)   # ← lock still held
        # ThreadPoolExecutor.__exit__ blocks until all futures complete
finally:
    fcntl.flock(lock_fd, fcntl.LOCK_UN)       # ← only now

Observed effect (reported on the issue): a 5-minute interval cron had a 68-minute gap between runs because a parallel Opus job was holding the lock. The repro is just "configure a few interval crons, fire a multi-minute delegation, watch subsequent ticks skip with Tick skipped — another instance holds the lock."

The lock only needs to serialize across tick instances long enough that advance_next_run() can bump each job's next_run_at. After that, any concurrent tick that picks up the lock will see those jobs as no-longer-due and exit cleanly — so at-most-once semantics are preserved without the lock spanning execution.

The fix

Split tick() so the fcntl/msvcrt lock wraps only get_due_jobs() and the advance_next_run() loop. The lock is released in a finally immediately after that critical section; job execution, MCP orphan cleanup, and result accounting all run lock-free.

Concretely, the inner block (worker resolution, _process_job, workdir vs parallel partitioning, the ThreadPoolExecutor, the MCP cleanup, and return sum(_results)) was lifted out of the try so it sits after the finally that releases the lock. No behavior change to get_due_jobs, advance_next_run, _process_job, the workdir-vs-parallel split, the HERMES_CRON_MAX_PARALLEL / cron.max_parallel_jobs resolution, or the existing MagicMock-compatible early-return on verbose and not due_jobs.

This is Fix 1 from the issue's proposal. Fix 2 (better grace / catch-up for interval jobs) and Fix 3 (cap max_parallel_jobs default) are deliberately not in scope here — they address separate root causes and would be better as their own PRs.

Test plan

• New regression test TestParallelTick::test_tick_releases_file_lock_before_running_jobs: patches fcntl.flock/msvcrt.locking to log LOCK_UN/LK_UNLCK events alongside run_job calls and asserts the lock is released before any job starts.
• Regression guard: against the pre-fix scheduler, the new test fails with the expected event order — [('run_job', 'long-1'), ('run_job', 'long-2'), ('lock_release',)]. With the fix applied the order flips and the assertion passes.
• Focused suite: uv run --with pytest --with pytest-xdist --with pytest-asyncio python3 -m pytest tests/cron/test_scheduler.py -v — 122 passed.
• Adjacent suite: full tests/cron/ — 348 passed.
• No production behaviour changes outside the lock-scope refactor: _process_job body, workdir vs parallel partitioning, as_completed(timeout=600) per-future handling (#5ed-style protection from 7a7e78a36), and the MCP orphan cleanup are all preserved verbatim.

Related

• Fixes cron: tick lock held for full job duration causes scheduler starvation and missed runs on long-running jobs #27485.
• Complementary to 7a7e78a36 (fix(cron): prevent parallel job result loss on exception) — that commit hardened result collection inside the pool; this PR pulls the pool itself out of the critical section.

Positioning vs #21901 (perf(cron): narrow file lock scope + heartbeat ticker)

Both PRs move job execution outside the fcntl critical section. The differences are scope and execution semantics — picking one over the other is a real call for maintainers, not a pure dedup:

This PR is intentionally the minimal change for the missed-runs symptom in #27485 — same fix-1 outcome, no change to execution semantics or workdir safety, plus the regression test that asserts the lock is released before any run_job() call. Happy to defer to #21901 if the broader execution-model + observability changes are wanted in one go.

Sibling code paths that may need the same fix: cron/jobs.py::compute_next_run grace window and the unbounded cron.max_parallel_jobs default are both called out in the issue as separate contributing factors to missed runs. Intentionally left out of this PR's scope to keep the diff small — happy to widen or follow up in a separate PR if preferred.