Follow-up: cross-host stale detection, a cross-language survey, and the security model

What @keller-mark's deployment tells us

NFS mount on a Unix HPC cluster (mix of devtmpfs, tmpfs, xfs, nfs). ~100 parallel slurm jobs sharing the same lock, most of them readers, a subset writers, writes on the order of tens of GB taking a few minutes.

Writer-preference is load-bearing. With a ~99/1 reader/writer mix, a non-preferring design lets a writer starve for the entire job queue. A two-phase writer-acquire (claim .write first, then drain readers) turns writer starvation into something the design rules out.

Cross-host stale detection is mandatory. Slurm schedules jobs on arbitrary compute nodes. A job on node-42 holds a lock and crashes; the pid+hostname check SoftFileLock ships with today evicts only locks held by dead processes on the same host. Every reader on every other node wedges until someone runs a cleanup script. The HPC case needs a TTL with heartbeat refresh, so any host can evict a stale lock.

SoftReadWriteLock moves past what SoftFileLock does today. The extra complexity buys correctness on HPC.

Cross-language survey

Nothing in this table solves the same problem.

Language / library	NFS-safe	Shared locks	Notes
Rust `fs2` / `file-guard` / `fd-lock` / `file-lock`	no (`fcntl`/`flock`/`LockFileEx`)	yes	Every crate wraps advisory byte-range locks. None survive NFS.
Java `java.nio.channels.FileLock`	no	yes	Oracle's docs: "great care should be taken when locking files that reside on network filesystems" and "NFS does not consistently support exclusive locks."
Java Apache Curator `InterProcessReadWriteLock`	yes	yes	Needs ZooKeeper. Not file-based. What production Java on HPC uses.
Node.js `proper-lockfile`	yes via `mkdir(2)`	no	The one file-based NFS-safe library in this survey. Uses `mkdir` because "O_EXCL is broken on NFS". Periodic `mtime` refresh for stale detection. Exclusive only.
Go `gofrs/flock`, stdlib `lockedfile`	no	yes	`syscall.Flock()` returns `EBADF` on NFS.
Go `distrilock`	yes	exclusive-style	Client-server daemon, not a library. Go's answer when flock breaks.
Perl `File::NFSLock`	yes via `link(2)`	yes	20-year CPAN standard. Uses a chmod mode-bit hack (`0600`/`0601`) to distinguish share from exclusive. umask, SELinux, and restrictive NFS exports all mangle the bit.
C / shell `dotlockfile`, `lockfile(1)`	yes via `link()`	no	De facto on every HPC cluster. Exclusive only.
Lustre `-o flock`	client-local by default; cluster-wide only if explicitly mounted	yes	"The default behavior of Lustre cluster is to not allow locking."

Three buckets cover every language's answer.

Need RW semantics over a network? Bring in ZooKeeper, Redis, or etcd. Pay the infra cost.
Need NFS-safe file locks? Settle for exclusive-only.
Run the usual file-lock library and hope it does not corrupt data on NFS.

Perl's File::NFSLock is the closest precedent for shared-over-NFS, and its chmod mode-bit hack stays fragile in practice. proper-lockfile's mkdir + periodic mtime refresh is the closest precedent for the heartbeat half of the design.

SoftReadWriteLock will be novel across languages. Fork-safety and heartbeat lifecycle get explicit handling in the design below.

Primitive choice

proper-lockfile uses mkdir(2) instead of O_EXCL because mkdir is atomic on every filesystem, including the older NFS implementations where O_EXCL races.

For the HPC target (Linux 2.6+, NFSv3+, the realistic slurm deployment), O_EXCL works and SoftFileLock already bets on it. Swapping to mkdir would buy broader compatibility at the cost of content storage: a marker file carries its token, pid, and hostname, so mkdir foo.lock.write.d/ followed by open foo.lock.write.d/meta becomes two ops where one used to suffice. concepts.rst will call mkdir out as a fallback, and an O_EXCL race report from the field would trigger a revisit.

The design

Trust boundary

The class protects three classes of attacker. Same-UID non-cooperating processes on one host. Cross-host same-UID processes (the slurm default). Same-host different-UID users, blocked via 0o600 and 0o700 permissions on every file and directory the class creates.

Outside the trust boundary: root compromise, NTP tampering on same-UID cross-host nodes, multi-tenant mounts where hostile co-tenants share the UID. The docs will say so.

Filesystem layout

graph LR
    subgraph locks [" path/to/foo.lock.* "]
        state["foo.lock.state<br/>SoftFileLock<br/>held for µs only"]
        write["foo.lock.write<br/>writer marker<br/>token + pid + host"]
        dir["foo.lock.readers/"]
        r1["host1.4711.abc"]
        r2["host1.4712.def"]
        r3["host2.8192.ghi"]
        dir --> r1
        dir --> r2
        dir --> r3
    end

Every marker file stores token\npid\nhostname\n. The token is a per-acquire secrets.token_hex(16).

Every marker open passes O_CREAT | O_EXCL | O_NOFOLLOW | O_WRONLY, mode 0o600. O_NOFOLLOW refuses to create over a pre-existing symlink. Without it, a hostile user who pre-creates foo.lock.write as a symlink to ~/.ssh/authorized_keys gets our token+pid+hostname written into the victim file. SoftFileLock._acquire at _soft.py:41 already ships this pattern.

mkdir(2) has no O_NOFOLLOW equivalent, so readers/ needs special handling. After mkdir(readers_path, 0o700) (or on EEXIST), the class lstats the path and aborts on a symlink. It then opens os.open(readers_path, O_RDONLY | O_DIRECTORY | O_NOFOLLOW) to get a dirfd, and every subsequent reader-file operation passes dir_fd=. Dirfd-relative I/O closes the check-then-use race.

The marker parser is bounded. It reads at most 1024 bytes, validates the token against ^[0-9a-f]{32}$, the pid against a bounded integer range, the hostname against ASCII ≤ 253 chars. Any parse failure returns None and the caller treats the marker as invalid. The parser catches OSError and ValueError; nothing escapes.

Heartbeat and TTL

Default heartbeat interval 30 s. Default stale threshold 90 s (3 × interval, following etcd's LeaseKeepAlive convention). Any process on any host that sees a marker whose mtime has not advanced in 90 s may evict it.

On every tick, the heartbeat thread parses its marker and refreshes mtime only when the token matches. hmac.compare_digest handles the equality check; local filesystem access has no timing-channel risk, but the call signals intent and quiets security linters. On token mismatch or a missing marker, the thread calls stop_event.set() on itself and exits. A thread that kept running after the lock was stolen would touch files it no longer owns.

Release runs stop_event.set() → thread.join() → os.unlink(marker, dir_fd=...), in that order. Unlink under suppress(FileNotFoundError).

The daemon-thread lock-renewal pattern comes from python-redis-lock and redis-py.

Writer acquire (two-phase, writer-preferring)

flowchart TD
    start([acquire_write]) --> p1[Phase 1: claim .write marker]
    p1 --> l1[take state mutex]
    l1 --> e1[break stale writer by TTL]
    e1 --> c1{writer<br/>present?}
    c1 -- yes --> d1[drop state mutex]
    d1 --> w1[sleep, backoff]
    w1 --> l1
    c1 -- no --> mark["create .write<br/>via O_CREAT | O_EXCL | O_NOFOLLOW"]
    mark --> hb1[start heartbeat]
    hb1 --> p2[Phase 2: drain readers]
    p2 --> l2[take state mutex]
    l2 --> e2[break stale readers by TTL]
    e2 --> c2{readers/<br/>empty?}
    c2 -- no --> d2[drop state mutex]
    d2 --> w2[sleep, backoff]
    w2 --> l2
    c2 -- yes --> held([held])

Phase 1 creates .write before phase 2 starts. Any reader arriving mid-drain sees the marker and waits. That rules out writer starvation.

Reader acquire

flowchart TD
    start([acquire_read]) --> lock[take state mutex]
    lock --> evict[break stale writer by TTL]
    evict --> check{writer<br/>present?}
    check -- yes --> drop[drop state mutex]
    drop --> wait[sleep, backoff]
    wait --> lock
    check -- no --> new["create readers/host.pid.uuid<br/>via O_CREAT | O_EXCL | O_NOFOLLOW"]
    new --> hb[start heartbeat]
    hb --> free[drop state mutex]
    free --> held([held])

Breaking a stale marker

Between "observed stale mtime" and "unlink", the legitimate holder's heartbeat can refresh the marker. A naive unlink drops a live lock and lets two writers run at once. The class borrows SoftFileLock._try_break_stale_lock (_soft.py:56) and adapts it for TTL:

Read the marker. Capture its fstat mtime.
Rename the marker to <marker>.break.<pid>.<nonce>.
Re-read the renamed file. Run hmac.compare_digest on the token and check mtime <= captured_mtime.
On match, unlink. On mismatch, the legitimate holder refreshed between step 1 and step 2, and step 3 sees either a different token or a newer mtime. Abort. Leave the .break.* file for TTL or atexit cleanup. Rollback-rename never runs; rollback is itself racy.

Fork-safety

Python threads do not survive fork(). A process holds a lock, forks, and the child inherits the marker files and the lock-level state but not the heartbeat thread. The parent keeps refreshing. The child believes it holds the lock and refreshes nothing. Both processes write. The result is silent data corruption.

PyMongo's connection-pool handling is the reference. Three hooks:

if hasattr(os, "register_at_fork"):
    os.register_at_fork(
        before=_acquire_all_internal_locks,
        after_in_parent=_release_all_internal_locks,
        after_in_child=_reset_all_after_fork,
    )

before= iterates _all_instances and takes each live instance's _internal_lock, freezing internal state across the fork boundary. after_in_parent= releases them. after_in_child= runs _reset_all_after_fork: clear _lock_level, _current_mode, _write_thread_id, _marker_name, _token, drop the _heartbeat_thread reference, release _internal_lock (it crossed fork() in the locked state), and set _fork_invalidated = True. Marker files stay on disk because the parent still owns them.

Once cleared, an outstanding with lock.read_lock(): block that unwinds in the child calls release() on a zero-level lock. A strict implementation would raise "lock not held". The fork-invalidated flag makes release() a no-op on that instance. The child can re-acquire if it wants its own slot.

Reentrancy, API shape, and shutdown

Mirrors ReadWriteLock: reentrant within the same mode, write locks pinned to the acquiring thread, upgrade and downgrade raise RuntimeError, singleton caching by resolved path.

Debug logs carry pid and hostname. The token never appears in a log line.

atexit walks _all_instances and calls release(force=True) inside with suppress(Exception). Mirrors _cleanup_connections at _read_write.py:26.

Documentation plan

The PR touches all four Diátaxis dimensions.

Dimension	File	What changes
Tutorial	`docs/tutorials.rst`	Walkthrough on an NFS path showing reader and writer context managers, with an `ls` of the sidecar files.
How-to	`docs/how-to.rst`	New task section: "Use read/write locks on network filesystems." Covers tuning `heartbeat_interval`, `stale_threshold`, `poll_interval`, the fork caveat, and when to pick this class over `ReadWriteLock`.
Explanation	`docs/concepts.rst`	New subsections on cross-host stale detection, fork semantics, and the trust boundary. Updates to the lock comparison table, the selection flowchart, and the mechanism overview.
Reference	`docs/api.rst`	autodoc picks the new classes up; the work is docstring quality.

Plus docs/index.rst (a new card in the lock-types grid) and docs/changelog.rst (a feat(rw) entry).

Out of scope for this PR

Backporting TTL-based cross-host stale detection to SoftFileLock itself. Separate PR, lands after SoftReadWriteLock to keep review focused.
Refactoring ReadWriteLock into a pluggable-backend base. Revisit after SoftReadWriteLock has shipped and settled.
A mkdir(2) or link(2) primitive variant for conservative NFS setups where O_EXCL races. Not needed for the Linux-2.6+/NFSv3+ HPC target. Documented as a fallback in concepts.rst.
Root compromise, NTP tampering on same-UID cross-host nodes, multi-tenant mounts where hostile co-tenants share the UID. Outside the trust boundary.

Uh oh!

Feature request: ReadWriteLock that is compatible with network filesystems #526

Description

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions