Summary

Adds two experimental Provide.Strategy modifiers (+unique and +entities) for large IPFS nodes that pin many overlapping DAGs, extends fast-provide to pin add/pin update, and adds a new --fast-provide-dag flag for full-DAG provide on write.

Motivation

Nodes hosting large, versioned datasets (e.g. dist.ipfs.tech) pin thousands of DAGs that share 90%+ of their blocks. Today, every reprovide cycle re-walks every shared subtree once per pin, wasting I/O and time. Meanwhile, announcing every internal file chunk to the DHT produces millions of provider records when only entity-level CIDs (files, directories) matter for discovery.

Separately, pin add and pin update never fast-provided the root CID, leaving a gap where newly pinned content was undiscoverable until the next reprovide cycle (up to 22h).

Changes

+unique strategy modifier

Appending +unique to a pinned, mfs, or pinned+mfs strategy enables bloom-filter deduplication across recursive pins within a single reprovide cycle.

• A shared BloomTracker (from boxo/dag/walker) is created per cycle, sized from the previous cycle's persisted count (stored in datastore at /reprovideLastUniqueCount).
• When a CID is already in the bloom, its entire subtree is skipped -- reducing traversal from O(pins * total_blocks) to O(unique_blocks).
• Memory: ~4 bytes/CID (vs ~75 bytes for an exact cid.Set), enabling dedup on repos with tens of millions of CIDs.

Example: Provide.Strategy = "pinned+mfs+unique"

+entities strategy modifier

Appending +entities announces only entity roots (file roots, directory roots, HAMT shard nodes), skipping internal file chunks. Implies +unique.

• Uses walker.WalkEntityRoots from boxo, which inspects UnixFS node types and stops descending into file chunk subtrees.
• Non-UnixFS content (e.g. dag-cbor) is still fully walked.
• Drastically fewer DHT provider records for repos with large files.

Example: Provide.Strategy = "pinned+mfs+entities"

pin add / pin update fast-provide

Both commands now announce the pinned root CID immediately after pinning (matching ipfs add and ipfs dag import behavior). New flags:

• --fast-provide-root (default: Import.FastProvideRoot, true)
• --fast-provide-dag (default: Import.FastProvideDAG, false)
• --fast-provide-wait (default: Import.FastProvideWait, false)

With Provide.Strategy=all (default), this is a no-op since the blockstore already provides every block on write.

--fast-provide-dag flag

New flag on ipfs add, ipfs dag import, ipfs pin add, and ipfs pin update. When enabled, walks and provides the full DAG immediately after write using the active Provide.Strategy to determine scope. A single bloom tracker is shared across all roots for dedup. Configurable via Import.FastProvideDAG (default: false).

With --fast-provide-dag, the DAG walk emits the root first (DFS pre-order), so --fast-provide-root is redundant and not needed.

Gate providingDagService behind --fast-provide-dag

The providingDagService wrapper (which provides every block during ipfs add writes) is now only active when --fast-provide-dag=true. Previously it was always on for pinned strategy, causing per-block DHT provides during add regardless of user intent. Now the default path only fast-provides the root CID after add completes, and the reprovide sweep handles the rest.

Hardened strategy parsing

ParseProvideStrategy now returns an error instead of silently ignoring unknown tokens. Catches:

• Unknown tokens (typos like "uniuqe")
• Empty tokens from malformed delimiters ("pinned+", "+pinned", "pinned++mfs")
• Invalid combinations ("all+pinned", "roots+unique")

Validated at startup via ValidateProvideConfig. Internal callers use MustParseProvideStrategy for already-validated strings.

Rename ExecuteFastProvide to ExecuteFastProvideRoot

Clarifies intent now that ExecuteFastProvideDAG exists alongside it.

Files changed

• config/provide.go -- +unique/+entities strategy constants, error-returning ParseProvideStrategy, MustParseProvideStrategy, validation in ValidateProvideConfig
• config/provide_test.go -- tests for valid strategies, unknown tokens, empty tokens, invalid combos, MustParseProvideStrategy, and config validation
• config/import.go -- DefaultFastProvideDAG, FastProvideDAG config field
• core/node/provider.go -- +unique/+entities reprovide cycle with bloom tracker, MFS entity-root walker, bloom count persistence
• core/commands/cmdenv/env.go -- ExecuteFastProvideRoot (rename), new ExecuteFastProvideDAG
• core/commands/add.go -- wire --fast-provide-dag flag
• core/commands/dag/dag.go, core/commands/dag/import.go -- wire --fast-provide-dag flag
• core/commands/pin/pin.go -- fast-provide flags on pin add and pin update
• core/coreapi/unixfs.go -- gate providingDagService behind FastProvideDAG setting
• core/coreiface/options/unixfs.go -- FastProvideDAG option
• core/node/core.go, core/node/storage.go -- switch to MustParseProvideStrategy
• docs/config.md -- document +unique, +entities modifiers and caveats
• docs/changelogs/v0.41.md -- changelog entries

Compatibility

• Default behavior (Provide.Strategy=all) is completely unchanged.
• +unique and +entities are opt-in modifiers.
• --fast-provide-dag defaults to false.
• Strategy parsing is now stricter; previously-ignored typos in config will now produce an error at startup.

Depends on

• boxo/dag/walker -- BloomTracker, WalkEntityRoots, WalkDAG, LinksFetcherFromBlockstore, NodeFetcherFromBlockstore
• boxo/pinning/dspin -- NewUniquePinnedProvider, NewPinnedEntityRootsProvider

Context

• Closes Improved Reprovider.Strategy for entity DAGs (HAMT/UnixFS dirs, big files) #8676