Skip to content

erigon snapshots reset fails on mounted chaindata directory and may break symlink setups #18135

@lemenkov

Description

@lemenkov

System information

Erigon version: 3.3.0

OS & Version: Fedora 43 (Aarch64)

Commit hash:

Erigon Command (with flags/config): erigon snapshots reset --datadir /var/lib/erigon/

Consensus Layer: N/A

Consensus Layer Command (with flags/config): N/A

Chain/Network: mainnet

Expected behaviour

erigon snapshots reset should remove the contents of the chaindata directory while preserving the directory itself, any mountpoint, or any symlink.

This is important because the [storage optimization documentation](https://docs.erigon.tech/fundamentals/optimizing-storage) recommends using symlinks:

Store datadir on the slow disk

Place the datadir on the slower disk. Then, create symbolic links (using ln -s) to the fast disk for the following sub-folders:

  • chaindata
  • snapshots/domain

This will speed up the execution of E3.

Actual behaviour

The command attempts to remove the chaindata directory itself rather than its contents. This causes two problems:

  1. Fails when chaindata is a mountpoint with error: removing chaindata dir: unlinkat /var/lib/erigon/chaindata: device or resource busy

  2. May silently break symlink setups - if chaindata is a symlink to fast NVMe storage and reset removes the symlink, the operator may not notice. On next startup, Erigon recreates chaindata as a regular directory on the slower base filesystem, causing significant performance degradation.

Steps to reproduce the behaviour

  1. Mount a separate disk at /var/lib/erigon/chaindata (or create a symlink to another location)
  2. Run erigon snapshots reset --datadir /var/lib/erigon/
  3. Observe failure with "device or resource busy" (mountpoint) or silent removal of symlink

Workaround: Manually clean the directory contents instead:

rm -rf /var/lib/erigon/chaindata/*

Backtrace

[root@curve-7 ~]# erigon snapshots reset --datadir /var/lib/erigon/
INFO[12-02|14:08:54.321] logging to file system                   log dir=/var/lib/erigon/logs file prefix=erigon log level=info json=false
INFO[12-02|14:08:54.321] resetting datadir                        path=/var/lib/erigon
INFO[12-02|14:08:54.324] [db] open                                label=chaindata sizeLimit=8TB pageSize=16KB
INFO[12-02|14:08:54.325] read chain name from config              chain=mainnet
INFO[12-02|14:08:54.325] Loading remote snapshot hashes 
INFO[12-02|14:08:55.918] Loaded preverified snapshots hashes      len=5512 chain=mainnet
INFO[12-02|14:08:55.918] Resetting snapshots directory            path=/var/lib/erigon/snapshots
INFO[12-02|14:08:56.617] Files NOT removed from snapshots directory torrents=1422 data=1422
INFO[12-02|14:08:56.617] Files removed from snapshots directory   torrents=90 data=235
INFO[12-02|14:08:56.617] Removing chaindata dir                   path=/var/lib/erigon/chaindata
removing chaindata dir: unlinkat /var/lib/erigon/chaindata: device or resource busy

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No fields configured for Bug.

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions