Skip to content

Commit 8d194c1

Browse files
hsiangkaohsiangkao
committed
erofs-snapshotter: make IMMUTABLE_FL optional
Enabling the IMMUTABLE_FL file attribute causes dirty data to be flushed synchronously at least on EXT4, which can greatly impact container launch performance. In contrast, the overlayfs snapshotter does not use syncfs by default. Most users may not need IMMUTABLE_FL, let's make IMMUTABLE_FL optional to align with the behavior of the overlayfs snapshotter and recover the original performance. 1. tensorflow Test commands: $ nerdctl image pull --snapshotter=X --unpack="false" tensorflow/tensorflow:2.19.0 $ time nerdctl container --snapshotter=X run -d tensorflow/tensorflow:2.19.0 /bin/sh Results: overlayfs | 0m18.748s erofs (no IMMUTABLE_FL) | 0m10.090s erofs (with IMMUTABLE_FL) | 0m21.074s 2. ubuntu 22.04 Test commands: $ nerdctl image pull --snapshotter=X --unpack="false" ubuntu:22.04 $ time nerdctl container --snapshotter=X run -d ubuntu:22.04 /bin/sh Results: overlayfs | 0m1.147s erofs (no IMMUTABLE_FL) | 0m0.795s erofs (with IMMUTABLE_FL) | 0m1.094s Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
1 parent 930b0f1 commit 8d194c1

3 files changed

Lines changed: 57 additions & 12 deletions

File tree

‎docs/snapshotters/erofs.md‎

Lines changed: 34 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -104,7 +104,7 @@ builds, as shown below:
104104

105105
``` toml
106106
[plugins."io.containerd.differ.v1.erofs"]
107-
mkfs_options = ["-T0 --mkfs-time"]
107+
mkfs_options = ["-T0", "--mkfs-time"]
108108
```
109109

110110
If erofs-utils is 1.8.2 or higher, it's preferred to append `--sort=none` to
@@ -113,7 +113,7 @@ improved performance, as shown below:
113113

114114
``` toml
115115
[plugins."io.containerd.differ.v1.erofs"]
116-
mkfs_options = ["-T0 --mkfs-time --sort=none"]
116+
mkfs_options = ["-T0", "--mkfs-time", "--sort=none"]
117117
```
118118

119119
### Running a container
@@ -128,6 +128,38 @@ $ # run the container with the provides snapshotter
128128
$ ctr run -rm -t --snapshotter erofs docker.io/library/busybox:latest hello sh
129129
```
130130

131+
## Data Integrity
132+
133+
The EROFS snapshotter provides two methods to consolidate data integrity:
134+
135+
### Data Integrity with Immutable File Attribute
136+
137+
By setting `set_immutable = true`, the EROFS snapshotter marks each layer blob
138+
with `IMMUTABLE_FL`. This ensures that dirty data is flushed immediately and the
139+
EROFS layer blob cannot be deleted, renamed, or modified.
140+
141+
The immutable file attribute is mainly used to ensure data persistence and
142+
prevent artificial data loss, but it cannot detect data corruption caused by
143+
hardware failures. Since it can flush in-memory dirty data, it may significantly
144+
increase the unpacking time it takes to launch a container: for example, the
145+
unpacking time for tensorflow:2.19.0 increases by 108.86% (from 10.090s to
146+
21.074s) on EXT4. However, it has no impact on runtime performance.
147+
148+
### Data Integrity with fs-verity
149+
150+
By setting `enable_fsverity = true`, the EROFS snapshotter will:
151+
152+
- Enable fs-verity on EROFS layers during commit;
153+
154+
- Verify the fs-verity status before mounting layers;
155+
156+
- Skip fs-verity if the filesystem or kernel does not support it.
157+
158+
The fs-verity method guarantees that EROFS blob layers never change, but it
159+
introduces additional runtime overhead since all container image reads from
160+
the container will be slower because it needs to verify the Merkle hash tree
161+
first.
162+
131163
## How It Works
132164

133165
For each layer, the EROFS snapshotter prepares a directory containing the
@@ -165,14 +197,6 @@ In other words, the EROFS differ can only be used with the EROFS snapshotter;
165197
otherwise, it will skip to the next differ. The EROFS snapshotter can work
166198
with or without the EROFS differ.
167199

168-
## Data Integrity with fsverity
169-
170-
The EROFS snapshotter supports fsverity for data integrity verification of EROFS layers.
171-
When enabled via `enable_fsverity = true`, the snapshotter will:
172-
- Enable fsverity on EROFS layers during commit
173-
- Verify fsverity status before mounting layers
174-
- Skip fsverity if the filesystem or kernel does not support it
175-
176200
## TODO
177201

178202
The EROFS Fsmerge feature is NOT supported in the current implementation

‎plugins/snapshots/erofs/erofs_linux.go‎

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,8 @@ type SnapshotterConfig struct {
4444
ovlOptions []string
4545
// enableFsverity enables fsverity for EROFS layers
4646
enableFsverity bool
47+
// setImmutable enables IMMUTABLE_FL file attribute for EROFS layers
48+
setImmutable bool
4749
}
4850

4951
// Opt is an option to configure the erofs snapshotter
@@ -63,6 +65,13 @@ func WithFsverity() Opt {
6365
}
6466
}
6567

68+
// WithImmutable enables IMMUTABLE_FL file attribute for EROFS layers
69+
func WithImmutable() Opt {
70+
return func(config *SnapshotterConfig) {
71+
config.setImmutable = true
72+
}
73+
}
74+
6675
type MetaStore interface {
6776
TransactionContext(ctx context.Context, writable bool) (context.Context, storage.Transactor, error)
6877
WithTransaction(ctx context.Context, writable bool, fn storage.TransactionCallback) error
@@ -74,6 +83,7 @@ type snapshotter struct {
7483
ms *storage.MetaStore
7584
ovlOptions []string
7685
enableFsverity bool
86+
setImmutable bool
7787
}
7888

7989
// check if EROFS kernel filesystem is registered or not
@@ -146,6 +156,7 @@ func NewSnapshotter(root string, opts ...Opt) (snapshots.Snapshotter, error) {
146156
ms: ms,
147157
ovlOptions: config.ovlOptions,
148158
enableFsverity: config.enableFsverity,
159+
setImmutable: config.setImmutable,
149160
}, nil
150161
}
151162

@@ -432,9 +443,12 @@ func (s *snapshotter) Commit(ctx context.Context, name, key string, opts ...snap
432443
return fmt.Errorf("failed to enable fsverity: %w", err)
433444
}
434445
}
446+
435447
// Set IMMUTABLE_FL on the EROFS layer to avoid artificial data loss
436-
if err := setImmutable(layerBlob, true); err != nil {
437-
log.G(ctx).WithError(err).Warnf("failed to set IMMUTABLE_FL for %s", layerBlob)
448+
if s.setImmutable {
449+
if err := setImmutable(layerBlob, true); err != nil {
450+
log.G(ctx).WithError(err).Warnf("failed to set IMMUTABLE_FL for %s", layerBlob)
451+
}
438452
}
439453
return nil
440454
})

‎plugins/snapshots/erofs/plugin/plugin_linux.go‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,9 @@ type Config struct {
3636

3737
// EnableFsverity enables fsverity for EROFS layers
3838
EnableFsverity bool `toml:"enable_fsverity"`
39+
40+
// If `SetImmutable` is enabled, IMMUTABLE_FL will be set on layer blobs.
41+
SetImmutable bool `toml:"set_immutable"`
3942
}
4043

4144
func init() {
@@ -65,6 +68,10 @@ func init() {
6568
opts = append(opts, erofs.WithFsverity())
6669
}
6770

71+
if config.SetImmutable {
72+
opts = append(opts, erofs.WithImmutable())
73+
}
74+
6875
ic.Meta.Exports[plugins.SnapshotterRootDir] = root
6976
return erofs.NewSnapshotter(root, opts...)
7077
},

0 commit comments

Comments
 (0)