I just ran a quick test comparing the size of erofs and squashfs. For the same rootfs directory, the erofs file was 14 MB, the squashfs file only 11 MB.
I think we should stick with squashfs...

Could you please elaborate on the test you performed? I'm wondering if the default EROFS LZMA compression is simply not as high as what we use for squashfs.

I configured erofs to use lzma at maximum compression level by using -zlzma,9

there are quite a few options, just tested some block-sizes and a single timestamp for all files and no xattr

host/bin - [master] » for size in 4096 8192 16384 32768 65536 131072 262144 524288 1048576; do ./mkfs.erofs \
  -C$size -d1 -zlzma,9 -x-1 -T$(date +%s) \ 
  test.$size.erofs \
  /home/mt/Projects/weimarnetz/openwrt/bin/targets/ath79/generic/rootfs; done

host/bin - [master] » ls -1vl test*
-rw-r--r-- 1 mt mt 5169152 31. Mai 01:02 test.erofs
-rw-r--r-- 1 mt mt 5169152 31. Mai 01:18 test.4096.erofs
-rw-r--r-- 1 mt mt 4919296 31. Mai 01:19 test.8192.erofs
-rw-r--r-- 1 mt mt 4763648 31. Mai 01:19 test.16384.erofs
-rw-r--r-- 1 mt mt 4644864 31. Mai 01:19 test.32768.erofs
-rw-r--r-- 1 mt mt 4583424 31. Mai 01:19 test.65536.erofs
-rw-r--r-- 1 mt mt 4526080 31. Mai 01:19 test.131072.erofs
-rw-r--r-- 1 mt mt 4497408 31. Mai 01:19 test.262144.erofs
-rw-r--r-- 1 mt mt 4485120 31. Mai 01:19 test.524288.erofs
-rw-r--r-- 1 mt mt 4485120 31. Mai 01:19 test.1048576.erofs

host/bin - [master] » du -sh /home/mt/Projects/weimarnetz/openwrt/bin/targets/ath79/generic/rootfs
16M     /home/mt/Projects/weimarnetz/openwrt/bin/targets/ath79/generic/rootfs

https://www.kernel.org/doc/html/latest/filesystems/erofs.html has some interesting things like cache_strategy=disabled which might help with 32mb ram devices.

generic/_openwrt-ath79-generic-tplink_archer-c60-v3-squashfs-factory.bin.extracted - [remotes/github/master] » ls -l
insgesamt 17397
drwxr-xr-x 16 mt mt      18 29. Mai 22:37 squashfs-root
drwxr-xr-x 16 mt mt      18 29. Mai 22:37 squashfs-root-0
-rw-r--r--  1 mt mt 7598807 31. Mai 02:22 2147
-rw-r--r--  1 mt mt 6099566 31. Mai 02:22 2147.7z
-rw-r--r--  1 mt mt 3831660 31. Mai 02:22 2233A6.squashfs

still quite a difference - so 3831660 for squashfs vs. 4497408 with clustersize 256k and 5169152 for defaults

Thank you for testing, could you please append the following parameters and test again?

-zlzma -C 65536 -Eztailpacking

I got some email feedback from the developer and he recommended squashfs for now. I'll keep erofs in mind for the future.

EDIT: I just did, now the storage difference for x86/64 rootfs is 3.6MB (erofs) vs 3.3MB (squashfs).

with -Eztailpacking

-rw-r--r-- 1 mt mt  4132864 31. Mai 11:14 test.1048576.9eztailpacking.erofs
-rw-r--r-- 1 mt mt  4132864 31. Mai 11:14 test.524288.9eztailpacking.erofs
-rw-r--r-- 1 mt mt  4141056 31. Mai 11:14 test.262144.9eztailpacking.erofs
-rw-r--r-- 1 mt mt  4177920 31. Mai 11:14 test.131072.9eztailpacking.erofs
-rw-r--r-- 1 mt mt  4218880 31. Mai 11:14 test.65536.9eztailpacking.erofs
-rw-r--r-- 1 mt mt  4284416 31. Mai 11:14 test.32768.9eztailpacking.erofs
-rw-r--r-- 1 mt mt  4386816 31. Mai 11:14 test.16384.9eztailpacking.erofs
-rw-r--r-- 1 mt mt  4534272 31. Mai 11:14 test.8192.9eztailpacking.erofs
-rw-r--r-- 1 mt mt  4726784 31. Mai 11:13 test.4096.9eztailpacking.erofs
-rw-r--r-- 1 mt mt  4485120 31. Mai 11:13 test.1048576.9.erofs
-rw-r--r-- 1 mt mt  4485120 31. Mai 11:13 test.524288.9.erofs
-rw-r--r-- 1 mt mt  4497408 31. Mai 11:13 test.262144.9.erofs
-rw-r--r-- 1 mt mt  4526080 31. Mai 11:13 test.131072.9.erofs
-rw-r--r-- 1 mt mt  4583424 31. Mai 11:13 test.65536.9.erofs
-rw-r--r-- 1 mt mt  4644864 31. Mai 11:13 test.32768.9.erofs
-rw-r--r-- 1 mt mt  4763648 31. Mai 11:13 test.16384.9.erofs
-rw-r--r-- 1 mt mt  4919296 31. Mai 11:13 test.8192.9.erofs
-rw-r--r-- 1 mt mt  5169152 31. Mai 11:12 test.4096.9.erofs

Is there a way to check how much RAM {squashfs4,erofs} kernel module needs?

If RAM usage of erofs is much lower than squashfs4 it might still be good for limited RAM/plenty ROM devices.

Is there a way to check how much RAM {squashfs4,erofs} kernel module needs?

not sure but cat /proc/meminfo -> MemAvailable might be useful - it's also possible to add cat /proc/slabinfo with CONFIG_KERNEL_SLABINFO=y in the openwrt config. There should be allocations from squashfs visible or at least more data than just /proc/meminfo.

I Attempted to boot openwrt with this patch on a wdr4300 but it didn't boot, have to take a look again - poc works in qemu so this should be enough to get an idea and some data.

The EROFS paper talks about improvements vs. squashfs: https://www.usenix.org/system/files/atc19-gao.pdf (section 2.2) but not sure how much of this applies to openwrt.

I asked on the erofs mailing list and they're working on improvements regarding size. I'd just leave this PR open and update it from time to time, this way people can run tests without duplicating the work.

It surely takes another year or so to become the default, if ever :)

not sure but cat /proc/meminfo -> MemAvailable might be useful - it's also possible to add cat /proc/slabinfo with CONFIG_KERNEL_SLABINFO=y in the openwrt config. There should be allocations from squashfs visible or at least more data than just /proc/meminfo.

Please report all finding in this PR

I Attempted to boot openwrt with this patch on a wdr4300 but it didn't boot

Sorry for the inconvenience!

Sorry for the inconvenience!

no worries! super cool that we have something to experiment and if it's not a good fit at least we tried :)
It's probably still lot's of work until this someday land in the repo either as default or option but it's more like a marathon than a sprint :)

thanks for your time working on this - I try to gather more data when I have some time again.

Is there a way to check how much RAM {squashfs4,erofs} kernel module needs?

not sure but cat /proc/meminfo -> MemAvailable might be useful - it's also possible to add cat /proc/slabinfo with CONFIG_KERNEL_SLABINFO=y in the openwrt config. There should be allocations from squashfs visible or at least more data than just /proc/meminfo.

Just notice this topic, but that is not simple. EROFS strategy is to reuse page cache rather than allocate bounced compressed buffers if suitable, so the whole benefit is reduced runtime temp buffers, the final MemAvailable isn't helpful to measure such data.

Instead, EROFS will cache some compressed data if needed if available memory is abundant since no need to trigger I/O anymore (if you use readaround or readahead cache_strategy). But in low memory scenarios, it will disable such cache and use inplace I/O only.

I guess it can be observed by allocstall_ in /proc/vmstat if under memory pressure or you could just test some boot speed, app lagging under memory pressure etc instead, anyway I guess MemAvailable is not helpful to this topic.

Also, MicroLZMA is still an experimental feature compared to LZ4, partially due to no stable XZ utils released yet, partially since I still have some remaining runtime tuning work to do for such slow algorithms. I didn't prioritize that also due to no new stable XZ utils till now.

I wonder if, aside from size increase problem, this fs would give us other benefits... Improvement to the boot time?

I’ll make a note that @stintel was wondering about reworking the whole thing using btrfs and subvolumes. Never got anywhere.

I'm very interested in moving to EROFS as well but unfortunately this didn't not on mt7621 either :(

I think with the right settings you could probably get very close to squashfs in size and I'd welcome it if this became a FS to choose from

-rw-r--r-- 1 ansuel ansuel  6236968 Jul  7 14:59 openwrt-ipq806x-generic-netgear_r7800-initramfs-uImage_erofs
-rw-r--r-- 1 ansuel ansuel  6207768 Jul  7 15:06 openwrt-ipq806x-generic-netgear_r7800-initramfs-uImage_squashfs

-rw-r--r-- 1 ansuel ansuel  8913025 Jul  7 14:59 openwrt-ipq806x-generic-netgear_r7800-squashfs-factory.img_erofs
-rw-r--r-- 1 ansuel ansuel  7733377 Jul  7 15:06 openwrt-ipq806x-generic-netgear_r7800-squashfs-factory.img_squashfs

-rw-r--r-- 1 ansuel ansuel  6912798 Jul  7 14:59 openwrt-ipq806x-generic-netgear_r7800-squashfs-sysupgrade.bin_erofs
-rw-r--r-- 1 ansuel ansuel  5724958 Jul  7 15:06 openwrt-ipq806x-generic-netgear_r7800-squashfs-sysupgrade.bin_squashfs

So yhea compression still needs some love

It'd probably be worth giving the option to use it since EROFS is supposed to be noticeably quicker and there's plenty of devices that do have enough storage.

My proposal would be to keep squashfs as default but give an EROFS option while configuring the build.
Alternatively let the device maintainer decide on what should be used as default? Probably a better idea after long-term testing

https://github.com/erofs/erofs-openbenchmark/wiki/linux_5.10_rc4-compression-FSes-benchmark

@timocapa i agree... if there are real benefits and someone doesn't care about the extra space than it seems a nice idea to me making erofs an alternative fs

Certainly good enough for me :p

4.2M	erofsc128kpclusterlzma.root
3.8M	squashfs.root

That said that'll be impossible for me to make work since I'm not a dev :(

I'll work on that

@Ansuel could you please help me backport the outstanding Kernel patches that add tailpacking?

@aparcar mhhh sure just tell me what you need :D Where can i find the patches?

@Ansuel I think it needs everything erofs related up until 5.17.

@aparcar oh so we need to backport each patch made to erofs? tedious work!

@Ansuel honestly I don't know, I can ask upstream devs what's really needed, I thought you may have a nice script which simplifies backports

@aparcar unfortunately no script :(

If lzma is a hard requirement (which to be fair does save ~1 MB) backports are needed - lz4(hc) should work OK with 5.15

I don't know if this would be allowed, but otherwise I would just do the whole EROFS kernel backporting in a single patch

Other than that you do need xz updates as well if lzma

@timocapa mhhh single patch backport are scary and a mess to handle with kernel bump. I prefer a well split patchset

@Ansuel We can also wait for the next Linux LTS release to be supported on OpenWrt, already includes lzma

(Sorry for the previous comment that i deleted, i thought 5.19 was LTS)

@timocapa are you sure 6.0 will be LTS? I expect a 6.1 version till December at least. We are at rc4, can't take that long for a stable release.

I mean if we really want I can experiment with adding 6.0 support to openwrt ehehehehhe

jesus my github app mad freaked out

@Ansuel Nah I'm not sure however it would fit the release cycle of what i know (every .5), but I've edited the post to say "Next LTS" instead

We can also just skip lzma for now and make LZ4(HC) work, lzma should be a drop in replacement for it (and if not we would still have the basic setup to use)

So 6.0 has been out for a week now, what's the plan? :p

@timocapa none for now? openwrt supports only LTS version and the next LTS is 6.1... anyway i have added support for 6.0 on openwrt but currently i only ported ipq806x

You had mentioned "I can experiment with adding 6.0 support to openwrt", so I was curious :)

Delivered :D https://git.openwrt.org/?p=openwrt/staging/ansuel.git;a=shortlog;h=refs/heads/test-6.0
but i didn't test erofs

This might be helpful in understanding why squashfs might always be smaller but perform slower:
https://blog.sigma-star.at/post/2022/07/squashfs-erofs/

Squashfs has larger chunksizes like 128KiB which increases compression but since they are variable your read speed suffers. EROFS limits itself to one chunk size only: 4KiB block size.
The article explains this way better than me though (especially why 4KiB improves read performance).

Limiting Squashfs to 4KiB appears to produce smaller output than erofs still so there is likely room for improvement. (the article tested outlandishly larger partitions in the gigabytes, so this might not apply to 6MiB images)
It would be cool to see them surpass squashfs in compression but I don't see that happening.

EROFS definitely has a few advantages over squashfs but I'm not sure it justifies running two types of filesystems in parallel on a release. (Squashfs will likely remain the default for low flash devices, I think)
I'm looking forward to further results once openwrt runs 6.1 LTS in testing :) (or earlier since there is that 6.0 test branch)

@Djfe the 6.0 testing branch got updated to 6.1... and 6.1 will be introduced when we will branch for a new release

erofs-utils 1.6 is released, could you check this with Linux 6.1/6.2 [1]?
-zlzma -E[all-]fragments[,dedupe] -C xxxxx is preferred if image sizes is the most important stuff.

[1] https://github.com/hsiangkao/erofs-utils/blob/dev/docs/PERFORMANCE.md

@Ansuel would you mind helping me with some backporting ;)?

@aparcar sure! what can we use to coordinate?

Is someone else excited to take care of this? Kernel 6.6 is around the corner so less back porting is needed anyway.

@aparcar did we understood if we should target 6.1 or 6.6?

Whatever you think is best :)