Pi netboot - 12.0.2 mounts NFS /storage as read-only

DavidGA · April 12, 2025 at 7:07 AM

When booting LibreELEC 12.0.2 on a Pi using NFS netboot, the /storage mount is mounted "ro", which breaks everything.

I added "textmode" to cmdline.txt and then did mount -o remount,rw /storage, which worked fine, which seems to rule out a server issue.

My cmdline.txt looks like this:
boot=NFS=<NFS server IP>:/path/to/boot disk=NFS=<NFS server IP>:/path/to/disk ssh ip=dhcp quiet console=ttyAMA10,115200 console=tty0

This works in 12.0.1, but is broken in:
12.0.2
12.0-nightly-20250404-5c24d2b
13.0-nightly-20250410-fe6b62d

Tested with both the Pi 4 and Pi 5 version, same results.

**chewitt** · April 12, 2025 at 1:10 PM

Can you bisect the issue using older LE12 nightlies to find when the breaking change was introduced?

**mglae** · April 12, 2025 at 4:02 PM

Did you read this thread?

DavidGA · April 12, 2025 at 9:39 PM

I checked out that thread, and tried adding vers=3 params, but that made no difference. (The mounts were vers=3 already, so I didn't expect it to.)

My NFS server doesn't support v4, if that's the issue, I can't test it. I hope that the problem is not "only NFSv4 is supported now" because lots of servers don't support v4. The "manage-gids" thing seems to be a red herring since that isn't enabled on my server.

Bisecting this is a lot of work but I'll get back to you if I find the time.

DavidGA · April 13, 2025 at 1:35 AM

LibreELEC-RPi5.aarch64-12.0-nightly-20241205-f894320 works

LibreELEC-RPi5.aarch64-12.0-nightly-20241217-32f8832 does not work

**heitbaum** · April 13, 2025 at 3:07 AM

Here is the changelog. So could be kernel 6.6.63 to 6.6.66. Nothing else stands out.

Comparing f894320...32f8832 · LibreELEC/LibreELEC.tv

Just enough OS for KODI. Contribute to LibreELEC/LibreELEC.tv development by creating an account on GitHub.

github.com

**mglae** · April 13, 2025 at 1:51 PM

The other thread is about NFSv4. vers=3 is still the default for kernel mounts.

Quote from DavidGA

and then did mount -o remount,rw /storage, which worked fine

Boot code in initramfs is using busybox mount while LE user mode is having util-linux mount + nfs-utils mount.nfs. There may be differences in behavior, but the tools are identical in the bisected versions.

Remain the kernel changes. For NFS (only) they are;

Code

> git --no-pager log --oneline v6.6.63...v6.6.66 -- fs/*nfs*
37dfc81266d3 nfsd: fix nfs4_openowner leak when concurrent nfsd4_open occur
1cecfdbc6bfc nfsd: make sure exp active before svc_export_show
065902117a5b nfs: ignore SB_RDONLY when mounting nfs
229a30ed42bb NFSv4.0: Fix a use-after-free problem in the asynchronous open()
dde654cad08f NFSD: Prevent a potential integer overflow
1dfc79bfc544 NFSD: Fix nfsd4_shutdown_copy()
bd8524148dd8 nfsd: release svc_expkey/svc_export with rcu_work
028417543535 NFSD: Cap the number of bytes copied by nfs4_reset_recoverydir()
eb51733ae5fc NFSD: Prevent NULL dereference in nfsd4_process_cb_update()

But I do see nothing obvious.

DavidGA · April 14, 2025 at 12:07 AM

I think "ignore SB_RDONLY when mounting nfs" is the problem:

See this discussion:

Making sure you're not a bot!

Relevant parts:

Quote

For NFSv3:
1) A single superblock is created for the initial mount.
2) When mounted read-only, this superblock carries the SB_RDONLY flag.
3) Before commit 52cb7f8f1778 ("nfs: ignore SB_RDONLY when mounting nfs"):
Subsequent rw mounts would not share the existing ro superblock due to
flag mismatch, creating a new superblock without SB_RDONLY.
After the commit:
The SB_RDONLY flag is ignored during superblock comparison, and this leads
to sharing the existing superblock even for rw mounts.
Ultimately results in write operations being rejected at the VFS layer.

Display More

Quote

When mounting the second NFS, the shared superblock is marked as ro,
causing the previous NFS mount to become read-only.
To resolve both issues, the ro flag is no longer applied to the superblock
during remount. Instead, the ro flag on the mount is used to control
whether the mount point is read-only.

I don't know exactly how this applies to the problem here, but it sure smells the same. In my case, both the boot and disk mounts are subdirectories of the same NFS share.

I can confirm that if you make boot= and disk= completely separate NFS mountpoints, the problem goes away.

If this can't be easily fixed, it should at least be documented.

**mglae** · April 14, 2025 at 4:20 PM

Good catch. Seem to be the issue (including NFSv4 not affected). Are you able to build an image with the patches?

With your remount finding and the mount-storage.sh feature here you can try to create this file in your boot= mount point to work around the issue:

Bash: mount-storage.sh

# Warning: if using "overlay" the target directory is not created

# Use default command to mount storage
mount_part "$disk" "/storage" "rw,noatime"
# force rw to work around kernel issue, see https://lore.kernel.org/lkml/20250304130533.549840-1-lilingfeng3@huawei.com/T/
mount -o remount,rw /storage

DavidGA · April 15, 2025 at 2:10 AM

I don't know how to build images, no.

But I'm happy with the workaround for now, it's OK to have /storage be a separate mountpoint on the server.