[RPi4 Netboot] When network booting, /storage is mounted read-only, causing a crash

    This might be a long one, but I hope the debugging info is worth it!

    I'm trying to set up LibreELEC to boot over the network, on a Raspberry Pi 4. The server is a Debian box running dnsmasq, a TFTP server and NFS for the SYSTEM image (rootfs) and storage.

    To do this I downloaded https://releases.libreelec.tv/LibreELEC-RPi4…4-12.0.2.img.gz, unpacked the FAT partition into /tftpboot/(pi serial number), and did the configuration needed to get the Pi bootrom to boot over the network. It tried to boot, loaded the kernel and mounted /storage, passed control to systemd, then hung at "Reached target kodi".

    The NFS export line on the server is nothing special:

    Code
    /mnt/zfs/netboot-libreelec      10.0.0.0/8(rw,sync,no_root_squash,no_subtree_check)

    And this was my cmdline.txt:

    Code
    toram boot=NFS=magrathea:/mnt/zfs/netboot-libreelec/boot disk=NFS=magrathea:/mnt/zfs/netboot-libreelec/@UID@ ip=dhcp

    I needed a shell, so I started reading the initrd and /init script, and ended up with this cmdline.txt:

    Code
    toram nosplash debugging progress boot=NFS=magrathea:/mnt/zfs/netboot-libreelec/boot disk=NFS=magrathea:/mnt/zfs/netboot-libreelec/@UID@ ip=dhcp systemd.debug_shell=1

    (systemd.debug.shell=1 is the important part and makes systemd open a debug shell if it fails to bring the system up. "nosplash" disables the splash screen, and "debugging progress" enables init script debugging and makes it display its progress)

    I hit enter when systemd had failed to boot the system to get a shell, typed "mount" and found that the "/storage" filesystem was mounted read-only:

    Code
    /mnt/zfs/netboot-libreelec/boot on /flash type nfs (ro,noatime,vers=3,rsize=32768,wsize=32768,namlen=255,hard,nolock,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.1.3.4,mountvers=3,mountproto=tcp,local_lock=all,addr=10.1.3.4)

loop0 on /sysroot type squashfs (ro,relatime,errors=continue)

/mnt/zfs/netboot-libreelec/12345678 on /stoage type nfs (ro,noatime,vers=3,rsize=32768,wsize=32768,namlen=255,hard,nolock,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.1.3.4,mountvers=3,mountproto=tcp,local_lock=all,addr=10.1.3.4)

    I've not been able to figure out why this is happening as the init script seems to be trying to mount it read-write -- sadly to make any more progress I think I'd need to modify the init script and thus the initrd.

    I found the "break=" debug option which allows a debug shell to be opened up at a specified point in the init script's execution, and this got me a little further (proving that the FS was being mounted read-only to begin with, not being remounted read-only later).

    If I set "debugging break=mount_storage" in cmdline.txt and run these commands:

    Code
    mount -o remount,rw /storage
exit (back to init script)
exit (back to init script again)
exit (back to init script once more for the road)

    Then LibreELEC mounts the NFS and boots successfully.

    As a workaround, I put this "mount-storage.sh" script in my NFS "boot" export, alongside the SYSTEM file (the LibreELEC rootfs image):

    Bash
    #!/bin/sh
mount_part "$disk" "/storage" "rw,noatime"
mount -o remount,rw /storage

    And now LibreELEC boots fine, and (with the storage directory empty) displays the installation prompts.

    I'm not sure what the proper fix is -- but something weird seems to be going on with NFS mounts during the init process. In the meantime the workaround works.