PXE Boot stuck on reboot/shutdown

    Quote from Goeran


    That is exactly the problem. But there is a solution. To circumvent connman to shutdown the interface on reboot/shutdown you need to prevent connman to manage the interface.
    Connman has an option to ignore interfaces via a blacklist in its configuration. The configuration is hard coded in squashfs image. But there is a script (/usr/lib/connman/connman-setup) that checks for a custom config before starting connman. Simply copy /etc/connman/main.conf to /storage/.config/connman_main.conf and add the network interface to NetworkInterfaceBlacklist option.

    That does work, but it causes other problems. With the ethernet device no longer managed by connmand, the DNS configuration can't be changed because it's hard coded into the systemd unit file for the connman service. DNS1 and DNS2 get set to Google's DNS servers. :(

    I tried to set the DNS servers in the connman_main.conf file and it failed to take because /etc/resolv.conf is actually a symlink to a connman dns config file. I don't want LibreELEC to be sending DNS queries to Google's servers: I have my own internal DNS server that I'd rather use and in fact have to because my database is centralized on a MariaDB KVM. With LibreELEC set to use Google's DNS servers, Kodi can't find the library.

    It also seems the default route isn't being set properly because LibreELEC can't reach the Internet (e.g. addon downloads fail).

    Is there another workaround I can try? I've been thinking that maybe I can setup a cron job on a server to check if the LibreELEC systems are up and if they are, to login and change the systemd config so that shutdowns and reboots work properly. The only problem with my craptastic idea is that I don't know for sure what's happening: is the file system being unmounted too soon or is the network shutting down too early?

    Quote from jau


    That does work, but it causes other problems. With the ethernet device no longer managed by connmand, the DNS configuration can't be changed because it's hard coded into the systemd unit file for the connman service. DNS1 and DNS2 get set to Google's DNS servers. :(


    Code
    cp /usr/lib/systemd/system/connman.service /storage/.config/system.d/

    And edit the file in /storage/.config/system.d to your needs.

    Quote from mglae
    Code
    cp /usr/lib/systemd/system/connman.service /storage/.config/system.d/

    And edit the file in /storage/.config/system.d to your needs.

    Damn! I didn't realize I could use customized systemd unit files in LibreELEC. Thanks for the pointer. I can finally fix the shutdown issue properly.

    Long time openelec pxe user here.

    I too have switched to libre and was suffering from this.

    I created an account here just to thank you guys and explicitly show the edits for future google searches :)

    Computer now reboots and shutdown like you would expect (FINALLY!!!)

    For the record : here's what I have (for an htpc with onboard lan)

    in .../nfsroot/libreELEC_krypton/.config :

    connman_main.conf

    Code
    NetworkInterfaceBlacklist = eth,vmnet,vboxnet,virbr,ifb,docker,veth,ve-,vb-

    and in .../nfsroot/libreELEC_krypton/.config/system.d

    copied

    Code
    [code]
cp /usr/lib/systemd/system/connman.service .../nfsroot/libreELEC_krypton/.config/system.d


    [/code]

    edited like this (added my home gateway first (192.168.0.1):

    Quote from mglae


    Code
    cp /usr/lib/systemd/system/connman.service /storage/.config/system.d/

    And edit the file in /storage/.config/system.d to your needs.

    That doesn't work anymore! :( So I poked around and saw the following in the README file under .confg/system.d/:

    Code
    $HOME/.config/systemd/user/*
/etc/systemd/user/*
/run/systemd/user/*
/usr/lib/systemd/user/*

    So I changed the path to .config/systemd/user and moved the connman.service file into that new directory. I restarted LibreELEC and while the DNS servers were set correctly, the search domains weren't added. WTF?

    I decided to change the ip of the secondary DNS server to see if the file is read at all, and sure enough, it isn't. It looks like connman is ignoring the device like I want it to, but the cusomt systemd unit file is being ignored.

    Maybe systemd is running on LibreELEC in a way that ignores the custom unit files? Here's what the README file says:

    It looks like systemd is running in "system mode" and is the reason why my custom unit file is being ignored. But, I could be wrong.

    Does anyone have any idea about what's going on?

    LibreELEC's systemd is patched to use /storage/.config/system.d/

    You can use systemctl status connman and systemctl show connman to check the configuration.

    For some reason, LibreELEC thinks the network is down:

    17:05:00.334 T:140685126657792 NOTICE: service.libraryautoupdate-1.0.0 : Starting network check

    17:05:00.366 T:140685126657792 NOTICE: service.libraryautoupdate-1.0.0 : Network down, not running

    17:06:00.379 T:140685126657792 NOTICE: service.libraryautoupdate-1.0.0 : Starting network check

    17:06:00.381 T:140685126657792 NOTICE: service.libraryautoupdate-1.0.0 : Network down, not running

    When I check the service, systemd is saying the service isn't enabled:

    ● connman.service - Connection service

    Loaded: loaded (/storage/.config/system.d/connman.service; disabled; vendor preset: disabled)

    Active: active (running) since Wed 2017-11-15 16:59:12 EST; 9min ago

    Is this normal behaviour? My custom network settings are being applied because I can ssh into the system and can see the DNS servers and search domains set correctly.

    I enabled connman to see what what would happen. The symlink was created and I restarted the service. But, the system still thought the network was down. Than I rebooted it, ran the status command to confirm the service was enabled still (it was) and saw in the logs the same "network down" message.


    I still don't know what's going on.

    I figured out some more: the default route wasn't set. Before 8.2, I didn't set the default route and everything just worked. That tells me the default route was being set outside of my customized connman unit file. But, with 8.2, it seems I have to explicitly create the default route.

    The mysql errors are now all gone from the kodi log. The "network down" errors are gone, too. Even the "inaccessible" errors for the NFS shares are gone. But, I still can't play any of my movies or tv-shows. :/

    I'll have to poke at it tomorrow...

    Welp, I powered it on this morning and everything is working. It seems I might have mixed up the order of my library testing and the creation of the default route yesterday.

    Anyway, the gist of it is if you have you're using a custom connman service unit file, make sure you include the addition of the default gateway.

    made a crude autostart script that works like a charm.

    Bash
    #!/bin/bash
/sbin/route add -net 0.0.0.0 gw 192.168.1.1
echo -e "nameserver 192.168.1.1\nnameserver 208.67.222.222\nnameserver 208.67.220.220" > /run/connman/resolv.conf

    /storage/.config/autostart.sh

    Network booting on RPi 3B+ is unreliable due to kernel and firmware issues that were resolved after the release of LibreELEC 8.2.5. Since we have no plans to issue further 8.2.y releases you'll need to switch to a LibreELEC 9.0 alpha release such as 8.90.003.

    I would like to configure my RPi3's to boot LibreElec over the network. I have a Synology with a working NFS share, but I am having trouble finding information on how to set this up. If somebody can point me to a Wiki article on how to set this up then I would appreciate the help. I'd also be willing to contribute a Wiki article if one doesn't exist yet once I get this all working. Thank you in advance!

    Thank you, chewitt! I saw that wiki article, but network booting on the Raspberry Pi's are done a little differently. Here is what I did to make it all work... Please let me know if I have provided any incorrect information. Thanks!

    First, download the LibreElec image files for all architectures of Raspberry Pi’s on the network (i.e. RPi1, RPI3)

    If your router acts as the DHCP server in your network then it will need to support network booting (most do not) if you want to continue to use it as the DHCP server, otherwise you will need to disable DHCP on your router and setup a DHCP server on your NAS. An alternative is to install a Linux firmware to your router if it’s possible (i.e. OpenWRT, DD-WRT, etc.) which is what I ended up doing.

    Synology Setup

    Enable TFTP service on your NAS. (Control Panel -> File Services -> TFTP) and select a root folder to use for network boot files. (i.e. shared folder called “public” with a folder inside called “pxeboot”.)

    Enable NFS service on your NAS. (Control Panel -> File Services -> SMB/AFP/NFS). Now click on “Shared Folder” and select the network share that you chose for network booting (i.e. “public”) and click the “Edit” button. Choose the NFS Permissions tab and click on the “Create” button. Use the following rules:

    Code
    Hostname or IP: *
Privilage: Read/Write
Squash: Map all users to admin
Security: sys

    Make sure the box is ticked for “Enable asynchronous”, “Allow connections from non-privilaged ports (ports higher than 1024)”, and “Allow users to access mounted subfolders” and click on the “OK” button. Click on the “OK” button to enable NFS on the selected share. From a Linux computer, test that the NFS share is now working (i.e. Try to access “nfs://synology/volume1/public”). If you are unable to access the NFS share from another computer on your network, then this issue will need to be resolved first before continuing.

    Inside the root folder that you chose for the network boot files (i.e. “pxeboot”), you will want to create a folder for all of the LibreElec files (i.e. “libreelec”) and then create two folders under that called “System” and “Storage”. Under the “System” folder you will want to create folders for each type of Raspberry Pi architecture that you will be using (i.e. “RPi1”, “RPi3”) and copy all the files from the “LIBREELEC” partition of the downloaded LibreElec image file to the corresponding RPi folder (i.e. “RPi3”). Copy the “bootcode.bin” file to the root folder (i.e. “pxeboot”). Edit the cmdline.txt file and change it to the following while substituting [synology] with the IP address of your NAS:

    Code
    boot=NFS=[synology]:/volume1/public/pxeboot/libreelec/System/RPi3 disk=NFS=[synology]:/volume1/public/pxeboot/libreelec/Storage/@UID@ quiet ssh ip=dhcp

    Do the same for any other Raspberry Pi architectures on the network.

    Router Setup

    SSH into your router running Linux firmware and add the following to the /etc/config/dhcp file substituting [synology] with the IP address of your NAS:

    Code
    config mac 'rasperrypi'
   option mac 'b8:27:eb:*:*:*'
   option networkid 'rasperrypi'
   list dhcp_option '66,[synology]'
   list dhcp_option '43,Raspberry Pi Boot'

    Reboot your router.

    Raspberry Pi Setup

    For Raspberry Pi 2’s and older: Before a Raspberry Pi will network boot, an SD card that contains at least the “bootcode.bin” file needs to be inserted.

    For Raspberry Pi 3’s only: Before a Raspberry Pi will network boot, it needs to be booted from an SD card with a config option to enable USB boot mode. This will set a bit in the OTP (One Time Programmable) memory in the Raspberry Pi SoC that enables network booting. Once this is done, the SD card is no longer required.

    Install Raspbian Lite (or Raspbian with Raspberry Pi Desktop) on the SD card in the usual way. Once booted up, log in and get the serial number of the Pi using the following command:

    $ cat /proc/cpuinfo

    Look for where it says “Serial” and write down the last 8 characters of this number because you will need it for later.

    Program USB boot mode (for Raspberry Pi 3 Only)

    First, prepare the /boot directory with the latest boot files:

    Code
    sudo apt-get update && sudo apt-get upgrade

    Now, enable USB boot mode with the following command:

    Code
    echo program_usb_boot_mode=1 | sudo tee -a /boot/config.txt

    This adds program_usb_boot_mode=1 to the end of /boot/config.txt. Reboot the Raspberry Pi with sudo reboot. Once the client Raspberry Pi has rebooted, check that the OTP has been programmed with:

    Code
    $ vcgencmd otp_dump | grep 17:
17:3020000a

    Ensure the output 0x3020000a is correct.

    The client configuration is almost done. The final thing to do is to remove the program_usb_boot_mode line from config.txt (make sure there is no blank line at the end). You can do this with sudo nano /boot/config.txt, for example. Finally, shut the client Raspberry Pi down with sudo poweroff.

    Finishing Up

    SSH into your Synology NAS and create a symlink for each Raspberry Pi that will be network booted substituting [arch] with the corresponding architecture (i.e. “RPi3”) and [serial] with the serial number of the Pi that you wrote down earlier:

    sudo ln -s /volume1/public/pxeboot/libreelec/System/[arch] /volume1/public/pxeboot/[serial]

    Also, create a folder under “Storage” for this Pi substituting [serial] with the serial number of the Pi that you wrote down earlier:

    mkdir /volume1/public/pxeboot/libreelec/Storage/[serial]

    Connect this Pi to the network and power it up. It should now network boot to LibreElec.

    Edited 3 times, last by powersurge (October 4, 2018 at 4:35 AM).

    Hi!

    I'm sorry for responding to a somewhat old thread but this was the most relevant thread I found so I registered just to share my solution.

    I've been struggling with networking booting LibreELEC over iSCSI (NFS works fine) since the shutdown process hangs, as many of you might know. Connman indeed seems to be the culprit (strange that with NFS it works fine).

    I tried different approaches suggested here. Many of them focus on disabling/circumventing connman completely but for me it causes the startup to take a bit longer and display a few errors since there are dependencies that would have to be edited too to make it work cleanly.

    Since the issue lies in the shutdown process, I simply ended up modifying connman.service by adding the following line to the unit-file:

    Code
    ExecStop=/usr/bin/killall -9 connmand

    This is still a hack, but it does prevent connman from interfering with the network by killing it instead of properly stopping it. (You could probably use something like ExecStop=/bin/true too as a dummy command for stopping it)

    To reiterate the steps I did to modify the unit-file:

    Code
    Copy the file "/usr/lib/systemd/system/connman.service" to "YOUR_STORAGE_PATH/.config/system.d"
Edit "YOUR_STORAGE_PATH/.config/system.d/connman.service" and add the ExecStop-line to the [Service] section
I then ran "systemctl daemon-reload" and "systemctl enable connman" to get the proper symlink created under "YOUR_STORAGE_PATH/.config/system.d...".
Check that "systemctl status connman" displays the new service-file path in the second line.

    A proper solution would be to rework the service dependencies or configure connman to somehow leave the connections to be. I took a quick look at it but it seems that systemd's shutdown order is always the reverse of the boot order and I didn't want to start messing with it that much so I came up with this hack instead.

    I hope this helps someone.

    I recently ran into that issue as well and PRed a fix for that: connman: ignore kernel-managed netdev and improve resolv.conf handling by HiassofT · Pull Request #3883 · LibreELEC/LibreELEC.tv · GitHub

    This fix is included in Milhouse nightly builds since #1002 and should make it's way into the next LibreELEC release if testing goes fine.

    LibreELEC Testbuilds for x86_64 (Kodi 19.0)

    LibreELEC Testbuilds for RaspberryPi (Kodi 19.0)

    Some feedback / test results would be highly appreciated.

    so long,

    Hias