LibreELEC on RPi4 messed up after repeated loss of electricity

    Hi,

    I have a setup where I use RPi 4 with LibreELEC for my Kodi experience. Last week there were several power cuts due to severe thunderstorms and as a result, I could see in the morning that the LibreELEC had been crashing during the night, and after the power returned, the TV was on with LibreELEC backdrop with all kinds of error messages concerning some or other process couldn't be killed and Kernel panic and so on. Sorry, but I am not very good at the Linux technical messages. So I couldn't figure out what was happening. So I tried several times to disconnect the power on the RPi4 and put it back on after a few minutes. This didn't help much. Every time it started to boot, the messages were back. After disconnecting everything and reconnecting all the wires and so on, I once got it back online and watched a movie with it. Then the next night there was another storm and the power cut occurred at least 3 more times (off and on again in a couple of seconds). And this time the same symptoms but now I can't revive the system no matter what. Every time it boots, there is this message, and the loading of the LibreELEC stalls.

    LibreELEC (official): 10.0.4

    [FAILED] Failed to start Connection service.

    [FAILED] Failed to start Samba NMB Daemon.

    [FAILED] Failed to start Samba SMB Daemon.

    .

    .

    .

    Endless repetition of the same lines scrolling down the screen!

    I have a lot of serials where I have been watching the episodes in random order so losing all the data about which episodes are watched is pretty important. Also a lot of movie details are also import when selecting the movies for the movie night. I would like to avoid the complete fresh install if I can help it. Is there any way I can repair the installation on the SD card that will repair/update the system but leave all my data untouched so that I can continue to use the system. Also any suggestions on how to avoid the similar problems in the future are most welcome (as long as they are not suggesting that I control the weather :D )

    FYI: this is my setup.

    RPi with 16GB SD card. LibreELEC installed. Media on a 2TB 2.5" HDD connected with USB3.0 cable (plus an additional USB2.0 power feed cable stealing the power from another USB port of the RPi). The system connected to my projector with a micro HDMI to HDMI cable and the whole thing is hidden deep out of sight to avoid the red light shining from the RPi box and the blinking lights on the HDD case when the data is read. This is another reason, why powering down the system every night to avoid the SD card data corruption due to power cuts is too cumbersome due to location of the whole RPi and HDD combo (hidden deep). There is a power supply that definitely supplies sufficient power (15W) for both RPi and the extra power required by the external HDD encloser (it's not much anyway since this is 2.5" HDD). I have setup the system nearly 1.5 years ago and never needed to touch it except a couple of time when I needed to power down the system and disconnect the HDD to transfer a large amount of data directly from the PC instead of over the air using wifi. And yes, during this time there have been many power cuts earlier too. But this is the first time the system has gotten this badly corrupted. Also during this 1.5 years, I have done couple of updates to the Kodi and some other things that the system was prompting me about. But I don't recall now what was the exact version of the Kodi or the LibreELEC because I can't even boot the system anymore. Please let me know if there are any other technical details I need to give to figure out a resolution to my problem (and also tell me where to find that detail without being able to boot the system).

    Any help is greatly appreciated! :)

    Hagar

  • Go to Best Answer

    Hopefully you have a system backup available. I recommend you do a fresh install and restore your backup if you do. You may have thrashed the file system with those unscheduled shutdowns/restarts so a fresh format of you SD card to ensure it’s in good condition, install LE and finally restore your backup. That will give you a starting point to get back to where you were prior to the power failures.

    It's hard to know what the issue is without seeing proper system or Kodi logs, but most likely something has gotten corrupted on the SD card. In that situation I'd recommend starting out with a clean install with LE11 (or an LE12 nightly) using a fresh SD card. This should get you back to a working (if not configured) system. From there you can connect the borked SD card using a USB > SD card adapter and see if the /storage partition on the card mounts to /mnt/<cardname>/storage or similar. If it does, restoring the general state of the previous setup is relatively simple and acheived by stopping Kodi, copying old userdata (database files, sources config and add-on settings) to the relevant location on the new card, starting Kodi up again, and (re)installing the add-ons that you were using. If you're familiar with some basic Linux fileystem and copy commands it's probably 5-10mins work to recover things. If you're not, it takes longer. If the old card fails to show up, then we need to see the system log to see what error messages are being generated. In that scenario it's probable that basic filesystem checks aren't going to get the card mounted and more complicated recovery processes might be required .. in which case "sorry, but now you understand the need for backups in the future!" is likely where we'll take a pause.

    Quote from chewitt

    It's hard to know what the issue is without seeing proper system or Kodi logs, but most likely something has gotten corrupted on the SD card. In that situation I'd recommend starting out with a clean install with LE11 (or an LE12 nightly) using a fresh SD card. This should get you back to a working (if not configured) system. From there you can connect the borked SD card using a USB > SD card adapter and see if the /storage partition on the card mounts to /mnt/<cardname>/storage or similar. If it does, restoring the general state of the previous setup is relatively simple and acheived by stopping Kodi, copying old userdata (database files, sources config and add-on settings) to the relevant location on the new card, starting Kodi up again, and (re)installing the add-ons that you were using. If you're familiar with some basic Linux fileystem and copy commands it's probably 5-10mins work to recover things. If you're not, it takes longer. If the old card fails to show up, then we need to see the system log to see what error messages are being generated. In that scenario it's probable that basic filesystem checks aren't going to get the card mounted and more complicated recovery processes might be required .. in which case "sorry, but now you understand the need for backups in the future!" is likely where we'll take a pause.

    The mention in the OP would suggest that the user is not familiar with Linux hence my suggestion to install from fresh and find the last working backup. That’s why I didn’t suggest using any form of tools to carry out repairs on the file system. More trouble than it’s worth 😂

    Thank you all for the excellent suggestions. Remote on/off support is a good suggestion for regular power down/up, but not going to be very helpful for those times when the power cut has already screwed up something on the SD card. For those times, I had to do disconnect all the cables and reboot everything in sequence to get the system up and running again.

    I like your suggestion a lot petediscrete, of fresh install and restoring from back-up. Now I just need to try and get the system up somehow to take my first back up. And next I need to figure out if there is a way that I can activate regular back ups automatically to some network drive or so. That way, I can have the automatic backup taken at regular intervals (say once a week) and if I need to restore from a fresh install, then at the most I will lose one week data of what has been already watched (or perhaps newly added stuff in that week). I have never done the backup earlier, but hope it means the backup of only the databases where the stuff like recently added, watched etc. is listed and NOT the whole media files! In such a case, I would assume that the data being backed up might be reasonably small that is worth doing every week. Also I don't want to flood my cloud storage with huge amount of backup files. Are these backups somehow incremental? Meaning, do I need to keep all the files from the beginning to be able to restore? Or they are cumulative, meaning I can always keep only the latest one and that will be sufficient to restore all the changes done up to that point? Also, is it possible to do the backups automatically to the network drive or is that a manual process?

    chewitt - I guess if I can't get the system up and running any other way, then your suggestion is the best one to rescue anything that is there on the current card. I have not tried if the SD card is completely fried or is it even possible to read something off it. I am not a master in Linux but I guess with the help of little surfing, I might be able to try some copy/paste operations, in case the file names/paths etc. are not changed a lot between what was on my card when I installed it nearly 2 years ago first time. If it looks very different and I try some plug and pray, I might end up with a system that is more messed up than now! :P To be honest, I don't have very high hopes of rescuing much from the old card if lot has changed. Let's see.

    • Best Answer
    Code
    systemctl stop kodi
mv /storage/.kodi /storage/.kodi-old
mkdir -p /storage/.kodi
cp -R /mnt/oldcard/STORAGE/.kodi/userdata /storage/.kodi/
systemctl start kodi

    ^ that will deliberately not import add-ons (which will need to be updated) but does get all their settings and any other userdata config so anything you've configured should restart using the pre-existing configuration. Databases will be updated on first restart, assuming they aren't corrupted (which is a possibility). If a file is corrupted and Kodi hangs you can "tail -f /storage/.kodi/temp/kodi.log" and see which DB file things hang on, then stop kodi, delete the offending DB file and then restart Kodi again and if there's an older/earlier DB file it will attempt to import that one instead. Art content like thumbs is best left to download again as the resulting cache is smaller and more efficient.

    It’s quite simple to do a manual backup. Just drop the backup onto a USB stick and you’re good to go. In the event of a complete system failure and after a fresh install of LE just drop the backup file from the USB stick onto the newly installed LE and restore it.

    I always do a backup just before a system upgrade and directly after it. Ensures there’s always something current to hand in an emergency. In fact I keep a spare SD card with LE installed close by. A simple swap out in the event of a failure ensures the evening’s viewing goes uninterrupted.

    I don’t use the cloud (not my scene) but a 4TB drive more than adequately covers my storage needs.