Orange Pi 3 - irregular system freezing

  • I have had a problem with LibreELEC / Kodi installed on the Orange Pi 3 board for a very long time (from the very beginning).

    The problem is that the whole system freezes. In this state, the board stops responding to any command, does not respond to the ping, the SSH session is interrupted and the image on the monitor freezes.

    The board responds only to the power button and, of course, to the power off. This problem occurs intermittently, usually within one hour of startup.

    I needed to find out whether the cause of the described problem is the nature of HW or SW.

    First, I tried to replace everything that could be replaced (SD card, cable, power supply). Unfortunately, no change.

    Then I tried to install Kodi in Armbian OS. It was unusable (very slow, just an old version of Kodi).

    Finally, I installed the Android 7 OS on the SD card and then Kodi. This is a combination you can live with, at least on a small monitor.

    I let it run for a week and everything was fine. The problem did not occur. I am aware that this is not entirely sufficient proof that the board is OK. But I only have one piece.

    What should I do ?

    kodi.log

    serial_console.log

  • I never experienced system freeze on OPi3. However, I did experience occasional Kodi crash on it. Other H6 boards/box that I have work fine. I tried to debug this problem several times, but I couldn't find anything useful, except that it crashes in ffmpeg during video decoding. This is elusive long standing bug on OPi3, which will obviously remain for time being. I'll take a look again in the future, but no promises.

    My problem could be very well related to yours. It could be due to too high memory clock or insufficient voltage or even due to some driver bug. In short, it's very hard to debug that.

  • Thanks for the answers, but it's sad for me.

    So I won't do a memory test yet.

    I understand that it will be difficult to find. No crash log because there is no crash ...

    However, such an unreliable system is unusable for normal operation.

    In my case, the problem may not be related to ffmpeg. I just restarted the system and then I didn't do anything.

    Unfortunately, this freezing also occurs when watching IPTV.

    Just for the purpose of this testing, I made a new SD card and did not install any add-ons at all (except for the screensaver).

    So this is an almost pure "factory" state as when it was first run.

    The course of the test was that I enabled the debug log and rebooted.

    The screensaver started automatically and the photos show that the freeze occurred at 06:50, while the last entry in the kodi.log was at 06:32:37.

    I was surprised that the serial console log contains only the system boot and then nothing.

    Would it be possible to make some debug image that would write in detail to the serial console?

  • In my case, the problem may not be related to ffmpeg. I just restarted the system and then I didn't do anything.

    This is probably something else, then.

    I was surprised that the serial console log contains only the system boot and then nothing.

    Printing on serial in theory slows down boot, if only for a second or less.

    Would it be possible to make some debug image that would write in detail to the serial console?

    no need, just remove quiet from extlinux.conf. If you want terminal on serial, also add systemd.debug-shell=ttyS0

  • no need, just remove quiet from extlinux.conf. If you want terminal on serial, also add systemd.debug-shell=ttyS0

    I had the quiet parameter removed before. Now I have also added systemd.debug-shell=ttyS0, so the serial console is fully functional.

    So now I can use a serial console to communicate with the LE in a similar way as via an SSH connection.

    Once it freezeed, I found that the system was not responding to the serial console either. Like when it's off.

    From what we know until now, it won't hurt to do the test.

    That's right, a memory test can't hurt. So I prepared an SD card with the Debian operating system. I found only one software for memory testing, a memtester.

    I did some tests, but all the results were OK. Debian worked well, there was no freezing.

    Then I did the same and with the same result with the Ubuntu operating system.

    So there is no indication that my board is defective.

    Nevertheless, I decided to buy another piece of the same board to see if the board was defective or not.

    I am aware that in one case I will have two good boards, which, however, cannot be used with LibreELEC ...

  • I have made considerable progress !

    It was like this: At first I made one desperate attempt. I took a new SD card and wrote the LE image for the Orange Pi One Plus on it.

    I inserted the card into the Orange Pi 3 and turned on the power. LE and Kodi started normally. I made some settings and installed (among other things) the PVR IPTV Simple Client add-on and my IPTV add-on. I used it for a few days, waiting for a total freeze to occur again. It didn't happen!

    So it looks like the freeze problem is probably related to either USB3 or WiFi / BT. To rule out at least one of these, I decided to run at least USB3 (the absence of USB3 bothered me much more than the absence of WiFi and BT). So I found the source file sun50i-h6-orangepi-one-plus.dts and tried to compile it into sun50i-h6-orangepi-one-plus.dtb. But I couldn't. I was still getting the error message "FATAL ERROR: Unable to parse input tree".

    So I did it differently. I took the sun50i-h6-orangepi-one-plus.dtb file located in the /flash folder and converted it to the sun50i-h6-orangepi-one-plus.dts file. There I enabled USB3 and did the conversion back to sun50i-h6-orangepi-one-plus.dtb.

    I replaced the original file in the /flash folder and turned on the device. It worked - USB3 started working.

    I used the device with the modified DT file again for a few days - it never froze!

    Then I took the next step. I returned to the original SD card I had earlier in Orange Pi 3. There I replaced the original sun50i-h6-orangepi-3.dtb file with the modified sun50i-h6-orangepi-one-plus.dtb file. After switching on the device, the expected is confirmed: no total freezing.

    Result: It seems that the total freezing of the Orange Pi 3 is related to either WiFi or BT. However, it is not enough if I use the original file sun50i-h6-orangepi-3.dtb and disable WiFi and BT in the settings of the LibreELEC add-on.

    How do I proceed to find the real cause of the freeze?

    Note: I'm still talking about "total freezing". So I say a state where the board did not respond to anything (except turning off the power or pressing the power button). This never happened while testing the device with a modified DT file. However, sometimes (always during IPTV playback) there was a temporary freeze (the image stopped and after a while Kodi restarted spontaneously). This is probably related to the mentioned ffmpeg. Interestingly, this crash never occurred on an Orange Pi 3 board with the original sun50i-h6-orangepi-3.dtb file.

  • Interesting analysis, thanks!

    I compared OrangePi One Plus and OrangePi 3 DTs and one detail caught my attention immediately. BLDO3 power supply is marked as unused and not connected anywhere in OrangePi 3 DT . However, schematic shows it should be used for VCC-PG, VCC-PM and VCC-WIFI-IO. I'm not sure if this is the culprit of instability but certainly something to look into.

    Unfortunately I'm busy with some other, more pressing issues. Would you please look into this matter a bit further and try to pinpoint exact issue? As you already noticed, OrangePi 3 and OrangePi One Plus DTs are pretty similar, so there is not much things to change and test. All you need to do is to make some changes in OrangePi 3 DT, create a patch, build your own image with that patch, update your system and test. Rinse and repeat until your board becomes stable. If you need help with that, just ask.

  • LuRu I think I found important difference. While it's true that wireless power supply is a mess, really important bit is CPU voltage range. I copied over One Plus values and my OPi3 seems to be much more stable. Please test this update:

    Index of /test/opi3-stability/

    Apply it to ordinary OrangePi 3 image.

    DT changes: http://ix.io/3yMQ

    If that doesn't work for you, I have one additional DT change to test. But I would really hope it's not the culprit.

  • LuRu I think I found important difference. While it's true that wireless power supply is a mess, really important bit is CPU voltage range. I copied over One Plus values and my OPi3 seems to be much more stable. Please test this update:

    Index of /test/opi3-stability/

    That sounds promising!

    As soon as I get home today, I'll try it right away. If it's okay, I'll get back to you in a few days. If it's not okay, I'll see it today or tomorrow at the latest.

    BTW: I've done some attempts in the meantime (especially on the weekend), but I haven't been successful.

  • Of course it's not certain yet (freezing is accidental), but it looks good!

    It has only one flaw. This is LE11 and updating or installing LE add-ons fails.

    How can I compile LE10 with the necessary changes?

    Note: I am able to perform a normal compilation (eg with git checkout 10.0.0).

    Edited once, last by LuRu (September 15, 2021 at 3:25 PM).

  • Of course it's not certain yet (freezing is accidental), but it looks good!

    Unfortunately, next day it didn't work anymore for me, even with CPU devfreq totaly thrown out. I'll borrow another OPi3 to check again. I have a feeling that this is not so common issue, otherwise there would be more complaints, either here or IRC #linux-sunxi.


    How can I compile LE10 with the necessary changes?

    just make patch file from link I posted above and put it in projects/Allwinner/devices/H6/patches/linux/ It should just work...

  • I can't compile it.

    I created a 0021-arm64-dts-allwinner-h6-OPi3-stability.patch file to which I copied http://ix.io/3yMQ.

    I placed this file in the location you specified.

    Compilation failed:

    Contents of the sun50i-h6-orangepi-3.dts.rej file:

  • So I was finally successful.

    Now I have a fully functional build LE10 for OPi3, including the necessary DT patch. And my experience so far is very positive.

    The board is completely stable, no system freezes. It seems to confirm that the official file sun50i-h6-orangepi-3.dts is not correct. In any case, it does not correspond to a publicly known wiring diagram.

    I have a feeling that this is not so common issue, otherwise there would be more complaints, either here or IRC #linux-sunxi.

    Yes, that also amazes me a lot.

    Here is the my patch for LE10: 0021-arm64-dts-allwinner-h6-OPi3-stability-LE10.patch