Kodi crash randomly overnight

1st Official Post

    Hello,

    I have LE 9.2.0 running on a RPi3.

    Recently, I realized that my system Kodi uptime was always a few hours, instead of days. I then noticed that every night the system reboots Kodi restarts by itself during the night. Normally, at the end of the evening, I just close the TV and let the RPi3 running.

    When trying to look at the kodi logs (in /storage/.kodi/temp), I found the following interesting files:

    Code
    -rw-r--r--    1 root     root       62.5K Jan 19 22:28 kodi_crashlog_20200119222837.log
-rw-r--r--    1 root     root       79.7K Jan 21 03:34 kodi_crashlog_20200121033352.log
-rw-r--r--    1 root     root      379.7K Jan 23 01:45 kodi_crashlog_20200123014549.log
-rw-r--r--    1 root     root       70.3K Jan 24 05:15 kodi_crashlog_20200124051550.log
-rw-r--r--    1 root     root       98.1K Jan 29 01:55 kodi_crashlog_20200129015542.log
-rw-r--r--    1 root     root       69.7K Jan 30 02:58 kodi_crashlog_20200130025802.log
-rw-r--r--    1 root     root       66.3K Feb  1 00:43 kodi_crashlog_20200201004302.log
-rw-r--r--    1 root     root      277.6K Feb 10 12:38 kodi_crashlog_20200210123826.log
-rw-r--r--    1 root     root      119.9K Feb 13 03:29 kodi_crashlog_20200213032858.log
-rw-r--r--    1 root     root       56.3K Feb 14 00:49 kodi_crashlog_20200214004853.log

    If it can be useful, here are the pastebin of the last 2 crashes:

    Let me know if more crash dumps would help?

    I briefly looked at the crash dumps, and the cause all seems to be inside the thread 1 of Kodi. Except the last crash, the previous ones all seem the same.

    How do I start from here?

    Thanks in advance!

    Edited once, last by mgouin: Correction: its not the whole system (RPi) that reboots, but only Kodi. I kept strikethrough to mark my changes. (February 29, 2020 at 8:28 PM).

    • Official Post
    Quote from mgouin

    How do I start from here?

    2019-04-11 12:28:42.297 T:1937336768 NOTICE: Disabled debug logging due to GUI setting. Level 0.

    It's always good to enable debugging so the Kodi log is more detailed. Do so, restart Kodi, and reproduce the error(s) for a new log file.

    Quote

    It's always good to enable debugging so the Kodi log is more detailed. Do so, restart Kodi, and reproduce the error(s) for a new log file.

    It's funny, I enabled debug log yesterday, restarted and my system ran overnight without a crash. It did not restart by itself.

    However, as a related question: is there a way to keep debug log enabled, but without the annoying top left overlay displayed constantly on the screen. I think I saw that in a custom xml, but I cannot find it back.

    As I cannot yet pinpoint to something specific I did to cause the numerous crash every night, I'd like to keep running in debug log to hopefully catch the issue again.

    Thank you in advance ! :)

    Hello,

    With debug log enabled I was able to catch another nighly crash. It ran for 4 days, so bear with the long debug log...

    dmesg: http://ix.io/2c68

    kodi.log: http://ix.io/2c69

    kodi.old.log: Kodi old log for post https://forum.libreelec.tv/thread/21384-kodi-crash-randomly-overnight-and-reboot-rpi3 · GitHub (I think I reached the paste limit)

    kodi_crashlog_20200218083716.log: kodi_crashlog_20200218083716.log · GitHub

    Let me know if this is sufficient.

    Thanks a lot!

    Hello,

    I again got a very similar crash. Let me know if you need more logs, but the crash dump was at the same place (pasted below).

    I do embedded programming as a living, so I tried to see if I could find anything.

    I found this:

    It seems as if the xbmc/GUIInfoLabel.cpp at master · xbmc/xbmc · GitHub : Get()) method is causing a too big string.

    From: libstdc++-v3 Source: basic_string.tcc Source File

    Code
    00462     _S_create(size_type __capacity, size_type __old_capacity,
00463           const _Alloc& __alloc)
00464     {
00465       typedef basic_string<_CharT, _Traits, _Alloc> __string_type;
00466       // _GLIBCXX_RESOLVE_LIB_DEFECTS
00467       // 83.  String::npos vs. string::max_size()
00468       if (__capacity > _S_max_size)
00469     __throw_length_error(__N("basic_string::_S_create"));

    Please anyone... Where do I continue from here?

    Thanks again!

    I created the following script to get every second the CPU, Temperature & Free RAM that I execute as a nohup.

    Then, I analyzed the data with a quick python script:

    Code
    import pandas as pd
import matplotlib.pyplot as plt

# Read to dataframe
df = pd.read_csv('all.log', index_col='DATE', parse_dates=['DATE'])

# Plot relevant information data
df[['%usr','%nice','%sys','%iowait','%soft','TEMP']].plot()
plt.show()

    Here is a screen shot of the plot, zoomed in where the crash occured:

    Imgur: The magic of the Internet

    Notice how the IOWait graph line jumps around the time my last crash dump was generated (kodi_crashlog_20200229052148.log)

    Is there any way to debug iowait?

    If it can help in debugging, I'm using the my LibreElec Pi as a DVR recorder with tvheadend running locally on the Pi. However, when the crash occurs, no recordings were in progress at this time.

    thanks

    Edited 2 times, last by mgouin (March 1, 2020 at 1:40 AM).

    Hello,

    Thanks for the reply. I'm curious, do you experience similar crashes?

    On my side, I did not get much progress beside a few more tests (refer to the github xbmc issue I raised).

    Regarding your suggestion, I don't understand what you want me to try. The post refers to the command line argument length, vs what I thought my issue was is an internal C++ string length.

    Could you please elaborate on your idea? I'm sincerely willing to try stuff as I'm mostly wandering in the dark with this issue...

    Thanks again for your help!

    Sorry for my first suggestion. I was up late last night and not fully awake. Well, I guess that happened later. I was multitasking all day yesterday. Lol!

    Quote from mgouin

    It seems as if the xbmc/GUIInfoLabel.cpp at master · xbmc/xbmc · GitHub : Get()) method is causing a too big string.

    Also, I only mostly read this stuff at this stage, and rarely try it. So, I may not know anything. But nevertheless it never hurts to try. I used git and cloned the source and tested this particular file GUIInfoLabel.cpp, which I suspect is where the problem originates.

    Here is what I have discovered. The includes are not on the right path.

    1hsfZIa.png

    As you can see the path for the GUIInfoLabel.h file is not the same as its location. It is looking for subfolders guilib/guiinfo/, but GUIInfoLabel.h is already inside the cwd.


    Using nano to change the path of the GUIInfoLabel.h, I find the next file that it cannot locate is FileItem.h, and this is in the source directory xbmc/xbmc/. Using find to track down the rest of the files proves it is more than just a couple incorrect paths.

    find /src/xbmc/ -name FileItem.h


    If you move this file it continues to prove an issue. I think perhaps this means you compiled it from source for one thing, and they do not realize the paths are not correct for another.

    So, I only wonder if that means I am correct and you compiled it from source? I wonder if it is a certain aspect of running the GUI that it is not capable of that way? What do you think?

    Because if you did not compile it from source, when was the last time Kodi itself was updated. But wait, you are on LibreELEC which is read-only. So, no then I would wonder when was Kodi last updated? And when did the crashing began? Could it be at the same time?

    I am only a tinkerer in this arena. I do not currently develop these things. I might give it a try though, and even fork Kodi. Most of my experience previously comes from web development, and embedded Linux is something I am trying to teach myself.

    To answer the question of whether Kodi crashes on me. The answer is easily yes, but that is only because I am learning how to develop it and testing different server software on it. Sometimes I want to know what it takes to crash it on a device. Many times I think it is another piece of software has been installed and has a conflict. This is probably easier to do with multimedia apps.

    For instance, if you have your own build of FFmpeg installed. You must guard it from being replaced or compromised. In the end, I decide to compile all my multimedia from source. Kodi uses my own build of FFmpeg, which you can compile with Kodi by changing the path in xbmc/CMakeLists.txt at master · xbmc/xbmc · GitHub.

    Scroll down to lines 50 to 60 and see where you can enter the path to FFmpeg. You can build FFmpeg with external codecs, and then build Kodi with FFmpeg. And if you insert your own advancedsettings.xml you are really rocking! If you build FFmpeg with Chromaprint, you can create autoexec.py and fetch metadata for music files from AcoustID. So, sorry for getting carried away with all the extra information. I rarely get to share it.

    Edited 6 times, last by w3techie (March 22, 2020 at 10:26 PM).

    Hey no problem! Thanks for sharing your passion :)

    Unfortunately, I did not go this deep in the development. I simply took the pre-compiled image for raspberry pi 3 from LibreELEC official download.

    I ended up looking at the source code because I was trying to analyze the crash dumps generated by kodi, especially the stack trace.

    Thanks!