Kodi crash randomly overnight

mgouin · February 15, 2020 at 1:58 AM

Hello,

I have LE 9.2.0 running on a RPi3.

Recently, I realized that my ~~system~~ Kodi uptime was always a few hours, instead of days. I then noticed that every night ~~the~~ ~~system reboots~~ Kodi restarts by itself during the night. Normally, at the end of the evening, I just close the TV and let the RPi3 running.

When trying to look at the kodi logs (in /storage/.kodi/temp), I found the following interesting files:

Code

-rw-r--r--    1 root     root       62.5K Jan 19 22:28 kodi_crashlog_20200119222837.log
-rw-r--r--    1 root     root       79.7K Jan 21 03:34 kodi_crashlog_20200121033352.log
-rw-r--r--    1 root     root      379.7K Jan 23 01:45 kodi_crashlog_20200123014549.log
-rw-r--r--    1 root     root       70.3K Jan 24 05:15 kodi_crashlog_20200124051550.log
-rw-r--r--    1 root     root       98.1K Jan 29 01:55 kodi_crashlog_20200129015542.log
-rw-r--r--    1 root     root       69.7K Jan 30 02:58 kodi_crashlog_20200130025802.log
-rw-r--r--    1 root     root       66.3K Feb  1 00:43 kodi_crashlog_20200201004302.log
-rw-r--r--    1 root     root      277.6K Feb 10 12:38 kodi_crashlog_20200210123826.log
-rw-r--r--    1 root     root      119.9K Feb 13 03:29 kodi_crashlog_20200213032858.log
-rw-r--r--    1 root     root       56.3K Feb 14 00:49 kodi_crashlog_20200214004853.log

If it can be useful, here are the pastebin of the last 2 crashes:

Let me know if more crash dumps would help?

I briefly looked at the crash dumps, and the cause all seems to be inside the thread 1 of Kodi. Except the last crash, the previous ones all seem the same.

Code

LibreELEC:~/.kodi/temp # grep -A 2 'Program terminated' kodi_crashlog_*.log
kodi_crashlog_20200119222837.log:Program terminated with signal SIGABRT, Aborted.
kodi_crashlog_20200119222837.log-#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
kodi_crashlog_20200119222837.log-[Current thread is 1 (Thread 0x737c1240 (LWP 2492))]
--
kodi_crashlog_20200121033352.log:Program terminated with signal SIGABRT, Aborted.
kodi_crashlog_20200121033352.log-#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
kodi_crashlog_20200121033352.log-[Current thread is 1 (Thread 0x7378d240 (LWP 4001))]
--
kodi_crashlog_20200123014549.log:Program terminated with signal SIGABRT, Aborted.
kodi_crashlog_20200123014549.log-#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
kodi_crashlog_20200123014549.log-[Current thread is 1 (Thread 0x7371a240 (LWP 5933))]
--
kodi_crashlog_20200124051550.log:Program terminated with signal SIGABRT, Aborted.
kodi_crashlog_20200124051550.log-#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
kodi_crashlog_20200124051550.log-[Current thread is 1 (Thread 0x737e79c0 (LWP 10246))]
--
kodi_crashlog_20200129015542.log:Program terminated with signal SIGABRT, Aborted.
kodi_crashlog_20200129015542.log-#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
kodi_crashlog_20200129015542.log-[Current thread is 1 (Thread 0x737cb9c0 (LWP 392))]
--
kodi_crashlog_20200130025802.log:Program terminated with signal SIGABRT, Aborted.
kodi_crashlog_20200130025802.log-#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
kodi_crashlog_20200130025802.log-[Current thread is 1 (Thread 0x7375e9c0 (LWP 5980))]
--
kodi_crashlog_20200201004302.log:Program terminated with signal SIGABRT, Aborted.
kodi_crashlog_20200201004302.log-#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
kodi_crashlog_20200201004302.log-[Current thread is 1 (Thread 0x737f69c0 (LWP 391))]
--
kodi_crashlog_20200210123826.log:Program terminated with signal SIGABRT, Aborted.
kodi_crashlog_20200210123826.log-#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
kodi_crashlog_20200210123826.log-[Current thread is 1 (Thread 0x7373e9c0 (LWP 392))]
--
kodi_crashlog_20200213032858.log:Program terminated with signal SIGABRT, Aborted.
kodi_crashlog_20200213032858.log-#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
kodi_crashlog_20200213032858.log-[Current thread is 1 (Thread 0x737789c0 (LWP 15190))]
--
kodi_crashlog_20200214004853.log:Program terminated with signal SIGSEGV, Segmentation fault.
kodi_crashlog_20200214004853.log-#0  0x76f56be0 in memcpy () from /usr/lib/libarmmem-v7l.so
kodi_crashlog_20200214004853.log-[Current thread is 1 (Thread 0x737969c0 (LWP 392))]

Display More

How do I start from here?

Thanks in advance!

**Klojum** · February 15, 2020 at 2:28 AM Official Post

Quote from mgouin

How do I start from here?

2019-04-11 12:28:42.297 T:1937336768 NOTICE: Disabled debug logging due to GUI setting. Level 0.

It's always good to enable debugging so the Kodi log is more detailed. Do so, restart Kodi, and reproduce the error(s) for a new log file.

mgouin · February 15, 2020 at 7:25 PM

Quote

It's always good to enable debugging so the Kodi log is more detailed. Do so, restart Kodi, and reproduce the error(s) for a new log file.

It's funny, I enabled debug log yesterday, restarted and my system ran overnight without a crash. It did not restart by itself.

However, as a related question: is there a way to keep debug log enabled, but without the annoying top left overlay displayed constantly on the screen. I think I saw that in a custom xml, but I cannot find it back.

As I cannot yet pinpoint to something specific I did to cause the numerous crash every night, I'd like to keep running in debug log to hopefully catch the issue again.

Thank you in advance !

mgouin · February 15, 2020 at 7:34 PM

Ok, I found it. [SOLVED] Reboot and Shutdown Crash

Posting it back in case it can help someone:

Code

<advancedsettings>
  <loglevel>1</loglevel>
</advancedsettings>

That should enable debug logging but without that annoying overlay on the screen.

I'll try that and see if it helps.

mgouin · February 19, 2020 at 1:50 AM

Hello,

With debug log enabled I was able to catch another nighly crash. It ran for 4 days, so bear with the long debug log...

dmesg: http://ix.io/2c68

kodi.log: http://ix.io/2c69

kodi.old.log: Kodi old log for post https://forum.libreelec.tv/thread/21384-kodi-crash-randomly-overnight-and-reboot-rpi3 · GitHub (I think I reached the paste limit)

kodi_crashlog_20200218083716.log: kodi_crashlog_20200218083716.log · GitHub

Let me know if this is sufficient.

Thanks a lot!

mgouin · February 23, 2020 at 3:56 AM

Hello,

I again got a very similar crash. Let me know if you need more logs, but the crash dump was at the same place (pasted below).

I do embedded programming as a living, so I tried to see if I could find anything.

I found this:

Code

Thread 1 (Thread 0x7378a9c0 (LWP 7099)):
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x74da9284 in __GI_abort () at abort.c:79
#2  0x74cc0938 in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/libstdc++.so.6
#3  0x74cbe630 in ?? () from /usr/lib/libstdc++.so.6
#4  0x74cbe6a4 in std::terminate() () from /usr/lib/libstdc++.so.6
#5  0x74cbea0c in __cxa_throw () from /usr/lib/libstdc++.so.6
#6  0x74ce788c in std::__throw_length_error(char const*) () from /usr/lib/libstdc++.so.6
#7  0x74d02b44 in std::string::_Rep::_S_create(unsigned int, unsigned int, std::allocator<char> const&) () from /usr/lib/libstdc++.so.6
#8  0x74d03b70 in std::string::_Rep::_M_clone(std::allocator<char> const&, unsigned int) () from /usr/lib/libstdc++.so.6
#9  0x74d03c2c in std::string::reserve(unsigned int) () from /usr/lib/libstdc++.so.6
#10 0x74d03d40 in std::string::append(std::string const&) () from /usr/lib/libstdc++.so.6
#11 0x008d4298 in ?? ()
#12 0x00669aa4 in KODI::GUILIB::GUIINFO::CGUIInfoLabel::CInfoPortion::Get() const ()
#13 0x00669c34 in KODI::GUILIB::GUIINFO::CGUIInfoLabel::CacheLabel(bool) const ()
#14 0x0066a55c in KODI::GUILIB::GUIINFO::CGUIInfoLabel::GetLabel(int, bool, std::string*) const ()
#15 0x00643a28 in CGUILabelControl::UpdateInfo(CGUIListItem const*) ()
#16 0x006391ac in CGUIControlGroup::Process(unsigned int, std::vector<CDirtyRegion, std::allocator<CDirtyRegion> >&) ()
#17 0x0064fb04 in CGUIControl::DoProcess(unsigned int, std::vector<CDirtyRegion, std::allocator<CDirtyRegion> >&) ()
#18 0x00640c74 in CGUIControlGroupList::Process(unsigned int, std::vector<CDirtyRegion, std::allocator<CDirtyRegion> >&) ()
#19 0x0064fb04 in CGUIControl::DoProcess(unsigned int, std::vector<CDirtyRegion, std::allocator<CDirtyRegion> >&) ()
#20 0x00640c74 in CGUIControlGroupList::Process(unsigned int, std::vector<CDirtyRegion, std::allocator<CDirtyRegion> >&) ()
#21 0x0064fb04 in CGUIControl::DoProcess(unsigned int, std::vector<CDirtyRegion, std::allocator<CDirtyRegion> >&) ()
#22 0x006391cc in CGUIControlGroup::Process(unsigned int, std::vector<CDirtyRegion, std::allocator<CDirtyRegion> >&) ()
#23 0x0064fb04 in CGUIControl::DoProcess(unsigned int, std::vector<CDirtyRegion, std::allocator<CDirtyRegion> >&) ()
#24 0x006391cc in CGUIControlGroup::Process(unsigned int, std::vector<CDirtyRegion, std::allocator<CDirtyRegion> >&) ()
#25 0x0064fb04 in CGUIControl::DoProcess(unsigned int, std::vector<CDirtyRegion, std::allocator<CDirtyRegion> >&) ()
#26 0x006391cc in CGUIControlGroup::Process(unsigned int, std::vector<CDirtyRegion, std::allocator<CDirtyRegion> >&) ()
#27 0x0064fb04 in CGUIControl::DoProcess(unsigned int, std::vector<CDirtyRegion, std::allocator<CDirtyRegion> >&) ()
#28 0x006391cc in CGUIControlGroup::Process(unsigned int, std::vector<CDirtyRegion, std::allocator<CDirtyRegion> >&) ()
#29 0x0064fb04 in CGUIControl::DoProcess(unsigned int, std::vector<CDirtyRegion, std::allocator<CDirtyRegion> >&) ()
#30 0x0060d8f8 in CGUIWindow::DoProcess(unsigned int, std::vector<CDirtyRegion, std::allocator<CDirtyRegion> >&) ()
#31 0x00606064 in CGUIWindowManager::Process(unsigned int) ()
#32 0x0057ab90 in CApplication::FrameMove(bool, bool) ()
#33 0x0050e1a4 in CXBApplicationEx::Run(CAppParamParser const&) ()
#34 0x006a256c in XBMC_Run ()
#35 0x002b774c in main ()

Display More

It seems as if the xbmc/GUIInfoLabel.cpp at master · xbmc/xbmc · GitHub : Get()) method is causing a too big string.

From: libstdc++-v3 Source: basic_string.tcc Source File

Code

00462     _S_create(size_type __capacity, size_type __old_capacity,
00463           const _Alloc& __alloc)
00464     {
00465       typedef basic_string<_CharT, _Traits, _Alloc> __string_type;
00466       // _GLIBCXX_RESOLVE_LIB_DEFECTS
00467       // 83.  String::npos vs. string::max_size()
00468       if (__capacity > _S_max_size)
00469     __throw_length_error(__N("basic_string::_S_create"));

Please anyone... Where do I continue from here?

Thanks again!

mgouin · February 29, 2020 at 5:50 PM

Hello,

Do you think it's worthwhile to open an issue at XBMC? Issues · xbmc/xbmc · GitHub

Thanks

mgouin · February 29, 2020 at 8:23 PM

I created the following script to get every second the CPU, Temperature & Free RAM that I execute as a nohup.

Code

#!/usr/bin/bash

logfile=all.log

echo DATE,TimeOnly,CPU,%usr,%nice,%sys,%iowait,%irq,%soft,%steal,%guest,%idle,FREE,TEMP >> $logfile

# Loop indefinitely
while ( [ 1 ] ) do
    # Date stamp
    d=$(date +"%Y-%m-%d %H:%M:%S")

    # Mem (free)
    m=$(free -w | tr -s ' ' | cut -d ' ' -f 8 | head -n 2 | tail -n 1)

    # Temp
    t=$(vcgencmd measure_temp | egrep -o '[0-9]*\.[0-9]*')

    # CPU (last because it includes the pause)
    c=$(mpstat 1 1 | grep -w all | grep -v Average | tr -s ' ' | tr ' ' ',')

    printf "%s,%s,%s,%s\n" "$d" "$c" "$m" "$t" >> $logfile
done

Display More

Then, I analyzed the data with a quick python script:

Code

import pandas as pd
import matplotlib.pyplot as plt

# Read to dataframe
df = pd.read_csv('all.log', index_col='DATE', parse_dates=['DATE'])

# Plot relevant information data
df[['%usr','%nice','%sys','%iowait','%soft','TEMP']].plot()
plt.show()

Here is a screen shot of the plot, zoomed in where the crash occured:

Imgur: The magic of the Internet

Notice how the IOWait graph line jumps around the time my last crash dump was generated (kodi_crashlog_20200229052148.log)

Is there any way to debug iowait?

If it can help in debugging, I'm using the my LibreElec Pi as a DVR recorder with tvheadend running locally on the Pi. However, when the crash occurs, no recordings were in progress at this time.

thanks

mgouin · March 1, 2020 at 1:33 AM

For reference, I opened the kodi issue: Kodi crashes randomly overnight · Issue #17420 · xbmc/xbmc · GitHub

w3techie · March 21, 2020 at 11:42 PM

Did you ever find out what caused this?

I was wondering if you had tried using env_size to perhaps test the theory the string is too big. Or at least the theory that is causing the crash.

Perhaps, in a manner like so?

kernel - What defines the maximum size for a command single argument? - Unix & Linux Stack Exchange

I am suggesting this because you are a capable embedded dev.

mgouin · March 22, 2020 at 1:37 AM

Hello,

Thanks for the reply. I'm curious, do you experience similar crashes?

On my side, I did not get much progress beside a few more tests (refer to the github xbmc issue I raised).

Regarding your suggestion, I don't understand what you want me to try. The post refers to the command line argument length, vs what I thought my issue was is an internal C++ string length.

Could you please elaborate on your idea? I'm sincerely willing to try stuff as I'm mostly wandering in the dark with this issue...

Thanks again for your help!

w3techie · March 22, 2020 at 6:48 PM

Sorry for my first suggestion. I was up late last night and not fully awake. Well, I guess that happened later. I was multitasking all day yesterday. Lol!

Quote from mgouin

It seems as if the xbmc/GUIInfoLabel.cpp at master · xbmc/xbmc · GitHub : Get()) method is causing a too big string.

Also, I only mostly read this stuff at this stage, and rarely try it. So, I may not know anything. But nevertheless it never hurts to try. I used git and cloned the source and tested this particular file GUIInfoLabel.cpp, which I suspect is where the problem originates.

Here is what I have discovered. The includes are not on the right path.

As you can see the path for the GUIInfoLabel.h file is not the same as its location. It is looking for subfolders guilib/guiinfo/, but GUIInfoLabel.h is already inside the cwd.

Using nano to change the path of the GUIInfoLabel.h, I find the next file that it cannot locate is FileItem.h, and this is in the source directory xbmc/xbmc/. Using find to track down the rest of the files proves it is more than just a couple incorrect paths.

find /src/xbmc/ -name FileItem.h

If you move this file it continues to prove an issue. I think perhaps this means you compiled it from source for one thing, and they do not realize the paths are not correct for another.

So, I only wonder if that means I am correct and you compiled it from source? I wonder if it is a certain aspect of running the GUI that it is not capable of that way? What do you think?

Because if you did not compile it from source, when was the last time Kodi itself was updated. But wait, you are on LibreELEC which is read-only. So, no then I would wonder when was Kodi last updated? And when did the crashing began? Could it be at the same time?

I am only a tinkerer in this arena. I do not currently develop these things. I might give it a try though, and even fork Kodi. Most of my experience previously comes from web development, and embedded Linux is something I am trying to teach myself.

To answer the question of whether Kodi crashes on me. The answer is easily yes, but that is only because I am learning how to develop it and testing different server software on it. Sometimes I want to know what it takes to crash it on a device. Many times I think it is another piece of software has been installed and has a conflict. This is probably easier to do with multimedia apps.

For instance, if you have your own build of FFmpeg installed. You must guard it from being replaced or compromised. In the end, I decide to compile all my multimedia from source. Kodi uses my own build of FFmpeg, which you can compile with Kodi by changing the path in xbmc/CMakeLists.txt at master · xbmc/xbmc · GitHub.

Scroll down to lines 50 to 60 and see where you can enter the path to FFmpeg. You can build FFmpeg with external codecs, and then build Kodi with FFmpeg. And if you insert your own advancedsettings.xml you are really rocking! If you build FFmpeg with Chromaprint, you can create autoexec.py and fetch metadata for music files from AcoustID. So, sorry for getting carried away with all the extra information. I rarely get to share it.

mgouin · March 22, 2020 at 7:10 PM

Hey no problem! Thanks for sharing your passion

Unfortunately, I did not go this deep in the development. I simply took the pre-compiled image for raspberry pi 3 from LibreELEC official download.

I ended up looking at the source code because I was trying to analyze the crash dumps generated by kodi, especially the stack trace.

Thanks!

w3techie · March 22, 2020 at 9:56 PM

I posted to the issue page as well, and already got one reply. I am still wondering particularly about the Get() method big string theory.

Kodi crash randomly overnight

Similar Threads

Libreelec crash/freeze with AMD 2200G

Kodi.bin memory usage grows and grows until crash on RPi3B

Widevine update crashes