video database corruption

1st Official Post

    I have the following setup:

    - media hosted on a NAS

    - MariaDB 10.5.5 on NAS hosting MyVideos116 and MyMusic72 databases

    - 4 x LibreELEC 9.2.4 on an RPi3

    - 3 x Macs running Kodi 18.8

    When the first RPi3 starts it modifies the database somehow that causes the 3 Macs running Kodi to not be able to start and just crash. The logs from the Mac states something about the database and then just crashes (or hangs).

    Code
    2020-09-21 22:34:49.019 T:123145534529536  NOTICE: MYSQL: Connected to version 10.5.5-MariaDB-1:10.5.5+maria~focal
2020-09-21 22:34:49.062 T:123145534529536   ERROR: SQL: [MyVideos116] An unknown error occurred
Query: SELECT idVersion FROM version
2020-09-21 22:34:49.062 T:123145534529536   ERROR: Process error processing job

    If I shutdown all LE players and then restore the database, the Mac Kodi players work just fine. It isn't until I turn on or restart one of my RPi3 running LE 9.2.4 that the database corrupts and the Mac players start crashing due to the database. Additionally, when this corruption happens, all of the media on the main screen of the LE players has the title of "0". The progress bars for in progress media is correct and even the watched totals are correct, but thumbnails, posters, titles, info, etc are all replaced with "0" on screen and I cannot select or play anything. Additionally, I cannot update the database and scan for media until I drop and restore the database. If I do not start up any of my Mac Kodi players, then all of the LE machines work off of the database it creates.

    So each player type works when using the database it creates/uses. But the other player type cannot use that database. This all worked fine under LE 9.2.3 and Kodi 18.7. Unfortunately the LE add on does not allow downgrading from 9.2.4 to 9.2.3. This database issue also happens with LE 9.2.4 (Kodi 18.8) and Kodi 18.7 on the Macs, which I find interesting because theoretically point releases should all work off the same database, only the major releases introduce breaking database changes as shown here: Databases - Official Kodi Wiki

    • Official Post
    Quote from squirtbrnr

    When the first RPi3 starts it modifies the database somehow

    That would be highly unlikely. If that really is the case, then I would like to see MariaDB's own log file with those 'changes'.

    Also, we like to see the entire kodi.log file (via a pastebin website). Four lines of the log is not gonna do it for us.

    Quote from squirtbrnr

    If I shutdown all LE players and then restore the database, the Mac Kodi players work just fine. It isn't until I turn on or restart one of my RPi3 running LE 9.2.4 that the database corrupts and the Mac players start crashing due to the database. Additionally, when this corruption happens, all of the media on the main screen of the LE players has the title of "0". The progress bars for in progress media is correct and even the watched totals are correct, but thumbnails, posters, titles, info, etc are all replaced with "0" on screen and I cannot select or play anything.

    The times when such behavior happens, usually some "naughty" add-ons has been installed. But, we'll see what happens in the kodi.log file first.

    Could it be related to this:

    [Leia] [backport] Bump mariadb connector version from 3.0.3 -> 3.1.9 by bkuhls · Pull Request #18171 · xbmc/xbmc · GitHub

    Quote

    Cherry-picked from #16275 and #18137

    Bump MariaDB connector version from 3.0.3 to latest released 3.1.9

    For Android 3.0.3 does not work properly with server versions 10.5.4, field types in result sets are not returned correctly therefore users see only "0" for all string fields.

    yes I should've added the full log. Sorry. Here is the log from the Mac after an LE machine has connected to the database and Kodi hangs on start and crashes.

    https://pastebin.com/raw/9m8AaWLH

    Do you want me to grab a log from the LE machine after I restore the database and start it for the first time?

    Here is the LE Kodi log from the LE machine I was testing with.

    http://ix.io/2yob

    Yes, I saw the release notes about the fix for MariaDB on Android, but I assumed that was specific to Android and should not have affected my two types of players (Mac and LE).

    • Official Post

    2020-09-21 22:34:48.437 T:4754218432 NOTICE: Disabled debug logging due to GUI setting. Level 0.

    This doesn't help.

    No clue yet about the error. Perhaps an update of the database went wrong.


    Also enable the Database component (at the bottom of the components listing) in the debugging section, restart Kodi, and do your thing, and provide a new log file. And try to locate your MariaDB log file, just in case.

    Here's my steps:

    - shutdown all Kodi players

    - Drop Kodi databases in MariaDB

    - start 1 LE machine and let it generate the database (http://ix.io/2yov)

    - shutdown LE machine

    - restore database in MariaDB

    - start 1 LE machine and database shows up (http://ix.io/2yoz)

    - start 1 Mac Kodi player and it just hangs and will not actually start (https://pastebin.com/raw/9j5cb7Cs)

    - restart 1 LE machine and database shows up (http://ix.io/2yoD)

    Then I did the opposite startup

    - shutdown all Kodi players

    - Drop Kodi databases in MariaDB

    - start 1 Mac Kodi player and let it generate the database (https://pastebin.com/raw/NzMFmsM0)

    - shutdown Mac Kodi player

    - restore database in MariaDB

    - start 1 Mac Kodi player (https://pastebin.com/raw/xRewtmSQ)

    - start 1 LE machine and database is corrupted and shows "0" for info (http://ix.io/2yoH)

    - reopen 1 Mac Kodi player and database shows up but cannot play any media and crashes (https://pastebin.com/raw/M0Y3Fu02)

    I run MariaDB in a docker container on the NAS. I believe the log is piped directly to the docker container log. I'm trying to to export that (Synology GUI doesn't play well with docker log, so I have to find the actual file it's reading).

    EDIT: And here is the only MariaDB log file I could find. https://pastebin.com/raw/dcamzQzA

    Edited once, last by squirtbrnr (September 22, 2020 at 4:12 PM).

    • Official Post

    - start 1 LE machine:

    2020-09-22 08:40:04.795 T:1807721344 ERROR: Unable to open database: MyMusic72 [1049](Unknown database 'MyMusic72')

    And have you enabled "Wait for network" on all LibreELEC clients?

    Code
    2020-09-22 9:11:56 5047 [Warning] Aborted connection 5047 to db: 'MyVideos116' user: 'kodi' host: '172.17.0.1' (Got an error reading communication packets)
2020-09-22 9:24:39 5054 [Warning] Aborted connection 5054 to db: 'unconnected' user: 'root' host: '172.17.0.1' (Got an error reading communication packets)
2020-09-22 9:25:56 0 [Note] mysqld (initiated by: unknown): Normal shutdown

    My guess is that your MariaDB server disconnects and/or simply shuts down. What OS/machine is your MariaDB server running on? Or mayb it is making separate connections for every query..? Shutting down the daemon sounds like stopping the database server.

    Yes, wait for network is enabled with the default timeout of 10 seconds on all LE machines. The reason it shows unknown database is because I had dropped the databases and was letting Kodi generate them from scratch (per the instructions).


    the aborted connection warnings show up in the MariaDB log whenever I close a MAC Kodi player or reboot/shutdown an LE machine. I run the MariaDB 10.5.5 official docker container on a Synology NAS. The database is not shutting down or restarting. I have one other application that writes to a completely separate database under a different user and it has not experienced any read/write errors or performance issues. Before this Kodi database issue, the MariaDB docker container had an up time of over 40 days.


    the mysqld shutdown is me actually restarting the database container because I could not drop the databases after I did my testing for some reason (that was the first time it had ever done that, it got hung dropping)

    just a reminder, this all worked under LE 9.2.3 and Kodi 18.7. MariaDB was upgraded from 10.5.4 to 10.5.5 at some point, but I could not tell you when that was. I believe it was prior to updating LE or Kodi. There was a part of that upgrade that seemed to fail but was fixed by running mysql-upgrade again. If I can recall, it was an issue with the sql_mode column in the mysql.proc table not of the correct type. After running the upgrade again, that error has not come up again.

    Edited 2 times, last by squirtbrnr (September 22, 2020 at 6:09 PM).

    brand new MariaDB 10.5.4 container with no previous databases existing. reinstalled LE 9.2.4 on a single RPi3. Reinstalled Kodi 18.8 on a single Mac. Launched Kodi on the Mac and created the new databases. Scanned in a folder of media. Powered on the RPi3 (http://ix.io/2ysv). Closed then opened Kodi on the Mac and it just crashes (https://pastebin.com/raw/H4jtv7G7). Rebooted the LE machine and it still works, but Kodi on the Mac crashes.

    Dropped all databases. Installed Kodi 18.7 on the Mac and downgraded LE to 9.2.3 (found an image backup I had). Powered on the RPi3 to create the new databases and scanned in a folder of media. Launched Kodi on the Mac and cannot play any media (https://pastebin.com/raw/vubS8VUA). Rebooted LE machine and database is fine and can play media (http://ix.io/2ysG). Reopened Kodi on the Mac and database partially shows up but won't play any media.

    MariaDB logs for all of this: https://pastebin.com/raw/TbY6jCBJ

    So on a whim, I decided to spin up a fresh MariaDB 10.5.1 container. LE 9.2.3, Kodi 18.7. Brand new databases, scanned in a folder of media. No issues. Rebooted both LE and the Mac and relaunched Kodi a number of times. No issues. I can select and play media on both. Restore my old database and still no issues on either. Upgrade LE to 9.2.4 and database shows up fine and I can play media. Update Kodi on Mac to 18.8 and no issues. Mac Kodi 18.8 log https://pastebin.com/raw/n1h6HmDC, LE 9.2.4 log http://ix.io/2ysT

    I'm now on the latest LE and Kodi release, but on a slightly older MariaDB release but the database is working again cross platform. I tried the same steps as above with MariaDB 10.5.2 and I get the database corruption showing "0" and Mac Kodi just crashes. I also tried doing an inline upgrade of MariaDB from 10.5.1 to 10.5.2 with the same crashing results.