Moving Kodi instances to using external DB - File no longer available error

1st Official Post

    Team. Probably a silly question. I have my main Kodi Instance where I mounting my shares using NFS within the autostart.sh and that is running on LibreELEC (Kodi 17.5). I move the library to an external SQL and everything works fine. Now I started with my 2nd instance and copied the advancedsettings.xml (FireStick - Kodi 17.6) and I can see the library and watched marks etc. but I'm unable to play any file. I know I'm missing something but not sure what is. Below my NFS mount on the main Kodi (LibreELEC)

    mount -t nfs 192.168.1.121:/volume1/video /storage/videos -o nolock; \

    mount -t nfs 192.168.1.121:/volume1/TV /storage/tvshows/ -o nolock; \


    I tried mounting my media sources on the 2nd Kodi instance via NFS the video add source within Kodi and didn't scan the content as I think it will add duplicates to my library if path is different (not sure). I name the mount volumes the same (/storage/videos and /storage/tvshows/) and still getting the file no longer available error message. If I go directly to the video source and select a file from there I can play the file no problem so the share is working fine.

    What I'm missing here?

    THanks in advanced.

    -embeleco

    I think that my problem is that on the main kodi (LibreELEC) I am doing the mount on the OS itself and then kodi is seeing the source as a local source and on the firestick kodi the source shows up as an NFS drive in Kodi. How can I fix that without changing my libreelec deployment. Below the source.xml file for both kodi instances:

    Kodi (LibreELEC)

    <sources>

    <programs>

    <default pathversion="1"></default>

    </programs>

    <video>

    <default pathversion="1"></default>

    <source>

    <name>Videos</name>

    <path pathversion="1">/storage/videos/</path>

    <allowsharing>true</allowsharing>

    </source>

    <source>

    <name>TV Shows</name>

    <path pathversion="1">/storage/tvshows/</path>

    <allowsharing>true</allowsharing>

    </source>

    Kodi (FireStick)

    <sources>

    <programs>

    <default pathversion="1"></default>

    </programs>

    <video>

    <default pathversion="1"></default>

    <source>

    <name>videos</name>

    <path pathversion="1">nfs://192.168.1.121/volume1/video/</path>

    <allowsharing>true</allowsharing>

    </source>

    Maybe using path substitution on the advancedsetting.xml but not sure how to use that.

    -embeleco

    Thanks milhouse.

    Instead of changing the LibreELEC client settings I opted for using the path substitution on the other kodi clients and they are working fine.. I deleted all the shares I have on my other kodi instances and with path substitution I can access now the actual files and playback is working surfing the central library on the mysql db.

    Now my last problem are the artwork / thumbnails. On the LibreELEC client (the one I did the library export / created the mysql db / import library) all the artwork / thumbnails are displayed correctly. But on my other kodi instances all of that is missing. Not sure how to tackle that problem. Any simple way to get that fix?

    -embeleco

    Yes, the simple way is to drop the database and start again using network shares! :)

    Everything else, including path substitution, is nothing but a barely working workaround, as you're discovering.

    I don't know how you've set up your path substitution but the problems with artwork are most likely because the artwork locations are once again inaccessible to the remote clients. You might be able to add more path substitution rules for the artwork, maybe. I personally wouldn't bother with path sub - you'll be dealing with this issue forever in one form or another, bite the bullet and fix it properly...

    The path substitution i’m doing is a simple one. I checked the path for the movies/tvshows that was displayed on the db and then did the substitution on the other kodi client for the actual NFS share where my actual media reside and started to work.

    But not sure what to do for the artwork / thumbnails / etc

    Edited once, last by embeleco (July 14, 2018 at 4:11 PM).

    Post your kodi debug log from the Firestick when using path substitution, I can't really comment further without knowing what the actual error is. Do the movies play successfully on the Firestick when using path substitution?

    Note that if you perform a library scan on the Firestick which added new content to the database, chances are the nfs:// paths will be added to the database so you'll end up with a mixture of /storage and nfs:// paths in your database. The nfs:// paths will probably still be accessible by the LibreELEC device (even without path substitution) but from an OCD perspective this would drive me up the wall... you could probably have re-scraped a clean, 100% nfs:// database by now.

    I'm not sure why you don't want to start afresh, but if you don't want to rescrape because you want to retain your watched statuses then there's a very simple solution for that which does not involve exporting the database. Also, exporting artwork is (IMHO) a really bad idea as the quality of the exported artwork is so much lower than the original artwork - again there is a solution for converting remote artwork to local artwork without losing quality. Unfortunately both solutions involve a command line, but if you're comfortable with that then you shouldn't have too many problems. I can post more details if this is of interest to you.

    Thanks for the help milhouse. Yes the movies play successfully on the firestick with path substitution. The only thing I don't have on the firestick is artwork / thumbnails / etc. I'm just trying to get it working as I don't want to loose my watched status / etc... but if the solution is to start fresh oh well.

    I don't plan to add a new content to the database from the firestick so nfs// path should not be on the database but again I'm not the expert here. Below the log I capture minutes ago

    hastebin

    -embeleco

    So you have these two path subs:

    Code
    21:52:34.954 T:18446744072517099920   DEBUG: Configuring path substitutions
21:52:34.954 T:18446744072517099920   DEBUG:   Registering substition pair:
21:52:34.955 T:18446744072517099920   DEBUG:     From: [/storage/videos/]
21:52:34.955 T:18446744072517099920   DEBUG:     To:   [nfs://192.168.1.121/volume1/video/]
21:52:34.955 T:18446744072517099920   DEBUG:   Registering substition pair:
21:52:34.955 T:18446744072517099920   DEBUG:     From: [/storage/tvshows/]
21:52:34.955 T:18446744072517099920   DEBUG:     To:   [nfs://192.168.1.121/volume1/TV/]

    but they're not working for your artwork as Kodi continues to retrieve the artwork files using the "local" path:

    Code
    21:52:39.020 T:18446744072529132944   DEBUG: GetImageHash - unable to stat url /storage/downloads/xbmc_videodb_2018-07-13/movies/Toy_Story_3_2010-poster.jpg

    What you'll need to do is add yet another path sub for /storage/downloads.

    Quote from embeleco

    I'm just trying to get it working as I don't want to loose my watched status

    If you just want to save watched status (ie. watched state, resume point and play count) then might I suggest using texturecache.py: [RELEASE] Texture Cache Maintenance utility

    You can install the script very easily into LibreELEC. In fact I include it as standard in my own LE9 test builds.

    To backup your watched status, on the LibreELEC client:

    Code
    texturecache.py watched movies backup /storage/watched-movies.dat
texturecache.py watched tvshows backup /storage/watched-tvshows.dat

    Now create your new library in MySQL using only nfs:// paths, then restore the watched status to the MySQL database:

    Code
    texturecache.py watched movies restore /storage/watched-movies.dat
texturecache.py watched tvshows restore /storage/watched-tvshows.dat

    Milhouse that is too advanced for me. :( Not sure how to work that texturecache utility and to do the nfs://paths on mysql as I did the library export using the client on libreelec. Kodi is not seeing the NFS path as I'm mounting the NFS shares within the libreelec OS using a script.

    Also weird as:

    21:52:39.020 T:18446744072529132944 DEBUG: GetImageHash - unable to stat url /storage/downloads/xbmc_videodb_2018-07-13/movies/Toy_Story_3_2010-poster.jpg

    that locations is where I did the export of the library (/storage/downloads/) on libreelec and what I used for importing the library back into libreelec kodi after I change kodi to use the external database using the advancedsettings.xml file. Not sure why it is actually using that path at all.

    Edited once, last by embeleco (July 15, 2018 at 3:39 AM).

    Quote from embeleco

    Milhouse that is too advanced for me.

    I'm really pretty sure it's not, particularly if you're willing to persist with path subs which are far more complicated IMHO! :)

    To use nfs:// paths just create an nfs:// source, in your case you'd create a Movies source with nfs://192.168.1.121/volume1/video/ and a TV Shows source with nfs://192.168.1.121/volume1/TV, then scrape the contents of each source into your new library.

    I'm sure you'll get it working with path subs eventually, but you'll always have a schizophrenic database that will try and break itself at every opportunity. Best of luck! :)

    I'll give it a shot with your texturecache script. But I have some doubts.. If I ran the script for getting the watch status/count/etc on the existing library and directory kodi is using.. once I move Kodi on libreELEC to use the new sources with NFS:// will the watch status / etc will be apply correctly? If that is the case.. then let me know if the order of steps is correct:

    1. Run texturecache script for getting all the watched status / count / etc for both existing local sources (Movies/TVs) [of course I need to know how to install the script on my existing libreELEC.

    2. drop the database all the database all together and start fresh

    3. point kodi on libreelec to the new mysql empty database

    4. delete the existing mount of NFS shares at the OS level with the autostart.sh

    5. add the NFS shares to kodi on libreelec and scrape the contents to the new library

    6. then ran the texturecache again to restore the watched status / count / resume / etc on kodi libreelec


    then what do I do on my other kodi instances besides pointing them to the new external database? Do I need to do something else to get the watched count etc on those boxes as well? Or will that start coming from the database?

    Also if the order is not right let me know ... I am a good listener and thanks a lot for helping me on this.

    Quote from embeleco

    will be apply correctly?

    Yes - so long as your movies and tv shows have the same names before and after (which they should). The script doesn't actually care where they are (or were) located when restoring the watched status, it only matches on the movie or tvshow names.

    Quote from embeleco

    then let me know if the order of steps is correct:

    Yep, you got it.

    Quote from embeleco

    [of course I need to know how to install the script on my existing libreELEC.

    It's explained in the first post of the thread I linked.

    If you want the super simple version, log in to LibreELEC and run:

    Code
    curl https://raw.githubusercontent.com/MilhouseVH/texturecache.py/master/texturecache.py -o /storage/texturecache.py
chmod +x /storage/texturecache.py

    Then to backup your watched status:

    Code
    /storage/texturecache.py watched movies backup /storage/watched-movies.dat
/storage/texturecache.py watched tvshows backup /storage/watched-tvshows.dat

    and to restore watched status (after you have rescraped the new library):

    Code
    /storage/texturecache.py watched movies restore /storage/watched-movies.dat
/storage/texturecache.py watched tvshows restore /storage/watched-tvshows.dat
    Quote from embeleco

    then what do I do on my other kodi instances besides pointing them to the new external database?

    Point them to the MySQL database by copying advancedsettings.xml.

    I'd also suggest copying sources.xml (in the same directory as advancedsettings.xml) from the LibreELEC client to your other clients just so that you have the same sources on all your clients. This step probably isn't essential, particularly if you don't plan on doing any scraping on those clients, but it may be useful.

    Quote from embeleco

    Do I need to do something else to get the watched count etc on those boxes as well? Or will that start coming from the database?

    Nope, that will all come from the central MySQL database.

    Once you have scraped your library on the LibreELEC machine you may find that the other non-LibreELEC clients don't have any artwork cached. They will cache the artwork as it is displayed, but this can mean navigating the GUI on those clients will be quite slow. A solution is to use texturecache.py running on the LibreELEC client to automate the caching of artwork on your other clients, but that's probably a story for another time (or read the first post of the linked thread, the details are all there, pretty much).

    Milhouse. It worked out perfectly :) As you said I ended copying the sources.xml file to all my other Kodi instances as if I add a file etc I needed to go to my home theater (LibreELEC) and update the library there... so now I can update the library from any kodi I'm connected to. Thanks a lot for all your help.

    I have two more problems.. I want your feedback.

    1. I'm running libreELEC ver 8.2.0 build from cVH as I needed some dvb drivers to work with my USB Tuner driver I used for OTA connected to the libreELEC HTPC running on an Intel NUC. Does your latest Leia builds have those drivers built in? I tried one Leia build not remember from who and I had to revert back as everything was broken :)

    2. I'm running Kodi 17.4 I think it is on libreELEC v8.2.0 on an Intel NUC 4th Generation Intel Core i3-4010U. Everytime I tried to run an HEVC file (H.265) kody start freezing and crashing. Almost sure it is too much for that CPU. I was wondering if getting a new 7th Generation Intel Core i5 NUC edition and move everything there (it is my main HTPC) or maybe there is other cheaper solution where I can put libreELEC, have my OTA Tuner working with Leia and of course play some decent HEVC and later on some 4K 10-bit HDR of course when time comes to that.

    I would like to hear your opinions and recommendations on the matter and again thank you very much for helping getting the external DB and watched settings replicated correctly and in an easy way.

    -Jose

    Glad you got it sorted.

    1. For RPi/RPi2 with 4.14.y kernel the extra DVB drivers are included. However for Generic (x86_64) with 4.17.y the additional DVB drivers are not currently included as they're not compatible, but will be when 4.18.y arrives in 2-3 weeks time. So, depending on your hardware the answer is "maybe".

    2. For an older generation Intel, the Skylake systems are fine and generally problem free, eg. NUC6i5SYH - they can play 4K videos without a problem, and also HEVC, but will never be able to output HDR. If you want 4K and 10-bit HDR then there's nothing from Intel right now that is guaranteed to support HDR, not even once Kodi, the Linux kernel and the various GPU drivers and sub-systems include 10-bit HDR support.

    Personally speaking, I can't recommend any current Intel systems given the uncertain support for HDR and other niggling issues that bedevil their hardware. The same applies to Nvidia, btw, but the uncertainty there is ongoing Kodi support which seems fairly unlikely post Kodi 18. My advice would be to look at an AMD Raven Ridge APU based solution.

    Or save your money and live with your current system a bit longer until Intel sort themselves out (which could take a while) - try disabling deinterlacing and/or video scaling options in case they're maxing out the i3 CPU. Finally, consider replace your Intel i3 with an RPi3+ which can handle 1080p HEVC without too many issues, and then buy x86_64 once you're sure what to get (yes, I'm serious!) :)