Copying exported library fails: "colons" in directory names conflicts with SMB & NTFS

1st Official Post

    Hi,

    I'm not sure whether this is a kodi related or a LibreELEC related issue, but as LibreELEC is responsible for the filesystem as the underlying OS I want to describe the issue here as well.

    When I export the video library (single file) to SD Card (ext 4) on my Odroid C2 everything is fine at first. If you then try to copy this directory for backup reasons - which contains fanart, coverart and so on of tv shows and movies - this fails. It fails via SMB, but I'm not sure if this affects also NFS or so. The same happens if I export the Library to my external NTFS USB drive.

    The issue is that some library directories have names like:

    "TRON:_Der_Aufstand"

    This is only one of at least 10 directories which have colons in its filenames, thus I cannot copy the Backup off the system to another system.
    As filenames containing a colon are not allowed on NTFS and in windows (are they allowed on ext4 drives?), this behavior damages the filesystem on my NTFS HDD (connected to Odroid C2 running LibreELEC). At first I didn't notice it, because the directories were build during library export process on my SD-Card. But when I connected to the drive via SMB and tried to copy the backup before formatting the SD-Card, it was not possible. And it is also not possible to rename these directories :/. Hence (at first I thought) I could only get rid of this directories if I format the drive (NTFS drive with 3TB of data).

    Later I noticed this:
    The faulty directories are there directly after exporting... I can see them in Kodi file manager and via SMB. But I get an error if I try to access them via SMB. But then suddenly when I connect this usb-device again to my windows Laptop afterwards, these directories are missing. I'm not sure what happens to them. Also a lot of other directories from this exported library-backup were not there anymore. I assume that all directories, which were written after the faulty one (containing this colon), are broken then and not visible or corrupted. I'm not sure if they occupy space on the disk, but I think they do, even though they cannot be accessed or found anymore, but they could be seen at first before remount of the drive.

    The result of this is: The library backup (export) is always broken/incomplete immediately after doing the export because those colon-library entries are gone and some other folders which were written afterwards are also gone.

    All this sounds a bit confusing (colon directories there at first, but not accessable and not deletable; then gone completely and taking other folders with them), but something strange is happening because of these colons in filenames, which should be avoided during creation of library (and backup) in the first place.

    So... is it possible to prohibit LibreELEC doing filenames, which contain colons?

    These filenames originally come due to overtaking the series names during scraping with the default thetvdb.com scraper addon. The above example shows:
    Tron: Der Aufstand TRON: Der Aufstand: Series Info
    Tron: Uprising Tron: Uprising: Series Info

    So LibreELEC could do some kind of validity check while making a library (to not even start doing such filenames). And then again a validy check while exporting a library for cases like these, thus it could then do a substitution for signs like colons. Windows for example substitutes a "colon" with "%3A".

    Edited once, last by infinity85 (October 5, 2016 at 1:40 PM).

    • Official Post

    It's not so much a Kodi or LibreELEC problem, but more a Linux vs Windows partition problem.

    The problem is indeed the colon character ":", Linux does accept the colon character in EXT4 file/folder names, and Windows partitions formatted like SMB/NTFS do not accept it.

    So, the solution would be to manually remove all colons from your folders & filenames, cleanup your video library and rescrape the "new" movies. Or, alternatively, you could format your backup drive as an EXT4, but of course your Windows computer(s) won't like that.

    Some scapers do accept usage of "-" (dash) instead of ":" (colon).
    I used a search and replace tool to make my movie collection more 'universal' by removing all colons from movie/folder names.

    Thank you both for your quick replies :)

    But hmm... I'm not sure about this... I think we misunderstand each other.

    The actual video files (e.g. mkv) are in directories which do not have colons in their names.

    After the scraping, the scraper adds all my movies and tvshows to the database/library. As I cannot find those directories in any folder of kodi after the scraping process, I guess that they are written directly into this file:
    /storage/.kodi/userdata/Database/MyVideos99.db

    So the issue might occur whenever I export my library to file (for backup reasons). Then those names (containing colons) are written out of this MyVideos99.db to file, thus also strictly overtaking the colons into file names. So if my assumptions are correct, there is nothing I could "rename" to follow your suggestion.
    If it is like I described, then LibreELEC should simply exchange all colons with dashes or "%3A" before writing to disk. And then it should obviosly also support those dashes / %3A when importing the library again on another machine, thus exchanging them into colons before writing/importing in the new MyVideos99.db.
    Or simply do always dashes/%3A after scraping and simply exchange the output/displaying in GUI to colons (would be safest and cleanes solution).

    So it is not about the actual directories containing the video files, which I always transfer myself from windows to the external drive connected to my odroid/raspberry (obviosly not containing any colons in names).

    Edited once, last by infinity85 (October 5, 2016 at 2:46 PM).

    • Official Post

    Hmm... It's a bug I think, unless the Kodi dev team has a remarkable explanation.

    I always use separate files myself when exporting the video library. Then, I can also choose whether to export fanart and stuff. With the single export file option, everything just gets exported without asking. Hope they fixed that as well in Krypton.

    But the single export option is weird. Example of the video library export via 'Single file':
    Fanart in export folder: Airwolf:_The_Movie_1984-fanart.jpg
    Movie in the NAS folder: /movies/Airwolf/Airwolf.hevc

    Whenever I do an export via Separate files, fanart is exported correctly: /movies/Airwolf/Airwolf-poster.jpg for example.
    I find that odd.

    I didn't know that filenames are different between separate and single file export.

    Your Airwolf example confirms it. The filename "Airwolf:_The_Movie_1984-fanart.jpg" is generated out of the scrapers database (in this case apparently you aren't using thetvdb.com) movie name. It is understandable that it takes this name, but it should not do it for exporting the library backup.

    Well, I've written about this some weeks ago in the kodi.tv forum. But nobody paid attention and theres also no bug/issues section in at forum.kodi.tv and none at their github either. How can I submit them this issue?