Kodi scrapes duplicate Movies pointing to the same files

Hello!

I have recently switched over to LibreELEC from OSMC (Raspberry Pi 4) and have had no issues so far. Now though, when I wanted to scrape my movie library which is shared via SMB from my PC, I encountered an annoying issue. The scraping works fine and everything is added, though it is added TWICE! I have two entries of every movie title, and both entries work and will play the same file...

I have tried removing the source and re-adding it, changing content type of source back and forth, I have even tried using another scraper...

When it scrapes, looking out the "progress notification", it seems to first scrape once and get everything, and then re-run the scraping, scraping everything once more! This is really odd as it works fine for my TV Shows which are shared via the same SMB but on another path...

MY SMB:

• Media Library
  • Movies
    • Movie1 file
    • movie2 file
    • Movie 3 folder
      • movie 3 file
    • ...
  • TV Shows
    • Show 1
      • Season 1
        • ep1
    • Show2
    • ...

In other words, my movies directory contains both single files and folders with files in them. Because of this I use recursive scan in the settings.

Here is my log file: https://filebin.ca/5Hx9QnIH6eUx/kodi.log (I tried pastebin, paste.kod.tv and paste.ubuntu.com but they would not have my paste... To large?)

any ideas?

Quote from Klojum

Multiple entries to the same video would suggest more than 1 source linking to the same video.

Have you checked all paths via the Information page of those videos?

Please note that any deviation, even an extra space, in a path is a different path to Kodi.

Hello and thanks for your comment.

You are right. There seems to be a devience in the path.

It is

*smb_path*/Media Library/Movies/Moviefile.mp4

vs

*smb_path*/Media Library/Movies//Moviefile.mp4

This makes kind of sense because I used to have a source that ended in double front slash (//) by mistake. Though I edited this and changed it to single front slash as the double front slash caused files not in the root of /Movies to not be scraped. Is there any manual way of deleting this source as it is not appearing in the UI? It appears to be correct in sources.xml (single front slash)

Thank you

Quote from chewitt

Have a look at /storage/.kodi/userdata/sources.xml

if you read my reply to Klojum you will see I already did that. I did it again though. Here is its full contents:

<sources>
    <programs>
        <default pathversion="1"></default>
    </programs>
    <video>
        <default pathversion="1"></default>
        <source>
            <name>TV Shows</name>
            <path pathversion="1">smb://192.168.1.111/Media Library/TV Shows/</path>
            <allowsharing>true</allowsharing>
        </source>
        <source>
            <name>Movies</name>
            <path pathversion="1">smb://192.168.1.111/Media Library/Movies/</path>
            <allowsharing>true</allowsharing>
        </source>
    </video>
    <music>
        <default pathversion="1"></default>
        <source>
            <name>Music</name>
            <path pathversion="1">/storage/music/</path>
            <allowsharing>true</allowsharing>
        </source>
    </music>
    <pictures>
        <default pathversion="1"></default>
        <source>
            <name>Pictures</name>
            <path pathversion="1">/storage/pictures/</path>
            <allowsharing>true</allowsharing>
        </source>
    </pictures>
    <files>
        <default pathversion="1"></default>
    </files>
    <games>
        <default pathversion="1"></default>
    </games>
</sources>

Quote from Klojum

Removing a faulty source from the sources.xml file is one thing, removing faulty entries from the video database is not that easy.

For the non-SQL person I'd say: do a video export, and reimport your video collection into a new video database.

For the SQL people, it will still be a challenge to pick out all these entries.

The 1st way is less messy.

Would you suggest export + reimport over simply deleting the database and rescanning? Also, I have seen how to export the library as xml, but how do I import it?

So I actually got it to work. When Klojum mentioned the SQL people thing it got me thinking. I do know SQL so I thought I would give it a shot.

I copied the MyVideos116.db file to my local machine where I had sqlite3 installed already (CLI).

From here I used SQLite Viewer to view all entries in the movie table. I quickly noticed at what idMovieI started getting duplicates so with a simple sql query of

delete from movie where idMovie > 67;

I got rid of all the duplicates (of which all had .../Movies//... in their c22 column (Path to file)). After this I simply transfered the modified .db file to my Pi and replaced the existing one. Now I have one of every movie and updating the library does not bring back the duplicates!

Thanks for the help everyone! I hope this helps someone else with a similar issue in the future. My query for deleting the duplicated entries was rather simple as, fortunately for me, they were all one after another. If they were all mixed up and I would have had to write a query that detects and deletes entries with // in the path, that would have become a bit more complicated

Quote from donbrew

Doesn't the // indicate an NFS path? Or, is it SMB? Anyway, / is for one of them and // is for the other. Or, maybe it is just the way my data paths are set up; one is set to the entire drive and the other is set to a directory on the drive.

This happens to me when I switch between smb and nfs, and refresh. Evidently the old (nfs) entries stay in the library until you start a new library or manually delete them.

I was switching back and forth experimenting, I ended up deciding they both work well.

I can access smb on a windows machine using \\COMPUTER-NAME
maybe that is what u mean? The double frontslash in my case was purely accidental. smb://COMPUTER-NAME I think is identical to \\COMPUTER-NAME