HDD corruption

    I'm hopefully onto my last problem area. The 6TB I'll be using on the RPi4 has been corrupted twice and I'm trying to find out why. I found this (https://www.linuxquestions.org/questions/linu…ved-4175536517/) thread which indicates that when Linux mounts a hibernated disk this can cause corruption. I've set my disk to go to sleep after 15 minutes inactivity so its entirely possible that it was that state when I took it off the Windows PC and connected onto the Pi.

    What does anyone here think? Likely or not?

    The 6TB disk is formatted to NTFS, moving to Ext4 is not a viable proposition, but I could move to ExFAT if people think that would be more stable.

    Windows does not hibernate external drives. And even if it would, Linux would refuse to mount them.

    The Linux NTFS driver is well tested and it is extremely improbable the simple uses cases a Kodi movie archive involves would trigger any overseen bug.

    Possible causes for corruption are:

    • Unplugging the drive during write operations.
      Solution: Make sure to use the safely remove feature of Kodi and Windows, wait for 3 seconds after confirmation from the OS before you unplug.
    • Insufficient power for write operations.
      Solution: Get a powered hub or a drive with an external power supply (due to the size of 6TB I guess the disk already has the later)
    • Vibration
      Solution: Don't put the drive on your Tumbler
    • Disk failure
      Solution: Check disk h ealth with smartcontrol or one of the many tools for Windows

    Things to check:

    Does it corrupt after being remove from Windows, from libreelec,

    or during operation on one of the systems?

    How does corruption show up in the kernel log? (dmesg)

    PS: Why does the forum censor the word h ealth

    Thanks - to answer your comments / questions

    "Windows does not hibernate external drives."

    True, however, Seagate provide a utility so you can tell the HDD to go to sleep after inactivity - I've set it to 15 minutes so having left it to copy 4TB overnight it was sleeping when I came back, turned the PC off and disconnected.

    "Unplugging the drive during write operations."

    Windows always tell it to eject or close PC down. RPi4 power off machine, but it shouldn't really be writing to the disk anyway - just reading videos.

    "Insufficient power for write operations."

    I'm pretty sure this was the first cause - power block with duff socket.That was fixed but I suppose its possible that the Pi was overloaded, couldn't read the OS from SSD and tried to create a new install on the HDD.

    "Vibration"

    Being a brit a tumbler is a glass (one of those things you drink out of) so might have trouble balancing the HDD on it

    "Disk failure"

    I'm about 23% through a full scann (HDDScan) right now - no bad sectors yet.

    "Does it corrupt after being remove from Windows, from libreelec,

    or during operation on one of the systems?"

    Good question - I only found it (both times) when I plugged back into my Windows PC - massive index corruption both times.

    "How does corruption show up in the kernel log? (dmesg)"

    No idea - the SSD where I presume it would be stored has been flashed several times since it happened.

    "PS: Why does the forum censor the word "

    Wish I knew - it also censors f00d

    My PS: any views on ExFAT vs NTFS?

    Quote from LybsterKodi

    True, however, Seagate provide a utility so you can tell the HDD to go to sleep after inactivity - I've set it to 15 minutes so having left it to copy 4TB overnight it was sleeping when I came back, turned the PC off and disconnected.

    yup, the disk fall to sleep, but, even when the Windows box hibernated [2], the disk might be still seen as mounted - I guess -.

    "disk sleep" is different from "disk unmounted"

    Try, if the Window has hibernated, to start the box WITH the disk still plugged and then disconnect the disk via right clicking on the USB icon in the right task bar area and something like "safely remove ..." or similar ... [1]

    reconnect the disk to see if it's still seen as corrupt

    [1]

    dito suggested for LE:

    either do it on a command line via "umount /dev/sd? " with a maybe one or two prior "sync" command(s)

    or

    shut down LE via power menu

    then the shutdown process cleanly unmounts the disk for you

    [2]

    if you're running your windows on an SSD you do NOT have much benefits, apart from "could carry on the work where you left it (apps still running), doing hibernation.

    search for "SSD wear leveling"

    Quote from LybsterKodi

    My PS: any views on ExFAT vs NTFS?

    there exists a comparison of filesystems in wikipedia, search for it

    Edited 3 times, last by GDPR-7 (August 17, 2022 at 9:46 PM).

    Quote from JoeAverage

    there exists a comparison of filesystems in wikipedia, search for it

    I did a lot of web searching before I posted here. The vast majority do a bit of a pro & con list with zero evidence to support anything. Wikipedia have a very detailed entry if I was interested in technical specs. I'm not.

    Quote from LybsterKodi

    I did a lot of web searching before I posted here. The vast majority do a bit of a pro & con list with zero evidence to support anything. Wikipedia have a very detailed entry if I was interested in technical specs. I'm not.

    If you are looking for cross platform compatibility, use NTFS. Both NTFS and exFAT are not native filesystems for Linux and NTFS implementation is older and more mature.

    Filesystem performance is less critical than device performance.

    Turn off hibernation in disk settings. If disks shutdowns on its own while being mounted, it can only cause mess. Disk should be stopped by OS and not other way. If your storage thinks that it is smarter than OS, you have to use synchronous file system without journal on it.

    Edited 2 times, last by tokul (August 18, 2022 at 8:41 PM).

    NTFS read has existed in the kernel for aeons, but write capabilities came from Paragon recently as part of a major update. It's a mature driver in the sense that Paragon have been using (and selling) a version of the driver for a long time; but equally it's rather new as their code was thoroughly bike-shedded during the submission process and what was finally merged is somewhat different from the original submission.

    IMHO.. always put big storage drives in a NAS box in the network and use Ethernet not Sneakernet to move files around between devices. The expense is worth it if you value reliability of data in use (good filesystems, no USB anywhere) and survivability of your data (put the drives in an appropriate redundant disk format).

    Quote from tokul

    somebody ran out of patience and attacked original poster.

    I'm convinced the "somebody" doesn't "attack original poster", but "attacks" the OPs wrong attitude what prevents to find the answer for his question(s) himself.

    please learn to read the differences in the sentences ! 8)

    Quote from JoeAverage

    okay, next time please mention in your questions that your're not really interesting in answers, safes some time here ...

    The question was

    "What does anyone here think? Likely or not?".

    If you are interested in answering that I'm interested.

    Quote from tokul

    If you are looking for cross platform compatibility, use NTFS. Both NTFS and exFAT are not native filesystems for Linux and NTFS implementation is older and more mature.

    Turn off hibernation in disk settings. If disks shutdowns on its own while being mounted, it can only cause mess. Disk should be stopped by OS and not other way. If your storage thinks that it is smarter than OS, you have to use synchronous file system without journal on it.

    Thanks.

    "Filesystem performance is less critical than device performance."

    fully agree with that

    Quote from JoeAverage

    I'm convinced the "somebody" doesn't "attack original poster", but "attacks" the OPs wrong attitude what prevents to find the answer for his question(s) himself.

    please learn to read the differences in the sentences ! 8)

    Interesting. I hadn't realised that asking questions here was forbidden. After all if I'm to find out the answer for myself I shouldn't be asking anyone. I was also unaware that you were watching and discounted my efforts googling for information and trying physical tests to see if I could recreate the problem.

    As a matter of interest. Did you look at the Wikipedia article, and if so how would it have helped answer my question? I ask because I'm always interested in learning - even at 70.

    Quote from chewitt

    The expense is worth it

    I would have agreed with you totally whilst I was still working and running a small IT Department. As a pensioner who had a large percentage of his savings disintegrated by the Credit Crunch my attitude has changed a bit <G>

    Edited 3 times, last by LybsterKodi: Merged a post created by LybsterKodi into this post. (August 19, 2022 at 7:30 AM).