HDD corruption

LybsterKodi · August 17, 2022 at 1:39 PM

I'm hopefully onto my last problem area. The 6TB I'll be using on the RPi4 has been corrupted twice and I'm trying to find out why. I found this (https://www.linuxquestions.org/questions/linu…ved-4175536517/) thread which indicates that when Linux mounts a hibernated disk this can cause corruption. I've set my disk to go to sleep after 15 minutes inactivity so its entirely possible that it was that state when I took it off the Windows PC and connected onto the Pi.

What does anyone here think? Likely or not?

The 6TB disk is formatted to NTFS, moving to Ext4 is not a viable proposition, but I could move to ExFAT if people think that would be more stable.

macros · August 17, 2022 at 2:23 PM

Windows does not hibernate external drives. And even if it would, Linux would refuse to mount them.

The Linux NTFS driver is well tested and it is extremely improbable the simple uses cases a Kodi movie archive involves would trigger any overseen bug.

Possible causes for corruption are:

Unplugging the drive during write operations.
Solution: Make sure to use the safely remove feature of Kodi and Windows, wait for 3 seconds after confirmation from the OS before you unplug.
Insufficient power for write operations.
Solution: Get a powered hub or a drive with an external power supply (due to the size of 6TB I guess the disk already has the later)
Vibration
Solution: Don't put the drive on your Tumbler
Disk failure
Solution: Check disk h ealth with smartcontrol or one of the many tools for Windows

Things to check:

Does it corrupt after being remove from Windows, from libreelec,

or during operation on one of the systems?

How does corruption show up in the kernel log? (dmesg)

PS: Why does the forum censor the word h ealth

LybsterKodi · August 17, 2022 at 4:11 PM

Thanks - to answer your comments / questions

"Windows does not hibernate external drives."

True, however, Seagate provide a utility so you can tell the HDD to go to sleep after inactivity - I've set it to 15 minutes so having left it to copy 4TB overnight it was sleeping when I came back, turned the PC off and disconnected.

"Unplugging the drive during write operations."

Windows always tell it to eject or close PC down. RPi4 power off machine, but it shouldn't really be writing to the disk anyway - just reading videos.

"Insufficient power for write operations."

I'm pretty sure this was the first cause - power block with duff socket.That was fixed but I suppose its possible that the Pi was overloaded, couldn't read the OS from SSD and tried to create a new install on the HDD.

"Vibration"

Being a brit a tumbler is a glass (one of those things you drink out of) so might have trouble balancing the HDD on it

"Disk failure"

I'm about 23% through a full scann (HDDScan) right now - no bad sectors yet.

"Does it corrupt after being remove from Windows, from libreelec,

or during operation on one of the systems?"

Good question - I only found it (both times) when I plugged back into my Windows PC - massive index corruption both times.

"How does corruption show up in the kernel log? (dmesg)"

No idea - the SSD where I presume it would be stored has been flashed several times since it happened.

"PS: Why does the forum censor the word "

Wish I knew - it also censors f00d

My PS: any views on ExFAT vs NTFS?

GDPR-7 · August 17, 2022 at 8:49 PM

Quote from LybsterKodi

True, however, Seagate provide a utility so you can tell the HDD to go to sleep after inactivity - I've set it to 15 minutes so having left it to copy 4TB overnight it was sleeping when I came back, turned the PC off and disconnected.

yup, the disk fall to sleep, but, even when the Windows box hibernated [2], the disk might be still seen as mounted - I guess -.

"disk sleep" is different from "disk unmounted"

Try, if the Window has hibernated, to start the box WITH the disk still plugged and then disconnect the disk via right clicking on the USB icon in the right task bar area and something like "safely remove ..." or similar ... [1]

reconnect the disk to see if it's still seen as corrupt

[1]

dito suggested for LE:

either do it on a command line via "umount /dev/sd? " with a maybe one or two prior "sync" command(s)

or

shut down LE via power menu

then the shutdown process cleanly unmounts the disk for you

[2]

if you're running your windows on an SSD you do NOT have much benefits, apart from "could carry on the work where you left it (apps still running), doing hibernation.

search for "SSD wear leveling"

Quote from LybsterKodi

My PS: any views on ExFAT vs NTFS?

there exists a comparison of filesystems in wikipedia, search for it

LybsterKodi · August 18, 2022 at 4:10 PM

Quote from JoeAverage

there exists a comparison of filesystems in wikipedia, search for it

I did a lot of web searching before I posted here. The vast majority do a bit of a pro & con list with zero evidence to support anything. Wikipedia have a very detailed entry if I was interested in technical specs. I'm not.

GDPR-7 · August 18, 2022 at 7:24 PM

Quote from LybsterKodi

Wikipedia have a very detailed entry if I was interested in technical specs. I'm not.

okay, next time please mention in your questions that your're not really interesting in answers, safes some time here ...

tokul · August 18, 2022 at 8:30 PM

Quote from LybsterKodi

I did a lot of web searching before I posted here. The vast majority do a bit of a pro & con list with zero evidence to support anything. Wikipedia have a very detailed entry if I was interested in technical specs. I'm not.

If you are looking for cross platform compatibility, use NTFS. Both NTFS and exFAT are not native filesystems for Linux and NTFS implementation is older and more mature.

Filesystem performance is less critical than device performance.

Turn off hibernation in disk settings. If disks shutdowns on its own while being mounted, it can only cause mess. Disk should be stopped by OS and not other way. If your storage thinks that it is smarter than OS, you have to use synchronous file system without journal on it.

GDPR-7 · August 18, 2022 at 8:53 PM

Quote from tokul

Both NTFS and exFAT are not native filesystems for Linux and NTFS implementation is older and more mature.

sure ?

both are in-kernel FS's

the exfat implementation is -AFAIK- from Samsung, where M$ exfat was thrown away last (?) year.

The New Microsoft exFAT File-System Driver Has Landed In Linux 5.7 - Phoronix

IIRC, a similar story happend with ntfs...

**chewitt** · August 19, 2022 at 1:52 AM

NTFS read has existed in the kernel for aeons, but write capabilities came from Paragon recently as part of a major update. It's a mature driver in the sense that Paragon have been using (and selling) a version of the driver for a long time; but equally it's rather new as their code was thoroughly bike-shedded during the submission process and what was finally merged is somewhat different from the original submission.

IMHO.. always put big storage drives in a NAS box in the network and use Ethernet not Sneakernet to move files around between devices. The expense is worth it if you value reliability of data in use (good filesystems, no USB anywhere) and survivability of your data (put the drives in an appropriate redundant disk format).

tokul · August 19, 2022 at 4:33 AM

Quote from JoeAverage

sure ?
both are in-kernel FS's
the exfat implementation is -AFAIK- from Samsung, where M$ exfat was thrown away last (?) year.
https://www.phoronix.com/news/Linux-5.7-New-exFAT-Lands
IIRC, a similar story happend with ntfs...

The only reason I posted on this thread is cause somebody ran out of patience and attacked original poster.

Please learn that having last post in the thread does not make you a winner.

GDPR-7 · August 19, 2022 at 5:16 AM

Quote from tokul

somebody ran out of patience and attacked original poster.

I'm convinced the "somebody" doesn't "attack original poster", but "attacks" the OPs wrong attitude what prevents to find the answer for his question(s) himself.

please learn to read the differences in the sentences !

LybsterKodi · August 19, 2022 at 7:15 AM

Quote from JoeAverage

okay, next time please mention in your questions that your're not really interesting in answers, safes some time here ...

The question was

"What does anyone here think? Likely or not?".

If you are interested in answering that I'm interested.

Quote from tokul

If you are looking for cross platform compatibility, use NTFS. Both NTFS and exFAT are not native filesystems for Linux and NTFS implementation is older and more mature.
Turn off hibernation in disk settings. If disks shutdowns on its own while being mounted, it can only cause mess. Disk should be stopped by OS and not other way. If your storage thinks that it is smarter than OS, you have to use synchronous file system without journal on it.

Thanks.

"Filesystem performance is less critical than device performance."

fully agree with that

Quote from JoeAverage

I'm convinced the "somebody" doesn't "attack original poster", but "attacks" the OPs wrong attitude what prevents to find the answer for his question(s) himself.
please learn to read the differences in the sentences !

Interesting. I hadn't realised that asking questions here was forbidden. After all if I'm to find out the answer for myself I shouldn't be asking anyone. I was also unaware that you were watching and discounted my efforts googling for information and trying physical tests to see if I could recreate the problem.

As a matter of interest. Did you look at the Wikipedia article, and if so how would it have helped answer my question? I ask because I'm always interested in learning - even at 70.

Quote from chewitt

The expense is worth it

I would have agreed with you totally whilst I was still working and running a small IT Department. As a pensioner who had a large percentage of his savings disintegrated by the Credit Crunch my attitude has changed a bit <G>

**Da Flex** · August 19, 2022 at 10:36 AM

Not LE specific anymore, so I move the thread to Off-Topic.