ext HDs not mounted after 11.01 update

Quote from chewitt

Copying binaries between different LE major versions or from other distros occasionally works but is never guaranteed. I'm going to have a look at Ubuntu 18.04 and what it's doing (need to find something with an Intel CPU for this though). I'm wondering if they have udev rules that run ntfs or they aliased a script that runs ntfsfix to fsck.ntfs. I'm 100% sure the util-linux tools support FAT/VFAT but do not support (and never have supported) NTFS; so the output from the fsck command needs explaining.

As far as the fsck output on Ubuntu 18.04 is concerned: that's my bad...

frankvw@dellfvw:~$ ls -l /sbin/fsck.ntfs
lrwxrwxrwx 1 root root 12 Sep 17  2018 /sbin/fsck.ntfs -> /bin/ntfsfix

Sorry to have missed that.

The version of ntfsfix on U18.04 is ntfsfix v2017.3.23. A quick grep for fsck or ntfsfix in /lib/udev/rules yields no results.

Ahh, that explains then. Thanks for clarifying. We're contemplating where we go with changes..

Quote from chewitt

Ahh, that explains then. Thanks for clarifying. We're contemplating where we go with changes..

Personally I'd be happy just with the /bin/ntfsfix binary (and the /sbin/fsck.ntfs symlink to it) being restored to LE 11.0.4!

The ntfsfix command could be linked to a button on a remote control or, since ntfsfix may not be able to solve all NTFS issues but I have yet to see it create a new one, and it's pretty quick to run, too, it might make sense to automate it. All we'd need would be a udev rule (say, /lib/udev/rules.d/99-usb-harddisk.rules) with something in it like

ACTION=="add", SUBSYSTEM=="block", KERNEL=="sd*", ATTRS{idVendor}=="YOUR_VENDOR_ID", ATTRS{idProduct}=="YOUR_PRODUCT_ID", RUN+="/sbin/somescript.sh %k"

The script would contain something like

DEVICE=$1
if blkid -t TYPE="ntfs" "/dev/$DEVICE"; then
for PARTITION in $(ls /dev/$DEVICE*); do
ntfsfix "$PARTITION"
done
fi

This isn't exactly copy-and-paste stuff of course, not to mention that it could be improved with an axe, but you see where I'm going with this.

On second thought, a separate rule file probably isn't necessary; I've had a quick look and an addition to /lib/udev/rules.d/95-udevil-mount.rules would probably do the trick.

[EDIT: or maybe it's 60-persistent-storage.rules - I'm not too sure at this point. I've never done a deep dig into udev rules before.]

Anyway - happy contemplating! Looking forward to the outcome.

In closing, thanks for all your help! It's greatly appreciated. You guys rock!

Don't get too excited about ntfsfix, I did a quick test on a corrupted NTFS filesystem, removing the dirty bit, and as expected it didn't help with the errors and the ntfsdriver set the dirty bit again after mounting it because the filesystem had issues.

so long,

Hias

Quote from HiassofT

Don't get too excited about ntfsfix, I did a quick test on a corrupted NTFS filesystem, removing the dirty bit, and as expected it didn't help with the errors and the ntfsdriver set the dirty bit again after mounting it because the filesystem had issues.

You're right: ntfsfix is severly limited. It always has been. In fact, the manpage specifically states that "ntfsfix is NOT a Linux version of chkdsk. It only repairs some fundamental NTFS inconsistencies, resets the NTFS journal file and schedules an NTFS consistency check for the first boot into Windows."

However, the issue under discussion here is that a simple unclean dismount (i.e. pulling the plug on an USB device without first dismounting it cleanly, having a power failure or other crash, or simply moving a cable with a dodgy plug or connector) will leave the "dirty bit" set on the NTFS partition regardless of whether or not anything has been corrupted. LE then refuses to remount it, ever.

As of LE11.0.3 there's no ntfsfix at all included to reset the dirty bit. That means that as soon as you have even the slightest hiccup with and NTFS disk on LE, you need to plug the HD into either a Linux box that has ntfsfix on it, or into a Windows box and run Chkdsk. That's a bit hard to live with, and simply restoring ntfsfix to LE will go a long way to fixing that problem.

If you have real NTFS corruption problems, no amount of ntfsfix will help you and only running Chkdsk on a Windows box will sort it out. But that's a Linux issue, not an LE issue. NTFS support for Linux has never been great (although these days it's sufficient for all regular purposes) but the lack of a proper Chkdsk alternative that can repair a corrupted NTFS has been a problem since NTFS support was first introduced in Linux.

As petediscrete already suggested above, testdisk may be a better alternative for ntfsfix. However, for daily LE use ntfsfix will be sufficient in most cases and infinitely better than nothing at all (which LE11.0.3 currently has).

Quote from frankvw

You're right: ntfsfix is severly limited. It always has been. In fact, the manpage specifically states that "ntfsfix is NOT a Linux version of chkdsk. It only repairs some fundamental NTFS inconsistencies, resets the NTFS journal file and schedules an NTFS consistency check for the first boot into Windows."

However, the issue under discussion here is that a simple unclean dismount (i.e. pulling the plug on an USB device without first dismounting it cleanly, having a power failure or other crash, or simply moving a cable with a dodgy plug or connector) will leave the "dirty bit" set on the NTFS partition regardless of whether or not anything has been corrupted. LE then refuses to remount it, ever.

As of LE11.0.3 there's no ntfsfix at all included to reset the dirty bit. That means that as soon as you have even the slightest hiccup with and NTFS disk on LE, you need to plug the HD into either a Linux box that has ntfsfix on it, or into a Windows box and run Chkdsk. That's a bit hard to live with, and simply restoring ntfsfix to LE will go a long way to fixing that problem.

If you have real NTFS corruption problems, no amount of ntfsfix will help you and only running Chkdsk on a Windows box will sort it out. But that's a Linux issue, not an LE issue. NTFS support for Linux has never been great (although these days it's sufficient for all regular purposes) but the lack of a proper Chkdsk alternative that can repair a corrupted NTFS has been a problem since NTFS support was first introduced in Linux.

As petediscrete already suggested above, testdisk may be a better alternative for ntfsfix. However, for daily LE use ntfsfix will be sufficient in most cases and infinitely better than nothing at all (which LE11.0.3 currently has).

Or else keep NTFS systems away from the Linux party. It really is a matter of re education for Windows users. With shares and other network solutions does anyone really need to be using an NTFS disc on LE anyone. And for those who do, with storage being so cheap these days a simple copy over of the contents of the NTFS disk to EXT4 would suffice before hooking up to the LE machine. Honestly what can’t be done in Linux that can be done in Windows. I prefer spending my time “consuming” my media instead of watching LE consuming my NTFS file system 😂

And still there is a bit of a difference in marketshares between desktop OS, Microsoft 72% and Linux less than 3%, just face it, most people isnt ready for Linux in their daily routines, most of what they do is done with Windows.

I'm currently looking into a different approach: let users choose to mount NTFS partitions read-only https://github.com/LibreELEC/LibreELEC.tv/pull/8308 - this would hopefully solve the issue of easy corruption in case of unclean shutdowns.

so long,

Hias

Quote from petediscrete

Or else keep NTFS systems away from the Linux party. It really is a matter of re education for Windows users. With shares and other network solutions does anyone really need to be using an NTFS disc on LE anyone. And for those who do, with storage being so cheap these days a simple copy over of the contents of the NTFS disk to EXT4 would suffice before hooking up to the LE machine. Honestly what can’t be done in Linux that can be done in Windows. I prefer spending my time “consuming” my media instead of watching LE consuming my NTFS file system 😂

I'm with you on that. In fact, when I come into power and become Ruler of All the World, the first thing I'll do is to banish Windows and NTFS from the face of the earth and make Linux skills a compulsory part of general education, and the Redmond crowd will be among the first against the wall. However, until that happy day comes around, we need to face the reality that just about every USB device sold comes with NTFS on it, and John & Jane Consumer have no idea what ext4 is, much less how to convert their newly bought USB harddisk to it. They simply plug it in and expect it to work. And most of the time it does. Except with LE, right now. And that's not a Good Thing. But it can be fixed, as per the above discussion, and it's not all that complicated. So my vote is to leave the Windows vs. Linux (and related technologies) holy War for what it is, face facts, and deal with it.

Quote from HiassofT

I'm currently looking into a different approach: let users choose to mount NTFS partitions read-only https://github.com/LibreELEC/LibreELEC.tv/pull/8308 - this would hopefully solve the issue of easy corruption in case of unclean shutdowns.

so long,

Hias

If that works, I could live with it. However,it wouldn't let anyone add media through SMB or sftp, and it would break add-ons like Transmission, though. Not a big issue for me but it might be for many others. However, compromises being what they are it's better than the current unreliability of NTFS devices.

Quote from MatteN

And still there is a bit of a difference in marketshares between desktop OS, Microsoft 72% and Linux less than 3%, just face it, most people isnt ready for Linux in their daily routines, most of what they do is done with Windows.

Remember the critique, lies, damn lies and statistics. Why do so many people here want to use LE I wonder 😂

Quote from petediscrete

Remember the critique, lies, damn lies and statistics. Why do so many people here want to use LE I wonder 😂

That's the point. They do want to use LE because it's easy, convenient and does not require any computer skills beyond the ability to follow step-by-step instructions on how to copy a distro image to an SD card or USB stick. They are, however, stuck with USB storage devices that come out of the box with NTFS on it, and they just expect that to work.

As one of Debian's Elder Geeks put it (about 25 years ago now): "If the user points the gun at his foot and pulls the trigger, it is our job to ensure the bullet gets where it's supposed to." It's much the same here: people install LE for its simplicity of use, then they plug in a USB storage device that comes right out of the box, and they aren't even aware that it comes with NTFS on it and the problems associated therewith, much less are they interested in converting it to ext4. They plug and expect play.

Now I'll grant you that there are limits to catering to all aspects of that expectation, but in this case LE's finicky response to uncleanly dismounted NTFS devices and a lack of a means to fix it within LE is serious enough for the uneducated user to consider LE broken in this regard. You and I understand what's going on, but from a UX standpoint (I'm trying to avoid using "the great unwashed masses" as a descriptive term here) this is a problem.

Which in my opinion can be fixed fairly easily by restoring ntfsfix to the next LE distro image and (as I would prefer) hooking it into a udev rule that automatically runs it on any NTFS partition at mount time.

Perhaps testdisk could also be added to the next image for cases in which ntfsfix is insufficient. I see that tools like udevadm (which are not exactly intended for Joe & Jane Average User) is there, so why not testdisk? It'll get used more often than the udev tools, would be my guess.

It still comes down to education. People shouldn't be uncleanly shutting down any device. Put LE on a ups to survive power outages and properly use the shutdown menu to power LE off. No more problem.

Quote from pauljw

It still comes down to education. People shouldn't be uncleanly shutting down any device. Put LE on a ups to survive power outages and properly use the shutdown menu to power LE off. No more problem.

You do have a point. Removing all the safeties from all equipment, removing insulation from live wires, removing the brake lights from cars, etc. will educate everyone very, very quickly and teach them not to do anything stupid with equipment that have could potential unwanted effects. But is that really the way to go?

Let's face reality here: people are going to install LE because of its combination of simplicity and rich features, then plug any off-the-shelf USB storage device in it and expect it to work. And those things will be dismounted uncleanly sooner or later, because accidents happen. If LE then can no longer use the device their UX is impacted and they'll blame LE for not doing what they can do with their average Windoze or Linux box every single day.

If that can be remedied fairly simply (and it can as per the above discussion) I believe that's worth doing. But that's just me.

Hi and thanks for an interesting thread : I had a power outage yesterday, and now 2/3 disks don't want to start. Have run them in Win10 without problems after chkdsk. Today I have LE 11.0.4 generic on a PC which worked flawlessly. Any ideas? Mvh Thore from Sweden

Quote from 2469

Hi and thanks for an interesting thread : I had a power outage yesterday, and now 2/3 disks don't want to start. Have run them in Win10 without problems after chkdsk. Today I have LE 11.0.4 generic on a PC which worked flawlessly. Any ideas? Mvh Thore from Sweden

Until ntfsfix is included again with LE using a Windoze box to clear the 'dirty' bit on the NTFS partitions is your only option, I'm afraid.

Quote from Reddirt

I'm surprised at the attention this issue attracted. I for one vote to use the ext4 system for all things LE and be done with it.

As I said above, people want to use LE because it's easy, convenient and does not require any computer skills beyond the ability to follow step-by-step instructions on how to copy a distro image to an SD card or USB stick. They are, however, stuck with USB storage devices that come out of the box with NTFS on it, and they just expect that to work.

Which, from a UX point of view, is not all that an unreasonable standpoint.

At this point in time any power failure or accidental unplugging of an NTFS based storage device will stop it from working, and LE has no option to fix that. You need a Linux box with ntfsfix on it just to get the partition to mount again, and a Windoze box to fix any real NTFS problems.

Keep in mind that, to all intents and purposes, all USB storage devices come out of the box with NTFS on it.

So as far as user experience goes, shaky NTFS support is not a good thing. Add to that the fact that this could be easily fixed to some degree (including ntfsfix again and having it triggered by a udev rule when an NTFS device is plugged in and I agree with you: we shouldn't still be just discussing this.

For those with an IT background the choice between NTFS and ext4 is a no-brainer. And I have reformatted my NTFS media tank to ext4 and now all is well. But for every person who knows what ext4 is, why NTFS should be avoided in a *ix environment and how to replace NTFS with ext4, there is a gazillion "normal" people out there who just buy a device, plug it in, an expect it to work. They reason that since it does work "everywhere else" (i.e. on every Windoze box) why won't it on LE? It's not an unreasonable question, in all fairness.

If we want to reserve LE for the IT-savvy elite then yes, let's ignore NTFS and consider its shortcomings the punisment for using it. But unless I'm very much mistaken, that's not what LE is all about, is it?

Quote from frankvw

Until ntfsfix is included again with LE using a Windoze box to clear the 'dirty' bit on the NTFS partitions is your only option, I'm afraid.

The ntfsfix utility has never been shipped with LE, and I've closed the pull-request that proposed adding it because it's an incomplete tool. From a support and data integrity perspective it's better to inconvenience users by forcing them to use the right tool (chkdsk.exe on Windows) than not-quite fixing the problems a filesystem has and risking data loss.

Please test with the latest LibreELEC 12 nightly builds, I recently noticed and fixed an issue which may have led to filesystem writes (updating last file/directory access timestamps) even if you only read from NTFS partitions

NTFS filesystems are mounted with incorrect options · Issue #8474 · LibreELEC/LibreELEC.tv

Automounting NTFS drives via udevil should mount NTFS filesystems with nosuid,noexec,nodev,noatime,fmask=0133,uid=0,gid=0 according to udevil.conf but on LE12…

github.com

In addition to that you can now configure LE to mount NTFS partitions read-only by default which should further help with accidential filesystem corruption in case of unclean shutdown, powerloss etc. If you want to change files eg via SMB share or shell just remount it read-write, eg via mount -o remount,rw /var/media/YOUR-NTFS-DRIVE and remount it again read-only after changing stuff via mount -o remount,ro ...:

Copy /etc/udevil/udevil.conf to /storage/.config/udevil.conf and change the default_options_ntfs line of /storage/.config/udevil.conf to contain the ro option. i.e. add , ro at the end of it:

LibreELEC:~ # cp /etc/udevil/udevil.conf /storage/.config/udevil.conf
LibreELEC:~ # nano /storage/.config/udevil.conf

...
default_options_ntfs      = nosuid, noexec, nodev, noatime, fmask=0133, uid=$UID, gid=$GID, ro
...

so long,

Hias