ext HDs not mounted after 11.01 update

    Yup. And the utility that's been dropped with NTFS-3G is "mkfs.ntfs" not "fsck.ntfs" .. and if you do some reading around this topic there is a conspicuous absence of fsck or similar tools for NTFS on Linux. There's an older "ntfsfix" which simply clears the dirty flag; which will probably allow the drive to mount in some/most cases, but risks causing damage in scenarios when proper chkdsk.exe is required to resolve a more serious problem with the filesystem (and due to that, isn't something we'd try to use). It looks like we (and the the rest of the Linux using world) is needing Paragon to clean-up and submit their fsck tool code.

    Quote from chewitt

    Yup. And the utility that's been dropped with NTFS-3G is "mkfs.ntfs" not "fsck.ntfs" .. and if you do some reading around this topic there is a conspicuous absence of fsck or similar tools for NTFS on Linux. There's an older "ntfsfix" which simply clears the dirty flag; which will probably allow the drive to mount in some/most cases, but risks causing damage in scenarios when proper chkdsk.exe is required to resolve a more serious problem with the filesystem (and due to that, isn't something we'd try to use). It looks like we (and the the rest of the Linux using world) is needing Paragon to clean-up and submit their fsck tool code.

    Agreed! But I'm not holding my breath.

    On my current (old Ubuntu 18.04) Linux laptop there is a /sbin/fsck.ntfs but that's just a symlink to /bin/ntfsfix, the manpage of which is quite honest about what it doesn't promise: "ntfsfix is a utility that fixes some common NTFS problems; ntfsfix is NOT a Linux version of chkdsk. It only repairs some fundamental NTFS inconsistencies, resets the NTFS journal file and schedules an NTFS consistency check for the first boot into Windows." Relevant command line options are limited to -b (Clear the list of bad sectors) and -d (Clear the volume dirty flag if the volume can be fixed and mounted). And that's it - not a whole heck of a lot. But as of right now that's about as good as it gets.

    And the fsck implementation in busybox (which I believe to be the one in LE, although I might be wrong) looks to be even more limited.


    In short, it looks like NTFS on LE11 is even less of an option than it is on other Linux platforms, and although I would say that from a UX standpoint proper and robust NTFS support is non-optional, I'm not sure it's realistically feasible at this point in time. :(

    ^ LE uses the full e2fsck binary not busybox.

    I built a Generic/GBM image here: https://chewitt.libreelec.tv/testing/LibreE…_64-11.80.0.tar which should reinstate mkfs.ntfs and include the ntfsfix tool from ntfsprogs. The image is not tested since I don't have any Intel CPU hardware around these days, but please have a play with the binaries to ensure they compiled okay and to see if ntfsfix can resolve the dirty-filesystem issues you're seeing?

    Quote from chewitt

    ^ LE uses the full e2fsck binary not busybox.

    I built a Generic/GBM image here: https://chewitt.libreelec.tv/testing/LibreE…_64-11.80.0.tar which should reinstate mkfs.ntfs and include the ntfsfix tool from ntfsprogs. The image is not tested since I don't have any Intel CPU hardware around these days, but please have a play with the binaries to ensure they compiled okay and to see if ntfsfix can resolve the dirty-filesystem issues you're seeing?

    I'm on the road this week; I can only download the image on Friday afternoon. Will revert then and let you know what happens. Meanwhile thank you so much for your support; you, sir, rock! :)

    I've downloaded it, but I'm not sure what to do. I'm not a hard-core developer and I'm not well-versed when it comes to things like Github commits. I work on in-house PHP website back-ends for a living. The above tar doesn't seem to be an image but contains a file system, so I'm not too sure how to install it. To have a look at the stable release image, thinking that the contents of the tar might replace some of it, I tried to download the current generic image for generic PC (no Nvidia support) which should in theory work on my Dell Inspiron 15 3576 laptop but that results in an error message:

    Code
    redigo: nil returned

    So I'm not sure how to proceed.

    I run LE 20.0.3 on an RPi4 (on which I experienced the above NTFS issues) and I've got my Linux laptop for work (still running Ubuntu 18.04, and I can't install LE on that one because I can't risk nuking my work system). That's pretty much it. Virtualization isn't really an option for testing NTFS issues since the virtualization is more than likely to skew the results.

    So. How now, brown cow? :)

    The download error was temporary (something being rebooted) but that's not any use since it won't contain ntfsfix. So what system do you want to test the change with? RPi4?

    Quote from chewitt

    The download error was temporary (something being rebooted) but that's not any use since it won't contain ntfsfix. So what system do you want to test the change with? RPi4?

    That would be great, yes! I can safely nuke my RPi4 without jeopardizing my work data or install a test image on a spare SD card. :) Tnx!

    chewitt :
    OK. I've put the test image through its paces. TL;DR version: no effect. Details below.

    First I replicated the issue on the LE11.0.3 image that I've been running for the past two weeks or so. The harddisk I'm using is a Seagate 8TB Archive SMR in external enclosure with USB3 interface (which also contains a 2-port USB3 hub). Partitions: 134MB FAT ("Microsoft reserved" rubbish that I never got around to eradicating) and the rest is NTFS.

    After a clean reboot with a previously cleanly unmounted harddisk all is well. Then I arranged for an unclean dismount of the USB harddisk. <yank> :)

    Relevant output of dmesg:

    Replugging the harddisk replicated the issue that started this whole thread: the NTFS partition is not visible from within Kodi.
    Relevant output of dmesg:

    Last two lines of mount output:

    Code
    /dev/sda2 on /var/media/Seagate_8TB type ntfs3(rw,relatime,uid=0,gid=0,fmask=37777600133,iocharset=utf8)
/dev/sdb1 on /var/media/FAT_134MB type vfat (rw,nosuid,nodev,noexec,noatime,fmask=0133,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,utf8,errors=remount-ro)

    However, while mount shows the partition as mounted, it is not visible from within Kodi and /var/media/Seagate_8TB/ is empty.

    Unplugging and replugging the USB drive (which seems the first option a non-technical user would think of) doesn't fix the problem, however the mount command now doesn't show /dev/sda2 at all.

    So let's run fsck /dev/sda2:

    Code
    fsck from util-linux 2.38.1

    Which is profoundly unhelpful and doesn't fix anything. So that's the problem reproduced with the standard 11.0.3 distribution image.

    Then I did the same with the LibreELEC (chewitt): 11.80.0 (RPi4.aarch64) image as per the download link above. This shows one (and only one) difference with the LE11.0.3 distro image, which is the output of the fsck command:

    Code
    fsck from util-linux 2.39.2

    Finally I plugged the USB harddisk into my Ubuntu 18.04 laptop. (Yes, I know, I'm way overdue for upgrading. Deal with it. :)) This system runs kernel version 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux.

    Strangely enough, the NTFS partition mounted fine. I unmounted it and ran fsck /dev/sdb2 on it:

    Code
    fsck from util-linux 2.31.1
Mounting volume... OK
Processing of $MFT and $MFTMirr completed successfully.
Checking the alternate boot sector... OK
NTFS volume version is 3.1.
NTFS partition /dev/sdb2 was processed successfully.

    I'm not sure why fsck on the old Linux box doesn't see the need to repair anything; on previous fsck runs on that partition it did come up with several messages of issues that were corrected. I wish I'd saved them at the time, but of course I didn't.

    Cleanly dismounting the USB disk from the Linux laptop did not make any difference on the LE side: when I plug it back in I still get the "volume is dirty" error in the dmesg output.

    So in summary:

    1. LE is more critical when it comes to refusing to mount a "dirty" NTFS partition, and
    2. The fsck from util-linux 2..31.1 gives more output and seems to do more than the ones in both the distro and the test LE images that I've tried.
    3. The fsck included in LE11 distro and the test image both don't seem to do much more than print its version number when applied to an NTFS partition.

    Item 1 could be kernel-related. 6.1.58 vs. 4.15.0 is not exactly trivial, although I wouldn't be able to explain why recent kernels would be less tolerant to NTFS issues... ?(

    It took plugging the HD into a Windows box and running chkdsk on it before LE would play ball with it again. Interestingly, chkdsk found no errors! But it still caused LE to mount the NTFS partition again. Uh-huh.

    I've had NTFS issues on the Linux box after a mid-write unclean dismount (files and directories that I couldn't read, write or delete anymore from Linux) and that needed a Windows box to fix. But the partition would still mount. LE seems much more picky.

    And that's it from me for today. Stay tuned - film at eleven... :P

    Edited 2 times, last by frankvw (November 9, 2023 at 2:31 PM).

    Connect up the damaged disk to your Ubuntu laptop and try running Test Disk. You’ll get a far better insight into what exactly is going on.

    Quote from petediscrete

    Connect up the damaged disk to your Ubuntu laptop and try running Test Disk. You’ll get a far better insight into what exactly is going on.

    Thanks for the tip! That may save me having to use my wife's Windows laptop. :)

    However that's not the issue we're discussing here. LE won't mount NTFS partitions that my Linux box handles without flinching and the on-board fsck is all there is but that doesn't do much more than print its version. So one unclean dismount and you can't get things going again on LE. And that's a problem. Regardless of whether you need an external Windoze box with Chkdsk or an external Linux box with testdisk.

    Maybe testdisk could be ported to LE if fsck can't be made to do more than just print a message and exiting. But that still leaves LE's finicky behavior with NTFS.

    Quote from chewitt

    The tool that I added to the image is "ntfsfix" not fsck. Please test that.

    chewitt :
    Okay! That works!

    After an unclean dismount the NTFS partition cannot be made to mount again, but 'ntfsfix -d /dev/sda2' solves that problem. Progress! :)

    It doesn't directly address the fact that LE11 appears far more critical than Ubuntu 18.04 when it comes to not remounting an unclean NTFS, much less the long-standing problems of Linux vs NTFS in general, but at least it's a usable work-around!

    If ntfsfix can be included in the next point release of LE11, a udev rule to run it every time the NTFS volume is plugged in would go a long way to easing this particular pain.

    Meanwhile I've extracted the /bin/ntfsfix binary from the image and I'll add it to my LE11.0.3 install.

    Tnx!

    Update:
    Simply copying the binary across doesn't seem to work. I put it in ~/.kodi/addons/virtual.system-tools/bin because that's in the path and it's too late in the evening to start figuring out how to remount / to be writeable. But simply running it (as it did on the test image) won't work here:

    Code
    LibreELEC:~/.kodi/addons/virtual.system-tools/bin # ls -l ntfsfix
-rwxr-xr-x    1 root     root        199360 Nov  9 23:00 ntfsfix

LibreELEC:~/.kodi/addons/virtual.system-tools/bin # sh ./ntfsfix
./ntfsfix: line 3: syntax error: unterminated quoted string

    Time to call it a night.... :sleepy:

    Edited 2 times, last by frankvw (November 9, 2023 at 9:19 PM).

    Copying binaries between different LE major versions or from other distros occasionally works but is never guaranteed. I'm going to have a look at Ubuntu 18.04 and what it's doing (need to find something with an Intel CPU for this though). I'm wondering if they have udev rules that run ntfs or they aliased a script that runs ntfsfix to fsck.ntfs. I'm 100% sure the util-linux tools support FAT/VFAT but do not support (and never have supported) NTFS; so the output from the fsck command needs explaining.