Usb Hard drive not recongize since 11.0.4

    Quote from VLouis

    "IF" your RPi will die (RPi's never dies ;)), you can boot your PC from a USB stick and live-linux (any kind of Ubuntu) and save your data from the EXT4 formatted disk. That's isn't a real problem. If you have linux (LE) and want stability use a linux partition.

    Nobody said isn't exist support for NTFS at all under linux. Just not guaranteed. Sometimes it's weak, sometimes enough (or it looks like enough for you).

    Thank's. Yes I can boot linux from USB drive and use it, but there are some people who can't, while I am away. Whether Rpis die or not, the problem is still there and I would really like to know, what's happening, where those errors are coming from and how this can be prevented. The current setup is too unreliable at the moment, for me to just leave it. At least I know it seems the HDD is not dying.

    Can those NTFS errors affect HDDs life and damage it finally?

    Seem like this is a known issue. There are older topics and talks about some kernerls being more stable than others, among other things.


    Kind regards

    There is force mount option in ntfs3, so you could eventually give it a try - not sure what is now the best way to add it for default options in LE, perhaps via autostart as described here.


    `````sudo mount -t ntfs3 -o force /dev/sda1 /media/ext

    • New

    As I understand it the issue is not with LE rather the Linux kernel that changed it's handling of read/write to devices. Actually I think MS made some NTFS code changes as I find many times my usb drives are reported to have issues when none exist. Some sort of error flag is being set for who knows why. That seems to cause HD mount issues in Linux so not really a LE issue.

    • New

    In recent releases LE changed from using the long-proven but rather slow "NTFS-3G" drivers which run under FUSE/userspace (hence the slowness) to newer and much faster (but less proven) in-kernel drivers. Hence there are some scenarios with "old releases didn't flag problems, newer ones do" as the drivers being used are completely different. The in-kernel ones are maturing without any major issues, but some users see problems with the filesystem of removable USB devices being marked dirty. There are no "fsck" drivers for NTFS filesystems so "fixing" that generally needs the drive to be reconnected to Windows. One of the major differences between Windows and Linux is that Windows will keep trying at all costs to mount a broken filesystem and make it accessible to the user, wheras Linux bails at the first sign of an issue and either refuses to mount the filesystem, or will force read-only mounting to prevent further damage.

    • New
    Quote from Marcin

    Is force mounting going to fix errors from LE?

    It won't fix anything, just mount it regardless these errors. The other (maybe better) option is to fix the disk in Windows system and change LE to mount it always as RO (the above url to autostart is good reference for details).

    BTW, the option to mount external contents disks as RO could be good extension for LE, but perhaps it happens so rarely that it is not worth the effort. I'm using RPi5 with latest LE12 and SSD disk formatted as NTFS and while I had that "dirty disc" errors in the past (probably around LE10 or LE9), it works now without problems. So this could be also some combination of disc type, NTFS and LE version (driver version).

    • New
    Quote from Marcin

    Seem like this is a known issue.

    Is questionable, if it's an issue or not. Like Chewitt explained above, it's more about the different OS handling a disk with active error flag. And as the NTFS is Microsoft proprietary, developed for Windows OS, I didn't think to have ever a "perfect" support under linux for it.

    For removable drives the best compatible way is the exFAT partition. A bit less good in error handling, but better support in all OS's. There's a lot of comparition between exFat and NTFS (and EXT4), just use the "google" if you want to know more. I'm using a "big" USB stick for storing media files (RPi4B, LE12), formatted to EXFAT, and (till now) didn't had any issue.

    • New
    Quote from VLouis

    Is questionable, if it's an issue or not. Like Chewitt explained above, it's more about the different OS handling a disk with active error flag. And as the NTFS is Microsoft proprietary, developed for Windows OS, I didn't think to have ever a "perfect" support under linux for it.

    For removable drives the best compatible way is the exFAT partition. A bit less good in error handling, but better support in all OS's. There's a lot of comparition between exFat and NTFS (and EXT4), just use the "google" if you want to know more. I'm using a "big" USB stick for storing media files (RPi4B, LE12), formatted to EXFAT, and (till now) didn't had any issue.

    Thank's for explaining chewitt post in more simple words :). I had difficulties understanding it. Not sure what an issue means for you, but it is without a doubt an issue for me, as my current setup is not usable, due to the HDD randomly just dissapearing from LE with no reason, no that I can think of.

    I bought RPi5 after my EMMC module in Odroid C2 died. I used the same HDD with odroid, only older LE version. I thought it's a good opportuinity for an upgrade. I never expected to run into such problems with the same external HDD and significanlty newer hardware and newer LE version. Shouldn't everyting be improving overtime?

    As for expectations for NTFS support, I was using Linux with NTFS drives on an old laptop for many, many years and never had such problem. I never saw this coming.

    RO mounting is not an option. I need to write to the disk as it is a storage place for all multimedia, and backup images from all home laptops.

    Reformating the drive to exFat will be a major problem, as I do not have a 10TB free space anywhere else. Besides, I can see, peaople report similar problems with exFAT also.


    EDIT

    Is there a way to force LE current version to "handle a disk with active error flag" in the same way like the older versions of LE did, where the drive was never dissapearing from LE?


    Kind regards

    Marcin

    Edited once, last by Marcin (October 22, 2024 at 7:50 PM).

    • New
    Quote from Marcin

    As for expectations for NTFS support, I was using Linux with NTFS drives on an old laptop for many, many years and never had such problem. I never saw this coming.

    I'm using too something like this, a dual boot (Win11 & Xubuntu) PC, and I had issues with the NTFS partitions, become better after I disabled hiberantion option in Windows. And yes, for the partition what I want to be "simple" and always usable in the both, I'm using exFAT. Mostly this linux-NTFS issues coming after a "dirty disconnect", like sudden disconnect/power down, what couldn't be expected in the case of the HDD assembled in a laptop.

    Quote from Marcin

    Not sure what an issue means for you, but it is without a doubt an issue for me, as my current setup is not usable, due to the HDD randomly just dissapearing from LE with no reason, no that I can think of.

    To simplify more this, I think the NTFS issue isn't LE related. It's in the linux kernel. So I think the right place for this could be somewhere in a linux development forum, kernel section. And about if it could be "resolved" soon or not... just try to think from another side, to report an issue in a Windows forum as you have difficulties with reading EXT4 linux partitions and you don't have full suport for it...

    You have concern about EXT4 and how to read it if your RPi with LE become defective. You have 10TB data on an external HDD, did you think ever, enough a little bigger mechanical shock and that disk could became defective, without chance to recover any data from it? I learned before, from my own bad luck and defectiv HDD, to don't keep important data in a single location.

    Sorry, but exists issues without a simple (or at all) solution and this probably is one of them.

    • New

    The main user-experience issue is that Linux has no "fsck.ntfs" equivalent to the "fsck.ext4" tool, so when errors occur the OS has no way to self-recover and continue. Similar issues occur with EXT4 filesystems, but we check the filesystem before mounting during boot and fsck if necessary, which clears dirty flags in most scenarios and allows a dirty drive to be mounted read-write.

    The workaround might be to self-compile an LE image that disables the in-kernel NTFS3 driver and revert to NTFS-3G, which should give the same experience as older releases. We aren't interested to do that in official releases because we are keen to keep moving forwards and continue supporting in-kernel driver development. The number of user issues here is overall quite low - despite the impacts users often being rather vocal.

    • New
    Quote from VLouis

    You have concern about EXT4 and how to read it if your RPi with LE become defective. You have 10TB data on an external HDD, did you think ever, enough a little bigger mechanical shock and that disk could became defective, without chance to recover any data from it? I learned before, from my own bad luck and defectiv HDD, to don't keep important data in a single location.

    Out of the 10TB, the most critical data is backed up.

    Quote from chewitt

    The workaround might be to self-compile an LE image that disables the in-kernel NTFS3 driver and revert to NTFS-3G, which should give the same experience as older releases.

    Yeah thanks chewitt, I can already see myself doing that. Half of what you write is rocket science for me ;)


    So to sum this up.

    This can happen whenever I restart RPi via SSH?

    This can happen whenever RPi or HDD looses supply due to power interruption?

    This was not happening on Odroid C2 because I was using older LE version with different kernel which was not doing that (flaging HDD as having errors and not mounting)?

    • New
    Quote from VLouis

    ...(RPi's never dies ;) ),...

    I understand your smiley face, so just FYI - unfortunately I had this bad experience with my RPi 4B/2G which died suddenly couple months ago, during the normal operation, without any obvious reason... Of course after the warranty period. It was properly cooled so never overheating.

    Just a black screen all of a sudden, did not boot anymore - most likely the Broadcom CPU itself died, pulling down the 3,3V to ~1,9V.

    • New
    Quote from Marcin

    This can happen whenever I restart RPi via SSH?

    It shouldn't happen as the volumes should be unmounted properly during the restart AFAIK. The safer way is to unmount the problematic volume manually (see umount --help ).

    Quote from Marcin

    This can happen whenever RPi or HDD looses supply due to power interruption?

    Yes. You should avoid it, also disconnecting the USB/SATA adapter cable from RPi without unmounting, when the RPi is running, can lead to your issue.

    Quote from Marcin

    This was not happening on Odroid C2 because I was using older LE version with different kernel which was not doing that (flaging HDD as having errors and not mounting)?

    I suppose so.

    • New

    I did a web search regarding ntfs-3g and ntfs3 mounts.

    Result is that ntfs3 does ever refuse to mount a FS with dirty bit set whereas ntfs-3g only refuse to mount FS with "real" errors.

    So for the user experience it is a regression because much more mounts fail after switching from ntfs-3g to ntfs3 mount.

    A said in the Arch wiki an equivalent mount behavior require a combination of ntfsfix -d and mount -t ntfs3. But this require further tests to get the boundary conditions.

    As a first step I will create a "NTFS-3g for udevil" addon (Coming Soon! :-)) that can be backported to LE12.

    • New

    mglae we looked at ntfsfix before but concluded it wasn't the right approach. I don't recall an option to clear only the dirty-flag last time I looked at the tool though, so not sure if that's a recent addition or I just missed it.

    • New

    chewitt No idea, I did update ntfs-3g to the current version 2022.10.3 first. ntfsfix "only repairs some fundamental NTFS inconsistencies" and ntfsck is deprecated.

    As said first step will be an addon allowing mounting with ntfs-3g.