external HDDs - random unmount of drives on rpi4

I have an external hdd tower with 5 hdds inside. The hdd tower has its own 12V 6.5A power supply and uses a single USB cable to connect to rpi4. I can access media on all 5 drives most of the time, however, the system seems to randomly unmount some of the disks from time to time. The issue is resolved when I unplug the HDD hub from rpi4, plug it into a windows PC (which automatically prompts me to scan hdds for errors), check the disks, gracefully unmount from windows, and then plug back into rpi4. The disks are not damaged or corrupt. All drives are mounted into kodi, and I can watch media sources on all drives.

...until it happens again - without any obvious reason or trigger. Wash, rinse, repeat. I am tired of having to do this unplug/scan for errors/unmount/remount process at random times, and maybe twice a week. Why is this happening?

dmesg output:

Quote from adam.h.

This sounds like power problem (despite the fact that your HDD tower is self-powered) - maybe some power peaks at night etc. I would suggest to add some reliable powered USB Hub between RPi and HDD tower (like this) and see if that makes any difference.

The hdd tower has its own ~7A external supply. It powers both the hdds and the SATA-USB bridge. It does not make sense to add a USB hub, and the manufacturer explicitly recommends not using a hub. Why add more USB in the mix? HDDs do not draw power over USB, they are externally powered.

All disks are NTFS. I am not aware of any hdd formatting issues with LE. Shouldn't all of the hdds equally have issues (all unmount instead of just 2-3)?

The hdd tower is JBOD using this hardware: https://www.amazon.com/gp/product/B06XK972L1

Maybe the tower is putting some inactive drives into sleep mode?

There's clearly some useful debugging in dmesg if someone could help interpret it.

[ 33.086977] ntfs3: sda5: volume is dirty and "force" flag is not set!
[ 33.141480] ntfs3: sdc2: volume is dirty and "force" flag is not set!
[ 33.178834] ntfs3: sde2: volume is dirty and "force" flag is not set!

I have one of those USB pass thru sticks to measure current draw. The hdd tower draws 0A from the rpi USB port, because it is externally powered. Everything is on UPS.

Looking into the specific hdd tower, some people mentioned in reviews it would put drives to sleep, and sometimes only some of the drives, not necessarily perfectly correlated to inactive drives. My guess is some chinese-special firmware that is randomly sleeping some of the drives, which may cause an un-graceful unmount and remount attempt, which only windows can resolve ...because NTFS. The hdd enclosure mfr has a firmware utility to modify the sleep function. I have just used it to disable sleep. We'll see if that does it. Since the unmount is an intermittent issue, it could be some time before I say for sure.

The NTFS issue I had no idea about until now. It had never been an issue for me in the past with several (the exact same) NTFS disks on older SATA-USB docking stations and several previous versions of LE (Leia, Matrix). Hopefully it's not the problem here since I'd have to buy a large new hdd to transfer/preserve data and reformat all disks. But I suppose that will be the next thing I try if the problem persists.

Well, the issue repeated. So the sleep setting on the hdd tower was not the issue.

There's one other complication to my story. After upgrading (clean fresh install) from LE10 to LE11 - all using the same hardware - that is when this issue first appeared. IIRC I had the same hdd tower on LE10 and did NOT have this unmount problem. I also think I was using docker and transmission on LE10, without issues.

But on current build, LE11, docker and transmission often give me problems with having to stop/restart the service and even sometimes remove and reinstall transmission. The symptom I get is, when adding a new magnet link to transmission, the free space on the target drive shows up as 1.99 GB, when the drive is actually some 100GB free space. That is how I know something has gone awry. This is sometimes coupled with the disk unmount problem. Typically the unmounted disk is the target download disk, plus sometimes additional disks. But it is hard to say if that itself is a symptom of a deeper problem or the cause. I've uninstalled transmission and I'm going to run without it for some time to see if the problem keeps happening without any docker services running.

It happened again. 2/5 hard drives unmounted. There is a crash log associated with the event (attached, pastebin flagged it for objectionable content for some reason). No docker, no transmission installed anymore, let alone running.

crashlog
kodi_crashlog_20240110052754.log

By 'media box', do you mean the hdd tower?

My only SBC in this scenario is rpi4. The hdd tower is a simple ATAPI bridge that turns 5x internal SATA hdds into a single USB device connected to the rpi4 by a single USB cable.

I _could_ turn this hdd tower into a NAS, but if the hdd tower is the problem, that presumably wouldn't fix the issue. You bought a multi-bay NAS that doesnt have this problem? What is the model so I can look it up and check its hardware specs?

It feels like overkill for my needs, but a true NAS would give me greater flexibility...on the other hand this feels like a situation where a simple one liner script to keep all 5 hdds awake might be the next step in identifying the true cause of the problem.

I spent a full day looking into proper NAS solutions. Synology looks like the way to go for an off the shelf ($700) solution, but I've decided to make my own for ~$300 with TrueNAS. Will update the thread with results, but I expect a proper NAS will fix the unmount issue.