It could be hardware, it could be how your shutting down
Its definitely how I'm shutting down - just flip the power switch. Having said that I do the same with the Pi3s and no problem. I pull USB drives out of the port on Windows (you know the OS everyone loves to slag <G>) without safely ejecting and no problem . In addition I live in the Scottish Highlands where power cuts or voltage drops are not unknown.
As far as not having the problem when taking a usb drive on holiday - how often have you written to the usb drive whilst its connected to the Pi4? If I leave it connected and do not change anything it works perfectly.
However, I do have some good news
Did you tried another format type or only NTFS and Ext4?
ExFAT looks like the answer, at least with my experiments with a 600GB USB drive.
I didn't try it because all my USB drives have been formatted to NTFS and all of the research I did said ExFAT is less stable. Stage 2 testing will commence after formatting the 6TB HDD.
another way: keep /storage on the SD, mount the hdd normal (e.g. as /var/media/storage2), delete the data directories (tvshows, videos, etc.) and make softlinks to the directories on your hdd under /storage
I'm not sure what this is meant to achieve.