- Are the two USB drives powered from the USB ports on the board or they have external PSU(s)?
- What is 5V/?A spec of the PSU?
- If you remove one of the drives does the remote appear reliably?
- If you scan the bus with lsusb -tv is the remote USB dongle listed; and does the remote now work?
If you have two drives powered from the board my hunch would be that when the board is booting and everything is in an active state the drives can pull enough power to result in the remote USB dongle not receiving enough to power on. This is probably a borderline case so it might still work occasionally. If you then remove/reconnect some time after boot the board is now in a less active and more power-stable state so the remote USB dongle reliably works.
If the hunch is in the correct direction, perhaps using a higher rated PSU might work, e.g. the 5V/5A spec for an RPi5. Or connecting the drives to a powered USB hub, or if one of the drives has as USB 'Y' lead (to draw power from multiple USB ports) connect the non-data plug to an external USB phone charger to reduce power draw from the 4B board.