This isn't really my area, but I'd imagine a USB drive that was too slow to enumerate may sometimes fail to boot and move to the next stage in booting (which may end up at the network install screen).
That doesn't really tie in with it also happening with sdcard.
What do you have BOOT_ORDER set to?
Have you removed the USB drive when testing with sdcard?
When at the network boot screen, it says "Press ESC to cancel and go to diagnostics screen" who does diagnostic screen show?
(if keyboard is not connected on boot, I believe it goes directly to diagnostics screen).