Incorrect device tree? No driver for internal memory variant? Hard to guess, so far there has not been a report of not working internal storage.
That's what aml_autoscript does when you press recovery button: it executes commands to replace boot command. You can execute the part that you need manually over console: LibreELEC.tv/aml_autoscript.src at libreelec-7.0 · kszaq/LibreELEC.tv · GitHub
The M96X doesn't have a recovery button, that's why I had to use the rs232 interface to boot in the first place... Installing to nand would probably be simplest if I can get /dev/system to show up. I'll try a different device tree file and see what happens.
Thanks
Edit: After trying a few dtb images, and still not seeing any /dev/system (or /dev/boot, etc) I'm starting to wonder if the internal flash is formatted properly and if the partitions are where they should be. Is there any way to reformat the internal flash properly from u-boot?