There is no support for 800x600 in video driver at the moment. Amlogic doesn't use X, it uses EGL and there is no xorg nor xrandr.
I can see that Odroid C2, with the same hardware, for both Ubuntu and Android has 800x600 support (but also with a lot of video tweaking settings), and a very convenient boot.ini configuration file:
en:c2_tips [Odroid Wiki]
There is no chance to implement this in LibreElec?