Looks like Full RGB mode is not active for your screen device output. I had the same problem which was solved by adding
to /storage/.config/autostart.sh (https://wiki.libreelec.tv/configuration/…wn#autostart.sh). Replace 239 with connector (video output) number specific to your system, which can be discovered by inspecting modetest text dump when ran without arguments.
Also make sure that your Settings / System / Display / Use limited color range option is not enabled.
Hello,
I would like to share my experience/knowledge as I don't fully agree to what you're saying with respect to avoiding the limited color range.
I have been using "proptest" to set the full/limited RGB range of my GPU since LibreELEC 11 has been released and graphics stack has changed from X11 to GBM. See RE: LibreELEC (Nexus) 11.0 for more details about how to use proptest. I was not aware of "modetest" but it seems like those commands do the same thing. Maybe "modetest" was not available at the initial release of LE 11?
Back to the color space. Unlike what you are saying I am using limited RGB in KODI, full RGB on my GPU and limited RGB on my TV. Why am I doing this? Because apparantly that is what is being recommended in the Kodi Wiki (see https://kodi.wiki/view/Video_levels_and_color_space). In section "Which Combination to Use" they recommend using "Limited, Full, Limited" as there is no video content scaling being performed. I have not had any issues so far after changing my setup to the recommended setting.