Raspberry Pi5 how to change color space

I am trying to use the Pi5 as my Kodi frontend, and it works great, except for 4K + HDR. It works - I see picture, and my projector is seeing that the video signal is 4K, HDR, and so on, but it's in RGB 12-bit format. I'm pretty sure this combination doesn't work well for my projector, but I instead need YCbCr with HDR instead. I've done a lot of searching, and all I found was a reference to a config.txt line

hdmi_pixel_encoding=4

and that caused my screen to just go blank when switching to 4k instead. Is there another way to do what I want? This is working perfectly other than this one issue. Also, is there a reference for what the various hdmi options are?

I'm using the latest LibreELEC 12 beta 2.

Thanks!

Well, I guess my projector's auto tone mapping doesn't do well in RGB mode. When I manually increased the brightness it looks fine, though it's hard to say how much black (or white) crush I accidentally introduced. Oh well...