These modechanges (modesets) are unavoidable as they are needed to push the changes through the video driver. And by changing max_bpc you are actually changing HDMI clocks - 12bit need 1.5x higher clocks than 8bit so the whole video output block needs to be reinitialized to set the correct HDMI output (which now runs 1.5 times faster than before).
BTW: not sure why you set Colorspace to 12, that would be DCI-P3_RGB_Theater?
My guess is you could leave Colorspace and max_bpc as they are and only change Broadcast RGB property.
so long,
Hias