I think this may be a firmware issue. https://github.com/raspberrypi/firmware/issues/1626
That bug report relates to the fkms driver (which I understand is like the default kms driver but with support for hdmi= settings in config.txt). I can't seem to get either kms using the new syntax in cmdline.txt or fkms using the old syntax in config.txt
The report mentions an older version of the firmware before the bug was introduced. I'm going to see if I can get that installed and try again with the fkms driver.