Thanks Da Flex!
Original post with edits follows below:
This puzzled me for some time until I switched the TV's HDMI range to Full. This solved the issue almost immediately and was a night and day difference - suddenly I could see all the detail I had been missing. The problem was, that normal, non-HDR content now appeared brightened and washed out, so one step forward, one step back.
Only quoting that part, as most of the rest is not relevant on that matter.
To keep it simple.
The TV is not creating the same level of brightness/saturation/color on sdr content when that is displayed with hdr as it most likely supports only a low level hdr on sdr content (mostly happens on lcd/led screens, oleds are far better to get a good sdr on hdr picture).
So the problem with the washed out colors is normal and might be lessend by adjusting stuff like brightness when it is displayed (but would also affect hdr as you change it in that mode).
You might search the net if people might have posted settings guides for that TV which you might like to try.
BTW. Switching the color range to full was needed as hdr formats like dolby vision only work with the enhanced color space.