Setting force_turbo=1 helped a bit, it solved the out of sync audio problem with some hevc movies.
The gpu_mem=512 crashed the hevc movie playback with every movie I tried, so I reverted back to 320.
I have carried out extensive tests with different downloaded hevc movies and with the Big Buck Bunny test video converted to hevc with different quality/bitrate settings. Based on the tests I think the RPI4 can play hevc videos smoothly with video bitrates below 80 Mbps, above 80 Mbps the playback stutters. Somewhere above 100 Mbps the RPI4 gets unresponsive and freezes.
Unfortunately lots of movies are converted/remuxed with parts or the whole movie above 80 Mbps, and the RPI4’s SoC is not capable / not powerful to play them.