Same issue here. I have a 2GB RPi4 with official case, PSU and HDMI cable. Clean install, room temperature is 20c and the Rpi4 idles at 70c and goes up to 75c during video playback.
The pi 4 temps are a known issue that they’re trying to fix. For the time being it might be wise to get the open air acrylic case with a fan. My pi 4 is stable under mild/moderate loads and hasn’t jumped past 40.5 in a house that is just under that temp wise. Doesn’t even take anything of note for a fan, either.