I would like to add I have the same issue with a Rpi 4 2GB. I'm not using audio passthrough, but HDMI direct into a Samsung UE22H5000.
This is the CID of the SD card I'm using "035344534233324780b7154b1e014a00". I believe it's a Sandisk 32GB Ultra.
I see this both on Youtube and on NFSd mkvs. It's completely intermittent, sometimes it will play perfectly, sometimes it will just start stuttering. The same file, the same YouTube video.
Glad to hear I am not alone at least. I am using audio passthrough and everything with that section in the settings is enabled. Using a Sandisk Extreme Plus SD Card 64GB
Disabling hardware acceleration did not work for me. Towards the last 5 minutes of the movie it began stuttering again after working fine for a long while after the last stutter period.