I would like to add I have the same issue with a Rpi 4 2GB. I'm not using audio passthrough, but HDMI direct into a Samsung UE22H5000.
This is the CID of the SD card I'm using "035344534233324780b7154b1e014a00". I believe it's a Sandisk 32GB Ultra.
I see this both on Youtube and on NFSd mkvs. It's completely intermittent, sometimes it will play perfectly, sometimes it will just start stuttering. The same file, the same YouTube video.