I'm seeing the same issue (Raspi 4)
It is definitely triggered by the ads in a stream. If I watch a stream that I subscribe to (doesn't have ads) there is no issue.
If I disable 'allow hardware acceleration with DRM PRIME' then the freezes don't happen, but there is a loss in quality. I guess it is something to do with switching to a different video 'mode' when the ads start.
I wonder who developes the hw accelerated codec?