I'll also repeat my findings:
1. It doesn't look like the issue is with the video files, the files that stutter on rpi4 play smoothly on other devices with similar settings.
2. Audio does have something to do with it, when switching the output to analogue only or hdmi+analgue, the issue seems to disappear. However, when subtitles are enabled there is still stuttering, regardless of audio settings.
3. The issue does not seem to be related to storage read speeds, CPU load or file source. I tried playing the video files from an internal SD card, from an SSD drive connected to USB 3 and via network from a different PC with the video file located on an SSD drive, both wired and wi-fi - the results were the same. While the file is playing the rpi4 CPU load is very low.
4. The issue is sometimes not easy to detect, it's very sporadic, but it's much more evident when TV refresh rate is adjusted to video file fps, which is 23.976 fps most of the time. Also, Kodi doesn't register skipped or dropped frames during this stuttering.