Update: I finally found what's doing it on my setup.
Video files I'm playing are mostly Mpeg2-TS 1080i
The default de-interlace setting (Auto) doesn't work properly: skips frames and sound get out-of-sync most of the time.
Manually trying each setting, I've found that both MMAL - Advanced are doing this, but the 2 MMAL - BOB work perfectly for what I need.