videos with 29.97 fps frame rate are still jerky, even though the TV is verifiably set to 29.97Hz.
The 29.97 and 30Hz modes should not normally be used. Just leave 60, 59.94, 24 and 23.98Hz modes in the whitelist and remove 29.97/30Hz modes.
if I force 23.98Hz resolution, then the 29.97 fps videos play absolutely smoothly, with Sync Playback to Display still set to Off (and I have a VERY critical eye - there is no judder at all). I don't get what's going on
Is it a footage that was originally shot on film at 24 fps? My guess is that Inverse Telecine is going on. If it is a DVD with a so-called "soft-telecine" encode (a 24p encode with pulldown flags) - Kodi probably just ignores the pulldown flags when the HDMI output is forced to 23.98Hz. That's why you get a smooth playback.