A few questions:
Is the non-smooth video blatant or subtle? Is it a frame drop every few seconds or multiple per second?
Do the stats (ctrl-shift-o) show increasing skip: and drop: counts (note: a few skips when overlay appears is normal, but it should then no longer increment. Also this doesn't seem to catch all forms of skipped frames).
If you disable "Adjust display HDR mode" in settings/player/video does it make the video smooth (at the cost of incorrect colour)?
Does your display switch to a 4kp24 mode when playing 4kp24 video?
Can you confirm the have the following settings:
system/display "Set GUI resolution limit": 1080
player/videos "Adjust display refresh rate" : "On start/stop"
player/videos "Sync playback to display" : "Off"
player/videos "Allow using DRM PRIME decoder" : "On"
player/videos "Allow hardware acceleration with DRM PRIME" : "On"
player/videos "PRIME Render Method" : "Direct to Plane"