Setup:
Fresh install of LibreELEC 9.2.1 on Raspberry Pi 4b.
Video information:
Format : AVC
Format/Info : Advanced Video Codec
Format profile : High@L4
Format settings : CABAC / 2 Ref Frames
Format settings, CABAC : Yes
Format settings, Reference frames : 2 frames
Codec ID : V_MPEG4/ISO/AVC
Duration : 1 h 53 min
Bit rate : 5 307 kb/s
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate mode : Constant
Frame rate : 23.976 (24000/1001) FPS
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 0.107
Stream size : 4.20 GiB (93%)
Writing library : x264 core 155 r2901 7d0ff22
Default : Yes
Forced : No
Color range : Limited
Color primaries : BT.709
Transfer characteristics : BT.709
Matrix coefficients : BT.709
My findings this far:
I've tried the playback both from a USB drive and a local file copied to the OS memory card, both give equal results. I have the debug overlay on and the CPU usage is never above ~35% (my original thought was that maybe its overheating and throttling the CPU). While playing the video file the FPS counter varies a lot, the values are between 15 and 60.
I apologize if this is common knowledge on how to fix this, but I've tried to search for possible solutions and haven't found any and would appreciate any tips on what to try next 
