Hi, thanks for the replies!
I finally figured it out. It was your screenshot Klojum that pointed me to the right direction.
I noticed near 0% CPU load while you were playing the 8K test video. Whatever video I played, be it 1080p, 4K or something else, it always hovered around 70% CPU usage, but not from Kodi.bin, but Xorg instead. I also noticed this behavior when just sitting in some menus. When I switched the output to 1080p, all was fine. The CPU usage of Xorg immediately went down to near 0% CPU usage.
The problem turned out to be the Pioneer AVR. On the Pioneer Android app I noticed the input video source being sent was YCbCr 4:2:0, even though Kodi was configured to do Full RGB. The BD input on the AVR was not configured to do YCbCr 4:4:4 at 4k60, but 4:2:0 instead. Once I configured the input to do the full 4:4:4, LibreElec started sending me RGB Full video, which was then being converted to YCbCr 4:4:4 as output. Once I did this, the menu and all videos immediately went back to being smooth. CPU usage was around 20% while playing videos. Xorg was probably converting the framebuffer from one colorspace to another, causing the high CPU usage.
I couldn't resist to stop messing around to get it even further down. I ended up modifying my xorg.conf, with the following changes:
Section "Screen"
Identifier "screen"
Device "nvidia"
DefaultDepth 24
Option "ColorRange" "Full"
Option "ColorSpace" "YCbCr444"
SubSection "Display"
Depth 24
EndSubSection
EndSection
The input source being sent is YCbCr 4:4:4 now, so the AVR isn't converting the signal anymore. Better yet, the CPU usage is now below 10% average.
Thanks for putting me in the right direction guys!