If you implemented cache fiddling on the current RPi4 setup; remove it and you'll likely discover things improve, esp. with LE13 images. Why? because there is a sizeable chasm between how most users think media streams and caching work, and how it actually works, which results in people with utterly moronic cache configurations. Unless your network is broken (in which case the solution is always to fix the network) there is no need for anything beyond Kodi defaults, no matter what some outdated reddit or Kodi forum thread armchair pundit claims or recommends. K22 has changed things quite a bit.
RPi5 is a noticeable update on RPi4. Generic PC stuff is a little faster again. There is no performance difference between 4/8/16 GB RAM in terms of GUI and playback because 2GB is enough for 4K and the GPU and Kodi desktop resolution (and key-repeat/scroll speed of the remote device) dictate how fluid things feel, not RAM size. These days 16GB seems to be the minimum DIMM size that you can so not sure anything smaller is even possible. The difference between DDR4 and DDR5 is likely irrelevant for LE use.
I have an RPi5 booting from an NVME drive (family daily-driver) and an Intel N150 box with eMMC storage (x86_64 dev work) and while the Intel box is technically a lot faster, the real-world difference is more marginal.