If it works on Lakka, we should be able to figure out why it doesn't work on Kodi. When I updated all game add-ons to their Lakka version (https://github.com/LibreELEC/LibreELEC.tv/pull/7705) it should have made both PSX cores build with the same flags that Lakka uses. I can check again and see if any flags have been updated.
The rewind feature slows down larger games a bunch. Definitely disable it for PSX cores.
The next likely cause is NEON/dynarec options. If either is disabled, you could see slowdowns like you do.
I have an RPi 3/4/5, so I'll try to reproduce when I get a chance.