Alright, I did a test with the latest stable LE12 from a clean install on a fresh SD card.
As more or less expected, this is working well, i.e. no delay anymore. Sadly, this does not hint a solution to the issue with my current installation.
Next step would probably be to rebuild my setup from this clean situation, step by step and checking every time if the issue re-appears.
But honestly, I cannot be bothered to do this - I just don't have capacities for this right now.
I guess, it's one of the not-so-many addons I use, but I have no idea which one and just deactivating them without full de-installation does not seem to help.
I hope that the update to LE13 (once the stable is released one fine day) will kind of erase my issue during the process.
If not, I will consider re-building my stuff with a fresh stable LE13 to that future date.