It seems VPU runs out of memory. A20 can use only part of memory for video decoding. What is the resolution of your video? Can you test smaller and/or just another videos?
Latest try with LibreELEC-A20.arm-9.80-nightly-20201014-74eb09f-bananapi:
Yes, I tested several videos H264 HD, SD XviD, AVC. (XviD videos are also jerky). In general, when I switch from a running video to another, playback switches to that new video, but the image is a still image from the original, only the audio from the new one continues. Same when I jump forward or backward during playback.
Logs on such jump:
http://ix.io/2ALJ
http://ix.io/2ALH
First I have to say that LibreELEC-A20.arm-9.80-nightly-20200426-65f515b-bananapi (oldest I have) works quite well. These test videos mostly play well (H264 SD HD) I can jump in a video, switch to another video during playback. There is only a problem: DVB-S2 H264 Full HD channels have delayed audio and fail after a while. O.K playback is on H264 SD channels.