Explanation of the ffmpeg patches

Chewi · December 26, 2023 at 11:27 AM

Hello! I'm here as a Gentoo Linux developer. We support a very broad range of hardware, but our RPi support has been a bit lacking in recent years, with old versions of the legacy stack still present in our package repository. I get the impression that not many people use video on RPi under Gentoo, but I'd like to change that.

I've started putting the changes together to throw out the legacy stack, and I've been testing the video performance against mpv and Kodi. I quickly became aware that vanilla ffmpeg was not going to cut it, even for some 1080p videos on a Pi 3b. I tried various branches from various forks against 6.0 and 6.1, and while these generally worked for mpv, only jc-kynesim's test/6.0.1/main branch worked for Kodi.

I have seen that you only apply this patchset specifically for your RPi build, while you apply Kwiboo's v4l2-request and lrusak's v4l2-drmprime patchsets on most other devices. I know that these patchsets cannot be combined, and I can see that while there are some similarities, the RPi set is not a superset. Does this mean the RPi patchset is useless on other devices? Or perhaps you're trying not to deviate from upstream more than you need to on those devices? Being a source-based distro, Gentoo does not need to force the same patches on everyone, but I'd rather only have one set to deal with here, if possible.

Carrying these patches has the potential to make life difficult for Gentoo maintainers, as we may want to move on from older ffmpeg versions before these patchsets are available for newer versions. I did start rebasing jc-kynesim's branch against 6.1, and while I didn't get stuck, it looked like it was going to take a while, and I didn't want to continue without knowing whether I would succeed unless I really had to. 6.0 is still quite recent though, so I'm not too worried.

I hope and believe that you are still interested in getting these efforts upstreamed. I appreciate this has been a long road, and there may still be some distance to go yet. I don't know whether I can help in this regard, but perhaps another distro adopting this puts some weight behind it. What is the strategy when we have seemingly competing patchsets though?

**chewitt** · December 26, 2023 at 2:55 PM

Great playback behaviour/performance with Kodi requires kernel, ffmpeg, and Kodi patches. The kernel patches are normally a mix of things that are already on upstream mailing lists and capabilities that are still work-in-progress. Over time LE accumulated several DRM maintainers and notable contributors as project staff, and we deliberately chat with folks from Collabora and Raspberry Pi who are involved with commercial work in the DRM/V4L2 ecosystem that overlaps with our needs: LE gains from their efforts to push code upstream and they benefit from LE/Kodi being media focussed so our devs/staff are good testers.

Our goal is to have zero Kodi patches (beyond basic distro packaging) but Kodi features for RPi currently depend on some kernel and ffmpeg capabilities which are not yet upstream, e.g. until the Red Hat sponsored 'HDR' working group met earlier this year the forward path for colorimetry and some related userspace bits were blocked pending decisions on how to implement; and as the ripple effects of agreement start to unblock things we are adapting and retooling where needed.

Upstreaming FFMpeg changes is the greatest challenge and we are mostly working through JC (who maintains ffmpeg and related things for Pi foundation) but the changes are often complex, the mailing list is opinionated, and the resulting merge rate is low so there's a large patch backlog. Kodi is using FFMpeg 6.0 currently and won't need to bump to 6.1 for a while, but I understand JC has now started work on that. One of the things we do for JC is test his branch over a range of SoC devices to ensure the changes for RPi don't break support for other SoC platforms (to help with upstreaming).

So as a general rule you should be able to pick and choose the media-related kernel patches LE has for Allwinner, Amlogic, Rockchip into a common 'arm64' codebase with FFMpeg 6.0 from JC and a few Kodi changes from LE. In our own codebase we still keep things a little separate to reduce interdependencies for individual SoC platform maintainers, but there are others that we chat with who are doing that; e.g. the developers behind MiniMyth (single image for all arm/arm64 devices) and Manjaro (similar..)

Chewi · December 26, 2023 at 6:31 PM

Thanks, chewitt, that helps a lot. At this point, I think we'd be happy with just "good", as opposed to "great", if it means we only have to patch ffmpeg.

Perhaps I'm looking in the wrong place, but I couldn't find any relevant kernel patches. We do provide the RPi kernel, as well as the mainline kernel, same as you. I suspect HDR might still require kernel patches, but that would just be icing on the cake at this point. I've been following this from a gaming perspective, and it seems to be coming together at a decent pace now. I imagine such kernel patches won't be needed for long.

I did see the Kodi patches, but they didn't look essential, and indeed, my H.264 test videos performed much better even with vanilla master. I'll be helping to maintain ffmpeg in Gentoo, but that won't extend to Kodi as I don't have a personal interest in it right now, so that would be up to a different maintainer. We do at least provide "live" packages for users who want them, so any changes that do get merged will be immediately available to them.

I'll therefore provide JC's ffmpeg patchset to all ARM and RISC-V users by default. Perhaps it won't be a perfect fit in every case, but it shouldn't make things any worse.

**chewitt** · December 26, 2023 at 10:40 PM

The Pi devs are good at upstreaming their changes these days but there's always a delay between them fixing or proving something and getting patches into an upstream kernel so we use their downstream fork for Pi devices. Allwinner and Rockchip SoC's maintain their own patches for DRM things (mostly codec or core IP support) but most of the V4L2 items are common and upstream.

NB: Be aware that Pi support for RPi 0/1/2/3/4 is stateful (so nothing to do with v4l2_request) for H264 and stateless (using v4l2_request) for HEVC on RPi 4/5 hardware. Stateful things in the upstream kernel are not as mature/robust as stateless.

I'd suggest you reach out to JC and let him know that you're packaging for Gentoo. Although his primary focus is the Raspberry Pi OS, the secondary mission is ensuring Pi hardware works great under any distro, so it's good for him to know how and where his patchsets are being used.

Chewi · December 26, 2023 at 11:19 PM

Quote from chewitt

NB: Be aware that Pi support for RPi 0/1/2/3/4 is stateful (so nothing to do with v4l2_request) for H264 and stateless (using v4l2_request) for HEVC on RPi 4/5 hardware. Stateful things in the upstream kernel are not as mature/robust as stateless.

Damn, that was quite a big thing for me to miss! I have plenty of H.265 videos here, but I hadn't tried one yet. There is evidently more than just v4l2-request stuff going on in these patches though, as they definitely made a difference. lrusak's v4l2-drmprime patchset alone didn't help.

Quote from chewitt

I'd suggest you reach out to JC and let him know that you're packaging for Gentoo. Although his primary focus is the Raspberry Pi OS, the secondary mission is ensuring Pi hardware works great under any distro, so it's good for him to know how and where his patchsets are being used.

Will do, thanks.

Chewi · December 27, 2023 at 6:09 PM

I tried HEVC on my Pi 3 and it did struggle a bit. I'm not sure whether that's expected to work or not. I gather it doesn't have the hardware. JC did work on a special decoder, but it doesn't seem to be in this patchset.

I then tried my Pi 4 for the first time. Kodi works fine, but mpv is slower than ever, which is really strange. It says it's using hardware decoding, but I think it's the display part that's slow.

Chewi · December 27, 2023 at 11:11 PM

Figured it out. It wasn't an entirely fair comparison. mpv has two DRM PRIME modes, one where it renders via EGL/VAAPI/Wayland, and one where it renders directly to an overlay plane. It tries the first before the second, but there isn't a way to explicitly select one of them at runtime. mpv tries to use desktop GL by default. This was failing under the Pi 3, so it fell back to the second mode. Desktop GL did work under the Pi 4, so it used the first mode. Telling it to use GLES instead made it slightly faster but not much. Eventually, I forced it to used the second mode by requesting GLES and setting MESA_GLES_VERSION_OVERRIDE=2.0. Now it works fine. I'll contact the mpv developers about this.

**chewitt** · December 28, 2023 at 7:52 AM

RPi3 and older have no hardware decode support for HEVC but the legacy display pipeline was tweaked (using never upstreamed patches) to offload some compute tasks to the GPU and this achieves enough headroom for low-bitrate 1080p HEVC adnd low-bitrate 720p DRM'd HEVC media to be software decoded. The current display pipeline hasn't reimplementated that (50,000 LOC) patchset becaue it would require a large effort and more importantly, the goal is run upstream-only or minimally-patched code and it was ultimately a clever hack that upstream maintainers aren't going to accept.

**emveepee** · December 31, 2023 at 4:57 PM

Quote from Chewi

I then tried my Pi 4 for the first time. Kodi works fine, but mpv is slower than ever, which is really strange. It says it's using hardware decoding, but I think it's the display part that's slow.

This may not be anything but I having trouble with mpv and FFmpeg 6 on my Rpi5, I can't get it to detect the odd rpi4_8 and rpi4_10 profiles FFmpeg detects in v5 to enable drm. With FFmpeg 5 this isn't an issue, drm works fine.

Chewi · December 31, 2023 at 6:03 PM

Probably not related, sorry. I also only have a Pi 3b and 4b to test with.

rellla · February 12, 2024 at 9:05 AM

Hi all,

i guess this is the right thread to ask my questions. I'll tell sth about the basics first:

With https://github.com/Zabrimus/VDRSternELEC we have a "fork" of LE/CE which applies additional packages and patches to LE to have a VDR (http://www.tvdr.de) as a fully useable "standalone" client of VDR next to kodi instead of just the VDR backend, like it's implemented in LE already. This works really fine already but i have some problems with the VDRs output-plugin still. The codebase (kernel, ffmpeg) is the same like in kodi.

The output plugin uses one DRM plane to render the OSD/GUI on (software or GLES), the video is hardware decoded with ffmpeg and is rendered to another DRM plane directly through prime. This works fine for Rockchip and Allwinner.

Atm I try to implement support for Rpi4 and Rpi5. The path is ok for hardware-decoded h264 on the RPi4.

mpeg2 on RPi4 and h264/ mpeg2 on Rpi5 is software decoded with ffmpeg and there I have some "blocky" display output and i still haven't found out, where the issue is.

My question is: What is the kodi-rendering way for software decoded video on Rpi4 and Rpi5? Is kodi using some memcpy to render to a buffer for DRM or does it render the video with GLES? If so, does kodi use separate primary and overlay planes for GUI and video or does it render both (GUI and video) onto one plane?

If GLES for video should be the path, which might be of succes, I'll try to implement it. But if kodi is getting the software-decoded video to DRM via some other path, I'll go on searching for the problem on my side.

Would be good, if you can give me some hints for the right path.

Btw., the blocky videos render in kodi just fine.

Thanks for your help

rellla

Explanation of the ffmpeg patches

Similar Threads

Unofficial LE for RK356x RK3328\RK3399 RK3588(s)

Native DSD on Raspberry Pi