Hardware accelerated video in 2022

    Rather global question, right now in the year of 2022, is there anything resembling general consensus on the unified Linux API for video hardware decoding and zero copy rendering?

    I’m only talking about mainline kernel and major opensource players/frameworks/players.

    Is it VAAPI only for big boy x64 AMD/Intel GPU’s? Browsers, players like mpv, kodi seem to use it by default.

    Stateless V4L and DRM for various ARM GPU’s? I see Libreelec has a lot of patches for kernel, ffmpeg and kodi, it’s not really mainlined yet?

    How does Vulkan video API fit in there?

    The x86_64 world is a little different to the ARM world. Intel/AMD standardised their hardware decoders around VAAPI so Kodi can call ffmpeg in a consistent way regardless of each, although at some point capabilities need to reflect which GPU/card and what the TV/panel supports. It all works reasonably well though advanced things like tonemapping still need to be plumbed in. Even nVidia finallly caved and added support for GBM/mesa to their latest drivers; and although Kodi does not support NVDEC (needed) someone wrote an NVDEC > VAAPI ship and we have a proof of concept implementation; although you need to run under Wayland which doesn't support fun stuff like "adjust-refresh" by design so it's not quite the solution nVidia users need (better than "no support" though)..

    The ARM world is domainated by ARM Mali GPUs (well supported with FOSS drivers now) with separate vendor-specific decoders. These have a mix of "stateless" and "stateful" types under the V4L2 banner. The stateless V4L2 "requests" uAPI is stable and supports a good range of codecs upstream; it's still up to individual drivers to support them. The stateful "mem2mem" side is not declared stable yet, and with the dependency on vendor firmware blobs is generally a bit futher behind. There is a lot of collaboration happening on testing and upstreaming to the kernel - and LE is heavly involved in that (Kodi is a good test app).

    On the userspace side most of the current activity on ffmpeg is focussed on making everything work. The process of usptreaming is looming on the horizon but it's still something to come. Again there's quite a bit of collaboration happening on this and we're involved.

    Vulkan is an alternative to things like OpenGL/OpenGLES .. so "standards" .. https://xkcd.com/927/

    Quote from chewitt

    although you need to run under Wayland which doesn't support fun stuff like "adjust-refresh" by design so it's not quite the solution nVidia users need (better than "no support" though)..

    I don't think wayland is a requirement for nvdec vaapi wrapper. However, I don't know if Kodi starts in GBM mode with new nvidia drivers.

    offbeat In short, I don't think there will ever be one single go-to video decoding API under Linux, mostly becasue supported HW is very diverse. However, I believe it will boil down to few different apis.

    Quote from chewitt

    The x86_64 world is a little different to the ARM world. Intel/AMD standardised their hardware decoders around VAAPI so Kodi can call ffmpeg in a consistent way regardless of each, although at some point capabilities need to reflect which GPU/card and what the TV/panel supports. It all works reasonably well though advanced things like tonemapping still need to be plumbed in. Even nVidia finallly caved and added support for GBM/mesa to their latest drivers; and although Kodi does not support NVDEC (needed) someone wrote an NVDEC > VAAPI ship and we have a proof of concept implementation; although you need to run under Wayland which doesn't support fun stuff like "adjust-refresh" by design so it's not quite the solution nVidia users need (better than "no support" though)..

    The ARM world is domainated by ARM Mali GPUs (well supported with FOSS drivers now) with separate vendor-specific decoders. These have a mix of "stateless" and "stateful" types under the V4L2 banner. The stateless V4L2 "requests" uAPI is stable and supports a good range of codecs upstream; it's still up to individual drivers to support them. The stateful "mem2mem" side is not declared stable yet, and with the dependency on vendor firmware blobs is generally a bit futher behind. There is a lot of collaboration happening on testing and upstreaming to the kernel - and LE is heavly involved in that (Kodi is a good test app).

    On the userspace side most of the current activity on ffmpeg is focussed on making everything work. The process of usptreaming is looming on the horizon but it's still something to come. Again there's quite a bit of collaboration happening on this and we're involved.

    Vulkan is an alternative to things like OpenGL/OpenGLES .. so "standards" .. https://xkcd.com/927/

    Maybe a bit off-topic and maybe too philosophical, but do you see the vendor-specific ARM-based stuff as the 'future' of Kodi? I see distros like OSMC and (maybe to a lesser extent) ZDMC, et. all, supporting stuff more quickly simply because they pack the vendor-specific blobs in there and it 'mostly' works... I know HDR stuff in the Linux kernel/rendering pipeline has been a nightmare so far for generic stuff.

    I ask mainly because I would love to build out another fanless x64 box for my home theater Kodi device (currently a Vero 4k+); I find the embedded systems to simply be under-powered.

    Quote from isamudaison

    I ask mainly because I would love to build out another fanless x64 box for my home theater Kodi device (currently a Vero 4k+); I find the embedded systems to simply be under-powered.

    You shouldn't need 10W+ to play media and run a pretty skin either, efficiency is king here IMHO

    Currently I think an S905X3 SoC is tough to beat in terms of 'sweet spot' capabilities and efficiency.

    Quote from powerarmour

    You shouldn't need 10W+ to play media and run a pretty skin either, efficiency is king here IMHO

    Currently I think an S905X3 SoC is tough to beat in terms of 'sweet spot' capabilities and efficiency.

    Simply not my experience when you throw in LiveTV, and a media library of multiple thousands of objects...

    Quote from isamudaison

    do you see the vendor-specific ARM-based stuff as the 'future' of Kodi?

    Team Kodi has been pursuing a "standards" based approach since 2016 and it's the reason Android now works very well. Linux is iinherently more complicated but a lot of the kernel frameworks are now in place and stable. Supporting older and current hardware has challenges as it means rework to existing codebases and commercial interests rarely fund work on old chips. However Google forcing Android device vendors to get ever-closer to upstream means future chip generations will increasingly have support written around the stable APIs and convergence will come. ARM devices are increasingly performant, and Intel stuff is still available (at a price) for anyone needing more.

    Quote from isamudaison

    Maybe a bit off-topic and maybe too philosophical, but do you see the vendor-specific ARM-based stuff as the 'future' of Kodi?

    Got two rather ancient old media devices running in my living room.

    First is media player, MIPS Sigma SOC. Fantastic hardware, plays any 8bit H.264 with dynamic Hz changing, every older codec, not a single visible framedrop and glitch. But welcome back to year 2007 and say hello to kernel 2.6.22. If you want to play something off network share, open up SMB1.

    Another one is DVB-S2 SAT receiver, MIPS Realtek SOC. Once again, vendor blobs, kernel 3.13, you get what you get. Want to plug additional DVB-T USB stick supported in mainline? Tough luck.

    Both devices got hardware beefy enough to run something like LibreElec (3D acceleration, hardware video decoding, enough RAM), yet both are rotting away, feature and security wise, waiting to become e-waste.

    Who wants such 'future'?

    Quote from offbeat

    Both devices got hardware beefy enough to run something like LibreElec (3D acceleration, hardware video decoding, enough RAM), yet both are rotting away, feature and security wise, waiting to become e-waste.

    Who wants such 'future'?

    Of course some old devices can have a long life, but you've got to have a developer who has the time to put in hard work and have it not be an utterly thankless task.

    A lot of old drivers won't even build these days, every dog has it's day unfortunately...

    Numerous Linux/X11 Display Drivers Can No Longer Even Properly Build - Phoronix

    Quote from powerarmour

    but you've got to have a developer who has the time to put in hard work

    Developers on payroll of Realtek and other vendors put their time in either way. Its management who wasn't interested in going extra mile mainlining code, adjusting own deadlines, etc.

    I'm happy that some companies are starting to see the benefits of mainlining things, others are getting pushed there by Google because of Android sertification.

    For us, end users, it will hopefully result less fragmentation, less e-waste. And maybe one or two less video decoding standard to care about, if we stay on topic.

    "Team Kodi" is more accurately described as "small group of developers doing their own thing with occasionally overlapping interests" and as is common with non-commercial FOSS projects there is not much product management and no schedule. One of the few things its developers have ever consistently agreed upon is the need to reduce the complexity and improve maintainability of the codebase. That's the prime driver and motivation for the V4L2 long-term technical direction. Linux is inherently a fragmented ecosystem so we need to follow standards and be an advocate and agent-of-change for the establishment of those standards.