Other solutions suggested "use a cable"
I now also tried the latest LE 12 nightly which should with a more recent kernel than 6.6.78 - but no luck. invalid key error still is their with the router being two meters away...maybe it also fails if the signal is too good?
Will try LE13 nightly next to see if this supports wifi.
And from what it looks like it seems to be a pure software problem/bug? The same PI SBC has no problems with the same wifi/router when running PiOS for example.