The infamous "invalid key" WiFi problem... I'd like to help, but which log files would be useful?

frankvw · October 16, 2023 at 9:27 AM

I just downloaded the latest LE11.0.3 image and installed it on my brand new RPI4. (Due to the RPi shortage they have been available here only since about 2 weeks ago.)

I'm not sure if it's relevant, but I'm on Linux and since there is currently no working GUI installer I used 'dd' to put the image on the SD card. This seems to have worked fine; LE started up without any problems.

Unfortunately LE is unable to connect to WiFi. I'm using a Huawei AX3 router (like the vast majority of all fibre users here in South Africa) set to WPA2/PSK. I'm entering the correct password but LE fails to connect and instead displays the notorious "invalid key" error.

I also tried running RaspberryOS on the same RPi4 and that connects without a hitch, so it is definitely a LE problem and not a hardware issue.

From what I see in the forum this is a problem that affects a number of users, but LE devs struggle with it:

Quote from chewitt

Based on the total number of issue reports to this forum and our IRC channel I'm calculating this problem is impacting ~0.0009% of the users running LE11 images.
[...]
As nobody on staff can reproduce this .. someone needs to start sharing connman and iwd debug logs. They will be verbose and sizeable but are often quite human-readable. The issue is likely in the comms between connman and iwd.

While there indeed may be a low number of users experiencing this problem, to those who do have it this extremely frustrating since it's essentially a showstopper in those situations where wired Internet is not practical (as in my case). However, I also understand the frustration of developers who are expected to "just fix this" without the proper data they need to work with, given that this appears to be a rather "glitchy" issue that most likely depends on a specific combination of hardware, routers and settings which makes it tricky to diagnose.

So I'd love to help. But given that I can't even ssh into my LE RPi I'll have to retrieve logfile data from the SD card, which I will happily do, but what log files will be useful, where do they live, and how should I submit them after having retrieved them manually?

// FvW

frankvw · October 16, 2023 at 1:23 PM

Update: several hours of testing have shown some progress: I set the WiFi router to use only IEEE802.11b/g and moved the RPi closer to the router (right next to it, in fact). Suddenly the problem disappeared.

Then I moved the RPi away from the router again (to the point where the WiFi signal was barely enough to be useable) and LE now complains again with the "invalid-key" error and requests that I enter the "correct" key.

So off-hand this does seem to be related to WiFi protocol implementations in some routers and/or signal strength/quality.

LybsterKodi · October 16, 2023 at 3:19 PM

Quote from frankvw

wired Internet is not practical (as in my case)

Just a thought - I live in a house built with foil backed plasterboard which isn't wifi friendly. My solution was powerline ethernet. Its nowhere as good as proper wired ethernet but it works consistently and goes over the various ring mains in the house albeit slower if it has to cross over.

**Da Flex** · October 16, 2023 at 6:52 PM

Quote from LybsterKodi

My solution was powerline ethernet.

Yep, or an Ethernet to WiFi adapter. The newer models have 5GHz support now. Benefit: Only Ethernet driver needed.

HarryH · October 16, 2023 at 7:35 PM

frankvw ,

do you have checked if it works if you limit the security settings in your router to WPA2? The important thing is to test it without PMF (Protected Managment Frames/802.11w). Is there an option to disable this at the Huawei AX3?

VLouis · October 17, 2023 at 6:58 AM

Quote from frankvw

Update: several hours of testing have shown some progress: I set the WiFi router to use only IEEE802.11b/g and moved the RPi closer to the router (right next to it, in fact). Suddenly the problem disappeared.

Then I moved the RPi away from the router again (to the point where the WiFi signal was barely enough to be useable) and LE now complains again with the "invalid-key" error and requests that I enter the "correct" key.
So off-hand this does seem to be related to WiFi protocol implementations in some routers and/or signal strength/quality.

Yeah, the tiny RPi onboard wifi antenna... probably a case again where the size is matter

I'm doesn't had the "invald-key" problem, only the weak signal and unstable connection... so I disabled the the onboard wifi (config.txt) and I'm using an USB wifi dongle with external antenna. Far better wifi connection. If you have an USB wifi dongle or can borrow one for testing, try it...

frakkin64 · October 17, 2023 at 10:52 PM

Other thing to look at, since it sounds like it is related to a signal problem, is your beacon loss counter on the client:

iw wlan0 station dump

beacon loss: 11

This one ^ was the pain in my ass, beacon loss 11 is actually great for this device (it was hundreds in a few hours with frequent disconnects), and iwd is pretty useless at giving you a clue what is wrong. You usually have to compile a separate kernel with debug turned on and turn on MLME debug to get a clue what's wrong. For me it turned out be beacon loss/timeout (but iwd reports everything as error 4, which is the AP disconnected). I ended up bumping the probe timeout to a pretty gross number of 2 seconds (normally it's 100ms), which seems to have helped. I probably ultimately need another AP to have full coverage, but it is working, and I still have those illusions of running ethernet in the house.

Other devices closer to the AP generally have no beacon losses.

birdr · October 22, 2023 at 12:28 AM

"Invalid Key" also happens for on Libreelec official 10.0.4 on a Raspberry Pi 4.

For me moving the Pi closer to the access point (TP link EAP-615) removes the problem.

I note that the signal is not actually that weak when the problem occurs - in the problem location my TV can stream via wifi in full UHD, and also my PS5 has no issues.

Indeed - if I allow libeelec to create a connection by having the Pi closer to the access point (only 1 metre closer - albeit maybe interference comes from the TV), and then return my Pi to the problem location, the connection remains fine, and it works.

However, if I then leave the Pi in the problem position and do either a hard or soft reboot, then usually (but not always) the "invalid key" problem has returned.

Happy to supply more info to enable resolution if someone can point me in the right direction. This is very reproducible for me.

These are some of the logs that frakkin64 mentioned.

Pi nearer access point and working for initial connection (mac removed):

~ # iw wlan0 station dump

Station xx:xx:xx:xx:xx:xx (on wlan0)

inactive time: 0 ms

rx bytes: 139418

rx packets: 201

tx bytes: 40943

tx packets: 262

tx failed: 0

signal: -73 dBm

tx bitrate: 195.0 MBit/s

rx bitrate: 234.0 MBit/s

authorized: yes

authenticated: yes

associated: yes

WMM/WME: yes

TDLS peer: yes

DTIM period: 1

beacon interval:100

short preamble: yes

connected time: 28 seconds

current time: 1416746254 ms

Moving away to problem location, leaves libreelec still working, with reduced rx bitrate.

Note - in multiple tests here I saw rx bitrate drop to ~14MBit/s, but also it would recover as high as 130Mbit/s )

~ # iw wlan0 station dump

Station xx:xx:xx:xx:xx:xx (on wlan0)

inactive time: 0 ms

rx bytes: 159469

rx packets: 366

tx bytes: 68123

tx packets: 413

tx failed: 8

signal: -72 dBm

tx bitrate: 117.0 MBit/s

rx bitrate: 65.0 MBit/s

authorized: yes

authenticated: yes

associated: yes

WMM/WME: yes

TDLS peer: yes

DTIM period: 1

beacon interval:100

short preamble: yes

connected time: 115 seconds

current time: 1416833347 ms

frankvw · October 22, 2023 at 8:29 AM

While all these replies are useful and appreciated, the idea here is to solve the WiFi problem, not to work around it with wired networking or external dongles. It worked fine in Leia on my RPi3b, and it works fine with RaspberryOS on my current RPi4b, so we're looking at a software issue here.

My original question remains: where do I find the system log files in LE? The Kodi log files are not relevant. The LE add-on has the option to share logs (although it is not clear which logs) but there's no option to extract logfile data and grep it (which would be my preferred initial approach). Just pastebinning it will not work, since that requires network access, and I can only reproduce the problem (and presumably have symptoms ending up in the log files) when there is no good network connection. It's a catch-22: reproducing the problem will prevent access to logfiles at that time, and restoring network and logfile access will make the symptom (and the log files?) go away.

Quote from HarryH

frankvw ,

do you have checked if it works if you limit the security settings in your router to WPA2? The important thing is to test it without PMF (Protected Managment Frames/802.11w). Is there an option to disable this at the Huawei AX3?

Nope. Nor is it necessary; putting the RPi close enough to the router solves the problem and removes the error message. Moving it further away can make it go completely haywire (switching connections twice a second) and also prevent it from connecting to my Android phone AP. In short, if the signal strength becomes low enough, connman and iwd (and whatever module may be involved) lose their sanity which obviously shouldn't happen. chewitt is right; this needs to be kicked upstairs to the developers of these modules; the LE developers can't be expected to resolve this.

frakkin64 · October 22, 2023 at 10:59 AM

Quote from frankvw

It's a catch-22: reproducing the problem will prevent access to logfiles at that time, and restoring network and logfile access will make the symptom (and the log files?) go away.

The log files are not going to be useful; they will indicate what you already know, and others have posted them already. If you want to know more then you need to re-compile the kernel with mac80211 & MLME debug on, turn on persistent logs, and reproduce the problem.

Quote from frankvw

this needs to be kicked upstairs to the developers of these modules; the LE developers can't be expected to resolve this.

The iwd mailing list is the place for that, LE developers would do that if they could reproduce it, but it seems based on prior threads that they can't reproduce it. It seems to me that you would have to bring that up on the mailing list, since no one can proxy for you without being able to reproduce the problem.

The difference between Raspberry Pi OS & Leia is iwd, both of them historically were using wpa_supplicant. As soon as LE (in 10) switched to iwd that's when these reports started rolling in. iwd does work great here with rtw88 devices & openwrt AP, so what that unique difference is unknown.

Quote from birdr

These are some of the logs that frakkin64 mentioned.

Looks like your probably using the built-in wifi? It doesn't indicate beacon loss, unfortunately. But noticed you did have some tx failed, which to me would suggest signal problems.

It's possible that invalid key is triggered off of response time, perhaps the authentication packet is sent out, it doesn't get a response within X time and that is considered an invalid key. I don't know if that's how it works, but it would possibly explain why it works sometimes or when it is closer.

frankvw · October 22, 2023 at 12:02 PM

Quote from frakkin64

The difference between Raspberry Pi OS & Leia is iwd, both of them historically were using wpa_supplicant. As soon as LE (in 10) switched to iwd that's when these reports started rolling in.

Aha. I hadn't realized that yet (I've been out of the picture for a while.) I now see reports of iwd issues on the RPi4 as early as 2019, then on Raspbian Buster. Unfortunately I can't see this being a big, high priority issue...

frakkin64 · October 22, 2023 at 12:43 PM

Quote from frankvw

Aha. I hadn't realized that yet (I've been out of the picture for a while.) I now see reports of iwd issues on the RPi4 as early as 2019, then on Raspbian Buster. Unfortunately I can't see this being a big, high priority issue..

That's probably folks that chose to use iwd on Raspberry Pi OS. That's the nice thing about Debian, you can try all kinds of alternatives from the standard installation. As far as I know Raspberry Pi didn't switch the default supplicant, even a fresh install of bookworm appears to still use wpa_supplicant:

$ ps fuxwa |grep wpa

root 542 0.0 0.1 17148 10628 ? Ss Oct20 0:14 /sbin/wpa_supplicant -u -s -O DIR=/run/wpa_supplicant GROUP=netdev

iwd is the new thing (I think it started somewhere around 2017 or 2018), and probably around 2019 is when it became compelling as a replacement for wpa_supplicant.

One other thought is perhaps trying a USB dongle. I don't know which ones are recommended for LE11, as I compile my own with rtw88 drivers backported from the mainline kernel. It might indicate if the common thread is the broadcom drivers, I don't know if anyone has collected reports and correlated the common thread here. I know Ethernet is what is recommended, and perhaps a lot of folks follow that as Wi Fi is a pain in the butt.