Posts by GamerBene19

GamerBene19

There was indeed an issue in the Linux kernel (which only occurred in combination with iwd). There is a patch (not by me) linked in my response on the GitHub Issue that resolves the DHCP issue.

I am now back to finding out what causes the invalid-key issue. So far, what I was able to find out is that my dongle enters some kind of failure state after some time (have seen it happen after ~1.5h, but also only after ~18h). In that state the dongle does not seem to send out packets anymore.

If I manually (try to) (re)connect to a wifi network dmesg output shows three tries and that the handshake timed out, iwd logs the authentication timeout and responds a failed connection attempt to connman which it interprets as it being an invalid key.

Judging from a packet capture via Wireshark, when in the 'failed' state, it looks like the dongle is not actually sending out authentication requests.

Unfortunately it's a little hard to diagnose this issue as
1) I don't have a known good/bad state to compare/bisect
2) It takes (potentially) a lot of time to replicate the issue
3) I can't be certain it's actually gone if it has not happened after x hours/minutes

I'm currently testing with wpa_supplicant. So far (~22h) it has not failed yet, which aligns with previous reports and with what chewitt mentioned about wpa_supplicant fixing the issue.

GamerBene19

Quote from bam

Hey utdrmac, could you share your script to provide complete solution?

Sorry for necrobumping. I've stumbled upon this thread through googling while I was debugging another issue.
In case this is is still relevant: If you have a LE image with wpa_supplicant you can use something like systemctl edit --full wpa_supplicant and then modify the ExecStart line accordingly.

GamerBene19

Quick update: I was able to narrow the range down a bit to the (upstream) commits

2ed7064934dda0766495eaaf6b26332eb63e7df8 (which is the first commit on 6.8.y that builds an LE image) issue does not appear with this commit
03ff51a768fc34ebbe51ad1a60bd566fc672eb46 (which is the first commit on 6.9.y that builds an LE image) issue does appear with this commit

I have also opened a issue in the raspberrypi linux kernel's GitHub repo: https://github.com/raspberrypi/linux/issues/7245

GamerBene19

So from what I was able to find out, the issue is that e8f897f4afef0031fe618a8e94127a0934896aba (which is a upstream linux commit) does not have the dts/overlay files the raspberry kernel has and that the LibreELEC build expects.

I tried adding them, some changes to Makefiles and some files from include/... from the HEAD of 6.8.y (which builds successfully) via a patch and I god a building kernel, but when I flash the image, the disk LED flashes a few and then the Pi simply turns off (both LEDs off, no display output).

Any advice on what would be a good way to find out which commit between 6.8.y and 6.9.y introduced my problem is appreciated.

GamerBene19

So I started bisecting between the two commits and need(ed) to test a merge base

Code

Bisecting: a merge base must be tested
[e8f897f4afef0031fe618a8e94127a0934896aba] Linux 6.8

Unfortunately the build for this commit fails with

Code

...
[QA CHECK] [linux] [safe_remove]:
path does not exist: /build/build.LibreELEC-RPi2.arm-12.2-devel/install_pkg/linux-e8f897f4afef0031fe618a8e94127a0934896aba/usr/share/bootloader/overlay_map.dtb
cp: cannot stat 'arch/arm/boot/dts/overlays/README': No such file or directory
FAILURE: scripts/build linux:target during makeinstall_target (package.mk)
*********** FAILED COMMAND ***********
cp -p arch/${TARGET_KERNEL_ARCH}/boot/dts/overlays/README ${INSTALL}/usr/share/bootloader/overlays
**************************************
*********** FAILED COMMAND ***********
${SCRIPTS}/build "${1}" "${PARENT_PKG}"
**************************************
FAILURE: scripts/install linux:target has failed!

Display More

There was a similar issue in

Post

RE: TV adapter drivers missing from 11.0.4?

*sigh* these are the moments I was afraid of, feeling like a total noob. I checked out 11.0.6 which was the most recent at the time. First build an image, went fine. Then I modified package.mk to reflect the bisect suggested commit. Build again and then:

[QA CHECK] [linux] [safe_remove]:
path does not exist: /home/rataplan_/http://LibreELEC.tv/build.LibreELEC-RPi4.arm-11.0-devel/install_pkg/linux-2368afd60f647889d90fa4a42c7b27548f77dbd9/usr/share/bootloader/overlay_map.dtb
cp: cannot stat…

Rataplan626

February 5, 2024 at 2:41 PM

Rataplan626 do you remember what you did to solve it (to bisect successfully)?

GamerBene19

Quote from GamerBene19

I'll try the different version of the rpi-kernel (rpi-6.{7,8,9,10,11}.y) next to narrow down when it breaks. Might be a few days until I report back though as I won't be home over the weekend.

I was able to test them before leaving.
rpi-6.7.y (which uses 6.7.12) and rpi-6.8.y (which uses 6.8.12) work, rpi-6.9.y (which uses 6.9.12) breaks.

For the exact config I used see the commits on this branch.

I'll try to git bisect the relevant rpi kernel commits once I get back.

GamerBene19

I'll try the different version of the rpi-kernel (rpi-6.{7,8,9,10,11}.y) next to narrow down when it breaks. Might be a few days until I report back though as I won't be home over the weekend.

GamerBene19

So I've bit the bullet bisected commits that touched linux kernel, drivers and LibreELEC options between 12.0.2 and 12.2.1 with
git bisect start '--' 'distributions/LibreELEC/options' 'packages/linux-drivers/' 'packages/linux'

According to bisect, this is the first bad commit

linux (RPi): update to 6.12.38-9c09b75 · LibreELEC/LibreELEC.tv@014e83d

Signed-off-by: Matthias Reichl <[email protected]>

github.com

This is my bisect log

Code

git bisect start '--' 'distributions/LibreELEC/options' 'packages/linux-drivers/' 'packages/linux'
# status: waiting for both good and bad commits
# bad: [6209f5a3b067c45fbf527286dd84ee483294b46a] Merge pull request #10633 from heitbaum/binaries-12.2
git bisect bad 6209f5a3b067c45fbf527286dd84ee483294b46a
# status: waiting for good commit(s), bad commit known
# good: [f3fdd11916f8a47dc5a11c3a4c99cb7c7ffac78b] Merge pull request #9707 from chewitt/kodi-tag-revert
git bisect good f3fdd11916f8a47dc5a11c3a4c99cb7c7ffac78b
# bad: [ba3527f093fc645566acb4a445a9f70d0573c59c] Merge pull request #10241 from heitbaum/uplift-kernel-projects-to-master
git bisect bad ba3527f093fc645566acb4a445a9f70d0573c59c
# good: [81c3943df84f052abc72b2571482f2eab5b4007f] mali-midgard: drop package
git bisect good 81c3943df84f052abc72b2571482f2eab5b4007f
# bad: [b9fbadfa84960761e9ab01c0da954cf49f41f533] linux: update to 6.15.6
git bisect bad b9fbadfa84960761e9ab01c0da954cf49f41f533
# skip: [1267bd8d3d8fa0305328bfab220e29803397375f] linux (RPi): apply rtw88 patches
git bisect skip 1267bd8d3d8fa0305328bfab220e29803397375f
# skip: [5d1274fe0966b4ce0400f10cd2c25cd35a078e0a] linux: add rtw88 patches
git bisect skip 5d1274fe0966b4ce0400f10cd2c25cd35a078e0a
# good: [1318b53abc8f210a683076270abcd0e6a97503b8] mali-utgard: drop package
git bisect good 1318b53abc8f210a683076270abcd0e6a97503b8
# bad: [014e83d91f5884dde6be828fb8effd5892487741] linux (RPi): update to 6.12.38-9c09b75
git bisect bad 014e83d91f5884dde6be828fb8effd5892487741
# good: [92c221ad0bbc12e981584e24805dc5eb40c6d7c0] linux: drop ancient patches that never went upstream
git bisect good 92c221ad0bbc12e981584e24805dc5eb40c6d7c0
# first bad commit: [014e83d91f5884dde6be828fb8effd5892487741] linux (RPi): update to 6.12.38-9c09b75

Display More

1267bd8 and 5d1274f do not build (which is also mentioned in https://github.com/LibreELEC/LibreELEC.tv/pull/10247) so i skipped them.

014e83d also did not compile at first due to a missing call_read_iter which is referenced in out-of-kernel driver RTL8192DU and was removed in kernel version 6.10. For my bisect I simply created this patch to get it compling.

Diff

diff --git a/os_dep/osdep_service.c b/os_dep/osdep_service.c
index fd36e3b..6a2ae13 100644
--- a/os_dep/osdep_service.c
+++ b/os_dep/osdep_service.c
@@ -56,7 +56,7 @@ static ssize_t new_sync_read(struct file *filp, void __user *buf, __kernel_size_
         kiocb.ki_pos = *ppos;
         iov_iter_init(&iter, READ, &iov, 1, len);

-        ret = call_read_iter(filp, &kiocb, &iter);
+        ret = filp->f_op->read_iter(&kiocb, &iter);
         BUG_ON(ret == -EIOCBQUEUED);
         *ppos = kiocb.ki_pos;
         return ret;

Display More

I have also opened a issue in the driver repo in case it is helpful to someone.

GamerBene19

Quote from GamerBene19

I'll try disabling RTL8192DU and RTL8812AU on 12.0.2 next, to see if disabling them somehow causes the issue (again, i don't expect it too, but you never know).

As expected it did not. I've simply cherry-picked the four commits from the PR that disable them onto the 12.0.2 tag in case anyone wants to replicate it.

GamerBene19

LE 13 with the kernel from 12.0.2 works too, so it seems to be a kernel/driver issue.

I have reset packages/devel/glibc/, packages/compress/xz/, packages/linux-drivers/, packages/linux/, distributions/LibreELEC/options and projects/RPi/devices/RPi2/linux/linux.arm.conf to the version from tag 12.0.2. I have also pushed these changes onto a branch too.

So from what I can tell the issue seems to be caused by either

1. Changes the LE project made between 12.0.2 and 12.2.1 to the (RPi2) kernel/drivers -- e.g. via (removal) of patches

or

2. Changes made upstream between kernel versions 6.6.70 (which LE 12.0.2 uses) and 6.12.56 (which LE 12.2.1 uses)

As mentioned previously in this thread, LE has disabled the out-of-tree 8192CU drivers in the past. These would have been used for my chipset in the past (and seem to still be used under RaspberryPi OS).
But those drivers were already disabled in 12.0.2 (and in 11.x too for that matter), so I don't think it's relevant here.

Compared to 12.0.2, 12.2.1 further disabled RTL8192DU and RTL8812AU, but I would expect that this has no effect for my dongle, as these drivers are for different chipsets (at least based on their name).

I'm not quite sure how to continue debugging at this point, perhaps someone has a good suggestion.

I'll try disabling RTL8192DU and RTL8812AU on 12.0.2 next, to see if disabling them somehow causes the issue (again, i don't expect it too, but you never know).

GamerBene19

DHCP on LibreELEC 12.0.2 with connman (2.0) and iw (6.17) versions from master works.

I had to patch some checksums and I've also pulled in iwd (for good measure) and nftables (marked as dependency of conman in master branch) along with a newer version libnftnl (needed by nftables). If anyone is interested, my I've pushed my changes to a separate branch in my fork.

I'll (try to) test 12.0.2 kernel/drivers on master/nightly next.

GamerBene19

I have opened a WiFi hotspot on my computer and have attached a Wireshark capture for both LE 12.0.2 and 13-nightly to test option 1 from above. In the image you can see a comparison of nightly (left), where DHCP is broken, and LE12.0.2 (right) where DHCP works. Notice how on the left (13-nightly) there are no data packets after the EAPOL messages (which are the WPA key exchange as far as I can tell).

Note that I don't think reception can be an issue here as the two WiFi dongles/antennas are literally <0.5m apart on my desk.

GamerBene19

Quote from VLouis

So, if you can't connect at all, maybe the main problem isn't in the LE code.

Depends on your definition of connecting. I can authenticate and associate but it seems like I can't send any traffic after that.

Quote from VLouis

What encryption mode is set on that AP?

WPA2. Although a WPA3-enabled AP seemed to work with the older LE12.0.2 version too.

GamerBene19

I think that the lack of of something like received DHCP packet xid 0x5da74051 in failing log means that either
1. DHCP discovery never actually gets sent
or
2. The reply is sent but never reaches the Pi
or
3. The reply gets sent, but something causes it to take over 5s, always missing the timeout

To rule out 3., I increased the DHCP discovery timeout to 30s; it still fails. The connection attempt is eventually aborted by another timeout and no LL address is configured in this case.

I also took a packet capture on my router (which is not the AP the Pi is connecting to) of the relevant interface. I did not see any DHCP related packets going in or out while the Pi tries to connect. So I think 2. is unlikely too.

I currently strongly suspect that the Pi has trouble actually sending out the DHCP discovery packets (aka option 1). Although I have no idea what could be causing it.

Looking at the relevant source code of connman, the packets are sent with the sendto syscall (and notably without iw). So for now I would rule out iw aswell, which (i think) would only leave the kernel and/or the rtl8xxxu module, but that assumption might be wrong.

I'll try to get LibreELEC 12.0.2 running with current iw and connman versions tomorrow.
chewitt Do you know if there are other dependencies I need to consider too or if I can simply upgrade iw and connman on 12.0.2?

I think another path forward could be git bisecting between 12.0.2 and 12.2.1, although there are ~950 commits between the two tags and I would need to rebuild the image for each bisect step. Also there might be commits in the history that do not build or break WiFi/networking in another way, so I'm not sure how viable it really is.

GamerBene19

I've enabled debug settings for connman with:

Code

LibreELEC:~ # cat /run/libreelec/debug/connman.conf
CONNMAN_DEBUG="-d"
CONNMAN_DHCP_DEBUG=1

and judging from what is logged about DHCP

Code

[~] <> * grep -i dhcp connman.log4
Feb 12 19:44:19 LibreELEC connmand[747]: ../src/network.c:set_connected_dhcp() network 0x1f17c18
Feb 12 19:44:19 LibreELEC connmand[747]: ../src/ipconfig.c:__connman_ipconfig_save() ipconfig 0x1f185c8 identifier wifi_74da38542906_496f54_managed_psk method dhcp
Feb 12 19:44:19 LibreELEC connmand[747]: ../src/dhcp.c:__connman_dhcp_start()
Feb 12 19:44:19 LibreELEC connmand[747]: ../src/ipconfig.c:__connman_ipconfig_ref_debug() 0x1f185c8 ref 3 by ../src/dhcp.c:708:__connman_dhcp_start()
Feb 12 19:44:19 LibreELEC connmand[747]: ../src/network.c:connman_network_ref_debug() 0x1f17c18 name IoT ref 4 by ../src/dhcp.c:712:__connman_dhcp_start()
Feb 12 19:44:19 LibreELEC connmand[747]: ../src/dhcp.c:dhcp_initialize() dhcp 0x1f0ebe8
Feb 12 19:44:19 LibreELEC connmand[747]: DHCP index 4: switch listening mode (0 ==> 1)
Feb 12 19:44:19 LibreELEC connmand[747]: DHCP index 4: sending DHCP discover request
Feb 12 19:44:24 LibreELEC connmand[747]: DHCP index 4: sending DHCP discover request
Feb 12 19:44:29 LibreELEC connmand[747]: DHCP index 4: sending DHCP discover request
Feb 12 19:44:34 LibreELEC connmand[747]: DHCP index 4: sending DHCP discover request
Feb 12 19:44:39 LibreELEC connmand[747]: DHCP index 4: sending DHCP discover request
Feb 12 19:44:44 LibreELEC connmand[747]: DHCP index 4: sending DHCP discover request
Feb 12 19:44:49 LibreELEC connmand[747]: ../src/dhcp.c:no_lease_cb() No lease available ipv4ll 0 client (nil)
Feb 12 19:44:49 LibreELEC connmand[747]: ../src/dhcp.c:dhcp_invalidate() dhcp 0x1f0ebe8 callback 0
Feb 12 19:44:49 LibreELEC connmand[747]: ../src/dhcp.c:dhcp_invalidate() last address (null)
Feb 12 19:44:59 LibreELEC connmand[747]: ../src/dhcp.c:ipv4ll_available_cb() IPV4LL available
Feb 12 19:44:59 LibreELEC connmand[747]: ../src/network.c:dhcp_success() lease acquired for ipconfig 0x1f185c8

Display More

and by looking through the connman code it looks like at most 6 DHCP discovery requests are sent out with a timeout of 5 seconds and if the threshold is hit an IPv4 LL address is configured.

Edit: Under LE 12.0.2 where DHCP works it looks like this

Code

[~] <> * grep -i dhcp connman.log5
Feb 12 23:52:51 LibreELEC connmand[1448]: ../src/network.c:set_connected_dhcp() network 0x11d0f90
Feb 12 23:52:51 LibreELEC connmand[1448]: ../src/dhcp.c:__connman_dhcp_start()
Feb 12 23:52:51 LibreELEC connmand[1448]: ../src/ipconfig.c:__connman_ipconfig_ref_debug() 0x11d1370 ref 3 by ../src/dhcp.c:708:__connman_dhcp_start()
Feb 12 23:52:51 LibreELEC connmand[1448]: ../src/network.c:connman_network_ref_debug() 0x11d0f90 name WGLAN ref 4 by ../src/dhcp.c:712:__connman_dhcp_start()
Feb 12 23:52:51 LibreELEC connmand[1448]: ../src/dhcp.c:dhcp_initialize() dhcp 0x1199c70
Feb 12 23:52:51 LibreELEC connmand[1448]: DHCP index 4: switch listening mode (0 ==> 1)
Feb 12 23:52:51 LibreELEC connmand[1448]: DHCP index 4: sending DHCP discover request
Feb 12 23:52:57 LibreELEC connmand[1448]: DHCP index 4: sending DHCP discover request
Feb 12 23:52:58 LibreELEC connmand[1448]: DHCP index 4: received DHCP packet xid 0x5da74051 (current state 0)
Feb 12 23:52:58 LibreELEC connmand[1448]: DHCP index 4: init ip 255.255.255.255 secs 1536 -> broadcast flag on
Feb 12 23:52:58 LibreELEC connmand[1448]: DHCP index 4: start request (retries 0)
Feb 12 23:52:58 LibreELEC connmand[1448]: DHCP index 4: sending DHCP request (state 2)
Feb 12 23:53:03 LibreELEC connmand[1448]: DHCP index 4: request timeout (retries 0)
Feb 12 23:53:03 LibreELEC connmand[1448]: DHCP index 4: start request (retries 1)
Feb 12 23:53:03 LibreELEC connmand[1448]: DHCP index 4: sending DHCP request (state 2)
Feb 12 23:53:03 LibreELEC connmand[1448]: DHCP index 4: received DHCP packet xid 0x5da74051 (current state 2)
Feb 12 23:53:03 LibreELEC connmand[1448]: DHCP index 4: switch listening mode (1 ==> 0)
Feb 12 23:53:03 LibreELEC connmand[1448]: ../src/dhcp.c:lease_available_cb() Lease available
Feb 12 23:53:03 LibreELEC connmand[1448]: ../src/dhcp.c:lease_available_cb() last address 10.0.40.245
Feb 12 23:53:03 LibreELEC connmand[1448]: ../src/dhcp.c:lease_available_cb() c_address (null)
Feb 12 23:53:03 LibreELEC connmand[1448]: ../src/network.c:dhcp_success() lease acquired for ipconfig 0x11d1370
Feb 12 23:53:03 LibreELEC connmand[1448]: ../src/ipconfig.c:__connman_ipconfig_save() ipconfig 0x11d1370 identifier wifi_74da38542906_57474c414e_managed_psk method dhcp
Feb 12 23:53:03 LibreELEC connmand[1448]: ../src/ipconfig.c:__connman_ipconfig_save() ipconfig 0x11d1370 identifier wifi_74da38542906_57474c414e_managed_psk method dhcp
Feb 12 23:53:03 LibreELEC connmand[1448]: DHCP index 4: start bound
Feb 12 23:53:03 LibreELEC connmand[1448]: DHCP index 4: processed DHCP packet (new state 3)
Feb 12 23:53:03 LibreELEC connmand[1448]: ../src/service.c:service_ip_bound() service 0x11d10f8 (wifi_74da38542906_57474c414e_managed_psk) type 1 (IPv4) ipconfig 0x11d1370 method 4 (dhcp)

Display More

GamerBene19

I'd like to make modifications to build.LibreELEC-RPi2.arm-13.0-devel/build/connman-1.45/src/dhcp.c to add additional debug prints to debug the DHCP issue.

Can I just edit the file and rerun make image, or is the process more involved?

If it is: What are the steps that I need to take to get my changes into the compiled image?

GamerBene19

Quote from chewitt

Make sure you set/configure the wireless regulatory domain for your country in LE settings, it can make a large difference to the radio spectrum that's usable.

I have no access to the devices display output at the moment, but at least via SSH it looks correct

Code

LibreELEC:~ # iw reg get
global
country DE: DFS-ETSI
	(2400 - 2483 @ 40), (N/A, 20), (N/A)
	(5150 - 5250 @ 80), (N/A, 23), (N/A), NO-OUTDOOR, AUTO-BW
	(5250 - 5350 @ 80), (N/A, 20), (0 ms), NO-OUTDOOR, DFS, AUTO-BW
	(5470 - 5725 @ 160), (N/A, 26), (0 ms), DFS
	(5725 - 5875 @ 80), (N/A, 13), (N/A)
	(5945 - 6425 @ 160), (N/A, 23), (N/A), NO-OUTDOOR
	(57000 - 66000 @ 2160), (N/A, 40), (N/A)

and I have written down the following in my documentation of the setup steps

so I am reasonably sure I've set it correctly.

GamerBene19

Quote from VLouis

Hi, that could be because of the "unstable" wifi source. Maybe one of your neighbor use the same wifi channel, you did a channel analysis to see the interferences? The possible existent connman-iwd issue I think appear at reconnecting. But even without the "reconnection issue", an unstable network and continuous streaming isn't the best option. I'm just wondering... that part of the stable/unstable wifi network could be the reason that only a few users reported this issue...

Sorry for not answering about that earlier. Yes I did look at channel utilization and signal strength of other APs. To me it looked fine.
I just looked again, this roughly is the current situation according to iw dev wlan0 scan:

Signal Strength	Channel
-56 dBm	1
-90 dBm	1
-90 dBm	1
-96 dBm	1
-82 dBm	6
-94 dBm	6
-82 dBm	11
-90 dBm	11
-94 dBm	6
-82 dBm	6
-96 dBm	1
-90 dBm	6
-77 dBm	5
-76 dBm	5
-86 dBm	6
-86 dBm	6
-98 dBm	10

I am connected to the network of the first row (interestingly iw wlan0 station dump shows a better dBm value for the connected WiFi; don't know why). It might be better to switch to channel 11, but I think overall this is situation is fine considering all other dBm values of the same channel are >90

Posts by GamerBene19

[RPi2B] Invalid Key/WiFi issues with USB dongle

[RPi2] Unable to connect to WiFi WPA/WPA2 Personal

[RPi2B] Invalid Key/WiFi issues with USB dongle

[RPi2B] Invalid Key/WiFi issues with USB dongle

[RPi2B] Invalid Key/WiFi issues with USB dongle

RE: TV adapter drivers missing from 11.0.4?

[RPi2B] Invalid Key/WiFi issues with USB dongle

[RPi2B] Invalid Key/WiFi issues with USB dongle

[RPi2B] Invalid Key/WiFi issues with USB dongle

[RPi2B] Invalid Key/WiFi issues with USB dongle

[RPi2B] Invalid Key/WiFi issues with USB dongle

[RPi2B] Invalid Key/WiFi issues with USB dongle

[RPi2B] Invalid Key/WiFi issues with USB dongle

[RPi2B] Invalid Key/WiFi issues with USB dongle

[RPi2B] Invalid Key/WiFi issues with USB dongle

[RPi2B] Invalid Key/WiFi issues with USB dongle

[RPi2B] Invalid Key/WiFi issues with USB dongle

[RPi2B] Invalid Key/WiFi issues with USB dongle

[RPi2B] Invalid Key/WiFi issues with USB dongle