WiFi lost, auto-reconnects only after minutes

1st Official Post

    Hi,

    I have setup an rpi4 with a TV Hat Tuner and everything is working pretty well, apart from a serious wifi connection issue:

    every now and then, the rpi disconnects from the wifi and reconnects almost immediately (it's a matter of few seconds) and this doesn't seem to impact anything: I can still watch my content (TV Recordings saved to my NAS) without issues, but two or three times during the day, randomly, the wifi cannot connect for minutes. It happened when I was watching a recorded video: playback stopped and even if I rebooted, I was seeing in the initial screen messages the "waiting for the network to be up". If you have patiente and wait a bit, then the reconnection will re-establish again.

    So, typically, quite many times a day I see the below (which is apparently not causing issues):

    By the way, I see in the logs of my FritzBox router that the device is deregistering itself and re-registering, sometimes with a 5Ghz band steering to 2,5Ghz despite the rpi is in the same room where the router is (at a distance of about 5 meters). Another things that I have noticed is that the maximum speed it gets at 5Ghz is 140Mbps, while other 5Ghz connected device get 866Mbps speed (my FireTV, for instance)

    Anyway, it's still good. Sometimes, though, the reconnection fails, like in the example below:

    As you see, at 07:31:18 I get a rejected connection, and then the wpa_supplicant scheduled scan doesn't work until 07:37 (6 minutes without wifi access)

    Code
    Jun 25 07:31:15 LibreELEC wpa_supplicant[410]: wlan0: Trying to associate with SSID 'TISCALI'
Jun 25 07:31:18 LibreELEC wpa_supplicant[410]: wlan0: CTRL-EVENT-ASSOC-REJECT bssid=00:00:00:00:00:00 status_code=16
Jun 25 07:32:00 LibreELEC wpa_supplicant[410]: wlan0: Failed to initiate sched scan
Jun 25 07:33:21 LibreELEC wpa_supplicant[410]: wlan0: Failed to initiate sched scan
Jun 25 07:37:24 LibreELEC wpa_supplicant[410]: wlan0: Failed to initiate sched scan
Jun 25 07:37:24 LibreELEC wpa_supplicant[410]: wlan0: Trying to associate with SSID 'TISCALI'
Jun 25 07:37:26 LibreELEC wpa_supplicant[410]: wlan0: Associated with dc:39:6f:f2:32:9d

    Then, everything comes back to normal again.

    WiFi is working fine in my house, and I have a FireTV stick connected to my TV, on the shelf below my rpi, and I have never had issues to connect to my NAS over WiFi

    I followed some hints about the regdomain which I set to IT (in Kodi), but this didn't do the trick. I also used the iw command to set it, but maybe the regdomain doesn't have any role.

    I don't exactly know how to troubleshoot it also because I am not familiar with LibreELEC, I used this OS because I thought I shouldn't have had to play with various configurations, but I have no problems with linux (I have two more rpi), so if you can give me some hints on where I should start things, that would be great.

    5 or 6 minutes of pause is a bit annoying, especially while watching TV, and I believe the same issue could happen while I am recording a TV program, which would be a disaster.

    Laying an ethernet cable across the room requires some effort and time, so I would definitely like to avoid it, unless strictly necessary.

    Thanks for your help!

    In the LibreElec addon config under network, did you set the Wireless Regulatory Domain to your country? I had issues with Wifi until I set this on my previous install. I know it doesn't make sense that this would do anything major but it does for some reason.

    Are you using the Rpi's bluetooth. I'd probably try disabling the onboard and see if that fixes the Wifi. The 3+ that I have would have issues with both WiFi and BT but this was a known issue. You can edit the /boot/config.txt file and add a line that is dtoverlay=disable-bt

    You'll have to do a search on how to edit the config file since libreelec is a bit different as you have to mount it as read/write file system and then edit the config.txt file and then mount as read only then reboot.

    I disabled mine and bought a BT dongle.

    Thank you very much for your hint! Didn't personally know it is a "known issue". Anyway, I have followed your advice and I will monitor the system to see how it goes. Normally, I get this scheduled scan issue three times a day, therefore I'll need to wait a bit and provide an update in a next post.

    [EDIT] it just happened again....

    Jun 29 12:30:15 LibreELEC wpa_supplicant[418]: wlan0: CTRL-EVENT-DISCONNECTED bssid=xxxx reason=0 locally_generated=1

    Jun 29 12:31:00 LibreELEC wpa_supplicant[418]: wlan0: Failed to initiate sched scan

    Jun 29 12:32:21 LibreELEC wpa_supplicant[418]: wlan0: Failed to initiate sched scan

    Jun 29 12:36:24 LibreELEC wpa_supplicant[418]: wlan0: Failed to initiate sched scan

    Jun 29 12:36:27 LibreELEC wpa_supplicant[418]: wlan0: CTRL-EVENT-CONNECTED - Connection to xxxx completed [id=0 id_str=]

    I was checking the firmware version of brcmfmac, as I saw someone upgraded its drivers and was succesful.

    Mine looks rather old as it is dated Feb 27, 2018

    Code
    LibreELEC:~/.config # dmesg | grep brcmfmac
[    3.823754] brcmfmac: F1 signature read @0x18000000=0x15264345
[    3.863820] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
[    3.868588] usbcore: registered new interface driver brcmfmac
[    4.106906] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
[    4.117506] brcmfmac: brcmf_c_preinit_dcmds: Firmware: BCM4345/6 wl0: Feb 27 2018 03:15:32 version 7.45.154 (r684107 CY) FWID 01-4fbe0b04
[    4.238615] brcmfmac: brcmf_cfg80211_set_power_mgmt: power save enabled
[    7.816953] brcmfmac: brcmf_cfg80211_set_power_mgmt: power save disabled

    Maybe there is a way to upgrade the firmware even if I am using LE? Do you think this could improve things?

    Quote from ciclista71

    Another things that I have noticed is that the maximum speed it gets at 5Ghz is 140Mbps

    Is your RPi overclocked? 140Mbps is impressive if that is an actual tested transfer rate with iperf3, since the on-board wireless on RPi4 is gated by the SDIO bus which is normally running at 41.6MHz clock rate w/ 4-bit bus, so at half-duplex your theoretical maximum is around 160Mbps. In my experience, I couldn't get more than 80Mbps on a 5GHz band with 80MHz bandwidth. Most people I have seen reporting actual numbers get 80Mbps, and maybe 110~120Mbps with overclocking (because I believe it bumps the SDIO bus clock rate, as it is using a frequency divider of the core clock rate).

    I guess this is one of the design tradeoffs of the RPi4 to maintain a certain cost point. I personally ended up buying a 802.11ac USB dongle and can pull 260Mbps similar to my other devices. (Which I have read is about typical for 802.11ac).

    I also presume you are looking at the link rate of 867Mbps, which isn't the actual speed in reality. I would suggest using iperf3 to test actual speeds, but it sounds like to me you are stating link rates.

    Thanks for your suggestions.

    I have no idea on how I should upgrade the driver, can you point me to any guide, also considering LE has specific commands or paths or limitations?

    As for the network speed, what I checked is the reported speed in my router, meaning the rpi4 uses 40Mhz 5Ghz bandwidth instead of 80Mhz, and connects using 802.1n instead of 802.1ac. All my other 5Ghz capable devices connect to my FritzBox router using 80Mhz and 802.1ac, but this looks impossible with the rpi4.

    Thanks, I really appreciate your help

    What version of LE are you running? It sounds pretty old to have a firmware from 2018. The reason I mention it is older versions of the firmware were not capable of connecting to 80MHz channels. There are also co-existence issues with WiFi & BT, and hardware interference issues with HDMI, as far as I know these were addressed through various firmware fixes, but most of those issues affect the 2.4GHz band from what I recall. The other issue because the SDIO bus is the bottleneck, if you are saturating that 80Mbps bus then you might see various timeouts like WPA supplicant is telling you, I think the timeout for scheduled scan is 2 seconds.

    I think, not certain, you would have to build your own image from source. From what I can tell it is part of the system image which is mounted via a loopback squash filesystem.

    I went to the LibreElec website and downloaded the version that comes with Kodi 18.9, and I did this around one month ago, I would say. It's 9.2.6

    Look, I have created a small scripts that checks every 30 seconds if there is a failure in the scheduled scan

    I have manually launched it and I have seen a strange interesting thing, see below:

    As you can notice, at 10:41:34 I get disconnected and at 10:41:37 I get a reject from my Router; right after, I see a "Failed to initiale sched scan". My program recognizes this problem at 10:42:06, and sees that there is no IP configured in the wlan0. It scans the WiFi networks and connects to mine, thus getting an IP address (the right one). I need to scan them because the connmanctl services does not show any network without a prior scan.

    Right after, wpa_supplicant connects again and everything is established.

    It happened again a couple of hours after:

    In this case, still there is a sheduled scan failure after I reconnect, but three seconds after I get it reconnected.

    So, what I have understood is that:

    - if I manually reconnect, it seems to work (stull under test)

    - the scheduled scan is very slow and does not seem to work properly, and there is no way to increase its pace

    - scan always happens after 45s from disconnection, then 1m10s after last attempt and 4m03s after the last attempt and then generally it reconnects spontaneously

    I would like to find a way to manually reconnect in a faster way but I need to think how. And of course, this is just a workaround and I need to test it more deeply to see it it really works; I would rather love to fix this issue, or testing the firmware upgrade.

    [EDIT] I have found some new firmware for the brcm, mainly two:

    One is called brcmfmac43455-sdio.bin and one is called brcmfmac43455-sdio.clm_blob. There is also a text file called brcmfmac43455-sdio.txt.

    I checked the content of the first .bin file and I see the same version you have, but I am not sure whether I should also use the other two files or not.

    If by change I have issues with this firmware, is there a way to roll-back?

    Edited once, last by ciclista71 (June 30, 2021 at 2:53 PM).

    • Official Post

    I'll add $0.02 to the conversation: If you want any form of response or interest from devs you need to test/use LE10 images. It's not that we don't care about LE 9.2.x, we do, but it's a codebase that is no longer in development so anything you find is not going to be fixed, and LE10 has significant-enough differences in software component versions to make comparisons low-value.

    I understand your point, but unfortunately I read that the new Kodi under LE10 only works with Python3 and many plugins do not work yet. Now, since I do not know what would work and what not, I decided to go for Kodi 18.9. On top, I read that LE10 has some limited support for Rpi4.

    This is the reason why I am using that version

    Edited once, last by ciclista71 (June 30, 2021 at 5:44 PM).

    I would just buy a USB dongle, and roll your own image, or find one that is already supported by LE 9.2.6. The on-board Wi-Fi is frankly useless if your using this for playing back FHD or higher media, and to make it work you will need to create a nice fat cache to smooth you through the high bit-rate scenes.

    I was just doing regular 1080p playback over the on-board Wi-Fi with Pi4, and on high bit-rate scenes kworker queues would consume 30% of CPU, and the SDIO bus was totally saturated, and the result was hangs, freezing, NFS read errors, etc. (Never wireless drop outs). I ended up doing 3 things, 1) buying a TP-Link T3U USB dongle, 2) building my own images with Realtek RTL88X2BU drivers, 3) bumping the cache:

    Code
    <advancedsettings version="1.0">
    <cache>
        <buffermode>1</buffermode>
        <memorysize>524288000</memorysize>
    </cache>
</advancedsettings>

    The cache is probably a bit excessive at 512M, but it buffers 2~3 minutes of FHD video and smooths out high bit-rate scenes. This I did before #1 & #2 and resolved my issues with stuttering and NFS read errors. I ended up doing #1 & #2 after, because eventually I might run UHD videos, and that will give me ~260Mbps transfer rates, which should be plenty hopefully (YouTube is suggesting 85Mbps for 60fps 2160p HDR).

    ciclista71 This post may help you with updating the firmware:

    Post

    RE: How to change files on Read-only file system

    […]

    I had thought about that as well. But it was puzzling me why adding firmware via .config/firmware shouldn't work so I gave it a try and - guess what - it worked as expected.

    I created an RPi2 image with the RPi wifi firmwares removed and added them in .config/firmware/brcm/:

    (Code, 12 lines)

    Then booted on a RPi3B (which needs the brcmfmac43430-sdio bin and txt files) with dyndbg="module firmware_class +p" on the kernel command line to trace the kernel firmware loader.

    The kernel first checks…
    HiassofT
    November 30, 2018 at 10:02 AM

    Basically what I read is you create a folder "/storage/.config/firmware/brcm/" and plop the 3 files you mentioned earlier in there. Reboot, check dmesg if the version reported is updated. If it doesn't work or creates more problems then just remove the files from that location.

    Only catch is if the firmware is paired with any other components (like the kernel for example). I would have no idea, and hope that they make it backwards compatible.

    Yesterday evening, at 21:10 CET, a Scheduled Recording started on my TvHeadEnd server and at 22:39 I decided to watch a Recorded Video (a different one, though this is not important). At 22:43 the Wi-Fi stopped working.

    The sequence is the following:

    I do not understand how I can lose connectivity on my loopback, though.

    I was pretty sure that my recording would fail as well, and instead, surprisingly, I haven't lost any packets at all, even while the wifi was out. If you look at my grafana, you'll notice TVH or Kodi cached the data and then sent them all as soon as the WiFi was re-established (I did tweak the advancedsettings.xml). Actually I do not have any issues in playing HD Video.

    My settings are the following, and I am using buffermore=4 to cache NFS as well. Odd enough, I am using exactly the same memory size you are using!

    Code
    <advancedsettings>
        <cache>
                <buffermode>4</buffermode>
                <memorysize>524288000</memorysize>
                <readfactor>5</readfactor>
                <curlclienttimeout>10</curlclienttimeout>
                <curllowspeedtime>10</curllowspeedtime>
        </cache>

    I think the next steps will be the following:

    1) improve the script to react more quickly (though I am not sure I will solve the TVH Client issue) and play with TVH settings to see if there is a way to speed up network recovery

    2) Update the firmware (now that I know I need to add all the three files) and see what happens

    3) If 1) and 2) are not working, I will try installing Buster Lite, add TVH and use Kodi on my FireTV as a player, to see if WiFi drops still occur

    4) If I still have issues, I will have to take the drill and start cabling my house

    I think trying to make it "reconnect faster" is a poor solution to your problem, the fact that you made this script is generally a bad idea. You should resolve the cause of the disconnects.

    I think from the earlier posts you have heard stories of various issues that cause WiFi drop outs, and in fact if you check the Raspberry Pi forums you will find many more, but to recap:

    - HDMI causes interference in the 2.4GHz band (known to affect WiFi channel 1, maybe BT?), firmware fix AFAIK. Never an issue for me, but I wasn't an early adopter.

    - USB3 causes interference with on-board Wi-Fi, no idea if this is still a thing or not but the RPi4 has 3 firmwares to my knowledge (Bootloader, USB, and Wifi/BT).

    - BT itself has interference issues with WiFi on the same chip, think there was firmware fixes & parameter changes to address it.

    - People have reported USB hubs or other devices being too close as the cause interference with WiFi.

    - Perhaps your tuner HAT is causing the issue, no idea, don't have one.

    That tells me you want to get on the latest software available for the platform with all of the firmware patches possible. You could have kernel related issues, or SDIO bus saturation which may manifest as problems as well [hard to know unless you can quantify your wireless performance at the time of the drop outs -- i.e. how much data is being sent/received]. So I think either running LE 10 or testing out a recent update of Raspberry Pi OS may help. It might also be worthwhile to poke in on the Raspberry Pi Forums and see if the engineers respond with any insight on your issue, a few of the Raspberry Pi engineers occasionally check in on issues and will reply.


    As I mentioned earlier, I am running a similar setup, RPi4, Tvheadend client, Kodi 19.1 (LE 10b5), and have zero issues. Even before I bought a $15 USB dongle, it managed fine with a larger cache. No wireless drop outs, ever.

    As a side note, your incorrect about the buffermode. 4 is the default, which means remote only, you want 1:

    #define CACHE_BUFFER_MODE_INTERNET 0

    #define CACHE_BUFFER_MODE_ALL 1

    #define CACHE_BUFFER_MODE_TRUE_INTERNET 2

    #define CACHE_BUFFER_MODE_NONE 3

    #define CACHE_BUFFER_MODE_REMOTE 4

    I don't think remote includes "sources", so this is something you might want to check is working right. There is a visual cue on the playback progress bar of the caching.

    Anyways, good luck.

    Edited once, last by frakkin64 (July 1, 2021 at 9:07 PM).

    Do you have a spare MicroSD? You could write one of the LE 10 test builds to it and see if you are still seeing the same dropout issues. If it doesn't work then you have your original microSD to pop back in. This is what I did and kept my Raspbian OS with Kodi 18.9 just in case something went south I would still have a system.