LibreELEC boot freezes on Wetek Hub for 7m30s

MaTTiQ · March 20, 2020 at 2:57 PM

Hey,

I'm struggling with an issue during booting LibreELEC on Wetek Hub. I was using latest LibreELEC 9.0.2 for a long time and recently I noticed issues where I'm getting "A start job is running for Load Kernel Modules", it starts from 0s / 1m30s and next it increases until 7m30s where after reaching 7m30s it goes further and Kodi is successfully opened and working except remote control. However to confirm if remote controller or IR module are broken I opened WeOS which I have in internal memory and remote control works completely fine. I installed Kore on my phone to use remote control in LibreELEC and moving in menu works fine as well as other functionalities including playback. I've done a bit of troubleshooting before I decided to write here:

1. I reflashed LibreELEC (tried 9.0.2 and 8.2.5) and it didn't help. It goes smoothly through first run with partition resizing, next there is automatic restart after 15 secs and again stuck for 7m30s with same message with load kernel modules.

2. I tried using different MicroSD cards, originally I was using Samsung Evo Plus 32 GB but I also tried Goodram 4 GB and other 2 GB card. For all of these 3 cards result is the same. Additionally to exclude this reason 100% I used different USB ports and MicroSD card readers.

3. I tried flashing using LibreELEC Tool and eventually Rufus, same result for both, unsuccessful.

4. I tried to run Wetek without either Ethernet cable connected with HDMI only or other way without HDMI connected but with Ethernet cable to exclude issues with these 2 interfaces but for both scenarios it didn't help and it still starts long time.

5. I checked power supply using multimeter, shows ca. 5.12 V so it's okay, power supply is 5V/2A.

Log generated after start after 7m30s on fresh install: [ 0.000000@0] Initializing cgroup subsys cpuset [ 0.000000@0] Initializin - Pastebin.com

I can see there is a pause in logs for a while and then it starts with 'sdio: error -22 whilst initialising SDIO card'. I tried to find something in Google but I couldn't found anything relevant.

To sum up I can't see any issue with MicroSD interface, HDMI or Ethernet. I only use these interfaces, I don't use USB or don't have WiFi antenna connected. Btw I'm using Wetek Play 2 on second TV and it works fine without problems. I'd like to have this problem fixed because waiting 7m30s and then using mobile remote controller is not convenient to me. I'd be grateful if you can take a look into logs I provided and help me with this issue.

Thank you!

**chewitt** · March 21, 2020 at 3:36 AM Official Post

I forgot how full of irrelevant verbose garbage 3.14 kernel logs are

For comparison purposes put LibreELEC-AMLGX.arm-9.80-devel-20200314152345-6c08564-box.img.gz on an SD card, boot and install and then share the logs. You'll need to set dtb=/dtb/meson-gxbb-khadas-play2.dtb in uEnv.ini.

Let's see if we get similar SDIO errors.

MaTTiQ · March 21, 2020 at 4:46 PM

Hi chewitt, I've just flashed my regularly used 32 GB Samsung card with version you mentioned and the only difference I made was I set dtb to 'meson-gxbb-wetek-hub.dtb' as I couldn't see in dtb folder dtb file you mentioned, additionally you mentioned dtb related to Wetek Play 2 but box with which I have a problem is Hub so I assumed this will be more proper.

Anyway I managed to run this version immediately without issues. Attaching logs I got: http://ix.io/2eRU

Still here I can see entry:

Code

[   53.474710] mmc2: error -22 whilst initialising SDIO card

Before that today I turned on my working Play 2 where I exported logs and next I did side-by-side diff of Hub and Play 2 logs to spot major differences as both were using 9.0.2. I could notice similar to above problem occuring on Hub which was not present on Play 2 and it was repeated a few times:

Code

[   75.953743@2] sdio: queuing unknown CIS tuple 0x80 (18 bytes)
[   75.991538@2] sdio: queuing unknown CIS tuple 0x19 (70 bytes)
[   76.119198@2] sdio: queuing unknown CIS tuple 0x7d (245 bytes)
[  140.955858@2] sdio: error -22 whilst initialising SDIO card
[  140.956882@0] [aml_sd_emmc_irq] sdio: resp_timeout,vstat:0xa1ff2800,virqc:3fff
[  140.956897@2] [aml_host_bus_fsm_show] sdio: err: wait for irq service, bus_fsm:0x8
[  140.956920@2] aml_sd_emmc_data_thread 2642 sdio: cmd:55
[  140.959058@0] [aml_sd_emmc_irq] sdio: resp_timeout,vstat:0xa1ff2800,virqc:3fff
[  140.959070@2] [aml_host_bus_fsm_show] sdio: err: wait for irq service, bus_fsm:0x8
[  140.959099@2] aml_sd_emmc_data_thread 2642 sdio: cmd:55
[  140.960723@0] [aml_sd_emmc_irq] sdio: resp_timeout,vstat:0xa1ff2800,virqc:3fff
[  140.960734@2] [aml_host_bus_fsm_show] sdio: err: wait for irq service, bus_fsm:0x8
[  140.960758@2] aml_sd_emmc_data_thread 2642 sdio: cmd:55
[  140.961792@0] [aml_sd_emmc_irq] sdio: resp_timeout,vstat:0xa1ff2800,virqc:3fff
[  140.961802@2] [aml_host_bus_fsm_show] sdio: err: wait for irq service, bus_fsm:0x8
[  140.961823@2] aml_sd_emmc_data_thread 2642 sdio: cmd:55
[  140.966537@0] [aml_sd_emmc_irq] sdio: resp_timeout,vstat:0xa1ff2800,virqc:3fff
[  140.966546@2] [aml_host_bus_fsm_show] sdio: err: wait for irq service, bus_fsm:0x8
[  140.966567@2] aml_sd_emmc_data_thread 2642 sdio: cmd:1
[  140.966590@2] sdio: error -110 whilst initialising MMC card
[  140.967692@0] [sdio_reinit] finish
[  142.959847@0] failed to power up DHD generic adapter, 2 retry left
[  142.980496@0] wifi_platform_set_power = 0
[  142.980559@0] ======== PULL WL_REG_ON LOW! ========
[  142.980606@0] wifi_platform_bus_enumerate device present 0
[  142.980647@0] ======== Card detection to remove SDIO card! ========

Display More

In general startup of your version looks really promising and it was faster than on previous version. Still I'm not sure if I should be worried about issues I have. I'm looking for your comment regarding to log file from 9.80 version.

Btw is it quite stable to use generic amlogic dev version on Wetek Hub/Play 2? I'm thinking about updating my working Play 2 where I would select dtb for Play 2.

Thank you!

**chewitt** · March 22, 2020 at 5:43 AM Official Post

If you're seeing the same issue on two different OS images it looks like the device has a failed SDIO wifi module; although the BT section (which is on the same module) looks working which is unusual. I guess if you're only using Ethernet it's not a big deal.

Boot times on mainline are way lower and I find the entire OS to be lighter and more focussed. The only thing that's still "work in progress" for an S905 box is hardware video playback which still needs some rework after recent kernel-side changes. If it works for you, you're welcome to use the nightly images. If not, stick to 3.14 for a while longer. WP2 will be broadly similar, the only exception is that 4K might work as it has 2GB RAM so the currently insane CMA memory allocation needed for that is possible vs. the Hub which is 1GB and needs to wait for some memory use optimisations.

MaTTiQ · March 22, 2020 at 1:20 PM

No, I can't see any issues on Play 2. Play 2 works completely fine. I only used Play 2 logs as a comparison to Hub to find difference in logs.

You're right in terms of problems with WiFi module. It looks like it doesn't work. I attached antenna and checked LE as well as WeOS 3.2 in NAND but networks are not found, even I click on WiFi in WeOS it stucks and I'm getting popup 'App Settings seems to not responding'. I don't know when it could stop to work as I'm not using WiFi at all, I prefer wired connection. Although Bluetooth works correctly. I assume BT may be separate module to WiFi.

In this case is it possible to exclude WiFi module load at all? I'm not an expert but I'm thinking about compiling device tree file (dts/dtsi) for Hub which would have excluded information about WiFi module as I don't need it really, ethernet connection is the most important to me.

Second thing I wanted to ask is about latest available LibreELEC version with Kodi Leia which I could install on Wetek Hub. LE 9.0.2 contains Kodi 18.2, I was thinking about version with Kodi 18.6, even developmental. I couldn't find it on website or Yadi.sk.

I haven't checked Kodi 19 in terms of video playback, I can check it in next days.

**chewitt** · March 22, 2020 at 6:10 PM Official Post

linux-amlogic/gxbb_p200_1G_wetek_hub.dts at amlogic-3.14.y · LibreELEC/linux-amlogic · GitHub

^ dropping that from the device tree should/might do the trick. LE moved all development away from the legacy kernels to mainline so you won't find anything from us. Hopefully everyone being locked up at home for a while stimulates the ffmpeg work to be finished off..

MaTTiQ · March 23, 2020 at 11:26 PM

Today I managed to resolve my problems, by 'resolving' I mean disabling WiFi with having Wetek booted fast (as it was before my problem appeared). At first I tried to amending device tree file (.dts), it helped but only partially. I was able to decrease boot time from 7m30s to around 1 minute but still with Load Kernel Modules prompt at boot. I checked using SSH status of systemd-modules-load.service what gave me information about failed modules load. In log there was an important information to me:

Code

Jan 01 00:01:03 LibreELEC systemd-modules-load[1875]: Failed to insert 'dhd': No such device

I figured out 'dhd' is responsible for WiFi module so I blacklisted this module from load in /etc/modprobe.conf. After all I did reboot and LibreELEC started to boot like a charm. After boot I checked in console using lsmod command if 'dhd' is still listed but it wasn't as expected, systemd-modules-load.service status shown entry with 'dhd' as skipped and whole Load Kernel Module process now has SUCCESS status, not FAILURE as before. I also did scenario where I only disabled 'dhd' module without amending device tree but it hasn't allowed me to get system booted as normal, still I needed to wait around 1 minute to start. So both actions were needed to have it running smoothly.

How to fix the issue:

1. Connect MicroSD card to computer and grab dtb.img file from card.

2. Decompile device tree blob (.dtb) on Linux using 'device-tree-compiler' to device tree source (.dts).

3. Start editing decompiled .dts file. Find 'sdio' section in file and amend 'status' tag value from 'okay' to 'disabled'. Example below:

Code

sdio {
    compatible = "amlogic, aml_sd_emmc";
    dev_name = "aml_newsd.0";
    status = "disabled"; // Change value here from okay to disabled
    reg = <0x0 0xd0070000 0x0 0x2000>;
    interrupts = <0x0 0xd8 0x1>;
    pinctrl-names = "sdio_clk_cmd_pins", "sdio_all_pins";

4. Save changes and compile source file to blob version (.dtb). Make sure you made backup of your original .dtb file. Save compiled version as dtb.img and overwrite it on MicroSD card. Place card back into MicroSD socket in Wetek. Turn on the device.

5. After successful bootup connect to console using SSH protocol and execute following command to make sure if there are failing modules:

Code

systemctl status systemd-modules-load.service

6. 'dhd' module should be found as a problematic one. Connect to Wetek using SFTP and go to /etc/modprobe.conf.

7. Create new file disable-wifi-module.conf with content:

Code

# Disables WiFi module (dhd) which blocks smooth start up of system

blacklist dhd

8. Save the file, make sure it has CHMOD 644 and reboot Wetek from SSH or just restart it using remote.

9. Now Wetek should start smooth without any delays related to broken WiFi module.

It took me a bit of effort to get this working but I'm happy about the result. In the meantime I even started to look at potential TV boxes to buy. I have seen Amlogic S922X was released some time ago and recently MINIX U22-XJ has been released. Do you think if S922X will be future-proof and worth buying or is it better to get RPi4? In general I'm happy from this Wetek so after resolving issue I had I won't change it in the nearest time but it's good to have knowledge because sometimes there maybe quick sale so then I might upgrade my S905. I have running RPi4 but I'm not able to try running LE because it has own purpose. I'm looking for your reply and I'd like to thank you very much for your help.

**chewitt** · March 24, 2020 at 1:41 AM Official Post

RPi4 is probably going to be the better supported device in the long-term, there will be millions more of tthem than S922X boxes, but I would always advocate seeing what's around in the future when you reach the point of wanting/needing to upgrade because it's a fast-moving world. In the next year S908X will probably take over from S922X as the high-end chip in boxes, although Linux support will lag a little behind if there's major changes to the SoC architecture (likely not, but until we see them it's unknown).

Glad you got it sorted out.