[UNOFFICIAL][RK3228/RK3229][box]Libreelec builds

  • oneillb, jock2 it's really strange. This kind of errors I catch them with the first versions (buggy mesa driver).

    Anyway try this build

    testing – Google Drive

    I'm working on the next version where

    - devices with emmc don't try to load the nand (and vice versa)

    - since libreelec is a multimedia system the gpu is fixed at the maximum clock rate (500Mhz)

    - updated kodi with some bugs fixed

    - updated mesa (gpu drivers)

    just try:

    - add more debugging info by appending

    Code
    debugging cma=256M

    on your kernel command line

    - launch it from sd card instead of internal memory

    Hope it helps

    Thank you very much

  • ilmich

    I've run up the latest build (LibreELEC-RK322x.arm-10.0-nightly-20220426-a0fa878-rk322x.img.gz) , from SD card, and the linux OS seems to be more resilient...

    I've turned on the debugging and set cma=256, as you suggested. I've turned off the ssv6051, and have no other usb devices attached, turned off samba, avahi etc. resolution set to 720p

    The kodi process again is crashing after a few minutes of streaming some 1080p .ts files over nfs, but they do not bring the OS down.

    When kodi.bin crashes, the process hangs around, and remains visible in ps.. Another kodi.bin starts, but can't initialise the display, which remains as it was when the 1st instance crashed... .

    kill and kill -9 are ineffective in killing the older kodi.bin processes, and issuing a reboot via ssh, does not reboot the box.. it brings down the network etc., but there is no reboot..

    Only cycling the power will result in a reboot, and the display resetting.

    I've attached some logs ...

    logs_20220426.zip

  • kill and kill -9 are ineffective in killing the older kodi.bin processes, and issuing a reboot via ssh, does not reboot the box.. it brings down the network etc., but there is no reboot..

    Try this command from SSH console to force reboot immediately (do it only if there's no other option than power cycle):

    Code
       echo b > /proc/sysrq-trigger
  • v88 4k when I update (/update) it tells me a black screen and it does not connect to the network, but if I install it directly from the sd Kodi it closes when I try to complete the steps it restarts and the third time it hangs

  • markosc please try to be more precise (like oneillb).

    for example what build are you using?!?! which device tree?! (if you read in the first post, by default there is a simple one but you can change it later). can you post some logs! ??!

    in any case update from libreelec 9.2 to 10 will never works, because they are completely different (legacy vs mainline)

    oneillb as soon as I can I try to take a look at your logs. Surely since I focus more on hardware than software, try to play the same file locally (no nfs).

    This is because if you have no problems, then you have to investigate kodi (which is not my field, however)

    Thank you very much

  • Hi ilmich,

    I think I may have made a breakthrough....

    I'm still using your last testing build (LibreELEC-RK322x.arm-10.0-nightly-20220426-a0fa878-rk322x.img.gz) from SD card, with the following disabled:

    ssv6051

    spdif

    cec

    samba

    avahi


    I bought a CP2014 USB -> Uart adapter, and I set it up to monitor the serial console output on the v329Q v3.0 board.. That instantly discovered my box has DDR3 330Mhz ram.. not DDR2 as I had believed.

    Code
    DDR Version V1.10 20190926
    In
    ID:0xFFF
    660MHz
    DDR3
    Bus Width=16 Col=11 Bank=8 Row=15 CS=1 Die Bus-Width=16 Size=1024MB
    mach:2
    OUT
    Boot1 Release Time: May 13 2019 17:02:59, version: 2.56
    ChipType = 0xc, 269


    I could see the u-boot and early kernel messages on the LE9.2 build, but the 10x build showed complete garbage... for some reason the BAUD rate changes from 115200 to 1500000 when u-boot kicks in... I think this is patched in by your 10.x sources here...

    projects/Rockchip/devices/RK322x/packages/tools/u-boot/patches/rockchip/u-boot-1000-rk322x-defconfig-dts.patch


    I changed my boot line to be...

    boot=UUID=2604-2146 disk=UUID=b528aa0c-4dc9-4853-bf9b-45a8c3fa13db console=uart8250,mmio32,0x11030000,115200n8 ignore_loglevel cma=256M debugging


    And now I get all the kernel debug out to the serial console... and can see the output when there is a panic.

    With DDR3, I went back to the dmc settings. The LE9.2 v88mars build for me was rock-solid. So I transplanted some of those CPU clock and voltage settings from that build into a new dtb for my LE10.x configuration.. Which slightly upped the CPU voltages for the higher clock frequencies, but this wasn't actually that effective. I then upped in a similar fashion the voltages at the lower frequencies, as this is actually where the box spends most of the time running, and things are getting much much better.

    This is the DDR3 and CPU frequency / voltages I'm now using..

    rk3229-box-sm1.zip


    However,


    I'm still getting the odd issue. I had a lockup occur after about 40minutes of playing locally stored .ts files... The LED is still responsive, but nothing else. Here is the console output ..

    librelec_10.x_rk332x_lockup_202205021651.txt

    Does this suggest I may need to up the voltages a little more? Or is it a different issue? Any help appreciated...

    Thanks

    Barry

  • oneillb

    Code
    [    3.520762] rk3228-dmc 11200000.dmc: memory type DDR3, timings (tCL, tRCD, tRP, tRAS): CL10-10-10-25 command rate: 1T (mcfg register: 0x50021)
    [    3.535269] rk3228-dmc 11200000.dmc: detected DDR3 memory

    The 1T Command Rate may be the reason you get your issue.

    DDR memories have this Command Rate parameter which specifies how many clock cycle are required to send a command to the DRAM banks. It can be 1 clock (1T) or 2 clocks (2T). 1T is faster but less compatible, 2T is slower but more compatible.

    Most boards works fine with 2T, but 1T may let the system become unstable.

    On the contrary, a very small set of boards (like my r329q v3.0 and some other boards out there) works fine with 1T Command Rate, but don't boot with 2T Command Rate. This is way disappointing because the board freezes during initial boot with no other chances to interact with it.

    Long story short: if you install a bad loader on NAND/eMMC on a small set of boards (like my r329q), you may brick it and you have to force maskrom mode (the serial console will print the infamous "Err!" message).

    If you used my 660MHz loader that I posted on armbian forum, I may have set it CR incorrectly to 1T. I shall remove that loader which is not exactly stable for all boards.

    Now, since the dram memory controller driver is available for mainline kernel, using that 660MHz loader is less than ideal for many reasons. I may suggest to you to erase the NAND using rkdeveloptool and let the LibreELEC image boot with its own loader (which uses 2T Command Rate, as like as LE9.2 builds) and see if your board is stable that way.

    If it is stable, I can give you a properly compatible loader for your board so you could also install and boot libreelec from internal NAND.

    Edited 2 times, last by jock2 (May 3, 2022 at 9:38 AM).

  • jock2 - the bootloader on the NAND of my r329Q v3 board is rk322x_loader_v1.10.256.bin which I flashed using rkdevelop over the USB OTG port. (I think you pointed me in that direction when booting LE from nand was not working. )

    The remainder of the NAND is populated with ilmich's latest LE9.2 image (LibreELEC-RK322x.arm-9.2-devel-20220124165557-3ce5dd4-rk3229-v88mars.img.gz) which was flashed using multitool.

    The board will boot from SD (which I'm using to test out the LE10 builds) and fall back to NAND without issue when the SD card is removed... My problem is with stability once things are up and running... If the bootloader puts the board into 1T mode at startup, is that mode maintained through u-boot and kernel startup, or can the kernel driver switch the mode later on?


  • The board will boot from SD (which I'm using to test out the LE10 builds) and fall back to NAND without issue when the SD card is removed... My problem is with stability once things are up and running... If the bootloader puts the board into 1T mode at startup, is that mode maintained through u-boot and kernel startup, or can the kernel driver switch the mode later on?

    Uhm, this makes my hypothesis invalid as the answer to your question is no: the kernel driver can't swtich from 1T to 2T.

    Once your box boots in 1T mode it can't switch to 2T mode.

  • jock2 , ilmich,

    I've attached some logs from last weekend when I spent some time testing the LE10 build (LibreELEC-RK322x.arm-10.0-nightly-20220426-a0fa878-rk322x.img.gz) from SD card, with the following disabled:

    ssv6051

    spdif

    cec

    samba

    avahi

    librelex_10.x_logs.zip

    Some runs are with dmc enabled , others with dmc disabled. The earlier logs are with the default , and later ones with slightly increased (by about 0.05v) cpu voltages.

    Many crashes happened within 10 minutes, but some of the latter ones ran for > 40 minutes. In many cases the crash happened with Kodi idle, in others whilst running playback of local .ts files

    There doesn't seem to be any consistent pattern. Have a look and see if there is anything you can spot..

    Also, is there anything extra I can log, which may help? Is it worth logging the latest LE9.2 build running for example, which runs really well on this board?

    Thanks,

    Barry

  • oneillb

    I own many 329q ( all ddr3 ) and some of those boxes are the basis of all investigations made in the last years since starting studies up to actual days. jock2 himself and me have studied on one of these ! you can check here the time date and from here all has born (december 2019 !!!!! )

    https://forum.armbian.com/topic/12401-lo…k3229-rockchip/
    ilmich owns a board IDENTICAL to yours and he is not suffering any of those problems you show here.

    Don't forget that those cheap tvbox are made with the last rubbish components found in china offices or plants so I wouldn't be surprised thinking about faulty components ( ram ? defective capacitors on power switching thus conseguentially vcc ripple,? bad solder joins that under heat/stress go faulty ? and the list could go growing).

    The fact that under certains conditions is working and under stress or/and idle moments the board crashes are the most evil problems to investigate, but I must say you many thanks because with your feedbacks and your logs you helping ourselves to better investigating :thumbup: :thumbup:

  • oneillb did you try to keep only a single CPU operating point in the device tree?

    On armbian I had many issues and crashes on a board of mine because the transition between the slowest (600 mhz) and highest (1.4 ghz) operating points was "too fast"; actually the legacy 4.4 kernel worked plenty well, but the mainline kernels crashed a lot in a matter of minutes or seconds.

    That what pretty strange on my case, but I solved by either using a single cpu frequency (thus not allowing the frequency switching), or raising enough the voltage of the slowest operating points to reduce the voltage gap between slowest and highest operating points.

    That board now runs pretty well with uptimes of several weeks! (this is the device tree overlay for the modifications I used in my case)

  • Is there a RAM testing tool available? Maybe this should be the first thing to run.

    On armbian you can install and use memtester. I guess the binary would work on libreelec too, but I don't know if there is the need for some dependencies.

  • jock2  fabiobassa  ilmich @daflex

    Thanks for all the pointers... On the back of that I've done googling etc...

    jock2 - The pointer to regulator-ramp-delay was very useful, you recommended a change from 12500 to 300, which is a massive change, given the parameter is a gradient.... I think you may have mixed up 2 different parameters regulator-ramp-delay and regulator-enable-ramp-delay.

    Looking at some other rockchip and arm 32bit builds and documentation, regulator-enable-ramp-delay is sometimes set to 200 - 600 µs, which is a fixed delay on each transition and regulator-ramp-delay is the gradient of the ramp, which determines a variable time delay, based on the voltage change. This seems to be usually set somewhere in the 6000 - 12500 µV/µs range.

    As a 1st pass, I've restored the original voltage settings from the v88mars.dtb bundled with the build, and modified / set the following on both the vdd_arm and vdd_log regulator definitions:

         regulator-enable-ramp-delay = 300

    regulator-ramp-delay = 6000

    memtester 4.5.0 is part of the base Librelec 10.x image, so I'm now running that now over 650MB 5x passes, which was the freemem reported by top, which should stress tings a little. Not sure how long that will take, but it's been running for about 60 minutes already, and is still on pass 1.

    Admittedly memtester is not complex processing, and the CPU cores are probably all completely maxed out, so not ramping up and down, but 60 minutes of uptime is already an improvement of sorts.


    I'll let you know how it works out.

    Thanks again:thumbup::thumbup::thumbup:,

    Barry

    Edited once, last by oneillb (May 5, 2022 at 6:48 PM).

  • OK - The memtester run from yesterday completed successfully.... eventually.

    I then did some calcs in a spreadsheet. The total delays using the default kernel values seem incredibly short.... unless regulator_enable_ramp_delay defaults to a non-zero value if undefined, but the documents don't suggest there is a default


    Defaulted Kernel dtb values

    dV dμV regulator_enable_ramp_delay (μs) regulator_ramp_delay (μV/μs) delay (μs)
    0.025 25000 0 12500 2
    0.05 50000 0 12500 4
    0.075 75000 0 12500 6
    0.1 100000 0 12500 8



    jock2 recommended the following, considerably longer delays generally.

    dV dμV regulator_enable_ramp_delay (μs) regulator_ramp_delay (μV/μs) delay (μs)
    0.025 25000 0 300 83.33333
    0.05 50000 0 300 166.6667
    0.075 75000 0 300 250
    0.1 100000 0 300 333.3333

    my initial "guess" values actually result in an overall delay in the same ballpark as jock2, and seem to work fairly well also....

    dV dμV regulator_enable_ramp_delay (μs) regulator_ramp_delay (μV/μs) delay (μs)
    0.025 25000 300 6000 304.1667
    0.05 50000 300 6000 308.3333
    0.075 75000 300 6000 312.5
    0.1 100000 300 6000 316.6667


    I then tried to find where the sweet-spot might be, working from low overall delay, and increasining to get more uptime... I've had my board running for > 10 hours with the following settings.... which is a massive improvement...

    dV dμV regulator_enable_ramp_delay (μs) regulator_ramp_delay (μV/μs) delay (μs)
    0.025 25000 100 5000 105
    0.05 50000 100 5000 110
    0.075 75000 100 5000 115
    0.1 100000 100 5000 120

    However.... Although the Librelec OS has been up > 10 hours, Kodi.bin partially crashed after 70minutes....

    The IR LED was still responsive, and I could ssh onto the box and actually shutdown kodi using systemctrl stop kodi, and then I ran another memtester over 825MBtest, which is nearly finished now...

    There was no kodi crashlog, as kodi was still running to some extent, but there were dmesg errors for the time the clock had stopped on the UI (09:15) . With 3 messages like this...

    Thread JobWorker 2835341952 terminating (autodelete)

    Thread JobWorker 2817503872 terminating (autodelete)

    Thread JobWorker 2809111168 terminating (autodelete)


    There was a corresponding kernel error...

    [ 4065.211853] Hardware name: Generic DT based system

    [ 4065.217217] PC is at 0x0 [ 4065.224612] LR is at xas_load+0xc/0x70

    [ 4065.228812] pc : [<00000000>] lr : [<c068b8e0>] psr: 680e0013

    [ 4065.235829] sp : c3c0fd18 ip : ffffc005 fp : ffffffff

    [ 4065.241675] r10: 00000008 r9 : 00000003 r8 : 00000402

    [ 4065.247527] r7 : 00000406 r6 : c3c0fd54 r5 : 00000003 r4 : 00000000

    [ 4065.254832] r3 : 97ec674a r2 : 97ec674a r1 : ffffffff r0 : 00000000

    [ 4065.262142] Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none

    [ 4065.270135] Control: 10c5387d Table: 637c406a DAC: 00000051

    [ 4065.276572] Process kodi.bin (pid: 611, stack limit = 0x60538f89)

    [ 4065.283395] Stack: (0xc3c0fd18 to 0xc3c10000)

    [ 4065.288275] fd00: 00000000 97ec674a

    [ 4065.297433] fd20: 00000000 c1f58db0 c3c0e000 c54d76c8 c482dec0 00000000 c4509b4c 0000003f

    [ 4065.306595] fd40: c37d9a00 c0758670 00000000 00000cc0 c4509a50 00000000 00000000 c54d7680

    [ 4065.315753] fd60: c3c0fdf4 97ec674a c54d7680 c54d7680 c3c0fdf4 00000000 c482dec0 c079d54c

    [ 4065.324908] fd80: c3d79800 c3c0e000 c1505434 00000000 c4509a50 c3c0e000 c3d79800 00000000

    [ 4065.334065] fda0: 00000000 c1d35100 0000e4c4 0000004a 00010000 97ec674a 00000000 c3c0fe74

    [ 4065.343223] fdc0: c2430468 c4509a50 c3c0e000 c4509800 c4833900 c54d7680 c4833a00 c079a238

    [ 4065.352380] fde0: 00ba3d08 c482d840 c54d76f0 00000110 c496ce40 c4833a00 00000001 00000000

    [ 4065.361545] fe00: c4509800 c4509a50 0000004a 00000000 00000000 00000004 c54d7680 97ec674a

    [ 4065.370704] fe20: 00000000 40306443 00000000 c0d88170 00000030 c3c0e000 c3c0fe74 00000043

    [ 4065.379863] fe40: c4833900 c075a260 0000e280 00000001 c105b378 c0170e34 bee45728 c3c0fe74

    [ 4065.389027] fe60: 00000030 c448a000 c0799fe8 00000051 00000000 00000000 00000001 0000004a

    [ 4065.398185] fe80: 00000110 01b518a8 00000000 bee45818 00000000 00000000 00000004 00000000

    [ 4065.407342] fea0: 00000000 c157a300 c157a300 680e0113 00000000 c01aab88 ef6cf8c8 ef6cf8f0

    [ 4065.416503] fec0: ef6cf918 c01c6110 c1504fd0 002b4fe3 00000000 ef6ca380 ffffe000 c1d35100

    [ 4065.425663] fee0: 00000009 c3c0e000 c16491a0 ef6ca380 c3c0e000 97ec674a 00000003 40306443

    [ 4065.434821] ff00: c448a001 40306443 c3c0e000 bee45728 c448a000 0000001b c24103c8 c0312c80

    [ 4065.443984] ff20: 0000000a c148a2dc c1495c40 c1495c40 00113c57 c1503d00 00400100 c9284000

    [ 4065.453147] ff40: c3c0ff90 c3c0e000 ffffe000 c3c0ff68 00000001 c9284000 c3c0ff90 c3c0ffb0

    [ 4065.462305] ff60: bee45748 bee45748 c1494f44 c1494f44 00000000 c0192688 c1505714 97ec674a

    [ 4065.471466] ff80: f0802000 01b37b10 bee45728 40306443 00000036 c01002c4 c3c0e000 00000036

    [ 4065.480625] ffa0: 00000000 c0100060 01b37b10 bee45728 0000001b 40306443 bee45728 00000000

    [ 4065.489784] ffc0: 01b37b10 bee45728 40306443 00000036 00fcab64 00fcb1fc 00000000 00000000

    [ 4065.498943] ffe0: b48fb114 bee4570c b48f1568 b476011c a80e0010 0000001b 00000000 00000000

    [ 4065.508120] [<c068b8e0>] (xas_load) from [<97ec674a>] (0x97ec674a)

    [ 4065.515342] Code: bad PC value

    [ 4065.518973] ---[ end trace 56b0a57b8af61b99 ]---



    The kodi and dmesg logs are attached..

    logs_202205060915.zip

    Does this look like an OS, or do you think its a Kodi issue?

    Any help would be great.

    Thanks,

    Barry

  • What about CPU temp? Errors in CPU calculations are very likely when operating at high temperatures.

    My RK3228A freezes every 20-30 min without radiator.

  • I have a monster (relatively) heatsink cemented on the CPU, which is 20x20x10mm , so roughly 2x the surface area, of the stock one..

    The CPU generally sits between 63 and 73 Celsius, playing back 1080p content.

    I've gone back to the LE9.2 builds which ran flawlessly on this board, and looked again at their device trees... The LE9.2 builds for v88mars and alfawise A8 had no reliability issues for me.. Both of these device trees have set the following parameter in the regulator definitions for the cpu and memory...

        regulator-settling-time-up-us = <250>;

    I think all the following parameters, which I've been playing with, largely do variations of the same thing, i.e introduce a processing delay when a regulator voltage change is required...

    regulator-settling-time-up-us

      regulator-enable-ramp-delay

    regulator-ramp-delay


    I think I'll also pull in the memory and cpu and ddr frequency / voltage maps for these also, and transplant them into a LE10.x dtb file, and see how that goes.