Posts by oneillb

I now have a build and DTB that work really well on my r329q v3.0 board... which I've bult an .img.gz image from.

I burned to SD and booting from SD everything is OK... I then burned my build to the NAND, and again everything is working fabulously.... left the box running overnight, no Kodi issues, nothing weird in dmesg and 12hours uptime, which is great.

Only 1 small issue when booting from NAND..... Librelec 10.x does not seem to recognise the onboard MMC/SD card slot, or any SD card plugged into the onboard slot... It looks to be defined in the device tree, but there is no /dev/mmcblk0 device created when an sd card is inserted.

The kernel detects it , and creates the necessary device and mounts the partitions when I boot from SD with the same image and DTB.

Is this a u-boot thing rather than the linux kernel?

I've taken on board your comments, and I'll try lowering the voltages from my baseline. I'll take 0.025V off every step, to see if things remain stable..

Temperature is something I am aware of, the 1st thing I did when I was donated this box was to change the heatsink to one that's about 2x the size.. The temperature is generally in the 65-75 range... and occasionally will get up to 82.

Thanks for all the advice!

Barry

ilmich, jock2, fabiobassa

I've been tweaking the CPU voltages, and I've a set which have transformed the r329q v3.0 board I have... from being so unstable that the setup wizard couldn't be completed, and crashing every few minutes. It's now up for several hours at a time ( I've not attempted to run for any longer, but I'll do a longer Kodi trial over the next few days, and another memtester run) ... It's amazing how 0.025V or 0.050V can make a massive difference!

I've also been able to remove, one by one the, other regulator ramp / delay parameters, and I was able to maintain the stability. And kept DMC enabled too - It's disabled in the v88mars DTB and it wasn't being detected during boot and switched on, I'm not sure how the DDR2 / DDR3 sensing works... but I had to force it to be enabled.

Over the next few days, I'll turn on the drivers I've blacklisted, and turn on the features which have been turned off (e.g samba, avahi etc.), I think this will be OK now.

I've forked your LE project in Github, and I'll probably send a pull request in the next few days for you to consider.

The changes are subtle, I've boosted the voltages at the slower CPU speeds, so there's a much more linear relationship now between frequency and voltage, and bumped the Kernel to Linux 5.10.113

I hope you decide to test / verify them on your board, and hopefully incorporate them into your build.

Maybe jock2 and fabiobassa can try these settings on your similar boards.

Thanks to everyone for all your help and ideas! I've learned a lot!

Barry

I have a monster (relatively) heatsink cemented on the CPU, which is 20x20x10mm , so roughly 2x the surface area, of the stock one..

The CPU generally sits between 63 and 73 Celsius, playing back 1080p content.

I've gone back to the LE9.2 builds which ran flawlessly on this board, and looked again at their device trees... The LE9.2 builds for v88mars and alfawise A8 had no reliability issues for me.. Both of these device trees have set the following parameter in the regulator definitions for the cpu and memory...

    regulator-settling-time-up-us = <250>;

I think all the following parameters, which I've been playing with, largely do variations of the same thing, i.e introduce a processing delay when a regulator voltage change is required...

regulator-settling-time-up-us

  regulator-enable-ramp-delay

regulator-ramp-delay

I think I'll also pull in the memory and cpu and ddr frequency / voltage maps for these also, and transplant them into a LE10.x dtb file, and see how that goes.

OK - The memtester run from yesterday completed successfully.... eventually.

I then did some calcs in a spreadsheet. The total delays using the default kernel values seem incredibly short.... unless regulator_enable_ramp_delay defaults to a non-zero value if undefined, but the documents don't suggest there is a default

Defaulted Kernel dtb values

jock2 recommended the following, considerably longer delays generally.

my initial "guess" values actually result in an overall delay in the same ballpark as jock2, and seem to work fairly well also....

I then tried to find where the sweet-spot might be, working from low overall delay, and increasining to get more uptime... I've had my board running for > 10 hours with the following settings.... which is a massive improvement...

However.... Although the Librelec OS has been up > 10 hours, Kodi.bin partially crashed after 70minutes....

The IR LED was still responsive, and I could ssh onto the box and actually shutdown kodi using systemctrl stop kodi, and then I ran another memtester over 825MBtest, which is nearly finished now...

There was no kodi crashlog, as kodi was still running to some extent, but there were dmesg errors for the time the clock had stopped on the UI (09:15) . With 3 messages like this...

Thread JobWorker 2835341952 terminating (autodelete)

Thread JobWorker 2817503872 terminating (autodelete)

Thread JobWorker 2809111168 terminating (autodelete)

There was a corresponding kernel error...

[ 4065.211853] Hardware name: Generic DT based system

[ 4065.217217] PC is at 0x0 [ 4065.224612] LR is at xas_load+0xc/0x70

[ 4065.228812] pc : [<00000000>] lr : [<c068b8e0>] psr: 680e0013

[ 4065.235829] sp : c3c0fd18 ip : ffffc005 fp : ffffffff

[ 4065.241675] r10: 00000008 r9 : 00000003 r8 : 00000402

[ 4065.247527] r7 : 00000406 r6 : c3c0fd54 r5 : 00000003 r4 : 00000000

[ 4065.254832] r3 : 97ec674a r2 : 97ec674a r1 : ffffffff r0 : 00000000

[ 4065.262142] Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none

[ 4065.270135] Control: 10c5387d Table: 637c406a DAC: 00000051

[ 4065.276572] Process kodi.bin (pid: 611, stack limit = 0x60538f89)

[ 4065.283395] Stack: (0xc3c0fd18 to 0xc3c10000)

[ 4065.288275] fd00: 00000000 97ec674a

[ 4065.297433] fd20: 00000000 c1f58db0 c3c0e000 c54d76c8 c482dec0 00000000 c4509b4c 0000003f

[ 4065.306595] fd40: c37d9a00 c0758670 00000000 00000cc0 c4509a50 00000000 00000000 c54d7680

[ 4065.315753] fd60: c3c0fdf4 97ec674a c54d7680 c54d7680 c3c0fdf4 00000000 c482dec0 c079d54c

[ 4065.324908] fd80: c3d79800 c3c0e000 c1505434 00000000 c4509a50 c3c0e000 c3d79800 00000000

[ 4065.334065] fda0: 00000000 c1d35100 0000e4c4 0000004a 00010000 97ec674a 00000000 c3c0fe74

[ 4065.343223] fdc0: c2430468 c4509a50 c3c0e000 c4509800 c4833900 c54d7680 c4833a00 c079a238

[ 4065.352380] fde0: 00ba3d08 c482d840 c54d76f0 00000110 c496ce40 c4833a00 00000001 00000000

[ 4065.361545] fe00: c4509800 c4509a50 0000004a 00000000 00000000 00000004 c54d7680 97ec674a

[ 4065.370704] fe20: 00000000 40306443 00000000 c0d88170 00000030 c3c0e000 c3c0fe74 00000043

[ 4065.379863] fe40: c4833900 c075a260 0000e280 00000001 c105b378 c0170e34 bee45728 c3c0fe74

[ 4065.389027] fe60: 00000030 c448a000 c0799fe8 00000051 00000000 00000000 00000001 0000004a

[ 4065.398185] fe80: 00000110 01b518a8 00000000 bee45818 00000000 00000000 00000004 00000000

[ 4065.407342] fea0: 00000000 c157a300 c157a300 680e0113 00000000 c01aab88 ef6cf8c8 ef6cf8f0

[ 4065.416503] fec0: ef6cf918 c01c6110 c1504fd0 002b4fe3 00000000 ef6ca380 ffffe000 c1d35100

[ 4065.425663] fee0: 00000009 c3c0e000 c16491a0 ef6ca380 c3c0e000 97ec674a 00000003 40306443

[ 4065.434821] ff00: c448a001 40306443 c3c0e000 bee45728 c448a000 0000001b c24103c8 c0312c80

[ 4065.443984] ff20: 0000000a c148a2dc c1495c40 c1495c40 00113c57 c1503d00 00400100 c9284000

[ 4065.453147] ff40: c3c0ff90 c3c0e000 ffffe000 c3c0ff68 00000001 c9284000 c3c0ff90 c3c0ffb0

[ 4065.462305] ff60: bee45748 bee45748 c1494f44 c1494f44 00000000 c0192688 c1505714 97ec674a

[ 4065.471466] ff80: f0802000 01b37b10 bee45728 40306443 00000036 c01002c4 c3c0e000 00000036

[ 4065.480625] ffa0: 00000000 c0100060 01b37b10 bee45728 0000001b 40306443 bee45728 00000000

[ 4065.489784] ffc0: 01b37b10 bee45728 40306443 00000036 00fcab64 00fcb1fc 00000000 00000000

[ 4065.498943] ffe0: b48fb114 bee4570c b48f1568 b476011c a80e0010 0000001b 00000000 00000000

[ 4065.508120] [<c068b8e0>] (xas_load) from [<97ec674a>] (0x97ec674a)

[ 4065.515342] Code: bad PC value

[ 4065.518973] ---[ end trace 56b0a57b8af61b99 ]---

The kodi and dmesg logs are attached..

logs_202205060915.zip

Does this look like an OS, or do you think its a Kodi issue?

Any help would be great.

Thanks,

Barry

jock2  fabiobassa  ilmich @daflex

Thanks for all the pointers... On the back of that I've done googling etc...

jock2 - The pointer to regulator-ramp-delay was very useful, you recommended a change from 12500 to 300, which is a massive change, given the parameter is a gradient.... I think you may have mixed up 2 different parameters regulator-ramp-delay and regulator-enable-ramp-delay.

Looking at some other rockchip and arm 32bit builds and documentation, regulator-enable-ramp-delay is sometimes set to 200 - 600 µs, which is a fixed delay on each transition and regulator-ramp-delay is the gradient of the ramp, which determines a variable time delay, based on the voltage change. This seems to be usually set somewhere in the 6000 - 12500 µV/µs range.

As a 1st pass, I've restored the original voltage settings from the v88mars.dtb bundled with the build, and modified / set the following on both the vdd_arm and vdd_log regulator definitions:

     regulator-enable-ramp-delay = 300

regulator-ramp-delay = 6000

memtester 4.5.0 is part of the base Librelec 10.x image, so I'm now running that now over 650MB 5x passes, which was the freemem reported by top, which should stress tings a little. Not sure how long that will take, but it's been running for about 60 minutes already, and is still on pass 1.

Admittedly memtester is not complex processing, and the CPU cores are probably all completely maxed out, so not ramping up and down, but 60 minutes of uptime is already an improvement of sorts.

I'll let you know how it works out.

Thanks again,

Barry

jock2 , ilmich,

I've attached some logs from last weekend when I spent some time testing the LE10 build (LibreELEC-RK322x.arm-10.0-nightly-20220426-a0fa878-rk322x.img.gz) from SD card, with the following disabled:

ssv6051

spdif

cec

samba

avahi

librelex_10.x_logs.zip

Some runs are with dmc enabled , others with dmc disabled. The earlier logs are with the default , and later ones with slightly increased (by about 0.05v) cpu voltages.

Many crashes happened within 10 minutes, but some of the latter ones ran for > 40 minutes. In many cases the crash happened with Kodi idle, in others whilst running playback of local .ts files

There doesn't seem to be any consistent pattern. Have a look and see if there is anything you can spot..

Also, is there anything extra I can log, which may help? Is it worth logging the latest LE9.2 build running for example, which runs really well on this board?

Thanks,

Barry

jock2 - the bootloader on the NAND of my r329Q v3 board is rk322x_loader_v1.10.256.bin which I flashed using rkdevelop over the USB OTG port. (I think you pointed me in that direction when booting LE from nand was not working. )

The remainder of the NAND is populated with ilmich's latest LE9.2 image (LibreELEC-RK322x.arm-9.2-devel-20220124165557-3ce5dd4-rk3229-v88mars.img.gz) which was flashed using multitool.

The board will boot from SD (which I'm using to test out the LE10 builds) and fall back to NAND without issue when the SD card is removed... My problem is with stability once things are up and running... If the bootloader puts the board into 1T mode at startup, is that mode maintained through u-boot and kernel startup, or can the kernel driver switch the mode later on?

Hi ilmich,

I think I may have made a breakthrough....

I'm still using your last testing build (LibreELEC-RK322x.arm-10.0-nightly-20220426-a0fa878-rk322x.img.gz) from SD card, with the following disabled:

ssv6051

spdif

cec

samba

avahi

I bought a CP2014 USB -> Uart adapter, and I set it up to monitor the serial console output on the v329Q v3.0 board.. That instantly discovered my box has DDR3 330Mhz ram.. not DDR2 as I had believed.

DDR Version V1.10 20190926
In
ID:0xFFF
660MHz
DDR3
Bus Width=16 Col=11 Bank=8 Row=15 CS=1 Die Bus-Width=16 Size=1024MB
mach:2
OUT
Boot1 Release Time: May 13 2019 17:02:59, version: 2.56
ChipType = 0xc, 269

I could see the u-boot and early kernel messages on the LE9.2 build, but the 10x build showed complete garbage... for some reason the BAUD rate changes from 115200 to 1500000 when u-boot kicks in... I think this is patched in by your 10.x sources here...

projects/Rockchip/devices/RK322x/packages/tools/u-boot/patches/rockchip/u-boot-1000-rk322x-defconfig-dts.patch

I changed my boot line to be...

boot=UUID=2604-2146 disk=UUID=b528aa0c-4dc9-4853-bf9b-45a8c3fa13db console=uart8250,mmio32,0x11030000,115200n8 ignore_loglevel cma=256M debugging

And now I get all the kernel debug out to the serial console... and can see the output when there is a panic.

With DDR3, I went back to the dmc settings. The LE9.2 v88mars build for me was rock-solid. So I transplanted some of those CPU clock and voltage settings from that build into a new dtb for my LE10.x configuration.. Which slightly upped the CPU voltages for the higher clock frequencies, but this wasn't actually that effective. I then upped in a similar fashion the voltages at the lower frequencies, as this is actually where the box spends most of the time running, and things are getting much much better.

CPU Time in State
408000 96934
600000 53380
816000 11680
1008000 2296
1200000 798
1300000 428
1400000 2808


CPU Trans Table
From  :    To
       :    408000    600000    816000   1008000   1200000   1300000   1400000
 408000:         0      7931         0         0         0         0         0
 600000:      7931         0      2393         0         0         0         0
 816000:         0      2393         0       347         0         0         0
1008000:         0         0       347         0       131         0         0
1200000:         0         0         0       131         0        54         0
1300000:         0         0         0         0        54         0        62
1400000:         0         0         0         0         0        62         0


MEM Time in State
From       :   To
           : 330000000 400000000 534000000 660000000   time(ms)
  330000000:         0         0         0        52     18400
  400000000:         6         0         0        31      7016
  534000000:        11        18         0        90     33200
* 660000000:        34        19       119         0   1623726
Total transition : 380

This is the DDR3 and CPU frequency / voltages I'm now using..

rk3229-box-sm1.zip

However,

I'm still getting the odd issue. I had a lockup occur after about 40minutes of playing locally stored .ts files... The LED is still responsive, but nothing else. Here is the console output ..

librelec_10.x_rk332x_lockup_202205021651.txt

Does this suggest I may need to up the voltages a little more? Or is it a different issue? Any help appreciated...

Thanks

Barry

ilmich

I've run up the latest build (LibreELEC-RK322x.arm-10.0-nightly-20220426-a0fa878-rk322x.img.gz) , from SD card, and the linux OS seems to be more resilient...

I've turned on the debugging and set cma=256, as you suggested. I've turned off the ssv6051, and have no other usb devices attached, turned off samba, avahi etc. resolution set to 720p

The kodi process again is crashing after a few minutes of streaming some 1080p .ts files over nfs, but they do not bring the OS down.

When kodi.bin crashes, the process hangs around, and remains visible in ps.. Another kodi.bin starts, but can't initialise the display, which remains as it was when the 1st instance crashed... .

kill and kill -9 are ineffective in killing the older kodi.bin processes, and issuing a reboot via ssh, does not reboot the box.. it brings down the network etc., but there is no reboot..

Only cycling the power will result in a reboot, and the display resetting.

I've attached some logs ...

logs_20220426.zip

jock2  ilmich

Thanks for all the info, I'm learning a lot... I've gone back to the base v88mars build, and removed everything I don't need, as you've recommended.

* svv6051 disabled

* no USB wifi dongle

* using ethernet only,

* no usb keyboard & mouse,

* cec disabled (not sure board has cec)

* avahi switched off

* samba switched off

In approx 60 minutes had 3 os lockups, and 3 other kodi crashes.

The kodi crash logs are attached as is the OS journal, which unfortunately leaves very little about the OS crashes.

logs_20220424.zip

Hopefully this is useful to you.

I limited the GUI resolution and changed the cma settings as recommended, but it didn't change things really, but I've left it that way anyway for now.

However, I think i have resolved the crashing I've been having with my box... which has DDR2 ram. I was using the v88mars.dtb , that board I think has DDR3... and the minimum enabled ram speed in the .dtb file is set to 533MHz

I've rebuilt a dtb specifically for my board, based on the v88mars one, but setting the max freq to 330MHz, by toggling the status= values in the dts and re-compiling. Initial impressions are good, I'll run with this for a day or 2 and see how things go.

Does this sound reasonable?

dmc-opp-table {                                              
                compatible = "operating-points-v2";                  
                phandle = <0x06>;                                        
                                                                     
                opp-330000000 {                                      
                        opp-hz = <0x00 0x13ab6680>;                  
                        opp-microvolt = <0x100590 0x100590 0x124f80>;
                        status = "okay";                             
                };                                                       
                                                                     
                opp-534000000 {                                      
                        opp-hz = <0x00 0x1fd43180>;                  
                        opp-microvolt = <0x100590 0x100590 0x124f80>;
                        status = "disabled";                         
                };                                                       
                                                                     
                opp-660000000 {                                          
                        opp-hz = <0x00 0x2756cd00>;                  
                        opp-microvolt = <0x10c8e0 0x100590 0x124f80>;
                        status = "disabled";                         
                };                                                       
                                                                     
                opp-786000000 {                                      
                        opp-hz = <0x00 0x2ed96880>;                  
                        opp-microvolt = <0x10c8e0 0x100590 0x124f80>;
                        status = "disabled";                         
                };                                                       
        };           

Hi ilmich ,

I've again needed to blacklist the ssv6051 to make things stable, as otherwise things crash after a few minutes.

There are still some Kodi crashes, I've attached some logs . I hope this helps.

logs_20220422.zip

ilmich - That's worked a treat your latest v88mars image (LibreELEC-RK322x.arm-9.2-devel-20220124165557-3ce5dd4-rk3229-v88mars.img.gz) worked straight out of the box......

As this is a sticky thread, perhaps the links on page 1 should point to your later builds, or at least offer a choice.

Would never have found the recent LE9.2 builds without your help.

Thanks all.

Sorry for the confusion... I've been using both LE9.2 (knaerzche's Alfawise A* worked really well from SD) from 2020

and your LE10.x builds from this year.

I see you have more recent LE9.2 builds .. I'll give the some of them a try..

Thanks again.

Hi,

I have an r329q v3.0 board with rk3228, DDR2, sv6051 wifi and NAND storage.

The LE9.2 build for the Alfawise A8 image (LibreELEC-RK322x.arm-9.2-devel-20200427213246-b7186bc-rk3228b-a8.img.gz) is the best for me on my hardware, and I have an SD card running beautifully and reliably including IR..

However, when it's up and running from SD card, I can't see the NAND device /dev/rknand0

When I write the image to NAND using Multitool the boot fails giving a "grep: /proc/net/pnp: No such file or directory" error and drops to a console prompt.

This is a legacy kernel image, do they all have rknand support? Do I need to do anything specific to turn it on?

I've compiled a custom dtb file, by cobbling bits from the dts (deconstructed from the dtb) files for the a8 and v88mars builds, and recompiling, to enable NAND but that didn't do the trick.

BTW . I've previously upgraded the idbloader on the NAND using rk322x_loader_v1.10.256.bin and have had a LE10.x build booting on the NAND previously. Just looking to DG to 9.2 whilst the 10.x builds stabilise.

Any help greatly appreciated.

Barry

Updating the loader using rkdeveloptool and rk322x_loader_v1.10.256.bin has sorted all the booting from NAND issues out.. I must have had a very old / unusual idbloader.

I've also successfully blacklisted ssv6051, I now also have more stable wireless networking (using a wifi dongle now based using a realtek 5390 chipset i think.. which I used previously on an early raspberry Pi) , and I've managed to get a trace in dmesg for a lockup, which was only kodi freeze, not a freeze of the Kernel / OS.

Actually I ran the r329q v3.0 board without Kodi (systemctrl stop kodi after boot) for several hour without issue, so I expect my issues no are with the kodi binary, not the underlying kernel and OS.

dmesg_202204181321.log

Hi again,

I'd say freeze, rather than crash there's no kernel panic or reboot ... with LE10.x, generally within 30-60mins of idling the box will freeze, sometimes more quickly. I'd originally put it down possibly to heat, but I've cemented a much bigger 20x20x10mm heat sink, to the CPU, and generally Kodi reports a temp in mid 60s and up to about 75C when more heavily loaded, which I think is OK.

By comparison the LE9.2 build for Alfawise A8 seems to be bullet proof... I've been running the box with Kodi idling for > 6 hours, playing some low res 360p mp4s over nfs. I have some 1080p .ts files also which it manages, although things struggle when the GUI overlayed.

The LE9.2 build for v88Mars freezes after about 30seconds after I get Kodi up. I don't get through the initial setup wizard.

I'm having a go a building myself from sources on a Ubuntu 21.04 x86_64 machine but I'm having issues with building m4-1.4.18 .

Have a great vacation.

Barry

I've made a little discovery which may be important to understand the root of the problem I have booting from NAND... on my r329q v3.0 board, which has DDR2 RAM, NAND and sv6051p wifi.

I re-wrote the NAND using Multitool and the latest LE 10.x image, as before and didn't boot.. exactly the same as before. However, I then booted off an SD card using the same 10.x image... It booted the kernel from SD, but mounted the rootfs from NAND.

Also, I still have a lot of instability with LE10.x even after I've blacklisted the ssv6051 wifi module, which may be down to dram timing / voltages. I've also been using the LE9.2 build , with the Alfawise a8.dtb (LibreELEC-RK322x.arm-9.2-devel-20200427213246-b7186bc-rk3228b-a8.img.gz ) stability on this build seems much better, with less frequent lockups.

How easy would it be to build a LE10.x dtb file for my board based on the dram configuration from the LE9.2 dtb file?