shell script to get the lastest nightly

  • as a command line user on LE and the currently fixed lack to have previous nightlies on a LE box for e.g. roll backs, I wrote a shell script.

    it supports the lastly changed directory structure on the nightly download server.


    I think it could be usefull for others and so I would like to share it here


    comments, bug reports and test results are welcome


    What is does

    - reads out /etc/os-release to get the nightly variant from your currently running LE box

    - checks the download server for last (for updates) and previous (for downgrades) nightlies

    - calculates need disk space (min. ~576 MB, nightly image de-compression factor is for Generic 3 and RPI* 5 !)

    - downloads the choosen nightly Image from the download server and places it under ~/.update

    - offers to reboot


    What it doesn't do

    - does NOT support boxes which are NOT Generic, Generic-legacy, RPi2, RPi4

    - does NOT support switching between Generic and Generic-legacy nightlies


    HowTo

    - copy the code in the code box in a file, lets say "get_nightly.sh"

    - place the get_nightly.sh on your LE box somewhere, but NOT in ~/.update or /tmp

    - run on linux (only once) "chmod u+x get_nightly.sh" to make the script executable


    starting the script:

    - ssh into the LE box

    - move to the folder where the script is stored

    - run the script via command "./get_nightly.sh"


    BE AWARE

    - NO WARRANTY by me and use at you own risk.

    Changelog

    Version 0.08

    - fix a bug with Generic-legacy

    - fix typo's


    Version 0.07

    - change in line 287: FACTOR=4 to FACTOR=3


    Version 0.06

    - bug fix (what is for update/downgrade) - hopefully fixed now, tested though ... -


    Version 0.05

    - fix a bug: un-commenting line 226


    Version 0.04

    - initial release


    ===

    DigiBit R1, NUC8i3BEH

    Edited 41 times, last by JoeAverage ().

  • So, I wanted to try on my rpi3b+ nightly installation (rpi2 variation of the image), but because the rpi folder (= what I am supposed to change on line 5 for arch) has 2 different images for rpi2 and rp4, I checked what the cut command actually does on line 19. And it returns the image for rpi4, which is wrong

    Code
    # wget https://test.libreelec.tv/11.0/RPi/
    Connecting to test.libreelec.tv (164.92.230.217:443)
    saving to 'index.html'
    index.html           100% |********************************|  4475  0:00:00 ETA
    'index.html' saved
    
    # cat index.html | cut -d " " -f2 | sed -e 's/href=//'g | sed -e 's/"//'g | grep nightly | sort -r | head -1
    LibreELEC-RPi4.arm-11.0-nightly-20220609-c66e09a.img.gz

    In other folders, e.g. in the rockchip one where there are like 10 different variations, things are worse. But on folders that there is one, e.g. the generic or the samsung one, it works as it should.


    ---edit

    Also, now that the build system sometimes creates 0 byte images, there should be a "fail safe" that e.g. deletes the downloaded file when it is 0 bytes.

    Edited once, last by jim_p ().

  • Also, now that the build system sometimes creates 0 byte images, there should be a "fail safe" that e.g. deletes the downloaded file when it is 0 bytes.

    as said I own a x86_64 box only, so I couldn't test others, but *try* to come up with a solution.


    are you willing to test then again ?


    P.S.

    I don't think to support Allwinner, Rocketship, etc. e.g. where some parts of the HW Variants are following the nightly git tag !

    DigiBit R1, NUC8i3BEH

    Edited once, last by JoeAverage ().

  • Sure, I am willing to try any suggestions. I also searched for a way to make it somehow return the board type, i.e. rpi2 in my case, but sadly I found none. The file /etc/issue seems to be the only one containing it in text.


    ---edit

    I did not notice the quoted text (rough day today, sorry). So, if the script does check for 0 byte files, please ignore my last suggestion.

  • I did not notice the quoted text (rough day today, sorry). So, if the script does check for 0 byte files, please ignore my last suggestion.

    your complain was valid !

    I left the empty nighly in ~/.update. now fixed.


    anyway, a new release of the script what now supports: Generic, RPi2, RPi4

    alas I only could test for Generic and also downloading the RPi*, but for those NOT the compare part (last nightly already on the box)


    could you please test ?


    you could also start the script with sh -vx ./get_nightly.sh

    DigiBit R1, NUC8i3BEH

    Edited 6 times, last by JoeAverage ().

  • Couple of suggestions:

    • don't use the '-r' option with 'rm' as you don't need it and if there's a bug in your script, potentially catastrophic
    • Use https://www.shellcheck.net/ (or the similarly named tool from your Linux distro if you have it) to check your script for errors/warnings and fix them. BEST TOOL EVAR!
      You may want to change the first line to '#!/bin/sh' as otherwise shellcheck will assume you use actual bash, which isn't included in LE.
    • Use 'mktemp' to create a temporary file(name), store that in a variable (I'll use 'TMP_DL_FILE')
    • use 'wget ${DOWNLOAD_URL} -O ${TMP_DL_FILE} to store 'index.html' as ${TMP_DL_FILE}
    • process ${TMP_DL_FILE} as you now used 'index.html'
    • when done with it, do "rm -f ${TMP_DL_FILE}"
    • Using, and after use cleaning up, your own file doesn't interfere with a potentially existing file with the same name. 'index.html' is far more common then what 'mktemp' produces
    • 'Advanced' tip: try to put the various (distinct) functionality in their own function ('check_for_updates', 'download_update', etc) (so things aren't in the 'global namespace')
  • diederik, thanks for suggestions.

    I've fixed the most mistakes I made via shellcheck, but not all.


    the fixed version of the script is exchanged now in comment #1 to not blow up this thread.


    * just a bad habit or in my fingers using rm with "-r".

    I couldn't find something "potentially catastrophic";

    I delete at three points in the script, always bundled with a true of the prev. command, e.g. cd ~/.update && rm -f *;

    that should be fine, cause if nothing is to update there shouldn't be anything in ~/.update (left from me).

    also I work in /tmp what is a) cleaned via reboot and b) I delete what I downloaded there


    * script runs on LE here with #!/bin/bash


    * 'mktemp'

    there is a variant using tmpfile with a process number at the end in bash (in sh too ?)

    currently can't remember hwo it extactly called ...


    * functions

    I usually do use them esp. when I need to run parts of the script more then once


    All in all the script only should automate some tasks one normally need to do by hand:

    - open the browser,

    - go to the downlaod page,

    - get last nightly link,

    - ssh to the LE box

    - cd to ~/.update,

    - wget last nightly

    - or the last three for samba users in one stroke: just place the nightly to the share: ~/.update/

    - or further when update via AddOn works the script has lost it's or most of it benefit.

    initial target was to keep elder nightly for a roll back; now they're available on the LE server, so what ?


    last:

    an hour ago I was thinking to extend the script with more hardware types and I discovered that I dynamically could read the info's I need from /etc/issue and kodi.log ... (me idiot :/ )

    but as of now it could update Generic and I hope others can do RPi* too (couldn't test; no HW here)

    DigiBit R1, NUC8i3BEH

    Edited once, last by JoeAverage ().

  • Sure, I am willing to try any suggestions. I also searched for a way to make it somehow return the board type, i.e. rpi2 in my case, but sadly I found none. The file /etc/issue seems to be the only one containing it in text.


    ---edit

    I did not notice the quoted text (rough day today, sorry). So, if the script does check for 0 byte files, please ignore my last suggestion.


    The 0 byte thing is now cleaned up. (But only after a workflow is completed. https://github.com/LibreELEC/a…10?check_suite_focus=true


    The pr that fixed this is: https://github.com/LibreELEC/actions/pull/3


    You can see the cause of the issue in the build-image logs.


    We are working to have a larger server for the .img.gz files. But until then - max 2 copies of each img.gz file and the cleanup is done after the workflow (not the job) completes.

  • Sorry for complaining about the 0 byte files. I totally missed if's -s parameter when reading the script. I read the new version quickly and I sshed to the rpi to test it. And it somehow fails, probably because of something on wget


    The file is on the server, it is not 0 bytes and I have changed hardware and arch as described in lines 37 to 39. Also, the -vx parameter of sh seems to just output the contents of the script in the terminal, just like cat.

  • Sorry for complaining about the 0 byte files. I totally missed if's -s parameter when reading the script. I read the new version quickly and I sshed to the rpi to test it. And it somehow fails, probably because of something on

    All good. Was annoying me too. There will probably be a number of updates and refinements to the actions code as it is new.

  • I've fixed the most mistakes I made via shellcheck, but not all.

    That's a judgment you ofc have to make yourself, but I normally assume that shellcheck is right and I'm not.

    Quote

    the fixed version of the script is exchanged now in comment #5 to not blow up this thread.

    Why not in #1? That's the easiest to find. If you want to keep it in #5, then link to it in #1?

    Quote

    * just a bad habit or in my fingers using rm with "-r".

    I'm not a fan of using -f 'by default' either, but it's easier to make the case where that is warranted

    Quote

    I couldn't find something "potentially catastrophic";

    I delete at three points in the script, always bundled with a true of the prev. command, e.g. cd ~/.update && rm -f *;

    that should be fine, cause if nothing is to update there shouldn't be anything in ~/.update (left from me).

    Oh, you'd be surprised in all the ways a 'seemingly' simple script can fail. See Bugs in Hello World to see how the simplest program 'of all times' can fail ;)

    Over time I've learned to create the ThisShouldNeverHappenException class (in Java) and you'd be surprised how many times that got triggered :-O

    I don't know about busybox's rm applet, but the 'standard' rm program added a check that rm -rf wasn't executed from '/' and errors out (unless you add an explicit parameter to override that) even though the user specified '-f' (force)! Too many people ran that command 'accidentally' from the wrong dir and thereby completely wiping out their system. Unrecoverable.


    It's good that you thought about defensive programming :thumbup:

    Quote

    also I work in /tmp what is a) cleaned via reboot and b) I delete what I downloaded there


    * 'mktemp'

    there is a variant using tmpfile with a process number at the end in bash (in sh too ?)

    currently can't remember hwo it extactly called ...

    You use wildcard ('*') instead of explicitly named file(s) (or directories), so you may be using/removing more then you expect(ed). If other programs/addons use /tmp too, you may be breaking their operations.


    I used mktemp as that is available on LE, but IIRC (on my Debian Sid box) its use is actually deprecated. The goal is to create a temporary file (or directory) yourself, work with that file/directory and when done, clean/remove that file/directory (explicitly, thus no use of wildcards). Which function is used to create to temporary file/directory, is not important.

    Quote

    * script runs on LE here with #!/bin/bash

    Yep and that works.

    But if you check the script with tools like shellcheck, then it verifies it against the actual bash program, which isn't what's used on LE. Changing the shebang to #!/bin/dash (the default non-interactive shell on Debian (based?) systems) may be useful too as it may reveal issues when it would be run under that shell. Even though you have no intention to actually run it against dash, it could improve the quality of your script nonetheless.

    Quote

    * functions

    I usually do use them esp. when I need to run parts of the script more then once

    That's indeed the most common reason to use functions.

    Another reason is to make the code better structured and (thereby) readable. I think that's a worthwhile goal in itself, but it also makes it easier for others to review the script

    When code is in the 'global namespace' it gets run no matter what, including after an error occurred which is probably not what you want. When code is put into functions, that code is only executed when explicitly called.

    And that call can be guarded/surrounded by various test/if-statements.


    If your code is structured like this:

    Code
    setup_working_dir() {}
    check_for_update() {}
    download_update() {}
    verify_downloaded_update() {}
    install_update() {}
    cleanup_working_dir() {}

    Then the code in the 'global namespace' would then just be calling those functions, guarded by checks.

    Very readable and code only gets executed when all the preconditions are met.

    There are (ofc) multiple ways to do that, but this (structure) is what I prefer. YMMV.

    Quote

    extend the script with more hardware types ... [like RPi]

    You may want to take into account that people could be using a 512MB or 1GB SD card on which downloads fail because of -ENOSPACE ;)


    HTH

  • The file is on the server, it is not 0 bytes and I have changed hardware and arch as described in lines 37 to 39. Also, the -vx parameter of sh seems to just output the contents of the script in the terminal, just like cat.

    yeah, my mistake, sorry !

    line 71:
    wget -nc "${SERVER_URL}"/"${NEW_NIGHTLY}"; should read

    wget -nc "${DOWNLOAD_URL}"/"${NEW_NIGHTLY}";


    please fix that in your script !


    and yeah "-vx" is some sort of debugging to show how the Variables the script uses gets filled/what there values are when the script runs


    You may want to take into account that people could be using a 512MB or 1GB SD card on which downloads fail because of -ENOSPACE

    very good point !

    need to think about how to manage that ...


    and I need to think somewhat more since "legacy"-variants hit the scene ...

    and after some more thinking I pulled the script completely !

    DigiBit R1, NUC8i3BEH

    Edited 5 times, last by JoeAverage: Merged a post created by JoeAverage into this post. ().

  • yeah, my mistake, sorry !

    line 71:
    wget -nc "${SERVER_URL}"/"${NEW_NIGHTLY}"; should read

    wget -nc "${DOWNLOAD_URL}"/"${NEW_NIGHTLY}";

    After this change, I would like to report that today the script worked flawlessly on my rpi3b+ and upgraded it to today's nightly.

  • jim_p, thanks for the report.

    But I told you in another thread: don't use the script anymore


    1. it doesn't decide between Generic and Generic-legacy and as you told in the other thread: switching between both isn't that flawless.

    2. there is a left over in ~/.update when hitting CRTL+C. maybe a half downloaded/damaged image.

    don't know what it exactly means, but I guess it won't harm (untested !) cause the update installer run a sort of check summing and should deny installing the update

    3. I nowhere check the needed disk space to download, copy the ~/backup and to handle the uncompression the update installer does.

    (kudos diederik :) )


    anyway, I rewrote the script to handle 1. and 2., but currently not 3.


    for 3.:

    I guess I need ~2.5 - 3x the image size as required disk space (1x update, 1-2x the uncomressed update, 1x copy in backup).

    therefore I need to fetch the image size from download server *before* I start downloading at all.


    for 1.:

    I currently fetch the HW; Arch, etc. sorta dynamically from the running box, but currently not for other images (as of now only: generic and generic-legacy is handled)


    The lack I have is the output of "cat /etc/issue" for esp. RPi's and others and of cource the HW to test (owning a Generic only)


    care to post that for your RPi ?

    maybe I get some more grip...


    I'm completely undecided to put the new script online again ..., until at least I get a fix for the disk space part ...

    DigiBit R1, NUC8i3BEH

    Edited once, last by JoeAverage ().

  • 3. I nowhere check the needed disk space to download, copy the ~/backup and to handle the uncompression the update installer does.

    for 3.:

    I guess I need ~2.5 - 3x the image size as required disk space (1x update, 1-2x the uncompressed update, 1x copy in backup).

    What I do is DL it to ~/backups/ and then copy it to ~/.update/. And then reboot it.

    I do NOT do any decompression myself (but on reboot I assume it does get decompressed (in RAM?)). I assume that the reboot+update also removes the copy from ~/.update/.

    Quote

    therefore I need to fetch the image size from download server *before* I start downloading at all.

    I don't know if it already exist, but a (small) downloadable file with the filename + size + checksum (to verify whether the DL succeeded) seems very useful and shouldn't be too hard. That's something LE needs to provide, but could become part of the build process?

    Quote

    I'm completely undecided to put the new script online again ..., until at least I get a fix for the disk space part ...

    Have you thought about putting it in a(n online) git repo? (and putting a link to that in your script thread)

    Code should be put under version control (git) and it allows others to create Pull (or Merge) Requests, one can use git's tool to see what changed between version, etc.

    If you put care into your git commit messages, that can help (future) you and others to understand why certain changes have been made. That also helps you in your learning process

  • Have you thought about putting it in a(n online) git repo? (and putting a link to that in your script thread)

    Huh, my small script is usable just for automated home task and shouldn't be blown up to a software project.

    Me has a shortage of "brain windings" for programming, cause I first wrote the favorite "Hello world" in C with the age of 30 (1991) and bought my first computer at that time (Intel 386 33Mhz, 2 MB RAM, 40 MB HD, :angel: )

    means: I started very late to train that and now it's long ago

    I guess my knowledge fot programming (I've been working as a OS supporter/(Co-) admin) is just sufficing to do some (small) scripting and sorta extended "Hello world" "projects".

    that's all

    I don't know if it already exist, but a (small) downloadable file with the filename + size + checksum (to verify whether the DL succeeded) seems very useful and shouldn't be too hard.

    +1

    best: a text file

    if I get it right: the update installer does a sorta check summing, if in RAM only - don't know ! -


    file size is there too, but I need to fetch that info out of the downloaded file "index.html", not done yet.

    and it seems the download directories are changing somehow: yesterday a new variant hit the scene ...


    and at least it should be clear:

    when the update installer from the LE-Addon is able to handle all files in the download server directories (think: image roll back; change between generic/generic-legacy) then the script is (mostly) dead anyway.

    DigiBit R1, NUC8i3BEH

  • But I told you in another thread: don't use the script anymore

    Well, I had to test its complete functionality, even for once!

    As for /etc/issue, it shows exactly what you see in the motd when you log in via ssh, e.g.

    Code
    ##############################################
    #                 LibreELEC                  #
    #            https://libreelec.tv            #
    ##############################################
    
    
    LibreELEC (community): nightly-20220612-363112b (RPi2.arm)

    The text in the parentheses is the only thing that mentions the architecture. On x64 it says "generic.x86_64".