shell script to get the latest nightly (old)

So, I wanted to try on my rpi3b+ nightly installation (rpi2 variation of the image), but because the rpi folder (= what I am supposed to change on line 5 for arch) has 2 different images for rpi2 and rp4, I checked what the cut command actually does on line 19. And it returns the image for rpi4, which is wrong

# wget https://test.libreelec.tv/11.0/RPi/
Connecting to test.libreelec.tv (164.92.230.217:443)
saving to 'index.html'
index.html           100% |********************************|  4475  0:00:00 ETA
'index.html' saved

# cat index.html | cut -d " " -f2 | sed -e 's/href=//'g | sed -e 's/"//'g | grep nightly | sort -r | head -1
LibreELEC-RPi4.arm-11.0-nightly-20220609-c66e09a.img.gz

In other folders, e.g. in the rockchip one where there are like 10 different variations, things are worse. But on folders that there is one, e.g. the generic or the samsung one, it works as it should.

---edit

Also, now that the build system sometimes creates 0 byte images, there should be a "fail safe" that e.g. deletes the downloaded file when it is 0 bytes.

Sure, I am willing to try any suggestions. I also searched for a way to make it somehow return the board type, i.e. rpi2 in my case, but sadly I found none. The file /etc/issue seems to be the only one containing it in text.

---edit

I did not notice the quoted text (rough day today, sorry). So, if the script does check for 0 byte files, please ignore my last suggestion.

Couple of suggestions:

• don't use the '-r' option with 'rm' as you don't need it and if there's a bug in your script, potentially catastrophic
• Use https://www.shellcheck.net/ (or the similarly named tool from your Linux distro if you have it) to check your script for errors/warnings and fix them. BEST TOOL EVAR!
  You may want to change the first line to '#!/bin/sh' as otherwise shellcheck will assume you use actual bash, which isn't included in LE.
• Use 'mktemp' to create a temporary file(name), store that in a variable (I'll use 'TMP_DL_FILE')
• use 'wget ${DOWNLOAD_URL} -O ${TMP_DL_FILE} to store 'index.html' as ${TMP_DL_FILE}
• process ${TMP_DL_FILE} as you now used 'index.html'
• when done with it, do "rm -f ${TMP_DL_FILE}"
• Using, and after use cleaning up, your own file doesn't interfere with a potentially existing file with the same name. 'index.html' is far more common then what 'mktemp' produces
• 'Advanced' tip: try to put the various (distinct) functionality in their own function ('check_for_updates', 'download_update', etc) (so things aren't in the 'global namespace')

Quote from jim_p

Sure, I am willing to try any suggestions. I also searched for a way to make it somehow return the board type, i.e. rpi2 in my case, but sadly I found none. The file /etc/issue seems to be the only one containing it in text.

---edit

I did not notice the quoted text (rough day today, sorry). So, if the script does check for 0 byte files, please ignore my last suggestion.

The 0 byte thing is now cleaned up. (But only after a workflow is completed. https://github.com/LibreELEC/acti…uite_focus=true

The pr that fixed this is: https://github.com/LibreELEC/actions/pull/3

You can see the cause of the issue in the build-image logs.

We are working to have a larger server for the .img.gz files. But until then - max 2 copies of each img.gz file and the cleanup is done after the workflow (not the job) completes.

Sorry for complaining about the 0 byte files. I totally missed if's -s parameter when reading the script. I read the new version quickly and I sshed to the rpi to test it. And it somehow fails, probably because of something on wget

# sh get_nightly.sh
Connecting to test.libreelec.tv (164.92.230.217:443)
saving to 'index.html'
index.html           100% |******************************************************************************|  3481  0:00:00 ETA
'index.html' saved


downloading LibreELEC-RPi2.arm-11.0-nightly-20220611-f89b380.img.gz
wait ...


wget: bad address ''


nightly is empty, please check on the Server:


echo

The file is on the server, it is not 0 bytes and I have changed hardware and arch as described in lines 37 to 39. Also, the -vx parameter of sh seems to just output the contents of the script in the terminal, just like cat.

Quote from jim_p

Sorry for complaining about the 0 byte files. I totally missed if's -s parameter when reading the script. I read the new version quickly and I sshed to the rpi to test it. And it somehow fails, probably because of something on

All good. Was annoying me too. There will probably be a number of updates and refinements to the actions code as it is new.

Quote from JoeAverage

I've fixed the most mistakes I made via shellcheck, but not all.

That's a judgment you ofc have to make yourself, but I normally assume that shellcheck is right and I'm not.

Quote

the fixed version of the script is exchanged now in comment #5 to not blow up this thread.

Why not in #1? That's the easiest to find. If you want to keep it in #5, then link to it in #1?

Quote

* just a bad habit or in my fingers using rm with "-r".

I'm not a fan of using -f 'by default' either, but it's easier to make the case where that is warranted

Quote

I couldn't find something "potentially catastrophic";

I delete at three points in the script, always bundled with a true of the prev. command, e.g. cd ~/.update && rm -f *;

that should be fine, cause if nothing is to update there shouldn't be anything in ~/.update (left from me).

Oh, you'd be surprised in all the ways a 'seemingly' simple script can fail. See Bugs in Hello World to see how the simplest program 'of all times' can fail

Over time I've learned to create the ThisShouldNeverHappenException class (in Java) and you'd be surprised how many times that got triggered :-O

I don't know about busybox's rm applet, but the 'standard' rm program added a check that rm -rf wasn't executed from '/' and errors out (unless you add an explicit parameter to override that) even though the user specified '-f' (force)! Too many people ran that command 'accidentally' from the wrong dir and thereby completely wiping out their system. Unrecoverable.

It's good that you thought about defensive programming

Quote

also I work in /tmp what is a) cleaned via reboot and b) I delete what I downloaded there

* 'mktemp'

there is a variant using tmpfile with a process number at the end in bash (in sh too ?)

currently can't remember hwo it extactly called ...

You use wildcard ('*') instead of explicitly named file(s) (or directories), so you may be using/removing more then you expect(ed). If other programs/addons use /tmp too, you may be breaking their operations.

I used mktemp as that is available on LE, but IIRC (on my Debian Sid box) its use is actually deprecated. The goal is to create a temporary file (or directory) yourself, work with that file/directory and when done, clean/remove that file/directory (explicitly, thus no use of wildcards). Which function is used to create to temporary file/directory, is not important.

Quote

* script runs on LE here with #!/bin/bash

Yep and that works.

But if you check the script with tools like shellcheck, then it verifies it against the actual bash program, which isn't what's used on LE. Changing the shebang to #!/bin/dash (the default non-interactive shell on Debian (based?) systems) may be useful too as it may reveal issues when it would be run under that shell. Even though you have no intention to actually run it against dash, it could improve the quality of your script nonetheless.

Quote

* functions

I usually do use them esp. when I need to run parts of the script more then once

That's indeed the most common reason to use functions.

Another reason is to make the code better structured and (thereby) readable. I think that's a worthwhile goal in itself, but it also makes it easier for others to review the script

When code is in the 'global namespace' it gets run no matter what, including after an error occurred which is probably not what you want. When code is put into functions, that code is only executed when explicitly called.

And that call can be guarded/surrounded by various test/if-statements.

If your code is structured like this:

setup_working_dir() {}
check_for_update() {}
download_update() {}
verify_downloaded_update() {}
install_update() {}
cleanup_working_dir() {}

Then the code in the 'global namespace' would then just be calling those functions, guarded by checks.

Very readable and code only gets executed when all the preconditions are met.

There are (ofc) multiple ways to do that, but this (structure) is what I prefer. YMMV.

Quote

extend the script with more hardware types ... [like RPi]

You may want to take into account that people could be using a 512MB or 1GB SD card on which downloads fail because of -ENOSPACE

HTH

Quote from JoeAverage

yeah, my mistake, sorry !

line 71:
wget -nc "${SERVER_URL}"/"${NEW_NIGHTLY}"; should read

wget -nc "${DOWNLOAD_URL}"/"${NEW_NIGHTLY}";

After this change, I would like to report that today the script worked flawlessly on my rpi3b+ and upgraded it to today's nightly.

Quote from JoeAverage

3. I nowhere check the needed disk space to download, copy the ~/backup and to handle the uncompression the update installer does.

for 3.:

I guess I need ~2.5 - 3x the image size as required disk space (1x update, 1-2x the uncompressed update, 1x copy in backup).

What I do is DL it to ~/backups/ and then copy it to ~/.update/. And then reboot it.

I do NOT do any decompression myself (but on reboot I assume it does get decompressed (in RAM?)). I assume that the reboot+update also removes the copy from ~/.update/.

Quote

therefore I need to fetch the image size from download server *before* I start downloading at all.

I don't know if it already exist, but a (small) downloadable file with the filename + size + checksum (to verify whether the DL succeeded) seems very useful and shouldn't be too hard. That's something LE needs to provide, but could become part of the build process?

Quote

I'm completely undecided to put the new script online again ..., until at least I get a fix for the disk space part ...

Have you thought about putting it in a(n online) git repo? (and putting a link to that in your script thread)

Code should be put under version control (git) and it allows others to create Pull (or Merge) Requests, one can use git's tool to see what changed between version, etc.

If you put care into your git commit messages, that can help (future) you and others to understand why certain changes have been made. That also helps you in your learning process

Quote from JoeAverage

But I told you in another thread: don't use the script anymore

Well, I had to test its complete functionality, even for once!

As for /etc/issue, it shows exactly what you see in the motd when you log in via ssh, e.g.

##############################################
#                 LibreELEC                  #
#            https://libreelec.tv            #
##############################################


LibreELEC (community): nightly-20220612-363112b (RPi2.arm)

The text in the parentheses is the only thing that mentions the architecture. On x64 it says "generic.x86_64".