2 init boot hooks and kernel argument to configure SYSTEM_TORAM_LIMIT (patch included)

1st Official Post

    By messing around with encryption of LibreELEC i patched init with 2 hooks to mount things/ask user for password and so on, so that i dont need to rebuild LE every time i change the init script.

    The second one is to configure how much RAM is needed to copy the SYSTEM file to RAM before mounting it. For me on RPi2 it workt very well with "SYSTEM_TORAM_LIMIT=680000" as kernel parameter. The hardcoded limnit in init is 1GB.

    So here it is init_patch.txt

    Thanks for reading my proposal

    Thanks for the suggestions, will think them over.

    How much of a benefit do you see when using RAM for Storage? To be honest I'd be inclined to drop this feature entirely as in testing (several years ago with OpenELEC) there was little obvious performance benefit, and it just wasted a chunk of RAM (100MB or more).

    I created it for OpenELEC 5 to get around some trouble with sd cards, so that my 3 RPi2 need the sd card only for booting and the storage was on nfs. It was to get the sd card out of the picture as much as possible.

    The days of SD corruption are long behind us, thankfully, so these days there really should be little reason to use the ramdisk in order to avoid accessing the SD card.

    You can also boot over NFS (although with the pre-v1.2 RPi2 you'll still need an SD card with bootcode.bin). init: mount per-client boot/disk if available & configured by MilhouseVH · Pull Request #621 · LibreELEC/LibreELEC.tv · GitHub has some details on how this can be accomplished.

    I haven't looked into whether a ramdisk is beneficial when booting over NFS - copying the SYSTEM image into RAM might possibly result in improved performance and might be worth investigating (your SYSTEM_TORAM_LIMIT patch would be useful as a method to enable ramdisk support when it would otherwise be disabled - you could just set SYSTEM_TORAM_LIMIT=0 to force it).

    Thanks for the informations. The UID feature is cool. And with PXE i did full diskless with OE 4 (i think) and my old Zotac min PC with Nvidia GPU.

    And now as you say it: yes the last sd card problem was a long time ago. I dont hang on the SYSTEM_TORAM_LIMIT feature. It was just there and i used it (i think) some years sind OE 5.

    I was just excided that i finally did take the time to share my work in the forum and i like giving it back. And sometimes a nice sideeffect is that i have something less to maintain.

    Can you tell me more about what you are doing with post_flash() and post_update(), and why they are needed? I'm wondering if they need to be additional boot steps but could be added at the end of mount_flash() and mount_storage(), which would mean post_update() is called before check_update().

    I've got some initial commits here: Comparing LibreELEC:master...milhousevh:le90_init_sky42_patches · LibreELEC/LibreELEC.tv · GitHub

    Can you review the changes and post back in this thread. I know there are three threads discussing different suggestions which are all bundled up in the one branch but keeping the discussion on topic in each of the threads might make things easier.

    I've taken the liberty of modifying your patch slightly:

    1) use "ramlimit=####" instead of SYSTEM_TORAM_LIMIT=####

    2) fold the post-* functions into mount_flash() and mount_storage(), eliminating the extra build steps

    3) use "/storage/.config/post-storage.sh" instead of "/storage/.config/post-update.sh"

    I'm still a little unclear on what you are using the "hooks" for so I've added them as a commit that can be easily dropped if necessary.

    Quote from sky42

    I was just excided that i finally did take the time to share my work in the forum and i like giving it back. And sometimes a nice sideeffect is that i have something less to maintain.

    Yep, everybody wins! :)

    Once you get the hang of it, submitting Pull Requests to github.com is actually easier than the forum, and is likely to catch the attention of a developer much sooner too.

    Sorry i am not yet familiar how to quote in the forum. I try to answer in order.

    What do i want to do with these hooks:

    - probably tweak the update process some how after flash is mounted, but no plans for that so far

    - when storage is mounted i will try to mount .kodi/userdata encrypted on a different partition (working on that), at the moment i have a solution to do that as systemd unit before kodi starts

    - my first PoC was getting a password from the user in the boot process (if the user wants) and post-storage.sh calls get-pw-init.sh.txt and then from autostart.sh do this mount-all.sh.txt done with OE 5 and truecrypt

    - the real goal is when there is encrypted storage (partition) and i can open it umount unecrypted storage and mount the encrypted one instead

    - after that mounting encrypted data disks with the help of a crypttab, that is mainly for offline usage on the road with no server to get the keys/data from (NUC and disks with some films in my luggage)

    - this need the password from the boot process

    - when all i need is in the LE image and not a addon (like truecrypt in OE 5) i can do it earlier and since there is no truecrypt anymore (and it is old and needs replacing) i use cryptsetup, that can mount also truecrypt/veracrypt volumes


    I am fine with 1- 3 and the commit looks fine


    And yes i know i need to lern git and start to use it. I still use cvs (dont laugh) for versioning my server configs/scripts. I need to sit down with one of our developers at work and learn how to use git, have my own github repo and doing pull requests. Sorry for the inconvenience with the patches.

    Oh OK, so this isn't about mounting an encyrpted /storage partition - instead you're mounting encrypted disks (ie. media drives) but need to capture the encryption password/passphrase during the boot sequence.

    Rather than mount disks in autostart.sh, you could replace the [email protected] with your own systemd mount service so that it mounts your encrypted disks as they are connected/reconnected. You could then also access your encrypted drives as automatically shared Samba shares.

    I just re-read your reply and see that mounting an encrypted /storage partition is also your goal.

    I've pushed an extra commit to the PR so that init will now call your script if it exists rather than mount /storage - it would be up to your script to mount /storage as $disk. Obviously the script must now be in /flash, and I've named it mount-storage.sh. I'm not sure where your crypto binaries are stored but presumably they will be accessible before /storage is mounted.

    If we ever wanted to support full disk encryption natively then I'm sure there would be a cleaner solution than this but I'm not really familiar with disk encryption or what packages are available, and also suspect this would be a pretty niche feature. :)

    So hopefully with the PR, you can now:

    1) capture your password/passphrase using /flash/post-flash.sh

    2) mount an encrypted /storage partition with /flash/mount-storage.sh

    3) automount encrypted external drives after replacing [email protected] with something suitable

    Edit: 2018-05-09 Sorry erro corrected: i did see cryptsetup needs lvm2 too

    Interessting change. I will try to incorporate the PR in my community build LE 8.2.5 with UHD 630 / Coffee Lake support and luks, lvm2, dm-raid and test it. When i checked all the changes i will report back.

    But the encryption needs one more feature request cryptsetup luks/truecrypt/veracrypt/loopaes, lvm2 and probably dm-raid too (patches included)

    There is a small kernel config change and the cryptsetup, lvm2 packages are needed. The package mdadm is in there too, because the kernel has all what is needs to do dm-raid.

    I am fairly familiar with encryption and hope to make it easy enough with some dokumentation and some scripts. With the help of a developer it should be not that hard to mount encrypted usb drives as they are attached. That will work too for e.g. veracrypt encryption and a exfat fs on there (created and useable on windows).

    Edited once, last by sky42 (May 9, 2018 at 5:45 PM).

    For building LibreELEC with encryption support we'll need to discuss that in a separate PR, which hopefully you'll be able to push yourself in due course. It's not likely to be a trivial change, so it will need careful thought/consideration - it's a niche feature, so we need to weigh that up.

    I'm happy to leave the post-flash and mount-storage hooks in my current PR, as they could in theory have a general applicability and be useful even without encryption, unless anyone posts an objection on the PR.

    To include encryption in the kernel we'd need to see what impact it has (kernel size, performance overhead) but that can all be discussed in a PR, so that would be the next step.

    • Official Post

    The historic reason for rejecting all attempts to add software RAID and drive encryption to LE is that there are many different schemes and the tiny minority of people who want this are quite religious about their personal choice being the sole divine approach to the problem with all others inferior; so if we allow one crypto or RAID scheme we'll end up being asked for more and we end up having to support all schemes which would be complex for the team to maintain and support as each scheme requires detailed functional knowledge. To avoid that we've simply said 'no' and left things in the hands of community creators to spin their own images and share it if they want. It's not my personal decision, but we had these requests since OE days and no "popular" community image ever emerged which I see as proof that this remains a super-niche topic. As such I don't see a strong reason to change our "less is more" approach. It's kept things sensible for a long time.

    @milhouse and chewitt thanks for the explanation. I did read all the old requests and expected such answer and understand it too. @milhouse you did kind of ask "is that all you need for encryption" and i could not resist to ask for my cryptsetup feature request and also practicly knowing what the answer will be. Sorry.

    I do like less is more too. At work i do whenever possible working by the KISS principal.

    How about giving the community the change to build addons with the encryption, lvm2, mdadm tools, but for that we need support in the kernel. Then i will lern to build addons. And there is less discussion why this tool gets into LE and that not. Most of them need kernel support.

    I checked out le master and build 5 differen le versions in 2 flavors

    PROJECT=Generic ARCH=x86_64

    PROJECT=RPi DEVICE=RPi ARCH=arm

    PROJECT=RPi DEVICE=RPi2 ARCH=arm

    PROJECT=Rockchip DEVICE=RK3399 ARCH=arm UBOOT_SYSTEM=rockpro64

    PROJECT=Amlogic DEVICE=S912 ARCH=arm

    1st full build with IMAGE_SUFFIX=git (fresh and clean from git)

    then i changed the kernel config for encryption, lvm2, raid

    2nd just make again with IMAGE_SUFFIX=git+md+crypt+lvm+raid

    2 concerns i did read about the kernel

    - size for the kernel: that changes less than 10KB and the system around 1MB

    - i am pretty sure there is no mesurable performance impact as long as you dont use the new modules

    You can see for your self here HiDrive all images i build and the git diff for the kernel changes

    ls -g kernel/system sizes le-9.0-testing-images-ls.txt

    kernel config patch le-9.0-kernel-config-git+md+crypt+lvm+raid.patch.txt

    Please consider the support in the kernel. I will build all the kernel config diff or more images for testing. Just ask and say what to build. Sorry for the patches again, i promise i will learn git.


    Thanks for reading

    sky42