[SOLVED] Default blocksize on storage partition

    Hi,
    I've noticed, that the blocksize on the storage partition is 1MB.

    I'm aware that this is not a bug, but on the other hand it would also not be a feature request, as this big blocksize is a drawback, which I'd strogly suggest to get changed. Kodi mediascraping stores a lot of tiny <1KB files to the storage partition, which is good so far, but every single file consumes 1MB of storage. So even a 16GB SD card, which is only there for the Library files (Coverart, MetaData, etc.), can get full if the movie library is a bit bigger. And if you start having a music library as well, then every single of those small 4MB mp3's consume the same amount of actually 1KB small coverarts getting inflated to many MegaBytes.

    I can understand that some SD-Cards (Raspberry, Odroid C2) don't like writing a lot of small files from performance point of view. But 1MB per file for a 1byte file (it's blown up by the crazy factor of 1.000.000) is not necessary at all I think. I can't believe that the performance can benefit from this blocksize.

    Also making a backup of the library or restoring the backup takes unproportionally long because of the pure filesizes which get generated here without a reason I could get behind of :/

    Could you please change the blocksize to something like 128KB or even less? Or if not, could you explain then why it is as it is? Perhaps there is a way to change this default behavior?

    Thanks a lot in advance for some statements about this :)

    Edited once, last by infinity85 (November 18, 2016 at 2:24 PM).

    Quote from escalade


    Probably no thought behind it at all, as they just go with the default:

    LibreELEC.tv/installer at master · LibreELEC/LibreELEC.tv · GitHub

    You can simply backup your /storage, overwrite the filesystem with the block size you want and then restore.


    Thank you for your reply :)

    Do you happen to know whether this will be changed somewhen as default?

    If I understand it correctly, you suggest I put the card into my Laptop (Windows machine), use e.g. MiniTool Partition Wizard or similar to reformat the second partition with some other blocksize, yes?

    Any recommendation which blocksize / filesystem would be best?

    Thank you very much escalade!

    Edited once, last by infinity85 (November 16, 2016 at 2:49 PM).

    Quote from infinity85


    Do you happen to know whether this will be changed somewhen as default?

    Not to my knowledge. I'm sure if you could document that a different default block size would be beneficial a change would be considered.

    Quote


    If I understand it correctly, you suggest I put the card into my Laptop (Windows machine), use e.g. MiniTool Partition Wizard or similar to reformat the second partition with some other blocksize, yes?

    No, I would never recommend using Windows to create Linux filesystems.

    Quote


    Any recommendation which blocksize / filesystem would be best?

    I have no idea, you are the one who brought this up so you should know :)

    Edited once, last by escalade (November 16, 2016 at 3:00 PM).

    Quote from escalade

    Not to my knowledge. I'm sure if you could document that a different default block size would be beneficial a change would be considered.

    No, I would never recommend using Windows to create Linux filesystems.

    I have no idea, you are the one who brought this up so you should know :)


    fair enough, will see what I can do :D...

    Thanks again ;)

    Quote from trogggy


    It's not a recommendation, rather an observation, but I've been using mini-toolPW for years to format for openelec and libreelec with no issues.


    Thanks! I've tried it, but I don't get it working. LibreELEC can't boot as soon as I reformat the second partition.

    I tried these two scenarios:

    1. Flashed the current v7.90.008 ALPHA and booted it the first time. LibreELEC expanded the second partition as expected. Then I reformatted this partition with MiniTool Partition Wizard Free to ext4 with 4kb blocksize. Put the card into my odroid c2 again and it refused to boot (I expected it, because .config etc. was also on this second "storage" partition before reformatting)

    2. I flashed the sd card from scratch with v7.90.008 ALPHA and then I reformatted the (now) unallocated space for the second partition (which is to be expanded during first boot) with ext4 4kb blocksize. It refuses to boot again.

    What is the correct way to get a new installation of LibreELEC onto a manually formatted disk? Or how do you reformat only the second partition as suggested by escalade in the first reply?

    Quote from infinity85


    Thanks! I've tried it, but I don't get it working. LibreELEC can't boot as soon as I reformat the second partition.

    I tried these two scenarios:

    1. Flashed the current v7.90.008 ALPHA and booted it the first time. LibreELEC expanded the second partition as expected. Then I reformatted this partition with MiniTool Partition Wizard Free to ext4 with 4kb blocksize. Put the card into my odroid c2 again and it refused to boot (I expected it, because .config etc. was also on this second "storage" partition before reformatting)

    2. I flashed the sd card from scratch with v7.90.008 ALPHA and then I reformatted the (now) unallocated space for the second partition (which is to be expanded during first boot) with ext4 4kb blocksize. It refuses to boot again.

    What is the correct way to get a new installation of LibreELEC onto a manually formatted disk? Or how do you reformat only the second partition as suggested by escalade in the first reply?


    You just motivated me to get around to sticking 7.9 on a pi3, so here's what I did.
    1. Wrote the image file to the sd card (I use win32diskimager but it makes no difference).
    2. Fired up MTPW. Deleted the 32MB ext4 mini-partition. Expanded the Fat32 partition (don't ask). Filled the rest of the sd card with an ext4 partition - primary, 4K sectors, label = LIBREELEC_DISK - leaving a bit of unallocated space at the end (handy if you're trying to restore an image to a card that's fractionally smaller).
    3. Stuck it in the pi.

    infinity85 please boot your Odroid C2 into LE after flashing with the latest official image, and while logged into the Odroid C2 provide the output of

    Code
    tune2fs -l /dev/mmcblk0p2 | pastebinit


    and paste the link.

    On Odroid the default ext4 block size should be 1024 bytes (1KB), on everything else (Wetek, RPi, Generic) it should be 4096 bytes (4KB).

    I'm not sure how you are seeing a block size of 1MB.

    @trogggy
    thanks a lot for the mini guide :)

    Quote from milhouse


    infinity85 please boot your Odroid C2 into LE after flashing with the latest official image, and while logged into the Odroid C2 provide the output of

    Code
    tune2fs -l /dev/mmcblk0p2 | pastebinit


    and paste the link.

    On Odroid the default ext4 block size should be 1024 bytes (1KB), on everything else (Wetek, RPi, Generic) it should be 4096 bytes (4KB).

    I'm not sure how you are seeing a block size of 1MB.

    Good question and point!
    Here's the output: NcXX
    It shows 1024, so this means it is bytes then?

    I was seeing it via SMB. Whenever I tried to make a manual backup of some folders I noticed that the size of lets say the thumbnails folder on my main installation is like this:

    Quote


    \\libreelec\Userdata\Thumbnails
    Contents: 3.212 Dateien, 18 Ordner
    Actual size: 212 MB (222.312.844 Bytes)
    Size on disk: 3,13 GB (3.369.074.688 Bytes)

    Thats like 14 times the filesize on the disk.

    Here's a screenshot of the by you suggested new v7.90.008 ALPHA installation on my C2:

    It implies that Blocksize is 1MB for every file. If I display those properties for one file, it will show 1MB respectively a multiple of 1MB.

    So does that mean that samba makes also some kind of abstraction for displaying and calculating filesizes (I know that it abstracts the actual filesystem)? That'd mean that I got fooled by samba :D :rolleyes: Is there any explanation for this?

    Edited once, last by infinity85 (November 17, 2016 at 11:58 PM).

    Quote from infinity85


    Here's the output: NcXX
    It shows 1024, so this means it is bytes then?

    Yes, 1024 bytes, or 1KB, which is perfectly normal and reasonable.

    Quote from infinity85


    Here's a screenshot of the by you suggested new v7.90.008 ALPHA installation on my C2:
    ...
    It implies that Blocksize is 1MB for every file.

    Not really sure what's going on there but that might suggest a client (ie. Windows) issue as the "Size on disk" is definitely not what it is telling you.

    This is Windows 7 SP1 connecting to the LE Samba share of an RPi3:

    A NUC running LE Generic behaves the same as the RPi3.

    Quote from infinity85


    Is there any explanation for this?


    Another Redmond bug, perhaps?

    @milhouse
    Thanks a lot to pointing to these facts! I feel reassured again somehow ;). Weird... then this might really be some new weird bug in Windows 10. I know that it is showing these big blocksizes since at least may, where I discovered it because I wanted to know how large a backup of my library could get. And I almost couldn't believe this discrepancy.

    Is the SMB.conf of v7.90.008 ALPHA for C2 the same as on your RPi and NUC then?


    EDIT

    have a look at this: Linux Performance - SambaWiki
    The third paragraph (from this URLs position) says that 1MB is default.

    Quote


    ...if the allocation size is set to 1MB (the default in Samba) then...


    I don't really understand that. Do the other systems use a different samba version then, so that your systems show the correct size?

    Edited once, last by infinity85 (November 18, 2016 at 12:29 AM).

    Quote from infinity85


    Is the SMB.conf of v7.90.008 ALPHA for C2 the same as on your RPi and NUC then?

    Yes, I'm using the default samba config - I haven't created a custom samba config in /storage/.config on either the RPi2 or NUC clients. Nor have I added any registry entries on the Windows 7 PC.

    Quote from infinity85


    have a look at this: Linux Performance - SambaWiki
    The third paragraph (from this URLs position) says that 1MB is default.


    I don't really understand that. Do the other systems use a different samba version then, so that your systems show the correct size?

    It appears that Windows 10 respects (or is confused by) the 1MB "allocation size" resulting in these wildly inaccurate "size on disk" calculations.

    This post suggests the problem was introduced with Windows 8.x.

    Try adding the following global entry to /storage/.config/samba.conf (copy the default from /etc/samba/smb.conf):

    Code
    allocation roundup size = 4096

    Restart your Odroid, and check if the size on disk reported by Windows is now a little more reasonable.

    This setting may need to be added to the default config in future.

    From smb.conf manual:

    Quote


    allocation roundup size (S)

    This parameter allows an administrator to tune the allocation size reported to Windows clients. The default size of 1Mb generally results in improved Windows client performance. However, rounding the allocation size may cause difficulties for some applications, e.g. MS Visual Studio. If the MS Visual Studio compiler starts to crash with an internal error, set this parameter to zero for this share.

    The integer parameter specifies the roundup size in bytes.

    Default: allocation roundup size = 1048576

    Example: allocation roundup size = 0 # (to disable roundups)

    Edited once, last by milhouse (November 18, 2016 at 7:17 AM).

    Here we are...

    Tried allocation roundup size = 4096
    Windows explorer showed me then 4KB instead of 1MB (for files below 4KB size)

    Then tried allocation roundup size = 0
    Windows explorer showed me then 1KB instead of 1MB (for files below 1KB size)

    So I guess for a default smb.conf in the future allocation roundup size = 0 would be the best.


    After reading your links and googling, my conclusion is that the actual "issue" is Linux default allocation roundup size = 1048576, whereas Windows 7 simply ignores it (I would call this a bug) and Windows >8.1 respects it (unfortunately).

    Perhaps you have the time to test allocation roundup size = 0 and allocation roundup size = 4096 with your Windows 7 machine? Who knows... may be the value =0 always results in 1KB (as a bug) or may be your system will simply ignore either setting. Just asking because I would like to know whether allocation roundup size = 0 corresponds to the filesystem (which is on my Odroid 1024 Byte as you stated) and on your Raspberry 4096 bytes then. If allocation roundup size = 0 shows 4096 on your raspberry (instead of 1024 like on my C2), then this setting is confirmed to be the best default solution :)

    And while we're talking about samba... noticed that setting samba-passwords containing special characters like ! or * in LibreELEC are somehow not working if trying to connect with windows explorer. Don't know whether this might be a windows bug, or a LibreELEC bug.

    Thanks milhouse, simpsons have always been one of the best out there :)

    Edited once, last by infinity85 (November 18, 2016 at 1:50 PM).

    Using LibreELEC (community) Version: 7.90.009 - Virtual.x86_64-7.90.009:

    Code
    samba.conf allocation roundup size = 0
Size on disk
Windows  7: 1.00 KB (1.024 bytes)
Windows 10: 1.00 KB (1,024 bytes)


samba.conf allocation roundup size = 4096
Size on disk
Windows  7: 1.00 KB (1.024 bytes)
Windows 10: 4.00 KB (4,096 bytes)

    thanks vpeter :)

    Then it's more of a bug in Windows 7 and a rather bad linux default setting of 1MB.

    I'd say the actual Issue is solved then ;)


    EDIT
    @vpeter
    I guess your VM has a filesystem Blocksize of 1024 bytes, right?
    Just asking, because my Windows 10 Laptop has 4096 bytes (If I believe the "size on disk" propertiestab of a <1KB file).

    Edited once, last by infinity85 (November 18, 2016 at 2:29 PM).

    Linux stat commands show: IO Block: 1024
    And I don't think we have somethnig different for Virtual builds.

    A little more technical view of data captured with Wireshark:

    samba.conf allocation roundup size = 0
    Allocation size: 1024
    iW30Oz.jpg

    samba.conf allocation roundup size = 4096
    Allocation size: 4096
    nKVYiG.jpg