Would it make sense to make the firmware files a separate download, as only those who use wireless Red or hostAPD need them? Tom > On May 11, 2018, at 4:28 PM, Michael Tremer wrote: > > Hey Matthias, > > are you still working on this? > > I am asking because I just noticed that the new next build is about 50MB larger > than the old one due to the larger kernel and loads more of firmware files. > > Compressing it better would of course help to save space on disk. > > Best, > -Michael > >> On Sun, 2018-02-11 at 19:57 +0000, Michael Tremer wrote: >> Hi, >> >>> On Sun, 2018-02-11 at 15:35 +0000, Matthias Fischer wrote: >>> Hi, >>> >>> On 06.02.2018 02:05, Michael Tremer wrote: >>>>> ... >>>>> ((XZ_MEM=(($HOST_MEM)/10)*7)) && XZ_MEM=${XZ_MEM}MiB >>>> >>>> You can write this even short: >>>> >>>> XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB" >>> >>> Oneliner! Yes. I was looking for something like that... ;-) >>> >>>> I swapped the order of the multiplication and division so that you would >>>> have >>>> a >>>> more exact result. >>>> >>>> For example if you system has 1024 MB or RAM, you divide this by 10 (102) >>>> and >>>> then multiply by 7 which gives you 714. The other order is 1024 * 7 = 7167 >>>> and >>>> then divided by 10 is 716 which is closer to the actual result of 716.8. >>>> Shouldn't matter here but just in general this is better :) >>> >>> Thanks, looks better - noted for the next time! >>> >>> Now it looks like this ('make.sh'): >>> >>> ... >>> # Host memory is ok, calculating XZ memory >>> XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB" >>> XZ_OPT="--threads=0 -8 $XZ_MEM" >>> echo XZ-Memory: $XZ_MEM >>> echo XZ-Options: $XZ_OPT >>> echo "XZ memory size is OK (must be at least 700MiB), starting build..." >>> ... >>> ["echoes" for debugging!] >>> >>> This is working. '$XZ_OPT' now gives me: --threads=0 -8 5450MiB >> >> Will there be some checks in make.sh now that check if at least 1024MB >> of memory are available for build? I think there should be a warning if >> not enough memory is available, but we should still try to build. >> >>> But. ;-) >>> >>>> You can put this probably next under "HOST_MEM=$(system_memory)" and then >>>> export >>>> this to the internal shell by adding the variable to lfsmake2 for the disk >>>> image >>>> and ipfiredist for the packages. >>> >>> Sorry, I must confess that I don't know exactly how to do this. As I wrote: >>> "How >>> do >>> I pass the 'XZ_MEM'-variable to 'cdrom' and 'Config'?" Or in this case, >>> XZ_OPT!? >>> >>>> Then in lfs/cdrom and lfs/Config for the dist >>>> target, you just need to add the parameters to the command line. >>> >>> In 'Config' I changed: >>> >>> ... >>> cd /install/packages/package/tmp/ && XZ_OPT=-T0 tar -c -p --numeric-owner -J >>> -f >>> /install/packages/package/files.tar.xz * >>> ... >>> >>> To: >>> ... >>> cd /install/packages/package/tmp/ && $XZ_OPT tar -c -p --numeric-owner -J -f >>> /install/packages/package/files.tar.xz * >>> ... >>> >>>> Maybe it is a good idea to have a variable XZ_OPT that adds everything >>>> together >>>> and is then used in both places. >>> >>> I added the other xz-options and created XZ_OPT as shown above, but in >>> 'cdrom' >>> there >>> is an 'export XZ_OPT = ...'-line and I don't know how to handle these line. >>> Is >>> it still necessary? >> >> No, that is no longer necessary and can be removed. We should just add >> it to the tar command just like it is done in lfs/Config. >> >>> Do I have to 'export' the XZ_OPT'-variable in 'make.sh'? >> >> No, because the chroot environment will throw away all environment >> variables from the host system and then set them again. That's why we >> have this enterchroot() function that does this with the env command. >> >>> And how do I "add" this variable to 'lfsmake2' and 'ipfiredist'(-function)? >> >> Just have a look at the functions. There should be loads of examples >> where we pass other variables and you just need to add this one, too. >> >>> Sorry if these questions sound simple, but I'm guessing a bit too much while >>> trying >>> to get a grip on this. >> >> This is absolutely fine to sort this out first and then send the patch >> :) That's what this list is for. >> >> Best, >> -Michael >> >>> >>> Best, >>> Matthias >>> >>>> >>>> Best, >>>> -Michael >>>> >>>>> ... >>>>> >>>>> Looks better. >>>>> >>>>> Best, >>>>> Matthias >>>>> >>>>>> On 03.02.2018 20:44, Matthias Fischer wrote: >>>>>> Hi, >>>>>> >>>>>>> On 30.01.2018 21:03, Michael Tremer wrote: >>>>>>> Hi, >>>>>>> >>>>>>> ... >>>>>>>> >>>>>>>> I'll consider how I would do this and send it in as soon as I get >>>>>>>> a >>>>>>>> grip >>>>>>>> on it. >>>>>>> >>>>>>> Well, I added some functions already somewhere. I suppose you just >>>>>>> need >>>>>>> to >>>>>>> do >>>>>>> the maths to set that to X% of the host system which is just a >>>>>>> simple >>>>>>> multiplication of the output of ${HOST_MEM} in make.sh. >>>>>> >>>>>> Ok, I take my heart in my hand. Here are my first results - but >>>>>> remember, >>>>>> I'm not familiar with this. I built a short test script which looks as >>>>>> if >>>>>> it >>>>>> is working as wanted, but... >>>>>> >>>>>>> And that needs to be passed to the lfs scripts that are supposed to >>>>>>> make >>>>>>> the >>>>>>> images. >>>>>> >>>>>> ...this is a problem I'm struggling with right now: >>>>>> How do I pass the 'XZ_MEM'-variable to 'cdrom' and 'Config'? >>>>>> >>>>>> I think this can be done using 'export XZ_MEM' or something else but >>>>>> I'm >>>>>> not >>>>>> sure. >>>>>> >>>>>> This is the script I'm testing with: >>>>>> >>>>>> ***SNIP*** >>>>>> #!/bin/bash >>>>>> >>>>>> system_memory() { >>>>>> local key val unit >>>>>> >>>>>> while read -r key val unit; do >>>>>> case "${key}" in >>>>>> MemTotal:*) >>>>>> # Convert to MB >>>>>> echo "$(( ${val} / 1024 ))" >>>>>> break >>>>>> ;; >>>>>> esac >>>>>> done < /proc/meminfo >>>>>> } >>>>>> >>>>>> HOST_MEM=$(system_memory) >>>>>> >>>>>> # Checking host memory >>>>>> if [ $HOST_MEM -ge 1024 ]; then >>>>>> echo Host-Memory: $HOST_MEM'MiB' >>>>>> echo "Host memory is OK (must be at least 1024MiB), starting >>>>>> build." >>>>>> else >>>>>> echo Host-Memory: $HOST_MEM'MiB' >>>>>> echo "Host memory too low (less than 1024MiB), building >>>>>> stopped." >>>>>> exit 1 >>>>>> fi >>>>>> >>>>>> # Host memory is ok, calculating needed XZ memory to about 70% of host >>>>>> memory >>>>>> ((XZ_MEM=(($HOST_MEM)/10)*7)) >>>>>> XZ_MEM=$XZ_MEM'Mib' >>>>>> echo XZ-Memory: $XZ_MEM >>>>>> echo "XZ memory size is OK (must be at least 700MiB), starting >>>>>> build." >>>>>> ***SNAP*** >>>>>> >>>>>> The 'echoes' are mostly for debugging. >>>>>> >>>>>> Calculation looks ok, 'MiB' must added without blanks (XZ man page: >>>>>> "In >>>>>> most >>>>>> places where an integer argument is expected, an optional suffix is >>>>>> supported >>>>>> to easily indicate large integers. There must be no space between the >>>>>> integer >>>>>> and the suffix.") >>>>>> >>>>>>> Please send questions where ever you need help. There is no reason >>>>>>> to do >>>>>>> this >>>>>>> alone. >>>>>> >>>>>> Please see above: >>>>>> >>>>>> 1. Is this code what you where thinking of? I would leave the other >>>>>> parameters >>>>>> as they are (--threads=0 -8). >>>>>> >>>>>> 2. Where should I add such code in the first place? I'd like to start >>>>>> with >>>>>> 'make.sh', so building stops as soon as possible if there is not >>>>>> enough >>>>>> memory. Where's >>>>>> the best place? Right after function 'system_memory() {...}'? >>>>>> >>>>>> 3. How to I pass contents of 'XZ_MEM' to 'cdrom' and 'Config'? >>>>>> I consider something like: >>>>>> ... >>>>>> export XZ_OPT = --threads=0 -8 --memory=$XZ_MEM >>>>>> ... >>>>>> >>>>>> Best, >>>>>> Matthias >>>>>> >>>>>>> Best, >>>>>>> -Michael >>>>>>> >>>>>>>> >>>>>>>> Best, >>>>>>>> Matthias >>>>>>>>> Best, >>>>>>>>> -Michael >>>>>>>>> >>>>>>>>>> On Wed, 2017-11-22 at 16:37 +0000, Michael Tremer wrote: >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>>> On Tue, 2017-11-21 at 18:35 +0100, Matthias Fischer wrote: >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>>> On 21.11.2017 15:42, Michael Tremer wrote: >>>>>>>>>>>> Hello, >>>>>>>>>>>> >>>>>>>>>>>> any progress on this? >>>>>>>>>>> >>>>>>>>>>> Not much, because I've been struggling with a flu for the >>>>>>>>>>> past >>>>>>>>>>> two >>>>>>>>>>> weeks, and its not over yet, but: >>>>>>>>>>> >>>>>>>>>>> In the meantime I ran several builds with different >>>>>>>>>>> parameters. >>>>>>>>>>> >>>>>>>>>>> IMHO we could use: >>>>>>>>>>> >>>>>>>>>>> 'export XZ_OPT = --threads=0 -8 --memory=700MiB' >>>>>>>>>>> >>>>>>>>>>> in 'lfs/cdrom' >>>>>>>>>>> >>>>>>>>>>> and >>>>>>>>>>> >>>>>>>>>>> cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8 -- >>>>>>>>>>> memory=700MiB" tar >>>>>>>>>>> -c >>>>>>>>>>> -p >>>>>>>>>>> --numeric-owner -J -f /install/packages/package/files.tar.xz >>>>>>>>>>> * >>>>>>>>>>> >>>>>>>>>>> in 'lfs/Config'. >>>>>>>>>>> >>>>>>>>>>> Using '9' or '--extreme'-parameter brought no further >>>>>>>>>>> improvements, >>>>>>>>>>> on the other hand, build time and RAM consumption increased >>>>>>>>>>> strongly. With '-8', minimal RAM consumption was about 680 >>>>>>>>>>> MiB, >>>>>>>>>>> using >>>>>>>>>>> '700 Mib' was always safe. So this is the minimum I would >>>>>>>>>>> recommend. >>>>>>>>>> >>>>>>>>>> Yes, I observed the same thing. -9 compresses a little bit >>>>>>>>>> better >>>>>>>>>> and >>>>>>>>>> extreme >>>>>>>>>> doesn't change a bit. >>>>>>>>>> >>>>>>>>>> -8 seems to be optimal since it won't require too much memory >>>>>>>>>> to >>>>>>>>>> decompress >>>>>>>>>> either. So let's go with that. >>>>>>>>>> >>>>>>>>>>> Results: >>>>>>>>>>> >>>>>>>>>>> (Buildtime: 5 hours 20 minutes 59 seconds) >>>>>>>>>>> >>>>>>>>>>> 255866764 ipfire-2.19.1gb-ext4.i586-full-core117.img.gz >>>>>>>>>>> 252262006 ipfire-2.19.1gb-ext4-scon.i586-full- >>>>>>>>>>> core117.img.gz >>>>>>>>>>> 175112192 ipfire-2.19.i586-full-core117.iso >>>>>>>>>>> >>>>>>>>>>>> -Michael >>>>>>>>>>>> >>>>>>>>>>>>> On Tue, 2017-11-07 at 22:47 +0000, Michael Tremer wrote: >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, 2017-11-07 at 19:41 +0100, Matthias Fischer >>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 07.11.2017 16:05, Michael Tremer wrote: >>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> maybe this function helps: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdif >>>>>>>>>>>>>>> f;h= >>>>>>>>>>>>>>> 51 >>>>>>>>>>>>>>> 90eea2 >>>>>>>>>>>>>>> 4f98 >>>>>>>>>>>>>>> 22 >>>>>>>>>>>>>>> a63d >>>>>>>>>>>>>>> c5d06d214b48f973b14f29 >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks. I'll take a look... >>>>>>>>>>> >>>>>>>>>>> This would help, indeed, but how to implement(!?), and... >>>>>>>>>>> (see >>>>>>>>>>> below) >>>>>>>>>> >>>>>>>>>> Just run it somewhere in the script, figure out what is about >>>>>>>>>> 80% >>>>>>>>>> of >>>>>>>>>> that >>>>>>>>>> and >>>>>>>>>> if >>>>>>>>>> that is smaller than 800 MB, we should just fail. >>>>>>>>>> >>>>>>>>>> It would be good to have a function that determines the xz >>>>>>>>>> parameters and >>>>>>>>>> we >>>>>>>>>> use >>>>>>>>>> that where ever we call xz. We don't want any duplicated code. >>>>>>>>>> >>>>>>>>>>>>>>> On Tue, 2017-11-07 at 10:46 +0000, Michael Tremer >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> On Mon, 2017-11-06 at 22:09 +0100, Matthias >>>>>>>>>>>>>>>> Fischer >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> ... >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I forgot: Tuning the XZ-parameters... ;-) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Good that you are bringing this up. So I think >>>>>>>>>>>>>>>> what we >>>>>>>>>>>>>>>> should >>>>>>>>>>>>>>>> do >>>>>>>>>>>>>>>> here is >>>>>>>>>>>>>>>> bringing back XZ compression with -8 so that we do >>>>>>>>>>>>>>>> not >>>>>>>>>>>>>>>> waste >>>>>>>>>>>>>>>> too >>>>>>>>>>>>>>>> much >>>>>>>>>>>>>>>> space >>>>>>>>>>>>>>>> when >>>>>>>>>>>>>>>> extracting the image again. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Right now I'm testing how much RAM I need. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Current 'next'-build is running with: >>>>>>>>>>>>>> >>>>>>>>>>>>>> 'lfs/cdrom': >>>>>>>>>>>>>> ... >>>>>>>>>>>>>> export XZ_OPT = --threads=0 -8 --memory=500MiB >>>>>>>>>>>>>> ... >>>>>>>>>>>>>> >>>>>>>>>>>>>> 'lfs/Config': >>>>>>>>>>>>>> ... >>>>>>>>>>>>>> cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8" >>>>>>>>>>>>>> ... >>>>>>>>>>>>>> ... >>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm waiting for the results. Takes a while. >>>>>>>>>>>>> >>>>>>>>>>>>> You can just run it like this and it will tell you: >>>>>>>>>>>>> >>>>>>>>>>>>> [ms(a)hughes ~]$ xz -vv8 -T0 >>>>>>>>>>>>> xz: Filter chain: -- >>>>>>>>>>>>> lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=b >>>>>>>>>>>>> t4,d >>>>>>>>>>>>> ep >>>>>>>>>>>>> th=0 >>>>>>>>>>>>> xz: Using up to 4 threads. >>>>>>>>>>>>> xz: 2,629 MiB of memory is required. The limiter is >>>>>>>>>>>>> disabled. >>>>>>>>>>>>> xz: Decompression will need 33 MiB of memory. >>>>>>>>>>>>> xz: Compressed data cannot be written to a terminal >>>>>>>>>>>>> xz: Try `xz --help' for more information. >>>>>>>>>>>>> >>>>>>>>>>>>>>>> To not run into memory limits we should detect how >>>>>>>>>>>>>>>> much >>>>>>>>>>>>>>>> memory >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>> build >>>>>>>>>>>>>>>> host >>>>>>>>>>>>>>>> has and set the memory limit to maybe 80% of that. >>>>>>>>>>>>>>>> xz >>>>>>>>>>>>>>>> will >>>>>>>>>>>>>>>> then >>>>>>>>>>>>>>>> automatically >>>>>>>>>>>>>>>> scale down to a number of parallel processes that >>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>> fit >>>>>>>>>>>>>>>> into >>>>>>>>>>>>>>>> memory. >>>>>>>>>>> >>>>>>>>>>> This is something I couldn't get to work. First, I couldn't >>>>>>>>>>> specify not >>>>>>>>>>> less >>>>>>>>>>> than '700 MiB'. If I specify '600 MiB', building stops and >>>>>>>>>>> tells >>>>>>>>>>> me it >>>>>>>>>>> needs >>>>>>>>>>> at least about 680 MiB. If I use '70%', building stops. And >>>>>>>>>>> I >>>>>>>>>>> never saw >>>>>>>>>>> that >>>>>>>>>>> 'xz' "automatically scaled down". It just stopped everytime. >>>>>>>>>> >>>>>>>>>> Yes, ~700M is the minimum. >>>>>>>>>> >>>>>>>>>> It will just reduce the number of threads again to make sure >>>>>>>>>> everything >>>>>>>>>> fits >>>>>>>>>> into memory. >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>> I'm a bit puzzled about this: I tried using ' >>>>>>>>>>>>>> --memory=70%' >>>>>>>>>>>>>> as >>>>>>>>>>>>>> mentioned >>>>>>>>>>>>>> in the xz man pages, but is seems to have no effect. >>>>>>>>>>>>>> It >>>>>>>>>>>>>> looks as >>>>>>>>>>>>>> if >>>>>>>>>>>>>> the >>>>>>>>>>>>>> percent parameter isn't recognized and ignored. It >>>>>>>>>>>>>> always >>>>>>>>>>>>>> ends up >>>>>>>>>>>>>> in >>>>>>>>>>>>>> "cannot allocate memory". Any hints on this? Do both >>>>>>>>>>>>>> 'lfs/cdrom' >>>>>>>>>>>>>> AND >>>>>>>>>>>>>> 'lfs/Config' need the '--memory'-parameter? >>>>>>>>>>> >>>>>>>>>>> This was the next problem: 'percent'-parameter does not work >>>>>>>>>>> as >>>>>>>>>>> expected. >>>>>>>>>> >>>>>>>>>> No, don't use it. Just determine yourself what it is. >>>>>>>>>> >>>>>>>>>> I think we could even try to set 100% of the memory, because >>>>>>>>>> as >>>>>>>>>> far >>>>>>>>>> as I >>>>>>>>>> can >>>>>>>>>> see >>>>>>>>>> this is worst case and xz won't use everything that we allow. >>>>>>>>>> >>>>>>>>>> If that does't work, let's scale down to 90%, or 80%, etc... >>>>>>>>>> >>>>>>>>>>>>> Yes, but it would be good to have one place where the >>>>>>>>>>>>> parameters are >>>>>>>>>>>>> being generated and then we just use them from a >>>>>>>>>>>>> variable. >>>>>>>>>>> >>>>>>>>>>> Probably beyond my capabilities for the moment. Any >>>>>>>>>>> thoughts? >>>>>>>>>>> >>>>>>>>>>>>>>>> ... >>>>>>>>>>>>>>>> We will at least need about half a GB which should >>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>> sensible >>>>>>>>>>>>>>>> requirement >>>>>>>>>>>>>>>> for a build system any ways. >>>>>>>>>>> >>>>>>>>>>> I'm afraid '500 MiB' are not enough... >>>>>>>>>> >>>>>>>>>> No, so we will now make 1G the minimum for a build. >>>>>>>>>> >>>>>>>>>> I tried to build the OS on an ARM board with only 512MB of RAM >>>>>>>>>> the >>>>>>>>>> other >>>>>>>>>> day >>>>>>>>>> and >>>>>>>>>> even with a lot of swap it doesn't work. So we are past this >>>>>>>>>> any >>>>>>>>>> way. >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>>> Agreed. Because of this I'm testing with '500Mib'. >>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Will you work on this bringing it into the build >>>>>>>>>>>>>>>> system?! >>>>>>>>>>>>>> >>>>>>>>>>>>>> "I'll do my very best!" :-)) >>>>>>>>>>>>> >>>>>>>>>>>>> Very well Miss Sophie... >>>>>>>>>>>>> ... >>>>>>>>>>> >>>>>>>>>>> Best™, >>>>>>>>>>> Matthias >>>>>>>>>> >>>>>>>>>> -Michael