Hey,
technically that is an idea, but I don't think it makes sense to do this.
People would need the firmware if they are connecting to the Internet using WiFi and then it gets really complicated to figure out what people are most likely to use. Same goes for some Ethernet adapters.
We don't really have a limit about the image size. But of course a smaller image saves disk space and bandwidth to download. It's just a bit about efficiency :)
Best, -Michael
On Fri, 2018-05-11 at 20:35 -0400, Tom Rymes wrote:
Would it make sense to make the firmware files a separate download, as only those who use wireless Red or hostAPD need them?
Tom
On May 11, 2018, at 4:28 PM, Michael Tremer michael.tremer@ipfire.org wrote:
Hey Matthias,
are you still working on this?
I am asking because I just noticed that the new next build is about 50MB larger than the old one due to the larger kernel and loads more of firmware files.
Compressing it better would of course help to save space on disk.
Best, -Michael
On Sun, 2018-02-11 at 19:57 +0000, Michael Tremer wrote: Hi,
On Sun, 2018-02-11 at 15:35 +0000, Matthias Fischer wrote: Hi,
On 06.02.2018 02:05, Michael Tremer wrote:
... ((XZ_MEM=(($HOST_MEM)/10)*7)) && XZ_MEM=${XZ_MEM}MiB
You can write this even short:
XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"
Oneliner! Yes. I was looking for something like that... ;-)
I swapped the order of the multiplication and division so that you would have a more exact result.
For example if you system has 1024 MB or RAM, you divide this by 10 (102) and then multiply by 7 which gives you 714. The other order is 1024 * 7 = 7167 and then divided by 10 is 716 which is closer to the actual result of 716.8. Shouldn't matter here but just in general this is better :)
Thanks, looks better - noted for the next time!
Now it looks like this ('make.sh'):
... # Host memory is ok, calculating XZ memory XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB" XZ_OPT="--threads=0 -8 $XZ_MEM" echo XZ-Memory: $XZ_MEM echo XZ-Options: $XZ_OPT echo "XZ memory size is OK (must be at least 700MiB), starting build..." ... ["echoes" for debugging!]
This is working. '$XZ_OPT' now gives me: --threads=0 -8 5450MiB
Will there be some checks in make.sh now that check if at least 1024MB of memory are available for build? I think there should be a warning if not enough memory is available, but we should still try to build.
But. ;-)
You can put this probably next under "HOST_MEM=$(system_memory)" and then export this to the internal shell by adding the variable to lfsmake2 for the disk image and ipfiredist for the packages.
Sorry, I must confess that I don't know exactly how to do this. As I wrote: "How do I pass the 'XZ_MEM'-variable to 'cdrom' and 'Config'?" Or in this case, XZ_OPT!?
Then in lfs/cdrom and lfs/Config for the dist target, you just need to add the parameters to the command line.
In 'Config' I changed:
... cd /install/packages/package/tmp/ && XZ_OPT=-T0 tar -c -p --numeric- owner -J -f /install/packages/package/files.tar.xz * ...
To: ... cd /install/packages/package/tmp/ && $XZ_OPT tar -c -p --numeric-owner -J -f /install/packages/package/files.tar.xz * ...
Maybe it is a good idea to have a variable XZ_OPT that adds everything together and is then used in both places.
I added the other xz-options and created XZ_OPT as shown above, but in 'cdrom' there is an 'export XZ_OPT = ...'-line and I don't know how to handle these line. Is it still necessary?
No, that is no longer necessary and can be removed. We should just add it to the tar command just like it is done in lfs/Config.
Do I have to 'export' the XZ_OPT'-variable in 'make.sh'?
No, because the chroot environment will throw away all environment variables from the host system and then set them again. That's why we have this enterchroot() function that does this with the env command.
And how do I "add" this variable to 'lfsmake2' and 'ipfiredist'(- function)?
Just have a look at the functions. There should be loads of examples where we pass other variables and you just need to add this one, too.
Sorry if these questions sound simple, but I'm guessing a bit too much while trying to get a grip on this.
This is absolutely fine to sort this out first and then send the patch :) That's what this list is for.
Best, -Michael
Best, Matthias
Best, -Michael
...
Looks better.
Best, Matthias
> On 03.02.2018 20:44, Matthias Fischer wrote: > Hi, > > > On 30.01.2018 21:03, Michael Tremer wrote: > > Hi, > > > > ... > > > > > > I'll consider how I would do this and send it in as soon as I > > > get > > > a > > > grip > > > on it. > > > > Well, I added some functions already somewhere. I suppose you > > just > > need > > to > > do > > the maths to set that to X% of the host system which is just a > > simple > > multiplication of the output of ${HOST_MEM} in make.sh. > > Ok, I take my heart in my hand. Here are my first results - but > remember, > I'm not familiar with this. I built a short test script which > looks as > if > it > is working as wanted, but... > > > And that needs to be passed to the lfs scripts that are supposed > > to > > make > > the > > images. > > ...this is a problem I'm struggling with right now: > How do I pass the 'XZ_MEM'-variable to 'cdrom' and 'Config'? > > I think this can be done using 'export XZ_MEM' or something else > but > I'm > not > sure. > > This is the script I'm testing with: > > ***SNIP*** > #!/bin/bash > > system_memory() { > local key val unit > > while read -r key val unit; do > case "${key}" in > MemTotal:*) > # Convert to MB > echo "$(( ${val} / 1024 ))" > break > ;; > esac > done < /proc/meminfo > } > > HOST_MEM=$(system_memory) > > # Checking host memory > if [ $HOST_MEM -ge 1024 ]; then > echo Host-Memory: $HOST_MEM'MiB' > echo "Host memory is OK (must be at least 1024MiB), starting > build." > else > echo Host-Memory: $HOST_MEM'MiB' > echo "Host memory too low (less than 1024MiB), building > stopped." > exit 1 > fi > > # Host memory is ok, calculating needed XZ memory to about 70% of > host > memory > ((XZ_MEM=(($HOST_MEM)/10)*7)) > XZ_MEM=$XZ_MEM'Mib' > echo XZ-Memory: $XZ_MEM > echo "XZ memory size is OK (must be at least 700MiB), starting > build." > ***SNAP*** > > The 'echoes' are mostly for debugging. > > Calculation looks ok, 'MiB' must added without blanks (XZ man > page: > "In > most > places where an integer argument is expected, an optional suffix > is > supported > to easily indicate large integers. There must be no space between > the > integer > and the suffix.") > > > Please send questions where ever you need help. There is no > > reason > > to do > > this > > alone. > > Please see above: > > 1. Is this code what you where thinking of? I would leave the > other > parameters > as they are (--threads=0 -8). > > 2. Where should I add such code in the first place? I'd like to > start > with > 'make.sh', so building stops as soon as possible if there is not > enough > memory. Where's > the best place? Right after function 'system_memory() {...}'? > > 3. How to I pass contents of 'XZ_MEM' to 'cdrom' and 'Config'? > I consider something like: > ... > export XZ_OPT = --threads=0 -8 --memory=$XZ_MEM > ... > > Best, > Matthias > > > Best, > > -Michael > > > > > > > > Best, > > > Matthias > > > > Best, > > > > -Michael > > > > > > > > > On Wed, 2017-11-22 at 16:37 +0000, Michael Tremer wrote: > > > > > Hi, > > > > > > > > > > > On Tue, 2017-11-21 at 18:35 +0100, Matthias Fischer > > > > > > wrote: > > > > > > Hi, > > > > > > > > > > > > > On 21.11.2017 15:42, Michael Tremer wrote: > > > > > > > Hello, > > > > > > > > > > > > > > any progress on this? > > > > > > > > > > > > Not much, because I've been struggling with a flu for > > > > > > the > > > > > > past > > > > > > two > > > > > > weeks, and its not over yet, but: > > > > > > > > > > > > In the meantime I ran several builds with different > > > > > > parameters. > > > > > > > > > > > > IMHO we could use: > > > > > > > > > > > > 'export XZ_OPT = --threads=0 -8 --memory=700MiB' > > > > > > > > > > > > in 'lfs/cdrom' > > > > > > > > > > > > and > > > > > > > > > > > > cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8 -- > > > > > > memory=700MiB" tar > > > > > > -c > > > > > > -p > > > > > > --numeric-owner -J -f > > > > > > /install/packages/package/files.tar.xz > > > > > > * > > > > > > > > > > > > in 'lfs/Config'. > > > > > > > > > > > > Using '9' or '--extreme'-parameter brought no further > > > > > > improvements, > > > > > > on the other hand, build time and RAM consumption > > > > > > increased > > > > > > strongly. With '-8', minimal RAM consumption was about > > > > > > 680 > > > > > > MiB, > > > > > > using > > > > > > '700 Mib' was always safe. So this is the minimum I > > > > > > would > > > > > > recommend. > > > > > > > > > > Yes, I observed the same thing. -9 compresses a little bit > > > > > better > > > > > and > > > > > extreme > > > > > doesn't change a bit. > > > > > > > > > > -8 seems to be optimal since it won't require too much > > > > > memory > > > > > to > > > > > decompress > > > > > either. So let's go with that. > > > > > > > > > > > Results: > > > > > > > > > > > > (Buildtime: 5 hours 20 minutes 59 seconds) > > > > > > > > > > > > 255866764 ipfire-2.19.1gb-ext4.i586-full-core117.img.gz > > > > > > 252262006 ipfire-2.19.1gb-ext4-scon.i586-full- > > > > > > core117.img.gz > > > > > > 175112192 ipfire-2.19.i586-full-core117.iso > > > > > > > > > > > > > -Michael > > > > > > > > > > > > > > > On Tue, 2017-11-07 at 22:47 +0000, Michael Tremer > > > > > > > > wrote: > > > > > > > > Hi, > > > > > > > > > > > > > > > > On Tue, 2017-11-07 at 19:41 +0100, Matthias Fischer > > > > > > > > wrote: > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > On 07.11.2017 16:05, Michael Tremer wrote: > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > maybe this function helps: > > > > > > > > > > > > > > > > > > > > https://git.ipfire.org/?p=ipfire-2.x.git;a=commi > > > > > > > > > > tdif > > > > > > > > > > f;h= > > > > > > > > > > 51 > > > > > > > > > > 90eea2 > > > > > > > > > > 4f98 > > > > > > > > > > 22 > > > > > > > > > > a63d > > > > > > > > > > c5d06d214b48f973b14f29 > > > > > > > > > > > > > > > > > > Thanks. I'll take a look... > > > > > > > > > > > > This would help, indeed, but how to implement(!?), > > > > > > and... > > > > > > (see > > > > > > below) > > > > > > > > > > Just run it somewhere in the script, figure out what is > > > > > about > > > > > 80% > > > > > of > > > > > that > > > > > and > > > > > if > > > > > that is smaller than 800 MB, we should just fail. > > > > > > > > > > It would be good to have a function that determines the xz > > > > > parameters and > > > > > we > > > > > use > > > > > that where ever we call xz. We don't want any duplicated > > > > > code. > > > > > > > > > > > > > > > On Tue, 2017-11-07 at 10:46 +0000, Michael > > > > > > > > > > Tremer > > > > > > > > > > wrote: > > > > > > > > > > > On Mon, 2017-11-06 at 22:09 +0100, Matthias > > > > > > > > > > > Fischer > > > > > > > > > > > wrote: > > > > > > > > > > > ... > > > > > > > > > > > > > > > > > > > > > > > > I forgot: Tuning the XZ-parameters... ;-) > > > > > > > > > > > > > > > > > > > > > > Good that you are bringing this up. So I think > > > > > > > > > > > what we > > > > > > > > > > > should > > > > > > > > > > > do > > > > > > > > > > > here is > > > > > > > > > > > bringing back XZ compression with -8 so that > > > > > > > > > > > we do > > > > > > > > > > > not > > > > > > > > > > > waste > > > > > > > > > > > too > > > > > > > > > > > much > > > > > > > > > > > space > > > > > > > > > > > when > > > > > > > > > > > extracting the image again. > > > > > > > > > > > > > > > > > > Right now I'm testing how much RAM I need. > > > > > > > > > > > > > > > > > > Current 'next'-build is running with: > > > > > > > > > > > > > > > > > > 'lfs/cdrom': > > > > > > > > > ... > > > > > > > > > export XZ_OPT = --threads=0 -8 --memory=500MiB > > > > > > > > > ... > > > > > > > > > > > > > > > > > > 'lfs/Config': > > > > > > > > > ... > > > > > > > > > cd /install/packages/package/tmp/ && XZ_OPT="-T0 > > > > > > > > > -8" > > > > > > > > > ... > > > > > > > > > ... > > > > > > > > > > > > > > > > > > I'm waiting for the results. Takes a while. > > > > > > > > > > > > > > > > You can just run it like this and it will tell you: > > > > > > > > > > > > > > > > [ms@hughes ~]$ xz -vv8 -T0 > > > > > > > > xz: Filter chain: -- > > > > > > > > lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64, > > > > > > > > mf=b > > > > > > > > t4,d > > > > > > > > ep > > > > > > > > th=0 > > > > > > > > xz: Using up to 4 threads. > > > > > > > > xz: 2,629 MiB of memory is required. The limiter is > > > > > > > > disabled. > > > > > > > > xz: Decompression will need 33 MiB of memory. > > > > > > > > xz: Compressed data cannot be written to a terminal > > > > > > > > xz: Try `xz --help' for more information. > > > > > > > > > > > > > > > > > > > To not run into memory limits we should detect > > > > > > > > > > > how > > > > > > > > > > > much > > > > > > > > > > > memory > > > > > > > > > > > the > > > > > > > > > > > build > > > > > > > > > > > host > > > > > > > > > > > has and set the memory limit to maybe 80% of > > > > > > > > > > > that. > > > > > > > > > > > xz > > > > > > > > > > > will > > > > > > > > > > > then > > > > > > > > > > > automatically > > > > > > > > > > > scale down to a number of parallel processes > > > > > > > > > > > that > > > > > > > > > > > it > > > > > > > > > > > can > > > > > > > > > > > fit > > > > > > > > > > > into > > > > > > > > > > > memory. > > > > > > > > > > > > This is something I couldn't get to work. First, I > > > > > > couldn't > > > > > > specify not > > > > > > less > > > > > > than '700 MiB'. If I specify '600 MiB', building stops > > > > > > and > > > > > > tells > > > > > > me it > > > > > > needs > > > > > > at least about 680 MiB. If I use '70%', building stops. > > > > > > And > > > > > > I > > > > > > never saw > > > > > > that > > > > > > 'xz' "automatically scaled down". It just stopped > > > > > > everytime. > > > > > > > > > > Yes, ~700M is the minimum. > > > > > > > > > > It will just reduce the number of threads again to make > > > > > sure > > > > > everything > > > > > fits > > > > > into memory. > > > > > > > > > > > > > > > > > > > > I'm a bit puzzled about this: I tried using ' > > > > > > > > > --memory=70%' > > > > > > > > > as > > > > > > > > > mentioned > > > > > > > > > in the xz man pages, but is seems to have no > > > > > > > > > effect. > > > > > > > > > It > > > > > > > > > looks as > > > > > > > > > if > > > > > > > > > the > > > > > > > > > percent parameter isn't recognized and ignored. It > > > > > > > > > always > > > > > > > > > ends up > > > > > > > > > in > > > > > > > > > "cannot allocate memory". Any hints on this? Do > > > > > > > > > both > > > > > > > > > 'lfs/cdrom' > > > > > > > > > AND > > > > > > > > > 'lfs/Config' need the '--memory'-parameter? > > > > > > > > > > > > This was the next problem: 'percent'-parameter does not > > > > > > work > > > > > > as > > > > > > expected. > > > > > > > > > > No, don't use it. Just determine yourself what it is. > > > > > > > > > > I think we could even try to set 100% of the memory, > > > > > because > > > > > as > > > > > far > > > > > as I > > > > > can > > > > > see > > > > > this is worst case and xz won't use everything that we > > > > > allow. > > > > > > > > > > If that does't work, let's scale down to 90%, or 80%, > > > > > etc... > > > > > > > > > > > > > Yes, but it would be good to have one place where > > > > > > > > the > > > > > > > > parameters are > > > > > > > > being generated and then we just use them from a > > > > > > > > variable. > > > > > > > > > > > > Probably beyond my capabilities for the moment. Any > > > > > > thoughts? > > > > > > > > > > > > > > > > > ... > > > > > > > > > > > We will at least need about half a GB which > > > > > > > > > > > should > > > > > > > > > > > be > > > > > > > > > > > a > > > > > > > > > > > sensible > > > > > > > > > > > requirement > > > > > > > > > > > for a build system any ways. > > > > > > > > > > > > I'm afraid '500 MiB' are not enough... > > > > > > > > > > No, so we will now make 1G the minimum for a build. > > > > > > > > > > I tried to build the OS on an ARM board with only 512MB of > > > > > RAM > > > > > the > > > > > other > > > > > day > > > > > and > > > > > even with a lot of swap it doesn't work. So we are past > > > > > this > > > > > any > > > > > way. > > > > > > > > > > > > > > > > > > > > Agreed. Because of this I'm testing with '500Mib'. > > > > > > > > > > > > > > > > > > > > Will you work on this bringing it into the > > > > > > > > > > > build > > > > > > > > > > > system?! > > > > > > > > > > > > > > > > > > "I'll do my very best!" :-)) > > > > > > > > > > > > > > > > Very well Miss Sophie... > > > > > > > > ... > > > > > > > > > > > > Best™, > > > > > > Matthias > > > > > > > > > > -Michael