Hi,

On Mon, 2017-11-06 at 22:06 +0100, Matthias Fischer wrote:

Hi,

On 06.11.2017 19:32, Michael Tremer wrote:

...

Hello,

so the big Core Update 115 has been shipped last week. Core Update 116 is coming today. So the next one should be a little bit calmer please :)

We'll see... ;-)

...

I do not have anything so far except some smaller bug fixes here and there. Does anyone have anything big? It doesn't have to be another big update. We could also just improve IPFire where ever it is needed with loads of smaller changes and start working on 3 and other sub-projects that are floating around...

Just back in town and getting a grip - I only have a few smaller fixes and features in mind:

'captive portal'-localization: missing german translation for 'unlimited' - patch is being prepared.

Okay.

https://forum.ipfire.org/viewtopic.php?f=17&t=19792 (Mini-Bug in 'captive portal'): couldn't test this by now

I fixed this one already:

https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=ad1204e4eb397b4f7d10...

Updates in progress: 'ethtool' to 4.13, 'smartmontools' to 6.6, 'tor' to 0.3.1.8 (interested?)

Okay. Could you also please review at least some of the patches that Marcel sent last week?

'squid-graph'-localization and package (if I get the time)

As I can see: nothing serious from my side... ;-)

That's good :)

Best, Matthias

Hi,

On 17.11.2017 13:59, Matthias Fischer wrote:

Hi,

On 07.11.2017 11:44, Michael Tremer wrote:

...

... Could you also please review at least some of the patches that Marcel sent last week? ...

Done.

Applied all patches (19 commits) from '2017-10-31' against the current 'next':

***SNIP*** ... On branch next Changes not staged for commit: (use "git add <file>..." to update what will be committed) (use "git checkout -- <file>..." to discard changes in working directory)

modified: config/rootfiles/common/boost modified: config/rootfiles/common/cmake modified: config/rootfiles/common/e2fsprogs modified: config/rootfiles/common/expat modified: config/rootfiles/common/gawk modified: config/rootfiles/common/glib modified: config/rootfiles/common/libarchive modified: config/rootfiles/common/libupnp modified: config/rootfiles/common/libxml2 modified: config/rootfiles/common/libxslt modified: config/rootfiles/common/mpfr modified: config/rootfiles/common/pciutils modified: config/rootfiles/common/tcl modified: config/rootfiles/common/texinfo modified: config/rootfiles/common/xfsprogs modified: config/rootfiles/packages/minidlna modified: config/rootfiles/packages/nmap modified: config/rootfiles/packages/stunnel modified: lfs/boost modified: lfs/cmake modified: lfs/e2fsprogs modified: lfs/expat modified: lfs/gawk modified: lfs/glib modified: lfs/libarchive modified: lfs/libupnp modified: lfs/libxml2 modified: lfs/libxslt modified: lfs/minidlna modified: lfs/mpfr modified: lfs/nmap modified: lfs/pciutils modified: lfs/rpcbind modified: lfs/stunnel modified: lfs/tcl modified: lfs/texinfo modified: lfs/xfsprogs

Untracked files: (use "git add <file>..." to include in what will be committed)

    src/patches/rpcbind-0.2.4-vulnerability_fixes-1.patch

... ***SNAP***

Results:

Source for 'e2fsprogs 1.43.6' were missing, copied them to git.ipfire.org. No problem. Besides, in the meantime there's an update to '1.43.7', will test that with a clean build, too.

'poppler 0.52.0' is complaining:

Changes in poppler-0.52.0 check rootfile! [see attachment]

Besides that, building was ok for now.

Final results:

All patches applied - 'poppler 0.52.0' needs rootfile fix.

The 'e2fsprogs'-update to 1.43.7 also applied - no further changes seem to be necessary for this update so far.

@Marcel: would you do these changes or shall I?

Found by chance:

I found about 70 'configure: WARNING: unrecognized options:'-notifications in '_build.base.log' and 'build.ipfire.log'.

These deal with '--enable-mpbsd', '--disable-nls' (gmp-6.1.2, mpfr-3.1.6, e.g.), '--disable-evms', --enable-multibyte' '--enable-zlib', '--disable-static' , '--disable-kernel-module' and so on.

I don't know why these option switches are still included or why they exist, but: shouldn't we take a look at this?

Best, Matthias

Hi,

On 17.11.2017 22:25, Michael Tremer wrote:

Hi,

and when you apply only a few? Are there any conflicts to expect?

Good question, I thought something like that. It's really a lot. But sorry, my knowledge isn't good enough to be able to assess whether conflicts will arise. BUILDING was ok, but BEHAVIOUR during productive use? Don't know, sorry. Hopefully Marcel has tested this prior to comitting? Any comments from him?

Surely I could test only with "only a few". Which ones would you have in mind? For a start, I can't assess the 'packages' - I never used any of these.

Best, Matthias

On Fri, 2017-11-17 at 20:37 +0100, Matthias Fischer wrote:

...

Hi,

On 17.11.2017 13:59, Matthias Fischer wrote:

...

Hi,

On 07.11.2017 11:44, Michael Tremer wrote:

...

... Could you also please review at least some of the patches that Marcel sent last week? ...

Done.

Applied all patches (19 commits) from '2017-10-31' against the current 'next':

***SNIP*** ... On branch next Changes not staged for commit: (use "git add <file>..." to update what will be committed) (use "git checkout -- <file>..." to discard changes in working directory)

modified: config/rootfiles/common/boost modified: config/rootfiles/common/cmake modified: config/rootfiles/common/e2fsprogs modified: config/rootfiles/common/expat modified: config/rootfiles/common/gawk modified: config/rootfiles/common/glib modified: config/rootfiles/common/libarchive modified: config/rootfiles/common/libupnp modified: config/rootfiles/common/libxml2 modified: config/rootfiles/common/libxslt modified: config/rootfiles/common/mpfr modified: config/rootfiles/common/pciutils modified: config/rootfiles/common/tcl modified: config/rootfiles/common/texinfo modified: config/rootfiles/common/xfsprogs modified: config/rootfiles/packages/minidlna modified: config/rootfiles/packages/nmap modified: config/rootfiles/packages/stunnel modified: lfs/boost modified: lfs/cmake modified: lfs/e2fsprogs modified: lfs/expat modified: lfs/gawk modified: lfs/glib modified: lfs/libarchive modified: lfs/libupnp modified: lfs/libxml2 modified: lfs/libxslt modified: lfs/minidlna modified: lfs/mpfr modified: lfs/nmap modified: lfs/pciutils modified: lfs/rpcbind modified: lfs/stunnel modified: lfs/tcl modified: lfs/texinfo modified: lfs/xfsprogs

Untracked files: (use "git add <file>..." to include in what will be committed)

    src/patches/rpcbind-0.2.4-vulnerability_fixes-1.patch

... ***SNAP***

Results:

Source for 'e2fsprogs 1.43.6' were missing, copied them to git.ipfire.org. No problem. Besides, in the meantime there's an update to '1.43.7', will test that with a clean build, too.

'poppler 0.52.0' is complaining:

Changes in poppler-0.52.0 check rootfile! [see attachment]

Besides that, building was ok for now.

Final results:

All patches applied - 'poppler 0.52.0' needs rootfile fix.

The 'e2fsprogs'-update to 1.43.7 also applied - no further changes seem to be necessary for this update so far.

@Marcel: would you do these changes or shall I?

Found by chance:

I found about 70 'configure: WARNING: unrecognized options:'-notifications in '_build.base.log' and 'build.ipfire.log'.

These deal with '--enable-mpbsd', '--disable-nls' (gmp-6.1.2, mpfr-3.1.6, e.g.), '--disable-evms', --enable-multibyte' '--enable-zlib', '--disable-static' , '--disable-kernel-module' and so on.

I don't know why these option switches are still included or why they exist, but: shouldn't we take a look at this?

Best, Matthias

Hi,

maybe this function helps:

https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=5190eea24f9822a63dc5...

On Tue, 2017-11-07 at 10:46 +0000, Michael Tremer wrote:

On Mon, 2017-11-06 at 22:09 +0100, Matthias Fischer wrote:

...

On 06.11.2017 19:32, Michael Tremer wrote:

...

Hello,

so the big Core Update 115 has been shipped last week. Core Update 116 is coming today. So the next one should be a little bit calmer please :)

I do not have anything so far except some smaller bug fixes here and there. Does anyone have anything big? It doesn't have to be another big update. We could also just improve IPFire where ever it is needed with loads of smaller changes and start working on 3 and other sub-projects that are floating around...

I forgot: Tuning the XZ-parameters... ;-)

Good that you are bringing this up. So I think what we should do here is bringing back XZ compression with -8 so that we do not waste too much space when extracting the image again.

To not run into memory limits we should detect how much memory the build host has and set the memory limit to maybe 80% of that. xz will then automatically scale down to a number of parallel processes that it can fit into memory.

We will at least need about half a GB which should be a sensible requirement for a build system any ways.

Will you work on this bringing it into the build system?!

-Michael

...

Best, Matthias

Hi,

On Tue, 2017-11-07 at 19:41 +0100, Matthias Fischer wrote:

Hi,

On 07.11.2017 16:05, Michael Tremer wrote:

...

Hi,

maybe this function helps:

https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=5190eea24f9822a63dc5...

Thanks. I'll take a look...

...

On Tue, 2017-11-07 at 10:46 +0000, Michael Tremer wrote:

...

On Mon, 2017-11-06 at 22:09 +0100, Matthias Fischer wrote: ...

...

I forgot: Tuning the XZ-parameters... ;-)

Good that you are bringing this up. So I think what we should do here is bringing back XZ compression with -8 so that we do not waste too much space when extracting the image again.

Right now I'm testing how much RAM I need.

Current 'next'-build is running with:

'lfs/cdrom': ... export XZ_OPT = --threads=0 -8 --memory=500MiB ...

'lfs/Config': ... cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8" ... ...

I'm waiting for the results. Takes a while.

You can just run it like this and it will tell you:

[ms@hughes ~]$ xz -vv8 -T0 xz: Filter chain: --lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0 xz: Using up to 4 threads. xz: 2,629 MiB of memory is required. The limiter is disabled. xz: Decompression will need 33 MiB of memory. xz: Compressed data cannot be written to a terminal xz: Try `xz --help' for more information.

...

...

To not run into memory limits we should detect how much memory the build host has and set the memory limit to maybe 80% of that. xz will then automatically scale down to a number of parallel processes that it can fit into memory.

I'm a bit puzzled about this: I tried using '--memory=70%' as mentioned in the xz man pages, but is seems to have no effect. It looks as if the percent parameter isn't recognized and ignored. It always ends up in "cannot allocate memory". Any hints on this? Do both 'lfs/cdrom' AND 'lfs/Config' need the '--memory'-parameter?

Yes, but it would be good to have one place where the parameters are being generated and then we just use them from a variable.

...

...

... We will at least need about half a GB which should be a sensible requirement for a build system any ways.

Agreed. Because of this I'm testing with '500Mib'.

...

...

Will you work on this bringing it into the build system?!

"I'll do my very best!" :-))

Very well Miss Sophie...

Best™, Matthias

Hi,

On 21.11.2017 15:42, Michael Tremer wrote:

Hello,

any progress on this?

Not much, because I've been struggling with a flu for the past two weeks, and its not over yet, but:

In the meantime I ran several builds with different parameters.

IMHO we could use:

'export XZ_OPT = --threads=0 -8 --memory=700MiB'

in 'lfs/cdrom'

and

cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8 --memory=700MiB" tar -c -p --numeric-owner -J -f /install/packages/package/files.tar.xz *

in 'lfs/Config'.

Using '9' or '--extreme'-parameter brought no further improvements, on the other hand, build time and RAM consumption increased strongly. With '-8', minimal RAM consumption was about 680 MiB, using '700 Mib' was always safe. So this is the minimum I would recommend.

Results:

(Buildtime: 5 hours 20 minutes 59 seconds)

255866764 ipfire-2.19.1gb-ext4.i586-full-core117.img.gz 252262006 ipfire-2.19.1gb-ext4-scon.i586-full-core117.img.gz 175112192 ipfire-2.19.i586-full-core117.iso

-Michael

On Tue, 2017-11-07 at 22:47 +0000, Michael Tremer wrote:

...

Hi,

On Tue, 2017-11-07 at 19:41 +0100, Matthias Fischer wrote:

...

Hi,

On 07.11.2017 16:05, Michael Tremer wrote:

...

Hi,

maybe this function helps:

https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=5190eea24f9822a63d c5d06d214b48f973b14f29

Thanks. I'll take a look...

This would help, indeed, but how to implement(!?), and... (see below)

...

...

...

On Tue, 2017-11-07 at 10:46 +0000, Michael Tremer wrote:

...

On Mon, 2017-11-06 at 22:09 +0100, Matthias Fischer wrote: ...

...

I forgot: Tuning the XZ-parameters... ;-)

Good that you are bringing this up. So I think what we should do here is bringing back XZ compression with -8 so that we do not waste too much space when extracting the image again.

Right now I'm testing how much RAM I need.

Current 'next'-build is running with:

'lfs/cdrom': ... export XZ_OPT = --threads=0 -8 --memory=500MiB ...

'lfs/Config': ... cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8" ... ...

I'm waiting for the results. Takes a while.

You can just run it like this and it will tell you:

[ms@hughes ~]$ xz -vv8 -T0 xz: Filter chain: -- lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0 xz: Using up to 4 threads. xz: 2,629 MiB of memory is required. The limiter is disabled. xz: Decompression will need 33 MiB of memory. xz: Compressed data cannot be written to a terminal xz: Try `xz --help' for more information.

...

...

...

To not run into memory limits we should detect how much memory the build host has and set the memory limit to maybe 80% of that. xz will then automatically scale down to a number of parallel processes that it can fit into memory.

This is something I couldn't get to work. First, I couldn't specify not less than '700 MiB'. If I specify '600 MiB', building stops and tells me it needs at least about 680 MiB. If I use '70%', building stops. And I never saw that 'xz' "automatically scaled down". It just stopped everytime.

...

...

I'm a bit puzzled about this: I tried using '--memory=70%' as mentioned in the xz man pages, but is seems to have no effect. It looks as if the percent parameter isn't recognized and ignored. It always ends up in "cannot allocate memory". Any hints on this? Do both 'lfs/cdrom' AND 'lfs/Config' need the '--memory'-parameter?

This was the next problem: 'percent'-parameter does not work as expected.

...

Yes, but it would be good to have one place where the parameters are being generated and then we just use them from a variable.

Probably beyond my capabilities for the moment. Any thoughts?

...

...

...

...

... We will at least need about half a GB which should be a sensible requirement for a build system any ways.

I'm afraid '500 MiB' are not enough...

...

...

Agreed. Because of this I'm testing with '500Mib'.

...

...

Will you work on this bringing it into the build system?!

"I'll do my very best!" :-))

Very well Miss Sophie... ...

Best™, Matthias

Hello Matthias,

any update on this? Are you still on this?

Best, -Michael

On Wed, 2017-11-22 at 16:37 +0000, Michael Tremer wrote:

Hi,

On Tue, 2017-11-21 at 18:35 +0100, Matthias Fischer wrote:

...

Hi,

On 21.11.2017 15:42, Michael Tremer wrote:

...

Hello,

any progress on this?

Not much, because I've been struggling with a flu for the past two weeks, and its not over yet, but:

In the meantime I ran several builds with different parameters.

IMHO we could use:

'export XZ_OPT = --threads=0 -8 --memory=700MiB'

in 'lfs/cdrom'

and

cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8 --memory=700MiB" tar -c -p --numeric-owner -J -f /install/packages/package/files.tar.xz *

in 'lfs/Config'.

Using '9' or '--extreme'-parameter brought no further improvements, on the other hand, build time and RAM consumption increased strongly. With '-8', minimal RAM consumption was about 680 MiB, using '700 Mib' was always safe. So this is the minimum I would recommend.

Yes, I observed the same thing. -9 compresses a little bit better and extreme doesn't change a bit.

-8 seems to be optimal since it won't require too much memory to decompress either. So let's go with that.

...

Results:

(Buildtime: 5 hours 20 minutes 59 seconds)

255866764 ipfire-2.19.1gb-ext4.i586-full-core117.img.gz 252262006 ipfire-2.19.1gb-ext4-scon.i586-full-core117.img.gz 175112192 ipfire-2.19.i586-full-core117.iso

...

-Michael

On Tue, 2017-11-07 at 22:47 +0000, Michael Tremer wrote:

...

Hi,

On Tue, 2017-11-07 at 19:41 +0100, Matthias Fischer wrote:

...

Hi,

On 07.11.2017 16:05, Michael Tremer wrote:

...

Hi,

maybe this function helps:

https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=5190eea24f98 22 a63d c5d06d214b48f973b14f29

Thanks. I'll take a look...

This would help, indeed, but how to implement(!?), and... (see below)

Just run it somewhere in the script, figure out what is about 80% of that and if that is smaller than 800 MB, we should just fail.

It would be good to have a function that determines the xz parameters and we use that where ever we call xz. We don't want any duplicated code.

...

...

...

...

...

On Tue, 2017-11-07 at 10:46 +0000, Michael Tremer wrote: > On Mon, 2017-11-06 at 22:09 +0100, Matthias Fischer wrote: > ... > > > > I forgot: Tuning the XZ-parameters... ;-) > > Good that you are bringing this up. So I think what we should do > here is > bringing back XZ compression with -8 so that we do not waste too > much > space > when > extracting the image again.

Right now I'm testing how much RAM I need.

Current 'next'-build is running with:

'lfs/cdrom': ... export XZ_OPT = --threads=0 -8 --memory=500MiB ...

'lfs/Config': ... cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8" ... ...

I'm waiting for the results. Takes a while.

You can just run it like this and it will tell you:

[ms@hughes ~]$ xz -vv8 -T0 xz: Filter chain: -- lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0 xz: Using up to 4 threads. xz: 2,629 MiB of memory is required. The limiter is disabled. xz: Decompression will need 33 MiB of memory. xz: Compressed data cannot be written to a terminal xz: Try `xz --help' for more information.

...

...

> To not run into memory limits we should detect how much memory the > build > host > has and set the memory limit to maybe 80% of that. xz will then > automatically > scale down to a number of parallel processes that it can fit into > memory.

This is something I couldn't get to work. First, I couldn't specify not less than '700 MiB'. If I specify '600 MiB', building stops and tells me it needs at least about 680 MiB. If I use '70%', building stops. And I never saw that 'xz' "automatically scaled down". It just stopped everytime.

Yes, ~700M is the minimum.

It will just reduce the number of threads again to make sure everything fits into memory.

...

...

...

...

I'm a bit puzzled about this: I tried using '--memory=70%' as mentioned in the xz man pages, but is seems to have no effect. It looks as if the percent parameter isn't recognized and ignored. It always ends up in "cannot allocate memory". Any hints on this? Do both 'lfs/cdrom' AND 'lfs/Config' need the '--memory'-parameter?

This was the next problem: 'percent'-parameter does not work as expected.

No, don't use it. Just determine yourself what it is.

I think we could even try to set 100% of the memory, because as far as I can see this is worst case and xz won't use everything that we allow.

If that does't work, let's scale down to 90%, or 80%, etc...

...

...

...

Yes, but it would be good to have one place where the parameters are being generated and then we just use them from a variable.

Probably beyond my capabilities for the moment. Any thoughts?

...

...

...

...

> ... > We will at least need about half a GB which should be a sensible > requirement > for a build system any ways.

I'm afraid '500 MiB' are not enough...

No, so we will now make 1G the minimum for a build.

I tried to build the OS on an ARM board with only 512MB of RAM the other day and even with a lot of swap it doesn't work. So we are past this any way.

...

...

...

...

Agreed. Because of this I'm testing with '500Mib'.

...

> Will you work on this bringing it into the build system?!

"I'll do my very best!" :-))

Very well Miss Sophie... ...

Best™, Matthias

-Michael

Hi,

On Tue, 2018-01-30 at 17:47 +0100, Matthias Fischer wrote:

On 29.01.2018 14:23, Michael Tremer wrote:

...

Hello Matthias,

Hi Michael,

...

any update on this? Are you still on this?

Sorry, no update, but its still on my list.

We had a tough time here with all kinds of diseases in the family during the holidays and beyond, so I was only able to work on some 'easier jobs'.

That's fine. I just wanted to know if you are still on this and that it isn't just forgotten.

Besides, I'm not really familiar with this kind of scripting, but I'll try. This is the next I'm at.

I'll consider how I would do this and send it in as soon as I get a grip on it.

Well, I added some functions already somewhere. I suppose you just need to do the maths to set that to X% of the host system which is just a simple multiplication of the output of ${HOST_MEM} in make.sh.

And that needs to be passed to the lfs scripts that are supposed to make the images.

Please send questions where ever you need help. There is no reason to do this alone.

Best, -Michael

Best, Matthias

...

Best, -Michael

On Wed, 2017-11-22 at 16:37 +0000, Michael Tremer wrote:

...

Hi,

On Tue, 2017-11-21 at 18:35 +0100, Matthias Fischer wrote:

...

Hi,

On 21.11.2017 15:42, Michael Tremer wrote:

...

Hello,

any progress on this?

Not much, because I've been struggling with a flu for the past two weeks, and its not over yet, but:

In the meantime I ran several builds with different parameters.

IMHO we could use:

'export XZ_OPT = --threads=0 -8 --memory=700MiB'

in 'lfs/cdrom'

and

cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8 --memory=700MiB" tar -c -p --numeric-owner -J -f /install/packages/package/files.tar.xz *

in 'lfs/Config'.

Using '9' or '--extreme'-parameter brought no further improvements, on the other hand, build time and RAM consumption increased strongly. With '-8', minimal RAM consumption was about 680 MiB, using '700 Mib' was always safe. So this is the minimum I would recommend.

Yes, I observed the same thing. -9 compresses a little bit better and extreme doesn't change a bit.

-8 seems to be optimal since it won't require too much memory to decompress either. So let's go with that.

...

Results:

(Buildtime: 5 hours 20 minutes 59 seconds)

255866764 ipfire-2.19.1gb-ext4.i586-full-core117.img.gz 252262006 ipfire-2.19.1gb-ext4-scon.i586-full-core117.img.gz 175112192 ipfire-2.19.i586-full-core117.iso

...

-Michael

On Tue, 2017-11-07 at 22:47 +0000, Michael Tremer wrote:

...

Hi,

On Tue, 2017-11-07 at 19:41 +0100, Matthias Fischer wrote: > Hi, > > On 07.11.2017 16:05, Michael Tremer wrote: > > Hi, > > > > maybe this function helps: > > > > https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=5190eea2 > > 4f98 > > 22 > > a63d > > c5d06d214b48f973b14f29 > > Thanks. I'll take a look...

This would help, indeed, but how to implement(!?), and... (see below)

Just run it somewhere in the script, figure out what is about 80% of that and if that is smaller than 800 MB, we should just fail.

It would be good to have a function that determines the xz parameters and we use that where ever we call xz. We don't want any duplicated code.

...

...

...

> > On Tue, 2017-11-07 at 10:46 +0000, Michael Tremer wrote: > > > On Mon, 2017-11-06 at 22:09 +0100, Matthias Fischer wrote: > > > ... > > > > > > > > I forgot: Tuning the XZ-parameters... ;-) > > > > > > Good that you are bringing this up. So I think what we should > > > do > > > here is > > > bringing back XZ compression with -8 so that we do not waste > > > too > > > much > > > space > > > when > > > extracting the image again. > > Right now I'm testing how much RAM I need. > > Current 'next'-build is running with: > > 'lfs/cdrom': > ... > export XZ_OPT = --threads=0 -8 --memory=500MiB > ... > > 'lfs/Config': > ... > cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8" ... > ... > > I'm waiting for the results. Takes a while.

You can just run it like this and it will tell you:

[ms@hughes ~]$ xz -vv8 -T0 xz: Filter chain: -- lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0 xz: Using up to 4 threads. xz: 2,629 MiB of memory is required. The limiter is disabled. xz: Decompression will need 33 MiB of memory. xz: Compressed data cannot be written to a terminal xz: Try `xz --help' for more information.

> > > To not run into memory limits we should detect how much memory > > > the > > > build > > > host > > > has and set the memory limit to maybe 80% of that. xz will > > > then > > > automatically > > > scale down to a number of parallel processes that it can fit > > > into > > > memory.

This is something I couldn't get to work. First, I couldn't specify not less than '700 MiB'. If I specify '600 MiB', building stops and tells me it needs at least about 680 MiB. If I use '70%', building stops. And I never saw that 'xz' "automatically scaled down". It just stopped everytime.

Yes, ~700M is the minimum.

It will just reduce the number of threads again to make sure everything fits into memory.

...

...

...

> I'm a bit puzzled about this: I tried using '--memory=70%' as > mentioned > in the xz man pages, but is seems to have no effect. It looks as > if > the > percent parameter isn't recognized and ignored. It always ends up > in > "cannot allocate memory". Any hints on this? Do both 'lfs/cdrom' > AND > 'lfs/Config' need the '--memory'-parameter?

This was the next problem: 'percent'-parameter does not work as expected.

No, don't use it. Just determine yourself what it is.

I think we could even try to set 100% of the memory, because as far as I can see this is worst case and xz won't use everything that we allow.

If that does't work, let's scale down to 90%, or 80%, etc...

...

...

...

Yes, but it would be good to have one place where the parameters are being generated and then we just use them from a variable.

Probably beyond my capabilities for the moment. Any thoughts?

...

...

> > > ... > > > We will at least need about half a GB which should be a > > > sensible > > > requirement > > > for a build system any ways.

I'm afraid '500 MiB' are not enough...

No, so we will now make 1G the minimum for a build.

I tried to build the OS on an ARM board with only 512MB of RAM the other day and even with a lot of swap it doesn't work. So we are past this any way.

...

...

...

> Agreed. Because of this I'm testing with '500Mib'. > > > > Will you work on this bringing it into the build system?! > > "I'll do my very best!" :-))

Very well Miss Sophie... ...

Best™, Matthias

-Michael

Hi,

Small addition:

I found another possibility for calculating XZ memory and adding the needed suffix:

... ((XZ_MEM=(($HOST_MEM)/10)*7)) && XZ_MEM=${XZ_MEM}MiB ...

Looks better.

Best, Matthias

On 03.02.2018 20:44, Matthias Fischer wrote:

Hi,

On 30.01.2018 21:03, Michael Tremer wrote:

...

Hi,

...

...

I'll consider how I would do this and send it in as soon as I get a grip on it.

Well, I added some functions already somewhere. I suppose you just need to do the maths to set that to X% of the host system which is just a simple multiplication of the output of ${HOST_MEM} in make.sh.

Ok, I take my heart in my hand. Here are my first results - but remember, I'm not familiar with this. I built a short test script which looks as if it is working as wanted, but...

...

And that needs to be passed to the lfs scripts that are supposed to make the images.

...this is a problem I'm struggling with right now: How do I pass the 'XZ_MEM'-variable to 'cdrom' and 'Config'?

I think this can be done using 'export XZ_MEM' or something else but I'm not sure.

This is the script I'm testing with:

***SNIP*** #!/bin/bash

system_memory() { local key val unit

while read -r key val unit; do case "${key}" in MemTotal:*) # Convert to MB echo "$(( ${val} / 1024 ))" break ;; esac done < /proc/meminfo }

HOST_MEM=$(system_memory)

# Checking host memory if [ $HOST_MEM -ge 1024 ]; then echo Host-Memory: $HOST_MEM'MiB' echo "Host memory is OK (must be at least 1024MiB), starting build." else echo Host-Memory: $HOST_MEM'MiB' echo "Host memory too low (less than 1024MiB), building stopped." exit 1 fi

# Host memory is ok, calculating needed XZ memory to about 70% of host memory ((XZ_MEM=(($HOST_MEM)/10)*7)) XZ_MEM=$XZ_MEM'Mib' echo XZ-Memory: $XZ_MEM echo "XZ memory size is OK (must be at least 700MiB), starting build." ***SNAP***

The 'echoes' are mostly for debugging.

Calculation looks ok, 'MiB' must added without blanks (XZ man page: "In most places where an integer argument is expected, an optional suffix is supported to easily indicate large integers. There must be no space between the integer and the suffix.")

...

Please send questions where ever you need help. There is no reason to do this alone.

Please see above:

1. Is this code what you where thinking of? I would leave the other parameters

as they are (--threads=0 -8).

2. Where should I add such code in the first place? I'd like to start with

'make.sh', so building stops as soon as possible if there is not enough memory. Where's the best place? Right after function 'system_memory() {...}'?

3. How to I pass contents of 'XZ_MEM' to 'cdrom' and 'Config'?

I consider something like: ... export XZ_OPT = --threads=0 -8 --memory=$XZ_MEM ...

Best, Matthias

...

Best, -Michael

...

Best, Matthias

...

Best, -Michael

On Wed, 2017-11-22 at 16:37 +0000, Michael Tremer wrote:

...

Hi,

On Tue, 2017-11-21 at 18:35 +0100, Matthias Fischer wrote:

...

Hi,

On 21.11.2017 15:42, Michael Tremer wrote: > Hello, > > any progress on this?

Not much, because I've been struggling with a flu for the past two weeks, and its not over yet, but:

In the meantime I ran several builds with different parameters.

IMHO we could use:

'export XZ_OPT = --threads=0 -8 --memory=700MiB'

in 'lfs/cdrom'

and

cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8 --memory=700MiB" tar -c -p --numeric-owner -J -f /install/packages/package/files.tar.xz *

in 'lfs/Config'.

Using '9' or '--extreme'-parameter brought no further improvements, on the other hand, build time and RAM consumption increased strongly. With '-8', minimal RAM consumption was about 680 MiB, using '700 Mib' was always safe. So this is the minimum I would recommend.

Yes, I observed the same thing. -9 compresses a little bit better and extreme doesn't change a bit.

-8 seems to be optimal since it won't require too much memory to decompress either. So let's go with that.

...

Results:

(Buildtime: 5 hours 20 minutes 59 seconds)

255866764 ipfire-2.19.1gb-ext4.i586-full-core117.img.gz 252262006 ipfire-2.19.1gb-ext4-scon.i586-full-core117.img.gz 175112192 ipfire-2.19.i586-full-core117.iso

> -Michael > > On Tue, 2017-11-07 at 22:47 +0000, Michael Tremer wrote: > > Hi, > > > > On Tue, 2017-11-07 at 19:41 +0100, Matthias Fischer wrote: > > > Hi, > > > > > > On 07.11.2017 16:05, Michael Tremer wrote: > > > > Hi, > > > > > > > > maybe this function helps: > > > > > > > > https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=5190eea2 > > > > 4f98 > > > > 22 > > > > a63d > > > > c5d06d214b48f973b14f29 > > > > > > Thanks. I'll take a look...

This would help, indeed, but how to implement(!?), and... (see below)

Just run it somewhere in the script, figure out what is about 80% of that and if that is smaller than 800 MB, we should just fail.

It would be good to have a function that determines the xz parameters and we use that where ever we call xz. We don't want any duplicated code.

...

> > > > On Tue, 2017-11-07 at 10:46 +0000, Michael Tremer wrote: > > > > > On Mon, 2017-11-06 at 22:09 +0100, Matthias Fischer wrote: > > > > > ... > > > > > > > > > > > > I forgot: Tuning the XZ-parameters... ;-) > > > > > > > > > > Good that you are bringing this up. So I think what we should > > > > > do > > > > > here is > > > > > bringing back XZ compression with -8 so that we do not waste > > > > > too > > > > > much > > > > > space > > > > > when > > > > > extracting the image again. > > > > > > Right now I'm testing how much RAM I need. > > > > > > Current 'next'-build is running with: > > > > > > 'lfs/cdrom': > > > ... > > > export XZ_OPT = --threads=0 -8 --memory=500MiB > > > ... > > > > > > 'lfs/Config': > > > ... > > > cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8" ... > > > ... > > > > > > I'm waiting for the results. Takes a while. > > > > You can just run it like this and it will tell you: > > > > [ms@hughes ~]$ xz -vv8 -T0 > > xz: Filter chain: -- > > lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0 > > xz: Using up to 4 threads. > > xz: 2,629 MiB of memory is required. The limiter is disabled. > > xz: Decompression will need 33 MiB of memory. > > xz: Compressed data cannot be written to a terminal > > xz: Try `xz --help' for more information. > > > > > > > To not run into memory limits we should detect how much memory > > > > > the > > > > > build > > > > > host > > > > > has and set the memory limit to maybe 80% of that. xz will > > > > > then > > > > > automatically > > > > > scale down to a number of parallel processes that it can fit > > > > > into > > > > > memory.

This is something I couldn't get to work. First, I couldn't specify not less than '700 MiB'. If I specify '600 MiB', building stops and tells me it needs at least about 680 MiB. If I use '70%', building stops. And I never saw that 'xz' "automatically scaled down". It just stopped everytime.

Yes, ~700M is the minimum.

It will just reduce the number of threads again to make sure everything fits into memory.

...

> > > I'm a bit puzzled about this: I tried using '--memory=70%' as > > > mentioned > > > in the xz man pages, but is seems to have no effect. It looks as > > > if > > > the > > > percent parameter isn't recognized and ignored. It always ends up > > > in > > > "cannot allocate memory". Any hints on this? Do both 'lfs/cdrom' > > > AND > > > 'lfs/Config' need the '--memory'-parameter?

This was the next problem: 'percent'-parameter does not work as expected.

No, don't use it. Just determine yourself what it is.

I think we could even try to set 100% of the memory, because as far as I can see this is worst case and xz won't use everything that we allow.

If that does't work, let's scale down to 90%, or 80%, etc...

...

> > Yes, but it would be good to have one place where the parameters are > > being generated and then we just use them from a variable.

Probably beyond my capabilities for the moment. Any thoughts?

> > > > > ... > > > > > We will at least need about half a GB which should be a > > > > > sensible > > > > > requirement > > > > > for a build system any ways.

I'm afraid '500 MiB' are not enough...

No, so we will now make 1G the minimum for a build.

I tried to build the OS on an ARM board with only 512MB of RAM the other day and even with a lot of swap it doesn't work. So we are past this any way.

...

> > > Agreed. Because of this I'm testing with '500Mib'. > > > > > > > > Will you work on this bringing it into the build system?! > > > > > > "I'll do my very best!" :-)) > > > > Very well Miss Sophie... > > ...

Best™, Matthias

-Michael

Hi,

On 06.02.2018 02:05, Michael Tremer wrote:

...

... ((XZ_MEM=(($HOST_MEM)/10)*7)) && XZ_MEM=${XZ_MEM}MiB

You can write this even short:

XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"

Oneliner! Yes. I was looking for something like that... ;-)

I swapped the order of the multiplication and division so that you would have a more exact result.

For example if you system has 1024 MB or RAM, you divide this by 10 (102) and then multiply by 7 which gives you 714. The other order is 1024 * 7 = 7167 and then divided by 10 is 716 which is closer to the actual result of 716.8. Shouldn't matter here but just in general this is better :)

Thanks, looks better - noted for the next time!

Now it looks like this ('make.sh'):

... # Host memory is ok, calculating XZ memory XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB" XZ_OPT="--threads=0 -8 $XZ_MEM" echo XZ-Memory: $XZ_MEM echo XZ-Options: $XZ_OPT echo "XZ memory size is OK (must be at least 700MiB), starting build..." ... ["echoes" for debugging!]

This is working. '$XZ_OPT' now gives me: --threads=0 -8 5450MiB

But. ;-)

You can put this probably next under "HOST_MEM=$(system_memory)" and then export this to the internal shell by adding the variable to lfsmake2 for the disk image and ipfiredist for the packages.

Sorry, I must confess that I don't know exactly how to do this. As I wrote: "How do I pass the 'XZ_MEM'-variable to 'cdrom' and 'Config'?" Or in this case, XZ_OPT!?

Then in lfs/cdrom and lfs/Config for the dist target, you just need to add the parameters to the command line.

In 'Config' I changed:

... cd /install/packages/package/tmp/ && XZ_OPT=-T0 tar -c -p --numeric-owner -J -f /install/packages/package/files.tar.xz * ...

To: ... cd /install/packages/package/tmp/ && $XZ_OPT tar -c -p --numeric-owner -J -f /install/packages/package/files.tar.xz * ...

Maybe it is a good idea to have a variable XZ_OPT that adds everything together and is then used in both places.

I added the other xz-options and created XZ_OPT as shown above, but in 'cdrom' there is an 'export XZ_OPT = ...'-line and I don't know how to handle these line. Is it still necessary?

Do I have to 'export' the XZ_OPT'-variable in 'make.sh'?

And how do I "add" this variable to 'lfsmake2' and 'ipfiredist'(-function)?

Sorry if these questions sound simple, but I'm guessing a bit too much while trying to get a grip on this.

Best, Matthias

Best, -Michael

...

...

Looks better.

Best, Matthias

On 03.02.2018 20:44, Matthias Fischer wrote:

...

Hi,

On 30.01.2018 21:03, Michael Tremer wrote:

...

Hi,

...

...

I'll consider how I would do this and send it in as soon as I get a grip on it.

Well, I added some functions already somewhere. I suppose you just need to do the maths to set that to X% of the host system which is just a simple multiplication of the output of ${HOST_MEM} in make.sh.

Ok, I take my heart in my hand. Here are my first results - but remember, I'm not familiar with this. I built a short test script which looks as if it is working as wanted, but...

...

And that needs to be passed to the lfs scripts that are supposed to make the images.

...this is a problem I'm struggling with right now: How do I pass the 'XZ_MEM'-variable to 'cdrom' and 'Config'?

I think this can be done using 'export XZ_MEM' or something else but I'm not sure.

This is the script I'm testing with:

***SNIP*** #!/bin/bash

system_memory() { local key val unit

while read -r key val unit; do case "${key}" in MemTotal:*) # Convert to MB echo "$(( ${val} / 1024 ))" break ;; esac done < /proc/meminfo }

HOST_MEM=$(system_memory)

# Checking host memory if [ $HOST_MEM -ge 1024 ]; then echo Host-Memory: $HOST_MEM'MiB' echo "Host memory is OK (must be at least 1024MiB), starting build." else echo Host-Memory: $HOST_MEM'MiB' echo "Host memory too low (less than 1024MiB), building stopped." exit 1 fi

# Host memory is ok, calculating needed XZ memory to about 70% of host memory ((XZ_MEM=(($HOST_MEM)/10)*7)) XZ_MEM=$XZ_MEM'Mib' echo XZ-Memory: $XZ_MEM echo "XZ memory size is OK (must be at least 700MiB), starting build." ***SNAP***

The 'echoes' are mostly for debugging.

Calculation looks ok, 'MiB' must added without blanks (XZ man page: "In most places where an integer argument is expected, an optional suffix is supported to easily indicate large integers. There must be no space between the integer and the suffix.")

...

Please send questions where ever you need help. There is no reason to do this alone.

Please see above:

1. Is this code what you where thinking of? I would leave the other

parameters as they are (--threads=0 -8).

2. Where should I add such code in the first place? I'd like to start with

'make.sh', so building stops as soon as possible if there is not enough memory. Where's the best place? Right after function 'system_memory() {...}'?

3. How to I pass contents of 'XZ_MEM' to 'cdrom' and 'Config'?

I consider something like: ... export XZ_OPT = --threads=0 -8 --memory=$XZ_MEM ...

Best, Matthias

...

Best, -Michael

...

Best, Matthias

...

Best, -Michael

On Wed, 2017-11-22 at 16:37 +0000, Michael Tremer wrote: > Hi, > > On Tue, 2017-11-21 at 18:35 +0100, Matthias Fischer wrote: > > Hi, > > > > On 21.11.2017 15:42, Michael Tremer wrote: > > > Hello, > > > > > > any progress on this? > > > > Not much, because I've been struggling with a flu for the past > > two > > weeks, and its not over yet, but: > > > > In the meantime I ran several builds with different parameters. > > > > IMHO we could use: > > > > 'export XZ_OPT = --threads=0 -8 --memory=700MiB' > > > > in 'lfs/cdrom' > > > > and > > > > cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8 -- > > memory=700MiB" tar > > -c > > -p > > --numeric-owner -J -f /install/packages/package/files.tar.xz * > > > > in 'lfs/Config'. > > > > Using '9' or '--extreme'-parameter brought no further > > improvements, > > on the other hand, build time and RAM consumption increased > > strongly. With '-8', minimal RAM consumption was about 680 MiB, > > using > > '700 Mib' was always safe. So this is the minimum I would > > recommend. > > Yes, I observed the same thing. -9 compresses a little bit better > and > extreme > doesn't change a bit. > > -8 seems to be optimal since it won't require too much memory to > decompress > either. So let's go with that. > > > Results: > > > > (Buildtime: 5 hours 20 minutes 59 seconds) > > > > 255866764 ipfire-2.19.1gb-ext4.i586-full-core117.img.gz > > 252262006 ipfire-2.19.1gb-ext4-scon.i586-full-core117.img.gz > > 175112192 ipfire-2.19.i586-full-core117.iso > > > > > -Michael > > > > > > On Tue, 2017-11-07 at 22:47 +0000, Michael Tremer wrote: > > > > Hi, > > > > > > > > On Tue, 2017-11-07 at 19:41 +0100, Matthias Fischer wrote: > > > > > Hi, > > > > > > > > > > On 07.11.2017 16:05, Michael Tremer wrote: > > > > > > Hi, > > > > > > > > > > > > maybe this function helps: > > > > > > > > > > > > https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h= > > > > > > 51 > > > > > > 90eea2 > > > > > > 4f98 > > > > > > 22 > > > > > > a63d > > > > > > c5d06d214b48f973b14f29 > > > > > > > > > > Thanks. I'll take a look... > > > > This would help, indeed, but how to implement(!?), and... (see > > below) > > Just run it somewhere in the script, figure out what is about 80% > of > that > and > if > that is smaller than 800 MB, we should just fail. > > It would be good to have a function that determines the xz > parameters and > we > use > that where ever we call xz. We don't want any duplicated code. > > > > > > > On Tue, 2017-11-07 at 10:46 +0000, Michael Tremer wrote: > > > > > > > On Mon, 2017-11-06 at 22:09 +0100, Matthias Fischer > > > > > > > wrote: > > > > > > > ... > > > > > > > > > > > > > > > > I forgot: Tuning the XZ-parameters... ;-) > > > > > > > > > > > > > > Good that you are bringing this up. So I think what we > > > > > > > should > > > > > > > do > > > > > > > here is > > > > > > > bringing back XZ compression with -8 so that we do not > > > > > > > waste > > > > > > > too > > > > > > > much > > > > > > > space > > > > > > > when > > > > > > > extracting the image again. > > > > > > > > > > Right now I'm testing how much RAM I need. > > > > > > > > > > Current 'next'-build is running with: > > > > > > > > > > 'lfs/cdrom': > > > > > ... > > > > > export XZ_OPT = --threads=0 -8 --memory=500MiB > > > > > ... > > > > > > > > > > 'lfs/Config': > > > > > ... > > > > > cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8" ... > > > > > ... > > > > > > > > > > I'm waiting for the results. Takes a while. > > > > > > > > You can just run it like this and it will tell you: > > > > > > > > [ms@hughes ~]$ xz -vv8 -T0 > > > > xz: Filter chain: -- > > > > lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,d > > > > ep > > > > th=0 > > > > xz: Using up to 4 threads. > > > > xz: 2,629 MiB of memory is required. The limiter is > > > > disabled. > > > > xz: Decompression will need 33 MiB of memory. > > > > xz: Compressed data cannot be written to a terminal > > > > xz: Try `xz --help' for more information. > > > > > > > > > > > To not run into memory limits we should detect how > > > > > > > much > > > > > > > memory > > > > > > > the > > > > > > > build > > > > > > > host > > > > > > > has and set the memory limit to maybe 80% of that. xz > > > > > > > will > > > > > > > then > > > > > > > automatically > > > > > > > scale down to a number of parallel processes that it > > > > > > > can > > > > > > > fit > > > > > > > into > > > > > > > memory. > > > > This is something I couldn't get to work. First, I couldn't > > specify not > > less > > than '700 MiB'. If I specify '600 MiB', building stops and tells > > me it > > needs > > at least about 680 MiB. If I use '70%', building stops. And I > > never saw > > that > > 'xz' "automatically scaled down". It just stopped everytime. > > Yes, ~700M is the minimum. > > It will just reduce the number of threads again to make sure > everything > fits > into memory. > > > > > > > > I'm a bit puzzled about this: I tried using '--memory=70%' > > > > > as > > > > > mentioned > > > > > in the xz man pages, but is seems to have no effect. It > > > > > looks as > > > > > if > > > > > the > > > > > percent parameter isn't recognized and ignored. It always > > > > > ends up > > > > > in > > > > > "cannot allocate memory". Any hints on this? Do both > > > > > 'lfs/cdrom' > > > > > AND > > > > > 'lfs/Config' need the '--memory'-parameter? > > > > This was the next problem: 'percent'-parameter does not work as > > expected. > > No, don't use it. Just determine yourself what it is. > > I think we could even try to set 100% of the memory, because as > far > as I > can > see > this is worst case and xz won't use everything that we allow. > > If that does't work, let's scale down to 90%, or 80%, etc... > > > > > Yes, but it would be good to have one place where the > > > > parameters are > > > > being generated and then we just use them from a variable. > > > > Probably beyond my capabilities for the moment. Any thoughts? > > > > > > > > > ... > > > > > > > We will at least need about half a GB which should be > > > > > > > a > > > > > > > sensible > > > > > > > requirement > > > > > > > for a build system any ways. > > > > I'm afraid '500 MiB' are not enough... > > No, so we will now make 1G the minimum for a build. > > I tried to build the OS on an ARM board with only 512MB of RAM the > other > day > and > even with a lot of swap it doesn't work. So we are past this any > way. > > > > > > > > Agreed. Because of this I'm testing with '500Mib'. > > > > > > > > > > > > Will you work on this bringing it into the build > > > > > > > system?! > > > > > > > > > > "I'll do my very best!" :-)) > > > > > > > > Very well Miss Sophie... > > > > ... > > > > Best™, > > Matthias > > -Michael

Hey Matthias,

are you still working on this?

I am asking because I just noticed that the new next build is about 50MB larger than the old one due to the larger kernel and loads more of firmware files.

Compressing it better would of course help to save space on disk.

Best, -Michael

On Sun, 2018-02-11 at 19:57 +0000, Michael Tremer wrote:

Hi,

On Sun, 2018-02-11 at 15:35 +0000, Matthias Fischer wrote:

...

Hi,

On 06.02.2018 02:05, Michael Tremer wrote:

...

...

... ((XZ_MEM=(($HOST_MEM)/10)*7)) && XZ_MEM=${XZ_MEM}MiB

You can write this even short:

XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"

Oneliner! Yes. I was looking for something like that... ;-)

...

I swapped the order of the multiplication and division so that you would have a more exact result.

For example if you system has 1024 MB or RAM, you divide this by 10 (102) and then multiply by 7 which gives you 714. The other order is 1024 * 7 = 7167 and then divided by 10 is 716 which is closer to the actual result of 716.8. Shouldn't matter here but just in general this is better :)

Thanks, looks better - noted for the next time!

Now it looks like this ('make.sh'):

... # Host memory is ok, calculating XZ memory XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB" XZ_OPT="--threads=0 -8 $XZ_MEM" echo XZ-Memory: $XZ_MEM echo XZ-Options: $XZ_OPT echo "XZ memory size is OK (must be at least 700MiB), starting build..." ... ["echoes" for debugging!]

This is working. '$XZ_OPT' now gives me: --threads=0 -8 5450MiB

Will there be some checks in make.sh now that check if at least 1024MB of memory are available for build? I think there should be a warning if not enough memory is available, but we should still try to build.

...

But. ;-)

...

You can put this probably next under "HOST_MEM=$(system_memory)" and then export this to the internal shell by adding the variable to lfsmake2 for the disk image and ipfiredist for the packages.

Sorry, I must confess that I don't know exactly how to do this. As I wrote: "How do I pass the 'XZ_MEM'-variable to 'cdrom' and 'Config'?" Or in this case, XZ_OPT!?

...

Then in lfs/cdrom and lfs/Config for the dist target, you just need to add the parameters to the command line.

In 'Config' I changed:

... cd /install/packages/package/tmp/ && XZ_OPT=-T0 tar -c -p --numeric-owner -J -f /install/packages/package/files.tar.xz * ...

To: ... cd /install/packages/package/tmp/ && $XZ_OPT tar -c -p --numeric-owner -J -f /install/packages/package/files.tar.xz * ...

...

Maybe it is a good idea to have a variable XZ_OPT that adds everything together and is then used in both places.

I added the other xz-options and created XZ_OPT as shown above, but in 'cdrom' there is an 'export XZ_OPT = ...'-line and I don't know how to handle these line. Is it still necessary?

No, that is no longer necessary and can be removed. We should just add it to the tar command just like it is done in lfs/Config.

...

Do I have to 'export' the XZ_OPT'-variable in 'make.sh'?

No, because the chroot environment will throw away all environment variables from the host system and then set them again. That's why we have this enterchroot() function that does this with the env command.

...

And how do I "add" this variable to 'lfsmake2' and 'ipfiredist'(-function)?

Just have a look at the functions. There should be loads of examples where we pass other variables and you just need to add this one, too.

...

Sorry if these questions sound simple, but I'm guessing a bit too much while trying to get a grip on this.

This is absolutely fine to sort this out first and then send the patch :) That's what this list is for.

Best, -Michael

...

Best, Matthias

...

Best, -Michael

...

...

Looks better.

Best, Matthias

On 03.02.2018 20:44, Matthias Fischer wrote:

...

Hi,

On 30.01.2018 21:03, Michael Tremer wrote:

...

Hi,

... > > I'll consider how I would do this and send it in as soon as I get > a > grip > on it.

Well, I added some functions already somewhere. I suppose you just need to do the maths to set that to X% of the host system which is just a simple multiplication of the output of ${HOST_MEM} in make.sh.

Ok, I take my heart in my hand. Here are my first results - but remember, I'm not familiar with this. I built a short test script which looks as if it is working as wanted, but...

...

And that needs to be passed to the lfs scripts that are supposed to make the images.

...this is a problem I'm struggling with right now: How do I pass the 'XZ_MEM'-variable to 'cdrom' and 'Config'?

I think this can be done using 'export XZ_MEM' or something else but I'm not sure.

This is the script I'm testing with:

***SNIP*** #!/bin/bash

system_memory() { local key val unit

while read -r key val unit; do case "${key}" in MemTotal:*) # Convert to MB echo "$(( ${val} / 1024 ))" break ;; esac done < /proc/meminfo }

HOST_MEM=$(system_memory)

# Checking host memory if [ $HOST_MEM -ge 1024 ]; then echo Host-Memory: $HOST_MEM'MiB' echo "Host memory is OK (must be at least 1024MiB), starting build." else echo Host-Memory: $HOST_MEM'MiB' echo "Host memory too low (less than 1024MiB), building stopped." exit 1 fi

# Host memory is ok, calculating needed XZ memory to about 70% of host memory ((XZ_MEM=(($HOST_MEM)/10)*7)) XZ_MEM=$XZ_MEM'Mib' echo XZ-Memory: $XZ_MEM echo "XZ memory size is OK (must be at least 700MiB), starting build." ***SNAP***

The 'echoes' are mostly for debugging.

Calculation looks ok, 'MiB' must added without blanks (XZ man page: "In most places where an integer argument is expected, an optional suffix is supported to easily indicate large integers. There must be no space between the integer and the suffix.")

...

Please send questions where ever you need help. There is no reason to do this alone.

Please see above:

1. Is this code what you where thinking of? I would leave the other

parameters as they are (--threads=0 -8).

2. Where should I add such code in the first place? I'd like to start

with 'make.sh', so building stops as soon as possible if there is not enough memory. Where's the best place? Right after function 'system_memory() {...}'?

3. How to I pass contents of 'XZ_MEM' to 'cdrom' and 'Config'?

I consider something like: ... export XZ_OPT = --threads=0 -8 --memory=$XZ_MEM ...

Best, Matthias

...

Best, -Michael

> > Best, > Matthias > > Best, > > -Michael > > > > On Wed, 2017-11-22 at 16:37 +0000, Michael Tremer wrote: > > > Hi, > > > > > > On Tue, 2017-11-21 at 18:35 +0100, Matthias Fischer wrote: > > > > Hi, > > > > > > > > On 21.11.2017 15:42, Michael Tremer wrote: > > > > > Hello, > > > > > > > > > > any progress on this? > > > > > > > > Not much, because I've been struggling with a flu for the > > > > past > > > > two > > > > weeks, and its not over yet, but: > > > > > > > > In the meantime I ran several builds with different > > > > parameters. > > > > > > > > IMHO we could use: > > > > > > > > 'export XZ_OPT = --threads=0 -8 --memory=700MiB' > > > > > > > > in 'lfs/cdrom' > > > > > > > > and > > > > > > > > cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8 -- > > > > memory=700MiB" tar > > > > -c > > > > -p > > > > --numeric-owner -J -f /install/packages/package/files.tar.xz > > > > * > > > > > > > > in 'lfs/Config'. > > > > > > > > Using '9' or '--extreme'-parameter brought no further > > > > improvements, > > > > on the other hand, build time and RAM consumption increased > > > > strongly. With '-8', minimal RAM consumption was about 680 > > > > MiB, > > > > using > > > > '700 Mib' was always safe. So this is the minimum I would > > > > recommend. > > > > > > Yes, I observed the same thing. -9 compresses a little bit > > > better > > > and > > > extreme > > > doesn't change a bit. > > > > > > -8 seems to be optimal since it won't require too much memory > > > to > > > decompress > > > either. So let's go with that. > > > > > > > Results: > > > > > > > > (Buildtime: 5 hours 20 minutes 59 seconds) > > > > > > > > 255866764 ipfire-2.19.1gb-ext4.i586-full-core117.img.gz > > > > 252262006 ipfire-2.19.1gb-ext4-scon.i586-full- > > > > core117.img.gz > > > > 175112192 ipfire-2.19.i586-full-core117.iso > > > > > > > > > -Michael > > > > > > > > > > On Tue, 2017-11-07 at 22:47 +0000, Michael Tremer wrote: > > > > > > Hi, > > > > > > > > > > > > On Tue, 2017-11-07 at 19:41 +0100, Matthias Fischer > > > > > > wrote: > > > > > > > Hi, > > > > > > > > > > > > > > On 07.11.2017 16:05, Michael Tremer wrote: > > > > > > > > Hi, > > > > > > > > > > > > > > > > maybe this function helps: > > > > > > > > > > > > > > > > https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdif > > > > > > > > f;h= > > > > > > > > 51 > > > > > > > > 90eea2 > > > > > > > > 4f98 > > > > > > > > 22 > > > > > > > > a63d > > > > > > > > c5d06d214b48f973b14f29 > > > > > > > > > > > > > > Thanks. I'll take a look... > > > > > > > > This would help, indeed, but how to implement(!?), and... > > > > (see > > > > below) > > > > > > Just run it somewhere in the script, figure out what is about > > > 80% > > > of > > > that > > > and > > > if > > > that is smaller than 800 MB, we should just fail. > > > > > > It would be good to have a function that determines the xz > > > parameters and > > > we > > > use > > > that where ever we call xz. We don't want any duplicated code. > > > > > > > > > > > On Tue, 2017-11-07 at 10:46 +0000, Michael Tremer > > > > > > > > wrote: > > > > > > > > > On Mon, 2017-11-06 at 22:09 +0100, Matthias > > > > > > > > > Fischer > > > > > > > > > wrote: > > > > > > > > > ... > > > > > > > > > > > > > > > > > > > > I forgot: Tuning the XZ-parameters... ;-) > > > > > > > > > > > > > > > > > > Good that you are bringing this up. So I think > > > > > > > > > what we > > > > > > > > > should > > > > > > > > > do > > > > > > > > > here is > > > > > > > > > bringing back XZ compression with -8 so that we do > > > > > > > > > not > > > > > > > > > waste > > > > > > > > > too > > > > > > > > > much > > > > > > > > > space > > > > > > > > > when > > > > > > > > > extracting the image again. > > > > > > > > > > > > > > Right now I'm testing how much RAM I need. > > > > > > > > > > > > > > Current 'next'-build is running with: > > > > > > > > > > > > > > 'lfs/cdrom': > > > > > > > ... > > > > > > > export XZ_OPT = --threads=0 -8 --memory=500MiB > > > > > > > ... > > > > > > > > > > > > > > 'lfs/Config': > > > > > > > ... > > > > > > > cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8" > > > > > > > ... > > > > > > > ... > > > > > > > > > > > > > > I'm waiting for the results. Takes a while. > > > > > > > > > > > > You can just run it like this and it will tell you: > > > > > > > > > > > > [ms@hughes ~]$ xz -vv8 -T0 > > > > > > xz: Filter chain: -- > > > > > > lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=b > > > > > > t4,d > > > > > > ep > > > > > > th=0 > > > > > > xz: Using up to 4 threads. > > > > > > xz: 2,629 MiB of memory is required. The limiter is > > > > > > disabled. > > > > > > xz: Decompression will need 33 MiB of memory. > > > > > > xz: Compressed data cannot be written to a terminal > > > > > > xz: Try `xz --help' for more information. > > > > > > > > > > > > > > > To not run into memory limits we should detect how > > > > > > > > > much > > > > > > > > > memory > > > > > > > > > the > > > > > > > > > build > > > > > > > > > host > > > > > > > > > has and set the memory limit to maybe 80% of that. > > > > > > > > > xz > > > > > > > > > will > > > > > > > > > then > > > > > > > > > automatically > > > > > > > > > scale down to a number of parallel processes that > > > > > > > > > it > > > > > > > > > can > > > > > > > > > fit > > > > > > > > > into > > > > > > > > > memory. > > > > > > > > This is something I couldn't get to work. First, I couldn't > > > > specify not > > > > less > > > > than '700 MiB'. If I specify '600 MiB', building stops and > > > > tells > > > > me it > > > > needs > > > > at least about 680 MiB. If I use '70%', building stops. And > > > > I > > > > never saw > > > > that > > > > 'xz' "automatically scaled down". It just stopped everytime. > > > > > > Yes, ~700M is the minimum. > > > > > > It will just reduce the number of threads again to make sure > > > everything > > > fits > > > into memory. > > > > > > > > > > > > > > I'm a bit puzzled about this: I tried using ' > > > > > > > --memory=70%' > > > > > > > as > > > > > > > mentioned > > > > > > > in the xz man pages, but is seems to have no effect. > > > > > > > It > > > > > > > looks as > > > > > > > if > > > > > > > the > > > > > > > percent parameter isn't recognized and ignored. It > > > > > > > always > > > > > > > ends up > > > > > > > in > > > > > > > "cannot allocate memory". Any hints on this? Do both > > > > > > > 'lfs/cdrom' > > > > > > > AND > > > > > > > 'lfs/Config' need the '--memory'-parameter? > > > > > > > > This was the next problem: 'percent'-parameter does not work > > > > as > > > > expected. > > > > > > No, don't use it. Just determine yourself what it is. > > > > > > I think we could even try to set 100% of the memory, because > > > as > > > far > > > as I > > > can > > > see > > > this is worst case and xz won't use everything that we allow. > > > > > > If that does't work, let's scale down to 90%, or 80%, etc... > > > > > > > > > Yes, but it would be good to have one place where the > > > > > > parameters are > > > > > > being generated and then we just use them from a > > > > > > variable. > > > > > > > > Probably beyond my capabilities for the moment. Any > > > > thoughts? > > > > > > > > > > > > > ... > > > > > > > > > We will at least need about half a GB which should > > > > > > > > > be > > > > > > > > > a > > > > > > > > > sensible > > > > > > > > > requirement > > > > > > > > > for a build system any ways. > > > > > > > > I'm afraid '500 MiB' are not enough... > > > > > > No, so we will now make 1G the minimum for a build. > > > > > > I tried to build the OS on an ARM board with only 512MB of RAM > > > the > > > other > > > day > > > and > > > even with a lot of swap it doesn't work. So we are past this > > > any > > > way. > > > > > > > > > > > > > > Agreed. Because of this I'm testing with '500Mib'. > > > > > > > > > > > > > > > > Will you work on this bringing it into the build > > > > > > > > > system?! > > > > > > > > > > > > > > "I'll do my very best!" :-)) > > > > > > > > > > > > Very well Miss Sophie... > > > > > > ... > > > > > > > > Best™, > > > > Matthias > > > > > > -Michael

Hey,

technically that is an idea, but I don't think it makes sense to do this.

People would need the firmware if they are connecting to the Internet using WiFi and then it gets really complicated to figure out what people are most likely to use. Same goes for some Ethernet adapters.

We don't really have a limit about the image size. But of course a smaller image saves disk space and bandwidth to download. It's just a bit about efficiency :)

Best, -Michael

On Fri, 2018-05-11 at 20:35 -0400, Tom Rymes wrote:

Would it make sense to make the firmware files a separate download, as only those who use wireless Red or hostAPD need them?

Tom

...

On May 11, 2018, at 4:28 PM, Michael Tremer michael.tremer@ipfire.org wrote:

Hey Matthias,

are you still working on this?

I am asking because I just noticed that the new next build is about 50MB larger than the old one due to the larger kernel and loads more of firmware files.

Compressing it better would of course help to save space on disk.

Best, -Michael

...

On Sun, 2018-02-11 at 19:57 +0000, Michael Tremer wrote: Hi,

...

On Sun, 2018-02-11 at 15:35 +0000, Matthias Fischer wrote: Hi,

On 06.02.2018 02:05, Michael Tremer wrote:

...

...

... ((XZ_MEM=(($HOST_MEM)/10)*7)) && XZ_MEM=${XZ_MEM}MiB

You can write this even short:

XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"

Oneliner! Yes. I was looking for something like that... ;-)

...

I swapped the order of the multiplication and division so that you would have a more exact result.

For example if you system has 1024 MB or RAM, you divide this by 10 (102) and then multiply by 7 which gives you 714. The other order is 1024 * 7 = 7167 and then divided by 10 is 716 which is closer to the actual result of 716.8. Shouldn't matter here but just in general this is better :)

Thanks, looks better - noted for the next time!

Now it looks like this ('make.sh'):

... # Host memory is ok, calculating XZ memory XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB" XZ_OPT="--threads=0 -8 $XZ_MEM" echo XZ-Memory: $XZ_MEM echo XZ-Options: $XZ_OPT echo "XZ memory size is OK (must be at least 700MiB), starting build..." ... ["echoes" for debugging!]

This is working. '$XZ_OPT' now gives me: --threads=0 -8 5450MiB

Will there be some checks in make.sh now that check if at least 1024MB of memory are available for build? I think there should be a warning if not enough memory is available, but we should still try to build.

...

But. ;-)

...

You can put this probably next under "HOST_MEM=$(system_memory)" and then export this to the internal shell by adding the variable to lfsmake2 for the disk image and ipfiredist for the packages.

Sorry, I must confess that I don't know exactly how to do this. As I wrote: "How do I pass the 'XZ_MEM'-variable to 'cdrom' and 'Config'?" Or in this case, XZ_OPT!?

...

Then in lfs/cdrom and lfs/Config for the dist target, you just need to add the parameters to the command line.

In 'Config' I changed:

... cd /install/packages/package/tmp/ && XZ_OPT=-T0 tar -c -p --numeric- owner -J -f /install/packages/package/files.tar.xz * ...

To: ... cd /install/packages/package/tmp/ && $XZ_OPT tar -c -p --numeric-owner -J -f /install/packages/package/files.tar.xz * ...

...

Maybe it is a good idea to have a variable XZ_OPT that adds everything together and is then used in both places.

I added the other xz-options and created XZ_OPT as shown above, but in 'cdrom' there is an 'export XZ_OPT = ...'-line and I don't know how to handle these line. Is it still necessary?

No, that is no longer necessary and can be removed. We should just add it to the tar command just like it is done in lfs/Config.

...

Do I have to 'export' the XZ_OPT'-variable in 'make.sh'?

No, because the chroot environment will throw away all environment variables from the host system and then set them again. That's why we have this enterchroot() function that does this with the env command.

...

And how do I "add" this variable to 'lfsmake2' and 'ipfiredist'(- function)?

Just have a look at the functions. There should be loads of examples where we pass other variables and you just need to add this one, too.

...

Sorry if these questions sound simple, but I'm guessing a bit too much while trying to get a grip on this.

This is absolutely fine to sort this out first and then send the patch :) That's what this list is for.

Best, -Michael

...

Best, Matthias

...

Best, -Michael

...

...

Looks better.

Best, Matthias

> On 03.02.2018 20:44, Matthias Fischer wrote: > Hi, > > > On 30.01.2018 21:03, Michael Tremer wrote: > > Hi, > > > > ... > > > > > > I'll consider how I would do this and send it in as soon as I > > > get > > > a > > > grip > > > on it. > > > > Well, I added some functions already somewhere. I suppose you > > just > > need > > to > > do > > the maths to set that to X% of the host system which is just a > > simple > > multiplication of the output of ${HOST_MEM} in make.sh. > > Ok, I take my heart in my hand. Here are my first results - but > remember, > I'm not familiar with this. I built a short test script which > looks as > if > it > is working as wanted, but... > > > And that needs to be passed to the lfs scripts that are supposed > > to > > make > > the > > images. > > ...this is a problem I'm struggling with right now: > How do I pass the 'XZ_MEM'-variable to 'cdrom' and 'Config'? > > I think this can be done using 'export XZ_MEM' or something else > but > I'm > not > sure. > > This is the script I'm testing with: > > ***SNIP*** > #!/bin/bash > > system_memory() { > local key val unit > > while read -r key val unit; do > case "${key}" in > MemTotal:*) > # Convert to MB > echo "$(( ${val} / 1024 ))" > break > ;; > esac > done < /proc/meminfo > } > > HOST_MEM=$(system_memory) > > # Checking host memory > if [ $HOST_MEM -ge 1024 ]; then > echo Host-Memory: $HOST_MEM'MiB' > echo "Host memory is OK (must be at least 1024MiB), starting > build." > else > echo Host-Memory: $HOST_MEM'MiB' > echo "Host memory too low (less than 1024MiB), building > stopped." > exit 1 > fi > > # Host memory is ok, calculating needed XZ memory to about 70% of > host > memory > ((XZ_MEM=(($HOST_MEM)/10)*7)) > XZ_MEM=$XZ_MEM'Mib' > echo XZ-Memory: $XZ_MEM > echo "XZ memory size is OK (must be at least 700MiB), starting > build." > ***SNAP*** > > The 'echoes' are mostly for debugging. > > Calculation looks ok, 'MiB' must added without blanks (XZ man > page: > "In > most > places where an integer argument is expected, an optional suffix > is > supported > to easily indicate large integers. There must be no space between > the > integer > and the suffix.") > > > Please send questions where ever you need help. There is no > > reason > > to do > > this > > alone. > > Please see above: > > 1. Is this code what you where thinking of? I would leave the > other > parameters > as they are (--threads=0 -8). > > 2. Where should I add such code in the first place? I'd like to > start > with > 'make.sh', so building stops as soon as possible if there is not > enough > memory. Where's > the best place? Right after function 'system_memory() {...}'? > > 3. How to I pass contents of 'XZ_MEM' to 'cdrom' and 'Config'? > I consider something like: > ... > export XZ_OPT = --threads=0 -8 --memory=$XZ_MEM > ... > > Best, > Matthias > > > Best, > > -Michael > > > > > > > > Best, > > > Matthias > > > > Best, > > > > -Michael > > > > > > > > > On Wed, 2017-11-22 at 16:37 +0000, Michael Tremer wrote: > > > > > Hi, > > > > > > > > > > > On Tue, 2017-11-21 at 18:35 +0100, Matthias Fischer > > > > > > wrote: > > > > > > Hi, > > > > > > > > > > > > > On 21.11.2017 15:42, Michael Tremer wrote: > > > > > > > Hello, > > > > > > > > > > > > > > any progress on this? > > > > > > > > > > > > Not much, because I've been struggling with a flu for > > > > > > the > > > > > > past > > > > > > two > > > > > > weeks, and its not over yet, but: > > > > > > > > > > > > In the meantime I ran several builds with different > > > > > > parameters. > > > > > > > > > > > > IMHO we could use: > > > > > > > > > > > > 'export XZ_OPT = --threads=0 -8 --memory=700MiB' > > > > > > > > > > > > in 'lfs/cdrom' > > > > > > > > > > > > and > > > > > > > > > > > > cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8 -- > > > > > > memory=700MiB" tar > > > > > > -c > > > > > > -p > > > > > > --numeric-owner -J -f > > > > > > /install/packages/package/files.tar.xz > > > > > > * > > > > > > > > > > > > in 'lfs/Config'. > > > > > > > > > > > > Using '9' or '--extreme'-parameter brought no further > > > > > > improvements, > > > > > > on the other hand, build time and RAM consumption > > > > > > increased > > > > > > strongly. With '-8', minimal RAM consumption was about > > > > > > 680 > > > > > > MiB, > > > > > > using > > > > > > '700 Mib' was always safe. So this is the minimum I > > > > > > would > > > > > > recommend. > > > > > > > > > > Yes, I observed the same thing. -9 compresses a little bit > > > > > better > > > > > and > > > > > extreme > > > > > doesn't change a bit. > > > > > > > > > > -8 seems to be optimal since it won't require too much > > > > > memory > > > > > to > > > > > decompress > > > > > either. So let's go with that. > > > > > > > > > > > Results: > > > > > > > > > > > > (Buildtime: 5 hours 20 minutes 59 seconds) > > > > > > > > > > > > 255866764 ipfire-2.19.1gb-ext4.i586-full-core117.img.gz > > > > > > 252262006 ipfire-2.19.1gb-ext4-scon.i586-full- > > > > > > core117.img.gz > > > > > > 175112192 ipfire-2.19.i586-full-core117.iso > > > > > > > > > > > > > -Michael > > > > > > > > > > > > > > > On Tue, 2017-11-07 at 22:47 +0000, Michael Tremer > > > > > > > > wrote: > > > > > > > > Hi, > > > > > > > > > > > > > > > > On Tue, 2017-11-07 at 19:41 +0100, Matthias Fischer > > > > > > > > wrote: > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > On 07.11.2017 16:05, Michael Tremer wrote: > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > maybe this function helps: > > > > > > > > > > > > > > > > > > > > https://git.ipfire.org/?p=ipfire-2.x.git;a=commi > > > > > > > > > > tdif > > > > > > > > > > f;h= > > > > > > > > > > 51 > > > > > > > > > > 90eea2 > > > > > > > > > > 4f98 > > > > > > > > > > 22 > > > > > > > > > > a63d > > > > > > > > > > c5d06d214b48f973b14f29 > > > > > > > > > > > > > > > > > > Thanks. I'll take a look... > > > > > > > > > > > > This would help, indeed, but how to implement(!?), > > > > > > and... > > > > > > (see > > > > > > below) > > > > > > > > > > Just run it somewhere in the script, figure out what is > > > > > about > > > > > 80% > > > > > of > > > > > that > > > > > and > > > > > if > > > > > that is smaller than 800 MB, we should just fail. > > > > > > > > > > It would be good to have a function that determines the xz > > > > > parameters and > > > > > we > > > > > use > > > > > that where ever we call xz. We don't want any duplicated > > > > > code. > > > > > > > > > > > > > > > On Tue, 2017-11-07 at 10:46 +0000, Michael > > > > > > > > > > Tremer > > > > > > > > > > wrote: > > > > > > > > > > > On Mon, 2017-11-06 at 22:09 +0100, Matthias > > > > > > > > > > > Fischer > > > > > > > > > > > wrote: > > > > > > > > > > > ... > > > > > > > > > > > > > > > > > > > > > > > > I forgot: Tuning the XZ-parameters... ;-) > > > > > > > > > > > > > > > > > > > > > > Good that you are bringing this up. So I think > > > > > > > > > > > what we > > > > > > > > > > > should > > > > > > > > > > > do > > > > > > > > > > > here is > > > > > > > > > > > bringing back XZ compression with -8 so that > > > > > > > > > > > we do > > > > > > > > > > > not > > > > > > > > > > > waste > > > > > > > > > > > too > > > > > > > > > > > much > > > > > > > > > > > space > > > > > > > > > > > when > > > > > > > > > > > extracting the image again. > > > > > > > > > > > > > > > > > > Right now I'm testing how much RAM I need. > > > > > > > > > > > > > > > > > > Current 'next'-build is running with: > > > > > > > > > > > > > > > > > > 'lfs/cdrom': > > > > > > > > > ... > > > > > > > > > export XZ_OPT = --threads=0 -8 --memory=500MiB > > > > > > > > > ... > > > > > > > > > > > > > > > > > > 'lfs/Config': > > > > > > > > > ... > > > > > > > > > cd /install/packages/package/tmp/ && XZ_OPT="-T0 > > > > > > > > > -8" > > > > > > > > > ... > > > > > > > > > ... > > > > > > > > > > > > > > > > > > I'm waiting for the results. Takes a while. > > > > > > > > > > > > > > > > You can just run it like this and it will tell you: > > > > > > > > > > > > > > > > [ms@hughes ~]$ xz -vv8 -T0 > > > > > > > > xz: Filter chain: -- > > > > > > > > lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64, > > > > > > > > mf=b > > > > > > > > t4,d > > > > > > > > ep > > > > > > > > th=0 > > > > > > > > xz: Using up to 4 threads. > > > > > > > > xz: 2,629 MiB of memory is required. The limiter is > > > > > > > > disabled. > > > > > > > > xz: Decompression will need 33 MiB of memory. > > > > > > > > xz: Compressed data cannot be written to a terminal > > > > > > > > xz: Try `xz --help' for more information. > > > > > > > > > > > > > > > > > > > To not run into memory limits we should detect > > > > > > > > > > > how > > > > > > > > > > > much > > > > > > > > > > > memory > > > > > > > > > > > the > > > > > > > > > > > build > > > > > > > > > > > host > > > > > > > > > > > has and set the memory limit to maybe 80% of > > > > > > > > > > > that. > > > > > > > > > > > xz > > > > > > > > > > > will > > > > > > > > > > > then > > > > > > > > > > > automatically > > > > > > > > > > > scale down to a number of parallel processes > > > > > > > > > > > that > > > > > > > > > > > it > > > > > > > > > > > can > > > > > > > > > > > fit > > > > > > > > > > > into > > > > > > > > > > > memory. > > > > > > > > > > > > This is something I couldn't get to work. First, I > > > > > > couldn't > > > > > > specify not > > > > > > less > > > > > > than '700 MiB'. If I specify '600 MiB', building stops > > > > > > and > > > > > > tells > > > > > > me it > > > > > > needs > > > > > > at least about 680 MiB. If I use '70%', building stops. > > > > > > And > > > > > > I > > > > > > never saw > > > > > > that > > > > > > 'xz' "automatically scaled down". It just stopped > > > > > > everytime. > > > > > > > > > > Yes, ~700M is the minimum. > > > > > > > > > > It will just reduce the number of threads again to make > > > > > sure > > > > > everything > > > > > fits > > > > > into memory. > > > > > > > > > > > > > > > > > > > > I'm a bit puzzled about this: I tried using ' > > > > > > > > > --memory=70%' > > > > > > > > > as > > > > > > > > > mentioned > > > > > > > > > in the xz man pages, but is seems to have no > > > > > > > > > effect. > > > > > > > > > It > > > > > > > > > looks as > > > > > > > > > if > > > > > > > > > the > > > > > > > > > percent parameter isn't recognized and ignored. It > > > > > > > > > always > > > > > > > > > ends up > > > > > > > > > in > > > > > > > > > "cannot allocate memory". Any hints on this? Do > > > > > > > > > both > > > > > > > > > 'lfs/cdrom' > > > > > > > > > AND > > > > > > > > > 'lfs/Config' need the '--memory'-parameter? > > > > > > > > > > > > This was the next problem: 'percent'-parameter does not > > > > > > work > > > > > > as > > > > > > expected. > > > > > > > > > > No, don't use it. Just determine yourself what it is. > > > > > > > > > > I think we could even try to set 100% of the memory, > > > > > because > > > > > as > > > > > far > > > > > as I > > > > > can > > > > > see > > > > > this is worst case and xz won't use everything that we > > > > > allow. > > > > > > > > > > If that does't work, let's scale down to 90%, or 80%, > > > > > etc... > > > > > > > > > > > > > Yes, but it would be good to have one place where > > > > > > > > the > > > > > > > > parameters are > > > > > > > > being generated and then we just use them from a > > > > > > > > variable. > > > > > > > > > > > > Probably beyond my capabilities for the moment. Any > > > > > > thoughts? > > > > > > > > > > > > > > > > > ... > > > > > > > > > > > We will at least need about half a GB which > > > > > > > > > > > should > > > > > > > > > > > be > > > > > > > > > > > a > > > > > > > > > > > sensible > > > > > > > > > > > requirement > > > > > > > > > > > for a build system any ways. > > > > > > > > > > > > I'm afraid '500 MiB' are not enough... > > > > > > > > > > No, so we will now make 1G the minimum for a build. > > > > > > > > > > I tried to build the OS on an ARM board with only 512MB of > > > > > RAM > > > > > the > > > > > other > > > > > day > > > > > and > > > > > even with a lot of swap it doesn't work. So we are past > > > > > this > > > > > any > > > > > way. > > > > > > > > > > > > > > > > > > > > Agreed. Because of this I'm testing with '500Mib'. > > > > > > > > > > > > > > > > > > > > Will you work on this bringing it into the > > > > > > > > > > > build > > > > > > > > > > > system?! > > > > > > > > > > > > > > > > > > "I'll do my very best!" :-)) > > > > > > > > > > > > > > > > Very well Miss Sophie... > > > > > > > > ... > > > > > > > > > > > > Best™, > > > > > > Matthias > > > > > > > > > > -Michael

Hi,

On Sat, 2018-05-12 at 16:14 +0200, Matthias Fischer wrote:

HI,

On 11.05.2018 22:27, Michael Tremer wrote:

...

Hey Matthias,

are you still working on this?

Its still on my list, but I must confess: I didn't get a grip on this yet.

...

I am asking because I just noticed that the new next build is about 50MB larger than the old one due to the larger kernel and loads more of firmware files.

Yes, I saw this too. I was waiting for you to ask... ;-)

:)

I'll try to explain what I did and where I'm stuck:

...

...

...

...

On 06.02.2018 02:05, Michael Tremer wrote:

...

...

... ((XZ_MEM=(($HOST_MEM)/10)*7)) && XZ_MEM=${XZ_MEM}MiB

You can write this even short:

XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"

Oneliner! Yes. I was looking for something like that... ;-)

This did work, no problem here.

...

...

...

... # Host memory is ok, calculating XZ memory XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB" XZ_OPT="--threads=0 -8 $XZ_MEM" echo XZ-Memory: $XZ_MEM echo XZ-Options: $XZ_OPT echo "XZ memory size is OK (must be at least 700MiB), starting build..." ... ["echoes" for debugging!]

This is working. '$XZ_OPT' now gives me: --threads=0 -8 5450MiB

Will there be some checks in make.sh now that check if at least 1024MB of memory are available for build? ...

For this, I changed the above ("read -p..."-lines are just for debugging):

***SNIP*** ... # Get the amount of memory in this build system HOST_MEM=$(system_memory)

# XZ-TUNING BEGIN # Checking host memory

if [ $HOST_MEM -ge 1024 ]; then print_build_stage "Host-Memory: $HOST_MEM MiB" print_build_stage "Host memory is OK (must be at least 1024 MiB), calculating XZ memory..."

# Host memory is ok, calculating XZ memory echo

XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB" XZ_OPT="--threads=0 -8 $XZ_MEM" print_build_stage "XZ-Memory:" $XZ_MEM print_build_stage "XZ-Options:" $XZ_OPT print_build_stage "XZ memory size is OK (must be at least 700 MiB), moving on..."

echo read -p "Press [Enter] key to continue..."

else

print_build_stage "Host-Memory: $HOST_MEM MiB" print_build_stage "Not enough host memory (less than 1024 MiB, consider upgrading)," print_build_stage "building will use standard XZ options." XZ_OPT="-T0" # alte Standardparameter einsetzen

echo read -p "Press [Enter] key to continue..."

fi ... # XZ-TUNING END

***SNAP***

I don't think that this needs to be so verbose. I just just ending the build script when there is less than 1024 MB of memory detected will do.

With that little memory you won't only have issues with XZ but also the rest of the build environment.

...

...

... I think there should be a warning if not enough memory is available, but we should still try to build.

As far as I can see, this could be done by replacing the above 'echoes' with 'print_build_stage'(?).

E.g.: print_build_stage "Not enough host memory (less than 1024MiB, consider upgrading)," print_build_stage "building will use standard XZ options."

...

...

...

...

But. ;-)

...

You can put this probably next under "HOST_MEM=$(system_memory)" and then export this to the internal shell by adding the variable to lfsmake2 for the disk image and ipfiredist for the packages.

Sorry, I must confess that I don't know exactly how to do this. As I wrote: "How do I pass the 'XZ_MEM'-variable to 'cdrom' and 'Config'?" Or in this case, XZ_OPT!?

And this is still my problem! I don't get this to work. How do I "pass" this variable?

...

...

...

...

Then in lfs/cdrom and lfs/Config for the dist target, you just need to add the parameters to the command line.

In 'Config' I changed:

... cd /install/packages/package/tmp/ && XZ_OPT=-T0 tar -c -p --numeric-owner -J -f /install/packages/package/files.tar.xz * ...

To: ... cd /install/packages/package/tmp/ && $XZ_OPT tar -c -p --numeric-owner -J -f /install/packages/package/files.tar.xz * ...

...

Maybe it is a good idea to have a variable XZ_OPT that adds everything together and is then used in both places.

I tried that with no luck. I added the variable '$XZ_OPT' to 'lfs/Config' in line 212:

cd /install/packages/package/tmp/ && $XZ_OPT tar -c -p --numeric-owner...

But this usually ends - like today - in '_build_packages.log' with:

***SNIP*** ... #Change xxxKVERxxx to Kernelversion sed -e "s/xxxKVERxxx/4.14.40/g" -i /install/packages/package/update.sh cd /install/packages/package && tar cf ../core-upgrade-2.19-$(basename core/121).ipfire \ update.sh files.tar.xz ROOTFILES rm -rf /install/packages/package sed -e "s/NAME/core-upgrade/g" \ -e "s/VER/2.19/g" \ -e "s/RELEASE/$(basename core/121)/g" \ -e "s/DEPS//g" \ -e "s/SIZE/`ls -l /install/packages/core-upgrade-2.19-$(basename core/121).ipfire | awk '{ print $5 }'`/g" \ < /usr/src/src/pakfire/meta > /install/packages/meta-core-upgrade-$(basename core/121) May 12 13:39:49: Building directfb DirectFB-1.7.7.tar.gz checksum OK

• cd /usr/src/lfs
• make -f directfb LFS_BASEDIR=/usr/src dist

'/usr/src/config/rootfiles/packages/i586/directfb' -> '/install/packages/package/ROOTFILES' /bin/sh: Z_OPT: command not found make: *** [directfb:60: dist] Error 127 ... ***SNAP***

Why "Z_OPT"?

Make required you to write the variable name in brackets: $(XZ_OPT) instead of $XZ_OPT. It's not shell.

...

...

...

I added the other xz-options and created XZ_OPT as shown above, but in 'cdrom' there is an 'export XZ_OPT = ...'-line and I don't know how to handle these line. Is it still necessary?

No, that is no longer necessary and can be removed. We should just add it to the tar command just like it is done in lfs/Config.

Understanding problem: What do mean with "adding it to the tar command"? For me, this sounds as if we need an "export $XZ_OPT"-line here...

Yes, unfortunately like that. Or you can just pass it before the tar command is executed. Like this:

XZ_OPT="$(XZ_OPT)" tar ...

That will also export the variable but only for that command and nothing else.

...

...

...

Do I have to 'export' the XZ_OPT'-variable in 'make.sh'?

No, because the chroot environment will throw away all environment variables from the host system and then set them again. That's why we have this enterchroot() function that does this with the env command.

...

And how do I "add" this variable to 'lfsmake2' and 'ipfiredist'(-function)?

Just have a look at the functions. There should be loads of examples where we pass other variables and you just need to add this one, too.

I think I've found these "examples" and tried several "local XZ_OPT='$XZ_OPT'" placements and versions in 'lfsmake2' and 'ipfiredist' but none seemed to work as I wanted.

Why not?

Just try to add "echo $(XZ_OPT)" to one of the lfs scripts and see if you can see the desired output.

Did you look at enterchroot() yet?

...

...

...

Sorry if these questions sound simple, but I'm guessing a bit too much while trying to get a grip on this.

This is absolutely fine to sort this out first and then send the patch :) That's what this list is for.

After several attempts I stopped guessing and concentrated on something I could handle. Somehow I've got the feeling I'm making some really simple mistakes - for reviewing I added my current test patches as attachments.

No this is really tricky. Although the patch in the will only be a few lines, this is really not just writing it down and sending it.

Please advise... ;-)

Done. Please send more questions.

Best, -Michael

Best, Matthias

Great.

Could you send the patch to the list so that we can comment on it?

Also, do you have any plans to migrate to OpenVPN 2.4? Not sure how long 2.3 will be patched.

Best, -Michael

On Fri, 2017-11-10 at 12:01 +0100, ummeegge wrote:

Hi Michael, have here some fixes from Bugzilla causing OpenVPN --> https://bugzilla.ipfire .org/show_bug.cgi?id=10482 a clean patch can be found in here --> https://gith ub.com/ummeegge/ipfire_core111_ovpn_fixes/commit/da7349461630ce9e38a9a5b47f41b c644e907ef8#diff-2011d5d928fd214cacb83844729c65cc but needs to be patched for Core 115 and this here --> https://bugzilla.ipfire.org/show_bug.cgi?id=11364 with a couples of patches whereby the "--ns-cert-type" patch is already merged .

If you think some of them are useful, let it me know ;-) .

Greetings,

Erik

Am 06.11.2017 um 19:32 schrieb Michael Tremer:

...

Hello,

so the big Core Update 115 has been shipped last week. Core Update 116 is coming today. So the next one should be a little bit calmer please :)

I do not have anything so far except some smaller bug fixes here and there. Does anyone have anything big? It doesn't have to be another big update. We could also just improve IPFire where ever it is needed with loads of smaller changes and start working on 3 and other sub-projects that are floating around...

Best, -Michael