Planning out Core Update 117

Planning out Core Update 117

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 492 bytes --]

Hello,

so the big Core Update 115 has been shipped last week. Core Update 116 is coming
today. So the next one should be a little bit calmer please :)

I do not have anything so far except some smaller bug fixes here and there. Does
anyone have anything big? It doesn't have to be another big update. We could
also just improve IPFire where ever it is needed with loads of smaller changes
and start working on 3 and other sub-projects that are floating around...

Best,
-Michael

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Planning out Core Update 117

[Matthias Fischer]

[-- Attachment #1: Type: text/plain, Size: 1109 bytes --]

Hi,

On 06.11.2017 19:32, Michael Tremer wrote:
> Hello,
> 
> so the big Core Update 115 has been shipped last week. Core Update 116 is coming
> today. So the next one should be a little bit calmer please :)

We'll see... ;-)

> I do not have anything so far except some smaller bug fixes here and there. Does
> anyone have anything big? It doesn't have to be another big update. We could
> also just improve IPFire where ever it is needed with loads of smaller changes
> and start working on 3 and other sub-projects that are floating around...

Just back in town and getting a grip - I only have a few smaller fixes
and features in mind:

'captive portal'-localization: missing german translation for
'unlimited' - patch is being prepared.

https://forum.ipfire.org/viewtopic.php?f=17&t=19792 (Mini-Bug in
'captive portal'): couldn't test this by now

Updates in progress: 'ethtool' to 4.13, 'smartmontools' to 6.6, 'tor' to
0.3.1.8 (interested?)

'squid-graph'-localization and package (if I get the time)

As I can see: nothing serious from my side... ;-)

Best,
Matthias

Re: Planning out Core Update 117

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 1507 bytes --]

Hi,

On Mon, 2017-11-06 at 22:06 +0100, Matthias Fischer wrote:
> Hi,
> 
> On 06.11.2017 19:32, Michael Tremer wrote:
> > Hello,
> > 
> > so the big Core Update 115 has been shipped last week. Core Update 116 is
> > coming
> > today. So the next one should be a little bit calmer please :)
> 
> We'll see... ;-)
> 
> > I do not have anything so far except some smaller bug fixes here and there.
> > Does
> > anyone have anything big? It doesn't have to be another big update. We could
> > also just improve IPFire where ever it is needed with loads of smaller
> > changes
> > and start working on 3 and other sub-projects that are floating around...
> 
> Just back in town and getting a grip - I only have a few smaller fixes
> and features in mind:
> 
> 'captive portal'-localization: missing german translation for
> 'unlimited' - patch is being prepared.

Okay.

> https://forum.ipfire.org/viewtopic.php?f=17&t=19792 (Mini-Bug in
> 'captive portal'): couldn't test this by now

I fixed this one already:

  https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=ad1204e4eb397b4f7d10c3bdfa9214aa8c0a0575

> Updates in progress: 'ethtool' to 4.13, 'smartmontools' to 6.6, 'tor' to
> 0.3.1.8 (interested?)

Okay. Could you also please review at least some of the patches that Marcel sent
last week?

> 'squid-graph'-localization and package (if I get the time)
> 
> As I can see: nothing serious from my side... ;-)

That's good :)

> 
> Best,
> Matthias

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Planning out Core Update 117

[Matthias Fischer]

[-- Attachment #1: Type: text/plain, Size: 2285 bytes --]

Hi,

On 07.11.2017 11:44, Michael Tremer wrote:
> ...
> Could you also please review at least some of the patches that Marcel sent
> last week?
> ...

Done.

Applied all patches (19 commits) from '2017-10-31' against the current 'next':

***SNIP***
...
On branch next
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

	modified:   config/rootfiles/common/boost
	modified:   config/rootfiles/common/cmake
	modified:   config/rootfiles/common/e2fsprogs
	modified:   config/rootfiles/common/expat
	modified:   config/rootfiles/common/gawk
	modified:   config/rootfiles/common/glib
	modified:   config/rootfiles/common/libarchive
	modified:   config/rootfiles/common/libupnp
	modified:   config/rootfiles/common/libxml2
	modified:   config/rootfiles/common/libxslt
	modified:   config/rootfiles/common/mpfr
	modified:   config/rootfiles/common/pciutils
	modified:   config/rootfiles/common/tcl
	modified:   config/rootfiles/common/texinfo
	modified:   config/rootfiles/common/xfsprogs
	modified:   config/rootfiles/packages/minidlna
	modified:   config/rootfiles/packages/nmap
	modified:   config/rootfiles/packages/stunnel
	modified:   lfs/boost
	modified:   lfs/cmake
	modified:   lfs/e2fsprogs
	modified:   lfs/expat
	modified:   lfs/gawk
	modified:   lfs/glib
	modified:   lfs/libarchive
	modified:   lfs/libupnp
	modified:   lfs/libxml2
	modified:   lfs/libxslt
	modified:   lfs/minidlna
	modified:   lfs/mpfr
	modified:   lfs/nmap
	modified:   lfs/pciutils
	modified:   lfs/rpcbind
	modified:   lfs/stunnel
	modified:   lfs/tcl
	modified:   lfs/texinfo
	modified:   lfs/xfsprogs

Untracked files:
  (use "git add <file>..." to include in what will be committed)

        src/patches/rpcbind-0.2.4-vulnerability_fixes-1.patch
...
***SNAP***

Results:

Source for 'e2fsprogs 1.43.6' were missing, copied them to git.ipfire.org. No problem.
Besides, in the meantime there's an update to '1.43.7', will test that with a clean build, too.

'poppler 0.52.0' is complaining:

Changes in poppler-0.52.0 check rootfile! [see attachment]

Besides that, building was ok for now.

HTH,
Matthias



[-- Attachment #2: poppler-0.52.0 --]
[-- Type: text/plain, Size: 8848 bytes --]

usr/bin/pdfdetach
usr/bin/pdffonts
usr/bin/pdfimages
usr/bin/pdfinfo
usr/bin/pdfseparate
usr/bin/pdftocairo
usr/bin/pdftohtml
usr/bin/pdftoppm
usr/bin/pdftops
usr/bin/pdftotext
usr/bin/pdfunite
#usr/include/poppler
#usr/include/poppler/Annot.h
#usr/include/poppler/Array.h
#usr/include/poppler/BuiltinFont.h
#usr/include/poppler/BuiltinFontTables.h
#usr/include/poppler/CMap.h
#usr/include/poppler/CachedFile.h
#usr/include/poppler/Catalog.h
#usr/include/poppler/CharCodeToUnicode.h
#usr/include/poppler/CharTypes.h
#usr/include/poppler/CompactFontTables.h
#usr/include/poppler/DateInfo.h
#usr/include/poppler/Decrypt.h
#usr/include/poppler/Dict.h
#usr/include/poppler/Error.h
#usr/include/poppler/ErrorCodes.h
#usr/include/poppler/FileSpec.h
#usr/include/poppler/FontEncodingTables.h
#usr/include/poppler/FontInfo.h
#usr/include/poppler/Form.h
#usr/include/poppler/Function.h
#usr/include/poppler/Gfx.h
#usr/include/poppler/GfxFont.h
#usr/include/poppler/GfxState.h
#usr/include/poppler/GfxState_helpers.h
#usr/include/poppler/GlobalParams.h
#usr/include/poppler/Hints.h
#usr/include/poppler/JArithmeticDecoder.h
#usr/include/poppler/JBIG2Stream.h
#usr/include/poppler/Lexer.h
#usr/include/poppler/Linearization.h
#usr/include/poppler/Link.h
#usr/include/poppler/LocalPDFDocBuilder.h
#usr/include/poppler/MarkedContentOutputDev.h
#usr/include/poppler/Movie.h
#usr/include/poppler/NameToCharCode.h
#usr/include/poppler/NameToUnicodeTable.h
#usr/include/poppler/Object.h
#usr/include/poppler/OptionalContent.h
#usr/include/poppler/Outline.h
#usr/include/poppler/OutputDev.h
#usr/include/poppler/PDFDoc.h
#usr/include/poppler/PDFDocBuilder.h
#usr/include/poppler/PDFDocEncoding.h
#usr/include/poppler/PDFDocFactory.h
#usr/include/poppler/PSOutputDev.h
#usr/include/poppler/PSTokenizer.h
#usr/include/poppler/Page.h
#usr/include/poppler/PageTransition.h
#usr/include/poppler/Parser.h
#usr/include/poppler/PopplerCache.h
#usr/include/poppler/PreScanOutputDev.h
#usr/include/poppler/ProfileData.h
#usr/include/poppler/Rendition.h
#usr/include/poppler/SecurityHandler.h
#usr/include/poppler/SignatureInfo.h
#usr/include/poppler/Sound.h
#usr/include/poppler/SplashOutputDev.h
#usr/include/poppler/StdinCachedFile.h
#usr/include/poppler/StdinPDFDocBuilder.h
#usr/include/poppler/Stream-CCITT.h
#usr/include/poppler/Stream.h
#usr/include/poppler/StructElement.h
#usr/include/poppler/StructTreeRoot.h
#usr/include/poppler/TextOutputDev.h
#usr/include/poppler/UTF.h
#usr/include/poppler/UTF8.h
#usr/include/poppler/UnicodeCClassTables.h
#usr/include/poppler/UnicodeCompTables.h
#usr/include/poppler/UnicodeDecompTables.h
#usr/include/poppler/UnicodeMap.h
#usr/include/poppler/UnicodeMapTables.h
#usr/include/poppler/UnicodeTypeTable.h
#usr/include/poppler/ViewerPreferences.h
#usr/include/poppler/XRef.h
#usr/include/poppler/XpdfPluginAPI.h
#usr/include/poppler/cpp
#usr/include/poppler/cpp/poppler-document.h
#usr/include/poppler/cpp/poppler-embedded-file.h
#usr/include/poppler/cpp/poppler-font.h
#usr/include/poppler/cpp/poppler-global.h
#usr/include/poppler/cpp/poppler-image.h
#usr/include/poppler/cpp/poppler-page-renderer.h
#usr/include/poppler/cpp/poppler-page-transition.h
#usr/include/poppler/cpp/poppler-page.h
#usr/include/poppler/cpp/poppler-rectangle.h
#usr/include/poppler/cpp/poppler-toc.h
#usr/include/poppler/cpp/poppler-version.h
#usr/include/poppler/fofi
#usr/include/poppler/fofi/FoFiBase.h
#usr/include/poppler/fofi/FoFiEncodings.h
#usr/include/poppler/fofi/FoFiIdentifier.h
#usr/include/poppler/fofi/FoFiTrueType.h
#usr/include/poppler/fofi/FoFiType1.h
#usr/include/poppler/fofi/FoFiType1C.h
+usr/include/poppler/glib
+usr/include/poppler/glib/poppler-action.h
+usr/include/poppler/glib/poppler-annot.h
+usr/include/poppler/glib/poppler-attachment.h
+usr/include/poppler/glib/poppler-date.h
+usr/include/poppler/glib/poppler-document.h
+usr/include/poppler/glib/poppler-enums.h
+usr/include/poppler/glib/poppler-features.h
+usr/include/poppler/glib/poppler-form-field.h
+usr/include/poppler/glib/poppler-layer.h
+usr/include/poppler/glib/poppler-media.h
+usr/include/poppler/glib/poppler-movie.h
+usr/include/poppler/glib/poppler-page.h
+usr/include/poppler/glib/poppler-structure-element.h
+usr/include/poppler/glib/poppler.h
#usr/include/poppler/goo
#usr/include/poppler/goo/FixedPoint.h
#usr/include/poppler/goo/GooHash.h
#usr/include/poppler/goo/GooLikely.h
#usr/include/poppler/goo/GooList.h
#usr/include/poppler/goo/GooMutex.h
#usr/include/poppler/goo/GooString.h
#usr/include/poppler/goo/GooTimer.h
#usr/include/poppler/goo/ImgWriter.h
#usr/include/poppler/goo/JpegWriter.h
#usr/include/poppler/goo/NetPBMWriter.h
#usr/include/poppler/goo/PNGWriter.h
#usr/include/poppler/goo/TiffWriter.h
#usr/include/poppler/goo/gfile.h
#usr/include/poppler/goo/gmem.h
#usr/include/poppler/goo/grandom.h
#usr/include/poppler/goo/gstrtod.h
#usr/include/poppler/goo/gtypes.h
#usr/include/poppler/poppler-config.h
#usr/include/poppler/splash
#usr/include/poppler/splash/Splash.h
#usr/include/poppler/splash/SplashBitmap.h
#usr/include/poppler/splash/SplashClip.h
#usr/include/poppler/splash/SplashErrorCodes.h
#usr/include/poppler/splash/SplashFTFont.h
#usr/include/poppler/splash/SplashFTFontEngine.h
#usr/include/poppler/splash/SplashFTFontFile.h
#usr/include/poppler/splash/SplashFont.h
#usr/include/poppler/splash/SplashFontEngine.h
#usr/include/poppler/splash/SplashFontFile.h
#usr/include/poppler/splash/SplashFontFileID.h
#usr/include/poppler/splash/SplashGlyphBitmap.h
#usr/include/poppler/splash/SplashMath.h
#usr/include/poppler/splash/SplashPath.h
#usr/include/poppler/splash/SplashPattern.h
#usr/include/poppler/splash/SplashScreen.h
#usr/include/poppler/splash/SplashState.h
#usr/include/poppler/splash/SplashT1Font.h
#usr/include/poppler/splash/SplashT1FontEngine.h
#usr/include/poppler/splash/SplashT1FontFile.h
#usr/include/poppler/splash/SplashTypes.h
#usr/include/poppler/splash/SplashXPath.h
#usr/include/poppler/splash/SplashXPathScanner.h
#usr/lib/libpoppler-cpp.a
#usr/lib/libpoppler-cpp.la
#usr/lib/libpoppler-cpp.so
usr/lib/libpoppler-cpp.so.0
usr/lib/libpoppler-cpp.so.0.3.0
+usr/lib/libpoppler-glib.a
+usr/lib/libpoppler-glib.la
+usr/lib/libpoppler-glib.so
+usr/lib/libpoppler-glib.so.8
+usr/lib/libpoppler-glib.so.8.8.0
#usr/lib/libpoppler.a
#usr/lib/libpoppler.la
#usr/lib/libpoppler.so
usr/lib/libpoppler.so.66
usr/lib/libpoppler.so.66.0.0
#usr/lib/pkgconfig/poppler-cairo.pc
#usr/lib/pkgconfig/poppler-cpp.pc
+usr/lib/pkgconfig/poppler-glib.pc
#usr/lib/pkgconfig/poppler-splash.pc
#usr/lib/pkgconfig/poppler.pc
+usr/share/gtk-doc/html/poppler
+usr/share/gtk-doc/html/poppler/PopplerAnnot.html
+usr/share/gtk-doc/html/poppler/PopplerAttachment.html
+usr/share/gtk-doc/html/poppler/PopplerDocument.html
+usr/share/gtk-doc/html/poppler/PopplerFormField.html
+usr/share/gtk-doc/html/poppler/PopplerLayer.html
+usr/share/gtk-doc/html/poppler/PopplerMedia.html
+usr/share/gtk-doc/html/poppler/PopplerMovie.html
+usr/share/gtk-doc/html/poppler/PopplerPage.html
+usr/share/gtk-doc/html/poppler/PopplerStructureElement.html
+usr/share/gtk-doc/html/poppler/annotation-glossary.html
+usr/share/gtk-doc/html/poppler/api-index-0-12.html
+usr/share/gtk-doc/html/poppler/api-index-0-14.html
+usr/share/gtk-doc/html/poppler/api-index-0-16.html
+usr/share/gtk-doc/html/poppler/api-index-0-18.html
+usr/share/gtk-doc/html/poppler/api-index-0-20.html
+usr/share/gtk-doc/html/poppler/api-index-0-22.html
+usr/share/gtk-doc/html/poppler/api-index-0-26.html
+usr/share/gtk-doc/html/poppler/api-index-0-33.html
+usr/share/gtk-doc/html/poppler/api-index-0-46.html
+usr/share/gtk-doc/html/poppler/api-index-deprecated.html
+usr/share/gtk-doc/html/poppler/api-index-full.html
+usr/share/gtk-doc/html/poppler/ch01.html
+usr/share/gtk-doc/html/poppler/home.png
+usr/share/gtk-doc/html/poppler/index.html
+usr/share/gtk-doc/html/poppler/left-insensitive.png
+usr/share/gtk-doc/html/poppler/left.png
+usr/share/gtk-doc/html/poppler/poppler-Error-handling.html
+usr/share/gtk-doc/html/poppler/poppler-PDF-Utility-functions.html
+usr/share/gtk-doc/html/poppler/poppler-PopplerAction.html
+usr/share/gtk-doc/html/poppler/poppler-PopplerColor.html
+usr/share/gtk-doc/html/poppler/poppler-Version-and-Features-Information.html
+usr/share/gtk-doc/html/poppler/poppler.devhelp2
+usr/share/gtk-doc/html/poppler/right-insensitive.png
+usr/share/gtk-doc/html/poppler/right.png
+usr/share/gtk-doc/html/poppler/style.css
+usr/share/gtk-doc/html/poppler/up-insensitive.png
+usr/share/gtk-doc/html/poppler/up.png
#usr/share/man/man1/pdfdetach.1
#usr/share/man/man1/pdffonts.1
#usr/share/man/man1/pdfimages.1
#usr/share/man/man1/pdfinfo.1
#usr/share/man/man1/pdfseparate.1
#usr/share/man/man1/pdftocairo.1
#usr/share/man/man1/pdftohtml.1
#usr/share/man/man1/pdftoppm.1
#usr/share/man/man1/pdftops.1
#usr/share/man/man1/pdftotext.1
#usr/share/man/man1/pdfunite.1

Re: Planning out Core Update 117

[Matthias Fischer]

[-- Attachment #1: Type: text/plain, Size: 3189 bytes --]

Hi,

On 17.11.2017 13:59, Matthias Fischer wrote:
> Hi,
> 
> On 07.11.2017 11:44, Michael Tremer wrote:
>> ...
>> Could you also please review at least some of the patches that Marcel sent
>> last week?
>> ...
> 
> Done.
> 
> Applied all patches (19 commits) from '2017-10-31' against the current 'next':
> 
> ***SNIP***
> ...
> On branch next
> Changes not staged for commit:
>   (use "git add <file>..." to update what will be committed)
>   (use "git checkout -- <file>..." to discard changes in working directory)
> 
> 	modified:   config/rootfiles/common/boost
> 	modified:   config/rootfiles/common/cmake
> 	modified:   config/rootfiles/common/e2fsprogs
> 	modified:   config/rootfiles/common/expat
> 	modified:   config/rootfiles/common/gawk
> 	modified:   config/rootfiles/common/glib
> 	modified:   config/rootfiles/common/libarchive
> 	modified:   config/rootfiles/common/libupnp
> 	modified:   config/rootfiles/common/libxml2
> 	modified:   config/rootfiles/common/libxslt
> 	modified:   config/rootfiles/common/mpfr
> 	modified:   config/rootfiles/common/pciutils
> 	modified:   config/rootfiles/common/tcl
> 	modified:   config/rootfiles/common/texinfo
> 	modified:   config/rootfiles/common/xfsprogs
> 	modified:   config/rootfiles/packages/minidlna
> 	modified:   config/rootfiles/packages/nmap
> 	modified:   config/rootfiles/packages/stunnel
> 	modified:   lfs/boost
> 	modified:   lfs/cmake
> 	modified:   lfs/e2fsprogs
> 	modified:   lfs/expat
> 	modified:   lfs/gawk
> 	modified:   lfs/glib
> 	modified:   lfs/libarchive
> 	modified:   lfs/libupnp
> 	modified:   lfs/libxml2
> 	modified:   lfs/libxslt
> 	modified:   lfs/minidlna
> 	modified:   lfs/mpfr
> 	modified:   lfs/nmap
> 	modified:   lfs/pciutils
> 	modified:   lfs/rpcbind
> 	modified:   lfs/stunnel
> 	modified:   lfs/tcl
> 	modified:   lfs/texinfo
> 	modified:   lfs/xfsprogs
> 
> Untracked files:
>   (use "git add <file>..." to include in what will be committed)
> 
>         src/patches/rpcbind-0.2.4-vulnerability_fixes-1.patch
> ...
> ***SNAP***
> 
> Results:
> 
> Source for 'e2fsprogs 1.43.6' were missing, copied them to git.ipfire.org. No problem.
> Besides, in the meantime there's an update to '1.43.7', will test that with a clean build, too.
> 
> 'poppler 0.52.0' is complaining:
> 
> Changes in poppler-0.52.0 check rootfile! [see attachment]
> 
> Besides that, building was ok for now.

Final results:

All patches applied - 'poppler 0.52.0' needs rootfile fix.

The 'e2fsprogs'-update to 1.43.7 also applied - no further changes seem
to be necessary for this update so far.

@Marcel: would you do these changes or shall I?

Found by chance:

I found about 70 'configure: WARNING: unrecognized
options:'-notifications in '_build.base.log' and 'build.ipfire.log'.

These deal with '--enable-mpbsd', '--disable-nls' (gmp-6.1.2,
mpfr-3.1.6, e.g.), '--disable-evms', --enable-multibyte'
'--enable-zlib', '--disable-static' , '--disable-kernel-module' and so on.

I don't know why these option switches are still included or why they
exist, but: shouldn't we take a look at this?

Best,
Matthias

Re: Planning out Core Update 117

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 3531 bytes --]

Hi,

and when you apply only a few? Are there any conflicts to expect?

On Fri, 2017-11-17 at 20:37 +0100, Matthias Fischer wrote:
> Hi,
> 
> On 17.11.2017 13:59, Matthias Fischer wrote:
> > Hi,
> > 
> > On 07.11.2017 11:44, Michael Tremer wrote:
> > > ...
> > > Could you also please review at least some of the patches that Marcel sent
> > > last week?
> > > ...
> > 
> > Done.
> > 
> > Applied all patches (19 commits) from '2017-10-31' against the current 'next':
> > 
> > ***SNIP***
> > ...
> > On branch next
> > Changes not staged for commit:
> >   (use "git add <file>..." to update what will be committed)
> >   (use "git checkout -- <file>..." to discard changes in working directory)
> > 
> > 	modified:   config/rootfiles/common/boost
> > 	modified:   config/rootfiles/common/cmake
> > 	modified:   config/rootfiles/common/e2fsprogs
> > 	modified:   config/rootfiles/common/expat
> > 	modified:   config/rootfiles/common/gawk
> > 	modified:   config/rootfiles/common/glib
> > 	modified:   config/rootfiles/common/libarchive
> > 	modified:   config/rootfiles/common/libupnp
> > 	modified:   config/rootfiles/common/libxml2
> > 	modified:   config/rootfiles/common/libxslt
> > 	modified:   config/rootfiles/common/mpfr
> > 	modified:   config/rootfiles/common/pciutils
> > 	modified:   config/rootfiles/common/tcl
> > 	modified:   config/rootfiles/common/texinfo
> > 	modified:   config/rootfiles/common/xfsprogs
> > 	modified:   config/rootfiles/packages/minidlna
> > 	modified:   config/rootfiles/packages/nmap
> > 	modified:   config/rootfiles/packages/stunnel
> > 	modified:   lfs/boost
> > 	modified:   lfs/cmake
> > 	modified:   lfs/e2fsprogs
> > 	modified:   lfs/expat
> > 	modified:   lfs/gawk
> > 	modified:   lfs/glib
> > 	modified:   lfs/libarchive
> > 	modified:   lfs/libupnp
> > 	modified:   lfs/libxml2
> > 	modified:   lfs/libxslt
> > 	modified:   lfs/minidlna
> > 	modified:   lfs/mpfr
> > 	modified:   lfs/nmap
> > 	modified:   lfs/pciutils
> > 	modified:   lfs/rpcbind
> > 	modified:   lfs/stunnel
> > 	modified:   lfs/tcl
> > 	modified:   lfs/texinfo
> > 	modified:   lfs/xfsprogs
> > 
> > Untracked files:
> >   (use "git add <file>..." to include in what will be committed)
> > 
> >         src/patches/rpcbind-0.2.4-vulnerability_fixes-1.patch
> > ...
> > ***SNAP***
> > 
> > Results:
> > 
> > Source for 'e2fsprogs 1.43.6' were missing, copied them to git.ipfire.org. No problem.
> > Besides, in the meantime there's an update to '1.43.7', will test that with a clean build, too.
> > 
> > 'poppler 0.52.0' is complaining:
> > 
> > Changes in poppler-0.52.0 check rootfile! [see attachment]
> > 
> > Besides that, building was ok for now.
> 
> Final results:
> 
> All patches applied - 'poppler 0.52.0' needs rootfile fix.
> 
> The 'e2fsprogs'-update to 1.43.7 also applied - no further changes seem
> to be necessary for this update so far.
> 
> @Marcel: would you do these changes or shall I?
> 
> Found by chance:
> 
> I found about 70 'configure: WARNING: unrecognized
> options:'-notifications in '_build.base.log' and 'build.ipfire.log'.
> 
> These deal with '--enable-mpbsd', '--disable-nls' (gmp-6.1.2,
> mpfr-3.1.6, e.g.), '--disable-evms', --enable-multibyte'
> '--enable-zlib', '--disable-static' , '--disable-kernel-module' and so on.
> 
> I don't know why these option switches are still included or why they
> exist, but: shouldn't we take a look at this?
> 
> Best,
> Matthias

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Planning out Core Update 117

[Matthias Fischer]

[-- Attachment #1: Type: text/plain, Size: 4186 bytes --]

Hi,

On 17.11.2017 22:25, Michael Tremer wrote:
> Hi,
> 
> and when you apply only a few? Are there any conflicts to expect?

Good question, I thought something like that. It's really a lot. But
sorry, my knowledge isn't good enough to be able to assess whether
conflicts will arise. BUILDING was ok, but BEHAVIOUR during productive
use? Don't know, sorry. Hopefully Marcel has tested this prior to
comitting? Any comments from him?

Surely I could test only with "only a few". Which ones would you have in
mind? For a start, I can't assess the 'packages' - I never used any of
these.

Best,
Matthias

> On Fri, 2017-11-17 at 20:37 +0100, Matthias Fischer wrote:
>> Hi,
>> 
>> On 17.11.2017 13:59, Matthias Fischer wrote:
>> > Hi,
>> > 
>> > On 07.11.2017 11:44, Michael Tremer wrote:
>> > > ...
>> > > Could you also please review at least some of the patches that Marcel sent
>> > > last week?
>> > > ...
>> > 
>> > Done.
>> > 
>> > Applied all patches (19 commits) from '2017-10-31' against the current 'next':
>> > 
>> > ***SNIP***
>> > ...
>> > On branch next
>> > Changes not staged for commit:
>> >   (use "git add <file>..." to update what will be committed)
>> >   (use "git checkout -- <file>..." to discard changes in working directory)
>> > 
>> > 	modified:   config/rootfiles/common/boost
>> > 	modified:   config/rootfiles/common/cmake
>> > 	modified:   config/rootfiles/common/e2fsprogs
>> > 	modified:   config/rootfiles/common/expat
>> > 	modified:   config/rootfiles/common/gawk
>> > 	modified:   config/rootfiles/common/glib
>> > 	modified:   config/rootfiles/common/libarchive
>> > 	modified:   config/rootfiles/common/libupnp
>> > 	modified:   config/rootfiles/common/libxml2
>> > 	modified:   config/rootfiles/common/libxslt
>> > 	modified:   config/rootfiles/common/mpfr
>> > 	modified:   config/rootfiles/common/pciutils
>> > 	modified:   config/rootfiles/common/tcl
>> > 	modified:   config/rootfiles/common/texinfo
>> > 	modified:   config/rootfiles/common/xfsprogs
>> > 	modified:   config/rootfiles/packages/minidlna
>> > 	modified:   config/rootfiles/packages/nmap
>> > 	modified:   config/rootfiles/packages/stunnel
>> > 	modified:   lfs/boost
>> > 	modified:   lfs/cmake
>> > 	modified:   lfs/e2fsprogs
>> > 	modified:   lfs/expat
>> > 	modified:   lfs/gawk
>> > 	modified:   lfs/glib
>> > 	modified:   lfs/libarchive
>> > 	modified:   lfs/libupnp
>> > 	modified:   lfs/libxml2
>> > 	modified:   lfs/libxslt
>> > 	modified:   lfs/minidlna
>> > 	modified:   lfs/mpfr
>> > 	modified:   lfs/nmap
>> > 	modified:   lfs/pciutils
>> > 	modified:   lfs/rpcbind
>> > 	modified:   lfs/stunnel
>> > 	modified:   lfs/tcl
>> > 	modified:   lfs/texinfo
>> > 	modified:   lfs/xfsprogs
>> > 
>> > Untracked files:
>> >   (use "git add <file>..." to include in what will be committed)
>> > 
>> >         src/patches/rpcbind-0.2.4-vulnerability_fixes-1.patch
>> > ...
>> > ***SNAP***
>> > 
>> > Results:
>> > 
>> > Source for 'e2fsprogs 1.43.6' were missing, copied them to git.ipfire.org. No problem.
>> > Besides, in the meantime there's an update to '1.43.7', will test that with a clean build, too.
>> > 
>> > 'poppler 0.52.0' is complaining:
>> > 
>> > Changes in poppler-0.52.0 check rootfile! [see attachment]
>> > 
>> > Besides that, building was ok for now.
>> 
>> Final results:
>> 
>> All patches applied - 'poppler 0.52.0' needs rootfile fix.
>> 
>> The 'e2fsprogs'-update to 1.43.7 also applied - no further changes seem
>> to be necessary for this update so far.
>> 
>> @Marcel: would you do these changes or shall I?
>> 
>> Found by chance:
>> 
>> I found about 70 'configure: WARNING: unrecognized
>> options:'-notifications in '_build.base.log' and 'build.ipfire.log'.
>> 
>> These deal with '--enable-mpbsd', '--disable-nls' (gmp-6.1.2,
>> mpfr-3.1.6, e.g.), '--disable-evms', --enable-multibyte'
>> '--enable-zlib', '--disable-static' , '--disable-kernel-module' and so on.
>> 
>> I don't know why these option switches are still included or why they
>> exist, but: shouldn't we take a look at this?
>> 
>> Best,
>> Matthias
> 

Re: Planning out Core Update 117

[Marcel Lorenz]

[-- Attachment #1: Type: text/plain, Size: 3618 bytes --]

Hi Matthias, hi Michael,

my test images work with this configure warnings. I think, we can 
certain remove
this old switches. I will send v2 patches for gmp/e2fsprogs and a 
cleanup patch for mpfr.

Some other packages brings this warnings: autoconf, automake, gzip, 
libtool, m4, patch

Best,
Marcel

Am 2017-11-17 20:37, schrieb Matthias Fischer:
> Hi,
> 
> On 17.11.2017 13:59, Matthias Fischer wrote:
>> Hi,
>> 
>> On 07.11.2017 11:44, Michael Tremer wrote:
>>> ...
>>> Could you also please review at least some of the patches that Marcel 
>>> sent
>>> last week?
>>> ...
>> 
>> Done.
>> 
>> Applied all patches (19 commits) from '2017-10-31' against the current 
>> 'next':
>> 
>> ***SNIP***
>> ...
>> On branch next
>> Changes not staged for commit:
>>   (use "git add <file>..." to update what will be committed)
>>   (use "git checkout -- <file>..." to discard changes in working 
>> directory)
>> 
>> 	modified:   config/rootfiles/common/boost
>> 	modified:   config/rootfiles/common/cmake
>> 	modified:   config/rootfiles/common/e2fsprogs
>> 	modified:   config/rootfiles/common/expat
>> 	modified:   config/rootfiles/common/gawk
>> 	modified:   config/rootfiles/common/glib
>> 	modified:   config/rootfiles/common/libarchive
>> 	modified:   config/rootfiles/common/libupnp
>> 	modified:   config/rootfiles/common/libxml2
>> 	modified:   config/rootfiles/common/libxslt
>> 	modified:   config/rootfiles/common/mpfr
>> 	modified:   config/rootfiles/common/pciutils
>> 	modified:   config/rootfiles/common/tcl
>> 	modified:   config/rootfiles/common/texinfo
>> 	modified:   config/rootfiles/common/xfsprogs
>> 	modified:   config/rootfiles/packages/minidlna
>> 	modified:   config/rootfiles/packages/nmap
>> 	modified:   config/rootfiles/packages/stunnel
>> 	modified:   lfs/boost
>> 	modified:   lfs/cmake
>> 	modified:   lfs/e2fsprogs
>> 	modified:   lfs/expat
>> 	modified:   lfs/gawk
>> 	modified:   lfs/glib
>> 	modified:   lfs/libarchive
>> 	modified:   lfs/libupnp
>> 	modified:   lfs/libxml2
>> 	modified:   lfs/libxslt
>> 	modified:   lfs/minidlna
>> 	modified:   lfs/mpfr
>> 	modified:   lfs/nmap
>> 	modified:   lfs/pciutils
>> 	modified:   lfs/rpcbind
>> 	modified:   lfs/stunnel
>> 	modified:   lfs/tcl
>> 	modified:   lfs/texinfo
>> 	modified:   lfs/xfsprogs
>> 
>> Untracked files:
>>   (use "git add <file>..." to include in what will be committed)
>> 
>>         src/patches/rpcbind-0.2.4-vulnerability_fixes-1.patch
>> ...
>> ***SNAP***
>> 
>> Results:
>> 
>> Source for 'e2fsprogs 1.43.6' were missing, copied them to 
>> git.ipfire.org. No problem.
>> Besides, in the meantime there's an update to '1.43.7', will test that 
>> with a clean build, too.
>> 
>> 'poppler 0.52.0' is complaining:
>> 
>> Changes in poppler-0.52.0 check rootfile! [see attachment]
>> 
>> Besides that, building was ok for now.
> 
> Final results:
> 
> All patches applied - 'poppler 0.52.0' needs rootfile fix.
> 
> The 'e2fsprogs'-update to 1.43.7 also applied - no further changes seem
> to be necessary for this update so far.
> 
> @Marcel: would you do these changes or shall I?
> 
> Found by chance:
> 
> I found about 70 'configure: WARNING: unrecognized
> options:'-notifications in '_build.base.log' and 'build.ipfire.log'.
> 
> These deal with '--enable-mpbsd', '--disable-nls' (gmp-6.1.2,
> mpfr-3.1.6, e.g.), '--disable-evms', --enable-multibyte'
> '--enable-zlib', '--disable-static' , '--disable-kernel-module' and so 
> on.
> 
> I don't know why these option switches are still included or why they
> exist, but: shouldn't we take a look at this?
> 
> Best,
> Matthias

-- 
Gesendet mit Roundcube

Re: Planning out Core Update 117

[Matthias Fischer]

[-- Attachment #1: Type: text/plain, Size: 599 bytes --]

On 06.11.2017 19:32, Michael Tremer wrote:
> Hello,
> 
> so the big Core Update 115 has been shipped last week. Core Update 116 is coming
> today. So the next one should be a little bit calmer please :)
> 
> I do not have anything so far except some smaller bug fixes here and there. Does
> anyone have anything big? It doesn't have to be another big update. We could
> also just improve IPFire where ever it is needed with loads of smaller changes
> and start working on 3 and other sub-projects that are floating around...

I forgot: Tuning the XZ-parameters... ;-)

Best,
Matthias

Re: Planning out Core Update 117

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 1311 bytes --]

On Mon, 2017-11-06 at 22:09 +0100, Matthias Fischer wrote:
> On 06.11.2017 19:32, Michael Tremer wrote:
> > Hello,
> > 
> > so the big Core Update 115 has been shipped last week. Core Update 116 is
> > coming
> > today. So the next one should be a little bit calmer please :)
> > 
> > I do not have anything so far except some smaller bug fixes here and there.
> > Does
> > anyone have anything big? It doesn't have to be another big update. We could
> > also just improve IPFire where ever it is needed with loads of smaller
> > changes
> > and start working on 3 and other sub-projects that are floating around...
> 
> I forgot: Tuning the XZ-parameters... ;-)

Good that you are bringing this up. So I think what we should do here is
bringing back XZ compression with -8 so that we do not waste too much space when
extracting the image again.

To not run into memory limits we should detect how much memory the build host
has and set the memory limit to maybe 80% of that. xz will then automatically
scale down to a number of parallel processes that it can fit into memory.

We will at least need about half a GB which should be a sensible requirement for
a build system any ways.

Will you work on this bringing it into the build system?!

-Michael

> 
> Best,
> Matthias

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Planning out Core Update 117

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 1596 bytes --]

Hi,

maybe this function helps:

https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=5190eea24f9822a63dc5d06d214b48f973b14f29

On Tue, 2017-11-07 at 10:46 +0000, Michael Tremer wrote:
> On Mon, 2017-11-06 at 22:09 +0100, Matthias Fischer wrote:
> > On 06.11.2017 19:32, Michael Tremer wrote:
> > > Hello,
> > > 
> > > so the big Core Update 115 has been shipped last week. Core Update 116 is
> > > coming
> > > today. So the next one should be a little bit calmer please :)
> > > 
> > > I do not have anything so far except some smaller bug fixes here and
> > > there.
> > > Does
> > > anyone have anything big? It doesn't have to be another big update. We
> > > could
> > > also just improve IPFire where ever it is needed with loads of smaller
> > > changes
> > > and start working on 3 and other sub-projects that are floating around...
> > 
> > I forgot: Tuning the XZ-parameters... ;-)
> 
> Good that you are bringing this up. So I think what we should do here is
> bringing back XZ compression with -8 so that we do not waste too much space
> when
> extracting the image again.
> 
> To not run into memory limits we should detect how much memory the build host
> has and set the memory limit to maybe 80% of that. xz will then automatically
> scale down to a number of parallel processes that it can fit into memory.
> 
> We will at least need about half a GB which should be a sensible requirement
> for
> a build system any ways.
> 
> Will you work on this bringing it into the build system?!
> 
> -Michael
> 
> > 
> > Best,
> > Matthias

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Planning out Core Update 117

[Matthias Fischer]

[-- Attachment #1: Type: text/plain, Size: 1784 bytes --]

Hi,

On 07.11.2017 16:05, Michael Tremer wrote:
> Hi,
> 
> maybe this function helps:
> 
> https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=5190eea24f9822a63dc5d06d214b48f973b14f29

Thanks. I'll take a look...

> On Tue, 2017-11-07 at 10:46 +0000, Michael Tremer wrote:
>> On Mon, 2017-11-06 at 22:09 +0100, Matthias Fischer wrote:
>> ...
>> > 
>> > I forgot: Tuning the XZ-parameters... ;-)
>> 
>> Good that you are bringing this up. So I think what we should do here is
>> bringing back XZ compression with -8 so that we do not waste too much space
>> when
>> extracting the image again.

Right now I'm testing how much RAM I need.

Current 'next'-build is running with:

'lfs/cdrom':
...
export XZ_OPT = --threads=0 -8 --memory=500MiB
...

'lfs/Config':
...
cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8" ...
...

I'm waiting for the results. Takes a while.

>> To not run into memory limits we should detect how much memory the build host
>> has and set the memory limit to maybe 80% of that. xz will then automatically
>> scale down to a number of parallel processes that it can fit into memory.

I'm a bit puzzled about this: I tried using '--memory=70%' as mentioned
in the xz man pages, but is seems to have no effect. It looks as if the
percent parameter isn't recognized and ignored. It always ends up in
"cannot allocate memory". Any hints on this? Do both 'lfs/cdrom' AND
'lfs/Config' need the '--memory'-parameter?

>> ...
>> We will at least need about half a GB which should be a sensible requirement
>> for a build system any ways.

Agreed. Because of this I'm testing with '500Mib'.

>> Will you work on this bringing it into the build system?!

"I'll do my very best!" :-))

Best™,
Matthias

Re: Planning out Core Update 117

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 2550 bytes --]

Hi,

On Tue, 2017-11-07 at 19:41 +0100, Matthias Fischer wrote:
> Hi,
> 
> On 07.11.2017 16:05, Michael Tremer wrote:
> > Hi,
> > 
> > maybe this function helps:
> > 
> > https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=5190eea24f9822a63dc5d06d214b48f973b14f29
> 
> Thanks. I'll take a look...
> 
> > On Tue, 2017-11-07 at 10:46 +0000, Michael Tremer wrote:
> > > On Mon, 2017-11-06 at 22:09 +0100, Matthias Fischer wrote:
> > > ...
> > > > 
> > > > I forgot: Tuning the XZ-parameters... ;-)
> > > 
> > > Good that you are bringing this up. So I think what we should do here is
> > > bringing back XZ compression with -8 so that we do not waste too much space
> > > when
> > > extracting the image again.
> 
> Right now I'm testing how much RAM I need.
> 
> Current 'next'-build is running with:
> 
> 'lfs/cdrom':
> ...
> export XZ_OPT = --threads=0 -8 --memory=500MiB
> ...
> 
> 'lfs/Config':
> ...
> cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8" ...
> ...
> 
> I'm waiting for the results. Takes a while.

You can just run it like this and it will tell you:

[ms(a)hughes ~]$ xz -vv8 -T0
xz: Filter chain: --lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
xz: Using up to 4 threads.
xz: 2,629 MiB of memory is required. The limiter is disabled.
xz: Decompression will need 33 MiB of memory.
xz: Compressed data cannot be written to a terminal
xz: Try `xz --help' for more information.

> > > To not run into memory limits we should detect how much memory the build host
> > > has and set the memory limit to maybe 80% of that. xz will then automatically
> > > scale down to a number of parallel processes that it can fit into memory.
> 
> I'm a bit puzzled about this: I tried using '--memory=70%' as mentioned
> in the xz man pages, but is seems to have no effect. It looks as if the
> percent parameter isn't recognized and ignored. It always ends up in
> "cannot allocate memory". Any hints on this? Do both 'lfs/cdrom' AND
> 'lfs/Config' need the '--memory'-parameter?

Yes, but it would be good to have one place where the parameters are
being generated and then we just use them from a variable.

> > > ...
> > > We will at least need about half a GB which should be a sensible requirement
> > > for a build system any ways.
> 
> Agreed. Because of this I'm testing with '500Mib'.
> 
> > > Will you work on this bringing it into the build system?!
> 
> "I'll do my very best!" :-))

Very well Miss Sophie...

> Best™,
> Matthias

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Planning out Core Update 117

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 2868 bytes --]

Hello,

any progress on this?

-Michael

On Tue, 2017-11-07 at 22:47 +0000, Michael Tremer wrote:
> Hi,
> 
> On Tue, 2017-11-07 at 19:41 +0100, Matthias Fischer wrote:
> > Hi,
> > 
> > On 07.11.2017 16:05, Michael Tremer wrote:
> > > Hi,
> > > 
> > > maybe this function helps:
> > > 
> > > https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=5190eea24f9822a63d
> > > c5d06d214b48f973b14f29
> > 
> > Thanks. I'll take a look...
> > 
> > > On Tue, 2017-11-07 at 10:46 +0000, Michael Tremer wrote:
> > > > On Mon, 2017-11-06 at 22:09 +0100, Matthias Fischer wrote:
> > > > ...
> > > > > 
> > > > > I forgot: Tuning the XZ-parameters... ;-)
> > > > 
> > > > Good that you are bringing this up. So I think what we should do here is
> > > > bringing back XZ compression with -8 so that we do not waste too much
> > > > space
> > > > when
> > > > extracting the image again.
> > 
> > Right now I'm testing how much RAM I need.
> > 
> > Current 'next'-build is running with:
> > 
> > 'lfs/cdrom':
> > ...
> > export XZ_OPT = --threads=0 -8 --memory=500MiB
> > ...
> > 
> > 'lfs/Config':
> > ...
> > cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8" ...
> > ...
> > 
> > I'm waiting for the results. Takes a while.
> 
> You can just run it like this and it will tell you:
> 
> [ms(a)hughes ~]$ xz -vv8 -T0
> xz: Filter chain: --
> lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
> xz: Using up to 4 threads.
> xz: 2,629 MiB of memory is required. The limiter is disabled.
> xz: Decompression will need 33 MiB of memory.
> xz: Compressed data cannot be written to a terminal
> xz: Try `xz --help' for more information.
> 
> > > > To not run into memory limits we should detect how much memory the build
> > > > host
> > > > has and set the memory limit to maybe 80% of that. xz will then
> > > > automatically
> > > > scale down to a number of parallel processes that it can fit into
> > > > memory.
> > 
> > I'm a bit puzzled about this: I tried using '--memory=70%' as mentioned
> > in the xz man pages, but is seems to have no effect. It looks as if the
> > percent parameter isn't recognized and ignored. It always ends up in
> > "cannot allocate memory". Any hints on this? Do both 'lfs/cdrom' AND
> > 'lfs/Config' need the '--memory'-parameter?
> 
> Yes, but it would be good to have one place where the parameters are
> being generated and then we just use them from a variable.
> 
> > > > ...
> > > > We will at least need about half a GB which should be a sensible
> > > > requirement
> > > > for a build system any ways.
> > 
> > Agreed. Because of this I'm testing with '500Mib'.
> > 
> > > > Will you work on this bringing it into the build system?!
> > 
> > "I'll do my very best!" :-))
> 
> Very well Miss Sophie...
> 
> > Best™,
> > Matthias

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Planning out Core Update 117

[Matthias Fischer]

[-- Attachment #1: Type: text/plain, Size: 4483 bytes --]

Hi,

On 21.11.2017 15:42, Michael Tremer wrote:
> Hello,
> 
> any progress on this?

Not much, because I've been struggling with a flu for the past two
weeks, and its not over yet, but:

In the meantime I ran several builds with different parameters.

IMHO we could use:

'export XZ_OPT = --threads=0 -8 --memory=700MiB'

in 'lfs/cdrom'

and

cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8 --memory=700MiB" tar -c -p --numeric-owner -J -f /install/packages/package/files.tar.xz *

in 'lfs/Config'.

Using '9' or '--extreme'-parameter brought no further improvements,
on the other hand, build time and RAM consumption increased
strongly. With '-8', minimal RAM consumption was about 680 MiB, using
'700 Mib' was always safe. So this is the minimum I would recommend.

Results:

(Buildtime: 5 hours 20 minutes 59 seconds)

255866764  ipfire-2.19.1gb-ext4.i586-full-core117.img.gz
252262006  ipfire-2.19.1gb-ext4-scon.i586-full-core117.img.gz
175112192  ipfire-2.19.i586-full-core117.iso
 
> -Michael
> 
> On Tue, 2017-11-07 at 22:47 +0000, Michael Tremer wrote:
>> Hi,
>> 
>> On Tue, 2017-11-07 at 19:41 +0100, Matthias Fischer wrote:
>> > Hi,
>> > 
>> > On 07.11.2017 16:05, Michael Tremer wrote:
>> > > Hi,
>> > > 
>> > > maybe this function helps:
>> > > 
>> > > https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=5190eea24f9822a63d
>> > > c5d06d214b48f973b14f29
>> > 
>> > Thanks. I'll take a look...

This would help, indeed, but how to implement(!?), and... (see below)

>> > > On Tue, 2017-11-07 at 10:46 +0000, Michael Tremer wrote:
>> > > > On Mon, 2017-11-06 at 22:09 +0100, Matthias Fischer wrote:
>> > > > ...
>> > > > > 
>> > > > > I forgot: Tuning the XZ-parameters... ;-)
>> > > > 
>> > > > Good that you are bringing this up. So I think what we should do here is
>> > > > bringing back XZ compression with -8 so that we do not waste too much
>> > > > space
>> > > > when
>> > > > extracting the image again.
>> > 
>> > Right now I'm testing how much RAM I need.
>> > 
>> > Current 'next'-build is running with:
>> > 
>> > 'lfs/cdrom':
>> > ...
>> > export XZ_OPT = --threads=0 -8 --memory=500MiB
>> > ...
>> > 
>> > 'lfs/Config':
>> > ...
>> > cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8" ...
>> > ...
>> > 
>> > I'm waiting for the results. Takes a while.
>> 
>> You can just run it like this and it will tell you:
>> 
>> [ms(a)hughes ~]$ xz -vv8 -T0
>> xz: Filter chain: --
>> lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
>> xz: Using up to 4 threads.
>> xz: 2,629 MiB of memory is required. The limiter is disabled.
>> xz: Decompression will need 33 MiB of memory.
>> xz: Compressed data cannot be written to a terminal
>> xz: Try `xz --help' for more information.
>> 
>> > > > To not run into memory limits we should detect how much memory the build
>> > > > host
>> > > > has and set the memory limit to maybe 80% of that. xz will then
>> > > > automatically
>> > > > scale down to a number of parallel processes that it can fit into
>> > > > memory.

This is something I couldn't get to work. First, I couldn't specify not less
than '700 MiB'. If I specify '600 MiB', building stops and tells me it needs
at least about 680 MiB. If I use '70%', building stops. And I never saw that
'xz' "automatically scaled down". It just stopped everytime.
 
>> > I'm a bit puzzled about this: I tried using '--memory=70%' as mentioned
>> > in the xz man pages, but is seems to have no effect. It looks as if the
>> > percent parameter isn't recognized and ignored. It always ends up in
>> > "cannot allocate memory". Any hints on this? Do both 'lfs/cdrom' AND
>> > 'lfs/Config' need the '--memory'-parameter?

This was the next problem: 'percent'-parameter does not work as expected.

>> Yes, but it would be good to have one place where the parameters are
>> being generated and then we just use them from a variable.

Probably beyond my capabilities for the moment. Any thoughts?

>> > > > ...
>> > > > We will at least need about half a GB which should be a sensible
>> > > > requirement
>> > > > for a build system any ways.

I'm afraid '500 MiB' are not enough...

>> > Agreed. Because of this I'm testing with '500Mib'.
>> > 
>> > > > Will you work on this bringing it into the build system?!
>> > 
>> > "I'll do my very best!" :-))
>> 
>> Very well Miss Sophie...
>> ...

Best™,
Matthias

Re: Planning out Core Update 117

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 6070 bytes --]

Hi,

On Tue, 2017-11-21 at 18:35 +0100, Matthias Fischer wrote:
> Hi,
> 
> On 21.11.2017 15:42, Michael Tremer wrote:
> > Hello,
> > 
> > any progress on this?
> 
> Not much, because I've been struggling with a flu for the past two
> weeks, and its not over yet, but:
> 
> In the meantime I ran several builds with different parameters.
> 
> IMHO we could use:
> 
> 'export XZ_OPT = --threads=0 -8 --memory=700MiB'
> 
> in 'lfs/cdrom'
> 
> and
> 
> cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8 --memory=700MiB" tar -c -p
> --numeric-owner -J -f /install/packages/package/files.tar.xz *
> 
> in 'lfs/Config'.
> 
> Using '9' or '--extreme'-parameter brought no further improvements,
> on the other hand, build time and RAM consumption increased
> strongly. With '-8', minimal RAM consumption was about 680 MiB, using
> '700 Mib' was always safe. So this is the minimum I would recommend.

Yes, I observed the same thing. -9 compresses a little bit better and extreme
doesn't change a bit.

-8 seems to be optimal since it won't require too much memory to decompress
either. So let's go with that.

> Results:
> 
> (Buildtime: 5 hours 20 minutes 59 seconds)
> 
> 255866764  ipfire-2.19.1gb-ext4.i586-full-core117.img.gz
> 252262006  ipfire-2.19.1gb-ext4-scon.i586-full-core117.img.gz
> 175112192  ipfire-2.19.i586-full-core117.iso
>  
> > -Michael
> > 
> > On Tue, 2017-11-07 at 22:47 +0000, Michael Tremer wrote:
> > > Hi,
> > > 
> > > On Tue, 2017-11-07 at 19:41 +0100, Matthias Fischer wrote:
> > > > Hi,
> > > > 
> > > > On 07.11.2017 16:05, Michael Tremer wrote:
> > > > > Hi,
> > > > > 
> > > > > maybe this function helps:
> > > > > 
> > > > > https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=5190eea24f9822
> > > > > a63d
> > > > > c5d06d214b48f973b14f29
> > > > 
> > > > Thanks. I'll take a look...
> 
> This would help, indeed, but how to implement(!?), and... (see below)

Just run it somewhere in the script, figure out what is about 80% of that and if
that is smaller than 800 MB, we should just fail.

It would be good to have a function that determines the xz parameters and we use
that where ever we call xz. We don't want any duplicated code.

> > > > > On Tue, 2017-11-07 at 10:46 +0000, Michael Tremer wrote:
> > > > > > On Mon, 2017-11-06 at 22:09 +0100, Matthias Fischer wrote:
> > > > > > ...
> > > > > > > 
> > > > > > > I forgot: Tuning the XZ-parameters... ;-)
> > > > > > 
> > > > > > Good that you are bringing this up. So I think what we should do
> > > > > > here is
> > > > > > bringing back XZ compression with -8 so that we do not waste too
> > > > > > much
> > > > > > space
> > > > > > when
> > > > > > extracting the image again.
> > > > 
> > > > Right now I'm testing how much RAM I need.
> > > > 
> > > > Current 'next'-build is running with:
> > > > 
> > > > 'lfs/cdrom':
> > > > ...
> > > > export XZ_OPT = --threads=0 -8 --memory=500MiB
> > > > ...
> > > > 
> > > > 'lfs/Config':
> > > > ...
> > > > cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8" ...
> > > > ...
> > > > 
> > > > I'm waiting for the results. Takes a while.
> > > 
> > > You can just run it like this and it will tell you:
> > > 
> > > [ms(a)hughes ~]$ xz -vv8 -T0
> > > xz: Filter chain: --
> > > lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
> > > xz: Using up to 4 threads.
> > > xz: 2,629 MiB of memory is required. The limiter is disabled.
> > > xz: Decompression will need 33 MiB of memory.
> > > xz: Compressed data cannot be written to a terminal
> > > xz: Try `xz --help' for more information.
> > > 
> > > > > > To not run into memory limits we should detect how much memory the
> > > > > > build
> > > > > > host
> > > > > > has and set the memory limit to maybe 80% of that. xz will then
> > > > > > automatically
> > > > > > scale down to a number of parallel processes that it can fit into
> > > > > > memory.
> 
> This is something I couldn't get to work. First, I couldn't specify not less
> than '700 MiB'. If I specify '600 MiB', building stops and tells me it needs
> at least about 680 MiB. If I use '70%', building stops. And I never saw that
> 'xz' "automatically scaled down". It just stopped everytime.

Yes, ~700M is the minimum.

It will just reduce the number of threads again to make sure everything fits
into memory.

>  
> > > > I'm a bit puzzled about this: I tried using '--memory=70%' as mentioned
> > > > in the xz man pages, but is seems to have no effect. It looks as if the
> > > > percent parameter isn't recognized and ignored. It always ends up in
> > > > "cannot allocate memory". Any hints on this? Do both 'lfs/cdrom' AND
> > > > 'lfs/Config' need the '--memory'-parameter?
> 
> This was the next problem: 'percent'-parameter does not work as expected.

No, don't use it. Just determine yourself what it is.

I think we could even try to set 100% of the memory, because as far as I can see
this is worst case and xz won't use everything that we allow.

If that does't work, let's scale down to 90%, or 80%, etc...

> > > Yes, but it would be good to have one place where the parameters are
> > > being generated and then we just use them from a variable.
> 
> Probably beyond my capabilities for the moment. Any thoughts?
> 
> > > > > > ...
> > > > > > We will at least need about half a GB which should be a sensible
> > > > > > requirement
> > > > > > for a build system any ways.
> 
> I'm afraid '500 MiB' are not enough...

No, so we will now make 1G the minimum for a build.

I tried to build the OS on an ARM board with only 512MB of RAM the other day and
even with a lot of swap it doesn't work. So we are past this any way.

> 
> > > > Agreed. Because of this I'm testing with '500Mib'.
> > > > 
> > > > > > Will you work on this bringing it into the build system?!
> > > > 
> > > > "I'll do my very best!" :-))
> > > 
> > > Very well Miss Sophie...
> > > ...
> 
> Best™,
> Matthias

-Michael

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Compress images with XZ (was: Planning out Core Update 117)

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 6625 bytes --]

Hello Matthias,

any update on this? Are you still on this?

Best,
-Michael

On Wed, 2017-11-22 at 16:37 +0000, Michael Tremer wrote:
> Hi,
> 
> On Tue, 2017-11-21 at 18:35 +0100, Matthias Fischer wrote:
> > Hi,
> > 
> > On 21.11.2017 15:42, Michael Tremer wrote:
> > > Hello,
> > > 
> > > any progress on this?
> > 
> > Not much, because I've been struggling with a flu for the past two
> > weeks, and its not over yet, but:
> > 
> > In the meantime I ran several builds with different parameters.
> > 
> > IMHO we could use:
> > 
> > 'export XZ_OPT = --threads=0 -8 --memory=700MiB'
> > 
> > in 'lfs/cdrom'
> > 
> > and
> > 
> > cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8 --memory=700MiB" tar -c
> > -p
> > --numeric-owner -J -f /install/packages/package/files.tar.xz *
> > 
> > in 'lfs/Config'.
> > 
> > Using '9' or '--extreme'-parameter brought no further improvements,
> > on the other hand, build time and RAM consumption increased
> > strongly. With '-8', minimal RAM consumption was about 680 MiB, using
> > '700 Mib' was always safe. So this is the minimum I would recommend.
> 
> Yes, I observed the same thing. -9 compresses a little bit better and extreme
> doesn't change a bit.
> 
> -8 seems to be optimal since it won't require too much memory to decompress
> either. So let's go with that.
> 
> > Results:
> > 
> > (Buildtime: 5 hours 20 minutes 59 seconds)
> > 
> > 255866764  ipfire-2.19.1gb-ext4.i586-full-core117.img.gz
> > 252262006  ipfire-2.19.1gb-ext4-scon.i586-full-core117.img.gz
> > 175112192  ipfire-2.19.i586-full-core117.iso
> >  
> > > -Michael
> > > 
> > > On Tue, 2017-11-07 at 22:47 +0000, Michael Tremer wrote:
> > > > Hi,
> > > > 
> > > > On Tue, 2017-11-07 at 19:41 +0100, Matthias Fischer wrote:
> > > > > Hi,
> > > > > 
> > > > > On 07.11.2017 16:05, Michael Tremer wrote:
> > > > > > Hi,
> > > > > > 
> > > > > > maybe this function helps:
> > > > > > 
> > > > > > https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=5190eea24f98
> > > > > > 22
> > > > > > a63d
> > > > > > c5d06d214b48f973b14f29
> > > > > 
> > > > > Thanks. I'll take a look...
> > 
> > This would help, indeed, but how to implement(!?), and... (see below)
> 
> Just run it somewhere in the script, figure out what is about 80% of that and
> if
> that is smaller than 800 MB, we should just fail.
> 
> It would be good to have a function that determines the xz parameters and we
> use
> that where ever we call xz. We don't want any duplicated code.
> 
> > > > > > On Tue, 2017-11-07 at 10:46 +0000, Michael Tremer wrote:
> > > > > > > On Mon, 2017-11-06 at 22:09 +0100, Matthias Fischer wrote:
> > > > > > > ...
> > > > > > > > 
> > > > > > > > I forgot: Tuning the XZ-parameters... ;-)
> > > > > > > 
> > > > > > > Good that you are bringing this up. So I think what we should do
> > > > > > > here is
> > > > > > > bringing back XZ compression with -8 so that we do not waste too
> > > > > > > much
> > > > > > > space
> > > > > > > when
> > > > > > > extracting the image again.
> > > > > 
> > > > > Right now I'm testing how much RAM I need.
> > > > > 
> > > > > Current 'next'-build is running with:
> > > > > 
> > > > > 'lfs/cdrom':
> > > > > ...
> > > > > export XZ_OPT = --threads=0 -8 --memory=500MiB
> > > > > ...
> > > > > 
> > > > > 'lfs/Config':
> > > > > ...
> > > > > cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8" ...
> > > > > ...
> > > > > 
> > > > > I'm waiting for the results. Takes a while.
> > > > 
> > > > You can just run it like this and it will tell you:
> > > > 
> > > > [ms(a)hughes ~]$ xz -vv8 -T0
> > > > xz: Filter chain: --
> > > > lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
> > > > xz: Using up to 4 threads.
> > > > xz: 2,629 MiB of memory is required. The limiter is disabled.
> > > > xz: Decompression will need 33 MiB of memory.
> > > > xz: Compressed data cannot be written to a terminal
> > > > xz: Try `xz --help' for more information.
> > > > 
> > > > > > > To not run into memory limits we should detect how much memory the
> > > > > > > build
> > > > > > > host
> > > > > > > has and set the memory limit to maybe 80% of that. xz will then
> > > > > > > automatically
> > > > > > > scale down to a number of parallel processes that it can fit into
> > > > > > > memory.
> > 
> > This is something I couldn't get to work. First, I couldn't specify not less
> > than '700 MiB'. If I specify '600 MiB', building stops and tells me it needs
> > at least about 680 MiB. If I use '70%', building stops. And I never saw that
> > 'xz' "automatically scaled down". It just stopped everytime.
> 
> Yes, ~700M is the minimum.
> 
> It will just reduce the number of threads again to make sure everything fits
> into memory.
> 
> >  
> > > > > I'm a bit puzzled about this: I tried using '--memory=70%' as
> > > > > mentioned
> > > > > in the xz man pages, but is seems to have no effect. It looks as if
> > > > > the
> > > > > percent parameter isn't recognized and ignored. It always ends up in
> > > > > "cannot allocate memory". Any hints on this? Do both 'lfs/cdrom' AND
> > > > > 'lfs/Config' need the '--memory'-parameter?
> > 
> > This was the next problem: 'percent'-parameter does not work as expected.
> 
> No, don't use it. Just determine yourself what it is.
> 
> I think we could even try to set 100% of the memory, because as far as I can
> see
> this is worst case and xz won't use everything that we allow.
> 
> If that does't work, let's scale down to 90%, or 80%, etc...
> 
> > > > Yes, but it would be good to have one place where the parameters are
> > > > being generated and then we just use them from a variable.
> > 
> > Probably beyond my capabilities for the moment. Any thoughts?
> > 
> > > > > > > ...
> > > > > > > We will at least need about half a GB which should be a sensible
> > > > > > > requirement
> > > > > > > for a build system any ways.
> > 
> > I'm afraid '500 MiB' are not enough...
> 
> No, so we will now make 1G the minimum for a build.
> 
> I tried to build the OS on an ARM board with only 512MB of RAM the other day
> and
> even with a lot of swap it doesn't work. So we are past this any way.
> 
> > 
> > > > > Agreed. Because of this I'm testing with '500Mib'.
> > > > > 
> > > > > > > Will you work on this bringing it into the build system?!
> > > > > 
> > > > > "I'll do my very best!" :-))
> > > > 
> > > > Very well Miss Sophie...
> > > > ...
> > 
> > Best™,
> > Matthias
> 
> -Michael

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Compress images with XZ

[Matthias Fischer]

[-- Attachment #1: Type: text/plain, Size: 7294 bytes --]

On 29.01.2018 14:23, Michael Tremer wrote:
> Hello Matthias,

Hi Michael,

> any update on this? Are you still on this?

Sorry, no update, but its still on my list.

We had a tough time here with all kinds of diseases in the family during
the holidays and beyond, so I was only able to work on some 'easier jobs'.

Besides, I'm not really familiar with this kind of scripting, but I'll
try. This is the next I'm at.

I'll consider how I would do this and send it in as soon as I get a grip
on it.

Best,
Matthias
 > Best,
> -Michael
> 
> On Wed, 2017-11-22 at 16:37 +0000, Michael Tremer wrote:
>> Hi,
>> 
>> On Tue, 2017-11-21 at 18:35 +0100, Matthias Fischer wrote:
>> > Hi,
>> > 
>> > On 21.11.2017 15:42, Michael Tremer wrote:
>> > > Hello,
>> > > 
>> > > any progress on this?
>> > 
>> > Not much, because I've been struggling with a flu for the past two
>> > weeks, and its not over yet, but:
>> > 
>> > In the meantime I ran several builds with different parameters.
>> > 
>> > IMHO we could use:
>> > 
>> > 'export XZ_OPT = --threads=0 -8 --memory=700MiB'
>> > 
>> > in 'lfs/cdrom'
>> > 
>> > and
>> > 
>> > cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8 --memory=700MiB" tar -c
>> > -p
>> > --numeric-owner -J -f /install/packages/package/files.tar.xz *
>> > 
>> > in 'lfs/Config'.
>> > 
>> > Using '9' or '--extreme'-parameter brought no further improvements,
>> > on the other hand, build time and RAM consumption increased
>> > strongly. With '-8', minimal RAM consumption was about 680 MiB, using
>> > '700 Mib' was always safe. So this is the minimum I would recommend.
>> 
>> Yes, I observed the same thing. -9 compresses a little bit better and extreme
>> doesn't change a bit.
>> 
>> -8 seems to be optimal since it won't require too much memory to decompress
>> either. So let's go with that.
>> 
>> > Results:
>> > 
>> > (Buildtime: 5 hours 20 minutes 59 seconds)
>> > 
>> > 255866764  ipfire-2.19.1gb-ext4.i586-full-core117.img.gz
>> > 252262006  ipfire-2.19.1gb-ext4-scon.i586-full-core117.img.gz
>> > 175112192  ipfire-2.19.i586-full-core117.iso
>> >  
>> > > -Michael
>> > > 
>> > > On Tue, 2017-11-07 at 22:47 +0000, Michael Tremer wrote:
>> > > > Hi,
>> > > > 
>> > > > On Tue, 2017-11-07 at 19:41 +0100, Matthias Fischer wrote:
>> > > > > Hi,
>> > > > > 
>> > > > > On 07.11.2017 16:05, Michael Tremer wrote:
>> > > > > > Hi,
>> > > > > > 
>> > > > > > maybe this function helps:
>> > > > > > 
>> > > > > > https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=5190eea24f98
>> > > > > > 22
>> > > > > > a63d
>> > > > > > c5d06d214b48f973b14f29
>> > > > > 
>> > > > > Thanks. I'll take a look...
>> > 
>> > This would help, indeed, but how to implement(!?), and... (see below)
>> 
>> Just run it somewhere in the script, figure out what is about 80% of that and
>> if
>> that is smaller than 800 MB, we should just fail.
>> 
>> It would be good to have a function that determines the xz parameters and we
>> use
>> that where ever we call xz. We don't want any duplicated code.
>> 
>> > > > > > On Tue, 2017-11-07 at 10:46 +0000, Michael Tremer wrote:
>> > > > > > > On Mon, 2017-11-06 at 22:09 +0100, Matthias Fischer wrote:
>> > > > > > > ...
>> > > > > > > > 
>> > > > > > > > I forgot: Tuning the XZ-parameters... ;-)
>> > > > > > > 
>> > > > > > > Good that you are bringing this up. So I think what we should do
>> > > > > > > here is
>> > > > > > > bringing back XZ compression with -8 so that we do not waste too
>> > > > > > > much
>> > > > > > > space
>> > > > > > > when
>> > > > > > > extracting the image again.
>> > > > > 
>> > > > > Right now I'm testing how much RAM I need.
>> > > > > 
>> > > > > Current 'next'-build is running with:
>> > > > > 
>> > > > > 'lfs/cdrom':
>> > > > > ...
>> > > > > export XZ_OPT = --threads=0 -8 --memory=500MiB
>> > > > > ...
>> > > > > 
>> > > > > 'lfs/Config':
>> > > > > ...
>> > > > > cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8" ...
>> > > > > ...
>> > > > > 
>> > > > > I'm waiting for the results. Takes a while.
>> > > > 
>> > > > You can just run it like this and it will tell you:
>> > > > 
>> > > > [ms(a)hughes ~]$ xz -vv8 -T0
>> > > > xz: Filter chain: --
>> > > > lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
>> > > > xz: Using up to 4 threads.
>> > > > xz: 2,629 MiB of memory is required. The limiter is disabled.
>> > > > xz: Decompression will need 33 MiB of memory.
>> > > > xz: Compressed data cannot be written to a terminal
>> > > > xz: Try `xz --help' for more information.
>> > > > 
>> > > > > > > To not run into memory limits we should detect how much memory the
>> > > > > > > build
>> > > > > > > host
>> > > > > > > has and set the memory limit to maybe 80% of that. xz will then
>> > > > > > > automatically
>> > > > > > > scale down to a number of parallel processes that it can fit into
>> > > > > > > memory.
>> > 
>> > This is something I couldn't get to work. First, I couldn't specify not less
>> > than '700 MiB'. If I specify '600 MiB', building stops and tells me it needs
>> > at least about 680 MiB. If I use '70%', building stops. And I never saw that
>> > 'xz' "automatically scaled down". It just stopped everytime.
>> 
>> Yes, ~700M is the minimum.
>> 
>> It will just reduce the number of threads again to make sure everything fits
>> into memory.
>> 
>> >  
>> > > > > I'm a bit puzzled about this: I tried using '--memory=70%' as
>> > > > > mentioned
>> > > > > in the xz man pages, but is seems to have no effect. It looks as if
>> > > > > the
>> > > > > percent parameter isn't recognized and ignored. It always ends up in
>> > > > > "cannot allocate memory". Any hints on this? Do both 'lfs/cdrom' AND
>> > > > > 'lfs/Config' need the '--memory'-parameter?
>> > 
>> > This was the next problem: 'percent'-parameter does not work as expected.
>> 
>> No, don't use it. Just determine yourself what it is.
>> 
>> I think we could even try to set 100% of the memory, because as far as I can
>> see
>> this is worst case and xz won't use everything that we allow.
>> 
>> If that does't work, let's scale down to 90%, or 80%, etc...
>> 
>> > > > Yes, but it would be good to have one place where the parameters are
>> > > > being generated and then we just use them from a variable.
>> > 
>> > Probably beyond my capabilities for the moment. Any thoughts?
>> > 
>> > > > > > > ...
>> > > > > > > We will at least need about half a GB which should be a sensible
>> > > > > > > requirement
>> > > > > > > for a build system any ways.
>> > 
>> > I'm afraid '500 MiB' are not enough...
>> 
>> No, so we will now make 1G the minimum for a build.
>> 
>> I tried to build the OS on an ARM board with only 512MB of RAM the other day
>> and
>> even with a lot of swap it doesn't work. So we are past this any way.
>> 
>> > 
>> > > > > Agreed. Because of this I'm testing with '500Mib'.
>> > > > > 
>> > > > > > > Will you work on this bringing it into the build system?!
>> > > > > 
>> > > > > "I'll do my very best!" :-))
>> > > > 
>> > > > Very well Miss Sophie...
>> > > > ...
>> > 
>> > Best™,
>> > Matthias
>> 
>> -Michael
> 

Re: Compress images with XZ

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 8744 bytes --]

Hi,

On Tue, 2018-01-30 at 17:47 +0100, Matthias Fischer wrote:
> On 29.01.2018 14:23, Michael Tremer wrote:
> > Hello Matthias,
> 
> Hi Michael,
> 
> > any update on this? Are you still on this?
> 
> Sorry, no update, but its still on my list.
> 
> We had a tough time here with all kinds of diseases in the family during
> the holidays and beyond, so I was only able to work on some 'easier jobs'.

That's fine. I just wanted to know if you are still on this and that it isn't
just forgotten.

> Besides, I'm not really familiar with this kind of scripting, but I'll
> try. This is the next I'm at.
> 
> I'll consider how I would do this and send it in as soon as I get a grip
> on it.

Well, I added some functions already somewhere. I suppose you just need to do
the maths to set that to X% of the host system which is just a simple
multiplication of the output of ${HOST_MEM} in make.sh.

And that needs to be passed to the lfs scripts that are supposed to make the
images.

Please send questions where ever you need help. There is no reason to do this
alone.

Best,
-Michael

> 
> Best,
> Matthias
>  > Best,
> > -Michael
> > 
> > On Wed, 2017-11-22 at 16:37 +0000, Michael Tremer wrote:
> > > Hi,
> > > 
> > > On Tue, 2017-11-21 at 18:35 +0100, Matthias Fischer wrote:
> > > > Hi,
> > > > 
> > > > On 21.11.2017 15:42, Michael Tremer wrote:
> > > > > Hello,
> > > > > 
> > > > > any progress on this?
> > > > 
> > > > Not much, because I've been struggling with a flu for the past two
> > > > weeks, and its not over yet, but:
> > > > 
> > > > In the meantime I ran several builds with different parameters.
> > > > 
> > > > IMHO we could use:
> > > > 
> > > > 'export XZ_OPT = --threads=0 -8 --memory=700MiB'
> > > > 
> > > > in 'lfs/cdrom'
> > > > 
> > > > and
> > > > 
> > > > cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8 --memory=700MiB" tar
> > > > -c
> > > > -p
> > > > --numeric-owner -J -f /install/packages/package/files.tar.xz *
> > > > 
> > > > in 'lfs/Config'.
> > > > 
> > > > Using '9' or '--extreme'-parameter brought no further improvements,
> > > > on the other hand, build time and RAM consumption increased
> > > > strongly. With '-8', minimal RAM consumption was about 680 MiB, using
> > > > '700 Mib' was always safe. So this is the minimum I would recommend.
> > > 
> > > Yes, I observed the same thing. -9 compresses a little bit better and
> > > extreme
> > > doesn't change a bit.
> > > 
> > > -8 seems to be optimal since it won't require too much memory to
> > > decompress
> > > either. So let's go with that.
> > > 
> > > > Results:
> > > > 
> > > > (Buildtime: 5 hours 20 minutes 59 seconds)
> > > > 
> > > > 255866764  ipfire-2.19.1gb-ext4.i586-full-core117.img.gz
> > > > 252262006  ipfire-2.19.1gb-ext4-scon.i586-full-core117.img.gz
> > > > 175112192  ipfire-2.19.i586-full-core117.iso
> > > >  
> > > > > -Michael
> > > > > 
> > > > > On Tue, 2017-11-07 at 22:47 +0000, Michael Tremer wrote:
> > > > > > Hi,
> > > > > > 
> > > > > > On Tue, 2017-11-07 at 19:41 +0100, Matthias Fischer wrote:
> > > > > > > Hi,
> > > > > > > 
> > > > > > > On 07.11.2017 16:05, Michael Tremer wrote:
> > > > > > > > Hi,
> > > > > > > > 
> > > > > > > > maybe this function helps:
> > > > > > > > 
> > > > > > > > https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=5190eea2
> > > > > > > > 4f98
> > > > > > > > 22
> > > > > > > > a63d
> > > > > > > > c5d06d214b48f973b14f29
> > > > > > > 
> > > > > > > Thanks. I'll take a look...
> > > > 
> > > > This would help, indeed, but how to implement(!?), and... (see below)
> > > 
> > > Just run it somewhere in the script, figure out what is about 80% of that
> > > and
> > > if
> > > that is smaller than 800 MB, we should just fail.
> > > 
> > > It would be good to have a function that determines the xz parameters and
> > > we
> > > use
> > > that where ever we call xz. We don't want any duplicated code.
> > > 
> > > > > > > > On Tue, 2017-11-07 at 10:46 +0000, Michael Tremer wrote:
> > > > > > > > > On Mon, 2017-11-06 at 22:09 +0100, Matthias Fischer wrote:
> > > > > > > > > ...
> > > > > > > > > > 
> > > > > > > > > > I forgot: Tuning the XZ-parameters... ;-)
> > > > > > > > > 
> > > > > > > > > Good that you are bringing this up. So I think what we should
> > > > > > > > > do
> > > > > > > > > here is
> > > > > > > > > bringing back XZ compression with -8 so that we do not waste
> > > > > > > > > too
> > > > > > > > > much
> > > > > > > > > space
> > > > > > > > > when
> > > > > > > > > extracting the image again.
> > > > > > > 
> > > > > > > Right now I'm testing how much RAM I need.
> > > > > > > 
> > > > > > > Current 'next'-build is running with:
> > > > > > > 
> > > > > > > 'lfs/cdrom':
> > > > > > > ...
> > > > > > > export XZ_OPT = --threads=0 -8 --memory=500MiB
> > > > > > > ...
> > > > > > > 
> > > > > > > 'lfs/Config':
> > > > > > > ...
> > > > > > > cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8" ...
> > > > > > > ...
> > > > > > > 
> > > > > > > I'm waiting for the results. Takes a while.
> > > > > > 
> > > > > > You can just run it like this and it will tell you:
> > > > > > 
> > > > > > [ms(a)hughes ~]$ xz -vv8 -T0
> > > > > > xz: Filter chain: --
> > > > > > lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
> > > > > > xz: Using up to 4 threads.
> > > > > > xz: 2,629 MiB of memory is required. The limiter is disabled.
> > > > > > xz: Decompression will need 33 MiB of memory.
> > > > > > xz: Compressed data cannot be written to a terminal
> > > > > > xz: Try `xz --help' for more information.
> > > > > > 
> > > > > > > > > To not run into memory limits we should detect how much memory
> > > > > > > > > the
> > > > > > > > > build
> > > > > > > > > host
> > > > > > > > > has and set the memory limit to maybe 80% of that. xz will
> > > > > > > > > then
> > > > > > > > > automatically
> > > > > > > > > scale down to a number of parallel processes that it can fit
> > > > > > > > > into
> > > > > > > > > memory.
> > > > 
> > > > This is something I couldn't get to work. First, I couldn't specify not
> > > > less
> > > > than '700 MiB'. If I specify '600 MiB', building stops and tells me it
> > > > needs
> > > > at least about 680 MiB. If I use '70%', building stops. And I never saw
> > > > that
> > > > 'xz' "automatically scaled down". It just stopped everytime.
> > > 
> > > Yes, ~700M is the minimum.
> > > 
> > > It will just reduce the number of threads again to make sure everything
> > > fits
> > > into memory.
> > > 
> > > >  
> > > > > > > I'm a bit puzzled about this: I tried using '--memory=70%' as
> > > > > > > mentioned
> > > > > > > in the xz man pages, but is seems to have no effect. It looks as
> > > > > > > if
> > > > > > > the
> > > > > > > percent parameter isn't recognized and ignored. It always ends up
> > > > > > > in
> > > > > > > "cannot allocate memory". Any hints on this? Do both 'lfs/cdrom'
> > > > > > > AND
> > > > > > > 'lfs/Config' need the '--memory'-parameter?
> > > > 
> > > > This was the next problem: 'percent'-parameter does not work as
> > > > expected.
> > > 
> > > No, don't use it. Just determine yourself what it is.
> > > 
> > > I think we could even try to set 100% of the memory, because as far as I
> > > can
> > > see
> > > this is worst case and xz won't use everything that we allow.
> > > 
> > > If that does't work, let's scale down to 90%, or 80%, etc...
> > > 
> > > > > > Yes, but it would be good to have one place where the parameters are
> > > > > > being generated and then we just use them from a variable.
> > > > 
> > > > Probably beyond my capabilities for the moment. Any thoughts?
> > > > 
> > > > > > > > > ...
> > > > > > > > > We will at least need about half a GB which should be a
> > > > > > > > > sensible
> > > > > > > > > requirement
> > > > > > > > > for a build system any ways.
> > > > 
> > > > I'm afraid '500 MiB' are not enough...
> > > 
> > > No, so we will now make 1G the minimum for a build.
> > > 
> > > I tried to build the OS on an ARM board with only 512MB of RAM the other
> > > day
> > > and
> > > even with a lot of swap it doesn't work. So we are past this any way.
> > > 
> > > > 
> > > > > > > Agreed. Because of this I'm testing with '500Mib'.
> > > > > > > 
> > > > > > > > > Will you work on this bringing it into the build system?!
> > > > > > > 
> > > > > > > "I'll do my very best!" :-))
> > > > > > 
> > > > > > Very well Miss Sophie...
> > > > > > ...
> > > > 
> > > > Best™,
> > > > Matthias
> > > 
> > > -Michael

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Compress images with XZ

[Matthias Fischer]

[-- Attachment #1: Type: text/plain, Size: 10512 bytes --]

Hi,

On 30.01.2018 21:03, Michael Tremer wrote:
> Hi,
> 
> ...
>> 
>> I'll consider how I would do this and send it in as soon as I get a grip
>> on it.
> 
> Well, I added some functions already somewhere. I suppose you just need to do
> the maths to set that to X% of the host system which is just a simple
> multiplication of the output of ${HOST_MEM} in make.sh.

Ok, I take my heart in my hand. Here are my first results - but remember,
I'm not familiar with this. I built a short test script which looks as if it
is working as wanted, but...
 
> And that needs to be passed to the lfs scripts that are supposed to make the
> images.

...this is a problem I'm struggling with right now:
How do I pass the 'XZ_MEM'-variable to 'cdrom' and 'Config'?

I think this can be done using 'export XZ_MEM' or something else but I'm not sure.

This is the script I'm testing with:

***SNIP***
#!/bin/bash

system_memory() {
	local key val unit

	while read -r key val unit; do
		case "${key}" in
			MemTotal:*)
				# Convert to MB
				echo "$(( ${val} / 1024 ))"
				break
				;;
		esac
	done < /proc/meminfo
}

HOST_MEM=$(system_memory)

# Checking host memory
if [ $HOST_MEM -ge 1024 ]; then
	echo Host-Memory: $HOST_MEM'MiB'
	echo "Host memory is OK (must be at least 1024MiB), starting build."
else
	echo Host-Memory: $HOST_MEM'MiB'
	echo "Host memory too low (less than 1024MiB), building stopped."
exit 1
fi

# Host memory is ok, calculating needed XZ memory to about 70% of host memory
((XZ_MEM=(($HOST_MEM)/10)*7))
	XZ_MEM=$XZ_MEM'Mib'
	echo XZ-Memory: $XZ_MEM
	echo "XZ memory size is OK (must be at least 700MiB), starting build."
***SNAP***

The 'echoes' are mostly for debugging.

Calculation looks ok, 'MiB' must added without blanks (XZ man page: "In most
places where an integer argument is expected, an optional suffix is supported
to easily indicate large integers. There must be no space between the integer
and the suffix.")

> Please send questions where ever you need help. There is no reason to do this
> alone.

Please see above:

1. Is this code what you where thinking of? I would leave the other parameters
as they are (--threads=0 -8).

2. Where should I add such code in the first place? I'd like to start with
'make.sh', so building stops as soon as possible if there is not enough memory. Where's
the best place? Right after function 'system_memory() {...}'?

3. How to I pass contents of 'XZ_MEM' to 'cdrom' and 'Config'?
I consider something like:
...
export XZ_OPT = --threads=0 -8 --memory=$XZ_MEM
...

Best,
Matthias
 
> Best,
> -Michael
> 
>> 
>> Best,
>> Matthias
>>  > Best,
>> > -Michael
>> > 
>> > On Wed, 2017-11-22 at 16:37 +0000, Michael Tremer wrote:
>> > > Hi,
>> > > 
>> > > On Tue, 2017-11-21 at 18:35 +0100, Matthias Fischer wrote:
>> > > > Hi,
>> > > > 
>> > > > On 21.11.2017 15:42, Michael Tremer wrote:
>> > > > > Hello,
>> > > > > 
>> > > > > any progress on this?
>> > > > 
>> > > > Not much, because I've been struggling with a flu for the past two
>> > > > weeks, and its not over yet, but:
>> > > > 
>> > > > In the meantime I ran several builds with different parameters.
>> > > > 
>> > > > IMHO we could use:
>> > > > 
>> > > > 'export XZ_OPT = --threads=0 -8 --memory=700MiB'
>> > > > 
>> > > > in 'lfs/cdrom'
>> > > > 
>> > > > and
>> > > > 
>> > > > cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8 --memory=700MiB" tar
>> > > > -c
>> > > > -p
>> > > > --numeric-owner -J -f /install/packages/package/files.tar.xz *
>> > > > 
>> > > > in 'lfs/Config'.
>> > > > 
>> > > > Using '9' or '--extreme'-parameter brought no further improvements,
>> > > > on the other hand, build time and RAM consumption increased
>> > > > strongly. With '-8', minimal RAM consumption was about 680 MiB, using
>> > > > '700 Mib' was always safe. So this is the minimum I would recommend.
>> > > 
>> > > Yes, I observed the same thing. -9 compresses a little bit better and
>> > > extreme
>> > > doesn't change a bit.
>> > > 
>> > > -8 seems to be optimal since it won't require too much memory to
>> > > decompress
>> > > either. So let's go with that.
>> > > 
>> > > > Results:
>> > > > 
>> > > > (Buildtime: 5 hours 20 minutes 59 seconds)
>> > > > 
>> > > > 255866764  ipfire-2.19.1gb-ext4.i586-full-core117.img.gz
>> > > > 252262006  ipfire-2.19.1gb-ext4-scon.i586-full-core117.img.gz
>> > > > 175112192  ipfire-2.19.i586-full-core117.iso
>> > > >  
>> > > > > -Michael
>> > > > > 
>> > > > > On Tue, 2017-11-07 at 22:47 +0000, Michael Tremer wrote:
>> > > > > > Hi,
>> > > > > > 
>> > > > > > On Tue, 2017-11-07 at 19:41 +0100, Matthias Fischer wrote:
>> > > > > > > Hi,
>> > > > > > > 
>> > > > > > > On 07.11.2017 16:05, Michael Tremer wrote:
>> > > > > > > > Hi,
>> > > > > > > > 
>> > > > > > > > maybe this function helps:
>> > > > > > > > 
>> > > > > > > > https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=5190eea2
>> > > > > > > > 4f98
>> > > > > > > > 22
>> > > > > > > > a63d
>> > > > > > > > c5d06d214b48f973b14f29
>> > > > > > > 
>> > > > > > > Thanks. I'll take a look...
>> > > > 
>> > > > This would help, indeed, but how to implement(!?), and... (see below)
>> > > 
>> > > Just run it somewhere in the script, figure out what is about 80% of that
>> > > and
>> > > if
>> > > that is smaller than 800 MB, we should just fail.
>> > > 
>> > > It would be good to have a function that determines the xz parameters and
>> > > we
>> > > use
>> > > that where ever we call xz. We don't want any duplicated code.
>> > > 
>> > > > > > > > On Tue, 2017-11-07 at 10:46 +0000, Michael Tremer wrote:
>> > > > > > > > > On Mon, 2017-11-06 at 22:09 +0100, Matthias Fischer wrote:
>> > > > > > > > > ...
>> > > > > > > > > > 
>> > > > > > > > > > I forgot: Tuning the XZ-parameters... ;-)
>> > > > > > > > > 
>> > > > > > > > > Good that you are bringing this up. So I think what we should
>> > > > > > > > > do
>> > > > > > > > > here is
>> > > > > > > > > bringing back XZ compression with -8 so that we do not waste
>> > > > > > > > > too
>> > > > > > > > > much
>> > > > > > > > > space
>> > > > > > > > > when
>> > > > > > > > > extracting the image again.
>> > > > > > > 
>> > > > > > > Right now I'm testing how much RAM I need.
>> > > > > > > 
>> > > > > > > Current 'next'-build is running with:
>> > > > > > > 
>> > > > > > > 'lfs/cdrom':
>> > > > > > > ...
>> > > > > > > export XZ_OPT = --threads=0 -8 --memory=500MiB
>> > > > > > > ...
>> > > > > > > 
>> > > > > > > 'lfs/Config':
>> > > > > > > ...
>> > > > > > > cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8" ...
>> > > > > > > ...
>> > > > > > > 
>> > > > > > > I'm waiting for the results. Takes a while.
>> > > > > > 
>> > > > > > You can just run it like this and it will tell you:
>> > > > > > 
>> > > > > > [ms(a)hughes ~]$ xz -vv8 -T0
>> > > > > > xz: Filter chain: --
>> > > > > > lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
>> > > > > > xz: Using up to 4 threads.
>> > > > > > xz: 2,629 MiB of memory is required. The limiter is disabled.
>> > > > > > xz: Decompression will need 33 MiB of memory.
>> > > > > > xz: Compressed data cannot be written to a terminal
>> > > > > > xz: Try `xz --help' for more information.
>> > > > > > 
>> > > > > > > > > To not run into memory limits we should detect how much memory
>> > > > > > > > > the
>> > > > > > > > > build
>> > > > > > > > > host
>> > > > > > > > > has and set the memory limit to maybe 80% of that. xz will
>> > > > > > > > > then
>> > > > > > > > > automatically
>> > > > > > > > > scale down to a number of parallel processes that it can fit
>> > > > > > > > > into
>> > > > > > > > > memory.
>> > > > 
>> > > > This is something I couldn't get to work. First, I couldn't specify not
>> > > > less
>> > > > than '700 MiB'. If I specify '600 MiB', building stops and tells me it
>> > > > needs
>> > > > at least about 680 MiB. If I use '70%', building stops. And I never saw
>> > > > that
>> > > > 'xz' "automatically scaled down". It just stopped everytime.
>> > > 
>> > > Yes, ~700M is the minimum.
>> > > 
>> > > It will just reduce the number of threads again to make sure everything
>> > > fits
>> > > into memory.
>> > > 
>> > > >  
>> > > > > > > I'm a bit puzzled about this: I tried using '--memory=70%' as
>> > > > > > > mentioned
>> > > > > > > in the xz man pages, but is seems to have no effect. It looks as
>> > > > > > > if
>> > > > > > > the
>> > > > > > > percent parameter isn't recognized and ignored. It always ends up
>> > > > > > > in
>> > > > > > > "cannot allocate memory". Any hints on this? Do both 'lfs/cdrom'
>> > > > > > > AND
>> > > > > > > 'lfs/Config' need the '--memory'-parameter?
>> > > > 
>> > > > This was the next problem: 'percent'-parameter does not work as
>> > > > expected.
>> > > 
>> > > No, don't use it. Just determine yourself what it is.
>> > > 
>> > > I think we could even try to set 100% of the memory, because as far as I
>> > > can
>> > > see
>> > > this is worst case and xz won't use everything that we allow.
>> > > 
>> > > If that does't work, let's scale down to 90%, or 80%, etc...
>> > > 
>> > > > > > Yes, but it would be good to have one place where the parameters are
>> > > > > > being generated and then we just use them from a variable.
>> > > > 
>> > > > Probably beyond my capabilities for the moment. Any thoughts?
>> > > > 
>> > > > > > > > > ...
>> > > > > > > > > We will at least need about half a GB which should be a
>> > > > > > > > > sensible
>> > > > > > > > > requirement
>> > > > > > > > > for a build system any ways.
>> > > > 
>> > > > I'm afraid '500 MiB' are not enough...
>> > > 
>> > > No, so we will now make 1G the minimum for a build.
>> > > 
>> > > I tried to build the OS on an ARM board with only 512MB of RAM the other
>> > > day
>> > > and
>> > > even with a lot of swap it doesn't work. So we are past this any way.
>> > > 
>> > > > 
>> > > > > > > Agreed. Because of this I'm testing with '500Mib'.
>> > > > > > > 
>> > > > > > > > > Will you work on this bringing it into the build system?!
>> > > > > > > 
>> > > > > > > "I'll do my very best!" :-))
>> > > > > > 
>> > > > > > Very well Miss Sophie...
>> > > > > > ...
>> > > > 
>> > > > Best™,
>> > > > Matthias
>> > > 
>> > > -Michael
> 

Re: Compress images with XZ

[Matthias Fischer]

[-- Attachment #1: Type: text/plain, Size: 11173 bytes --]

Hi,

Small addition:

I found another possibility for calculating XZ memory and adding the
needed suffix:

...
((XZ_MEM=(($HOST_MEM)/10)*7)) && XZ_MEM=${XZ_MEM}MiB
...

Looks better.

Best,
Matthias

On 03.02.2018 20:44, Matthias Fischer wrote:
> Hi,
> 
> On 30.01.2018 21:03, Michael Tremer wrote:
>> Hi,
>> 
>> ...
>>> 
>>> I'll consider how I would do this and send it in as soon as I get a grip
>>> on it.
>> 
>> Well, I added some functions already somewhere. I suppose you just need to do
>> the maths to set that to X% of the host system which is just a simple
>> multiplication of the output of ${HOST_MEM} in make.sh.
> 
> Ok, I take my heart in my hand. Here are my first results - but remember,
> I'm not familiar with this. I built a short test script which looks as if it
> is working as wanted, but...
>  
>> And that needs to be passed to the lfs scripts that are supposed to make the
>> images.
> 
> ...this is a problem I'm struggling with right now:
> How do I pass the 'XZ_MEM'-variable to 'cdrom' and 'Config'?
> 
> I think this can be done using 'export XZ_MEM' or something else but I'm not sure.
> 
> This is the script I'm testing with:
> 
> ***SNIP***
> #!/bin/bash
> 
> system_memory() {
> 	local key val unit
> 
> 	while read -r key val unit; do
> 		case "${key}" in
> 			MemTotal:*)
> 				# Convert to MB
> 				echo "$(( ${val} / 1024 ))"
> 				break
> 				;;
> 		esac
> 	done < /proc/meminfo
> }
> 
> HOST_MEM=$(system_memory)
> 
> # Checking host memory
> if [ $HOST_MEM -ge 1024 ]; then
> 	echo Host-Memory: $HOST_MEM'MiB'
> 	echo "Host memory is OK (must be at least 1024MiB), starting build."
> else
> 	echo Host-Memory: $HOST_MEM'MiB'
> 	echo "Host memory too low (less than 1024MiB), building stopped."
> exit 1
> fi
> 
> # Host memory is ok, calculating needed XZ memory to about 70% of host memory
> ((XZ_MEM=(($HOST_MEM)/10)*7))
> 	XZ_MEM=$XZ_MEM'Mib'
> 	echo XZ-Memory: $XZ_MEM
> 	echo "XZ memory size is OK (must be at least 700MiB), starting build."
> ***SNAP***
> 
> The 'echoes' are mostly for debugging.
> 
> Calculation looks ok, 'MiB' must added without blanks (XZ man page: "In most
> places where an integer argument is expected, an optional suffix is supported
> to easily indicate large integers. There must be no space between the integer
> and the suffix.")
> 
>> Please send questions where ever you need help. There is no reason to do this
>> alone.
> 
> Please see above:
> 
> 1. Is this code what you where thinking of? I would leave the other parameters
> as they are (--threads=0 -8).
> 
> 2. Where should I add such code in the first place? I'd like to start with
> 'make.sh', so building stops as soon as possible if there is not enough memory. Where's
> the best place? Right after function 'system_memory() {...}'?
> 
> 3. How to I pass contents of 'XZ_MEM' to 'cdrom' and 'Config'?
> I consider something like:
> ...
> export XZ_OPT = --threads=0 -8 --memory=$XZ_MEM
> ...
> 
> Best,
> Matthias
>  
>> Best,
>> -Michael
>> 
>>> 
>>> Best,
>>> Matthias
>>>  > Best,
>>> > -Michael
>>> > 
>>> > On Wed, 2017-11-22 at 16:37 +0000, Michael Tremer wrote:
>>> > > Hi,
>>> > > 
>>> > > On Tue, 2017-11-21 at 18:35 +0100, Matthias Fischer wrote:
>>> > > > Hi,
>>> > > > 
>>> > > > On 21.11.2017 15:42, Michael Tremer wrote:
>>> > > > > Hello,
>>> > > > > 
>>> > > > > any progress on this?
>>> > > > 
>>> > > > Not much, because I've been struggling with a flu for the past two
>>> > > > weeks, and its not over yet, but:
>>> > > > 
>>> > > > In the meantime I ran several builds with different parameters.
>>> > > > 
>>> > > > IMHO we could use:
>>> > > > 
>>> > > > 'export XZ_OPT = --threads=0 -8 --memory=700MiB'
>>> > > > 
>>> > > > in 'lfs/cdrom'
>>> > > > 
>>> > > > and
>>> > > > 
>>> > > > cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8 --memory=700MiB" tar
>>> > > > -c
>>> > > > -p
>>> > > > --numeric-owner -J -f /install/packages/package/files.tar.xz *
>>> > > > 
>>> > > > in 'lfs/Config'.
>>> > > > 
>>> > > > Using '9' or '--extreme'-parameter brought no further improvements,
>>> > > > on the other hand, build time and RAM consumption increased
>>> > > > strongly. With '-8', minimal RAM consumption was about 680 MiB, using
>>> > > > '700 Mib' was always safe. So this is the minimum I would recommend.
>>> > > 
>>> > > Yes, I observed the same thing. -9 compresses a little bit better and
>>> > > extreme
>>> > > doesn't change a bit.
>>> > > 
>>> > > -8 seems to be optimal since it won't require too much memory to
>>> > > decompress
>>> > > either. So let's go with that.
>>> > > 
>>> > > > Results:
>>> > > > 
>>> > > > (Buildtime: 5 hours 20 minutes 59 seconds)
>>> > > > 
>>> > > > 255866764  ipfire-2.19.1gb-ext4.i586-full-core117.img.gz
>>> > > > 252262006  ipfire-2.19.1gb-ext4-scon.i586-full-core117.img.gz
>>> > > > 175112192  ipfire-2.19.i586-full-core117.iso
>>> > > >  
>>> > > > > -Michael
>>> > > > > 
>>> > > > > On Tue, 2017-11-07 at 22:47 +0000, Michael Tremer wrote:
>>> > > > > > Hi,
>>> > > > > > 
>>> > > > > > On Tue, 2017-11-07 at 19:41 +0100, Matthias Fischer wrote:
>>> > > > > > > Hi,
>>> > > > > > > 
>>> > > > > > > On 07.11.2017 16:05, Michael Tremer wrote:
>>> > > > > > > > Hi,
>>> > > > > > > > 
>>> > > > > > > > maybe this function helps:
>>> > > > > > > > 
>>> > > > > > > > https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=5190eea2
>>> > > > > > > > 4f98
>>> > > > > > > > 22
>>> > > > > > > > a63d
>>> > > > > > > > c5d06d214b48f973b14f29
>>> > > > > > > 
>>> > > > > > > Thanks. I'll take a look...
>>> > > > 
>>> > > > This would help, indeed, but how to implement(!?), and... (see below)
>>> > > 
>>> > > Just run it somewhere in the script, figure out what is about 80% of that
>>> > > and
>>> > > if
>>> > > that is smaller than 800 MB, we should just fail.
>>> > > 
>>> > > It would be good to have a function that determines the xz parameters and
>>> > > we
>>> > > use
>>> > > that where ever we call xz. We don't want any duplicated code.
>>> > > 
>>> > > > > > > > On Tue, 2017-11-07 at 10:46 +0000, Michael Tremer wrote:
>>> > > > > > > > > On Mon, 2017-11-06 at 22:09 +0100, Matthias Fischer wrote:
>>> > > > > > > > > ...
>>> > > > > > > > > > 
>>> > > > > > > > > > I forgot: Tuning the XZ-parameters... ;-)
>>> > > > > > > > > 
>>> > > > > > > > > Good that you are bringing this up. So I think what we should
>>> > > > > > > > > do
>>> > > > > > > > > here is
>>> > > > > > > > > bringing back XZ compression with -8 so that we do not waste
>>> > > > > > > > > too
>>> > > > > > > > > much
>>> > > > > > > > > space
>>> > > > > > > > > when
>>> > > > > > > > > extracting the image again.
>>> > > > > > > 
>>> > > > > > > Right now I'm testing how much RAM I need.
>>> > > > > > > 
>>> > > > > > > Current 'next'-build is running with:
>>> > > > > > > 
>>> > > > > > > 'lfs/cdrom':
>>> > > > > > > ...
>>> > > > > > > export XZ_OPT = --threads=0 -8 --memory=500MiB
>>> > > > > > > ...
>>> > > > > > > 
>>> > > > > > > 'lfs/Config':
>>> > > > > > > ...
>>> > > > > > > cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8" ...
>>> > > > > > > ...
>>> > > > > > > 
>>> > > > > > > I'm waiting for the results. Takes a while.
>>> > > > > > 
>>> > > > > > You can just run it like this and it will tell you:
>>> > > > > > 
>>> > > > > > [ms(a)hughes ~]$ xz -vv8 -T0
>>> > > > > > xz: Filter chain: --
>>> > > > > > lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
>>> > > > > > xz: Using up to 4 threads.
>>> > > > > > xz: 2,629 MiB of memory is required. The limiter is disabled.
>>> > > > > > xz: Decompression will need 33 MiB of memory.
>>> > > > > > xz: Compressed data cannot be written to a terminal
>>> > > > > > xz: Try `xz --help' for more information.
>>> > > > > > 
>>> > > > > > > > > To not run into memory limits we should detect how much memory
>>> > > > > > > > > the
>>> > > > > > > > > build
>>> > > > > > > > > host
>>> > > > > > > > > has and set the memory limit to maybe 80% of that. xz will
>>> > > > > > > > > then
>>> > > > > > > > > automatically
>>> > > > > > > > > scale down to a number of parallel processes that it can fit
>>> > > > > > > > > into
>>> > > > > > > > > memory.
>>> > > > 
>>> > > > This is something I couldn't get to work. First, I couldn't specify not
>>> > > > less
>>> > > > than '700 MiB'. If I specify '600 MiB', building stops and tells me it
>>> > > > needs
>>> > > > at least about 680 MiB. If I use '70%', building stops. And I never saw
>>> > > > that
>>> > > > 'xz' "automatically scaled down". It just stopped everytime.
>>> > > 
>>> > > Yes, ~700M is the minimum.
>>> > > 
>>> > > It will just reduce the number of threads again to make sure everything
>>> > > fits
>>> > > into memory.
>>> > > 
>>> > > >  
>>> > > > > > > I'm a bit puzzled about this: I tried using '--memory=70%' as
>>> > > > > > > mentioned
>>> > > > > > > in the xz man pages, but is seems to have no effect. It looks as
>>> > > > > > > if
>>> > > > > > > the
>>> > > > > > > percent parameter isn't recognized and ignored. It always ends up
>>> > > > > > > in
>>> > > > > > > "cannot allocate memory". Any hints on this? Do both 'lfs/cdrom'
>>> > > > > > > AND
>>> > > > > > > 'lfs/Config' need the '--memory'-parameter?
>>> > > > 
>>> > > > This was the next problem: 'percent'-parameter does not work as
>>> > > > expected.
>>> > > 
>>> > > No, don't use it. Just determine yourself what it is.
>>> > > 
>>> > > I think we could even try to set 100% of the memory, because as far as I
>>> > > can
>>> > > see
>>> > > this is worst case and xz won't use everything that we allow.
>>> > > 
>>> > > If that does't work, let's scale down to 90%, or 80%, etc...
>>> > > 
>>> > > > > > Yes, but it would be good to have one place where the parameters are
>>> > > > > > being generated and then we just use them from a variable.
>>> > > > 
>>> > > > Probably beyond my capabilities for the moment. Any thoughts?
>>> > > > 
>>> > > > > > > > > ...
>>> > > > > > > > > We will at least need about half a GB which should be a
>>> > > > > > > > > sensible
>>> > > > > > > > > requirement
>>> > > > > > > > > for a build system any ways.
>>> > > > 
>>> > > > I'm afraid '500 MiB' are not enough...
>>> > > 
>>> > > No, so we will now make 1G the minimum for a build.
>>> > > 
>>> > > I tried to build the OS on an ARM board with only 512MB of RAM the other
>>> > > day
>>> > > and
>>> > > even with a lot of swap it doesn't work. So we are past this any way.
>>> > > 
>>> > > > 
>>> > > > > > > Agreed. Because of this I'm testing with '500Mib'.
>>> > > > > > > 
>>> > > > > > > > > Will you work on this bringing it into the build system?!
>>> > > > > > > 
>>> > > > > > > "I'll do my very best!" :-))
>>> > > > > > 
>>> > > > > > Very well Miss Sophie...
>>> > > > > > ...
>>> > > > 
>>> > > > Best™,
>>> > > > Matthias
>>> > > 
>>> > > -Michael
>> 
> 
> 

Re: Compress images with XZ

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 13865 bytes --]

Hi,

On Sat, 2018-02-03 at 22:09 +0100, Matthias Fischer wrote:
> Hi,
> 
> Small addition:
> 
> I found another possibility for calculating XZ memory and adding the
> needed suffix:
> 
> ...
> ((XZ_MEM=(($HOST_MEM)/10)*7)) && XZ_MEM=${XZ_MEM}MiB

You can write this even short:

XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"

I swapped the order of the multiplication and division so that you would have a
more exact result.

For example if you system has 1024 MB or RAM, you divide this by 10 (102) and
then multiply by 7 which gives you 714. The other order is 1024 * 7 = 7167 and
then divided by 10 is 716 which is closer to the actual result of 716.8.
Shouldn't matter here but just in general this is better :)

You can put this probably next under "HOST_MEM=$(system_memory)" and then export
this to the internal shell by adding the variable to lfsmake2 for the disk image
and ipfiredist for the packages. Then in lfs/cdrom and lfs/Config for the dist
target, you just need to add the parameters to the command line.

Maybe it is a good idea to have a variable XZ_OPT that adds everything together
and is then used in both places.

Best,
-Michael

> ...
> 
> Looks better.
> 
> Best,
> Matthias
> 
> On 03.02.2018 20:44, Matthias Fischer wrote:
> > Hi,
> > 
> > On 30.01.2018 21:03, Michael Tremer wrote:
> > > Hi,
> > > 
> > > ...
> > > > 
> > > > I'll consider how I would do this and send it in as soon as I get a grip
> > > > on it.
> > > 
> > > Well, I added some functions already somewhere. I suppose you just need to
> > > do
> > > the maths to set that to X% of the host system which is just a simple
> > > multiplication of the output of ${HOST_MEM} in make.sh.
> > 
> > Ok, I take my heart in my hand. Here are my first results - but remember,
> > I'm not familiar with this. I built a short test script which looks as if it
> > is working as wanted, but...
> >  
> > > And that needs to be passed to the lfs scripts that are supposed to make
> > > the
> > > images.
> > 
> > ...this is a problem I'm struggling with right now:
> > How do I pass the 'XZ_MEM'-variable to 'cdrom' and 'Config'?
> > 
> > I think this can be done using 'export XZ_MEM' or something else but I'm not
> > sure.
> > 
> > This is the script I'm testing with:
> > 
> > ***SNIP***
> > #!/bin/bash
> > 
> > system_memory() {
> > 	local key val unit
> > 
> > 	while read -r key val unit; do
> > 		case "${key}" in
> > 			MemTotal:*)
> > 				# Convert to MB
> > 				echo "$(( ${val} / 1024 ))"
> > 				break
> > 				;;
> > 		esac
> > 	done < /proc/meminfo
> > }
> > 
> > HOST_MEM=$(system_memory)
> > 
> > # Checking host memory
> > if [ $HOST_MEM -ge 1024 ]; then
> > 	echo Host-Memory: $HOST_MEM'MiB'
> > 	echo "Host memory is OK (must be at least 1024MiB), starting build."
> > else
> > 	echo Host-Memory: $HOST_MEM'MiB'
> > 	echo "Host memory too low (less than 1024MiB), building stopped."
> > exit 1
> > fi
> > 
> > # Host memory is ok, calculating needed XZ memory to about 70% of host
> > memory
> > ((XZ_MEM=(($HOST_MEM)/10)*7))
> > 	XZ_MEM=$XZ_MEM'Mib'
> > 	echo XZ-Memory: $XZ_MEM
> > 	echo "XZ memory size is OK (must be at least 700MiB), starting build."
> > ***SNAP***
> > 
> > The 'echoes' are mostly for debugging.
> > 
> > Calculation looks ok, 'MiB' must added without blanks (XZ man page: "In most
> > places where an integer argument is expected, an optional suffix is
> > supported
> > to easily indicate large integers. There must be no space between the
> > integer
> > and the suffix.")
> > 
> > > Please send questions where ever you need help. There is no reason to do
> > > this
> > > alone.
> > 
> > Please see above:
> > 
> > 1. Is this code what you where thinking of? I would leave the other
> > parameters
> > as they are (--threads=0 -8).
> > 
> > 2. Where should I add such code in the first place? I'd like to start with
> > 'make.sh', so building stops as soon as possible if there is not enough
> > memory. Where's
> > the best place? Right after function 'system_memory() {...}'?
> > 
> > 3. How to I pass contents of 'XZ_MEM' to 'cdrom' and 'Config'?
> > I consider something like:
> > ...
> > export XZ_OPT = --threads=0 -8 --memory=$XZ_MEM
> > ...
> > 
> > Best,
> > Matthias
> >  
> > > Best,
> > > -Michael
> > > 
> > > > 
> > > > Best,
> > > > Matthias
> > > >  > Best,
> > > > > -Michael
> > > > > 
> > > > > On Wed, 2017-11-22 at 16:37 +0000, Michael Tremer wrote:
> > > > > > Hi,
> > > > > > 
> > > > > > On Tue, 2017-11-21 at 18:35 +0100, Matthias Fischer wrote:
> > > > > > > Hi,
> > > > > > > 
> > > > > > > On 21.11.2017 15:42, Michael Tremer wrote:
> > > > > > > > Hello,
> > > > > > > > 
> > > > > > > > any progress on this?
> > > > > > > 
> > > > > > > Not much, because I've been struggling with a flu for the past two
> > > > > > > weeks, and its not over yet, but:
> > > > > > > 
> > > > > > > In the meantime I ran several builds with different parameters.
> > > > > > > 
> > > > > > > IMHO we could use:
> > > > > > > 
> > > > > > > 'export XZ_OPT = --threads=0 -8 --memory=700MiB'
> > > > > > > 
> > > > > > > in 'lfs/cdrom'
> > > > > > > 
> > > > > > > and
> > > > > > > 
> > > > > > > cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8 --
> > > > > > > memory=700MiB" tar
> > > > > > > -c
> > > > > > > -p
> > > > > > > --numeric-owner -J -f /install/packages/package/files.tar.xz *
> > > > > > > 
> > > > > > > in 'lfs/Config'.
> > > > > > > 
> > > > > > > Using '9' or '--extreme'-parameter brought no further
> > > > > > > improvements,
> > > > > > > on the other hand, build time and RAM consumption increased
> > > > > > > strongly. With '-8', minimal RAM consumption was about 680 MiB,
> > > > > > > using
> > > > > > > '700 Mib' was always safe. So this is the minimum I would
> > > > > > > recommend.
> > > > > > 
> > > > > > Yes, I observed the same thing. -9 compresses a little bit better
> > > > > > and
> > > > > > extreme
> > > > > > doesn't change a bit.
> > > > > > 
> > > > > > -8 seems to be optimal since it won't require too much memory to
> > > > > > decompress
> > > > > > either. So let's go with that.
> > > > > > 
> > > > > > > Results:
> > > > > > > 
> > > > > > > (Buildtime: 5 hours 20 minutes 59 seconds)
> > > > > > > 
> > > > > > > 255866764  ipfire-2.19.1gb-ext4.i586-full-core117.img.gz
> > > > > > > 252262006  ipfire-2.19.1gb-ext4-scon.i586-full-core117.img.gz
> > > > > > > 175112192  ipfire-2.19.i586-full-core117.iso
> > > > > > >  
> > > > > > > > -Michael
> > > > > > > > 
> > > > > > > > On Tue, 2017-11-07 at 22:47 +0000, Michael Tremer wrote:
> > > > > > > > > Hi,
> > > > > > > > > 
> > > > > > > > > On Tue, 2017-11-07 at 19:41 +0100, Matthias Fischer wrote:
> > > > > > > > > > Hi,
> > > > > > > > > > 
> > > > > > > > > > On 07.11.2017 16:05, Michael Tremer wrote:
> > > > > > > > > > > Hi,
> > > > > > > > > > > 
> > > > > > > > > > > maybe this function helps:
> > > > > > > > > > > 
> > > > > > > > > > > https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=51
> > > > > > > > > > > 90eea2
> > > > > > > > > > > 4f98
> > > > > > > > > > > 22
> > > > > > > > > > > a63d
> > > > > > > > > > > c5d06d214b48f973b14f29
> > > > > > > > > > 
> > > > > > > > > > Thanks. I'll take a look...
> > > > > > > 
> > > > > > > This would help, indeed, but how to implement(!?), and... (see
> > > > > > > below)
> > > > > > 
> > > > > > Just run it somewhere in the script, figure out what is about 80% of
> > > > > > that
> > > > > > and
> > > > > > if
> > > > > > that is smaller than 800 MB, we should just fail.
> > > > > > 
> > > > > > It would be good to have a function that determines the xz
> > > > > > parameters and
> > > > > > we
> > > > > > use
> > > > > > that where ever we call xz. We don't want any duplicated code.
> > > > > > 
> > > > > > > > > > > On Tue, 2017-11-07 at 10:46 +0000, Michael Tremer wrote:
> > > > > > > > > > > > On Mon, 2017-11-06 at 22:09 +0100, Matthias Fischer
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > ...
> > > > > > > > > > > > > 
> > > > > > > > > > > > > I forgot: Tuning the XZ-parameters... ;-)
> > > > > > > > > > > > 
> > > > > > > > > > > > Good that you are bringing this up. So I think what we
> > > > > > > > > > > > should
> > > > > > > > > > > > do
> > > > > > > > > > > > here is
> > > > > > > > > > > > bringing back XZ compression with -8 so that we do not
> > > > > > > > > > > > waste
> > > > > > > > > > > > too
> > > > > > > > > > > > much
> > > > > > > > > > > > space
> > > > > > > > > > > > when
> > > > > > > > > > > > extracting the image again.
> > > > > > > > > > 
> > > > > > > > > > Right now I'm testing how much RAM I need.
> > > > > > > > > > 
> > > > > > > > > > Current 'next'-build is running with:
> > > > > > > > > > 
> > > > > > > > > > 'lfs/cdrom':
> > > > > > > > > > ...
> > > > > > > > > > export XZ_OPT = --threads=0 -8 --memory=500MiB
> > > > > > > > > > ...
> > > > > > > > > > 
> > > > > > > > > > 'lfs/Config':
> > > > > > > > > > ...
> > > > > > > > > > cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8" ...
> > > > > > > > > > ...
> > > > > > > > > > 
> > > > > > > > > > I'm waiting for the results. Takes a while.
> > > > > > > > > 
> > > > > > > > > You can just run it like this and it will tell you:
> > > > > > > > > 
> > > > > > > > > [ms(a)hughes ~]$ xz -vv8 -T0
> > > > > > > > > xz: Filter chain: --
> > > > > > > > > lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,dep
> > > > > > > > > th=0
> > > > > > > > > xz: Using up to 4 threads.
> > > > > > > > > xz: 2,629 MiB of memory is required. The limiter is disabled.
> > > > > > > > > xz: Decompression will need 33 MiB of memory.
> > > > > > > > > xz: Compressed data cannot be written to a terminal
> > > > > > > > > xz: Try `xz --help' for more information.
> > > > > > > > > 
> > > > > > > > > > > > To not run into memory limits we should detect how much
> > > > > > > > > > > > memory
> > > > > > > > > > > > the
> > > > > > > > > > > > build
> > > > > > > > > > > > host
> > > > > > > > > > > > has and set the memory limit to maybe 80% of that. xz
> > > > > > > > > > > > will
> > > > > > > > > > > > then
> > > > > > > > > > > > automatically
> > > > > > > > > > > > scale down to a number of parallel processes that it can
> > > > > > > > > > > > fit
> > > > > > > > > > > > into
> > > > > > > > > > > > memory.
> > > > > > > 
> > > > > > > This is something I couldn't get to work. First, I couldn't
> > > > > > > specify not
> > > > > > > less
> > > > > > > than '700 MiB'. If I specify '600 MiB', building stops and tells
> > > > > > > me it
> > > > > > > needs
> > > > > > > at least about 680 MiB. If I use '70%', building stops. And I
> > > > > > > never saw
> > > > > > > that
> > > > > > > 'xz' "automatically scaled down". It just stopped everytime.
> > > > > > 
> > > > > > Yes, ~700M is the minimum.
> > > > > > 
> > > > > > It will just reduce the number of threads again to make sure
> > > > > > everything
> > > > > > fits
> > > > > > into memory.
> > > > > > 
> > > > > > >  
> > > > > > > > > > I'm a bit puzzled about this: I tried using '--memory=70%'
> > > > > > > > > > as
> > > > > > > > > > mentioned
> > > > > > > > > > in the xz man pages, but is seems to have no effect. It
> > > > > > > > > > looks as
> > > > > > > > > > if
> > > > > > > > > > the
> > > > > > > > > > percent parameter isn't recognized and ignored. It always
> > > > > > > > > > ends up
> > > > > > > > > > in
> > > > > > > > > > "cannot allocate memory". Any hints on this? Do both
> > > > > > > > > > 'lfs/cdrom'
> > > > > > > > > > AND
> > > > > > > > > > 'lfs/Config' need the '--memory'-parameter?
> > > > > > > 
> > > > > > > This was the next problem: 'percent'-parameter does not work as
> > > > > > > expected.
> > > > > > 
> > > > > > No, don't use it. Just determine yourself what it is.
> > > > > > 
> > > > > > I think we could even try to set 100% of the memory, because as far
> > > > > > as I
> > > > > > can
> > > > > > see
> > > > > > this is worst case and xz won't use everything that we allow.
> > > > > > 
> > > > > > If that does't work, let's scale down to 90%, or 80%, etc...
> > > > > > 
> > > > > > > > > Yes, but it would be good to have one place where the
> > > > > > > > > parameters are
> > > > > > > > > being generated and then we just use them from a variable.
> > > > > > > 
> > > > > > > Probably beyond my capabilities for the moment. Any thoughts?
> > > > > > > 
> > > > > > > > > > > > ...
> > > > > > > > > > > > We will at least need about half a GB which should be a
> > > > > > > > > > > > sensible
> > > > > > > > > > > > requirement
> > > > > > > > > > > > for a build system any ways.
> > > > > > > 
> > > > > > > I'm afraid '500 MiB' are not enough...
> > > > > > 
> > > > > > No, so we will now make 1G the minimum for a build.
> > > > > > 
> > > > > > I tried to build the OS on an ARM board with only 512MB of RAM the
> > > > > > other
> > > > > > day
> > > > > > and
> > > > > > even with a lot of swap it doesn't work. So we are past this any
> > > > > > way.
> > > > > > 
> > > > > > > 
> > > > > > > > > > Agreed. Because of this I'm testing with '500Mib'.
> > > > > > > > > > 
> > > > > > > > > > > > Will you work on this bringing it into the build
> > > > > > > > > > > > system?!
> > > > > > > > > > 
> > > > > > > > > > "I'll do my very best!" :-))
> > > > > > > > > 
> > > > > > > > > Very well Miss Sophie...
> > > > > > > > > ...
> > > > > > > 
> > > > > > > Best™,
> > > > > > > Matthias
> > > > > > 
> > > > > > -Michael
> > 
> > 
> 
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Compress images with XZ

[Matthias Fischer]

[-- Attachment #1: Type: text/plain, Size: 16164 bytes --]

Hi,

On 06.02.2018 02:05, Michael Tremer wrote:
> > ...
> > ((XZ_MEM=(($HOST_MEM)/10)*7)) && XZ_MEM=${XZ_MEM}MiB
> 
> You can write this even short:
> 
> XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"

Oneliner! Yes. I was looking for something like that... ;-)

> I swapped the order of the multiplication and division so that you would have
> a
> more exact result.
> 
> For example if you system has 1024 MB or RAM, you divide this by 10 (102) and
> then multiply by 7 which gives you 714. The other order is 1024 * 7 = 7167 and
> then divided by 10 is 716 which is closer to the actual result of 716.8.
> Shouldn't matter here but just in general this is better :)

Thanks, looks better - noted for the next time!

Now it looks like this ('make.sh'):

...
# Host memory is ok, calculating XZ memory
	XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"
	XZ_OPT="--threads=0 -8 $XZ_MEM"
	echo XZ-Memory: $XZ_MEM
	echo XZ-Options: $XZ_OPT
	echo "XZ memory size is OK (must be at least 700MiB), starting build..."
...
["echoes" for debugging!]

This is working. '$XZ_OPT' now gives me: --threads=0 -8 5450MiB

But. ;-)

> You can put this probably next under "HOST_MEM=$(system_memory)" and then
> export
> this to the internal shell by adding the variable to lfsmake2 for the disk
> image
> and ipfiredist for the packages.

Sorry, I must confess that I don't know exactly how to do this. As I wrote: "How
do
I pass the 'XZ_MEM'-variable to 'cdrom' and 'Config'?" Or in this case, XZ_OPT!?

> Then in lfs/cdrom and lfs/Config for the dist
> target, you just need to add the parameters to the command line.

In 'Config' I changed:

...
cd /install/packages/package/tmp/ && XZ_OPT=-T0 tar -c -p --numeric-owner -J -f
/install/packages/package/files.tar.xz *
...

To:
...
cd /install/packages/package/tmp/ && $XZ_OPT tar -c -p --numeric-owner -J -f
/install/packages/package/files.tar.xz *
...

> Maybe it is a good idea to have a variable XZ_OPT that adds everything
> together
> and is then used in both places.

I added the other xz-options and created XZ_OPT as shown above, but in 'cdrom'
there
is an 'export XZ_OPT = ...'-line and I don't know how to handle these line. Is
it still necessary?

Do I have to 'export' the XZ_OPT'-variable in 'make.sh'?

And how do I "add" this variable to 'lfsmake2' and 'ipfiredist'(-function)?

Sorry if these questions sound simple, but I'm guessing a bit too much while
trying
to get a grip on this.

Best,
Matthias

> 
> Best,
> -Michael
> 
> > ...
> > 
> > Looks better.
> > 
> > Best,
> > Matthias
> > 
> > On 03.02.2018 20:44, Matthias Fischer wrote:
> > > Hi,
> > > 
> > > On 30.01.2018 21:03, Michael Tremer wrote:
> > > > Hi,
> > > > 
> > > > ...
> > > > > 
> > > > > I'll consider how I would do this and send it in as soon as I get a
> > > > > grip
> > > > > on it.
> > > > 
> > > > Well, I added some functions already somewhere. I suppose you just need
> > > > to
> > > > do
> > > > the maths to set that to X% of the host system which is just a simple
> > > > multiplication of the output of ${HOST_MEM} in make.sh.
> > > 
> > > Ok, I take my heart in my hand. Here are my first results - but remember,
> > > I'm not familiar with this. I built a short test script which looks as if
> > > it
> > > is working as wanted, but...
> > >  
> > > > And that needs to be passed to the lfs scripts that are supposed to make
> > > > the
> > > > images.
> > > 
> > > ...this is a problem I'm struggling with right now:
> > > How do I pass the 'XZ_MEM'-variable to 'cdrom' and 'Config'?
> > > 
> > > I think this can be done using 'export XZ_MEM' or something else but I'm
> > > not
> > > sure.
> > > 
> > > This is the script I'm testing with:
> > > 
> > > ***SNIP***
> > > #!/bin/bash
> > > 
> > > system_memory() {
> > > 	local key val unit
> > > 
> > > 	while read -r key val unit; do
> > > 		case "${key}" in
> > > 			MemTotal:*)
> > > 				# Convert to MB
> > > 				echo "$(( ${val} / 1024 ))"
> > > 				break
> > > 				;;
> > > 		esac
> > > 	done < /proc/meminfo
> > > }
> > > 
> > > HOST_MEM=$(system_memory)
> > > 
> > > # Checking host memory
> > > if [ $HOST_MEM -ge 1024 ]; then
> > > 	echo Host-Memory: $HOST_MEM'MiB'
> > > 	echo "Host memory is OK (must be at least 1024MiB), starting build."
> > > else
> > > 	echo Host-Memory: $HOST_MEM'MiB'
> > > 	echo "Host memory too low (less than 1024MiB), building stopped."
> > > exit 1
> > > fi
> > > 
> > > # Host memory is ok, calculating needed XZ memory to about 70% of host
> > > memory
> > > ((XZ_MEM=(($HOST_MEM)/10)*7))
> > > 	XZ_MEM=$XZ_MEM'Mib'
> > > 	echo XZ-Memory: $XZ_MEM
> > > 	echo "XZ memory size is OK (must be at least 700MiB), starting build."
> > > ***SNAP***
> > > 
> > > The 'echoes' are mostly for debugging.
> > > 
> > > Calculation looks ok, 'MiB' must added without blanks (XZ man page: "In
> > > most
> > > places where an integer argument is expected, an optional suffix is
> > > supported
> > > to easily indicate large integers. There must be no space between the
> > > integer
> > > and the suffix.")
> > > 
> > > > Please send questions where ever you need help. There is no reason to do
> > > > this
> > > > alone.
> > > 
> > > Please see above:
> > > 
> > > 1. Is this code what you where thinking of? I would leave the other
> > > parameters
> > > as they are (--threads=0 -8).
> > > 
> > > 2. Where should I add such code in the first place? I'd like to start with
> > > 'make.sh', so building stops as soon as possible if there is not enough
> > > memory. Where's
> > > the best place? Right after function 'system_memory() {...}'?
> > > 
> > > 3. How to I pass contents of 'XZ_MEM' to 'cdrom' and 'Config'?
> > > I consider something like:
> > > ...
> > > export XZ_OPT = --threads=0 -8 --memory=$XZ_MEM
> > > ...
> > > 
> > > Best,
> > > Matthias
> > >  
> > > > Best,
> > > > -Michael
> > > > 
> > > > > 
> > > > > Best,
> > > > > Matthias
> > > > >  > Best,
> > > > > > -Michael
> > > > > > 
> > > > > > On Wed, 2017-11-22 at 16:37 +0000, Michael Tremer wrote:
> > > > > > > Hi,
> > > > > > > 
> > > > > > > On Tue, 2017-11-21 at 18:35 +0100, Matthias Fischer wrote:
> > > > > > > > Hi,
> > > > > > > > 
> > > > > > > > On 21.11.2017 15:42, Michael Tremer wrote:
> > > > > > > > > Hello,
> > > > > > > > > 
> > > > > > > > > any progress on this?
> > > > > > > > 
> > > > > > > > Not much, because I've been struggling with a flu for the past
> > > > > > > > two
> > > > > > > > weeks, and its not over yet, but:
> > > > > > > > 
> > > > > > > > In the meantime I ran several builds with different parameters.
> > > > > > > > 
> > > > > > > > IMHO we could use:
> > > > > > > > 
> > > > > > > > 'export XZ_OPT = --threads=0 -8 --memory=700MiB'
> > > > > > > > 
> > > > > > > > in 'lfs/cdrom'
> > > > > > > > 
> > > > > > > > and
> > > > > > > > 
> > > > > > > > cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8 --
> > > > > > > > memory=700MiB" tar
> > > > > > > > -c
> > > > > > > > -p
> > > > > > > > --numeric-owner -J -f /install/packages/package/files.tar.xz *
> > > > > > > > 
> > > > > > > > in 'lfs/Config'.
> > > > > > > > 
> > > > > > > > Using '9' or '--extreme'-parameter brought no further
> > > > > > > > improvements,
> > > > > > > > on the other hand, build time and RAM consumption increased
> > > > > > > > strongly. With '-8', minimal RAM consumption was about 680 MiB,
> > > > > > > > using
> > > > > > > > '700 Mib' was always safe. So this is the minimum I would
> > > > > > > > recommend.
> > > > > > > 
> > > > > > > Yes, I observed the same thing. -9 compresses a little bit better
> > > > > > > and
> > > > > > > extreme
> > > > > > > doesn't change a bit.
> > > > > > > 
> > > > > > > -8 seems to be optimal since it won't require too much memory to
> > > > > > > decompress
> > > > > > > either. So let's go with that.
> > > > > > > 
> > > > > > > > Results:
> > > > > > > > 
> > > > > > > > (Buildtime: 5 hours 20 minutes 59 seconds)
> > > > > > > > 
> > > > > > > > 255866764  ipfire-2.19.1gb-ext4.i586-full-core117.img.gz
> > > > > > > > 252262006  ipfire-2.19.1gb-ext4-scon.i586-full-core117.img.gz
> > > > > > > > 175112192  ipfire-2.19.i586-full-core117.iso
> > > > > > > >  
> > > > > > > > > -Michael
> > > > > > > > > 
> > > > > > > > > On Tue, 2017-11-07 at 22:47 +0000, Michael Tremer wrote:
> > > > > > > > > > Hi,
> > > > > > > > > > 
> > > > > > > > > > On Tue, 2017-11-07 at 19:41 +0100, Matthias Fischer wrote:
> > > > > > > > > > > Hi,
> > > > > > > > > > > 
> > > > > > > > > > > On 07.11.2017 16:05, Michael Tremer wrote:
> > > > > > > > > > > > Hi,
> > > > > > > > > > > > 
> > > > > > > > > > > > maybe this function helps:
> > > > > > > > > > > > 
> > > > > > > > > > > > https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=
> > > > > > > > > > > > 51
> > > > > > > > > > > > 90eea2
> > > > > > > > > > > > 4f98
> > > > > > > > > > > > 22
> > > > > > > > > > > > a63d
> > > > > > > > > > > > c5d06d214b48f973b14f29
> > > > > > > > > > > 
> > > > > > > > > > > Thanks. I'll take a look...
> > > > > > > > 
> > > > > > > > This would help, indeed, but how to implement(!?), and... (see
> > > > > > > > below)
> > > > > > > 
> > > > > > > Just run it somewhere in the script, figure out what is about 80%
> > > > > > > of
> > > > > > > that
> > > > > > > and
> > > > > > > if
> > > > > > > that is smaller than 800 MB, we should just fail.
> > > > > > > 
> > > > > > > It would be good to have a function that determines the xz
> > > > > > > parameters and
> > > > > > > we
> > > > > > > use
> > > > > > > that where ever we call xz. We don't want any duplicated code.
> > > > > > > 
> > > > > > > > > > > > On Tue, 2017-11-07 at 10:46 +0000, Michael Tremer wrote:
> > > > > > > > > > > > > On Mon, 2017-11-06 at 22:09 +0100, Matthias Fischer
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > ...
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > I forgot: Tuning the XZ-parameters... ;-)
> > > > > > > > > > > > > 
> > > > > > > > > > > > > Good that you are bringing this up. So I think what we
> > > > > > > > > > > > > should
> > > > > > > > > > > > > do
> > > > > > > > > > > > > here is
> > > > > > > > > > > > > bringing back XZ compression with -8 so that we do not
> > > > > > > > > > > > > waste
> > > > > > > > > > > > > too
> > > > > > > > > > > > > much
> > > > > > > > > > > > > space
> > > > > > > > > > > > > when
> > > > > > > > > > > > > extracting the image again.
> > > > > > > > > > > 
> > > > > > > > > > > Right now I'm testing how much RAM I need.
> > > > > > > > > > > 
> > > > > > > > > > > Current 'next'-build is running with:
> > > > > > > > > > > 
> > > > > > > > > > > 'lfs/cdrom':
> > > > > > > > > > > ...
> > > > > > > > > > > export XZ_OPT = --threads=0 -8 --memory=500MiB
> > > > > > > > > > > ...
> > > > > > > > > > > 
> > > > > > > > > > > 'lfs/Config':
> > > > > > > > > > > ...
> > > > > > > > > > > cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8" ...
> > > > > > > > > > > ...
> > > > > > > > > > > 
> > > > > > > > > > > I'm waiting for the results. Takes a while.
> > > > > > > > > > 
> > > > > > > > > > You can just run it like this and it will tell you:
> > > > > > > > > > 
> > > > > > > > > > [ms(a)hughes ~]$ xz -vv8 -T0
> > > > > > > > > > xz: Filter chain: --
> > > > > > > > > > lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,d
> > > > > > > > > > ep
> > > > > > > > > > th=0
> > > > > > > > > > xz: Using up to 4 threads.
> > > > > > > > > > xz: 2,629 MiB of memory is required. The limiter is
> > > > > > > > > > disabled.
> > > > > > > > > > xz: Decompression will need 33 MiB of memory.
> > > > > > > > > > xz: Compressed data cannot be written to a terminal
> > > > > > > > > > xz: Try `xz --help' for more information.
> > > > > > > > > > 
> > > > > > > > > > > > > To not run into memory limits we should detect how
> > > > > > > > > > > > > much
> > > > > > > > > > > > > memory
> > > > > > > > > > > > > the
> > > > > > > > > > > > > build
> > > > > > > > > > > > > host
> > > > > > > > > > > > > has and set the memory limit to maybe 80% of that. xz
> > > > > > > > > > > > > will
> > > > > > > > > > > > > then
> > > > > > > > > > > > > automatically
> > > > > > > > > > > > > scale down to a number of parallel processes that it
> > > > > > > > > > > > > can
> > > > > > > > > > > > > fit
> > > > > > > > > > > > > into
> > > > > > > > > > > > > memory.
> > > > > > > > 
> > > > > > > > This is something I couldn't get to work. First, I couldn't
> > > > > > > > specify not
> > > > > > > > less
> > > > > > > > than '700 MiB'. If I specify '600 MiB', building stops and tells
> > > > > > > > me it
> > > > > > > > needs
> > > > > > > > at least about 680 MiB. If I use '70%', building stops. And I
> > > > > > > > never saw
> > > > > > > > that
> > > > > > > > 'xz' "automatically scaled down". It just stopped everytime.
> > > > > > > 
> > > > > > > Yes, ~700M is the minimum.
> > > > > > > 
> > > > > > > It will just reduce the number of threads again to make sure
> > > > > > > everything
> > > > > > > fits
> > > > > > > into memory.
> > > > > > > 
> > > > > > > >  
> > > > > > > > > > > I'm a bit puzzled about this: I tried using '--memory=70%'
> > > > > > > > > > > as
> > > > > > > > > > > mentioned
> > > > > > > > > > > in the xz man pages, but is seems to have no effect. It
> > > > > > > > > > > looks as
> > > > > > > > > > > if
> > > > > > > > > > > the
> > > > > > > > > > > percent parameter isn't recognized and ignored. It always
> > > > > > > > > > > ends up
> > > > > > > > > > > in
> > > > > > > > > > > "cannot allocate memory". Any hints on this? Do both
> > > > > > > > > > > 'lfs/cdrom'
> > > > > > > > > > > AND
> > > > > > > > > > > 'lfs/Config' need the '--memory'-parameter?
> > > > > > > > 
> > > > > > > > This was the next problem: 'percent'-parameter does not work as
> > > > > > > > expected.
> > > > > > > 
> > > > > > > No, don't use it. Just determine yourself what it is.
> > > > > > > 
> > > > > > > I think we could even try to set 100% of the memory, because as
> > > > > > > far
> > > > > > > as I
> > > > > > > can
> > > > > > > see
> > > > > > > this is worst case and xz won't use everything that we allow.
> > > > > > > 
> > > > > > > If that does't work, let's scale down to 90%, or 80%, etc...
> > > > > > > 
> > > > > > > > > > Yes, but it would be good to have one place where the
> > > > > > > > > > parameters are
> > > > > > > > > > being generated and then we just use them from a variable.
> > > > > > > > 
> > > > > > > > Probably beyond my capabilities for the moment. Any thoughts?
> > > > > > > > 
> > > > > > > > > > > > > ...
> > > > > > > > > > > > > We will at least need about half a GB which should be
> > > > > > > > > > > > > a
> > > > > > > > > > > > > sensible
> > > > > > > > > > > > > requirement
> > > > > > > > > > > > > for a build system any ways.
> > > > > > > > 
> > > > > > > > I'm afraid '500 MiB' are not enough...
> > > > > > > 
> > > > > > > No, so we will now make 1G the minimum for a build.
> > > > > > > 
> > > > > > > I tried to build the OS on an ARM board with only 512MB of RAM the
> > > > > > > other
> > > > > > > day
> > > > > > > and
> > > > > > > even with a lot of swap it doesn't work. So we are past this any
> > > > > > > way.
> > > > > > > 
> > > > > > > > 
> > > > > > > > > > > Agreed. Because of this I'm testing with '500Mib'.
> > > > > > > > > > > 
> > > > > > > > > > > > > Will you work on this bringing it into the build
> > > > > > > > > > > > > system?!
> > > > > > > > > > > 
> > > > > > > > > > > "I'll do my very best!" :-))
> > > > > > > > > > 
> > > > > > > > > > Very well Miss Sophie...
> > > > > > > > > > ...
> > > > > > > > 
> > > > > > > > Best™,
> > > > > > > > Matthias
> > > > > > > 
> > > > > > > -Michael

Re: Compress images with XZ

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 17976 bytes --]

Hi,

On Sun, 2018-02-11 at 15:35 +0000, Matthias Fischer wrote:
> Hi,
> 
> On 06.02.2018 02:05, Michael Tremer wrote:
> > > ...
> > > ((XZ_MEM=(($HOST_MEM)/10)*7)) && XZ_MEM=${XZ_MEM}MiB
> > 
> > You can write this even short:
> > 
> > XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"
> 
> Oneliner! Yes. I was looking for something like that... ;-)
> 
> > I swapped the order of the multiplication and division so that you would have
> > a
> > more exact result.
> > 
> > For example if you system has 1024 MB or RAM, you divide this by 10 (102) and
> > then multiply by 7 which gives you 714. The other order is 1024 * 7 = 7167 and
> > then divided by 10 is 716 which is closer to the actual result of 716.8.
> > Shouldn't matter here but just in general this is better :)
> 
> Thanks, looks better - noted for the next time!
> 
> Now it looks like this ('make.sh'):
> 
> ...
> # Host memory is ok, calculating XZ memory
> 	XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"
> 	XZ_OPT="--threads=0 -8 $XZ_MEM"
> 	echo XZ-Memory: $XZ_MEM
> 	echo XZ-Options: $XZ_OPT
> 	echo "XZ memory size is OK (must be at least 700MiB), starting build..."
> ...
> ["echoes" for debugging!]
> 
> This is working. '$XZ_OPT' now gives me: --threads=0 -8 5450MiB

Will there be some checks in make.sh now that check if at least 1024MB
of memory are available for build? I think there should be a warning if
not enough memory is available, but we should still try to build.

> But. ;-)
> 
> > You can put this probably next under "HOST_MEM=$(system_memory)" and then
> > export
> > this to the internal shell by adding the variable to lfsmake2 for the disk
> > image
> > and ipfiredist for the packages.
> 
> Sorry, I must confess that I don't know exactly how to do this. As I wrote: "How
> do
> I pass the 'XZ_MEM'-variable to 'cdrom' and 'Config'?" Or in this case, XZ_OPT!?
> 
> > Then in lfs/cdrom and lfs/Config for the dist
> > target, you just need to add the parameters to the command line.
> 
> In 'Config' I changed:
> 
> ...
> cd /install/packages/package/tmp/ && XZ_OPT=-T0 tar -c -p --numeric-owner -J -f
> /install/packages/package/files.tar.xz *
> ...
> 
> To:
> ...
> cd /install/packages/package/tmp/ && $XZ_OPT tar -c -p --numeric-owner -J -f
> /install/packages/package/files.tar.xz *
> ...
> 
> > Maybe it is a good idea to have a variable XZ_OPT that adds everything
> > together
> > and is then used in both places.
> 
> I added the other xz-options and created XZ_OPT as shown above, but in 'cdrom'
> there
> is an 'export XZ_OPT = ...'-line and I don't know how to handle these line. Is
> it still necessary?

No, that is no longer necessary and can be removed. We should just add
it to the tar command just like it is done in lfs/Config.

> Do I have to 'export' the XZ_OPT'-variable in 'make.sh'?

No, because the chroot environment will throw away all environment
variables from the host system and then set them again. That's why we
have this enterchroot() function that does this with the env command.

> And how do I "add" this variable to 'lfsmake2' and 'ipfiredist'(-function)?

Just have a look at the functions. There should be loads of examples
where we pass other variables and you just need to add this one, too.

> Sorry if these questions sound simple, but I'm guessing a bit too much while
> trying
> to get a grip on this.

This is absolutely fine to sort this out first and then send the patch
:) That's what this list is for.

Best,
-Michael

> 
> Best,
> Matthias
> 
> > 
> > Best,
> > -Michael
> > 
> > > ...
> > > 
> > > Looks better.
> > > 
> > > Best,
> > > Matthias
> > > 
> > > On 03.02.2018 20:44, Matthias Fischer wrote:
> > > > Hi,
> > > > 
> > > > On 30.01.2018 21:03, Michael Tremer wrote:
> > > > > Hi,
> > > > > 
> > > > > ...
> > > > > > 
> > > > > > I'll consider how I would do this and send it in as soon as I get a
> > > > > > grip
> > > > > > on it.
> > > > > 
> > > > > Well, I added some functions already somewhere. I suppose you just need
> > > > > to
> > > > > do
> > > > > the maths to set that to X% of the host system which is just a simple
> > > > > multiplication of the output of ${HOST_MEM} in make.sh.
> > > > 
> > > > Ok, I take my heart in my hand. Here are my first results - but remember,
> > > > I'm not familiar with this. I built a short test script which looks as if
> > > > it
> > > > is working as wanted, but...
> > > >  
> > > > > And that needs to be passed to the lfs scripts that are supposed to make
> > > > > the
> > > > > images.
> > > > 
> > > > ...this is a problem I'm struggling with right now:
> > > > How do I pass the 'XZ_MEM'-variable to 'cdrom' and 'Config'?
> > > > 
> > > > I think this can be done using 'export XZ_MEM' or something else but I'm
> > > > not
> > > > sure.
> > > > 
> > > > This is the script I'm testing with:
> > > > 
> > > > ***SNIP***
> > > > #!/bin/bash
> > > > 
> > > > system_memory() {
> > > > 	local key val unit
> > > > 
> > > > 	while read -r key val unit; do
> > > > 		case "${key}" in
> > > > 			MemTotal:*)
> > > > 				# Convert to MB
> > > > 				echo "$(( ${val} / 1024 ))"
> > > > 				break
> > > > 				;;
> > > > 		esac
> > > > 	done < /proc/meminfo
> > > > }
> > > > 
> > > > HOST_MEM=$(system_memory)
> > > > 
> > > > # Checking host memory
> > > > if [ $HOST_MEM -ge 1024 ]; then
> > > > 	echo Host-Memory: $HOST_MEM'MiB'
> > > > 	echo "Host memory is OK (must be at least 1024MiB), starting build."
> > > > else
> > > > 	echo Host-Memory: $HOST_MEM'MiB'
> > > > 	echo "Host memory too low (less than 1024MiB), building stopped."
> > > > exit 1
> > > > fi
> > > > 
> > > > # Host memory is ok, calculating needed XZ memory to about 70% of host
> > > > memory
> > > > ((XZ_MEM=(($HOST_MEM)/10)*7))
> > > > 	XZ_MEM=$XZ_MEM'Mib'
> > > > 	echo XZ-Memory: $XZ_MEM
> > > > 	echo "XZ memory size is OK (must be at least 700MiB), starting build."
> > > > ***SNAP***
> > > > 
> > > > The 'echoes' are mostly for debugging.
> > > > 
> > > > Calculation looks ok, 'MiB' must added without blanks (XZ man page: "In
> > > > most
> > > > places where an integer argument is expected, an optional suffix is
> > > > supported
> > > > to easily indicate large integers. There must be no space between the
> > > > integer
> > > > and the suffix.")
> > > > 
> > > > > Please send questions where ever you need help. There is no reason to do
> > > > > this
> > > > > alone.
> > > > 
> > > > Please see above:
> > > > 
> > > > 1. Is this code what you where thinking of? I would leave the other
> > > > parameters
> > > > as they are (--threads=0 -8).
> > > > 
> > > > 2. Where should I add such code in the first place? I'd like to start with
> > > > 'make.sh', so building stops as soon as possible if there is not enough
> > > > memory. Where's
> > > > the best place? Right after function 'system_memory() {...}'?
> > > > 
> > > > 3. How to I pass contents of 'XZ_MEM' to 'cdrom' and 'Config'?
> > > > I consider something like:
> > > > ...
> > > > export XZ_OPT = --threads=0 -8 --memory=$XZ_MEM
> > > > ...
> > > > 
> > > > Best,
> > > > Matthias
> > > >  
> > > > > Best,
> > > > > -Michael
> > > > > 
> > > > > > 
> > > > > > Best,
> > > > > > Matthias
> > > > > >  > Best,
> > > > > > > -Michael
> > > > > > > 
> > > > > > > On Wed, 2017-11-22 at 16:37 +0000, Michael Tremer wrote:
> > > > > > > > Hi,
> > > > > > > > 
> > > > > > > > On Tue, 2017-11-21 at 18:35 +0100, Matthias Fischer wrote:
> > > > > > > > > Hi,
> > > > > > > > > 
> > > > > > > > > On 21.11.2017 15:42, Michael Tremer wrote:
> > > > > > > > > > Hello,
> > > > > > > > > > 
> > > > > > > > > > any progress on this?
> > > > > > > > > 
> > > > > > > > > Not much, because I've been struggling with a flu for the past
> > > > > > > > > two
> > > > > > > > > weeks, and its not over yet, but:
> > > > > > > > > 
> > > > > > > > > In the meantime I ran several builds with different parameters.
> > > > > > > > > 
> > > > > > > > > IMHO we could use:
> > > > > > > > > 
> > > > > > > > > 'export XZ_OPT = --threads=0 -8 --memory=700MiB'
> > > > > > > > > 
> > > > > > > > > in 'lfs/cdrom'
> > > > > > > > > 
> > > > > > > > > and
> > > > > > > > > 
> > > > > > > > > cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8 --
> > > > > > > > > memory=700MiB" tar
> > > > > > > > > -c
> > > > > > > > > -p
> > > > > > > > > --numeric-owner -J -f /install/packages/package/files.tar.xz *
> > > > > > > > > 
> > > > > > > > > in 'lfs/Config'.
> > > > > > > > > 
> > > > > > > > > Using '9' or '--extreme'-parameter brought no further
> > > > > > > > > improvements,
> > > > > > > > > on the other hand, build time and RAM consumption increased
> > > > > > > > > strongly. With '-8', minimal RAM consumption was about 680 MiB,
> > > > > > > > > using
> > > > > > > > > '700 Mib' was always safe. So this is the minimum I would
> > > > > > > > > recommend.
> > > > > > > > 
> > > > > > > > Yes, I observed the same thing. -9 compresses a little bit better
> > > > > > > > and
> > > > > > > > extreme
> > > > > > > > doesn't change a bit.
> > > > > > > > 
> > > > > > > > -8 seems to be optimal since it won't require too much memory to
> > > > > > > > decompress
> > > > > > > > either. So let's go with that.
> > > > > > > > 
> > > > > > > > > Results:
> > > > > > > > > 
> > > > > > > > > (Buildtime: 5 hours 20 minutes 59 seconds)
> > > > > > > > > 
> > > > > > > > > 255866764  ipfire-2.19.1gb-ext4.i586-full-core117.img.gz
> > > > > > > > > 252262006  ipfire-2.19.1gb-ext4-scon.i586-full-core117.img.gz
> > > > > > > > > 175112192  ipfire-2.19.i586-full-core117.iso
> > > > > > > > >  
> > > > > > > > > > -Michael
> > > > > > > > > > 
> > > > > > > > > > On Tue, 2017-11-07 at 22:47 +0000, Michael Tremer wrote:
> > > > > > > > > > > Hi,
> > > > > > > > > > > 
> > > > > > > > > > > On Tue, 2017-11-07 at 19:41 +0100, Matthias Fischer wrote:
> > > > > > > > > > > > Hi,
> > > > > > > > > > > > 
> > > > > > > > > > > > On 07.11.2017 16:05, Michael Tremer wrote:
> > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > 
> > > > > > > > > > > > > maybe this function helps:
> > > > > > > > > > > > > 
> > > > > > > > > > > > > https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=
> > > > > > > > > > > > > 51
> > > > > > > > > > > > > 90eea2
> > > > > > > > > > > > > 4f98
> > > > > > > > > > > > > 22
> > > > > > > > > > > > > a63d
> > > > > > > > > > > > > c5d06d214b48f973b14f29
> > > > > > > > > > > > 
> > > > > > > > > > > > Thanks. I'll take a look...
> > > > > > > > > 
> > > > > > > > > This would help, indeed, but how to implement(!?), and... (see
> > > > > > > > > below)
> > > > > > > > 
> > > > > > > > Just run it somewhere in the script, figure out what is about 80%
> > > > > > > > of
> > > > > > > > that
> > > > > > > > and
> > > > > > > > if
> > > > > > > > that is smaller than 800 MB, we should just fail.
> > > > > > > > 
> > > > > > > > It would be good to have a function that determines the xz
> > > > > > > > parameters and
> > > > > > > > we
> > > > > > > > use
> > > > > > > > that where ever we call xz. We don't want any duplicated code.
> > > > > > > > 
> > > > > > > > > > > > > On Tue, 2017-11-07 at 10:46 +0000, Michael Tremer wrote:
> > > > > > > > > > > > > > On Mon, 2017-11-06 at 22:09 +0100, Matthias Fischer
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > ...
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > I forgot: Tuning the XZ-parameters... ;-)
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > Good that you are bringing this up. So I think what we
> > > > > > > > > > > > > > should
> > > > > > > > > > > > > > do
> > > > > > > > > > > > > > here is
> > > > > > > > > > > > > > bringing back XZ compression with -8 so that we do not
> > > > > > > > > > > > > > waste
> > > > > > > > > > > > > > too
> > > > > > > > > > > > > > much
> > > > > > > > > > > > > > space
> > > > > > > > > > > > > > when
> > > > > > > > > > > > > > extracting the image again.
> > > > > > > > > > > > 
> > > > > > > > > > > > Right now I'm testing how much RAM I need.
> > > > > > > > > > > > 
> > > > > > > > > > > > Current 'next'-build is running with:
> > > > > > > > > > > > 
> > > > > > > > > > > > 'lfs/cdrom':
> > > > > > > > > > > > ...
> > > > > > > > > > > > export XZ_OPT = --threads=0 -8 --memory=500MiB
> > > > > > > > > > > > ...
> > > > > > > > > > > > 
> > > > > > > > > > > > 'lfs/Config':
> > > > > > > > > > > > ...
> > > > > > > > > > > > cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8" ...
> > > > > > > > > > > > ...
> > > > > > > > > > > > 
> > > > > > > > > > > > I'm waiting for the results. Takes a while.
> > > > > > > > > > > 
> > > > > > > > > > > You can just run it like this and it will tell you:
> > > > > > > > > > > 
> > > > > > > > > > > [ms(a)hughes ~]$ xz -vv8 -T0
> > > > > > > > > > > xz: Filter chain: --
> > > > > > > > > > > lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,d
> > > > > > > > > > > ep
> > > > > > > > > > > th=0
> > > > > > > > > > > xz: Using up to 4 threads.
> > > > > > > > > > > xz: 2,629 MiB of memory is required. The limiter is
> > > > > > > > > > > disabled.
> > > > > > > > > > > xz: Decompression will need 33 MiB of memory.
> > > > > > > > > > > xz: Compressed data cannot be written to a terminal
> > > > > > > > > > > xz: Try `xz --help' for more information.
> > > > > > > > > > > 
> > > > > > > > > > > > > > To not run into memory limits we should detect how
> > > > > > > > > > > > > > much
> > > > > > > > > > > > > > memory
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > build
> > > > > > > > > > > > > > host
> > > > > > > > > > > > > > has and set the memory limit to maybe 80% of that. xz
> > > > > > > > > > > > > > will
> > > > > > > > > > > > > > then
> > > > > > > > > > > > > > automatically
> > > > > > > > > > > > > > scale down to a number of parallel processes that it
> > > > > > > > > > > > > > can
> > > > > > > > > > > > > > fit
> > > > > > > > > > > > > > into
> > > > > > > > > > > > > > memory.
> > > > > > > > > 
> > > > > > > > > This is something I couldn't get to work. First, I couldn't
> > > > > > > > > specify not
> > > > > > > > > less
> > > > > > > > > than '700 MiB'. If I specify '600 MiB', building stops and tells
> > > > > > > > > me it
> > > > > > > > > needs
> > > > > > > > > at least about 680 MiB. If I use '70%', building stops. And I
> > > > > > > > > never saw
> > > > > > > > > that
> > > > > > > > > 'xz' "automatically scaled down". It just stopped everytime.
> > > > > > > > 
> > > > > > > > Yes, ~700M is the minimum.
> > > > > > > > 
> > > > > > > > It will just reduce the number of threads again to make sure
> > > > > > > > everything
> > > > > > > > fits
> > > > > > > > into memory.
> > > > > > > > 
> > > > > > > > >  
> > > > > > > > > > > > I'm a bit puzzled about this: I tried using '--memory=70%'
> > > > > > > > > > > > as
> > > > > > > > > > > > mentioned
> > > > > > > > > > > > in the xz man pages, but is seems to have no effect. It
> > > > > > > > > > > > looks as
> > > > > > > > > > > > if
> > > > > > > > > > > > the
> > > > > > > > > > > > percent parameter isn't recognized and ignored. It always
> > > > > > > > > > > > ends up
> > > > > > > > > > > > in
> > > > > > > > > > > > "cannot allocate memory". Any hints on this? Do both
> > > > > > > > > > > > 'lfs/cdrom'
> > > > > > > > > > > > AND
> > > > > > > > > > > > 'lfs/Config' need the '--memory'-parameter?
> > > > > > > > > 
> > > > > > > > > This was the next problem: 'percent'-parameter does not work as
> > > > > > > > > expected.
> > > > > > > > 
> > > > > > > > No, don't use it. Just determine yourself what it is.
> > > > > > > > 
> > > > > > > > I think we could even try to set 100% of the memory, because as
> > > > > > > > far
> > > > > > > > as I
> > > > > > > > can
> > > > > > > > see
> > > > > > > > this is worst case and xz won't use everything that we allow.
> > > > > > > > 
> > > > > > > > If that does't work, let's scale down to 90%, or 80%, etc...
> > > > > > > > 
> > > > > > > > > > > Yes, but it would be good to have one place where the
> > > > > > > > > > > parameters are
> > > > > > > > > > > being generated and then we just use them from a variable.
> > > > > > > > > 
> > > > > > > > > Probably beyond my capabilities for the moment. Any thoughts?
> > > > > > > > > 
> > > > > > > > > > > > > > ...
> > > > > > > > > > > > > > We will at least need about half a GB which should be
> > > > > > > > > > > > > > a
> > > > > > > > > > > > > > sensible
> > > > > > > > > > > > > > requirement
> > > > > > > > > > > > > > for a build system any ways.
> > > > > > > > > 
> > > > > > > > > I'm afraid '500 MiB' are not enough...
> > > > > > > > 
> > > > > > > > No, so we will now make 1G the minimum for a build.
> > > > > > > > 
> > > > > > > > I tried to build the OS on an ARM board with only 512MB of RAM the
> > > > > > > > other
> > > > > > > > day
> > > > > > > > and
> > > > > > > > even with a lot of swap it doesn't work. So we are past this any
> > > > > > > > way.
> > > > > > > > 
> > > > > > > > > 
> > > > > > > > > > > > Agreed. Because of this I'm testing with '500Mib'.
> > > > > > > > > > > > 
> > > > > > > > > > > > > > Will you work on this bringing it into the build
> > > > > > > > > > > > > > system?!
> > > > > > > > > > > > 
> > > > > > > > > > > > "I'll do my very best!" :-))
> > > > > > > > > > > 
> > > > > > > > > > > Very well Miss Sophie...
> > > > > > > > > > > ...
> > > > > > > > > 
> > > > > > > > > Best™,
> > > > > > > > > Matthias
> > > > > > > > 
> > > > > > > > -Michael

Re: Compress images with XZ

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 20270 bytes --]

Hey Matthias,

are you still working on this?

I am asking because I just noticed that the new next build is about 50MB larger
than the old one due to the larger kernel and loads more of firmware files.

Compressing it better would of course help to save space on disk.

Best,
-Michael

On Sun, 2018-02-11 at 19:57 +0000, Michael Tremer wrote:
> Hi,
> 
> On Sun, 2018-02-11 at 15:35 +0000, Matthias Fischer wrote:
> > Hi,
> > 
> > On 06.02.2018 02:05, Michael Tremer wrote:
> > > > ...
> > > > ((XZ_MEM=(($HOST_MEM)/10)*7)) && XZ_MEM=${XZ_MEM}MiB
> > > 
> > > You can write this even short:
> > > 
> > > XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"
> > 
> > Oneliner! Yes. I was looking for something like that... ;-)
> > 
> > > I swapped the order of the multiplication and division so that you would
> > > have
> > > a
> > > more exact result.
> > > 
> > > For example if you system has 1024 MB or RAM, you divide this by 10 (102)
> > > and
> > > then multiply by 7 which gives you 714. The other order is 1024 * 7 = 7167
> > > and
> > > then divided by 10 is 716 which is closer to the actual result of 716.8.
> > > Shouldn't matter here but just in general this is better :)
> > 
> > Thanks, looks better - noted for the next time!
> > 
> > Now it looks like this ('make.sh'):
> > 
> > ...
> > # Host memory is ok, calculating XZ memory
> > 	XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"
> > 	XZ_OPT="--threads=0 -8 $XZ_MEM"
> > 	echo XZ-Memory: $XZ_MEM
> > 	echo XZ-Options: $XZ_OPT
> > 	echo "XZ memory size is OK (must be at least 700MiB), starting build..."
> > ...
> > ["echoes" for debugging!]
> > 
> > This is working. '$XZ_OPT' now gives me: --threads=0 -8 5450MiB
> 
> Will there be some checks in make.sh now that check if at least 1024MB
> of memory are available for build? I think there should be a warning if
> not enough memory is available, but we should still try to build.
> 
> > But. ;-)
> > 
> > > You can put this probably next under "HOST_MEM=$(system_memory)" and then
> > > export
> > > this to the internal shell by adding the variable to lfsmake2 for the disk
> > > image
> > > and ipfiredist for the packages.
> > 
> > Sorry, I must confess that I don't know exactly how to do this. As I wrote:
> > "How
> > do
> > I pass the 'XZ_MEM'-variable to 'cdrom' and 'Config'?" Or in this case,
> > XZ_OPT!?
> > 
> > > Then in lfs/cdrom and lfs/Config for the dist
> > > target, you just need to add the parameters to the command line.
> > 
> > In 'Config' I changed:
> > 
> > ...
> > cd /install/packages/package/tmp/ && XZ_OPT=-T0 tar -c -p --numeric-owner -J
> > -f
> > /install/packages/package/files.tar.xz *
> > ...
> > 
> > To:
> > ...
> > cd /install/packages/package/tmp/ && $XZ_OPT tar -c -p --numeric-owner -J -f
> > /install/packages/package/files.tar.xz *
> > ...
> > 
> > > Maybe it is a good idea to have a variable XZ_OPT that adds everything
> > > together
> > > and is then used in both places.
> > 
> > I added the other xz-options and created XZ_OPT as shown above, but in
> > 'cdrom'
> > there
> > is an 'export XZ_OPT = ...'-line and I don't know how to handle these line.
> > Is
> > it still necessary?
> 
> No, that is no longer necessary and can be removed. We should just add
> it to the tar command just like it is done in lfs/Config.
> 
> > Do I have to 'export' the XZ_OPT'-variable in 'make.sh'?
> 
> No, because the chroot environment will throw away all environment
> variables from the host system and then set them again. That's why we
> have this enterchroot() function that does this with the env command.
> 
> > And how do I "add" this variable to 'lfsmake2' and 'ipfiredist'(-function)?
> 
> Just have a look at the functions. There should be loads of examples
> where we pass other variables and you just need to add this one, too.
> 
> > Sorry if these questions sound simple, but I'm guessing a bit too much while
> > trying
> > to get a grip on this.
> 
> This is absolutely fine to sort this out first and then send the patch
> :) That's what this list is for.
> 
> Best,
> -Michael
> 
> > 
> > Best,
> > Matthias
> > 
> > > 
> > > Best,
> > > -Michael
> > > 
> > > > ...
> > > > 
> > > > Looks better.
> > > > 
> > > > Best,
> > > > Matthias
> > > > 
> > > > On 03.02.2018 20:44, Matthias Fischer wrote:
> > > > > Hi,
> > > > > 
> > > > > On 30.01.2018 21:03, Michael Tremer wrote:
> > > > > > Hi,
> > > > > > 
> > > > > > ...
> > > > > > > 
> > > > > > > I'll consider how I would do this and send it in as soon as I get
> > > > > > > a
> > > > > > > grip
> > > > > > > on it.
> > > > > > 
> > > > > > Well, I added some functions already somewhere. I suppose you just
> > > > > > need
> > > > > > to
> > > > > > do
> > > > > > the maths to set that to X% of the host system which is just a
> > > > > > simple
> > > > > > multiplication of the output of ${HOST_MEM} in make.sh.
> > > > > 
> > > > > Ok, I take my heart in my hand. Here are my first results - but
> > > > > remember,
> > > > > I'm not familiar with this. I built a short test script which looks as
> > > > > if
> > > > > it
> > > > > is working as wanted, but...
> > > > >  
> > > > > > And that needs to be passed to the lfs scripts that are supposed to
> > > > > > make
> > > > > > the
> > > > > > images.
> > > > > 
> > > > > ...this is a problem I'm struggling with right now:
> > > > > How do I pass the 'XZ_MEM'-variable to 'cdrom' and 'Config'?
> > > > > 
> > > > > I think this can be done using 'export XZ_MEM' or something else but
> > > > > I'm
> > > > > not
> > > > > sure.
> > > > > 
> > > > > This is the script I'm testing with:
> > > > > 
> > > > > ***SNIP***
> > > > > #!/bin/bash
> > > > > 
> > > > > system_memory() {
> > > > > 	local key val unit
> > > > > 
> > > > > 	while read -r key val unit; do
> > > > > 		case "${key}" in
> > > > > 			MemTotal:*)
> > > > > 				# Convert to MB
> > > > > 				echo "$(( ${val} / 1024 ))"
> > > > > 				break
> > > > > 				;;
> > > > > 		esac
> > > > > 	done < /proc/meminfo
> > > > > }
> > > > > 
> > > > > HOST_MEM=$(system_memory)
> > > > > 
> > > > > # Checking host memory
> > > > > if [ $HOST_MEM -ge 1024 ]; then
> > > > > 	echo Host-Memory: $HOST_MEM'MiB'
> > > > > 	echo "Host memory is OK (must be at least 1024MiB), starting
> > > > > build."
> > > > > else
> > > > > 	echo Host-Memory: $HOST_MEM'MiB'
> > > > > 	echo "Host memory too low (less than 1024MiB), building
> > > > > stopped."
> > > > > exit 1
> > > > > fi
> > > > > 
> > > > > # Host memory is ok, calculating needed XZ memory to about 70% of host
> > > > > memory
> > > > > ((XZ_MEM=(($HOST_MEM)/10)*7))
> > > > > 	XZ_MEM=$XZ_MEM'Mib'
> > > > > 	echo XZ-Memory: $XZ_MEM
> > > > > 	echo "XZ memory size is OK (must be at least 700MiB), starting
> > > > > build."
> > > > > ***SNAP***
> > > > > 
> > > > > The 'echoes' are mostly for debugging.
> > > > > 
> > > > > Calculation looks ok, 'MiB' must added without blanks (XZ man page:
> > > > > "In
> > > > > most
> > > > > places where an integer argument is expected, an optional suffix is
> > > > > supported
> > > > > to easily indicate large integers. There must be no space between the
> > > > > integer
> > > > > and the suffix.")
> > > > > 
> > > > > > Please send questions where ever you need help. There is no reason
> > > > > > to do
> > > > > > this
> > > > > > alone.
> > > > > 
> > > > > Please see above:
> > > > > 
> > > > > 1. Is this code what you where thinking of? I would leave the other
> > > > > parameters
> > > > > as they are (--threads=0 -8).
> > > > > 
> > > > > 2. Where should I add such code in the first place? I'd like to start
> > > > > with
> > > > > 'make.sh', so building stops as soon as possible if there is not
> > > > > enough
> > > > > memory. Where's
> > > > > the best place? Right after function 'system_memory() {...}'?
> > > > > 
> > > > > 3. How to I pass contents of 'XZ_MEM' to 'cdrom' and 'Config'?
> > > > > I consider something like:
> > > > > ...
> > > > > export XZ_OPT = --threads=0 -8 --memory=$XZ_MEM
> > > > > ...
> > > > > 
> > > > > Best,
> > > > > Matthias
> > > > >  
> > > > > > Best,
> > > > > > -Michael
> > > > > > 
> > > > > > > 
> > > > > > > Best,
> > > > > > > Matthias
> > > > > > >  > Best,
> > > > > > > > -Michael
> > > > > > > > 
> > > > > > > > On Wed, 2017-11-22 at 16:37 +0000, Michael Tremer wrote:
> > > > > > > > > Hi,
> > > > > > > > > 
> > > > > > > > > On Tue, 2017-11-21 at 18:35 +0100, Matthias Fischer wrote:
> > > > > > > > > > Hi,
> > > > > > > > > > 
> > > > > > > > > > On 21.11.2017 15:42, Michael Tremer wrote:
> > > > > > > > > > > Hello,
> > > > > > > > > > > 
> > > > > > > > > > > any progress on this?
> > > > > > > > > > 
> > > > > > > > > > Not much, because I've been struggling with a flu for the
> > > > > > > > > > past
> > > > > > > > > > two
> > > > > > > > > > weeks, and its not over yet, but:
> > > > > > > > > > 
> > > > > > > > > > In the meantime I ran several builds with different
> > > > > > > > > > parameters.
> > > > > > > > > > 
> > > > > > > > > > IMHO we could use:
> > > > > > > > > > 
> > > > > > > > > > 'export XZ_OPT = --threads=0 -8 --memory=700MiB'
> > > > > > > > > > 
> > > > > > > > > > in 'lfs/cdrom'
> > > > > > > > > > 
> > > > > > > > > > and
> > > > > > > > > > 
> > > > > > > > > > cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8 --
> > > > > > > > > > memory=700MiB" tar
> > > > > > > > > > -c
> > > > > > > > > > -p
> > > > > > > > > > --numeric-owner -J -f /install/packages/package/files.tar.xz
> > > > > > > > > > *
> > > > > > > > > > 
> > > > > > > > > > in 'lfs/Config'.
> > > > > > > > > > 
> > > > > > > > > > Using '9' or '--extreme'-parameter brought no further
> > > > > > > > > > improvements,
> > > > > > > > > > on the other hand, build time and RAM consumption increased
> > > > > > > > > > strongly. With '-8', minimal RAM consumption was about 680
> > > > > > > > > > MiB,
> > > > > > > > > > using
> > > > > > > > > > '700 Mib' was always safe. So this is the minimum I would
> > > > > > > > > > recommend.
> > > > > > > > > 
> > > > > > > > > Yes, I observed the same thing. -9 compresses a little bit
> > > > > > > > > better
> > > > > > > > > and
> > > > > > > > > extreme
> > > > > > > > > doesn't change a bit.
> > > > > > > > > 
> > > > > > > > > -8 seems to be optimal since it won't require too much memory
> > > > > > > > > to
> > > > > > > > > decompress
> > > > > > > > > either. So let's go with that.
> > > > > > > > > 
> > > > > > > > > > Results:
> > > > > > > > > > 
> > > > > > > > > > (Buildtime: 5 hours 20 minutes 59 seconds)
> > > > > > > > > > 
> > > > > > > > > > 255866764  ipfire-2.19.1gb-ext4.i586-full-core117.img.gz
> > > > > > > > > > 252262006  ipfire-2.19.1gb-ext4-scon.i586-full-
> > > > > > > > > > core117.img.gz
> > > > > > > > > > 175112192  ipfire-2.19.i586-full-core117.iso
> > > > > > > > > >  
> > > > > > > > > > > -Michael
> > > > > > > > > > > 
> > > > > > > > > > > On Tue, 2017-11-07 at 22:47 +0000, Michael Tremer wrote:
> > > > > > > > > > > > Hi,
> > > > > > > > > > > > 
> > > > > > > > > > > > On Tue, 2017-11-07 at 19:41 +0100, Matthias Fischer
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > 
> > > > > > > > > > > > > On 07.11.2017 16:05, Michael Tremer wrote:
> > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > maybe this function helps:
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdif
> > > > > > > > > > > > > > f;h=
> > > > > > > > > > > > > > 51
> > > > > > > > > > > > > > 90eea2
> > > > > > > > > > > > > > 4f98
> > > > > > > > > > > > > > 22
> > > > > > > > > > > > > > a63d
> > > > > > > > > > > > > > c5d06d214b48f973b14f29
> > > > > > > > > > > > > 
> > > > > > > > > > > > > Thanks. I'll take a look...
> > > > > > > > > > 
> > > > > > > > > > This would help, indeed, but how to implement(!?), and...
> > > > > > > > > > (see
> > > > > > > > > > below)
> > > > > > > > > 
> > > > > > > > > Just run it somewhere in the script, figure out what is about
> > > > > > > > > 80%
> > > > > > > > > of
> > > > > > > > > that
> > > > > > > > > and
> > > > > > > > > if
> > > > > > > > > that is smaller than 800 MB, we should just fail.
> > > > > > > > > 
> > > > > > > > > It would be good to have a function that determines the xz
> > > > > > > > > parameters and
> > > > > > > > > we
> > > > > > > > > use
> > > > > > > > > that where ever we call xz. We don't want any duplicated code.
> > > > > > > > > 
> > > > > > > > > > > > > > On Tue, 2017-11-07 at 10:46 +0000, Michael Tremer
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > On Mon, 2017-11-06 at 22:09 +0100, Matthias
> > > > > > > > > > > > > > > Fischer
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > ...
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > I forgot: Tuning the XZ-parameters... ;-)
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > Good that you are bringing this up. So I think
> > > > > > > > > > > > > > > what we
> > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > do
> > > > > > > > > > > > > > > here is
> > > > > > > > > > > > > > > bringing back XZ compression with -8 so that we do
> > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > waste
> > > > > > > > > > > > > > > too
> > > > > > > > > > > > > > > much
> > > > > > > > > > > > > > > space
> > > > > > > > > > > > > > > when
> > > > > > > > > > > > > > > extracting the image again.
> > > > > > > > > > > > > 
> > > > > > > > > > > > > Right now I'm testing how much RAM I need.
> > > > > > > > > > > > > 
> > > > > > > > > > > > > Current 'next'-build is running with:
> > > > > > > > > > > > > 
> > > > > > > > > > > > > 'lfs/cdrom':
> > > > > > > > > > > > > ...
> > > > > > > > > > > > > export XZ_OPT = --threads=0 -8 --memory=500MiB
> > > > > > > > > > > > > ...
> > > > > > > > > > > > > 
> > > > > > > > > > > > > 'lfs/Config':
> > > > > > > > > > > > > ...
> > > > > > > > > > > > > cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8"
> > > > > > > > > > > > > ...
> > > > > > > > > > > > > ...
> > > > > > > > > > > > > 
> > > > > > > > > > > > > I'm waiting for the results. Takes a while.
> > > > > > > > > > > > 
> > > > > > > > > > > > You can just run it like this and it will tell you:
> > > > > > > > > > > > 
> > > > > > > > > > > > [ms(a)hughes ~]$ xz -vv8 -T0
> > > > > > > > > > > > xz: Filter chain: --
> > > > > > > > > > > > lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=b
> > > > > > > > > > > > t4,d
> > > > > > > > > > > > ep
> > > > > > > > > > > > th=0
> > > > > > > > > > > > xz: Using up to 4 threads.
> > > > > > > > > > > > xz: 2,629 MiB of memory is required. The limiter is
> > > > > > > > > > > > disabled.
> > > > > > > > > > > > xz: Decompression will need 33 MiB of memory.
> > > > > > > > > > > > xz: Compressed data cannot be written to a terminal
> > > > > > > > > > > > xz: Try `xz --help' for more information.
> > > > > > > > > > > > 
> > > > > > > > > > > > > > > To not run into memory limits we should detect how
> > > > > > > > > > > > > > > much
> > > > > > > > > > > > > > > memory
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > build
> > > > > > > > > > > > > > > host
> > > > > > > > > > > > > > > has and set the memory limit to maybe 80% of that.
> > > > > > > > > > > > > > > xz
> > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > then
> > > > > > > > > > > > > > > automatically
> > > > > > > > > > > > > > > scale down to a number of parallel processes that
> > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > fit
> > > > > > > > > > > > > > > into
> > > > > > > > > > > > > > > memory.
> > > > > > > > > > 
> > > > > > > > > > This is something I couldn't get to work. First, I couldn't
> > > > > > > > > > specify not
> > > > > > > > > > less
> > > > > > > > > > than '700 MiB'. If I specify '600 MiB', building stops and
> > > > > > > > > > tells
> > > > > > > > > > me it
> > > > > > > > > > needs
> > > > > > > > > > at least about 680 MiB. If I use '70%', building stops. And
> > > > > > > > > > I
> > > > > > > > > > never saw
> > > > > > > > > > that
> > > > > > > > > > 'xz' "automatically scaled down". It just stopped everytime.
> > > > > > > > > 
> > > > > > > > > Yes, ~700M is the minimum.
> > > > > > > > > 
> > > > > > > > > It will just reduce the number of threads again to make sure
> > > > > > > > > everything
> > > > > > > > > fits
> > > > > > > > > into memory.
> > > > > > > > > 
> > > > > > > > > >  
> > > > > > > > > > > > > I'm a bit puzzled about this: I tried using '
> > > > > > > > > > > > > --memory=70%'
> > > > > > > > > > > > > as
> > > > > > > > > > > > > mentioned
> > > > > > > > > > > > > in the xz man pages, but is seems to have no effect.
> > > > > > > > > > > > > It
> > > > > > > > > > > > > looks as
> > > > > > > > > > > > > if
> > > > > > > > > > > > > the
> > > > > > > > > > > > > percent parameter isn't recognized and ignored. It
> > > > > > > > > > > > > always
> > > > > > > > > > > > > ends up
> > > > > > > > > > > > > in
> > > > > > > > > > > > > "cannot allocate memory". Any hints on this? Do both
> > > > > > > > > > > > > 'lfs/cdrom'
> > > > > > > > > > > > > AND
> > > > > > > > > > > > > 'lfs/Config' need the '--memory'-parameter?
> > > > > > > > > > 
> > > > > > > > > > This was the next problem: 'percent'-parameter does not work
> > > > > > > > > > as
> > > > > > > > > > expected.
> > > > > > > > > 
> > > > > > > > > No, don't use it. Just determine yourself what it is.
> > > > > > > > > 
> > > > > > > > > I think we could even try to set 100% of the memory, because
> > > > > > > > > as
> > > > > > > > > far
> > > > > > > > > as I
> > > > > > > > > can
> > > > > > > > > see
> > > > > > > > > this is worst case and xz won't use everything that we allow.
> > > > > > > > > 
> > > > > > > > > If that does't work, let's scale down to 90%, or 80%, etc...
> > > > > > > > > 
> > > > > > > > > > > > Yes, but it would be good to have one place where the
> > > > > > > > > > > > parameters are
> > > > > > > > > > > > being generated and then we just use them from a
> > > > > > > > > > > > variable.
> > > > > > > > > > 
> > > > > > > > > > Probably beyond my capabilities for the moment. Any
> > > > > > > > > > thoughts?
> > > > > > > > > > 
> > > > > > > > > > > > > > > ...
> > > > > > > > > > > > > > > We will at least need about half a GB which should
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > sensible
> > > > > > > > > > > > > > > requirement
> > > > > > > > > > > > > > > for a build system any ways.
> > > > > > > > > > 
> > > > > > > > > > I'm afraid '500 MiB' are not enough...
> > > > > > > > > 
> > > > > > > > > No, so we will now make 1G the minimum for a build.
> > > > > > > > > 
> > > > > > > > > I tried to build the OS on an ARM board with only 512MB of RAM
> > > > > > > > > the
> > > > > > > > > other
> > > > > > > > > day
> > > > > > > > > and
> > > > > > > > > even with a lot of swap it doesn't work. So we are past this
> > > > > > > > > any
> > > > > > > > > way.
> > > > > > > > > 
> > > > > > > > > > 
> > > > > > > > > > > > > Agreed. Because of this I'm testing with '500Mib'.
> > > > > > > > > > > > > 
> > > > > > > > > > > > > > > Will you work on this bringing it into the build
> > > > > > > > > > > > > > > system?!
> > > > > > > > > > > > > 
> > > > > > > > > > > > > "I'll do my very best!" :-))
> > > > > > > > > > > > 
> > > > > > > > > > > > Very well Miss Sophie...
> > > > > > > > > > > > ...
> > > > > > > > > > 
> > > > > > > > > > Best™,
> > > > > > > > > > Matthias
> > > > > > > > > 
> > > > > > > > > -Michael

Re: Compress images with XZ

[Tom Rymes]

[-- Attachment #1: Type: text/plain, Size: 17510 bytes --]

Would it make sense to make the firmware files a separate download, as only those who use wireless Red or hostAPD need them?

Tom

> On May 11, 2018, at 4:28 PM, Michael Tremer <michael.tremer(a)ipfire.org> wrote:
> 
> Hey Matthias,
> 
> are you still working on this?
> 
> I am asking because I just noticed that the new next build is about 50MB larger
> than the old one due to the larger kernel and loads more of firmware files.
> 
> Compressing it better would of course help to save space on disk.
> 
> Best,
> -Michael
> 
>> On Sun, 2018-02-11 at 19:57 +0000, Michael Tremer wrote:
>> Hi,
>> 
>>> On Sun, 2018-02-11 at 15:35 +0000, Matthias Fischer wrote:
>>> Hi,
>>> 
>>> On 06.02.2018 02:05, Michael Tremer wrote:
>>>>> ...
>>>>> ((XZ_MEM=(($HOST_MEM)/10)*7)) && XZ_MEM=${XZ_MEM}MiB
>>>> 
>>>> You can write this even short:
>>>> 
>>>> XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"
>>> 
>>> Oneliner! Yes. I was looking for something like that... ;-)
>>> 
>>>> I swapped the order of the multiplication and division so that you would
>>>> have
>>>> a
>>>> more exact result.
>>>> 
>>>> For example if you system has 1024 MB or RAM, you divide this by 10 (102)
>>>> and
>>>> then multiply by 7 which gives you 714. The other order is 1024 * 7 = 7167
>>>> and
>>>> then divided by 10 is 716 which is closer to the actual result of 716.8.
>>>> Shouldn't matter here but just in general this is better :)
>>> 
>>> Thanks, looks better - noted for the next time!
>>> 
>>> Now it looks like this ('make.sh'):
>>> 
>>> ...
>>> # Host memory is ok, calculating XZ memory
>>>    XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"
>>>    XZ_OPT="--threads=0 -8 $XZ_MEM"
>>>    echo XZ-Memory: $XZ_MEM
>>>    echo XZ-Options: $XZ_OPT
>>>    echo "XZ memory size is OK (must be at least 700MiB), starting build..."
>>> ...
>>> ["echoes" for debugging!]
>>> 
>>> This is working. '$XZ_OPT' now gives me: --threads=0 -8 5450MiB
>> 
>> Will there be some checks in make.sh now that check if at least 1024MB
>> of memory are available for build? I think there should be a warning if
>> not enough memory is available, but we should still try to build.
>> 
>>> But. ;-)
>>> 
>>>> You can put this probably next under "HOST_MEM=$(system_memory)" and then
>>>> export
>>>> this to the internal shell by adding the variable to lfsmake2 for the disk
>>>> image
>>>> and ipfiredist for the packages.
>>> 
>>> Sorry, I must confess that I don't know exactly how to do this. As I wrote:
>>> "How
>>> do
>>> I pass the 'XZ_MEM'-variable to 'cdrom' and 'Config'?" Or in this case,
>>> XZ_OPT!?
>>> 
>>>> Then in lfs/cdrom and lfs/Config for the dist
>>>> target, you just need to add the parameters to the command line.
>>> 
>>> In 'Config' I changed:
>>> 
>>> ...
>>> cd /install/packages/package/tmp/ && XZ_OPT=-T0 tar -c -p --numeric-owner -J
>>> -f
>>> /install/packages/package/files.tar.xz *
>>> ...
>>> 
>>> To:
>>> ...
>>> cd /install/packages/package/tmp/ && $XZ_OPT tar -c -p --numeric-owner -J -f
>>> /install/packages/package/files.tar.xz *
>>> ...
>>> 
>>>> Maybe it is a good idea to have a variable XZ_OPT that adds everything
>>>> together
>>>> and is then used in both places.
>>> 
>>> I added the other xz-options and created XZ_OPT as shown above, but in
>>> 'cdrom'
>>> there
>>> is an 'export XZ_OPT = ...'-line and I don't know how to handle these line.
>>> Is
>>> it still necessary?
>> 
>> No, that is no longer necessary and can be removed. We should just add
>> it to the tar command just like it is done in lfs/Config.
>> 
>>> Do I have to 'export' the XZ_OPT'-variable in 'make.sh'?
>> 
>> No, because the chroot environment will throw away all environment
>> variables from the host system and then set them again. That's why we
>> have this enterchroot() function that does this with the env command.
>> 
>>> And how do I "add" this variable to 'lfsmake2' and 'ipfiredist'(-function)?
>> 
>> Just have a look at the functions. There should be loads of examples
>> where we pass other variables and you just need to add this one, too.
>> 
>>> Sorry if these questions sound simple, but I'm guessing a bit too much while
>>> trying
>>> to get a grip on this.
>> 
>> This is absolutely fine to sort this out first and then send the patch
>> :) That's what this list is for.
>> 
>> Best,
>> -Michael
>> 
>>> 
>>> Best,
>>> Matthias
>>> 
>>>> 
>>>> Best,
>>>> -Michael
>>>> 
>>>>> ...
>>>>> 
>>>>> Looks better.
>>>>> 
>>>>> Best,
>>>>> Matthias
>>>>> 
>>>>>> On 03.02.2018 20:44, Matthias Fischer wrote:
>>>>>> Hi,
>>>>>> 
>>>>>>> On 30.01.2018 21:03, Michael Tremer wrote:
>>>>>>> Hi,
>>>>>>> 
>>>>>>> ...
>>>>>>>> 
>>>>>>>> I'll consider how I would do this and send it in as soon as I get
>>>>>>>> a
>>>>>>>> grip
>>>>>>>> on it.
>>>>>>> 
>>>>>>> Well, I added some functions already somewhere. I suppose you just
>>>>>>> need
>>>>>>> to
>>>>>>> do
>>>>>>> the maths to set that to X% of the host system which is just a
>>>>>>> simple
>>>>>>> multiplication of the output of ${HOST_MEM} in make.sh.
>>>>>> 
>>>>>> Ok, I take my heart in my hand. Here are my first results - but
>>>>>> remember,
>>>>>> I'm not familiar with this. I built a short test script which looks as
>>>>>> if
>>>>>> it
>>>>>> is working as wanted, but...
>>>>>> 
>>>>>>> And that needs to be passed to the lfs scripts that are supposed to
>>>>>>> make
>>>>>>> the
>>>>>>> images.
>>>>>> 
>>>>>> ...this is a problem I'm struggling with right now:
>>>>>> How do I pass the 'XZ_MEM'-variable to 'cdrom' and 'Config'?
>>>>>> 
>>>>>> I think this can be done using 'export XZ_MEM' or something else but
>>>>>> I'm
>>>>>> not
>>>>>> sure.
>>>>>> 
>>>>>> This is the script I'm testing with:
>>>>>> 
>>>>>> ***SNIP***
>>>>>> #!/bin/bash
>>>>>> 
>>>>>> system_memory() {
>>>>>>    local key val unit
>>>>>> 
>>>>>>    while read -r key val unit; do
>>>>>>        case "${key}" in
>>>>>>            MemTotal:*)
>>>>>>                # Convert to MB
>>>>>>                echo "$(( ${val} / 1024 ))"
>>>>>>                break
>>>>>>                ;;
>>>>>>        esac
>>>>>>    done < /proc/meminfo
>>>>>> }
>>>>>> 
>>>>>> HOST_MEM=$(system_memory)
>>>>>> 
>>>>>> # Checking host memory
>>>>>> if [ $HOST_MEM -ge 1024 ]; then
>>>>>>    echo Host-Memory: $HOST_MEM'MiB'
>>>>>>    echo "Host memory is OK (must be at least 1024MiB), starting
>>>>>> build."
>>>>>> else
>>>>>>    echo Host-Memory: $HOST_MEM'MiB'
>>>>>>    echo "Host memory too low (less than 1024MiB), building
>>>>>> stopped."
>>>>>> exit 1
>>>>>> fi
>>>>>> 
>>>>>> # Host memory is ok, calculating needed XZ memory to about 70% of host
>>>>>> memory
>>>>>> ((XZ_MEM=(($HOST_MEM)/10)*7))
>>>>>>    XZ_MEM=$XZ_MEM'Mib'
>>>>>>    echo XZ-Memory: $XZ_MEM
>>>>>>    echo "XZ memory size is OK (must be at least 700MiB), starting
>>>>>> build."
>>>>>> ***SNAP***
>>>>>> 
>>>>>> The 'echoes' are mostly for debugging.
>>>>>> 
>>>>>> Calculation looks ok, 'MiB' must added without blanks (XZ man page:
>>>>>> "In
>>>>>> most
>>>>>> places where an integer argument is expected, an optional suffix is
>>>>>> supported
>>>>>> to easily indicate large integers. There must be no space between the
>>>>>> integer
>>>>>> and the suffix.")
>>>>>> 
>>>>>>> Please send questions where ever you need help. There is no reason
>>>>>>> to do
>>>>>>> this
>>>>>>> alone.
>>>>>> 
>>>>>> Please see above:
>>>>>> 
>>>>>> 1. Is this code what you where thinking of? I would leave the other
>>>>>> parameters
>>>>>> as they are (--threads=0 -8).
>>>>>> 
>>>>>> 2. Where should I add such code in the first place? I'd like to start
>>>>>> with
>>>>>> 'make.sh', so building stops as soon as possible if there is not
>>>>>> enough
>>>>>> memory. Where's
>>>>>> the best place? Right after function 'system_memory() {...}'?
>>>>>> 
>>>>>> 3. How to I pass contents of 'XZ_MEM' to 'cdrom' and 'Config'?
>>>>>> I consider something like:
>>>>>> ...
>>>>>> export XZ_OPT = --threads=0 -8 --memory=$XZ_MEM
>>>>>> ...
>>>>>> 
>>>>>> Best,
>>>>>> Matthias
>>>>>> 
>>>>>>> Best,
>>>>>>> -Michael
>>>>>>> 
>>>>>>>> 
>>>>>>>> Best,
>>>>>>>> Matthias
>>>>>>>>> Best,
>>>>>>>>> -Michael
>>>>>>>>> 
>>>>>>>>>> On Wed, 2017-11-22 at 16:37 +0000, Michael Tremer wrote:
>>>>>>>>>> Hi,
>>>>>>>>>> 
>>>>>>>>>>> On Tue, 2017-11-21 at 18:35 +0100, Matthias Fischer wrote:
>>>>>>>>>>> Hi,
>>>>>>>>>>> 
>>>>>>>>>>>> On 21.11.2017 15:42, Michael Tremer wrote:
>>>>>>>>>>>> Hello,
>>>>>>>>>>>> 
>>>>>>>>>>>> any progress on this?
>>>>>>>>>>> 
>>>>>>>>>>> Not much, because I've been struggling with a flu for the
>>>>>>>>>>> past
>>>>>>>>>>> two
>>>>>>>>>>> weeks, and its not over yet, but:
>>>>>>>>>>> 
>>>>>>>>>>> In the meantime I ran several builds with different
>>>>>>>>>>> parameters.
>>>>>>>>>>> 
>>>>>>>>>>> IMHO we could use:
>>>>>>>>>>> 
>>>>>>>>>>> 'export XZ_OPT = --threads=0 -8 --memory=700MiB'
>>>>>>>>>>> 
>>>>>>>>>>> in 'lfs/cdrom'
>>>>>>>>>>> 
>>>>>>>>>>> and
>>>>>>>>>>> 
>>>>>>>>>>> cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8 --
>>>>>>>>>>> memory=700MiB" tar
>>>>>>>>>>> -c
>>>>>>>>>>> -p
>>>>>>>>>>> --numeric-owner -J -f /install/packages/package/files.tar.xz
>>>>>>>>>>> *
>>>>>>>>>>> 
>>>>>>>>>>> in 'lfs/Config'.
>>>>>>>>>>> 
>>>>>>>>>>> Using '9' or '--extreme'-parameter brought no further
>>>>>>>>>>> improvements,
>>>>>>>>>>> on the other hand, build time and RAM consumption increased
>>>>>>>>>>> strongly. With '-8', minimal RAM consumption was about 680
>>>>>>>>>>> MiB,
>>>>>>>>>>> using
>>>>>>>>>>> '700 Mib' was always safe. So this is the minimum I would
>>>>>>>>>>> recommend.
>>>>>>>>>> 
>>>>>>>>>> Yes, I observed the same thing. -9 compresses a little bit
>>>>>>>>>> better
>>>>>>>>>> and
>>>>>>>>>> extreme
>>>>>>>>>> doesn't change a bit.
>>>>>>>>>> 
>>>>>>>>>> -8 seems to be optimal since it won't require too much memory
>>>>>>>>>> to
>>>>>>>>>> decompress
>>>>>>>>>> either. So let's go with that.
>>>>>>>>>> 
>>>>>>>>>>> Results:
>>>>>>>>>>> 
>>>>>>>>>>> (Buildtime: 5 hours 20 minutes 59 seconds)
>>>>>>>>>>> 
>>>>>>>>>>> 255866764  ipfire-2.19.1gb-ext4.i586-full-core117.img.gz
>>>>>>>>>>> 252262006  ipfire-2.19.1gb-ext4-scon.i586-full-
>>>>>>>>>>> core117.img.gz
>>>>>>>>>>> 175112192  ipfire-2.19.i586-full-core117.iso
>>>>>>>>>>> 
>>>>>>>>>>>> -Michael
>>>>>>>>>>>> 
>>>>>>>>>>>>> On Tue, 2017-11-07 at 22:47 +0000, Michael Tremer wrote:
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Tue, 2017-11-07 at 19:41 +0100, Matthias Fischer
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On 07.11.2017 16:05, Michael Tremer wrote:
>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> maybe this function helps:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> https://git.ipfire.org/?p=ipfire-2.x.git;a=commitdif
>>>>>>>>>>>>>>> f;h=
>>>>>>>>>>>>>>> 51
>>>>>>>>>>>>>>> 90eea2
>>>>>>>>>>>>>>> 4f98
>>>>>>>>>>>>>>> 22
>>>>>>>>>>>>>>> a63d
>>>>>>>>>>>>>>> c5d06d214b48f973b14f29
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks. I'll take a look...
>>>>>>>>>>> 
>>>>>>>>>>> This would help, indeed, but how to implement(!?), and...
>>>>>>>>>>> (see
>>>>>>>>>>> below)
>>>>>>>>>> 
>>>>>>>>>> Just run it somewhere in the script, figure out what is about
>>>>>>>>>> 80%
>>>>>>>>>> of
>>>>>>>>>> that
>>>>>>>>>> and
>>>>>>>>>> if
>>>>>>>>>> that is smaller than 800 MB, we should just fail.
>>>>>>>>>> 
>>>>>>>>>> It would be good to have a function that determines the xz
>>>>>>>>>> parameters and
>>>>>>>>>> we
>>>>>>>>>> use
>>>>>>>>>> that where ever we call xz. We don't want any duplicated code.
>>>>>>>>>> 
>>>>>>>>>>>>>>> On Tue, 2017-11-07 at 10:46 +0000, Michael Tremer
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> On Mon, 2017-11-06 at 22:09 +0100, Matthias
>>>>>>>>>>>>>>>> Fischer
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> I forgot: Tuning the XZ-parameters... ;-)
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Good that you are bringing this up. So I think
>>>>>>>>>>>>>>>> what we
>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>> do
>>>>>>>>>>>>>>>> here is
>>>>>>>>>>>>>>>> bringing back XZ compression with -8 so that we do
>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>> waste
>>>>>>>>>>>>>>>> too
>>>>>>>>>>>>>>>> much
>>>>>>>>>>>>>>>> space
>>>>>>>>>>>>>>>> when
>>>>>>>>>>>>>>>> extracting the image again.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Right now I'm testing how much RAM I need.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Current 'next'-build is running with:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 'lfs/cdrom':
>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>> export XZ_OPT = --threads=0 -8 --memory=500MiB
>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 'lfs/Config':
>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>> cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8"
>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I'm waiting for the results. Takes a while.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> You can just run it like this and it will tell you:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> [ms(a)hughes ~]$ xz -vv8 -T0
>>>>>>>>>>>>> xz: Filter chain: --
>>>>>>>>>>>>> lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=b
>>>>>>>>>>>>> t4,d
>>>>>>>>>>>>> ep
>>>>>>>>>>>>> th=0
>>>>>>>>>>>>> xz: Using up to 4 threads.
>>>>>>>>>>>>> xz: 2,629 MiB of memory is required. The limiter is
>>>>>>>>>>>>> disabled.
>>>>>>>>>>>>> xz: Decompression will need 33 MiB of memory.
>>>>>>>>>>>>> xz: Compressed data cannot be written to a terminal
>>>>>>>>>>>>> xz: Try `xz --help' for more information.
>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> To not run into memory limits we should detect how
>>>>>>>>>>>>>>>> much
>>>>>>>>>>>>>>>> memory
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> build
>>>>>>>>>>>>>>>> host
>>>>>>>>>>>>>>>> has and set the memory limit to maybe 80% of that.
>>>>>>>>>>>>>>>> xz
>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>> automatically
>>>>>>>>>>>>>>>> scale down to a number of parallel processes that
>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>> fit
>>>>>>>>>>>>>>>> into
>>>>>>>>>>>>>>>> memory.
>>>>>>>>>>> 
>>>>>>>>>>> This is something I couldn't get to work. First, I couldn't
>>>>>>>>>>> specify not
>>>>>>>>>>> less
>>>>>>>>>>> than '700 MiB'. If I specify '600 MiB', building stops and
>>>>>>>>>>> tells
>>>>>>>>>>> me it
>>>>>>>>>>> needs
>>>>>>>>>>> at least about 680 MiB. If I use '70%', building stops. And
>>>>>>>>>>> I
>>>>>>>>>>> never saw
>>>>>>>>>>> that
>>>>>>>>>>> 'xz' "automatically scaled down". It just stopped everytime.
>>>>>>>>>> 
>>>>>>>>>> Yes, ~700M is the minimum.
>>>>>>>>>> 
>>>>>>>>>> It will just reduce the number of threads again to make sure
>>>>>>>>>> everything
>>>>>>>>>> fits
>>>>>>>>>> into memory.
>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>>>> I'm a bit puzzled about this: I tried using '
>>>>>>>>>>>>>> --memory=70%'
>>>>>>>>>>>>>> as
>>>>>>>>>>>>>> mentioned
>>>>>>>>>>>>>> in the xz man pages, but is seems to have no effect.
>>>>>>>>>>>>>> It
>>>>>>>>>>>>>> looks as
>>>>>>>>>>>>>> if
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> percent parameter isn't recognized and ignored. It
>>>>>>>>>>>>>> always
>>>>>>>>>>>>>> ends up
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>> "cannot allocate memory". Any hints on this? Do both
>>>>>>>>>>>>>> 'lfs/cdrom'
>>>>>>>>>>>>>> AND
>>>>>>>>>>>>>> 'lfs/Config' need the '--memory'-parameter?
>>>>>>>>>>> 
>>>>>>>>>>> This was the next problem: 'percent'-parameter does not work
>>>>>>>>>>> as
>>>>>>>>>>> expected.
>>>>>>>>>> 
>>>>>>>>>> No, don't use it. Just determine yourself what it is.
>>>>>>>>>> 
>>>>>>>>>> I think we could even try to set 100% of the memory, because
>>>>>>>>>> as
>>>>>>>>>> far
>>>>>>>>>> as I
>>>>>>>>>> can
>>>>>>>>>> see
>>>>>>>>>> this is worst case and xz won't use everything that we allow.
>>>>>>>>>> 
>>>>>>>>>> If that does't work, let's scale down to 90%, or 80%, etc...
>>>>>>>>>> 
>>>>>>>>>>>>> Yes, but it would be good to have one place where the
>>>>>>>>>>>>> parameters are
>>>>>>>>>>>>> being generated and then we just use them from a
>>>>>>>>>>>>> variable.
>>>>>>>>>>> 
>>>>>>>>>>> Probably beyond my capabilities for the moment. Any
>>>>>>>>>>> thoughts?
>>>>>>>>>>> 
>>>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>>>> We will at least need about half a GB which should
>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>> sensible
>>>>>>>>>>>>>>>> requirement
>>>>>>>>>>>>>>>> for a build system any ways.
>>>>>>>>>>> 
>>>>>>>>>>> I'm afraid '500 MiB' are not enough...
>>>>>>>>>> 
>>>>>>>>>> No, so we will now make 1G the minimum for a build.
>>>>>>>>>> 
>>>>>>>>>> I tried to build the OS on an ARM board with only 512MB of RAM
>>>>>>>>>> the
>>>>>>>>>> other
>>>>>>>>>> day
>>>>>>>>>> and
>>>>>>>>>> even with a lot of swap it doesn't work. So we are past this
>>>>>>>>>> any
>>>>>>>>>> way.
>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>>>> Agreed. Because of this I'm testing with '500Mib'.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Will you work on this bringing it into the build
>>>>>>>>>>>>>>>> system?!
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> "I'll do my very best!" :-))
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Very well Miss Sophie...
>>>>>>>>>>>>> ...
>>>>>>>>>>> 
>>>>>>>>>>> Best™,
>>>>>>>>>>> Matthias
>>>>>>>>>> 
>>>>>>>>>> -Michael

Re: Compress images with XZ

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 24645 bytes --]

Hey,

technically that is an idea, but I don't think it makes sense to do this.

People would need the firmware if they are connecting to the Internet using WiFi
and then it gets really complicated to figure out what people are most likely to
use. Same goes for some Ethernet adapters.

We don't really have a limit about the image size. But of course a smaller image
saves disk space and bandwidth to download. It's just a bit about efficiency :)

Best,
-Michael

On Fri, 2018-05-11 at 20:35 -0400, Tom Rymes wrote:
> Would it make sense to make the firmware files a separate download, as only
> those who use wireless Red or hostAPD need them?
> 
> Tom
> 
> > On May 11, 2018, at 4:28 PM, Michael Tremer <michael.tremer(a)ipfire.org>
> > wrote:
> > 
> > Hey Matthias,
> > 
> > are you still working on this?
> > 
> > I am asking because I just noticed that the new next build is about 50MB
> > larger
> > than the old one due to the larger kernel and loads more of firmware files.
> > 
> > Compressing it better would of course help to save space on disk.
> > 
> > Best,
> > -Michael
> > 
> > > On Sun, 2018-02-11 at 19:57 +0000, Michael Tremer wrote:
> > > Hi,
> > > 
> > > > On Sun, 2018-02-11 at 15:35 +0000, Matthias Fischer wrote:
> > > > Hi,
> > > > 
> > > > On 06.02.2018 02:05, Michael Tremer wrote:
> > > > > > ...
> > > > > > ((XZ_MEM=(($HOST_MEM)/10)*7)) && XZ_MEM=${XZ_MEM}MiB
> > > > > 
> > > > > You can write this even short:
> > > > > 
> > > > > XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"
> > > > 
> > > > Oneliner! Yes. I was looking for something like that... ;-)
> > > > 
> > > > > I swapped the order of the multiplication and division so that you
> > > > > would
> > > > > have
> > > > > a
> > > > > more exact result.
> > > > > 
> > > > > For example if you system has 1024 MB or RAM, you divide this by 10
> > > > > (102)
> > > > > and
> > > > > then multiply by 7 which gives you 714. The other order is 1024 * 7 =
> > > > > 7167
> > > > > and
> > > > > then divided by 10 is 716 which is closer to the actual result of
> > > > > 716.8.
> > > > > Shouldn't matter here but just in general this is better :)
> > > > 
> > > > Thanks, looks better - noted for the next time!
> > > > 
> > > > Now it looks like this ('make.sh'):
> > > > 
> > > > ...
> > > > # Host memory is ok, calculating XZ memory
> > > >    XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"
> > > >    XZ_OPT="--threads=0 -8 $XZ_MEM"
> > > >    echo XZ-Memory: $XZ_MEM
> > > >    echo XZ-Options: $XZ_OPT
> > > >    echo "XZ memory size is OK (must be at least 700MiB), starting
> > > > build..."
> > > > ...
> > > > ["echoes" for debugging!]
> > > > 
> > > > This is working. '$XZ_OPT' now gives me: --threads=0 -8 5450MiB
> > > 
> > > Will there be some checks in make.sh now that check if at least 1024MB
> > > of memory are available for build? I think there should be a warning if
> > > not enough memory is available, but we should still try to build.
> > > 
> > > > But. ;-)
> > > > 
> > > > > You can put this probably next under "HOST_MEM=$(system_memory)" and
> > > > > then
> > > > > export
> > > > > this to the internal shell by adding the variable to lfsmake2 for the
> > > > > disk
> > > > > image
> > > > > and ipfiredist for the packages.
> > > > 
> > > > Sorry, I must confess that I don't know exactly how to do this. As I
> > > > wrote:
> > > > "How
> > > > do
> > > > I pass the 'XZ_MEM'-variable to 'cdrom' and 'Config'?" Or in this case,
> > > > XZ_OPT!?
> > > > 
> > > > > Then in lfs/cdrom and lfs/Config for the dist
> > > > > target, you just need to add the parameters to the command line.
> > > > 
> > > > In 'Config' I changed:
> > > > 
> > > > ...
> > > > cd /install/packages/package/tmp/ && XZ_OPT=-T0 tar -c -p --numeric-
> > > > owner -J
> > > > -f
> > > > /install/packages/package/files.tar.xz *
> > > > ...
> > > > 
> > > > To:
> > > > ...
> > > > cd /install/packages/package/tmp/ && $XZ_OPT tar -c -p --numeric-owner
> > > > -J -f
> > > > /install/packages/package/files.tar.xz *
> > > > ...
> > > > 
> > > > > Maybe it is a good idea to have a variable XZ_OPT that adds everything
> > > > > together
> > > > > and is then used in both places.
> > > > 
> > > > I added the other xz-options and created XZ_OPT as shown above, but in
> > > > 'cdrom'
> > > > there
> > > > is an 'export XZ_OPT = ...'-line and I don't know how to handle these
> > > > line.
> > > > Is
> > > > it still necessary?
> > > 
> > > No, that is no longer necessary and can be removed. We should just add
> > > it to the tar command just like it is done in lfs/Config.
> > > 
> > > > Do I have to 'export' the XZ_OPT'-variable in 'make.sh'?
> > > 
> > > No, because the chroot environment will throw away all environment
> > > variables from the host system and then set them again. That's why we
> > > have this enterchroot() function that does this with the env command.
> > > 
> > > > And how do I "add" this variable to 'lfsmake2' and 'ipfiredist'(-
> > > > function)?
> > > 
> > > Just have a look at the functions. There should be loads of examples
> > > where we pass other variables and you just need to add this one, too.
> > > 
> > > > Sorry if these questions sound simple, but I'm guessing a bit too much
> > > > while
> > > > trying
> > > > to get a grip on this.
> > > 
> > > This is absolutely fine to sort this out first and then send the patch
> > > :) That's what this list is for.
> > > 
> > > Best,
> > > -Michael
> > > 
> > > > 
> > > > Best,
> > > > Matthias
> > > > 
> > > > > 
> > > > > Best,
> > > > > -Michael
> > > > > 
> > > > > > ...
> > > > > > 
> > > > > > Looks better.
> > > > > > 
> > > > > > Best,
> > > > > > Matthias
> > > > > > 
> > > > > > > On 03.02.2018 20:44, Matthias Fischer wrote:
> > > > > > > Hi,
> > > > > > > 
> > > > > > > > On 30.01.2018 21:03, Michael Tremer wrote:
> > > > > > > > Hi,
> > > > > > > > 
> > > > > > > > ...
> > > > > > > > > 
> > > > > > > > > I'll consider how I would do this and send it in as soon as I
> > > > > > > > > get
> > > > > > > > > a
> > > > > > > > > grip
> > > > > > > > > on it.
> > > > > > > > 
> > > > > > > > Well, I added some functions already somewhere. I suppose you
> > > > > > > > just
> > > > > > > > need
> > > > > > > > to
> > > > > > > > do
> > > > > > > > the maths to set that to X% of the host system which is just a
> > > > > > > > simple
> > > > > > > > multiplication of the output of ${HOST_MEM} in make.sh.
> > > > > > > 
> > > > > > > Ok, I take my heart in my hand. Here are my first results - but
> > > > > > > remember,
> > > > > > > I'm not familiar with this. I built a short test script which
> > > > > > > looks as
> > > > > > > if
> > > > > > > it
> > > > > > > is working as wanted, but...
> > > > > > > 
> > > > > > > > And that needs to be passed to the lfs scripts that are supposed
> > > > > > > > to
> > > > > > > > make
> > > > > > > > the
> > > > > > > > images.
> > > > > > > 
> > > > > > > ...this is a problem I'm struggling with right now:
> > > > > > > How do I pass the 'XZ_MEM'-variable to 'cdrom' and 'Config'?
> > > > > > > 
> > > > > > > I think this can be done using 'export XZ_MEM' or something else
> > > > > > > but
> > > > > > > I'm
> > > > > > > not
> > > > > > > sure.
> > > > > > > 
> > > > > > > This is the script I'm testing with:
> > > > > > > 
> > > > > > > ***SNIP***
> > > > > > > #!/bin/bash
> > > > > > > 
> > > > > > > system_memory() {
> > > > > > >    local key val unit
> > > > > > > 
> > > > > > >    while read -r key val unit; do
> > > > > > >        case "${key}" in
> > > > > > >            MemTotal:*)
> > > > > > >                # Convert to MB
> > > > > > >                echo "$(( ${val} / 1024 ))"
> > > > > > >                break
> > > > > > >                ;;
> > > > > > >        esac
> > > > > > >    done < /proc/meminfo
> > > > > > > }
> > > > > > > 
> > > > > > > HOST_MEM=$(system_memory)
> > > > > > > 
> > > > > > > # Checking host memory
> > > > > > > if [ $HOST_MEM -ge 1024 ]; then
> > > > > > >    echo Host-Memory: $HOST_MEM'MiB'
> > > > > > >    echo "Host memory is OK (must be at least 1024MiB), starting
> > > > > > > build."
> > > > > > > else
> > > > > > >    echo Host-Memory: $HOST_MEM'MiB'
> > > > > > >    echo "Host memory too low (less than 1024MiB), building
> > > > > > > stopped."
> > > > > > > exit 1
> > > > > > > fi
> > > > > > > 
> > > > > > > # Host memory is ok, calculating needed XZ memory to about 70% of
> > > > > > > host
> > > > > > > memory
> > > > > > > ((XZ_MEM=(($HOST_MEM)/10)*7))
> > > > > > >    XZ_MEM=$XZ_MEM'Mib'
> > > > > > >    echo XZ-Memory: $XZ_MEM
> > > > > > >    echo "XZ memory size is OK (must be at least 700MiB), starting
> > > > > > > build."
> > > > > > > ***SNAP***
> > > > > > > 
> > > > > > > The 'echoes' are mostly for debugging.
> > > > > > > 
> > > > > > > Calculation looks ok, 'MiB' must added without blanks (XZ man
> > > > > > > page:
> > > > > > > "In
> > > > > > > most
> > > > > > > places where an integer argument is expected, an optional suffix
> > > > > > > is
> > > > > > > supported
> > > > > > > to easily indicate large integers. There must be no space between
> > > > > > > the
> > > > > > > integer
> > > > > > > and the suffix.")
> > > > > > > 
> > > > > > > > Please send questions where ever you need help. There is no
> > > > > > > > reason
> > > > > > > > to do
> > > > > > > > this
> > > > > > > > alone.
> > > > > > > 
> > > > > > > Please see above:
> > > > > > > 
> > > > > > > 1. Is this code what you where thinking of? I would leave the
> > > > > > > other
> > > > > > > parameters
> > > > > > > as they are (--threads=0 -8).
> > > > > > > 
> > > > > > > 2. Where should I add such code in the first place? I'd like to
> > > > > > > start
> > > > > > > with
> > > > > > > 'make.sh', so building stops as soon as possible if there is not
> > > > > > > enough
> > > > > > > memory. Where's
> > > > > > > the best place? Right after function 'system_memory() {...}'?
> > > > > > > 
> > > > > > > 3. How to I pass contents of 'XZ_MEM' to 'cdrom' and 'Config'?
> > > > > > > I consider something like:
> > > > > > > ...
> > > > > > > export XZ_OPT = --threads=0 -8 --memory=$XZ_MEM
> > > > > > > ...
> > > > > > > 
> > > > > > > Best,
> > > > > > > Matthias
> > > > > > > 
> > > > > > > > Best,
> > > > > > > > -Michael
> > > > > > > > 
> > > > > > > > > 
> > > > > > > > > Best,
> > > > > > > > > Matthias
> > > > > > > > > > Best,
> > > > > > > > > > -Michael
> > > > > > > > > > 
> > > > > > > > > > > On Wed, 2017-11-22 at 16:37 +0000, Michael Tremer wrote:
> > > > > > > > > > > Hi,
> > > > > > > > > > > 
> > > > > > > > > > > > On Tue, 2017-11-21 at 18:35 +0100, Matthias Fischer
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > Hi,
> > > > > > > > > > > > 
> > > > > > > > > > > > > On 21.11.2017 15:42, Michael Tremer wrote:
> > > > > > > > > > > > > Hello,
> > > > > > > > > > > > > 
> > > > > > > > > > > > > any progress on this?
> > > > > > > > > > > > 
> > > > > > > > > > > > Not much, because I've been struggling with a flu for
> > > > > > > > > > > > the
> > > > > > > > > > > > past
> > > > > > > > > > > > two
> > > > > > > > > > > > weeks, and its not over yet, but:
> > > > > > > > > > > > 
> > > > > > > > > > > > In the meantime I ran several builds with different
> > > > > > > > > > > > parameters.
> > > > > > > > > > > > 
> > > > > > > > > > > > IMHO we could use:
> > > > > > > > > > > > 
> > > > > > > > > > > > 'export XZ_OPT = --threads=0 -8 --memory=700MiB'
> > > > > > > > > > > > 
> > > > > > > > > > > > in 'lfs/cdrom'
> > > > > > > > > > > > 
> > > > > > > > > > > > and
> > > > > > > > > > > > 
> > > > > > > > > > > > cd /install/packages/package/tmp/ && XZ_OPT="-T0 -8 --
> > > > > > > > > > > > memory=700MiB" tar
> > > > > > > > > > > > -c
> > > > > > > > > > > > -p
> > > > > > > > > > > > --numeric-owner -J -f
> > > > > > > > > > > > /install/packages/package/files.tar.xz
> > > > > > > > > > > > *
> > > > > > > > > > > > 
> > > > > > > > > > > > in 'lfs/Config'.
> > > > > > > > > > > > 
> > > > > > > > > > > > Using '9' or '--extreme'-parameter brought no further
> > > > > > > > > > > > improvements,
> > > > > > > > > > > > on the other hand, build time and RAM consumption
> > > > > > > > > > > > increased
> > > > > > > > > > > > strongly. With '-8', minimal RAM consumption was about
> > > > > > > > > > > > 680
> > > > > > > > > > > > MiB,
> > > > > > > > > > > > using
> > > > > > > > > > > > '700 Mib' was always safe. So this is the minimum I
> > > > > > > > > > > > would
> > > > > > > > > > > > recommend.
> > > > > > > > > > > 
> > > > > > > > > > > Yes, I observed the same thing. -9 compresses a little bit
> > > > > > > > > > > better
> > > > > > > > > > > and
> > > > > > > > > > > extreme
> > > > > > > > > > > doesn't change a bit.
> > > > > > > > > > > 
> > > > > > > > > > > -8 seems to be optimal since it won't require too much
> > > > > > > > > > > memory
> > > > > > > > > > > to
> > > > > > > > > > > decompress
> > > > > > > > > > > either. So let's go with that.
> > > > > > > > > > > 
> > > > > > > > > > > > Results:
> > > > > > > > > > > > 
> > > > > > > > > > > > (Buildtime: 5 hours 20 minutes 59 seconds)
> > > > > > > > > > > > 
> > > > > > > > > > > > 255866764  ipfire-2.19.1gb-ext4.i586-full-core117.img.gz
> > > > > > > > > > > > 252262006  ipfire-2.19.1gb-ext4-scon.i586-full-
> > > > > > > > > > > > core117.img.gz
> > > > > > > > > > > > 175112192  ipfire-2.19.i586-full-core117.iso
> > > > > > > > > > > > 
> > > > > > > > > > > > > -Michael
> > > > > > > > > > > > > 
> > > > > > > > > > > > > > On Tue, 2017-11-07 at 22:47 +0000, Michael Tremer
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > On Tue, 2017-11-07 at 19:41 +0100, Matthias Fischer
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > On 07.11.2017 16:05, Michael Tremer wrote:
> > > > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > maybe this function helps:
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > https://git.ipfire.org/?p=ipfire-2.x.git;a=commi
> > > > > > > > > > > > > > > > tdif
> > > > > > > > > > > > > > > > f;h=
> > > > > > > > > > > > > > > > 51
> > > > > > > > > > > > > > > > 90eea2
> > > > > > > > > > > > > > > > 4f98
> > > > > > > > > > > > > > > > 22
> > > > > > > > > > > > > > > > a63d
> > > > > > > > > > > > > > > > c5d06d214b48f973b14f29
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > Thanks. I'll take a look...
> > > > > > > > > > > > 
> > > > > > > > > > > > This would help, indeed, but how to implement(!?),
> > > > > > > > > > > > and...
> > > > > > > > > > > > (see
> > > > > > > > > > > > below)
> > > > > > > > > > > 
> > > > > > > > > > > Just run it somewhere in the script, figure out what is
> > > > > > > > > > > about
> > > > > > > > > > > 80%
> > > > > > > > > > > of
> > > > > > > > > > > that
> > > > > > > > > > > and
> > > > > > > > > > > if
> > > > > > > > > > > that is smaller than 800 MB, we should just fail.
> > > > > > > > > > > 
> > > > > > > > > > > It would be good to have a function that determines the xz
> > > > > > > > > > > parameters and
> > > > > > > > > > > we
> > > > > > > > > > > use
> > > > > > > > > > > that where ever we call xz. We don't want any duplicated
> > > > > > > > > > > code.
> > > > > > > > > > > 
> > > > > > > > > > > > > > > > On Tue, 2017-11-07 at 10:46 +0000, Michael
> > > > > > > > > > > > > > > > Tremer
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > On Mon, 2017-11-06 at 22:09 +0100, Matthias
> > > > > > > > > > > > > > > > > Fischer
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > ...
> > > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > > I forgot: Tuning the XZ-parameters... ;-)
> > > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > Good that you are bringing this up. So I think
> > > > > > > > > > > > > > > > > what we
> > > > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > do
> > > > > > > > > > > > > > > > > here is
> > > > > > > > > > > > > > > > > bringing back XZ compression with -8 so that
> > > > > > > > > > > > > > > > > we do
> > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > waste
> > > > > > > > > > > > > > > > > too
> > > > > > > > > > > > > > > > > much
> > > > > > > > > > > > > > > > > space
> > > > > > > > > > > > > > > > > when
> > > > > > > > > > > > > > > > > extracting the image again.
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > Right now I'm testing how much RAM I need.
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > Current 'next'-build is running with:
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > 'lfs/cdrom':
> > > > > > > > > > > > > > > ...
> > > > > > > > > > > > > > > export XZ_OPT = --threads=0 -8 --memory=500MiB
> > > > > > > > > > > > > > > ...
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > 'lfs/Config':
> > > > > > > > > > > > > > > ...
> > > > > > > > > > > > > > > cd /install/packages/package/tmp/ && XZ_OPT="-T0
> > > > > > > > > > > > > > > -8"
> > > > > > > > > > > > > > > ...
> > > > > > > > > > > > > > > ...
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > I'm waiting for the results. Takes a while.
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > You can just run it like this and it will tell you:
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > [ms(a)hughes ~]$ xz -vv8 -T0
> > > > > > > > > > > > > > xz: Filter chain: --
> > > > > > > > > > > > > > lzma2=dict=32MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,
> > > > > > > > > > > > > > mf=b
> > > > > > > > > > > > > > t4,d
> > > > > > > > > > > > > > ep
> > > > > > > > > > > > > > th=0
> > > > > > > > > > > > > > xz: Using up to 4 threads.
> > > > > > > > > > > > > > xz: 2,629 MiB of memory is required. The limiter is
> > > > > > > > > > > > > > disabled.
> > > > > > > > > > > > > > xz: Decompression will need 33 MiB of memory.
> > > > > > > > > > > > > > xz: Compressed data cannot be written to a terminal
> > > > > > > > > > > > > > xz: Try `xz --help' for more information.
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > To not run into memory limits we should detect
> > > > > > > > > > > > > > > > > how
> > > > > > > > > > > > > > > > > much
> > > > > > > > > > > > > > > > > memory
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > build
> > > > > > > > > > > > > > > > > host
> > > > > > > > > > > > > > > > > has and set the memory limit to maybe 80% of
> > > > > > > > > > > > > > > > > that.
> > > > > > > > > > > > > > > > > xz
> > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > then
> > > > > > > > > > > > > > > > > automatically
> > > > > > > > > > > > > > > > > scale down to a number of parallel processes
> > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > fit
> > > > > > > > > > > > > > > > > into
> > > > > > > > > > > > > > > > > memory.
> > > > > > > > > > > > 
> > > > > > > > > > > > This is something I couldn't get to work. First, I
> > > > > > > > > > > > couldn't
> > > > > > > > > > > > specify not
> > > > > > > > > > > > less
> > > > > > > > > > > > than '700 MiB'. If I specify '600 MiB', building stops
> > > > > > > > > > > > and
> > > > > > > > > > > > tells
> > > > > > > > > > > > me it
> > > > > > > > > > > > needs
> > > > > > > > > > > > at least about 680 MiB. If I use '70%', building stops.
> > > > > > > > > > > > And
> > > > > > > > > > > > I
> > > > > > > > > > > > never saw
> > > > > > > > > > > > that
> > > > > > > > > > > > 'xz' "automatically scaled down". It just stopped
> > > > > > > > > > > > everytime.
> > > > > > > > > > > 
> > > > > > > > > > > Yes, ~700M is the minimum.
> > > > > > > > > > > 
> > > > > > > > > > > It will just reduce the number of threads again to make
> > > > > > > > > > > sure
> > > > > > > > > > > everything
> > > > > > > > > > > fits
> > > > > > > > > > > into memory.
> > > > > > > > > > > 
> > > > > > > > > > > > 
> > > > > > > > > > > > > > > I'm a bit puzzled about this: I tried using '
> > > > > > > > > > > > > > > --memory=70%'
> > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > mentioned
> > > > > > > > > > > > > > > in the xz man pages, but is seems to have no
> > > > > > > > > > > > > > > effect.
> > > > > > > > > > > > > > > It
> > > > > > > > > > > > > > > looks as
> > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > percent parameter isn't recognized and ignored. It
> > > > > > > > > > > > > > > always
> > > > > > > > > > > > > > > ends up
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > "cannot allocate memory". Any hints on this? Do
> > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > 'lfs/cdrom'
> > > > > > > > > > > > > > > AND
> > > > > > > > > > > > > > > 'lfs/Config' need the '--memory'-parameter?
> > > > > > > > > > > > 
> > > > > > > > > > > > This was the next problem: 'percent'-parameter does not
> > > > > > > > > > > > work
> > > > > > > > > > > > as
> > > > > > > > > > > > expected.
> > > > > > > > > > > 
> > > > > > > > > > > No, don't use it. Just determine yourself what it is.
> > > > > > > > > > > 
> > > > > > > > > > > I think we could even try to set 100% of the memory,
> > > > > > > > > > > because
> > > > > > > > > > > as
> > > > > > > > > > > far
> > > > > > > > > > > as I
> > > > > > > > > > > can
> > > > > > > > > > > see
> > > > > > > > > > > this is worst case and xz won't use everything that we
> > > > > > > > > > > allow.
> > > > > > > > > > > 
> > > > > > > > > > > If that does't work, let's scale down to 90%, or 80%,
> > > > > > > > > > > etc...
> > > > > > > > > > > 
> > > > > > > > > > > > > > Yes, but it would be good to have one place where
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > parameters are
> > > > > > > > > > > > > > being generated and then we just use them from a
> > > > > > > > > > > > > > variable.
> > > > > > > > > > > > 
> > > > > > > > > > > > Probably beyond my capabilities for the moment. Any
> > > > > > > > > > > > thoughts?
> > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > ...
> > > > > > > > > > > > > > > > > We will at least need about half a GB which
> > > > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > sensible
> > > > > > > > > > > > > > > > > requirement
> > > > > > > > > > > > > > > > > for a build system any ways.
> > > > > > > > > > > > 
> > > > > > > > > > > > I'm afraid '500 MiB' are not enough...
> > > > > > > > > > > 
> > > > > > > > > > > No, so we will now make 1G the minimum for a build.
> > > > > > > > > > > 
> > > > > > > > > > > I tried to build the OS on an ARM board with only 512MB of
> > > > > > > > > > > RAM
> > > > > > > > > > > the
> > > > > > > > > > > other
> > > > > > > > > > > day
> > > > > > > > > > > and
> > > > > > > > > > > even with a lot of swap it doesn't work. So we are past
> > > > > > > > > > > this
> > > > > > > > > > > any
> > > > > > > > > > > way.
> > > > > > > > > > > 
> > > > > > > > > > > > 
> > > > > > > > > > > > > > > Agreed. Because of this I'm testing with '500Mib'.
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > > Will you work on this bringing it into the
> > > > > > > > > > > > > > > > > build
> > > > > > > > > > > > > > > > > system?!
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > "I'll do my very best!" :-))
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > Very well Miss Sophie...
> > > > > > > > > > > > > > ...
> > > > > > > > > > > > 
> > > > > > > > > > > > Best™,
> > > > > > > > > > > > Matthias
> > > > > > > > > > > 
> > > > > > > > > > > -Michael

Re: Compress images with XZ

[Matthias Fischer]

[-- Attachment #1: Type: text/plain, Size: 6894 bytes --]

HI,

On 11.05.2018 22:27, Michael Tremer wrote:
> Hey Matthias,
> 
> are you still working on this?

Its still on my list, but I must confess: I didn't get a grip on this yet.
 
> I am asking because I just noticed that the new next build is about 50MB larger
> than the old one due to the larger kernel and loads more of firmware files.

Yes, I saw this too. I was waiting for you to ask... ;-)

I'll try to explain what I did and where I'm stuck:

> ...
>> > 
>> > On 06.02.2018 02:05, Michael Tremer wrote:
>> > > > ...
>> > > > ((XZ_MEM=(($HOST_MEM)/10)*7)) && XZ_MEM=${XZ_MEM}MiB
>> > > 
>> > > You can write this even short:
>> > > 
>> > > XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"
>> > 
>> > Oneliner! Yes. I was looking for something like that... ;-)

This did work, no problem here.

>> > ...
>> > # Host memory is ok, calculating XZ memory
>> > 	XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"
>> > 	XZ_OPT="--threads=0 -8 $XZ_MEM"
>> > 	echo XZ-Memory: $XZ_MEM
>> > 	echo XZ-Options: $XZ_OPT
>> > 	echo "XZ memory size is OK (must be at least 700MiB), starting build..."
>> > ...
>> > ["echoes" for debugging!]
>> > 
>> > This is working. '$XZ_OPT' now gives me: --threads=0 -8 5450MiB
>> 
>> Will there be some checks in make.sh now that check if at least 1024MB
>> of memory are available for build?
>> ...

For this, I changed the above ("read -p..."-lines are just for debugging):

***SNIP***
...
# Get the amount of memory in this build system
HOST_MEM=$(system_memory)

# XZ-TUNING BEGIN
# Checking host memory

if [ $HOST_MEM -ge 1024 ]; then
	print_build_stage "Host-Memory: $HOST_MEM MiB"
	print_build_stage "Host memory is OK (must be at least 1024 MiB), calculating XZ memory..."

# Host memory is ok, calculating XZ memory
echo

	XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"
	XZ_OPT="--threads=0 -8 $XZ_MEM"
	print_build_stage "XZ-Memory:" $XZ_MEM
	print_build_stage "XZ-Options:" $XZ_OPT
	print_build_stage "XZ memory size is OK (must be at least 700 MiB), moving on..."

echo
read -p "Press [Enter] key to continue..."

else

	print_build_stage "Host-Memory: $HOST_MEM MiB"
	print_build_stage "Not enough host memory (less than 1024 MiB, consider upgrading),"
	print_build_stage "building will use standard XZ options."
	XZ_OPT="-T0" # alte Standardparameter einsetzen

echo
read -p "Press [Enter] key to continue..."

fi
...
# XZ-TUNING END

***SNAP***

>> ...
>>  I think there should be a warning if
>>  not enough memory is available, but we should still try to build.

As far as I can see, this could be done by replacing the above 'echoes'
with 'print_build_stage'(?).

E.g.:
print_build_stage "Not enough host memory (less than 1024MiB, consider upgrading),"
print_build_stage "building will use standard XZ options."

>> ...
>> > But. ;-)
>> > 
>> > > You can put this probably next under "HOST_MEM=$(system_memory)" and then
>> > > export this to the internal shell by adding the variable to lfsmake2 for
>> > > the disk image and ipfiredist for the packages.
>> > 
>> > Sorry, I must confess that I don't know exactly how to do this. As I wrote:
>> > "How do I pass the 'XZ_MEM'-variable to 'cdrom' and 'Config'?" Or in this case,
>> > XZ_OPT!?

And this is still my problem! I don't get this to work. How do I "pass" this variable?

>> > > Then in lfs/cdrom and lfs/Config for the dist
>> > > target, you just need to add the parameters to the command line.
>> > 
>> > In 'Config' I changed:
>> > 
>> > ...
>> > cd /install/packages/package/tmp/ && XZ_OPT=-T0 tar -c -p --numeric-owner -J
>> > -f
>> > /install/packages/package/files.tar.xz *
>> > ...
>> > 
>> > To:
>> > ...
>> > cd /install/packages/package/tmp/ && $XZ_OPT tar -c -p --numeric-owner -J -f
>> > /install/packages/package/files.tar.xz *
>> > ...
>> > 
>> > > Maybe it is a good idea to have a variable XZ_OPT that adds everything
>> > > together and is then used in both places.

I tried that with no luck. I added the variable '$XZ_OPT' to 'lfs/Config' in line
212:

cd /install/packages/package/tmp/ && $XZ_OPT tar -c -p --numeric-owner...
 
But this usually ends - like today - in '_build_packages.log' with:

***SNIP***
...
#Change xxxKVERxxx to Kernelversion
sed -e "s/xxxKVERxxx/4.14.40/g" -i /install/packages/package/update.sh
cd /install/packages/package && tar cf ../core-upgrade-2.19-$(basename core/121).ipfire \
	update.sh files.tar.xz ROOTFILES
rm -rf /install/packages/package
sed -e "s/NAME/core-upgrade/g" \
	-e "s/VER/2.19/g" \
	-e "s/RELEASE/$(basename core/121)/g" \
	-e "s/DEPS//g" \
	-e "s/SIZE/`ls -l \/install\/packages\/core-upgrade-2.19-$(basename core/121).ipfire | awk '{ print $5 }'`/g" \
	< /usr/src/src/pakfire/meta > /install/packages/meta-core-upgrade-$(basename core/121)
May 12 13:39:49: Building directfb DirectFB-1.7.7.tar.gz checksum OK
+ cd /usr/src/lfs
+ make -f directfb LFS_BASEDIR=/usr/src dist
'/usr/src/config/rootfiles/packages/i586/directfb' -> '/install/packages/package/ROOTFILES'
/bin/sh: Z_OPT: command not found
make: *** [directfb:60: dist] Error 127
...
***SNAP***

Why "Z_OPT"?

>> > 
>> > I added the other xz-options and created XZ_OPT as shown above, but in
>> > 'cdrom' there is an 'export XZ_OPT = ...'-line and I don't know how to
>> > handle these line.
>> > Is it still necessary?
>> 
>> No, that is no longer necessary and can be removed. We should just add
>> it to the tar command just like it is done in lfs/Config.

Understanding problem:
What do mean with "adding it to the tar command"? For me, this sounds as if
we need an "export $XZ_OPT"-line here...

>> > Do I have to 'export' the XZ_OPT'-variable in 'make.sh'?
>> 
>> No, because the chroot environment will throw away all environment
>> variables from the host system and then set them again. That's why we
>> have this enterchroot() function that does this with the env command.
>> 
>> > And how do I "add" this variable to 'lfsmake2' and 'ipfiredist'(-function)?
>> 
>> Just have a look at the functions. There should be loads of examples
>> where we pass other variables and you just need to add this one, too.

I think I've found these "examples" and tried several "local XZ_OPT='$XZ_OPT'"
placements and versions in 'lfsmake2' and 'ipfiredist' but none seemed to work
as I wanted.

>> > Sorry if these questions sound simple, but I'm guessing a bit too much while
>> > trying to get a grip on this.
>> > 
>> This is absolutely fine to sort this out first and then send the patch
>> :) That's what this list is for.

After several attempts I stopped guessing and concentrated on something
I could handle. Somehow I've got the feeling I'm making some really simple
mistakes - for reviewing I added my current test patches as attachments.

Please advise... ;-)

Best,
Matthias



[-- Attachment #2: 2018-05-12_xz-tuning.patch --]
[-- Type: text/plain, Size: 3470 bytes --]

--- cdrom	Sun May 06 20:15:10 2018
+++ cdrom	Mon Feb 19 20:35:38 2018
@@ -35,9 +35,6 @@
 	TAR_OPTIONS = --xz
 endif
 
-# Enable multi-threaded compression for LZMA
-export XZ_OPT = --threads=0
-
 ###############################################################################
 # Top-level Rules
 ###############################################################################
--- Config	Sun May 06 20:15:10 2018
+++ Config	Sun Feb 18 14:12:48 2018
@@ -209,7 +209,7 @@
 	tar xf /install/packages/package/files.tmp -C /install/packages/package/tmp/ \
 		-p --numeric-owner
 	rm -f /install/packages/package/files.tmp
-	cd /install/packages/package/tmp/ && XZ_OPT=-T0 tar -c -p --numeric-owner -J -f /install/packages/package/files.tar.xz *
+	cd /install/packages/package/tmp/ && $XZ_OPT tar -c -p --numeric-owner -J -f /install/packages/package/files.tar.xz *
 	rm -r /install/packages/package/tmp
 	-cat /install/packages/package/ROOTFILES | grep -v "#" > /install/packages/package/ROOTFILES.tmp
 	mv /install/packages/package/ROOTFILES.tmp /install/packages/package/ROOTFILES
--- make.sh	Thu May 10 09:53:08 2018
+++ make.sh	Sat May 12 16:03:55 2018
@@ -632,6 +632,10 @@
 
 	local PS1='\u:\w$ '
 
+# XZ-TUNING BEGIN
+	local XZ_OPT='$XZ_OPT'
+# XZ-TUNING END
+
 	enterchroot \
 		${EXTRA_PATH}bash -x -c "cd /usr/src/lfs && \
 			MAKETUNING=${MAKETUNING} \
@@ -653,6 +657,10 @@
 
 	local PS1='\u:\w$ '
 
+# XZ-TUNING BEGIN
+	local XZ_OPT='$XZ_OPT'
+# XZ-TUNING END
+
 	enterchroot \
 		bash -x -c "cd /usr/src/lfs && make -f $* LFS_BASEDIR=/usr/src dist" \
 		>> ${LOGFILE} 2>&1 &
@@ -860,6 +868,38 @@
 # Get the amount of memory in this build system
 HOST_MEM=$(system_memory)
 
+# XZ-TUNING BEGIN
+# Checking host memory
+
+if [ $HOST_MEM -ge 1024 ]; then
+	print_build_stage "Host-Memory: $HOST_MEM MiB"
+	print_build_stage "Host memory is OK (must be at least 1024 MiB), calculating XZ memory..."
+
+# Host memory is ok, calculating XZ memory
+echo
+
+	XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"
+	XZ_OPT="--threads=0 -8 $XZ_MEM"
+	print_build_stage "XZ-Memory:" $XZ_MEM
+	print_build_stage "XZ-Options:" $XZ_OPT
+	print_build_stage "XZ memory size is OK (must be at least 700 MiB), moving on..."
+
+echo
+read -p "Press [Enter] key to continue..."
+
+else
+
+	print_build_stage "Host-Memory: $HOST_MEM MiB"
+	print_build_stage "Not enough host memory (less than 1024 MiB, consider upgrading),"
+	print_build_stage "building will use standard XZ options."
+	XZ_OPT="-T0" # alte Standardparameter einsetzen
+
+echo
+read -p "Press [Enter] key to continue..."
+
+fi
+# XZ-TUNING END
+
 if [ -n "${BUILD_ARCH}" ]; then
 	configure_build "${BUILD_ARCH}"
 else
@@ -1760,7 +1800,7 @@
 	buildtoolchain
 	echo "`date -u '+%b %e %T'`: Create toolchain image for ${BUILD_ARCH}" | tee -a $LOGFILE
 	test -d $BASEDIR/cache/toolchains || mkdir -p $BASEDIR/cache/toolchains
-	cd $BASEDIR && XZ_OPT="-T0 -8" tar -Jc --exclude='log/_build.*.log' -f cache/toolchains/$SNAME-$VERSION-toolchain-$TOOLCHAINVER-${BUILD_ARCH}.tar.xz \
+	cd $BASEDIR && $XZ_OPT tar -Jc --exclude='log/_build.*.log' -f cache/toolchains/$SNAME-$VERSION-toolchain-$TOOLCHAINVER-${BUILD_ARCH}.tar.xz \
 		build/${TOOLS_DIR} build/bin/sh log >> $LOGFILE
 	md5sum cache/toolchains/$SNAME-$VERSION-toolchain-$TOOLCHAINVER-${BUILD_ARCH}.tar.xz \
 		> cache/toolchains/$SNAME-$VERSION-toolchain-$TOOLCHAINVER-${BUILD_ARCH}.md5

Re: Compress images with XZ

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 8393 bytes --]

Hi,

On Sat, 2018-05-12 at 16:14 +0200, Matthias Fischer wrote:
> HI,
> 
> On 11.05.2018 22:27, Michael Tremer wrote:
> > Hey Matthias,
> > 
> > are you still working on this?
> 
> Its still on my list, but I must confess: I didn't get a grip on this yet.
>  
> > I am asking because I just noticed that the new next build is about 50MB larger
> > than the old one due to the larger kernel and loads more of firmware files.
> 
> Yes, I saw this too. I was waiting for you to ask... ;-)

:)

> I'll try to explain what I did and where I'm stuck:
> 
> > ...
> > > > 
> > > > On 06.02.2018 02:05, Michael Tremer wrote:
> > > > > > ...
> > > > > > ((XZ_MEM=(($HOST_MEM)/10)*7)) && XZ_MEM=${XZ_MEM}MiB
> > > > > 
> > > > > You can write this even short:
> > > > > 
> > > > > XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"
> > > > 
> > > > Oneliner! Yes. I was looking for something like that... ;-)
> 
> This did work, no problem here.
> 
> > > > ...
> > > > # Host memory is ok, calculating XZ memory
> > > > 	XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"
> > > > 	XZ_OPT="--threads=0 -8 $XZ_MEM"
> > > > 	echo XZ-Memory: $XZ_MEM
> > > > 	echo XZ-Options: $XZ_OPT
> > > > 	echo "XZ memory size is OK (must be at least 700MiB), starting build..."
> > > > ...
> > > > ["echoes" for debugging!]
> > > > 
> > > > This is working. '$XZ_OPT' now gives me: --threads=0 -8 5450MiB
> > > 
> > > Will there be some checks in make.sh now that check if at least 1024MB
> > > of memory are available for build?
> > > ...
> 
> For this, I changed the above ("read -p..."-lines are just for debugging):
> 
> ***SNIP***
> ...
> # Get the amount of memory in this build system
> HOST_MEM=$(system_memory)
> 
> # XZ-TUNING BEGIN
> # Checking host memory
> 
> if [ $HOST_MEM -ge 1024 ]; then
> 	print_build_stage "Host-Memory: $HOST_MEM MiB"
> 	print_build_stage "Host memory is OK (must be at least 1024 MiB), calculating XZ memory..."
> 
> # Host memory is ok, calculating XZ memory
> echo
> 
> 	XZ_MEM="$(( HOST_MEM * 7 / 10 ))MiB"
> 	XZ_OPT="--threads=0 -8 $XZ_MEM"
> 	print_build_stage "XZ-Memory:" $XZ_MEM
> 	print_build_stage "XZ-Options:" $XZ_OPT
> 	print_build_stage "XZ memory size is OK (must be at least 700 MiB), moving on..."
> 
> echo
> read -p "Press [Enter] key to continue..."
> 
> else
> 
> 	print_build_stage "Host-Memory: $HOST_MEM MiB"
> 	print_build_stage "Not enough host memory (less than 1024 MiB, consider upgrading),"
> 	print_build_stage "building will use standard XZ options."
> 	XZ_OPT="-T0" # alte Standardparameter einsetzen
> 
> echo
> read -p "Press [Enter] key to continue..."
> 
> fi
> ...
> # XZ-TUNING END
> 
> ***SNAP***

I don't think that this needs to be so verbose. I just just ending the
build script when there is less than 1024 MB of memory detected will
do.

With that little memory you won't only have issues with XZ but also the
rest of the build environment.

> > > ...
> > >  I think there should be a warning if
> > >  not enough memory is available, but we should still try to build.
> 
> As far as I can see, this could be done by replacing the above 'echoes'
> with 'print_build_stage'(?).
> 
> E.g.:
> print_build_stage "Not enough host memory (less than 1024MiB, consider upgrading),"
> print_build_stage "building will use standard XZ options."
> 
> > > ...
> > > > But. ;-)
> > > > 
> > > > > You can put this probably next under "HOST_MEM=$(system_memory)" and then
> > > > > export this to the internal shell by adding the variable to lfsmake2 for
> > > > > the disk image and ipfiredist for the packages.
> > > > 
> > > > Sorry, I must confess that I don't know exactly how to do this. As I wrote:
> > > > "How do I pass the 'XZ_MEM'-variable to 'cdrom' and 'Config'?" Or in this case,
> > > > XZ_OPT!?
> 
> And this is still my problem! I don't get this to work. How do I "pass" this variable?
> 
> > > > > Then in lfs/cdrom and lfs/Config for the dist
> > > > > target, you just need to add the parameters to the command line.
> > > > 
> > > > In 'Config' I changed:
> > > > 
> > > > ...
> > > > cd /install/packages/package/tmp/ && XZ_OPT=-T0 tar -c -p --numeric-owner -J
> > > > -f
> > > > /install/packages/package/files.tar.xz *
> > > > ...
> > > > 
> > > > To:
> > > > ...
> > > > cd /install/packages/package/tmp/ && $XZ_OPT tar -c -p --numeric-owner -J -f
> > > > /install/packages/package/files.tar.xz *
> > > > ...
> > > > 
> > > > > Maybe it is a good idea to have a variable XZ_OPT that adds everything
> > > > > together and is then used in both places.
> 
> I tried that with no luck. I added the variable '$XZ_OPT' to 'lfs/Config' in line
> 212:
> 
> cd /install/packages/package/tmp/ && $XZ_OPT tar -c -p --numeric-owner...
>  
> But this usually ends - like today - in '_build_packages.log' with:
> 
> ***SNIP***
> ...
> #Change xxxKVERxxx to Kernelversion
> sed -e "s/xxxKVERxxx/4.14.40/g" -i /install/packages/package/update.sh
> cd /install/packages/package && tar cf ../core-upgrade-2.19-$(basename core/121).ipfire \
> 	update.sh files.tar.xz ROOTFILES
> rm -rf /install/packages/package
> sed -e "s/NAME/core-upgrade/g" \
> 	-e "s/VER/2.19/g" \
> 	-e "s/RELEASE/$(basename core/121)/g" \
> 	-e "s/DEPS//g" \
> 	-e "s/SIZE/`ls -l \/install\/packages\/core-upgrade-2.19-$(basename core/121).ipfire | awk '{ print $5 }'`/g" \
> 	< /usr/src/src/pakfire/meta > /install/packages/meta-core-upgrade-$(basename core/121)
> May 12 13:39:49: Building directfb DirectFB-1.7.7.tar.gz checksum OK
> + cd /usr/src/lfs
> + make -f directfb LFS_BASEDIR=/usr/src dist
> '/usr/src/config/rootfiles/packages/i586/directfb' -> '/install/packages/package/ROOTFILES'
> /bin/sh: Z_OPT: command not found
> make: *** [directfb:60: dist] Error 127
> ...
> ***SNAP***
> 
> Why "Z_OPT"?

Make required you to write the variable name in brackets: $(XZ_OPT)
instead of $XZ_OPT. It's not shell.

> 
> > > > 
> > > > I added the other xz-options and created XZ_OPT as shown above, but in
> > > > 'cdrom' there is an 'export XZ_OPT = ...'-line and I don't know how to
> > > > handle these line.
> > > > Is it still necessary?
> > > 
> > > No, that is no longer necessary and can be removed. We should just add
> > > it to the tar command just like it is done in lfs/Config.
> 
> Understanding problem:
> What do mean with "adding it to the tar command"? For me, this sounds as if
> we need an "export $XZ_OPT"-line here...

Yes, unfortunately like that. Or you can just pass it before the tar
command is executed. Like this:

  XZ_OPT="$(XZ_OPT)" tar ...

That will also export the variable but only for that command and
nothing else.

> 
> > > > Do I have to 'export' the XZ_OPT'-variable in 'make.sh'?
> > > 
> > > No, because the chroot environment will throw away all environment
> > > variables from the host system and then set them again. That's why we
> > > have this enterchroot() function that does this with the env command.
> > > 
> > > > And how do I "add" this variable to 'lfsmake2' and 'ipfiredist'(-function)?
> > > 
> > > Just have a look at the functions. There should be loads of examples
> > > where we pass other variables and you just need to add this one, too.
> 
> I think I've found these "examples" and tried several "local XZ_OPT='$XZ_OPT'"
> placements and versions in 'lfsmake2' and 'ipfiredist' but none seemed to work
> as I wanted.

Why not?

Just try to add "echo $(XZ_OPT)" to one of the lfs scripts and see if
you can see the desired output.

Did you look at enterchroot() yet?

> > > > Sorry if these questions sound simple, but I'm guessing a bit too much while
> > > > trying to get a grip on this.
> > > > 
> > > 
> > > This is absolutely fine to sort this out first and then send the patch
> > > :) That's what this list is for.
> 
> After several attempts I stopped guessing and concentrated on something
> I could handle. Somehow I've got the feeling I'm making some really simple
> mistakes - for reviewing I added my current test patches as attachments.

No this is really tricky. Although the patch in the will only be a few
lines, this is really not just writing it down and sending it.

> Please advise... ;-)

Done. Please send more questions.

Best,
-Michael

> 
> Best,
> Matthias

Re: Planning out Core Update 117

[ummeegge]

[-- Attachment #1: Type: text/plain, Size: 1140 bytes --]

Hi Michael,
have here some fixes from Bugzilla causing OpenVPN --> https://bugzilla.ipfire.org/show_bug.cgi?id=10482 a clean patch can be found in here --> https://github.com/ummeegge/ipfire_core111_ovpn_fixes/commit/da7349461630ce9e38a9a5b47f41bc644e907ef8#diff-2011d5d928fd214cacb83844729c65cc but needs to be patched for Core 115 and this here --> https://bugzilla.ipfire.org/show_bug.cgi?id=11364 with a couples of patches whereby the "--ns-cert-type" patch is already merged .

If you think some of them are useful, let it me know ;-) .

Greetings,

Erik


Am 06.11.2017 um 19:32 schrieb Michael Tremer:

> Hello,
> 
> so the big Core Update 115 has been shipped last week. Core Update 116 is coming
> today. So the next one should be a little bit calmer please :)
> 
> I do not have anything so far except some smaller bug fixes here and there. Does
> anyone have anything big? It doesn't have to be another big update. We could
> also just improve IPFire where ever it is needed with loads of smaller changes
> and start working on 3 and other sub-projects that are floating around...
> 
> Best,
> -Michael

Re: Planning out Core Update 117

[Michael Tremer]

[-- Attachment #1: Type: text/plain, Size: 1479 bytes --]

Great.

Could you send the patch to the list so that we can comment on it?

Also, do you have any plans to migrate to OpenVPN 2.4? Not sure how long 2.3
will be patched.

Best,
-Michael

On Fri, 2017-11-10 at 12:01 +0100, ummeegge wrote:
> Hi Michael,
> have here some fixes from Bugzilla causing OpenVPN --> https://bugzilla.ipfire
> .org/show_bug.cgi?id=10482 a clean patch can be found in here --> https://gith
> ub.com/ummeegge/ipfire_core111_ovpn_fixes/commit/da7349461630ce9e38a9a5b47f41b
> c644e907ef8#diff-2011d5d928fd214cacb83844729c65cc but needs to be patched for
> Core 115 and this here --> https://bugzilla.ipfire.org/show_bug.cgi?id=11364
> with a couples of patches whereby the "--ns-cert-type" patch is already merged
> .
> 
> If you think some of them are useful, let it me know ;-) .
> 
> Greetings,
> 
> Erik
> 
> 
> Am 06.11.2017 um 19:32 schrieb Michael Tremer:
> 
> > Hello,
> > 
> > so the big Core Update 115 has been shipped last week. Core Update 116 is
> > coming
> > today. So the next one should be a little bit calmer please :)
> > 
> > I do not have anything so far except some smaller bug fixes here and there.
> > Does
> > anyone have anything big? It doesn't have to be another big update. We could
> > also just improve IPFire where ever it is needed with loads of smaller
> > changes
> > and start working on 3 and other sub-projects that are floating around...
> > 
> > Best,
> > -Michael
> 
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Planning out Core Update 117

[ummeegge]

[-- Attachment #1: Type: text/plain, Size: 1423 bytes --]

Hi Michael,

> Great.
> 
> Could you send the patch to the list so that we can comment on it?
OK will adapt the patches to Core 116 and send it as soon as possible to the dev list.

> 
> Also, do you have any plans to migrate to OpenVPN 2.4? Not sure how long 2.3
> will be patched.
Have builded and also tested the 2.4 from Alpha version to the until now actual 2.4.4 and have had no issues at all. The development in there --> https://forum.ipfire.org/viewtopic.php?f=50&t=18067#p104536 is relatively lonely since i do the testings on my own accept one user which made a very quick positive test. Since i normally want some more testers before i go for a merge request i am there a little lost until now since there is no feedback at all.
I developed a minimal solution which includes the needed changes to bring the 2.4.x version to life on IPFire but there is also an extended one with several new features also available over the ovpnmain.cgi.

Larger testing round can also be found in here --> https://forum.ipfire.org/viewtopic.php?f=50&t=17656 whereby i tried to explain also the new and for interesting features also al little.

I think 2.3 will be supported for longer time but 2.4 has some very good improvements under the hood and above and i think it might be nice for IPFire even some more testers might be great for that may it is nevertheless time to push it to mainline ?

Best,

Erik