From mboxrd@z Thu Jan 1 00:00:00 1970 From: Matthias Fischer To: development@lists.ipfire.org Subject: Re: Compress images with XZ Date: Sat, 12 May 2018 16:14:07 +0200 Message-ID: <606450a7-50ef-d1f3-4e8f-3f7c745b9ed7@ipfire.org> In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============4207879685927666968==" List-Id: --===============4207879685927666968== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable HI, On 11.05.2018 22:27, Michael Tremer wrote: > Hey Matthias, >=20 > are you still working on this? Its still on my list, but I must confess: I didn't get a grip on this yet. =20 > I am asking because I just noticed that the new next build is about 50MB la= rger > than the old one due to the larger kernel and loads more of firmware files. Yes, I saw this too. I was waiting for you to ask... ;-) I'll try to explain what I did and where I'm stuck: > ... >> >=20 >> > On 06.02.2018 02:05, Michael Tremer wrote: >> > > > ... >> > > > ((XZ_MEM=3D(($HOST_MEM)/10)*7)) && XZ_MEM=3D${XZ_MEM}MiB >> > >=20 >> > > You can write this even short: >> > >=20 >> > > XZ_MEM=3D"$(( HOST_MEM * 7 / 10 ))MiB" >> >=20 >> > Oneliner! Yes. I was looking for something like that... ;-) This did work, no problem here. >> > ... >> > # Host memory is ok, calculating XZ memory >> > XZ_MEM=3D"$(( HOST_MEM * 7 / 10 ))MiB" >> > XZ_OPT=3D"--threads=3D0 -8 $XZ_MEM" >> > echo XZ-Memory: $XZ_MEM >> > echo XZ-Options: $XZ_OPT >> > echo "XZ memory size is OK (must be at least 700MiB), starting build..." >> > ... >> > ["echoes" for debugging!] >> >=20 >> > This is working. '$XZ_OPT' now gives me: --threads=3D0 -8 5450MiB >>=20 >> Will there be some checks in make.sh now that check if at least 1024MB >> of memory are available for build? >> ... For this, I changed the above ("read -p..."-lines are just for debugging): ***SNIP*** ... # Get the amount of memory in this build system HOST_MEM=3D$(system_memory) # XZ-TUNING BEGIN # Checking host memory if [ $HOST_MEM -ge 1024 ]; then print_build_stage "Host-Memory: $HOST_MEM MiB" print_build_stage "Host memory is OK (must be at least 1024 MiB), calculatin= g XZ memory..." # Host memory is ok, calculating XZ memory echo XZ_MEM=3D"$(( HOST_MEM * 7 / 10 ))MiB" XZ_OPT=3D"--threads=3D0 -8 $XZ_MEM" print_build_stage "XZ-Memory:" $XZ_MEM print_build_stage "XZ-Options:" $XZ_OPT print_build_stage "XZ memory size is OK (must be at least 700 MiB), moving o= n..." echo read -p "Press [Enter] key to continue..." else print_build_stage "Host-Memory: $HOST_MEM MiB" print_build_stage "Not enough host memory (less than 1024 MiB, consider upgr= ading)," print_build_stage "building will use standard XZ options." XZ_OPT=3D"-T0" # alte Standardparameter einsetzen echo read -p "Press [Enter] key to continue..." fi ... # XZ-TUNING END ***SNAP*** >> ... >> I think there should be a warning if >> not enough memory is available, but we should still try to build. As far as I can see, this could be done by replacing the above 'echoes' with 'print_build_stage'(?). E.g.: print_build_stage "Not enough host memory (less than 1024MiB, consider upgrad= ing)," print_build_stage "building will use standard XZ options." >> ... >> > But. ;-) >> >=20 >> > > You can put this probably next under "HOST_MEM=3D$(system_memory)" and= then >> > > export this to the internal shell by adding the variable to lfsmake2 f= or >> > > the disk image and ipfiredist for the packages. >> >=20 >> > Sorry, I must confess that I don't know exactly how to do this. As I wro= te: >> > "How do I pass the 'XZ_MEM'-variable to 'cdrom' and 'Config'?" Or in thi= s case, >> > XZ_OPT!? And this is still my problem! I don't get this to work. How do I "pass" this = variable? >> > > Then in lfs/cdrom and lfs/Config for the dist >> > > target, you just need to add the parameters to the command line. >> >=20 >> > In 'Config' I changed: >> >=20 >> > ... >> > cd /install/packages/package/tmp/ && XZ_OPT=3D-T0 tar -c -p --numeric-ow= ner -J >> > -f >> > /install/packages/package/files.tar.xz * >> > ... >> >=20 >> > To: >> > ... >> > cd /install/packages/package/tmp/ && $XZ_OPT tar -c -p --numeric-owner -= J -f >> > /install/packages/package/files.tar.xz * >> > ... >> >=20 >> > > Maybe it is a good idea to have a variable XZ_OPT that adds everything >> > > together and is then used in both places. I tried that with no luck. I added the variable '$XZ_OPT' to 'lfs/Config' in = line 212: cd /install/packages/package/tmp/ && $XZ_OPT tar -c -p --numeric-owner... =20 But this usually ends - like today - in '_build_packages.log' with: ***SNIP*** ... #Change xxxKVERxxx to Kernelversion sed -e "s/xxxKVERxxx/4.14.40/g" -i /install/packages/package/update.sh cd /install/packages/package && tar cf ../core-upgrade-2.19-$(basename core/1= 21).ipfire \ update.sh files.tar.xz ROOTFILES rm -rf /install/packages/package sed -e "s/NAME/core-upgrade/g" \ -e "s/VER/2.19/g" \ -e "s/RELEASE/$(basename core/121)/g" \ -e "s/DEPS//g" \ -e "s/SIZE/`ls -l \/install\/packages\/core-upgrade-2.19-$(basename core/121= ).ipfire | awk '{ print $5 }'`/g" \ < /usr/src/src/pakfire/meta > /install/packages/meta-core-upgrade-$(basename= core/121) May 12 13:39:49: Building directfb DirectFB-1.7.7.tar.gz checksum OK + cd /usr/src/lfs + make -f directfb LFS_BASEDIR=3D/usr/src dist '/usr/src/config/rootfiles/packages/i586/directfb' -> '/install/packages/pack= age/ROOTFILES' /bin/sh: Z_OPT: command not found make: *** [directfb:60: dist] Error 127 ... ***SNAP*** Why "Z_OPT"? >> >=20 >> > I added the other xz-options and created XZ_OPT as shown above, but in >> > 'cdrom' there is an 'export XZ_OPT =3D ...'-line and I don't know how to >> > handle these line. >> > Is it still necessary? >>=20 >> No, that is no longer necessary and can be removed. We should just add >> it to the tar command just like it is done in lfs/Config. Understanding problem: What do mean with "adding it to the tar command"? For me, this sounds as if we need an "export $XZ_OPT"-line here... >> > Do I have to 'export' the XZ_OPT'-variable in 'make.sh'? >>=20 >> No, because the chroot environment will throw away all environment >> variables from the host system and then set them again. That's why we >> have this enterchroot() function that does this with the env command. >>=20 >> > And how do I "add" this variable to 'lfsmake2' and 'ipfiredist'(-functio= n)? >>=20 >> Just have a look at the functions. There should be loads of examples >> where we pass other variables and you just need to add this one, too. I think I've found these "examples" and tried several "local XZ_OPT=3D'$XZ_OP= T'" placements and versions in 'lfsmake2' and 'ipfiredist' but none seemed to work as I wanted. >> > Sorry if these questions sound simple, but I'm guessing a bit too much w= hile >> > trying to get a grip on this. >> >=20 >> This is absolutely fine to sort this out first and then send the patch >> :) That's what this list is for. After several attempts I stopped guessing and concentrated on something I could handle. Somehow I've got the feeling I'm making some really simple mistakes - for reviewing I added my current test patches as attachments. Please advise... ;-) Best, Matthias --===============4207879685927666968== Content-Type: text/plain Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="2018-05-12_xz-tuning.patch" MIME-Version: 1.0 LS0tIGNkcm9tCVN1biBNYXkgMDYgMjA6MTU6MTAgMjAxOA0KKysrIGNkcm9tCU1vbiBGZWIgMTkg MjA6MzU6MzggMjAxOA0KQEAgLTM1LDkgKzM1LDYgQEANCiAJVEFSX09QVElPTlMgPSAtLXh6DQog ZW5kaWYNCiANCi0jIEVuYWJsZSBtdWx0aS10aHJlYWRlZCBjb21wcmVzc2lvbiBmb3IgTFpNQQ0K LWV4cG9ydCBYWl9PUFQgPSAtLXRocmVhZHM9MA0KLQ0KICMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMN CiAjIFRvcC1sZXZlbCBSdWxlcw0KICMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMNCi0tLSBDb25maWcJ U3VuIE1heSAwNiAyMDoxNToxMCAyMDE4DQorKysgQ29uZmlnCVN1biBGZWIgMTggMTQ6MTI6NDgg MjAxOA0KQEAgLTIwOSw3ICsyMDksNyBAQA0KIAl0YXIgeGYgL2luc3RhbGwvcGFja2FnZXMvcGFj a2FnZS9maWxlcy50bXAgLUMgL2luc3RhbGwvcGFja2FnZXMvcGFja2FnZS90bXAvIFwNCiAJCS1w IC0tbnVtZXJpYy1vd25lcg0KIAlybSAtZiAvaW5zdGFsbC9wYWNrYWdlcy9wYWNrYWdlL2ZpbGVz LnRtcA0KLQljZCAvaW5zdGFsbC9wYWNrYWdlcy9wYWNrYWdlL3RtcC8gJiYgWFpfT1BUPS1UMCB0 YXIgLWMgLXAgLS1udW1lcmljLW93bmVyIC1KIC1mIC9pbnN0YWxsL3BhY2thZ2VzL3BhY2thZ2Uv ZmlsZXMudGFyLnh6ICoNCisJY2QgL2luc3RhbGwvcGFja2FnZXMvcGFja2FnZS90bXAvICYmICRY Wl9PUFQgdGFyIC1jIC1wIC0tbnVtZXJpYy1vd25lciAtSiAtZiAvaW5zdGFsbC9wYWNrYWdlcy9w YWNrYWdlL2ZpbGVzLnRhci54eiAqDQogCXJtIC1yIC9pbnN0YWxsL3BhY2thZ2VzL3BhY2thZ2Uv dG1wDQogCS1jYXQgL2luc3RhbGwvcGFja2FnZXMvcGFja2FnZS9ST09URklMRVMgfCBncmVwIC12 ICIjIiA+IC9pbnN0YWxsL3BhY2thZ2VzL3BhY2thZ2UvUk9PVEZJTEVTLnRtcA0KIAltdiAvaW5z dGFsbC9wYWNrYWdlcy9wYWNrYWdlL1JPT1RGSUxFUy50bXAgL2luc3RhbGwvcGFja2FnZXMvcGFj a2FnZS9ST09URklMRVMNCi0tLSBtYWtlLnNoCVRodSBNYXkgMTAgMDk6NTM6MDggMjAxOA0KKysr IG1ha2Uuc2gJU2F0IE1heSAxMiAxNjowMzo1NSAyMDE4DQpAQCAtNjMyLDYgKzYzMiwxMCBAQA0K IA0KIAlsb2NhbCBQUzE9J1x1Olx3JCAnDQogDQorIyBYWi1UVU5JTkcgQkVHSU4NCisJbG9jYWwg WFpfT1BUPSckWFpfT1BUJw0KKyMgWFotVFVOSU5HIEVORA0KKw0KIAllbnRlcmNocm9vdCBcDQog CQkke0VYVFJBX1BBVEh9YmFzaCAteCAtYyAiY2QgL3Vzci9zcmMvbGZzICYmIFwNCiAJCQlNQUtF VFVOSU5HPSR7TUFLRVRVTklOR30gXA0KQEAgLTY1Myw2ICs2NTcsMTAgQEANCiANCiAJbG9jYWwg UFMxPSdcdTpcdyQgJw0KIA0KKyMgWFotVFVOSU5HIEJFR0lODQorCWxvY2FsIFhaX09QVD0nJFha X09QVCcNCisjIFhaLVRVTklORyBFTkQNCisNCiAJZW50ZXJjaHJvb3QgXA0KIAkJYmFzaCAteCAt YyAiY2QgL3Vzci9zcmMvbGZzICYmIG1ha2UgLWYgJCogTEZTX0JBU0VESVI9L3Vzci9zcmMgZGlz dCIgXA0KIAkJPj4gJHtMT0dGSUxFfSAyPiYxICYNCkBAIC04NjAsNiArODY4LDM4IEBADQogIyBH ZXQgdGhlIGFtb3VudCBvZiBtZW1vcnkgaW4gdGhpcyBidWlsZCBzeXN0ZW0NCiBIT1NUX01FTT0k KHN5c3RlbV9tZW1vcnkpDQogDQorIyBYWi1UVU5JTkcgQkVHSU4NCisjIENoZWNraW5nIGhvc3Qg bWVtb3J5DQorDQoraWYgWyAkSE9TVF9NRU0gLWdlIDEwMjQgXTsgdGhlbg0KKwlwcmludF9idWls ZF9zdGFnZSAiSG9zdC1NZW1vcnk6ICRIT1NUX01FTSBNaUIiDQorCXByaW50X2J1aWxkX3N0YWdl ICJIb3N0IG1lbW9yeSBpcyBPSyAobXVzdCBiZSBhdCBsZWFzdCAxMDI0IE1pQiksIGNhbGN1bGF0 aW5nIFhaIG1lbW9yeS4uLiINCisNCisjIEhvc3QgbWVtb3J5IGlzIG9rLCBjYWxjdWxhdGluZyBY WiBtZW1vcnkNCitlY2hvDQorDQorCVhaX01FTT0iJCgoIEhPU1RfTUVNICogNyAvIDEwICkpTWlC Ig0KKwlYWl9PUFQ9Ii0tdGhyZWFkcz0wIC04ICRYWl9NRU0iDQorCXByaW50X2J1aWxkX3N0YWdl ICJYWi1NZW1vcnk6IiAkWFpfTUVNDQorCXByaW50X2J1aWxkX3N0YWdlICJYWi1PcHRpb25zOiIg JFhaX09QVA0KKwlwcmludF9idWlsZF9zdGFnZSAiWFogbWVtb3J5IHNpemUgaXMgT0sgKG11c3Qg YmUgYXQgbGVhc3QgNzAwIE1pQiksIG1vdmluZyBvbi4uLiINCisNCitlY2hvDQorcmVhZCAtcCAi UHJlc3MgW0VudGVyXSBrZXkgdG8gY29udGludWUuLi4iDQorDQorZWxzZQ0KKw0KKwlwcmludF9i dWlsZF9zdGFnZSAiSG9zdC1NZW1vcnk6ICRIT1NUX01FTSBNaUIiDQorCXByaW50X2J1aWxkX3N0 YWdlICJOb3QgZW5vdWdoIGhvc3QgbWVtb3J5IChsZXNzIHRoYW4gMTAyNCBNaUIsIGNvbnNpZGVy IHVwZ3JhZGluZyksIg0KKwlwcmludF9idWlsZF9zdGFnZSAiYnVpbGRpbmcgd2lsbCB1c2Ugc3Rh bmRhcmQgWFogb3B0aW9ucy4iDQorCVhaX09QVD0iLVQwIiAjIGFsdGUgU3RhbmRhcmRwYXJhbWV0 ZXIgZWluc2V0emVuDQorDQorZWNobw0KK3JlYWQgLXAgIlByZXNzIFtFbnRlcl0ga2V5IHRvIGNv bnRpbnVlLi4uIg0KKw0KK2ZpDQorIyBYWi1UVU5JTkcgRU5EDQorDQogaWYgWyAtbiAiJHtCVUlM RF9BUkNIfSIgXTsgdGhlbg0KIAljb25maWd1cmVfYnVpbGQgIiR7QlVJTERfQVJDSH0iDQogZWxz ZQ0KQEAgLTE3NjAsNyArMTgwMCw3IEBADQogCWJ1aWxkdG9vbGNoYWluDQogCWVjaG8gImBkYXRl IC11ICcrJWIgJWUgJVQnYDogQ3JlYXRlIHRvb2xjaGFpbiBpbWFnZSBmb3IgJHtCVUlMRF9BUkNI fSIgfCB0ZWUgLWEgJExPR0ZJTEUNCiAJdGVzdCAtZCAkQkFTRURJUi9jYWNoZS90b29sY2hhaW5z IHx8IG1rZGlyIC1wICRCQVNFRElSL2NhY2hlL3Rvb2xjaGFpbnMNCi0JY2QgJEJBU0VESVIgJiYg WFpfT1BUPSItVDAgLTgiIHRhciAtSmMgLS1leGNsdWRlPSdsb2cvX2J1aWxkLioubG9nJyAtZiBj YWNoZS90b29sY2hhaW5zLyRTTkFNRS0kVkVSU0lPTi10b29sY2hhaW4tJFRPT0xDSEFJTlZFUi0k e0JVSUxEX0FSQ0h9LnRhci54eiBcDQorCWNkICRCQVNFRElSICYmICRYWl9PUFQgdGFyIC1KYyAt LWV4Y2x1ZGU9J2xvZy9fYnVpbGQuKi5sb2cnIC1mIGNhY2hlL3Rvb2xjaGFpbnMvJFNOQU1FLSRW RVJTSU9OLXRvb2xjaGFpbi0kVE9PTENIQUlOVkVSLSR7QlVJTERfQVJDSH0udGFyLnh6IFwNCiAJ CWJ1aWxkLyR7VE9PTFNfRElSfSBidWlsZC9iaW4vc2ggbG9nID4+ICRMT0dGSUxFDQogCW1kNXN1 bSBjYWNoZS90b29sY2hhaW5zLyRTTkFNRS0kVkVSU0lPTi10b29sY2hhaW4tJFRPT0xDSEFJTlZF Ui0ke0JVSUxEX0FSQ0h9LnRhci54eiBcDQogCQk+IGNhY2hlL3Rvb2xjaGFpbnMvJFNOQU1FLSRW RVJTSU9OLXRvb2xjaGFpbi0kVE9PTENIQUlOVkVSLSR7QlVJTERfQVJDSH0ubWQ1DQo= --===============4207879685927666968==--