Hi,
I started building my own packages for IPFire and could use a little guidance.
The package I want to add to pakfire has a number of dependencies. It goes like this:
* Several are already part of IPFire (libpcap, libtool, autoconf tools, and some more). I don't need to do anything in most cases, but what if my package requires a newer version than the one available in pakfire? * At least one dependency is not already available (it's redis). I already created a pakfire package for this one and tested it on my own IPFire box, seems to work. * Yet some other dependencies are typically downloaded and compiled as part of the make step of the package build. My understanding of why it is this way is because of the tight integration between the software and those dependencies. My question is this: is it necessary to create pakfire packages for the ones usually downloaded and compiled in make process, or is it ok to let the make process take care of it? One more thing worth mentioning is that some of those packages automatically downloaded and compiled are available in IPFire (like rrdtool). Will that create conflicts or can it safely coexists?
Some more questions:
* When install my redis pakfire package I get the message "/opt/pakfire/lib/functions.sh: line 91: /etc/init.d/: is a directory", why is that? * Should a make test step (if there is one) be included in the lfs definition? * Again in the lfs definition, what's the difference between using "install -v -m644 $(DIR_SRC)/config/redis/redis.conf /etc/redis/redis.conf" and "cp $(DIR_SRC)/config/redis/redis.conf /etc/redis/redis.conf". Which one should I use? Which one should I use for initscripts? I've looked at several other lfs files but seems things are done slightly differently which makes it a bit harder to understand. * I don't see anywhere in the lfs definition how I would run a process as a new non-root user. I see many add-ons runb as root which is not as secure as it could be. What is best practice here and how to go about it.
Regards.
Hello,
I am not sure if I completely understood what you are asking, but I will have a try at answering them. If something is still not clear, please try to give some more detail...
I also assume that we are talking about IPFire 2 here.
On Sat, 2014-07-05 at 17:45 +0800, Ghislain Hachey wrote:
Hi,
I started building my own packages for IPFire and could use a little guidance.
The package I want to add to pakfire has a number of dependencies. It goes like this:
- Several are already part of IPFire (libpcap, libtool, autoconf tools,
and some more). I don't need to do anything in most cases, but what if my package requires a newer version than the one available in pakfire?
In that case, you should update them and send us the patch. Just make sure that the new version is completely backwards-compatible and does not break anything in the build process or at run time. Sometimes we keep older versions of the libraries around (check out the lfs/*-compat files for an example).
- At least one dependency is not already available (it's redis). I
already created a pakfire package for this one and tested it on my own IPFire box, seems to work.
Great.
- Yet some other dependencies are typically downloaded and compiled as
part of the make step of the package build. My understanding of why it is this way is because of the tight integration between the software and those dependencies. My question is this: is it necessary to create pakfire packages for the ones usually downloaded and compiled in make process, or is it ok to let the make process take care of it? One more thing worth mentioning is that some of those packages automatically downloaded and compiled are available in IPFire (like rrdtool). Will that create conflicts or can it safely coexists?
I don't really know what you want to know here. Tools like rrdtool are not packaged because they belong to the core system which is not managed by pakfire. pakfire only manages add-ons.
You can use those things if they are around and you can assume that they are always there when they are part of the core system. Otherwise just add them as a dependency.
If you are adding some things that are not already there, please create an extra file (= package) in the lfs/ directory. Don't build them all in one step.
Some more questions:
- When install my redis pakfire package I get the message
"/opt/pakfire/lib/functions.sh: line 91: /etc/init.d/: is a directory", why is that?
This is because the install.sh script requires some environment variables to be set which is not the case when you execute it manually. pakfire will do that.
- Should a make test step (if there is one) be included in the lfs
definition?
You may add the commands but please comment them out. This is a huge problem of the build system of IPFire 2 that you need to rebuild the entire distribution from scratch every time you change some tiny bit. So that would make the build process even longer and basically not feasible on architectures like ARM. In IPFire 3 we require to execute the testsuite when existant.
- Again in the lfs definition, what's the difference between using
"install -v -m644 $(DIR_SRC)/config/redis/redis.conf /etc/redis/redis.conf" and "cp $(DIR_SRC)/config/redis/redis.conf /etc/redis/redis.conf". Which one should I use? Which one should I use for initscripts? I've looked at several other lfs files but seems things are done slightly differently which makes it a bit harder to understand.
I prefer the install command which ensures that permissions and a possible ownership of the files are correct. In the end it is not important which one to use really.
- I don't see anywhere in the lfs definition how I would run a process
as a new non-root user. I see many add-ons runb as root which is not as secure as it could be. What is best practice here and how to go about it.
Most of the add-on processes are executed as root and drop their privileges shortly after startup. openvpn for example needs to bind to a privileged port and add routes which requires root privileges. This is also the preferred way to do it.
In the case that your software does not support this you may either patch the software or use sudo.
Regards.
Hope this helps. If not, please feel free to post any more questions and even include some files or parts of the files.
Best, -Michael
On 7/5/14, 22:15, Michael Tremer wrote:
- Yet some other dependencies are typically downloaded and compiled as
part of the make step of the package build. My understanding of why it is this way is because of the tight integration between the software and those dependencies. My question is this: is it necessary to create pakfire packages for the ones usually downloaded and compiled in make process, or is it ok to let the make process take care of it? One more thing worth mentioning is that some of those packages automatically downloaded and compiled are available in IPFire (like rrdtool). Will that create conflicts or can it safely coexists?
I don't really know what you want to know here. Tools like rrdtool are not packaged because they belong to the core system which is not managed by pakfire. pakfire only manages add-ons.
Thanks, great. Yes that mostly covers everything. Except the above. What I meant is that the software I am trying to package packages its own rrdtool, JITLua, amongst a few others. They are dependencies but installed by the make step in a normal build process (also compiling those from source). Now I know that, for example, rrdtool is already part of IPFire and want to know if it will be ok to have two copies installed (one by pakfire and the other by the make step of the software I am packaging).
Either way, I think its best I just give it a go and come back here with more specifics.
Thanks,
-- GH
On Sun, 2014-07-06 at 07:01 +0800, Ghislain Hachey wrote:
On 7/5/14, 22:15, Michael Tremer wrote:
- Yet some other dependencies are typically downloaded and compiled as
part of the make step of the package build. My understanding of why it is this way is because of the tight integration between the software and those dependencies. My question is this: is it necessary to create pakfire packages for the ones usually downloaded and compiled in make process, or is it ok to let the make process take care of it? One more thing worth mentioning is that some of those packages automatically downloaded and compiled are available in IPFire (like rrdtool). Will that create conflicts or can it safely coexists?
I don't really know what you want to know here. Tools like rrdtool are not packaged because they belong to the core system which is not managed by pakfire. pakfire only manages add-ons.
Thanks, great. Yes that mostly covers everything. Except the above. What I meant is that the software I am trying to package packages its own rrdtool, JITLua, amongst a few others. They are dependencies but installed by the make step in a normal build process (also compiling those from source). Now I know that, for example, rrdtool is already part of IPFire and want to know if it will be ok to have two copies installed (one by pakfire and the other by the make step of the software I am packaging).
That's what we call bundled packages (very often libraries) and which are extremely discouraged. The problem that comes with that is that when a component gets updated to resolve a certain issue this problem is still in the twenty other copies of the same software. Imagine that for things like Heartbleed. It also consumes space, increases the build time and so on.
You should use the provided versions of those tools and libraries or modify them if that is required. All other components that are missing should be created as individual packages.
Either way, I think its best I just give it a go and come back here with more specifics.
Thanks,
-- GH
On 7/6/14, 16:57, Michael Tremer wrote:
That's what we call bundled packages (very often libraries) and which are extremely discouraged. The problem that comes with that is that when a component gets updated to resolve a certain issue this problem is still in the twenty other copies of the same software. Imagine that for things like Heartbleed. It also consumes space, increases the build time and so on.
You should use the provided versions of those tools and libraries or modify them if that is required. All other components that are missing should be created as individual packages.
Yeah, I understand that. My only fear is that the software in question is including those third party libs as deps because they are needed in a very particular shape (specific versions, some source changes, etc.) to make it all work properly as a whole. I will check with the developers upstream regarding this and if not I will see if I can not modify the build process to include those deps as individual packages in IPFire.
Regarding IPFire version 3, If there are specific things I can do to help, sure. I'll go through information in the links to get started.
On Sun, 2014-07-06 at 17:44 +0800, Ghislain Hachey wrote:
On 7/6/14, 16:57, Michael Tremer wrote:
That's what we call bundled packages (very often libraries) and which are extremely discouraged. The problem that comes with that is that when a component gets updated to resolve a certain issue this problem is still in the twenty other copies of the same software. Imagine that for things like Heartbleed. It also consumes space, increases the build time and so on.
You should use the provided versions of those tools and libraries or modify them if that is required. All other components that are missing should be created as individual packages.
Yeah, I understand that. My only fear is that the software in question is including those third party libs as deps because they are needed in a very particular shape (specific versions, some source changes, etc.) to make it all work properly as a whole. I will check with the developers upstream regarding this and if not I will see if I can not modify the build process to include those deps as individual packages in IPFire.
Bundled libs are a real worry and in the case some software requires a certain version I would consider this software as seriously broken.
There is a reason why we have dynamic libraries and that those are replaceable. We can easily fix bugs and security issues and those fixes will get rolled out to the entire system. Therefore it is a must.
I wonder why I cannot find any documentation about this topic on our wiki...
Regarding IPFire version 3, If there are specific things I can do to help, sure. I'll go through information in the links to get started.
Hi Devs,
i build actual MySQL 5.6.19 for IPFire. A Question is, how i can bring up the tar.gz to the ipfire repository? Did you download and reupload it? or can i safe insert the original url for the tar.gz ?
best regards
Sascha
Hi,
i have successful compiled and installed MySQL 5.6.19. I provide the patch after i have runned some tests. I have a little Patch for make.sh too, to build single packages (little bit tricky if you have a fresh checkout and not full compiled the entire ipfire.
best regards
Sascha
Am Dienstag, den 08.07.2014, 12:35 +0200 schrieb Sascha Kilian:
Hi Devs,
i build actual MySQL 5.6.19 for IPFire. A Question is, how i can bring up the tar.gz to the ipfire repository? Did you download and reupload it? or can i safe insert the original url for the tar.gz ?
best regards
Sascha
Development mailing list Development@lists.ipfire.org http://lists.ipfire.org/mailman/listinfo/development
Hi,
I am trying to achieve something here I don't know if this is supported. I've built a custom addon for ntopng, a traffic monitoring tool which makes use of LuaJIT for its web UI scripting language (which is great for embedded devices). When starting the service on my home IPFire I quickly get the following error message.
PANIC: unprotected error in call to Lua API (runtime code generation failed, restricted kernel?)
Which leads me to think that a JIT is not supported in the kernel configuration of IPFire, or am I way off? This is a tool I'd really like to get working on my IPFire, any ideas?
Regards,
-- GH
On 2014-07-29 10:13, Ghislain Hachey wrote:
Hi,
I am trying to achieve something here I don't know if this is supported. I've built a custom addon for ntopng, a traffic monitoring tool which makes use of LuaJIT for its web UI scripting language (which is great for embedded devices). When starting the service on my home IPFire I quickly get the following error message.
PANIC: unprotected error in call to Lua API (runtime code generation failed, restricted kernel?)
Which leads me to think that a JIT is not supported in the kernel configuration of IPFire, or am I way off? This is a tool I'd really like to get working on my IPFire, any ideas?
Check the kernel log. If PaX mprotect has stopped the execution it will be logged and in most cases the protection can disabled for single binaries with paxctl.
Arne
On 7/6/14, 16:57, Michael Tremer wrote:
That's what we call bundled packages (very often libraries) and which are extremely discouraged. The problem that comes with that is that when a component gets updated to resolve a certain issue this problem is still in the twenty other copies of the same software. Imagine that for things like Heartbleed. It also consumes space, increases the build time and so on.
You should use the provided versions of those tools and libraries or modify them if that is required. All other components that are missing should be created as individual packages.
Hey,
So I've got 4 new addons built and in the process noticed that apr and apr-util are both not included as separate packages. I think that httpd makes use of them but provides the sources bundled which seems to go against what you recommend above. Like for instance, in Debian apr and apr-util are both provided as modular separate packages. So this is what I did while playing around. I built two addons for both apr and apr-util because subversion also has those as dependencies. I then specified --with-apr and --with-apr-util to my newly added apr and apr-util packages when ./configure'ing subversion. It all builds nicely and I'm just waiting on my clean/build to test packages on my live IPFire box.
Long story, but I'm curious about why apache makes used of bundled third party sources here, is this a special case that requires to go against what's encouraged?
For the record, I don't believe that something like subversion belongs to a nice trimmed down firewall OS, but I needed it for quick checkout of sources on my box and thought it might be occasionally useful.
Regards,
On Mon, 2014-07-07 at 16:49 +0800, Ghislain Hachey wrote:
On 7/6/14, 16:57, Michael Tremer wrote:
That's what we call bundled packages (very often libraries) and which are extremely discouraged. The problem that comes with that is that when a component gets updated to resolve a certain issue this problem is still in the twenty other copies of the same software. Imagine that for things like Heartbleed. It also consumes space, increases the build time and so on.
You should use the provided versions of those tools and libraries or modify them if that is required. All other components that are missing should be created as individual packages.
Hey,
So I've got 4 new addons built and in the process noticed that apr and apr-util are both not included as separate packages. I think that httpd makes use of them but provides the sources bundled which seems to go against what you recommend above. Like for instance, in Debian apr and apr-util are both provided as modular separate packages. So this is what I did while playing around. I built two addons for both apr and apr-util because subversion also has those as dependencies. I then specified --with-apr and --with-apr-util to my newly added apr and apr-util packages when ./configure'ing subversion. It all builds nicely and I'm just waiting on my clean/build to test packages on my live IPFire box.
Are you sure that subversion needs apr at runtime? It certainly needs the library, but I think that everything that is needed by subversion at runtime should be in the core system already.
We had a subversion package which was dropped in 2008, because the IPFire project switched to git and there was no one who wanted to maintain subversion any more.
http://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=a91ca65e82da80139f0e0...
Long story, but I'm curious about why apache makes used of bundled third party sources here, is this a special case that requires to go against what's encouraged?
APR is a part of apache. Some distributions simply create a subpackage for apr as it is usually not needed at runtime, just for development.
For the record, I don't believe that something like subversion belongs to a nice trimmed down firewall OS, but I needed it for quick checkout of sources on my box and thought it might be occasionally useful.
I would say we can include that as long as you are going to maintain the package.
The command line tools would certainly never hurt any one. Just make sure that mod_dav_svn is not loaded by apache by default and needs to be enabled in some way.
Regards,
-Michael
On 7/7/14, 17:22, Michael Tremer wrote:
Are you sure that subversion needs apr at runtime? It certainly needs the library, but I think that everything that is needed by subversion at runtime should be in the core system already.
We had a subversion package which was dropped in 2008, because the IPFire project switched to git and there was no one who wanted to maintain subversion any more.
http://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=a91ca65e82da80139f0e0...
Long story, but I'm curious about why apache makes used of bundled third party sources here, is this a special case that requires to go against what's encouraged?
APR is a part of apache. Some distributions simply create a subpackage for apr as it is usually not needed at runtime, just for development.
OK, that all makes sense. So I might as well just point the subversion ./configure script to use the ones in httpd sources for the build and be done with it. I don't mind maintaining the package (and other orphan packages that I may want to use from time to time). What does this usually entail? Just following the upstream project for bug and security fixes and re-building new sources when that happens?
-- GH
On Mon, 2014-07-07 at 17:50 +0800, Ghislain Hachey wrote:
On 7/7/14, 17:22, Michael Tremer wrote:
Are you sure that subversion needs apr at runtime? It certainly needs the library, but I think that everything that is needed by subversion at runtime should be in the core system already.
We had a subversion package which was dropped in 2008, because the IPFire project switched to git and there was no one who wanted to maintain subversion any more.
http://git.ipfire.org/?p=ipfire-2.x.git;a=commitdiff;h=a91ca65e82da80139f0e0...
Long story, but I'm curious about why apache makes used of bundled third party sources here, is this a special case that requires to go against what's encouraged?
APR is a part of apache. Some distributions simply create a subpackage for apr as it is usually not needed at runtime, just for development.
OK, that all makes sense. So I might as well just point the subversion ./configure script to use the ones in httpd sources for the build and be done with it. I don't mind maintaining the package (and other orphan packages that I may want to use from time to time). What does this usually entail? Just following the upstream project for bug and security fixes and re-building new sources when that happens?
Yes, it is basically all that. Not much work actually, but it needs to be done. If everyone does a little slice of it we will have a great distribution here.
Following upstream especially for security announcements, updating the package accordingly or every once in a while. It is not necessary to follow every single release, but it is nice to have a recent version on the system.
And if there are some users experiencing problems with the package, maybe give them some help. That does not contain explaining them how to use SVN for example, but if there is something that works on other systems but is broken on IPFire, it would be great looking into that.
What other packages/add-ons did you build?
-Michael
-- GH
Development mailing list Development@lists.ipfire.org http://lists.ipfire.org/mailman/listinfo/development
On 7/7/14, 18:35, Michael Tremer wrote:
What other packages/add-ons did you build?
* redis * apr and apr-util (which I will remove as discussed) * subversion * and I was going to try ntopng which is the one that packages several third party libs as discussed. The list of third party libs are: hiredis (c binding for redis), sqlite, json-c, LuaJIT, mongoose, patricia, redis-lua, rrdtool, zeromq. I have not yet been in touch with ntopng developers to see if it is safe to break down everything into several package.
-- GH
On 7/7/14, 18:35, Michael Tremer wrote:
What other packages/add-ons did you build?
Hey,
In the course of playing around with my subversion add-on I notice that the svn client will not work with http:// and https:// protocols which for me is a problem. I read that it must be built with serf (an HTTP Client lib). serf does not seem to be available in IPFire as is. I am not sure how to proceed with such a dep? Is it needed as an add-on? Is it only needed at build time? What is the best course of action here?
Also,
While I fight with getting all this built on ipfire-2.x I'd like to get started on preparing the same addons for ipfre-3.x. Is the process similar on ipfire-3.x? My understanding from your post is that this process is more modular, but I can't find any information on the wiki to get me started. Any basic pointers for me?
Thank you,
On Sun, 2014-07-06 at 16:51 +0800, Ghislain Hachey wrote:
Also,
While I fight with getting all this built on ipfire-2.x I'd like to get started on preparing the same addons for ipfre-3.x. Is the process similar on ipfire-3.x? My understanding from your post is that this process is more modular, but I can't find any information on the wiki to get me started. Any basic pointers for me?
The build process of IPFire 3 packages is very different. The is no such thing like a core system any more and everything is a package. pakfire is much more powerful in resolving dependencies and detecting conflicts between packages.
However, IPFire 3 is very far from production state so that it would not make sense by now to add additional software. But we would be very happy if you help us working on the base system.
Some pointers:
The GIT repository with all the sources: http://git.ipfire.org/?p=ipfire-3.x.git;a=summary
The Pakfire Build Service: https://pakfire.ipfire.org/
Some documentation about that: http://wiki.ipfire.org/devel/pakfire/start
And the packaging guidelines: http://wiki.ipfire.org/devel/packaging_guidelines/start
Thank you,
You're welcome.
-
Arne Fitzenreiter -
Ghislain Hachey -
Michael Tremer -
Sascha Kilian