Testing of IPFire packages, and possibly automating testing

["Alf Høgemark" <alf@i100.no>]

[-- Attachment #1: Type: text/plain, Size: 3137 bytes --]

Hi

Judging on the mail traffic to the developer list, it seems to me that 
if I have a few hours to spare, I should rather spend them testing than
sending patches for updating packages and components of IPFire to the 
latest versions, since there seems to several people doing that already.

I've tried to find information on the wiki pages about testing, and what 
test steps should be done for each package, but I cannot find any 
information.

Do you have a check list of steps that should be tested when doing a 
version upgrade of a package, for example
install the package, test that the service (if any)  runs.
reboot ipfire, test that the service runs
uninstall, check that the service is stopped and all files are removed
etc, etc

Or is it up to each developer to do the testing deemed necessary ?

I am running IPFire at home, and do not want to do testing on that 
instance itself, so I would rather have to set up another virtual IPFire 
instance.
Do you have any description of a "typical system" (the number of network 
cards, include wifi or not etc) to use when doing testing ? Or is this 
so different from package to package that it is impossible to describe 
such a typical system.


For example, what testing would be needed to update the PHP version from 
5.3.x to 5.6.x ? This is a rather complex package, or the use of it can 
be very different from system to system.

Have there been discussions earlier about having a "Testing" subsection 
on the wiki, beneath the "Development" section, to guide people that 
want to contribute to testing parts of IPFire ?


Do you have any automatic testing done on IPFire for each version built ?


A bit accidentally, I stumbled across https://github.com/qca/boardfarm 
while looking the OpenSUSE 2016 videos.
That boardfarm tool to me seems like it could enable a long list of 
automatic tests for IPFire.
The tool is made for OpenWRT, but by reading the documentation and 
source code for an hour, it seems to me like this is something that 
could be very useful also in a IPFire context, assuming you do not 
already have a solution for automatic testing.

My thinking is to run IPFire virtually, and to use this boardfarm to 
test that IPFire instance, for example by making a test for each 
package, to check that it installs correctly, is running correctly, and 
can be uninstalled correctly.
Using the selenium part of the boardfarm, it seems like this could also 
be used for doing at least "smoke tests" of the whole IPFire web ui.
Also the rest of the IPFire seems to be testable using this boardfarm.

Do you have any initial comments to the boardfarm, and its' potential 
use to testing IPFire ?

I hope I have not bored you developers with this rather long email. 
Being a developer myself, I know that the testing part is not always the 
most exciting, but this boardfarm looks exciting to me, so I wanted to 
share it.


(Completely unrelated, I also had a look at 
https://events.opensuse.org/conference/oSC16/program/proposal/774#2, 
which is a czech high-preformance opensource router, interesting to see 
what others are doing)


Regards
Alf Høgemark