Testing of IPFire packages, and possibly automating testing

Testing of IPFire packages, and possibly automating testing

[Alf Høgemark]

[-- Attachment #1: Type: text/plain, Size: 3137 bytes --]

Hi

Judging on the mail traffic to the developer list, it seems to me that 
if I have a few hours to spare, I should rather spend them testing than
sending patches for updating packages and components of IPFire to the 
latest versions, since there seems to several people doing that already.

I've tried to find information on the wiki pages about testing, and what 
test steps should be done for each package, but I cannot find any 
information.

Do you have a check list of steps that should be tested when doing a 
version upgrade of a package, for example
install the package, test that the service (if any)  runs.
reboot ipfire, test that the service runs
uninstall, check that the service is stopped and all files are removed
etc, etc

Or is it up to each developer to do the testing deemed necessary ?

I am running IPFire at home, and do not want to do testing on that 
instance itself, so I would rather have to set up another virtual IPFire 
instance.
Do you have any description of a "typical system" (the number of network 
cards, include wifi or not etc) to use when doing testing ? Or is this 
so different from package to package that it is impossible to describe 
such a typical system.


For example, what testing would be needed to update the PHP version from 
5.3.x to 5.6.x ? This is a rather complex package, or the use of it can 
be very different from system to system.

Have there been discussions earlier about having a "Testing" subsection 
on the wiki, beneath the "Development" section, to guide people that 
want to contribute to testing parts of IPFire ?


Do you have any automatic testing done on IPFire for each version built ?


A bit accidentally, I stumbled across https://github.com/qca/boardfarm 
while looking the OpenSUSE 2016 videos.
That boardfarm tool to me seems like it could enable a long list of 
automatic tests for IPFire.
The tool is made for OpenWRT, but by reading the documentation and 
source code for an hour, it seems to me like this is something that 
could be very useful also in a IPFire context, assuming you do not 
already have a solution for automatic testing.

My thinking is to run IPFire virtually, and to use this boardfarm to 
test that IPFire instance, for example by making a test for each 
package, to check that it installs correctly, is running correctly, and 
can be uninstalled correctly.
Using the selenium part of the boardfarm, it seems like this could also 
be used for doing at least "smoke tests" of the whole IPFire web ui.
Also the rest of the IPFire seems to be testable using this boardfarm.

Do you have any initial comments to the boardfarm, and its' potential 
use to testing IPFire ?

I hope I have not bored you developers with this rather long email. 
Being a developer myself, I know that the testing part is not always the 
most exciting, but this boardfarm looks exciting to me, so I wanted to 
share it.


(Completely unrelated, I also had a look at 
https://events.opensuse.org/conference/oSC16/program/proposal/774#2, 
which is a czech high-preformance opensource router, interesting to see 
what others are doing)


Regards
Alf Høgemark

Re: Testing of IPFire packages, and possibly automating testing

[R. W. Rodolico]

[-- Attachment #1: Type: text/plain, Size: 3780 bytes --]

Alf,

We NEVER have enough testers. IMHO, the greatest deficiencies in the
project are testing and documentation. It appears we have some excellent
developers and package maintainers, but documentation and testing are
what we need.

Michael or someone will know more if there is a gap he needs covered in
developing or package maintenance, but if not, yes, help!!!

Rod

On 07/17/2016 07:29 AM, Alf Høgemark wrote:
> Hi
> 
> Judging on the mail traffic to the developer list, it seems to me that
> if I have a few hours to spare, I should rather spend them testing than
> sending patches for updating packages and components of IPFire to the
> latest versions, since there seems to several people doing that already.
> 
> I've tried to find information on the wiki pages about testing, and what
> test steps should be done for each package, but I cannot find any
> information.
> 
> Do you have a check list of steps that should be tested when doing a
> version upgrade of a package, for example
> install the package, test that the service (if any)  runs.
> reboot ipfire, test that the service runs
> uninstall, check that the service is stopped and all files are removed
> etc, etc
> 
> Or is it up to each developer to do the testing deemed necessary ?
> 
> I am running IPFire at home, and do not want to do testing on that
> instance itself, so I would rather have to set up another virtual IPFire
> instance.
> Do you have any description of a "typical system" (the number of network
> cards, include wifi or not etc) to use when doing testing ? Or is this
> so different from package to package that it is impossible to describe
> such a typical system.
> 
> 
> For example, what testing would be needed to update the PHP version from
> 5.3.x to 5.6.x ? This is a rather complex package, or the use of it can
> be very different from system to system.
> 
> Have there been discussions earlier about having a "Testing" subsection
> on the wiki, beneath the "Development" section, to guide people that
> want to contribute to testing parts of IPFire ?
> 
> 
> Do you have any automatic testing done on IPFire for each version built ?
> 
> 
> A bit accidentally, I stumbled across https://github.com/qca/boardfarm
> while looking the OpenSUSE 2016 videos.
> That boardfarm tool to me seems like it could enable a long list of
> automatic tests for IPFire.
> The tool is made for OpenWRT, but by reading the documentation and
> source code for an hour, it seems to me like this is something that
> could be very useful also in a IPFire context, assuming you do not
> already have a solution for automatic testing.
> 
> My thinking is to run IPFire virtually, and to use this boardfarm to
> test that IPFire instance, for example by making a test for each
> package, to check that it installs correctly, is running correctly, and
> can be uninstalled correctly.
> Using the selenium part of the boardfarm, it seems like this could also
> be used for doing at least "smoke tests" of the whole IPFire web ui.
> Also the rest of the IPFire seems to be testable using this boardfarm.
> 
> Do you have any initial comments to the boardfarm, and its' potential
> use to testing IPFire ?
> 
> I hope I have not bored you developers with this rather long email.
> Being a developer myself, I know that the testing part is not always the
> most exciting, but this boardfarm looks exciting to me, so I wanted to
> share it.
> 
> 
> (Completely unrelated, I also had a look at
> https://events.opensuse.org/conference/oSC16/program/proposal/774#2,
> which is a czech high-preformance opensource router, interesting to see
> what others are doing)
> 
> 
> Regards
> Alf Høgemark
> 

-- 
Rod Rodolico
Daily Data, Inc.
POB 140465
Dallas TX 75214-0465
214.827.2170
http://www.dailydata.net