Hi
Judging on the mail traffic to the developer list, it seems to me that if I have a few hours to spare, I should rather spend them testing than sending patches for updating packages and components of IPFire to the latest versions, since there seems to several people doing that already.
I've tried to find information on the wiki pages about testing, and what test steps should be done for each package, but I cannot find any information.
Do you have a check list of steps that should be tested when doing a version upgrade of a package, for example install the package, test that the service (if any) runs. reboot ipfire, test that the service runs uninstall, check that the service is stopped and all files are removed etc, etc
Or is it up to each developer to do the testing deemed necessary ?
I am running IPFire at home, and do not want to do testing on that instance itself, so I would rather have to set up another virtual IPFire instance. Do you have any description of a "typical system" (the number of network cards, include wifi or not etc) to use when doing testing ? Or is this so different from package to package that it is impossible to describe such a typical system.
For example, what testing would be needed to update the PHP version from 5.3.x to 5.6.x ? This is a rather complex package, or the use of it can be very different from system to system.
Have there been discussions earlier about having a "Testing" subsection on the wiki, beneath the "Development" section, to guide people that want to contribute to testing parts of IPFire ?
Do you have any automatic testing done on IPFire for each version built ?
A bit accidentally, I stumbled across https://github.com/qca/boardfarm while looking the OpenSUSE 2016 videos. That boardfarm tool to me seems like it could enable a long list of automatic tests for IPFire. The tool is made for OpenWRT, but by reading the documentation and source code for an hour, it seems to me like this is something that could be very useful also in a IPFire context, assuming you do not already have a solution for automatic testing.
My thinking is to run IPFire virtually, and to use this boardfarm to test that IPFire instance, for example by making a test for each package, to check that it installs correctly, is running correctly, and can be uninstalled correctly. Using the selenium part of the boardfarm, it seems like this could also be used for doing at least "smoke tests" of the whole IPFire web ui. Also the rest of the IPFire seems to be testable using this boardfarm.
Do you have any initial comments to the boardfarm, and its' potential use to testing IPFire ?
I hope I have not bored you developers with this rather long email. Being a developer myself, I know that the testing part is not always the most exciting, but this boardfarm looks exciting to me, so I wanted to share it.
(Completely unrelated, I also had a look at https://events.opensuse.org/conference/oSC16/program/proposal/774#2, which is a czech high-preformance opensource router, interesting to see what others are doing)
Regards Alf Høgemark
Alf,
We NEVER have enough testers. IMHO, the greatest deficiencies in the project are testing and documentation. It appears we have some excellent developers and package maintainers, but documentation and testing are what we need.
Michael or someone will know more if there is a gap he needs covered in developing or package maintenance, but if not, yes, help!!!
Rod
On 07/17/2016 07:29 AM, Alf Høgemark wrote:
Hi
Judging on the mail traffic to the developer list, it seems to me that if I have a few hours to spare, I should rather spend them testing than sending patches for updating packages and components of IPFire to the latest versions, since there seems to several people doing that already.
I've tried to find information on the wiki pages about testing, and what test steps should be done for each package, but I cannot find any information.
Do you have a check list of steps that should be tested when doing a version upgrade of a package, for example install the package, test that the service (if any) runs. reboot ipfire, test that the service runs uninstall, check that the service is stopped and all files are removed etc, etc
Or is it up to each developer to do the testing deemed necessary ?
I am running IPFire at home, and do not want to do testing on that instance itself, so I would rather have to set up another virtual IPFire instance. Do you have any description of a "typical system" (the number of network cards, include wifi or not etc) to use when doing testing ? Or is this so different from package to package that it is impossible to describe such a typical system.
For example, what testing would be needed to update the PHP version from 5.3.x to 5.6.x ? This is a rather complex package, or the use of it can be very different from system to system.
Have there been discussions earlier about having a "Testing" subsection on the wiki, beneath the "Development" section, to guide people that want to contribute to testing parts of IPFire ?
Do you have any automatic testing done on IPFire for each version built ?
A bit accidentally, I stumbled across https://github.com/qca/boardfarm while looking the OpenSUSE 2016 videos. That boardfarm tool to me seems like it could enable a long list of automatic tests for IPFire. The tool is made for OpenWRT, but by reading the documentation and source code for an hour, it seems to me like this is something that could be very useful also in a IPFire context, assuming you do not already have a solution for automatic testing.
My thinking is to run IPFire virtually, and to use this boardfarm to test that IPFire instance, for example by making a test for each package, to check that it installs correctly, is running correctly, and can be uninstalled correctly. Using the selenium part of the boardfarm, it seems like this could also be used for doing at least "smoke tests" of the whole IPFire web ui. Also the rest of the IPFire seems to be testable using this boardfarm.
Do you have any initial comments to the boardfarm, and its' potential use to testing IPFire ?
I hope I have not bored you developers with this rather long email. Being a developer myself, I know that the testing part is not always the most exciting, but this boardfarm looks exciting to me, so I wanted to share it.
(Completely unrelated, I also had a look at https://events.opensuse.org/conference/oSC16/program/proposal/774#2, which is a czech high-preformance opensource router, interesting to see what others are doing)
Regards Alf Høgemark
-
Alf Høgemark -
R. W. Rodolico