From mboxrd@z Thu Jan 1 00:00:00 1970 From: "R. W. Rodolico" To: development@lists.ipfire.org Subject: Re: Testing of IPFire packages, and possibly automating testing Date: Sun, 17 Jul 2016 14:46:40 -0500 Message-ID: <578BE0A0.7010401@dailydata.net> In-Reply-To: <578B7A25.1020006@i100.no> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============2852822610065475303==" List-Id: --===============2852822610065475303== Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit Alf, We NEVER have enough testers. IMHO, the greatest deficiencies in the project are testing and documentation. It appears we have some excellent developers and package maintainers, but documentation and testing are what we need. Michael or someone will know more if there is a gap he needs covered in developing or package maintenance, but if not, yes, help!!! Rod On 07/17/2016 07:29 AM, Alf Høgemark wrote: > Hi > > Judging on the mail traffic to the developer list, it seems to me that > if I have a few hours to spare, I should rather spend them testing than > sending patches for updating packages and components of IPFire to the > latest versions, since there seems to several people doing that already. > > I've tried to find information on the wiki pages about testing, and what > test steps should be done for each package, but I cannot find any > information. > > Do you have a check list of steps that should be tested when doing a > version upgrade of a package, for example > install the package, test that the service (if any) runs. > reboot ipfire, test that the service runs > uninstall, check that the service is stopped and all files are removed > etc, etc > > Or is it up to each developer to do the testing deemed necessary ? > > I am running IPFire at home, and do not want to do testing on that > instance itself, so I would rather have to set up another virtual IPFire > instance. > Do you have any description of a "typical system" (the number of network > cards, include wifi or not etc) to use when doing testing ? Or is this > so different from package to package that it is impossible to describe > such a typical system. > > > For example, what testing would be needed to update the PHP version from > 5.3.x to 5.6.x ? This is a rather complex package, or the use of it can > be very different from system to system. > > Have there been discussions earlier about having a "Testing" subsection > on the wiki, beneath the "Development" section, to guide people that > want to contribute to testing parts of IPFire ? > > > Do you have any automatic testing done on IPFire for each version built ? > > > A bit accidentally, I stumbled across https://github.com/qca/boardfarm > while looking the OpenSUSE 2016 videos. > That boardfarm tool to me seems like it could enable a long list of > automatic tests for IPFire. > The tool is made for OpenWRT, but by reading the documentation and > source code for an hour, it seems to me like this is something that > could be very useful also in a IPFire context, assuming you do not > already have a solution for automatic testing. > > My thinking is to run IPFire virtually, and to use this boardfarm to > test that IPFire instance, for example by making a test for each > package, to check that it installs correctly, is running correctly, and > can be uninstalled correctly. > Using the selenium part of the boardfarm, it seems like this could also > be used for doing at least "smoke tests" of the whole IPFire web ui. > Also the rest of the IPFire seems to be testable using this boardfarm. > > Do you have any initial comments to the boardfarm, and its' potential > use to testing IPFire ? > > I hope I have not bored you developers with this rather long email. > Being a developer myself, I know that the testing part is not always the > most exciting, but this boardfarm looks exciting to me, so I wanted to > share it. > > > (Completely unrelated, I also had a look at > https://events.opensuse.org/conference/oSC16/program/proposal/774#2, > which is a czech high-preformance opensource router, interesting to see > what others are doing) > > > Regards > Alf Høgemark > -- Rod Rodolico Daily Data, Inc. POB 140465 Dallas TX 75214-0465 214.827.2170 http://www.dailydata.net --===============2852822610065475303==--