Some code coverage (unit test coverage) tools:

EMMA - open source

Cobertura - open source

Clover - Atlassian

Hansel & Gretel - only found info on an IBM developerworks article

Quilt - open source

Jester - open source

Jester takes a very interesting approach. It actually changes to code and then sees if your tests break. For instance it might change: if (x>y) to if (false)

All of the above target Java. What about for other languages like Perl, PHP, Python, and Ruby? What about for Smalltalk?

PHPCoverage - Spikesource

(here is an interesting list of tools for PHP)

rcov - for Ruby

http://www.semdesigns.com/products/testcoverage/PHPTestCoverage.html

heckle is a Ruby version of Jester.  Link here. Another link to an article on heckle

Of course there’s controversy about the value of code coverage tools, but really it’s an issue of misusing them. They are useful, and they give people something to aim for. A more interesting idea is a “functionality” coverage tool — which would have to be more manually built. An interesting article mentions rspec but that’s not really what I meant, though still an interesting idea.

A requirements coverage matrix shouldn’t be a crutch any more than a code coverage report, but the combination could be powerful.

It’s just adding a slider control to rotate an image, based on the first example I saw, but the exercise of looking through the API docs and figuring out how to create an event handler helped me understand what was going on and gave me confidence about being able to learn on the fly.

See it here on the Fluffy QA Site: Flash Source

<?xml version="1.0" encoding="utf-8"?>
<mx:Application xmlns:mx="http://www.adobe.com/2006/mxml" layout="absolute">
    <mx:Image id="shell" source="@Embed('C:/users/Aaron/Pictures/shell2.png')" height="100" top="225" left="208">
    <mx:Script>
        <![CDATA[
            import mx.events.SliderEvent;
            import mx.controls.Text;

            private function sliderHandler(event:SliderEvent) :void {
                degrees.text = event.value.toString();
                shell.rotation = event.value;
            }

        ]]>
    </mx:Script>
    <mx:filters>
      <mx:DropShadowFilter />
    </mx:filters>
  </mx:Image>
  <mx:HSlider x="25" y="42" id="slider" minimum="-360" maximum="360" snapInterval="10" tickInterval="360" change="sliderHandler(event)">

  </mx:HSlider>
  <mx:Label x="208" y="42" text="degrees:" id="lbl_degrees"/>
  <mx:TextInput x="271" y="40" width="45" id="degrees"/>
  <mx:Text x="25" y="10" text="Rotator" fontSize="16"/>
</mx:Application>

Now I’ll go read the manual.

Tinderbox is the Mozilla project build tool. It’s designed to run builds and tests on multiple environments.

Here are some starter links:

http://www.mozilla.org/tinderbox.html

http://wiki.mozilla.org/Tinderbox

http://www.mozilla.org/projects/tinderbox/

http://www.johnkeiser.com/mozilla/tbox3.html

http://wiki.mozilla.org/Tinderbox:Tbox3_Setup

I don’t want to have a Java Appserver running on a QA site. It takes too much memory and isn’t stable enough. Especially if it’s running on VM with limited memory and CPU. So I want something that can run as a CGI or FastCGI or apache module (mod_perl, mod_php, mod_python).

So the field is narrowed down quite a bit:

Tinderbox

Tinderbox is Mozilla’s build system. What I know about it is your central server outsources the build to a “tinderbox” which I assume can also run tests. It’s written in perl. I don’t know how up to date it is, but the idea of a collection of VMs running tests for a project is appealing. Especially if the build/test machines aren’t the same as the project server. If tinderbox has this built in, it goes to the top of the list. Anything else, I’d have to cobble something together.

Buildbot

Buildbot is written in python. I think mod_python is stable enough, but if not, CherryPy could probably run it. This opens up remoting potential as well, but I’m going to guess it doesn’t have the distributed nature I’m hoping for from Tinderbox. Since I haven’t heard much about Buildbot, I’m skeptical it will have all the features I might want.

Cerberus

I saw one review that said Cerberus doesn’t have a web interface. That might not be a deal breaker, but it makes it less likely to get a quick trial. Maybe Cerberus + CI::Reporter and ant JUnitHTMLReports would work. I’m also uncertain if Cerberus has a daemon. Somewhere it said it runs via cron, but that might be out of date. I think Luntbuild might do something similar. That might be acceptable, have cron run every 10 minutes and check for checkins and non-blocking. Or instead of cron-triggered have it checkin triggered via a SVN hook. But that makes reporting a bit more difficult, since something would apparently have to poll for the cerberus reports as well.

I’d love to hear any feedback.  Corrections, implementations, or other tools.

Here’s a link to Ruby CI apps:

http://www.degrunt.net/articles/2006/08/27/cerberus-continious-integration-for-rails/

• Cerberus
• Damage Control - defunct?
• CruiseControl.rb
• Continuous Builder

More links:

http://www.degrunt.net/articles/2006/08/27/cerberus-continious-integration-for-rails/

http://www.cuberick.com/2006/09/continuous-integration-for-ruby.html

http://confluence.public.thoughtworks.org/display/CC/CI+Feature+Matrix

http://www.paperplanes.de/archives/2007/8/22/continuous_integration_for_ruby_and/http://www.paperplanes.de/archives/2007/8/22/continuous_integration_for_ruby_and/

http://rossniemi.files.wordpress.com/2007/03/ci-ruby-on-rails-1_0.pdf

Also look at CI::Reporter to convert Rake output Rspec and Test::Unit to a format Ant’s JUnitReport can handle.  That opens up potential use of the Java CI servers and CruiseControl.NET

http://blog.nicksieger.com/articles/2007/01/06/continuous-integration-goodness-tm-for-your-ruby-project

http://blog.huikau.com/2008/01/09/jruby-ruby-continuous-integration-with-hudson/

Here’s a link to a large matrix of CI tools:

http://confluence.public.thoughtworks.org/display/CC/CI+Feature+Matrix

Here’s my own list with basic impressions

• CruiseControl - a bit old
• Continuum - Apache
• Luntbuild - Dead simple, but not that intuitive or extendible. Usually end up cloning and tweak
• Hudson - Looks very interst
• Anthill - not open source?
• Bamboo - Atlassian
• Teamcity - Intellij
• CruiseControl.NET - .NET based, I hear it’s much more featureful than regular cruisecontrol.

Others to look at:

• Tinderbox - Mozilla
• Pulse
• BuildBot - Python
• Cerberus - Ruby based

When I finally got bugzilla passing checksetup, I go to http://localhost/bugzilla and get plain text. Obviously not executing CGI.

Add the following to httpd.conf:

<Directory C:/Apache/httpd/2.2/htdocs/bugzilla>
AddHandler cgi-script .cgi
Options ExecCGI
AllowOverride Limit
DirectoryIndex index.cgi
</Directory>

-

Then I get a 500

Internal Server Error

with the following in the Apache log file:

[error] [client 127.0.0.1] (OS 3)The system cannot find the path specified. : couldn’t spawn child process: C:/Apache/httpd/2.2/htdocs/bugzilla/index.cgi

A quick search turns up this:

http://www.nabble.com/Internal-Server-Error-td14693066.html

You can either set perl to be the .cgi handler in the registry and set the directive

ScriptInterpreterSource register

Alternately, install perl where it is expected by index.cgi

: #!/usr/bin/perl -wT

or edit the script to expect perl in the proper location

C:/Perl/bin/perl.exe

or just #!perl

I think last time (on my laptop) I copied C:/Perl/bin/perl.exe to C:/usr/bin/perl.exe

This looks interesting:

http://oss.segetech.com/bugzilla-svn-wiki.html

It sounds a lot like part of my idea for a QA site.

Allows bug reports, wiki documents, and commit messages to reference each other, by creating hyperlinks based on the identifiers:

• BugID:
• VCS:
• Wiki:

So a wiki page can reference a bug report and vice versa, a commit log can reference bug numbers or requirements, etc. and it’s all recorded in a mailing list (mailman) archive.

I’d work on smoother integration, as well as integration with a task application, and tests.

I’m installing Bugzilla (v 3.0.4) on Windows again, and I must say it’s going very nicely.

I installed ActivePerl 5.8.8 build 822 and added the theoryx PPM repository:

ppm repo add theory58S http://theoryx5.uwinnipeg.ca/ppms

ppm install TimeDate + 2.22
ppm install Email-Send + 2.185
ppm install Template-Toolkit +v2.19
ppm install Email-MIME-Modifier +1.442

ppm install DBD-Pg + 1.45
ppm install DBD-mysql +4.005

ppm install GDGraph + 1.4308
ppm install Template-GD + 2.66
ppm install Chart + 2.3
ppm install XML-Twig + 3.32
ppm install MIME-tools + 5.4.20
ppm install PatchReader + 0.9.5
ppm install perl-ldap + 0.34
ppm install HTML-Scrubber + 0.88
ppm install Email-MIME-Attachment-Stripper + 1.313
ppm install Email-Reply + 1.202

I don’t need all these, but I thought I’d see how the install worked on each.

None of them had any trouble. DBD-mysql didn’t have any upgrade trouble, Chart didn’t need a custom install, template-toolkit didn’t choke on dependencies — these are among the issues I remember in the past.

Thanks alot to the Theoryx folks at the University of Winnipeg.

PerlMagick is included with ImageMagick these days, so I installed ImageMagick 6.4.2-1 O16 but it’s PerlMagick was for ActivePerl 5.10.[something] and I couldn’t find older versions on their site. If deliberate, I understand the motive, but such draconian measures hurt more than they help.

Anyway, another Google search turned up the following link:

http://www.perlmonks.org/?node_id=451710

which led to:

http://www.bribes.org/perl/ppmdir.html

so I typed:

ppm install http://www.bribes.org/perl/ppm/Image-Magick.pp

This installed PerlMagick successfully, at least to the point of passing checksetup.pl and a simple script that just looks like:

use Image::Magick;

So thanks again Theoryx, thanks Google, thanks PerlMonks, thanks Bribes.org, thanks ImageMagick, thanks Netscape, thanks Bugzilla team, thanks Larry & co, thanks Linus and RMS, Tim B-L, DARPA, & etc.

I’m just writing to mention a new project I’m working on, “Fluffy”.  It’s a research project looking at Flex, Flash, ActionScript, etc.  and looking at testing tools and strategies.

I set up a Fluffy QA site with all the regular tools: subversion, bugzilla, projectpier, trac with  blogs, forum, and wiki.

I’m using a wordpress-mu blog as kind of a project journal, but I’m not happy with it.  That’s one project I’d really like to work on.