Integrating JUnit tests with HP/Mercury Quality Center

Part 1: initial analysis and annotation

A while back I did some work for a client who had two sets of test cases. On the one hand, they had hundreds of manual test cases documented in Quality Center. On the other hand, they had hundreds of automated test cases written using JUnit. I was given the task of correlating the two sets of test cases and determining to what extent they overlapped and, if possible, updating the test results in QC with the JUnit test case results.

This represented a significant burden for the client because:

1. They were not capable of determining direct correlation between the tests
2. Given the results of the JUnit tests, it would be a laborious task to update test results manually in Quality Center every week

My solution was to analyze the JUnit test cases and compare them with summaries of the QC test cases. Because I did not have sufficient domain knowledge (at first), I was not capable of properly determining correlation. However, I had the assistance of a domain expert who, although not an experienced Java developer, was sufficiently technical to be able to edit the source code and commit changes to subversion with a little help. I could interpret the source code and he could then identified the corresponding QC test case (or cases).

For the first stage, we would add an annotation to each JUnit test method: @QCTestCases

This was a simple annotation and looked something like this:

	@Target({ElementType.METHOD, ElementType.TYPE})
	@Retention(RetentionPolicy.RUNTIME)
	public @interface QCTestCases {

	    String[] covered() default "";
	    String[] related() default "";
	    
	}

This allowed us to add a list of covered test cases as well as a list of related test cases to each test method:

	@Test
	@QCTestCases(covered = { "QC-TEST-1" }, related = { "QC-TEST-2", "QC-TEST-3", "QC-TEST-4" }
	public void testSomething() {
		//implementation...
	}

My definition of “covered” was a test that definitely corresponded such that if the JUnit test fails, then the qc test case should also fail; “related” tests were a bit more ambiguious and would require further analysis.

In practice it turned out that related tests were an unknown bucket that with some refinement could be eliminated or tweaked to provide direct coverage, but it was a useful categorization at this early stage when we didn’t want to have too many false failures. In an ideal situation there would be a direct 1-to-1 mapping, but that’s seldom the case.

Once I had annotations in place, I thought it would be fairly straightforward to map the cases and report the results. Using reflection, I could get the test names and results, and then using my mapping to upload test results to QC using the OTA (Open Test Architecture) API, the HP/Mercury public interface to Quality Center.

The initial analysis was tedious work but everything else depended on getting it right. It turned out to take several iterations before we had accurate coverage and a workable solution. But the strategy was in place.

In my next post, I’ll describe how I used the annotations to get results for mapping test cases.