Integrating JUnit with HP Quality Center – part 2

Integrating JUnit tests with HP/Mercury Quality Center

Part 2: reporting annotation converage using a base class

In my previous post I talked about adding an annotation to JUnit test cases that identified corresponding manual test cases defined in Quality Center. In this post, I’ll describe how I used those annotations to create a coverage report by having my annotated test cases extend a base class.

Once the test classes were properly annotated, every unit test was made to extend a base class. (This only works with JUnit 4 because JUnit 3 requires extending junit.framework.TestCase). The base class uses reflection to get the test name and report coverage from the annotations.

public class TestBase {
	@Rule
	public TestName testName = new TestName();
	protected static String previousTestName;
	protected static boolean isFirstRunMethod; // this is to check for a class with more than one method

	protected static final Logger log = Logger.getLogger("qcCoverageReport");

	protected static final String COVERAGE_REPORT_FILENAME = "qcCoverageReport.csv";
	protected static final String COVERAGE_REPORT_DELIMITER = ",";
	protected static final boolean COVERAGE_REPORT_APPEND = true;


	@BeforeClass
	public static void init() {
		PropertyConfigurator.configure("log4j.properties");
		isFirstRunMethod = true;
	}

	@Before
	public static void setUp() {
		if (! executeTests()) {
		   fail("creating coverage report");
		}
	}

	@After
	public void tearDown() {
		printQCTestCaseCoverage();
		writeCoverageReport(buildCoverageReport());
		isFirstRunMethod = false;
		previousTestName = testName.getMethodName();
	}

	// this is a simple method that just writes test coverage to a log file
	private void printQCTestCaseCoverage() {
		try {
			Class clazz = Class.forName(this.getClass().getName());
			Method method = clazz.getMethod(testName.getMethodName());
			if (method.isAnnotationPresent(QCTestCases.class)) {
				log.info("Class [" + clazz.getName() + "] test method [" + method.getName() + "].");
				QCTestCases qcTestCases = method.getAnnotation(QCTestCases.class);
				for (String element : qcTestCases.covered()) {
					log.info("QC Test Cases Covered [" + element.toString() + "].");
				}
				for (String element : qcTestCases.related()) {
					log.info("QC Test Cases Related [" + element.toString() + "].");
				}
			}
		} catch (Throwable t) {
			t.printStackTrace(System.err);
		}
	}

	// this is a more complex method that builds a collection and eliminates duplicates 
	public StringBuilder buildCoverageReport() {
		StringBuilder coverage = new StringBuilder();

		// get test case information via reflection
		String packageName = this.getClass().getPackage().getName();
		String className = this.getClass().getSimpleName();
		String methodName = testName.getMethodName();
		Boolean isSameAsLastMethod = false;

		// see if it's the same test run again (e.g. parameterized)
		if (methodName.equals(previousTestName)) {
			isSameAsLastMethod = true;
		}

		// check whether this is the first test case for this class
		if (isFirstRunMethod && !isSameAsLastMethod) {
			// write package name in the 1st column
			coverage.append("\n");
			coverage.append(packageName);

			// write class name in 2nd column
			coverage.append("\n,");
			coverage.append(className);
		}

		if (!isSameAsLastMethod) {
			// write method name in 3rd column
			coverage.append("\n,,");
			coverage.append(methodName);

		for (String coveredTestCase : getCoveredQCTestCases()) {
			if (!coveredTestCase.isEmpty()) {
				// Write covered test cases in the 4th column
				coverage.append("\n,,,");
				coverage.append(coveredTestCase);

				// Write 'covered' in the 5th column
				coverage.append(",covered");
			}
		}

		for (String relatedTestCase : getRelatedQCTestCases()) {
			if (!relatedTestCase.isEmpty()) {
				// Write related test cases in the 4th column
				coverage.append("\n,,,");
				coverage.append(relatedTestCase);

				// Write 'related' in the 5th column
				coverage.append(",, related");
			}
		}
	}

		return coverage;
	}

	public List getCoveredQCTestCases() {
		List coveredTestCases = new ArrayList();

		try {
			Class clazz = Class.forName(this.getClass().getName());
			Method method = clazz.getMethod(testName.getMethodName());

			if (method.isAnnotationPresent(QCTestCases.class)) {
				QCTestCases qcTestCases = method.getAnnotation(QCTestCases.class);

				for (String testCase : qcTestCases.covered()) {
					coveredTestCases.add(testCase);
				}
			}
		} catch (ClassNotFoundException e) {
			e.printStackTrace(System.err);
		} catch (NoSuchMethodException e) {
			e.printStackTrace(System.err);
		}

		return coveredTestCases;
	}


	public List getRelatedQCTestCases() {
		// Identical to getCoveredQCTestCases except calling qcTestCases.related()
		// It could have been refactored into a common method
	}

	public boolean executeTests() {
		// Set this to false if you just want to generate a coverage report.
		// We actually determine this from test properties but that's not important to this example
		return false;
	}

	public void writeCoverageReport(StringBuilder coverageReport) {
		try {
			FileWriter writer = new FileWriter(COVERAGE_REPORT_FILENAME, COVERAGE_REPORT_APPEND);
			writer.append(coverageReport);
			writer.close();
		} catch (IOException e) {
			e.printStackTrace();
		}
	}
}

The @Rule annotation is a newer feature of JUnit 4. One built in rule is TestName which allows you to get the test name from inside a test case.

There are actually two ways to get test coverage.

The simpler method [printQCTestCaseCoverage] just writes to a log file after every test case executes. It outputs the test case name, and a list of covered and related test cases.

The more complex method [buildCoverageReport] compares the test with previous test methods and checks for mutiples in coverage to avoid duplication. It uses some ugly logic hackery to get there, and all this will actually end up refactored out, so just look at printQCTestCaseCoverage for the basics of using reflection to get the test case name and annotation.

You can now have your junit test cases extend TestBase and get a csv report of test coverage.

public class MyTest extends TestBase {

	@Test
	@QCTestCases(covered = { "QC-TEST-1", "QC-TEST-2" }, related = { "QC-TEST-3", "QC-TEST-4", "QC-TEST-5" })
	public void testSomething() {
		//implementation...
	}

	@Test
	@QCTestCases(covered = { "QC-TEST-6"} })
	public void testSomethingElse() {
		//implementation...
	}
}

This will generate a CSV report that looks like this [qcCoverageReport.csv]:

com.mycompany,MyTest,,,
,,testSomething,,
,,,QC-TEST-1, covered
,,,QC-TEST-2, covered
,,,QC-TEST-3, related
,,,QC-TEST-4, related
,,,QC-TEST-5, related
,,testSomethingElse,,
,,,QC-TEST-6,covered
,AnotherTest,,,
,,TestThis,QC-TEST-7,covered
,,TestThat,QC-TEST-8,covered
,,,QC-TEST-9,related

which ends up looking like this if you open it in Excel:

I could just as easily have included package, class, and method name on every line by eliminating some newlines. This is, cheap (hacky) report generation, but serves our purposes here. I might use a CSV library to handle things like properly escaping fields, etc. but by the time I got to that point, I had refactored the reporting completely out of the base class.

My next post will talk about how I went from reporting coverage to reporting results — which turned out to be tricker than I thought.