Coverage testing in Python: combining the unittest and trace modules

Just testing is not enough, you need to know how much of your code you are actually testing to obtain any sort of confidence. In this article I show you my approach to combining the built-in unittest and trace modules to perform unit testing and coverage testing in one go, without introducing external dependencies.

Coverage tests in Python

If you want to check how much of your code is covered by your tests you can of course use any number of available modules (coverage.py is an excellent starting point) but this does introduce a dependency on external code, something you might want to avoid.

Checking coverage however, is just as simple as building unit test with Python's bundled modules. In the example below we use unittest to write our unit tests and use the less known trace module to tally any lines executed when we execute our test runner.

Example unit test

The code to test is a rather silly example of a function that has three branches. The bulk of the code is about the unit tests and the final (from line 28 onward) shows how we can execute the test runner (unittest.main()) within the trace framework.

def f(a):
 if a>0: 
  return a
 elif a==0:
  raise ValueError()
 return -a
# end of coverage

if __name__ == "__main__":
 import unittest
 import trace
 from os.path import dirname,abspath
 from collections import defaultdict
 from re import compile,match
 
 eoc=compile(r'^\s*# end of coverage')
 
 class TestFunction(unittest.TestCase):
  
  def test_int(self):
   self.assertEqual(3,f(3))
   self.assertEqual(-3,f(-3))
  
  def test_float(self):
   self.assertEqual(3.0,f(3.0))
   self.assertEqual(-3.0,f(-3.0))

 t=trace.Trace(count=1,trace=0)
 t.runfunc(unittest.main,exit=False)
 r=t.results()
 
 linecount = defaultdict(int)
 for line in r.counts:
  if line[0]==__file__:
   linecount[line[1]]=r.counts[line]
 
 with open(__file__) as f:
  for linenumber,line in enumerate(f,start=1):
   if eoc.match(line) : break 
   print("%02d %s"%(linecount[linenumber],line),end='')

Using the trace module

We first create an instance of a Trace object initialized with a trace parameter set to zero to prevent printing every line it encounters. (Full documentation is available on the Python site.) Then we invoke the runfunc() method with unittest.main as its first parameter. This will trace the test runner and collect the results in the Trace instance. The extra exit=False is passed to the test runner and will prevent it from stopping the program once the tests are done (after all, we still want to do something with the results from the trace).

The next step is to retrieve the results (line 30). These results are available as a dictionary which keys are (filename,linenumber) tuples and the corresponding values the number of times this line is executed. We convert this to a default dictionary for those lines that are present in the current source file (line 34). This dictionary is indexed by line numbers.

The final step is to iterate over the lines in the current source file until we encounter a line that signal the end of the tested code (line 39). We print those lines with the number of times they were executes. The result may look like this

..
----------------------------------------
Ran 2 tests in 0.031s

OK
00 def f(a):
04      if a>0:
02              return a
02      elif a==0:
00              raise ValueError()
02      return -a

The tests ran ok but as is immediately obvious from line 10 is that we missed an important branch that is not covered by our unit tests. Note that the line with the def statement is also not executed by our tests but this is to be expected as function definitions are only executed when the function is compiled, not when it is run.

Remarks

We do use more than just the document API of the trace module because the trace module as published has a major drawback: it not only collects statistics but only offers a public API that immediately prints those results. This not convenient when you want to postprocess the results or present them in a different way. We are therefore more or less forced to access the underlying data (the counts attribute in the object returned by the results() method.