unit testing in MATLAB
Likewise to the last post on programming style guidelines, this post also relates to the quest for beautiful code: “code that is more likely to be correct, understandable, sharable and maintainable” (Richard Johnson). However, while we did focus more on the “understandable” and “sharable” part last time, the focus this time will be more on “correctness” and “maintainability”: unit testing.
Whenever we write some code, of course, we would like to protect ourselves against unintended behavior and bugs in our program. This is where most beginners immediately think of errors, stopping the execution of the code. However, such errors are only one part of possible bugs in our code – actually, they are just the easy part. With errors, at least you know that something in your code has gone wrong. But how would we know, if our code just returns wrong results silently, without any indications through errors at run time?
A first ad-hoc approach that everybody is able to come up with himself is manually testing individual parts of the code. We plug in some values, and skip through the code line by line. Slightly more sophisticated, at the next step on this ladder – and it already took me quite some time to get there – one might discover the power of debugging interfaces (MATLAB, for example, has incredibly awesome debugging capabilities!). Still, however, we would like to have a more thought out approach for testing code, with higher automatism and better structure. And this is where we could make use of some of the experiences the computer science community already made. Kind of like keeping with the motto: “You must learn from the mistakes of others. You can’t possibly live long enough to make them all yourself.” (Sam Levenson)
One such approach to testing code that the computer science community did come up with is unit testing. In short, the idea is to decompose code into small functionalities, and test each of these functionalities for its own. Given such small units, it should be possible to come up with some exemplary input, where you are able to manually determine the output. Hence, you can write tests that check whether certain units do come up with the correct output for some given input. So far, this very much sounds identical to just testing code manually – so where’s the difference? The main point is, that you now keep your manually created test cases, and store them in a script, in order to be able to re-run them automatically whenever you make any changes to your code. This way, you can assure yourself, that any new changes to your code do not break some previously working functionality. And, there are a lot of positive additional side effects to this procedure as well:
- you are forced to modularize your code into smaller units of less complexity (as a side note: check out MATLAB’s M-Lint functionality to calculate the cyclomatic complexity, which is used to measure how complex a certain part of code is)
- you will end up with less duplicity in your code
- you are encouraged to more rigorous testing of your code through a more automated and user-friendly interface
At this point, however, I want to stop with my own short introduction to unit testing, and instead restrict myself to just pointing out some further resources from more sophisticated software developers. “If someone else already said it best, and you can’t top it, just steal from them”.
In 2009, Steve Eddins, a software development manager at the MathWorks, posted the xUnit framework to the MATLAB File exchange. xUnit is a free implementation of the unit testing approach in MATLAB, and comes with an extensive documentation. It has gained high popularity over the years, and probably still is the most used MATLAB unit testing framework so far. In addition to the documentation, Steve Eddins also did publish a paper called Automated Software Testing for Matlab, which is an excellent introduction and explanation of the most important features of xUnit.
Starting now, with the current MATLAB release of R2013a, MATLAB also ships with its own built-in implementation of the unit testing framework. This framework is build on object oriented programming, and you can easily gain some insight into its interface through the video on MATLAB Unit Testing Framework.
If you are indecisive about which of both frameworks to use, you might additionally check Steve Eddins’ blog entry on old and new software testing tools. Therein, he announces that he does “not plan to work on it anymore”, initiating the “retirement phase” of xUnit. Also, he mentions some additional functionality of the new framework: it entails “setup and teardown methods that get executed just once for all the methods in a test file, instead of being executed once for every individual test method”.
Finally, one last remark on unit testing in statistics in general. One major problem in statistical applications is that functions quite often entail random components. Hence, for a given input, you can not determine the output of the function up-front, because it depends on the exact state of the interior random number generator. For example, simulation routines, bootstrapping techniques or genetic optimization algorithms with random mutations all have stochastic components, and hence will not return one unique output. As far as I know, there does not exist one best solution for accounting for randomness, yet. However, I did stumble upon a paper called Automated Testing of Stochastic Systems: A Statistically Grounded Approach, which could be a good starting point in this field.
QI as content management:
Last but not least, a re-post of the link to the paper Automated Software Testing for Matlab. This time with my own annotations included, in order to allow for a fast overview of the paper the next time I look at it. This is something I always add primarily for myself, as I use the blog as kind of a content management system for myself, too. But who knows, maybe it will help someone else as well…