Why do your programs fail? There are several valid reasons: the program is being used in a way that you didn’t intend, there are faults in your development or production environment, or your client could have omitted key features in their requirements specification. Or, you could’ve made a mistake.
Whether it is a simple typo or a crudely designed algorithm, it is a problem that you have to solve. Between simply trusting your code will work, and mathematically proving that your code will work, you could test it. A test suite will diagnose these problems, allowing you to immediately fix them. Of course, testing is commonplace in the software industry among both hobbyists and professionals. Writing a successful test suite, however, is not trivial.
Usually, unit testing is the main topic when discussing software testing. Unit testing is simply testing each atomic ‘unit’ of your program, usually each method if you’re using an object-oriented programming language. Within unit testing, there are two major schools of thought that dictate the design of test suites: black-box testing, and glass box testing.
Black Box Testing
Black box testing derives all of the test cases from the specification of the program. It tests on what the program should do, without knowledge of the code structure of the program. Function specifications or signatures would dictate the tests. Test-driven development is a common example of black-box testing, where the tests are written before the actual program is. In this case, it is impossible to know the code structure of the program.
At a basic level, follow the following steps to write a competent black-box test suite:
Partition Input and Output values into Equivalence Classes
You will need to consider the expected input and output values, and sort them into equivalence classes. Equivalence classes are groups of input and output values that exhibit the same behaviour. For example, 1+1 and 100+100 produce different answers but exhibit the same behaviour: addition. Therefore, there is no need to write different tests for these cases. The same should go for each of your equivalence classes.
The next step is to write two sets of tests for each equivalence class: a set of typical tests and a set of boundary tests.
Typical Tests
Typical tests cover what you would expect, ‘typical’ cases depending on the unit, and it’s possible to input and output values. You should aim to test each possibility for the unit, including your error handling. Test everything that you think that a typical user would do or make use of. There should be equivalence classes for all of these, including typical errors that may occur.
Boundary Tests
Boundary tests test the extreme limits of your equivalence class, ensuring the unit can handle it. Overlooking boundary tests is a common mistake. What happens to your unit when it is tested with the minimum or maximum input values? Make sure to test on things like extremely large inputs, or input values of 0.
Glass Box Testing
Also known as white box testing, glass box test suites are written with the structure of the program in mind. The programmer that is writing the tests has intimate knowledge of the code that they are testing and writes the tests to accommodate this. This is often performed if a test suite is needed for an existing program that doesn’t have one.
In order to write a glass box testing suite, there is only one step: consider the structure of the code, and aim to test every segment of it. However, depending on the project or testing requirements, you may want to choose your tests based on one or more of the following strategies:
- Statement Coverage - execute every statement (line of code)
- Branch Coverage - each branch condition must be true and false at least once
- Path Coverage - execute all control-flow paths through the code
There are more strategies than these (though there is a lot of overlap), but these should show the basic methodology of glass box testing.
Limitations
Test suites do have limitations, which make the old-fashioned mathematical methods irreplaceable. Limitations in test suites are often related to how exhaustive they need to be, in order to avoid every possible issue that may arise. For a large project, it is infeasible to write an exhaustive test suite, as they will tend towards diminishing returns. It is just easier and cheaper to find and solve fringe errors when they occur, rather than testing for them.
Of course, both forms of testing will be essential to any critical, non-trivial project. In most cases, it will be beneficial to write multiple test suites, in order to most comprehensively test a program. A thorough test suite may improve your confidence that your code will work, but can never prove the absence of errors. Of course, for extremely critical projects, mathematically proving the code will always be necessary.