Important Marc Clifton Testing Types/Patterns
The next version of Microsoft's Visual Studio will include tools to automate refactoring. Tools that automate unit test generation would address some of the issues concerning maintenance and cost.
However, to achieve this acceptance, unit testing must be formalized so that it becomes a real engineering discipline rather than an ad hoc approach that relies on the capabilities of the programmer. Many people balk at the idea of writing the unit test first--it places them in the uncomfortable position of having to do up front design work.
There is no formal unit test engineering discipline.
Two things are needed--a formalization of unit testing by establishing unit test patterns, and the early adoption of object oriented design patterns in the developing application to specifically target the needs of unit testing.
One of the goals is a tool suite that can automatically generate unit tests, both as a reverse and forward engineering process. With the latter, it should be possible to generate the method stubs for the code under test.
Patterns
The patterns that I have identified so far can be loosely categorized as:
pass/fail patterns
collection management patterns
data driven patterns
performance patterns
process patterns
simulation patterns
multithreading patterns
stress test patterns
These are broad brush strokes.
Pass/Fail Patterns
These patterns are deceptive in what they tell you about the code.
The Simple-Test Pattern [yes]
When a unit test passes a simple test, all it does is tell me that the code under test will work if I give it exactly the same input as the unit test. A unit test that exercises an error trap is similar--it only tells me that, given the same condition as the unit test, the code will correctly trap the error. In both cases, I have no confidence that the code will work correctly with any other set of conditions, nor that it will correctly trap errors under any other error conditions. This really just basic logic. However, on these grounds you can hear a lot of people shouting "it passed!" as all the nodes on the unit test tree turn green.
The Parameter-Range Pattern [Yes]
The Parameter-Range pattern does this by feeding the Code-Path pattern with more than a single parameter set.
Data Driven Test Patterns
Constructing Parameter-Range unit tests is doable for certain kinds of testing, but it becomes inefficient and complicated to test at a piece of code with a complex set of permutations generated by the unit test itself. The data driven test patterns reduce this complexity by separating the test data from the test. The test data can now be generated (which in itself might be a time consuming task) and modified independent of the test.
The Simple-Test-Data Pattern
In the simplest case, a set of test data is iterated through to test the code and a straightforward result (either pass or fail) is expected. Computing the result can be done in the unit test itself or can be supplied with the data set. Variances in the result are not permitted. Examples of this kind of of Simple-Test-Data pattern include checksum calculations, mathematical algorithms, and simple business math calculations. More complex examples include encryption algorithms and lossless encoding or compression algorithms.
The Data-Transformation-Test Pattern
Data Transaction Patterns
Data transaction patterns are a start at embracing the issues of data persistence and communication. More on this topic is discussed under "Simulation Patterns". Also, these patterns intentionally omit stress testing, for example, loading on the server. This will be discussed under "Stress-Test Patterns".
The Simple-Data-I/O Pattern
This is a simple data transaction pattern, doing little more than verifying the read/write functions of the service. It may be coupled with the Simple-Test-Data pattern so that a set of data can be handed to the service and read back, making the transaction tests a little bit more robust.
The Constraint-Data Pattern
The Constraint-Data pattern adds robustness to the Simple-Data-I/O pattern by testing more aspects of the service and any rules that the service may incorporate. Constraints typically include:
can be null
must be unique
default value
foreign key relationship
cascade on update
cascade on delete
As the diagram illustrates, these constraints are modeled after those typically found in a database service and are "write" oriented. This unit test is really oriented in verifying the service implementation itself, whether a DB schema, web service, or other model that uses constraints to improve the integrity of the data.
The Rollback Pattern
The rollback pattern is an adjunct to the other transaction testing patterns. While unit tests are supposed to be executed without regard to order, this poses a problem when working with a database or other persistent storage service. One unit test may alter the dataset causing another unit test to inappropriately fail. Most transactional unit tests should incorporate the ability to rollback the dataset to a known state. This may also require setting the dataset into a known state at the beginning of the unit test. For performance reasons, it is probably better to configure the dataset to a known state at the beginning of the test suite rather than in each test and use the service's rollback function to restore that state for each test (assuming the service provides rollback capability).
Collection Management Patterns
A lot of what applications do is manage collections of information. While there are a variety of collections available to the programmer, it is important to verify (and thus document) that the code is using the correct collection. This affects ordering and constraints.
The Collection-Order Pattern
This is a simple pattern that verifies the expected results when given an unordered list. The test validates that the result is as expected:
unordered
ordered
same sequence as input
This provides the implementer with crucial information as to how the container is expected to manage the collection.
The Collection-Constraint Pattern
This pattern verifies that the container handles constraint violations: null values and inserting duplicate keys. This pattern typically applies only to key-value pair collections.
The Collection-Indexing Pattern
The indexing tests verify and document the indexing methods that the collection container must support--by index and/or by key. In addition, they verify that update and delete transactions that utilize indexing are working properly and are protected against missing indexes.
Performance Patterns
Unit testing should not just be concerned with function but also with form. How efficiently does the code under test perform its function? How fast? How much memory does it use? Does it trade off data insertion for data retrieval effectively? Does it free up resources correctly? These are all things that are under the purview of unit testing. By including performance patterns in the unit test, the implementer has a goal to reach, which results in better code, a better application, and a happier customer.
Process Patterns
The Process-Sequence Pattern
This pattern verifies the expected behavior when the code is performed in sequence, and it validates that problems when code is executed out of sequence are properly trapped. The Process-Sequence pattern also applies to the Data-Transaction pattern--rather than forcing a rollback, resetting the dataset, or loading in a completely new dataset, a process can build on the work of the previous step, improving performance and maintainability of the unit test structure.
Simulation Patterns
Mock objects can come to the rescue by simulating the database, web service, user event, connection, and/or hardware with which the code is transacting. Mock objects also have the ability to create failure conditions that are very difficult to reproduce in the real world--a lossy connection, a slow server, a failed network hub, etc. However, to properly use mock objects the code must make use of certain factory patterns to instantiate the correct instance--either the real thing or the simulation. All too often I have seen code that creates a database connection and fires off an SQL statement to a database, all embedded in the presentation or business layer! This kind of code makes it impossible to simulate without all the supporting systems--a preconfigured database, a database server, a connection to the database, etc. Furthermore, testing the result of the data transaction requires another transaction, creating another failure point. As much as possible, a unit test should not in itself be subject to failures outside of the code it is trying to test.
Mock-Object Pattern
In order to properly use mock objects, a factory pattern must be used to instantiate the service connection, and a base class must be used so that all interactions with the service can be managed using virtual methods. (Yes, alternatively, Aspect Oriented Programming practices can be used to establish a pointcut, but AOP is not available in many languages).
To achieve this construct, a certain amount of foresight and discipline is needed in the coding process. Classes need to be abstracted, objects must be constructed in factories rather than directly instantiated in code, facades and bridges need to be used to support abstraction, and data transactions need to be extracted from the presentation and business layers. These are good programming practices to begin with and result in a more flexible and modular implementation.
The Service-Simulation Pattern
This test simulates the connection and I/O methods of a service. In addition to simulating an existing service, this pattern is useful when developing large applications in which functional pieces are yet to be implemented.
The Bit-Error-Simulation Pattern
I have only used this pattern in limited applications such as simulating bit errors induced by rain-fade in satellite communications. However, it is important to at least consider where errors are going to be handled in the data stream--are they handled by the transport layer or by higher level code? If you're writing a transport layer, then this is a very relevant test pattern.
The Component-Simulation Pattern
In this pattern, the mock object simulates a component failure, such as a network cable, hub, or other device. After a suitable time, the mock object can do a variety of things:
throw an exception
return incomplete or completely missing data
return a "timeout' error
Again, this unit test documents that the code under test needs to handle these conditions.
Presentation Layer Patterns
