Table of Contents
QTest has been released under the terms of Version
2.0 of the Artistic License, a copy of which appears in the
file Artistic
in the source distribution.
QTest was originally created in 1993 as part of BCS, the Baseline Configuration System, which has since become inactive. I would like to acknowledge my previous employers Apex CoVantage and Argon ST for allowing me to republish modifications that I made to QTest on “company time” while under their employment.
QTest is implemented in Perl. It doesn't use any modules that aren't distributed with Perl. QTest requires Perl version 5.8 or newer. It also requires GNU diff. As long as these requirements are satisfied, it should work fine on any UNIX-like system.
To use QTest on Windows, you can either use Cygwin (which is really more like a UNIX environment anyway), or, starting with QTest version 1.4, you may use ActiveState Perl along with MSYS (from the MinGW project). Although QTest itself passes its own test suite in this environment, there are a number of caveats to keep in mind. See Chapter 3, Using QTest on Windows for details.
To use QTest, extract the distribution to the location of your
choice and make sure that qtest-driver, located
in the bin
directory of the QTest
distribution, is in your path. Alternatively, on systems that
support symbolic links, you may put a symbolic link to
qtest-driver in your path. The
qtest-driver program looks for the
TestDriver.pm
module in the
module
directory of the QTest distribution.
[1]
You don't have to take any specific action to get
TestDriver.pm
in your path:
qtest-driver does this automatically when it
invokes your test suites.
If you are going to use QTest's coverage system (see Chapter 7, Automated Test Coverage), you will need to ensure that you have a suitable implementation of the coverage call available in your development environment. Sample implementations are provided for several programming languages.
If you would like to embed QTest in your own application's source
tree, it is sufficient to create a directory with
bin
and module
directories. Copy bin/qtest-driver
into
bin
and
module/TestDriver.pm
into
module
. These are the only files required to
actually run qtest-driver. The
qtest-driver program looks for
TestDriver.pm
in
../module
relative to itself. If you use
coverage from your test suite, you may wish to copy some of the
coverage implementations from the QTC
directory as well. You can place those anywhere as long as your
application knows where to find them. You should also include some
indication of QTest's licensing terms and some acknowledgment that
you are using it.
[1] The qtest-driver command finds the grandparent directory of the absolute path to itself and treats that as the top of the QTest distribution. It looks for the module directory there.
Table of Contents
In this chapter, we describe QTest's basic functionality and describe how to create a test suite.
There are a variety of approaches to automated software testing, each of which has its own advantages and disadvantages. The QTest automated test framework supports the approach of designing for testability and subsequent creation of what might be called self-testing software. When using this approach, deciding how a specific area of functionality will be tested is central to the design. Every requirement and every piece of code must be testable, and the way in which it will be tested must be planned before any implementation is done.
Applications or modules implemented in this fashion will often have an embedded test mode in which the application itself or an external driver that “lives” with the software and is closely linked with it may exercise the code in a preset way, comparing the software's output with an established baseline. The software may contain code whose sole purpose is to assist with the testing process. This type of testing is performed largely by the developer. Test suites are coded incrementally at the same time as the software is being written. As defects in the software are found and corrected, the test suite is modified to exercise the defect before the problem is fixed, thus ensuring that the problem stays fixed. After all, the fact that the test suite did not enable the bug to be caught in the first place can well be considered a bug in the test suite!
This form of testing is complementary to the more traditional software testing that may be performed manually by a testing team or automatically using external tools to play back sequences of prerecorded events. It is also complementary to the normal Alpha and Beta testing cycles that should precede most software releases. All of these types of testing should be used together to ensure maximum software quality.
An important aspect of QTest's philosophy is that as many tests as possible should be able to be boiled down comparing generated text with known text. This approach is very powerful: designing your software in a way that makes it this easy to test is generally good for the software as well as for the testing process. Software that is easier to test is also easier to understand and maintain, and simple but thorough test cases are an invaluable tool to new programmers who are trying to become familiar with the software. This approach can be applied to a wide range of software including GUIs, multithreaded software, distributed software, and others as well. It's all about properly decomposing the problem and providing visibility into the internals of a system.
There are some types of testing that do not lend themselves to this approach. For example, if one has to determine whether an audio stream is intelligible, there is no substitute for having someone actually listen to the stream. Likewise, usability testing for graphical interfaces must be done by human operators. The challenge, then, is to decompose the problem and organize the software in such a way that the vast majority of testing can be automated, and the part of the testing that must be done manually is minimal and simplistic. Figuring out how to structure code and testing to support this goal is why writing good test suites is an art as well as a science. In this document, we focus primarily on the science: we describe the structure of the test suite, the features of the test framework, how to organize test suites, etc. We will also include some discussion in the form of examples and anecdotes that should help to provide insights into the art portion as well, though there is no substitute for experience when it comes to creating good test suites.
The QTest framework essentially gives you the tools to create what amounts to a script that runs your software in a particular way and examines its output. If you would be able to write a script to test your code, then you will be able to write a QTest-based test suite. The QTest framework invokes your software from the command line and studies the output that it creates. It may be complementary to tools such as cppunit or junit but does not require anything like that to operate.
When we say that a test suite passes, we must be very clear about exactly what this means. A good test suite must have the property that an overall “pass” is a very strong indicator of correctly working software. The overall pass condition must be defined in such a way that any unexpected error condition in the application or in the test suite itself will cause the test suite to fail. We define a test suite as passing when all of the following conditions are met:
Each individual test case passes.
The expected number of test cases is run.
The test driver itself terminates normally.
An individual test case is considered to pass when its output and exit status match those expected by the test suite.
Notice that the conditions described above are stronger than just saying that the test suite passes when all individual test cases pass. If there is a logic error in the test suite itself that causes a test case to be skipped entirely, this should make the test suite fail. Likewise, if the test driver crashes part way through but all tests up to that point have passed, the test suite should fail. By requiring the test driver to terminate normally after executing the correct number of test cases, we add some extra security to the test suite.
In the context of a test suite in this framework, a test case is simply the comparison of an input to an output. The input may be a fixed string, the contents of a file, or the output of a program. As part of the input specification, any of the above may be passed through a specified filter. The output may either be a fixed string, a file, or a perl-compatible regular expression that must match the input. If the input is the output of running a program, the expected output must also include the expected exit status of the program. The simple comparison of input and output strings may also be modified by telling the test system to normalize newlines and/or whitespace. Normally, a test case is expected to pass. There are some situations, however, in which it may be desirable to mark a test case as being known to fail. Examples may include test cases that exercise known bugs or regressions that have not yet been fixed, place-holder test cases for features that have not yet been completed, or test cases that are conditional upon some optional component of the system or some platform-specific functionality. If a test case is marked as expected to fail, the test driver requires that test case to fail in order for the overall test suite to pass. This alleviates the need to do things like “comment out” problematic test cases that you're not going to fix right away. An expected failure serves as a reminder that there is still something to do but is less strong than leaving an actual failing test case in the suite since an overall test suite passing may be a prerequisite to some other operation such as making a release.
A coverage condition is simply a situation inside of the software that must happen. Analysis of coverage conditions as included in this test framework should not be confused with coverage analysis as performed by standard code coverage tools. This framework is concerned with checking for specific programmer-specified conditions. Essentially, the developer writes a line of code that says, “This condition just happened,” and that line of code logs this fact to a file that is analyzed by the test framework itself. The test coverage system adds still further security to the test suite by ensuring not just that we have seen the correct number of test cases, but also that they have actually exercised the code as the developer intended. For further detail about this test coverage feature, please see Chapter 7, Automated Test Coverage.
A QTest test suite is a perl script whose name ends with the
extension .test
. A template for a QTest test
suite may be found in the file template
in
the misc
directory of the QTest source or
installation directory. The test framework invokes the perl
script with its include path set such that it can use the
statement require TestDriver;
to include the
TestDriver
object. The
TestDriver
object is a perl module that
includes several public methods:
notify
notify the user by printing a string
emphasize
notify the user by printing a string with emphasis
prompt
read input from the user by specifying a prompt, an override environment variable, and a default value
get_start_dir
return the directory from which the test suite was originally invoked
runtest
run an actual test case
report
report that a test suite has concluded, and specify the number of test cases that should have been run
The purpose of reporting the number of test cases with
report
is to perform an extra check to make
sure that the test suite itself didn't have a logic error that
caused some test cases to be skipped. The argument to
report
should therefore be a hard-coded
number or a number computed only from static features in the test
suite. It should not be a number that is counted up during the
process of running the test suite. Computing this number as a
side effect of running test cases would defeat the purpose of the
number. For example, if the test suite consists of an array of
test cases, and the test suite code iterates through that loop
and calls runtest
twice for each element, it
would be reasonable to pass an expression that calculated twice
the size of the array as an argument to
report
, but it would not be appropriate to
have a variable that is incremented each time
runtest
is called and then passed to
report
.
For details on how to use runtest
itself,
please see the extensive comments in
module/TestDriver.pm
in the test framework's
source or installation directory. The
TestDriver.pm
source code contains comments
on how to use all the public methods. To locate
TestDriver.pm
on your system, run
qtest-driver --print-path. This prints the
top-level directory of the QTest installation. The
TestDriver.pm
file can be found in the
module
directory.
The intended mode of invoking a program's test suite is that the
check or test target of the
build system should be run. The build system should enable support
for the QTest framework if a qtest
directory
exists. Each file in the qtest
directory that
ends with the .test
extension is expected to
be a test suite. The TESTS
environment variable may
be set to a space-separated list of words to specify which tests
should be run. (TESTS
="one two"
would result in qtest/one.test
and
qtest/two.test
being run.) Additionally,
QTest sets the IN_TESTSUITE
environment variable to
1
while running test suites. This can be useful
for programs that have to behave slightly differently in a test
suite, such as suppressing timestamps, simulating failure
conditions, etc.
Note that the qtest
directory contains the
test suites and any ancillary files required by the test suites
including data, filter scripts, etc. The code being tested does
not live in the qtest
directory. If set up
properly, the test system invokes the test driver so that all
programs being tested are at the beginning of the path.
If you are invoking programs by hard-coding relative paths to them, particularly if those paths contain the names of platform-specific object file directories, you are probably doing something wrong, or there is a problem with the way the test system is configured.
It is very important when writing test suites to be sure to avoid putting things in your test output files such as user names, absolute paths names, etc. Filters can be used to avoid these traps. Please see the example test suite (Chapter 5, Examples) for details.
The QTest test framework creates three output files in the
directory from which it is invoked:
qtest.log
,
qtest-results.xml
, and
TEST-qtest.xml
. They contain a summary of the
test results along with details about any test case that
unexpectedly failed. The qtest.log
file is
the primary file you will use to study the results of failing test
suites.
The qtest-results.xml
file contains the same
information as qtest.log
, but it is in an xml
format. The TEST-qtest.xml
file, introduced
in version 1.5, contains JUnit-compatible test output, which can
be consumed by many other tools. You can change the name of this
file by passing the -junit-suffix
option to
qtest-driver. Run qtest-driver
--help for additional details. The JUnit-compatible
output does not contain as much information as
qtest-results.xml
(most of the information is
there but just included in the failure information rather than in
specific xml elements), but its format is recognized by many
continuous integration systems and other tools that consume test
results. For details on the format of
qtest-results.xml
, please see the file
misc/qtest-results.dtd
in the source
directory. Although it is possible that, under some unusual
circumstances (such as if a disk fills up or the user interrupts a
test run), QTest may not create valid xml, QTest makes an effort
to create valid xml as much as possible. In particular, if a
user-supplied test suite exits abnormally, QTest will still
generate a valid xml file in that situation.
If coverage (see Chapter 7, Automated Test Coverage) is active for a
given run of the test suite, then the file called
will
also be created. You don't ordinarily need to look at this file;
its primary purpose is to be read by the coverage analyzer to
determine whether all coverage cases have been exercised. If
coverage analysis passes, the coverage output file is copied to
scope
.cov_out
.
If coverage analysis fails in a subsequent invocation of the test
suite, you can compare the contents of that file with the new
coverage output file to figure out where the missing cases used to
be covered.
scope
-passed.cov_out
QTest is known to pass its test suite on Windows both when running
under Cygwin and when running with MSYS and ActiveState Perl.
Although QTest itself works on Windows, it was originally designed
with a UNIX environment in mind. As such, you would be best to
have a UNIX-like environment. With Cygwin, this is not a problem.
If QTest detects that it is running under ActiveState Perl, it
assumes that it is running in an MSYS/MinGW environment. In this
case, it will attempt to discover the Windows path to
/bin/sh
, and it will use this explicitly when
invoking commands. This is to enable UNIX-like command syntax to
work properly in the arguments to runtest
.
If you wish to have your test suite work with ActiveState Perl, you must take care to avoid UNIX-specific constructs in your test suite. Here is a list of issues you should keep in mind:
Using backquotes (`...`
),
system
, or fork
and
exec
are not likely to work properly.
ActiveState Perl (at least as of 5.10.1) appears to use
cmd.exe to run the commands regardless of the
value of COMSPEC
. In particular, this means
there is no way to run shell scripts that start with
#!
from system
or
`...`
. QTest emulates this itself using a
function called winrun
, which you can find
in TestDriver.pm
. This function writes out
a shell script and then runs it using
/bin/sh.exe. Note that running commands
using QTest's own runtest
method works fine
even if those commands include embedded `...`
calls. This is exercised in the test suite.
Use of exit values of
SIG:
signals is
unreliable. Although signals are somewhat emulated inside the
MSYS environment, you really can't count on them since they
aren't really supported in Windows.
n
If QTest detects that it is running on a terminal, it will try
to use ANSI colors for PASSED
,
FAILED
, etc., in its output. When running
MSYS's sh.exe without
rxvt, the ANSI terminal codes are not
properly interpreted. There is a module available from CPAN
called Win32::Console::ANSI
. If you have
it installed, QTest will use it, and you will see the nice,
pretty colors. If you don't have it installed, you should
probably set the environment variable
ANSI_COLORS_DISABLED
to 1
to
avoid seeing “garbage” characters around your test
outcome indications.
In general, there are other issues to keep in mind about using ActiveState perl. It's really a native Windows perl, and Perl does little to shield you from the underlying operating system. However, if you are writing software that is supposed to run in both UNIX and native Windows environments, you are probably well aware of the issues. You generally have to manually handle differences between Windows paths and UNIX paths and make sure to avoid doing things that don't work in a Windows environment. There is plenty of documentation on this included with ActiveState's Perl distribution and in the perlwin32 manual page included with the Perl distribution.
Testing multithreaded programs is inherently difficult because the order in which output is generated may vary from one run to another. Within a single thread, however, the order of output should be predictable. The problem with simply separating output by thread is that important sequence information is lost. Although it may not be possible to predict the strict sequence of output messages, it should be possible to indicate a strict order among specific groups of items that span across threads. For example, if one thread reacts to an event in another thread, the reaction must follow the original event. QTest provides tools to assist with testing multithreaded programs by providing a mechanism to separate a program's output by thread and to create sequence groups within the output for output lines that span thread boundaries but must appear in a fixed relative order.
In order to make use of these facilities, each thread must be
identified by a unique and predictable string, and that thread's
name must be prepended to each line of output that could
potentially be immediately followed by output from a different
thread. The exact syntax that QTest expects is that each line is
prepended with
[[
where
id
]]:id
is the name of the thread that
generated the output. For example:
[[id]]:output line
A good strategy for doing this is to create a logging thread that logs messages on behalf of other threads and identifies the thread on whose behalf the log message is created. Having log messages written to a thread-safe queue to be read by a specific logging thread ensures that log messages from different threads will not be arbitrarily interleaved (even within a single line) without requiring expensive locking of the calls that actually perform the log operations themselves.
When using QTest's multithreaded testing capabilities, the expected output file for a test case must contain lines that are prepended by the name of the thread that generated each line, but the lines do not have to be in the exact same sequence that they will be output by the program being tested. The test framework will require all lines labeled with the same thread ID to appear in the proper relative order but will compensate for interleaving of other lines. Additionally, if it is necessary to enforce that certain lines in one thread appear in a specific relative sequence with other lines in other threads, a sequence group identifier may be inserted in the line after the thread id. The exact syntax is
[[id
]]:((seq1
,seq2
,...))output line
When QTest reads the expected output file, it actually splits it into separate expected output files for each thread and for each sequence group. When QTest reads the actual output, it uses the thread identifier to match the actual output line with the next expected line from the expected output file. If the next line from the expected output file is marked with sequence identifiers, the sequence identifiers are stripped from the line before the comparison is made. Then, in addition to comparing the line against the thread-specific expected output file, it is also compared with the expected output file for the sequence group. If there are any mismatches or errors during this process, sequence checking is abandoned for the remainder of the test case.
This is all best explained with an example. Suppose
event1
from thread A
must precede both event2
from thread
B
and event3
from
thread C
and that no other ordering
constraints are present including the relative order of
event2
and event3
.
The combined output could look like this:
[[A]]:msgA1 [[C]]:msgC1 [[C]]:msgC2 [[A]]:msgA2 [[B]]:msgB1 [[A]]:event1 [[B]]:msgB2 [[C]]:event3 [[B]]:event2 [[B]]:msgB3 [[C]]:msgC3 [[A]]:msgA3
or like this:
[[A]]:msgA1 [[A]]:msgA2 [[A]]:event1 [[C]]:msgC1 [[B]]:msgB1 [[B]]:msgB2 [[B]]:event2 [[C]]:msgC2 [[C]]:event3 [[A]]:msgA3 [[B]]:msgB3 [[C]]:msgC3
or numerous other possibilities. In each case, all the
A
lines came in the right order, all the
B
lines came in the right order, all the
C
lines came in the right order, and
A
's event1
came
before both B
's
event2
and C
's
event3
. Here is an example expected output
file that illustrates this:
[[A]]:msgA1 [[A]]:msgA2 [[A]]:((a,b))event1 [[B]]:msgB1 [[B]]:msgB2 [[B]]:((a))event2 [[C]]:msgC1 [[C]]:msgC2 [[C]]:((b))event3 [[A]]:msgA3 [[B]]:msgB3 [[C]]:msgC3
Notice that each line is labeled with the thread that generates it.
The sequence group identifiers label event1
and
event2
as being in group
a
and event1
and
event3
as being in group
b
. In this way, we ensure that
event1
precedes both event2
and event3
without imposing any order
constraints on event2
and
event3
relative to each other. Note that the
overall order of lines is different in our expected output. When
designing multithreaded programs, the author generally has some
conceptual notion of the order in which things should happen, even
if some interleaving is possible. The easiest way to create
combined multithreaded output files is to just group lines relating
to a specific event together for a specific thread. In the above
example, we could conceptually view the software as generating two
messages from thread A
relating to event1,
two messages from thread B
relating to
event2
, and two messages from thread
C
relating to event3
.
Perhaps threads A
, B
,
and C
immediately start generating messages,
but threads B
and C
block waiting for event1
before generating
event2
and event3
. Then
after each thread does its event, it generates one more message.
The earlier messages could arbitrarily interleaved, and
msgA3
could come before
event2
or event3
, but our
expected output file enables us to ignore this complexity and
pretend everything is nice and neat.
In order for the test framework to interpret an expected output
file as input to a multithreaded test case, the
THREAD_DATA
key has to be passed in as part of
the output hash passed to runtest
. It must
contain a list of all the threads and sequence groups that are
expected to appear. Please see TestDriver.pm
and the example test suite for the exact syntax of the call to
runtest
. The test driver then generates
t+s+3 test cases where
t is the number of threads and
s is the number of sequence groups. This
includes a test that runs the actual program and, if applicable,
checks its exit status, one output comparison test for each thread,
one comparison test for each sequence group, one test to indicate
that we saw the correct threads and sequence groups, and a final
test that dumps the unfiltered actual output in comparison to the
expected output.
[2]
Being able to compare the actual output of the program with the
expected output may be useful to help figure out what went wrong,
but keep in mind that it will not compensate for reordering due to
interleaving of messages from different threads and will therefore
likely contain irrelevant differences. (These are, in fact, the
differences that this system is designed to work around, but when
the test case fails, experience shows that it's sometimes easier to
find the failure in context when the actual output is available
than by looking separately at each individual thread and sequence
group.)
[2] The final test runs only if at least one of the other test fails. Otherwise, it is an empty test case to keep the number of test cases consistent.
There is an example test suite in the example
directory of the QTest distribution. You can run the test suite
there by typing make check from the example
directory. You must have the qtest-driver
program in your path. The README.txt
file in
that directory contains a guided tour of the example test suite
with plenty of commentary to explain how to use all the features of
the test system.
Additionally, QTest has its own test suite implemented in itself.
The test framework's test suite may be a bit confusing at first
glance because it actually contains other test suites within it.
The test framework's test suite combined with the self-contained
test suites that are used within it exercise the complete
functionality of the test framework and also illustrate some
techniques that are useful for designing software for use with this
test framework such as filtering out date strings and other things
that interfere with storing expected program output. Please see
the qtest
directory within the test
framework's source code to find the test suites. The top-level
Makefile
in the
test_suite
directory in the QTest sources
contains an invocation of the test suite in its
check target.
Table of Contents
Here we discuss a few anecdotes and suggestions of things to keep in mind while designing test cases. These examples come from issues encountered over years of using this test framework.
Make sure a program's output is meaningful and detailed enough to make sure that the right thing is happening for the right reason. Here are some examples of test suite problems that have resulted from not having adequately meaningful output.
An application was designed to do automated transfer of files from one area to another. The file transfers were arranged in stages where each stage was identified by a regular expression. The transfer application would transfer all files whose names matched the regular expression for the first stage. Then, once it finished, it would scan until it found a file whose name matched the next stage, at which point it would switch to the next stage and start the process over again. One feature of this application was support of optional stages. Say, for example, that stage 1 and 3 were mandatory and stage 2 was optional. When the application finished transferring all stage 1 files, it would look not only for stage 2 files but also for stage 3 files. If it found no stage 2 files and at least one stage 3 file, it would skip stage 2 because stage 2 was optional. To test this program, the test suite created several stage 1 files, several stage 3 files, and no stage 2 files, and verified that the program transferred the files in the correct order: all stage 1 files, then all stage 3 files. There was a subtle bug in the program: the stage 2 regular expression accidentally matched the stage 3 files, so all the stage 3 files transferred in stage 2 instead of stage 3, which meant the test suite passed even though the optional nature of stage 2 had not been exercised. If the software had printed diagnostic messages indicating which stage was currently in progress, the correct output would have been generated only if the files were transferred in the correct stages, not just in the correct order.
An application that was supposed to check a file for correctness reported errors if it found errors but reported nothing if it found no errors. The test case in the test suite for the normal case expected no output. If the program actually failed to run any checks at all, the test case would still pass. The program should have been tested in some kind of verbose mode that would ensure that all its checks were actually being run. Alternatively, some kind of coverage cases should have been generated. Be especially careful about this with programs named test as you may accidentally be running /bin/test instead.
Another common issue that comes up with testing is handling of information that is not constant but is instead a function of some transient aspect of the environment. Here are some examples of this.
If a program generates timestamps, you might be tempted to just skip over the timestamp generation entirely and replace it with some fixed string when running in the test environment. Be careful if you do this to make sure that the timestamp function itself is tested. For example, have the function that creates the timestamp itself be callable separately from some testing hook. Have one test case that generates a timestamp and compares it to a regular expression and perhaps subjects it to some additional verification to make sure that it is a valid and accurate timestamp. Then, when running in the test suite, have the program generate some phony but predictable string so that the test suite's output doesn't actually contain any time strings. QTest's own test suite uses this technique. An alternative technique is to have a filter that recognizes valid timestamps and replaces them with fixed strings.
It is not uncommon for a program to contain absolute path names
or users' login names. In this case, it is useful to have a
filter in the test suite that replaces the current directory with
--CWD--
or the current username with
--USER--
, for example. This makes the test
output independent of who is running the test suite or where it
is being run from. To be sure, it's a good idea to run test
suites from different locations or with different login
credentials. Similar techniques may be desirable for normalizing
path separators across platform boundaries.
Many programs will contain some aspects that can't be tested automatically. In these cases, it should be a goal to structure the software in such a way as to minimize the amount of testing that has to fall into that category. Here are some examples of how certain test suites have handled this problem. One common theme in these examples is that the test suite would prompt the user to perform the manual tests and would tell the user exactly what to do and what to expect. In this way, having the test suites include a few manual tests didn't compromise the goal of having the tests be runnable by people who were not familiar with the software.
A driver was written that processed certain types of input to send them to the printer. For the most part, it was sufficient to just check the data that was actually going to the printer to ensure that it was correct, but at some point, it was also necessary to make sure that the page actually printed. The driver's test suite itself had one test case that actually prompted the user for a printer name and sent a page to the printer. The test suite then asked the user whether the page printed properly by asking the user to verify certain features on the page. Then, for all other test cases for the driver and for applications that use the driver, the driver implemented a special spool directory from which the application could pick up the data that was being sent to the printer. In this way, by verifying that the one test case that actually printed a page was working properly, all other applications that used the print driver could be tested automatically without actually having to print pages, and those applications' test suites could actually inspect the data from the special spool directory.
An emacs mode for highlighting text was developed. It included a feature that would enable the user to hit n and p to move to the next and previous highlighted block. The application software would log to a file the positions within the file being edited at which the highlighting started and stopped. However, at some point, it was necessary to have an operator actually verify that the highlighted text was actually highlighted and readable. This test suite included one manual test case in which it put the operator into a file and told the operator to hit n and p in a certain way and to verify that certain text was highlighted. The operator would then verify whether the system behaved according to a specifically described behavior pattern. Once it was known that the actual highlighting code itself was working, all other tests could be run by simply inspecting the numerical file positions of the highlighted text.
A very important rule to follow when creating test output is to hand-create all your expected output files. This can be difficult and tedious, but in the end, it is a big time saver. Countless incorrect test results have been “locked in” by people who just eyeballed the output of a program and, if it looked right, saved it in the test suite. Although, in an ideal world, we would never compromise on this rule, realistically there are going to be circumstances under which this is impossible. In such cases, there are several things that can be done instead. In some cases, it may make sense to use a different program to help with generating the test output as long as the other program is not exercising exactly the same code. For example, using complex keyboard macros in emacs to generate test output may be completely legitimate if the actual software is generating the output another way. Another possibility is to have the test suite not actually directly match the output but rather to have a filter that processes the output and generates a summary. This technique can be used for processing audio files or image files looking for certain features. Finally, if generation of the output is simply too difficult, the test suite developer can create a written list in advance of what they will do to inspect the output. This is safer than just looking at it and making sure it looks right.
Table of Contents
All software test enthusiasts know the value and importance of thorough automated test suites. By requiring every piece of code to have an automated test suite, we enforce a designing for testability mentality among our developers and produce software whose quality is much higher than the norm. To test software thoroughly, it is necessary to build test suites that include not only black box tests, but also white box, or more aptly named clear box tests that are carefully crafted to exercise the code in particular ways. By ensuring that our test suites fully exercise our code, we can produce code that is as close as possible to being bug-free.
Even the most thorough black box/clear box test suite may be subject to a few weaknesses. Consider the following scenarios:
If the last test case were removed from any passing test suite, all prior cases would still pass. How would you know whether any functionality exercised by the last test case was still being exercised by the test suite?
Suppose a particular test case were crafted to exercise a specific boundary condition, but internal parameters were changed in a way that moved the boundary. The test case may still generate the expected results and therefore still pass. How would you know whether the boundary condition is still being tested?
If a programmer who is maintaining a piece of software modifies a test case or a test output file, how can you be sure the modification hasn't undermined an existing test case, making it no longer exercise the code it was designed to test?
If you are reviewing someone's code, how can you easily guarantee that a particular condition is and will always be exercised by the test suite?
The automated test coverage system included in QTest addresses these and other issues by creating a tight coupling between the test suite and the conditions in the code that the test suite is supposed to exercise. Specifically, it introduces a mechanism by which the overall test suite will fail unless all tested scenarios are exercised, even if every individual test case in the test suite passes. This is achieved with minimal impact on existing test suites and minor impact on the code being tested, and it works in a wide range of languages and platforms without the need for expensive third-party tools.
The techniques discussed here are programming and testing techniques: they require explicit implementation by software developers while a system is being coded. These techniques are distinct from (and complementary to) the use of “black-box” tools that analyze running software for code coverage in the traditional sense; i.e., which “lines of code” are covered during a specific collection of program executions. Although such coverage tools are valuable, their goals are orthogonal to the goals of QTest's coverage system.
We define the following terms as they are used in this document:
any “event” within the software that we wish to detect using the automated test coverage system
a way of assigning coverage cases to a specific test suite, allowing multiple test suites to exercise different coverage conditions within one body of code
The automated test coverage system consists of three components:
function calls to the coverage system made from inside the code being tested to tell the system that a specific coverage case has occurred
a file that lists all the coverage cases that belong to a given scope
the program that compares the coverage calls and their results to the coverage case registry
The three components of the coverage system are implemented transparently inside of the QTest framework.
Before discussing the details of the coverage system, we present a minimal coverage example. Here is a simple C++ code fragment that tests a condition and takes one of two actions:
if (condition) { do_something(val1); } else { do_something_else(); }
Any test coverage system could ensure that both halves of this
branch were executed in a given test suite. Suppose, however,
that we wanted not only to ensure that both branches had been
executed, but that the first branch had been executed at least
once with the value of val1
greater than 5 and
at least once with the value of val1
not
greater than 5. Using our coverage system, we could add coverage
calls to encapsulate these conditions. The resulting code
fragment would then look like this:
if (condition) { QTC::TC("example", "something", (val1 > 5 ? 0 : 1)); do_something(val1); } else { QTC::TC("example", "something else"); do_something_else(); }
The call to the function QTC::TC()
is an
example of a coverage call. It tells the coverage system that a
specific event occurred, perhaps under a specific set of
circumstances. We will now explore coverage calls and the rest of
the system in depth.
The test coverage system is integrated with QTest. It will be
active if exactly one file that has the
.testcov
extension appears in the directory
from which qtest-driver is invoked.
Alternatively, if qtest-driver is being invoked
from other than the source directory (such as from a
platform-specific object file directory),
qtest-driver can be invoked with the
-covdir
flag to tell it where to look for
coverage files. These details are ordinarily hidden in the
integration of QTest with a specific build environment.
In order to use the coverage system, you will need an
implementation of the coverage calls. In the QTest distribution
you can find a Perl implementation in QTC.pm
in the top-level directory. Additional implementations for Java,
C/C++, and GNU Make can be found in the QTC
directory. You can easily provide your own coverage call
implementation in any language. We provide a simplified example
implementation in both Perl and C++ in this document (see Section 7.10, “Coverage Call Implementation”). The actual
implementations included in the QTC
directory
may be somewhat more portable or complete.
Here we describe each component of the test coverage system in detail.
Each individual coverage case appears in the code as a call to
the function QTC::TC()
. This function
takes three parameters: a string literal containing the name of
the scope, a string literal containing the name of the coverage
case, and an optional numeric argument, which may be an
expression. The numeric argument is used to distinguish
different circumstances under which a specific coverage call may
be made. In our first example above:
QTC::TC("example", "something", (val1 > 5 ? 0 : 1));
the scope is example
, the coverage case
name is something
, and the numeric
argument is dependent upon the value of the variable
val1
. With QTest, we require the scope name
and coverage case name to be string literals
and to appear lexically within one line. As of qtest 1.8, they no
longer have to be on the same line as the coverage call.
We also require the coverage call to be preceded only by optional
whitespace.
[3]
The numeric argument may appear on a subsequent line of the
source file if desired. This enables the coverage system to
distinguish between whether this part of the code was executed
with the value of val1
greater than 5 or not
and to require that both cases occur in the test suite.
In this scheme, the code being tested is sprinkled with test coverage calls such as the one above. These calls serve both as the interface from the code into the test coverage system and as a promise to the reader that the code in question is exercised by the test suite. In effect, this enables the code to test the test suite in addition to having the test suite test the code. It also enables the developer to add coverage calls to the code while it's being implemented. This is when the list of boundary conditions, confusing corner cases, etc., is as fresh in the programmer's mind as it ever will be. The programmer could just keep a running list of ideas for the test suite, but by coding a coverage case, the programmer guarantees that the code will be exercised in the test suite in exactly the intended way. It is often the case that development of the code stays a few steps ahead of development of the test suite. Having coverage cases for new sections of code that have not net been exercised in the test suite prevents the test suite from passing and thereby prevents the developer from forgetting to test those new sections of code.
Although addition of coverage cases is intended to be done by the programmer, this functionality can also be very useful for the reviewer. When reviewing someone's code, insertion of a test coverage case into the code is a great way to determine whether the test suite is exercising the code in a particular way. It also ensures that if it is not, this missing functionality will be added to the test suite before it will pass again.
All coverage cases have to be registered
in the coverage case registry. The
coverage case registry lists all the coverage cases that are
expected to appear in a specific coverage scope along with the
maximum expected value of the numeric argument passed as the
third argument to the coverage call. The coverage case registry
is an ASCII file named
,
where scope
.testcovscope
is the scope whose cases
it lists.
Exactly one coverage call must appear in the code for every
coverage case in a given scope. That call must be made at least
once with each numeric value from zero up to and including the
number indicated in the coverage case registry. In the above
example, the numeric argument to the
something
coverage case in the
example
scope may have the value of either
0 or 1 under normal conditions. Therefore, the file
example.testcov
would contain the following
line:
something 1
This indicates that this coverage call must be called at least once with the value 0 and at least once with the value 1.
In addition to the list of coverage cases in this scope, the
registry file contains a list of other scopes to allow but ignore
in the code. Any scope that appears in a coverage call in the
code other than the primary scope or the explicitly declared
other scopes generates an error. The ability to declare other
ignored scopes allows multiple scopes to be present in a
collection of source files while still ensuring that coverage
cases won't be overlooked as a result of a typographical error in
the scope name. If the example above also contained coverage
calls for a scope called other
, we would
expect to see this line:
ignored-scope: other
in the coverage registry file as well. This tells the system
that it may safely ignore coverage calls in the
other
scope.
The remaining component of the coverage system is the coverage analyzer. The coverage analyzer is a short block of code that is run once at the beginning and once at the end of the test suite. The coverage analyzer is built into the QTest test framework. At the beginning of the test suite, the coverage analyzer performs the following tasks:
Ensure that every coverage case listed in the registry is unique.
Ensure that every coverage call in the source code either belongs to the current scope and appears in the registry or that it belongs to one of the explicitly ignored scopes.
Ensure that every coverage case in the current scope appears exactly one time in the code.
If all of these checks succeed, the test coverage system is activated. Otherwise, the test suite is aborted.
QTest imposes lexical constraints upon coverage calls so that the analyzer can find them without having to have any ability to parse the programming languages that may be in use. We impose the following restrictions:
The names of the scope and coverage case must be string literals and must appear within a single line, though as of qtest 1.8, this no longer has to be the same line as the name of the coverage function.
The name of the coverage function may be preceded on its line by whitespace only.
No restrictions are placed on the numeric argument. It may appear on the same line or a different line, and it may be a numeric literal or any expression with a numeric value.
At the end of the test suite, the coverage analyzer ensures that every coverage call has been called at least once with each required numeric argument. It then reports a list of all missing and extra coverage cases. The coverage analyzer's checks are cumulative throughout the entire test suite run.
Here we present a simple example application that uses our test coverage system. This is a program that finds a number in a sorted array using either a straight linear scan or a binary search, depending upon the size of the array. This is a toy program for purposes of illustrating the test coverage system, but it has enough interesting cases to be illustrative. Here is the code without any coverage calls:
#include <vector> static unsigned int const SCAN_THRESHOLD = 10; // Find an item by doing a linear scan int scan(std::vector<int> const& v, int to_find) { int nitems = v.size(); for (int i = 0; i < nitems; ++i) { if (v[i] == to_find) { return i; } } return -1; } // Find the item with a binary search int bsearch(std::vector<int> const& v, int to_find) { int low = 0; int high = v.size() - 1; while (low <= high) { int test = (low + high) / 2; if (v[test] == to_find) { return test; } else if (v[test] < to_find) { low = test + 1; } else { high = test - 1; } } return -1; } // Call one of scan() or bsearch() based on the size of the vector int search(std::vector<int> const& v, int to_find) { if (v.size() <= SCAN_THRESHOLD) { return scan(v, to_find); } else { return bsearch(v, to_find); } }
The function's implementation is quite straightforward: if the list is smaller than a certain threshold, do a linear scan; otherwise, do a binary search. The binary search algorithm is just a classic textbook implementation of binary search. We don't handle cases like detecting that the array isn't sorted, handling duplicate items, etc. This code is to demonstrate test coverage, not to talk about searching algorithms!
Although this code is simple and straightforward, there are a lot of conditions that a good test suite should exercise. For example:
Did both types of search get exercised both for numbers that were found and numbers that were not found?
Did we have numbers that were not found because they were too low, too high, and in the right range but just not there?
Did all three branches of the binary search code get exercised (increasing the lower bound, decreasing the upper bound, and finding the value)?
This code also presents a good example of how a test case may
later no longer test the condition for which it was originally
designed. For example, consider the
SCAN_THRESHOLD
variable. If we make it too low
or too high, we may find that cases designed to exercise a
condition in one type of search suddenly find themselves getting
processed with the other type of search. What we really need is
some assurance that the test suite exercises the conditions in the
code that we want it to. The test coverage system allows us to
make these assurances into a permanent part of the test suite.
Here is the code again. This time, we're also including a small
main()
routine to test the
search()
function as well as several coverage
calls. The main routine just creates an array of consecutive
multiples of 3. In a real application, we might want to test with
other arrays, but again, our purpose here is only to illustrate
the test coverage system.
01 #include <iostream> 02 #include <stdlib.h> 03 #include <vector> 04 #include "QTC.hh" 05 // If you change this threshold to something greater than 15 or less 06 // than 4, the test suite passes but coverage fails. 07 static unsigned int const SCAN_THRESHOLD = 10; 08 // Find an item by doing a linear scan 09 int scan(std::vector<int> const& v, int to_find) 10 { 11 int nitems = v.size(); 12 for (int i = 0; i < nitems; ++i) 13 { 14 if (v[i] == to_find) 15 { 16 QTC::TC("search", "linear scan found"); 17 return i; 18 } 19 } 20 QTC::TC("search", "linear scan not found", 21 ((to_find < v[0]) ? 0 : // too low 22 (to_find > v.back()) ? 1 : // too high 23 2)); // just not there 24 return -1; 25 } 26 // Find the item with a binary search 27 int bsearch(std::vector<int> const& v, int to_find) 28 { 29 int low = 0; 30 int high = v.size() - 1; 31 while (low <= high) 32 { 33 int test = (low + high) / 2; 34 if (v[test] == to_find) 35 { 36 QTC::TC("search", "bsearch found"); 37 return test; 38 } 39 else if (v[test] < to_find) 40 { 41 QTC::TC("search", "bsearch increased low"); 42 low = test + 1; 43 } 44 else 45 { 46 QTC::TC("search", "bsearch decreased high"); 47 high = test - 1; 48 } 49 } 50 QTC::TC("search", "bsearch not found", 51 ((to_find < v[0]) ? 0 : // too low 52 (to_find > v.back()) ? 1 : // too high 53 2)); // just not there 54 return -1; 55 } 56 // Call one of scan() or bsearch() based on the size of the vector 57 int search(std::vector<int> const& v, int to_find) 58 { 59 if (v.size() <= SCAN_THRESHOLD) 60 { 61 return scan(v, to_find); 62 } 63 else 64 { 65 return bsearch(v, to_find); 66 } 67 } 68 int main(int argc, char* argv[]) 69 { 70 if (argc != 3) 71 { 72 std::cerr << "Usage: search n-items to-find" << std::endl; 73 exit(2); 74 } 75 int nitems = atoi(argv[1]); 76 if (nitems < 1) 77 { 78 QTC::TC("search", "nitems < 1"); 79 std::cerr << "search: n-items must be >= 1" << std::endl; 80 exit(2); 81 } 82 int to_find = atoi(argv[2]); 83 std::vector<int> v; 84 for (int i = 0; i < nitems; ++i) 85 { 86 v.push_back(i * 3); 87 } 88 int idx = search(v, to_find); 89 QTC::TC("search", "idx location", 90 ((idx == -1) ? 0 : // not found 91 (idx == 0) ? 1 : // first item 92 (idx < nitems - 1) ? 2 : // somewhere in the middle 93 (idx == nitems - 1) ? 3 : // last item 94 999)); // can't happen 95 std::cout << "index: " << idx << std::endl; 96 return 0; 97 }
Observe that we have added eight test coverage calls, shown above
in boldface type. These calls are all in the
search
scope. Here is the
search.testcov
file that would accompany this
code:
linear scan found 0 linear scan not found 2 bsearch found 0 bsearch increased low 0 bsearch decreased high 0 bsearch not found 2 idx location 3 nitems < 1 0
Now we'll look at some specific coverage calls to study what they
do. Of the eight calls, five of them (lines 16, 36, 41, 46, and
78) are simple coverage calls with the numeric argument omitted.
(The numeric argument's value defaults to 0
.)
These calls are just inserted into the normal flow of the code to
tell the system that the part of the code that contains them has
been executed. You can see that this type of coverage call has
been placed at each significant branch in the code as well as for
the one error condition that we want to be sure to exercise in the
test suite.
The other three coverage cases make use of the numeric argument to
the coverage call. There is a coverage case at the end of each of
the two lower-level search functions (lines 20–23 and
50–53) that gets invoked when the search function fails to
find the number in the array. These calls both distinguish
between the case of the value being less than the first item,
greater than the last item, or otherwise. This ensures that the
search routines are both tested with numbers that fall out of
range in each direction as well as with in-range numbers that are
not in the array. These are important boundary conditions. The
last coverage call in the main routine (lines 89–94) also
uses the numeric argument. In this case, we're making sure that
other important boundary conditions are tested including finding
the first and last numbers in the array. Notice that this
coverage call may potentially pass the value 999 as the numeric
argument (line 94), but the coverage case is registered with a
maximum numeric argument of 3. This illustrates a useful
technique: use numbers greater than the registered maximum for
cases that can't happen. If that case were ever to happen, it
would result in the test coverage system reporting idx
location 999
as an extra coverage
case. Just as it's often better to have an explicit
else
clause when coding conditionals, it's
often better to use this type of technique when coding expressions
to be used as the numeric argument to a coverage call. This is
safer than just assuming you haven't missed any possibilities.
This code, along with its accompanying test suite, can be found in
the example/search
directory in the QTest
distribution. See README.txt
in that
directory for details on how to run it.
Notice that some of the test coverage cases here are called inside of a loop. Although coverage calls have a nominal performance impact when the coverage system is inactive (they just check an environment variable and return), in critical or time-sensitive code, even this function call overhead could be undesirable. In some cases, there may be security implications of having the coverage calls accessible by the end user—a user who knew which environment variables to set could cause the application to append data to any file the application had write access to.
In cases such as these, it would be possible to use conditional compilation (if supported by the language) to omit the test coverage calls for a releasable build of the software. If this is done, it is especially important that the coverage calls have no side effects (a good idea in any case). To be sure, test suites should be designed so that they can still be run on releasable versions of the code with coverage analysis turned off.
In this section, we discuss three examples of actual uses of the test coverage system to detect and correct faults in test suites.
A perl program was generating XML from the output of an OCR program. The OCR program sometimes put unprintable characters in its output. This perl program had a line of code to replace unprintable characters with the “?” character.
One of the input files was constructed to have several unprintable characters. A coverage case ensured that the perl substitution operator returned true (indicating that it replaced some characters) at least once in the test suite by using the return value of the substitution operator in the numeric argument to the coverage call.
During an update to the software, a junior programmer edited that test file to exercise some condition unrelated to the unprintable characters, and in doing so, removed the line that contained the unprintable characters. He made the corresponding edit in the expected output file, removing the line of output that contained the “?” characters.
Although that test case passed, the character stripping code was
no longer exercised in the test suite. This caused a failure to
be reported by the test coverage system, thus alerting the
developer that his change had undermined the original purpose of
the test case. By comparing the
scope.cov_out
file with the
scope-passed.cov_out
file from an earlier
run, it was possible to discover which test case used to exercise
the coverage case. The test case was redesigned so that both the
new and old conditions could be tested.
A program was generating XML for some but not all of the articles in a journal issue. Certain articles would be excluded based on a series of rules. Sometimes an issue would have no articles to be generated because all of the articles met the exclusion conditions. A special case in the code existed to test this valid situation, and a coverage case ensured that this happened in the test suite. Only one issue in the test suite exercised this condition.
Six months later, the exclusion rules were changed. As it happened, one of the articles in the special test issue now no longer met the exclusion conditions. The test coverage system alerted us to the fact that the special “all articles excluded” case was no longer being exercised, thus enabling us to remove that article from the test issue. The coverage system further ensured that removing that article didn't undermine any other test conditions.
A multithreaded image manipulation program had a worker thread that calculated information that was not needed to draw the screen but that was needed to do certain specific manipulations on the image. Upon performing the first operation that required the results of the calculation, the main thread would block waiting for the worker thread to finish. This case was exercised in the test suite by having an input image for which the worker thread's calculations took a long time. A coverage case was associated with code that got executed only when the worker thread was still running when the main thread first needed the information.
A year later, this code was tested on a much faster machine. The
machine was fast enough that the worker thread finished its
calculations on the test image before the main thread needed
them. All test cases passed, but the coverage system alerted us
to the fact that the thread waiting code was no longer being
exercised. The code was then updated to force an artificial
delay when the IN_TESTSUITE
variable was set, thus
guaranteeing that the condition would be tested on a machine of
any speed. This turned out to be particularly important because
the faster machine was using a newer implementation of the
underlying threading system with which this application had not
been previously tested.
Here are a few tips and tricks that can be used with the test coverage system to further increase its usefulness.
Pick names for coverage cases that start with the name of the source file in which they appear. This way, the lexically sorted listing of missing or extra coverage cases reported by the coverage analyzer will be grouped together by source file. Using naming conventions in this way to cause related coverage cases to be grouped together can also be useful when running only part of the test suite. You can inspect the list of missing coverage cases and make sure that none of them are for the part of the code that you're currently working on.
Agree on naming conventions for certain kinds of cases. For
example, put the word ERR
after the source file
name in all coverage cases that are associated with issuing error
messages. This causes all coverage cases that should happen in
error conditions to be grouped together within the cases for a
given source file.
Add coverage calls to the code while it is being written—that's when you're best able to think of all the detailed clear-box test cases you couldn't have thought of during design. This makes it impossible to forget to exercise certain functionality in the test suite.
Make coverage calls themselves conditional upon the circumstances under which you want to ensure that the event occurs, even if this information has to be passed in explicitly by the test driver. For example, the test suite for a configuration file parser may want to make sure that a particular construct appears in a file that contains no errors. It may not be sufficient to simply avoid making the coverage call if no errors have yet been detected because an error may still appear in the configuration file after the construct has already been seen. Although the application may have no way to know that, the test suite knows in advance whether a specific run is supposed to have errors or not. It can pass this information to the program being tested through an environment variable or command-line argument. The code being tested can make the coverage call conditional upon that information instead. In other words, don't be afraid to exploit knowledge that the test suite may have beyond what the application knows even if this means writing code in the application that exists solely to help it be tested. These are clear-box test cases after all.
Use numerical arguments not only for conditions that are expected to happen but also for conditions that are expected not to happen. Expected cases are numbered consecutively from 0 to n, where n is the number listed in the registry. Unexpected cases are numbered greater than n. That way, missing conditions generate missing test coverage cases, and extra conditions generate extra test coverage cases. For example, this coverage call:
QTC::TC("image", "border placement", ((left && (! right)) ? 0 : (right && (! left)) ? 1 : 999)); // 999 not expected
when accompanied by this registry line:
border placement 1
would cause generation of the extra coverage case
“border placement 999
” if
left
and right
were either
both true or both false. This would result in a coverage failure.
To understand how this coverage system works, it is useful to know exactly what the coverage call actually does. In our implementation, the coverage call determines whether coverage is active in the scope passed as its first argument. If so, then it appends the name of the coverage case and the numeric value to an output file. The names of the scope and output file are read from environment variables. These environment variables will have been set by the test framework based on the results of the first run of the coverage analyzer. At the end of the test suite, the coverage analyzer will read this file to find out which coverage cases were seen.
Here are sample C++ and perl implementations of the test coverage call:
// C++ implementation // Determine whether coverage is active in the current scope static bool tc_active(char const* const scope) { std::string value; // The return value of get_env indicates whether an environment // variable is set. If the second argument is non-null, it is // initialized with the value of the environment variable. return (get_env("TC_SCOPE", &value) && (value == scope)); } void QTC::TC(char const* const scope, char const* const ccase, int n /*= 0*/) { // Return immediately if coverage is not active for this scope. if (! tc_active(scope)) { return; } // Get the name of the output file. std::string filename; if (! get_env("TC_FILENAME", &filename)) { return; } // Append data from this call to the output file. FILE* tc = fopen(filename.c_str(), "ab"); if (tc) { fprintf(tc, "%s %d\n", ccase, n); fclose(tc); } }
# Perl implementation package QTC; sub TC { my ($scope, $case, $n) = @_; local $!; $n = 0 unless defined $n; return unless ($scope eq ($ENV{'TC_SCOPE'} || "")); my $filename = $ENV{'TC_FILENAME'} || return; my $fh = new FileHandle(">>$filename") or return; binmode $fh; print $fh "$case $n\n"; $fh->close(); }
In these implementations, we activate the test coverage system by
setting the environment variable TC_SCOPE
to the
coverage scope and TC_FILENAME
to the absolute path
of the file that the coverage system will append to.
In perl code, the QTC.pm
file has to be
loaded at runtime. If you don't ship a copy of
QTC.pm
with your code, you will want to make
sure that your code will work whether or not
QTC.pm
can be found. Rather than a simple
require QTC;
, you can use the following block
of code to load QTC
if present or fall back
to an empty stub implementation otherwise:
# If QTC is not available, create a stub so we can run without it. { # local scope my $have_qtc = 0; eval { require QTC; $have_qtc = 1; }; if (! $have_qtc) { eval { &QTC::TC(); $have_qtc = 1; } } if (! $have_qtc) { *QTC::TC = sub { } } }
If you use emacs, you may find this emacs lisp function useful. To use it, make sure the testcov file has been loaded into an emacs buffer. Then you can position the cursor on the line with a test coverage call and run this function to automatically add the coverage call to the coverage case registry. If you want a numeric argument of other than zero, run CTRL-U n M-x qtc-register to specify the value.
(defun qtc-register (num) (interactive "P") (setq num (cond ((null num) 0) ((listp num) (car num)) (t num) ) ) (let (comment) (save-excursion (goto-char (line-beginning-position)) (skip-chars-forward " \t") (setq comment (skip-chars-forward "/# \t")) (let (scope case) (cond ((looking-at "&?QTC\\(::\\|\\.\\)TC(\"\\([^\"]+\\)\", \"\\([^\"]+\\)\"") (setq scope (buffer-substring-no-properties (match-beginning 2) (match-end 2)) case (buffer-substring-no-properties (match-beginning 3) (match-end 3)) ) ) ((looking-at "[^#\n]*\\$(call QTC\\.TC,\\([^,]+\\),\\([^,)]+\\)") (setq scope (buffer-substring-no-properties (match-beginning 1) (match-end 1)) case (buffer-substring-no-properties (match-beginning 2) (match-end 2)) ) ) (t nil ) ) (if (not (null scope)) (let* ((bufname (concat scope ".testcov")) (buffer (get-buffer bufname)) ) (if (null buffer) (error (concat "no buffer " bufname " exists")) ) (if comment (delete-region (- (point) comment) (point)) ) (set-buffer buffer) (goto-char (point-max)) (insert case " " (int-to-string num) "\n") (save-buffer) ) (error "this line does not contain a test coverage call") ) ) ) ) )
If you would like to copy this code into your emacs environment,
you can find the code in a CDATA block in the XML sources of the
QTest manual, manual-src/qtest-manual.xml
in
the QTest source distribution.
[3] This is because the coverage analyzer uses a lexical approach to verifying that all coverage cases are registered. We discuss the registry and analyzer later in this section.