help testcase()

Title

testcase -- A Mata class for code testing

Contents

Introduction
Syntax
Example
Creating tests
testcase functions
testcase variables
Failures and errors
Diagnostics

Introduction

testcase is a Mata class for testing Mata code. The class provides an
"xUnit" type of testing framework (see Wikipedia "xUnit" article). To create
tests, create a subclass of testcase and write test functions
within that subclass. Users who are new to Mata class programming should
probably read m2_class.

Syntax

For a test class called (for example) "mytestclass" and a test instance called
"mytest", usage would look like this:

class mytestclass extends testcase {
void test_1()
... [other declarations]
}

mytestclass::test_1()
{
...
}

... [other function definitions]

mytest = mytestclass()

mytest.name = "mytest"
mytest.test_names = (
"test_1",
...
)

mytest.run()

Example

Suppose you've made a function that tries to return the sum of any two inputs
it receives.

function add(a, b)
{
return(a + b)
}

Now you want to write tests for this function, either to verify that it
behaves as intended, or to make sure that it will behave as intended after
changes are made to the code. We'll walk through how you can
make these tests using testcase. As a preview, the completed tests
will look like this:

class add_test extends testcase {
void test_errors()
void test_output()
}

void add_test::test_errors()
{
pointer(function) scalar fp

fp = &add()

this.assert_error(3250, fp, (&0, &"a"))
this.assert_error(3250, fp, (&"a", &0))
this.assert_error(., fp, (&(1, 2, 3), &(1, 2)))
}

void add_test::test_output()
{
this.assert_equal(add("a", "b"), "ab")
this.assert_equal(add(0, 1), 1)
this.assert_equal(add((0, 1), (1, 0)), (1, 1))
}

mytest = add_test()

mytest.name = "mytest"
mytest.test_names = (
"test_errors",
"test_output"
)

mytest.run()

In this example, all of our tests
of the add() function will reside within a single class, which we'll
call add_test:

class add_test extends testcase {
...
}

We'll write tests to verify that add() raises errors when we
expect it to and otherwise returns the value we expect. We'll put
tests for errors within a single function called test_errors(), and
we'll put the other tests in test_output(). These will be functions within
our add_test class, so our class declaration now looks like

class add_test extends testcase {
void test_errors()
void test_output()
}

The tests will use two of the
assert methods provided by testcase: assert_error() and
assert_equal() (see testcase functions
below).

void add_test::test_errors()
{
pointer(function) scalar fp

fp = &add

this.assert_error(3250, fp, (&0, &"a"))
this.assert_error(3250, fp, (&"a", &0))
this.assert_error(., fp, (&(1, 2, 3), &(1, 2)))
}

void add_test::test_output()
{
this.assert_equal(add("a", "b"), "ab")
this.assert_equal(add(0, 1), 1)
this.assert_equal(add((0, 1), (1, 0)), (1, 1))
}

The majority of the work is finished. All that we need to do now is
create an instance of this class

mytest = add_test()

then record the instance name and test names (so the
class knows which methods comprise the tests)

mytest.name = "mytest"
mytest.test_names = (
"test_errors",
"test_output"
)

Finally, we run the tests with

mytest.run()

and the output it produces looks like this:

test_errors ... ok
test_output ... ok

Creating tests

As demonstrated in the example above, you use
testcase by

1) subclassing testcase,

2) creating test functions within your subclass, and

3) creating sub-tests with the inherited assert functions.

The test functions you create should have a declaration like

void mytestfunction()

In other words, these functions should not return any values and should not
take any arguments.

As with any Mata class, you must declare the test functions in the (sub)class
declaration and then define them. With test functions there is an additional
requirement. You must put the name of the function in the test_names
variable (see example above or
testcase variables below).

testcase functions

Assert functions

void assert(real scalar v)

void assert_equal(transmorphic a, transmorphic b)

void assert_unequal(transmorphic a, transmorphic b)

void assert_equal_contents(transmorphic a, transmorphic b [, real scalar samecount])

void assert_close(numeric a, numeric b [, real scalar tolerance])

void assert_all(real a)

void assert_any(real a)

void assert_error(

real scalar err,

pointer(function) scalar func_ptr

[, pointer matrix arg_ptrs]

)

void assert_method_error(

real scalar err,

string scalar class_name,

string scalar func_ptr

[, pointer matrix arg_ptrs]

)

assert() is used to test that a condition is true. In Mata "true" is
encoded as 1 (or non-zero real scalar), and "false" is encoded as 0. Thus,
assert() only accepts a real scalar as input. It records an "error" if
the input is not a real scalar, a "fail" if the input is zero, or a "pass"
otherwise.

assert_equal() is used to test that its two inputs are equal. It records
a "pass" if the inputs are equal or a "fail" otherwise.

assert_unequal() is used to test that its two inputs are not equal. It
records a "pass" if the inputs are not equal or a "fail" otherwise.

assert_equal_contents() is used to test that two matrices (or scalars)
a and b have the same (equal) contents, regardless of how they are arranged.
For example, the matrices

(9, 10, 11), (10, 11, 9), and (11 \ 10 \ 9)

all have the same
contents. The optional input samecount determines whether the same elements
must occur the same number of times. For example, with samecount zero,

(9, 10, 11) and (9, 9, 10 \ 11, 11, 11)

are judged to have the same contents. The default value for samecount is 1,
i.e., each element must occur the same number of times. assert_equal_contents()
records an "error" if a or b is (a matrix of) class (classes instances
are difficult to compare). It records a "fail" if a and b do not
have the same contents (as described above). It records a "pass" otherwise.

assert_close() is used to test that its first two inputs a, b
are nearly the same, element by element. More specifically, it checks that

mreldif(a, b) < tolerance

where tolerance is the third, optional input (default value is 1e-12).
assert_close() records an "error" if either a or b is not numeric,
or if the tolerance (if specified) is not a real scalar. It records a "fail"
if any of the corresponding elements are not close or a and b have
different dimensions. It records a "pass" otherwise.

assert_all() is used to test that all elements of a matrix evaluate to
"true", i.e., are real and non-zero. It records an "error" if its input is not
real, a "fail" if any of the input's elements is zero, or a "pass" otherwise.

assert_any() is used to test that any element of a matrix evaluates to
"true", i.e., is real and non-zero. It records an "error" if its input is not
real, a "fail" if all of the input's elements are zero, or a "pass" otherwise.

assert_error() is used to test that a function raises an error when
expected to. The first input should be the intended error code (or zero for "no
error") or a missing value. If the first input is a missing value, the test
passes when the function raises any (non-zero) error. assert_error()
records an "error" if any of the inputs is miss-specified, a "fail" if the
function does not raise the intended error, or a "pass" otherwise.

assert_method_error() is used to test that a class member function raises
an error when expected to. The first input should be the intended error code
(or zero for "no error") or a missing value. If the first input is a missing
value, the test passes when the function raises any (non-zero) error. It records
an "error" if any of the inputs is miss-specified, a "fail" if the function does
not raise the intended error, or a "pass" otherwise.

Running tests and viewing results

void run()

void print_summary()

run() instructs the class to "run the tests". Each of the functions named in
test_names is called, information about passed tests, failures, and errors
is collected, and, if verbose is not zero (see
testcase variables below), a summary is printed.

print_summary() instructs the class to print a summary of the previous run.
The printed summary is the same as when using run() with nonzero verbose.

Pre- and post-test functions

virtual void setup()

virtual void setup_once()

virtual void teardown()

virtual void teardown_once()

void new()

setup() is used for any kind of preparation that is
desired to occur before each test function in your testcase
subclass is called. setup() is automatically called before each test
function.

setup_once() is used for any kind of preparation that is
desired to occur just once, before any test function in your
testcase subclass is called. setup_once() is automatically called
before the first test function is called (and before the first setup()).

teardown() is used for any kind of activity that is desired
to occur after each test function in your
testcase subclass is called. teardown() is automatically called after
each test function.

teardown_once() is used for any kind of post-test activity that is
desired to occur just once, after all of the test functions in your
testcase subclass have been called. teardown_once() is automatically
called after the last test function is called (and after the last teardown()).

new(), as with any Mata class, is used during class creation.
Use this function for any kind of preparation or customization that you
wish to occur when your testcase subclass is created.

testcase variables

Required

string scalar name

string vector test_names

name should be equal to the name of the class instance, as in the
example above. name must be supplied for each
class instance.

test_names should be a vector containing the names of all of the class
functions that serve as tests, as in the example
above. You may define other class functions if desired and not include them,
but the class will only know to run the tests in test_names.
test_names must be supplied for each class instance.

Optional

real scalar verbose

real scalar noisy_capture

real scalar noisy_assert

real scalar exit_setup_error

real scalar exit_teardown_error

verbose determines whether there is any output when run() is invoked.
Set mytest.verbose = 0 to silence output. Set mytest.verbose = 1 (or
any non-zero real value) to see output. The default value is 1. To see
output when verbose is zero, use print_summary().

noisy_capture determines whether the default Mata error messages are shown
when there is an unexpected error in a test. (See example below in failures and errors.) Set
mytest.noisy_capture = 0 to silence these error messages. Set
mytest.noisy_capture = 1 (or any non-zero real value) to see these error
messages. The default value is 1.

noisy_assert determines whether the default Mata error messages are shown
for the assert_error() and assert_method_error() functions. This will
not usually be helpful, since errors are usually expected with these functions,
but may be helpful when diagnosing an unexpected result. Set
mytest.noisy_assert = 0 to silence these error messages. Set
mytest.noisy_assert = 1 (or any non-zero real value) to see these error
messages. The default value is 0.

exit_setup_error determines whether testing should continue after an
error in setup() or setup_once(). A non-zero value signals that
testing should be aborted, a zero value signals that testing should continue. The
default value is 1.

exit_teardown_error determines whether testing should continue after an
error in teardown() (testing is complete after teardown_once()
regardless of errors). A non-zero value signals that testing should be aborted,
a zero value signals that testing should continue. The default value is 1.

Failures and errors

Let's add a couple lines to the test_output() function from the example above.

void add_test::test_output()
{
this.assert_equal(add("a", "b"), "ab")
this.assert_equal(add(0, 1), 1)
this.assert_equal(add((0, 1), (1, 0)), (1, 1))

this.assert_equal(add(0, 1), "a")
this.assert(add("a", "b"))
add()
}

The new lines are in bold, and add

1) an assertion that fails (0 + 1 does not equal "a"),

2) an assertion with an error (assert() expects a real scalar input), and

3) an error outside of an assertion.

When we run the tests now, we get output that shows failure and errors (if
continuing from the previous example, you'll have to drop mytest and
redefine it first):

: mytest.run()
add(): 3001 expected 2 arguments but received 0
add_test::test_output(): - function returned error
<istmt>: - function returned error

test_errors ... ok
test_output
subtest 4: assert_equal()
values are not equal
> element types differ: real vs string
subtest 5: assert()
error: input was string scalar, should be real scalar
** unexpected error 3001 after subtest 5 **

The error message immediately after mytest.run() corresponds to the
"unexpected error" in the last line. If we want to turn off the default Mata
error message, we set noisy_capture to zero:

: mytest.noisy_capture = 0

: mytest.run()

test_errors ... ok
test_output
subtest 4: assert_equal()
values are not equal
> element types differ: real vs string
subtest 5: assert()
error: input was string scalar, should be real scalar
** unexpected error 3001 after subtest 5 **

Diagnostics

If you get an error message when you invoke run(), like

: mytest.run()
type mismatch: exp.exp: transmorphic found where struct expected

make sure that you've set the name variable, and that you've set it to the
name of the instance and not, for example, the name of the class.

If you implement the optional setup(), setup_once(), teardown(),
or teardown_once() functions, you might have difficulty dropping your test
class and/or testcase(). The only remedy at present is to clear Mata.

There are several variables and functions used internally that are not listed
above. To prevent collision with existing functions and variables, avoid names
beginning with double underscore. Alternately, if wanting to use a name that
begins with double underscore, verify that the name is not in use by checking
the class declaration in testcase.mata.

Source code

testcase.mata

Author

James Fiedler

jrfiedler@gmail.com