Test Methods

Composition and Execution of Test Methods

Table of Contents

Handling Throwables in your Test Methods
JUnit Failures vs. Errors

The smallest groupings of test expressions are the methods that you put them in. Whether you use JUnit or not, you need to put your test expressions into Java methods, so you might as well group the expressions, according to any criteria you want, into methods. If you designate such a test method with the "@org.junit.Test" annotation, then JUnit will execute it as explained in the Test Classes and the JUnit Test Execution Environment chapter (and that is the ultimate purpose of doing so). For this document, I call such a method a test method.

The general recommendation is to test only one aspect of behavior in a method. (I know how vague this is). If that can be done with a single expression, then your test method should contain just one expression. There are two main reasons for keeping test methods very fine-grained.

First, your report output is only as granular as the test methods. If you have 10 methods each of which tests handling of a different type of input character, your test report will have a count (at least) for the success or failure of each character type. If you put the expressions to test all ten character types into a single test method, you will only get a single test result for all character types. The difference is drastic. If your first test fails, none of the other tests will even execute in the single-method scenario.

Secondly, setup and execution of one test may cause unintended consequences for a test which follows. This risk is usually acceptable for a single-author class, but for jointly developed software, or test classes which will outlast the tenure of the original author, the risk can be considerable.

With JUnit version 4, we implement test methods as follows.

Example 4.1. Implementation of a JUnit Test Method

    @org.junit.Test
    public void testDoubler() {
        TargetClass targetObject = new TargetClass();
        targetObject.init();   // Any setup needed to prepare the target object
        assertEquals("Target object failed to double 'xyz'",
                "xyzxyz", targetObject.double("xyz"));
    }

There are several important observations to be made here. It must be an instance method of type void with no parameters. Unlike JUnit 3.x, the name of the method doesn't matter to the JUnit 4.x test-runners (or to anything else). So, you can implement your own method naming convention. I didn't prefix the method name with "test" to comply with any JUnit rule, but because it makes the methods easy to recognize (both for human readers, and for scripts, Ant build files, etc.). Your test methods are allowed to throw anything by the rules of JUnit, but a good test method won't declare any Throwables (more on this topic below).

Test methods are single-threaded, but different test methods must be independent and can't depend on state (including values of instance or static variables) set by another test method. If you are going to make use of instance variables, you must take pains to initialize the variable values for each test method run. The following chapter explains how the test-runners set up and execute your test methods, and how you can write shared setup and shut down code.

Handling Throwables in your Test Methods

	Caution
	Be careful when catching `Throwable`s or `Error`s in your test methods. If you catch `java.lang.AssertionError` and don't rethrow it, you will have intercepted JUnit's test failure indication.

If your test method calls other methods which declare checked exceptions, you should not just declare it in your test method, but should handle it in either normal Java try/catch fashion, or with the JUnit expected attribute.

If you have elicited a Throwable on purpose, you can catch it to detect that, but if your test method is fine-grained (as it should be), you can accomplish the same thing more gracefully by telling JUnit that your test method should throw the specified Throwable.

    @org.junit.Test(expected=IOException.class)
    public void myTestMethod() {

In this example, we assume the source file has an import statement for java.io.IOException.

If, on the other hand, production of the Throwable indicates a test failure, then you should catch it from a try block and fail the test with something like Assert.fail(String).

Here is a code snippet that handles the very common case where the code being tested could throw any of various Exceptions, and they should all be considered test failures, not an error in the test system. This idiom can save you a lot of work and still create well-behaved tests.

Example 4.2. Generically treating generated Exceptions as test Failures

    public void testWithParams() {
        try {
            rb1.validate();
            rb2.validate();
            rb1.setMissingPosValueBehavior(
                    ValidatingResourceBundle.EMPTYSTRING_BEHAVIOR);
            rb2.setMissingPosValueBehavior(
                ValidatingResourceBundle.NOOP_BEHAVIOR);
            assertEquals(SUBSTITUTED_CONN_MSG,
                    rb1.getString(SqltoolRB.JDBC_ESTABLISHED, testParams));
            assertEquals(SUBSTITUTED_CONN_MSG,
                    rb2.getString(SqltoolRB.JDBC_ESTABLISHED, testParams));
            rb1.setMissingPosValueBehavior(
                 RefCapablePropertyResourceBundle.THROW_BEHAVIOR);
            assertEquals(SUBSTITUTED_CONN_MSG,
                    rb1.getString(SqltoolRB.JDBC_ESTABLISHED, testParams));
        } catch (Exception e) {
            AssertionError ae = new AssertionError(
                "RB system failed to generate end-user message");
            ae.initCause(e);
            throw ae;
        }
  }

Notice that I made my own AssertionErorr instead of just executing fail(). This is so that the JUnit failure report will identify the source of the original problem instead of the location of the fail() call, whether the original Throwable is generated by your test code (like a NPE) or fifty levels deep in the call stack.

You can easily insert additional catch statements for specialized reporting of especially-expected exception types; and you can also broaden it for code which is known to throw non-Exception Throwables. In that case, you must specifically catch and re-throw AssertionErrors, or you will incapacitate JUnit.

JUnit Failures vs. Errors

JUnit test reports differentiate failures vs. errors. A failure is a test which your code has explicitly failed by using the mechanisms for that purpose (as described above). Generation of a failure indicates that your time investment is paying off-- it points you right to an anticipated problem in the program that is the target of your testing. A JUnit error, on the other hand, indicates an unanticipated problem. It is either a resource problem such as is normally the domain of Java unchecked throwables; or it is a problem with your implementation of the test. When you run a test and get an error, it means you really need to fix something, and usually not just the program that you intended to test. Either a problem has occurred outside of your test method (like in test class instantiation, or the test setup methods described in the next chapter), or your test method has thrown an exception (as described above, your methods should only throw unchecked exceptions if you have used the expected mechanism described above).

(With version 3, there used to be lifecycle side-effects when Errors were produced. I believe that these problems have been fixed with version 4).


Chapter 3. Class org.junit.Assert		Chapter 5. Test Classes and the JUnit Test Execution Environment