Overview Parameterized Tests with JUnit

Gerard Meszaros's describes in his great book with the title xUnit Test Patterns a pattern with the name Parameterized Test. This blog post describes how the pattern can be implemented in a JUnit 4.X test. The post compares three different options.

1. JUnit Parameterized Test

The JUnit framework comes with a runner for parameterized tests. The parameters are defined by a static function which is marked with the annotation @Parameters. The JUnit Runner is called Parameterized.

Example: 
Output - Eclipse JUnit View: 

Advantages
No extra framework or library for parameterized tests is needed. The JUnit view in eclipse and also in other IDEs works fine. One test with one defined parameter set can be invoked via the JUnit view in eclipse.

Disadvantages
The output from the tests is not clear. The output shows only the index number of the used test parameter. Only one test data model and parameter set per test class.

2. More JUnit Parameterized Test in a Test Class

It is possible to have more then one parameterized JUnit test in one test class by using the experimental runner "Enclosed".

Example: 
Output - Eclipse JUnit View:

Advantages
Grouping logic tests together in one class. Each test can be run from the JUnit view, a single test can be executed for debugging.

Disadvantages
Lots of boilerplate code of the embedded classes.

3. TwiP

TwiP is JUnit extension for parameterized tests. The library brings a JUnit runner, which is named "TwiP".

Example: 
Output - Eclipse JUnit View: 

Advantages
More then one parameterized test in a class is possible. Mixing parameterized and not parameterized tests is possible in one test class. Clear test output toString() method is used for the test parameters.

Disadvantages
A single test could not be chosen from the JUnit output (Eclipse JUnit view) and could not be invoked. This makes debugging the tests difficult because always all test combination must be executed to debug one failing test with one special combination.

4. JUnit Params

JUnit Params is another JUnit extension for parameterized tests like TwiP. The essential difference between TwiP and JUnit Params are the syntax how to define the parameterized tests.

Example: 
Output - Eclipse JUnit View: 

Advantages
Same advantages as TwiP and a little bit clearer test output then TwiP.

Disadvantages
Same disadvantages as TwiP a single test could not be invoked.

5. Summary

A perfect solution for a parameterized test in JUnit does not exists. TwiP and JUnitParams provided the way I like to write parameterized tests. But TwiP and also JUnitParams test cannot be debugged a single test can not be executed. I think the problems lies in the JUnit design for the descriptions and not in the TwiP or the JUnitParams project. Hope there will be a better solution in the future. So it's a difficult choice

Mockito the Java Standard Mocking Framework?

I used a long time JMock and I really love it, I also used a lot EasyMock because of the answer and capture feature. About a year ago I found the mocking framework mockito. After that I was able to inspire a few work colleagues to use mockito in their projets. I play around with it again and my conclusion is: Mockito is at the moment the best mocking framework for Java and I'm not alone.

"We decided during the main conference that we should use JUnit 4 and Mockito because we think they are the future of TDD and mocking in Java" - Dan North, the originator of BDD

Mockito has a clean syntax and the patterns and ideas behind mockito make tests more maintainable. Mockito has the power of EasyMock but a smarter syntax. Today I build for my self a simple JUnit test as mockito cheat sheet. So you convince yourself with Mockito and the smart API see the demo test bellow.


Links

• Mockito Project Site
  http://code.google.com/p/mockito/
• Documentation Mockito
  http://docs.mockito.googlecode.com/hg/latest/org/mockito/Mockito.html
• GitHub Project with the Cheat Sheet Test
  https://github.com/tux2323/mockito-cheat-sheet

Test Doubles and Mocking Patterns

Yesterday I refactored a unit test with over 2000 lines of code, now the test has around 900 lines of code. One pattern I found in the test was that for every kind of test double a mock object was used.

I prefer the following test design rule for mock objects and test doubles. Object which contains logic e.g. of type service, factory or repository should be mocked instead of DTOs, value objects or entities (plain Java Beans) with getters and setters e.g. configuration objects this kind of object should never be mocked.

Here small example, a test with to much mocking, please don't write such kind of tests:


This simple unit test above has to much mocking logic and should be refactored to this test:


If you like you could use the builder pattern for the simple objects like sale to get the test a little bit nicer (see the links for eclipse tooling). A smell is when you need to mock classes, in most cases then you should use another test double pattern for this kind of objects, or something with the design of the SUT is wrong.

What I like to say is you should not always use our preferred mocking framework to create a mock object as test double for everything. There are situations where mock object I believe are a test design anti pattern / smell. Think about stubs and dummy object before creating a mock object and read the great xUnit test pattern book “xUnit Test Patterns” from Gerard Meszaros. Then you are on the right way to get an agile tester ;-)

Links

• xUnit Paterns - http://xunitpatterns.com/
• Test Double - http://xunitpatterns.com/Test%20Double.html
• Builder Eclipse Plugin - http://code.google.com/p/fluent-builders-generator-eclipse-plugin/

JUnit > 4.10 Rules

It's been a long time ago that I last wrote about JUnit. A number of improvements and releases have been made. There have been lot improvements around the topic JUnit rules added. Today I had little bit free time to look into this new JUnit features.

Class Rules
Now there are also class rules in JUnit. A class rule extends the idea of test method-level rules and can be used to add logic, which should be invoked for or after all test methods. A simple use-case could be, when you need a HTTP server in the test, you could have a class rule to start and stop the server. Here a simple code snippet that starts and stops a HTTP server.



Test Sequence

1. Start HTTP Server
2. Run Test : invokeServletWithRequestOne()
3. Run Test : invokeServletWithRequestTwo()
4. Stop HTTP Server

Composition of JUnit Rules with RuleChain Feature
In JUnit 4.10 now you can order rules via a rule chain. This is really nice to reuse rules and combine them. Here a simple RuleChain code snippet.



Test Sequence

1. Start new Jetty Server (rule = ServerRule)
2. Create new Servlet Instance and add it to the Server (rule = ServletRule)
3. Run Test : invokeServletWithRequestOne()
4. Remove Servlet from Server (rule = ServletRule)
5. Stop Jetty Server (rule = ServerRule)
   
6. Start new Jetty Server (rule = ServerRule)
7. Create new Servlet Instance and add it to the Server (rule = ServletRule)
8. Run Test : invokeServletWithRequestTwo()
9. Remove Servlet from Server (rule = ServletRule)
10. Stop Jetty Server (rule = ServerRule)

Interface MethodRule is deprecated
The type MethodRule is now deprecated because the name makes no sense anymore because now we have class and method-level rules. A rule now should be of the type TestRule.

Thanks
Thanks at David Saff and Kent Beck for the work on JUnit and the cool rule features we have now in JUnit.

Links

• JUnit GitHub Project - https://github.com/KentBeck/junit
• JUnit Portal - http://www.junit.org/
• Blog Post about JUnit Rule and Logging - http://tux2323.blogspot.com/2011/06/test-logging-via-junit-rule.html

Sikuli – GUI Test Automation with Java Robot API and Images

The Sikuli project provides a simple tool for automate and test graphical user interfaces (GUI) using images (screenshots). The idea is to find an element on the screen by a screenshot and not by XPath or ID. The tool can be used for all GUIs also web applications. It’s a nice idea below you can see a simple example.

The SIKULI project is based on the Java VM and also provides an API for writing the automation in Java here a JUnit sample test.


For real world web testing it is not the right tool I think, because it depends on the style of the elements. When the button style changed you need a new screenshot of the button. But it is a nice tool for simple automation tasks. Have also a look at screen casts on the project homepage.

Links:
- Project Home - http://sikuli.org/
- How to use Sikuli Script in your JAVA programs - http://sikuli.org/docx/faq/030-java-dev.html

Logging in JUnit Tests

With a JUnit 4.7 rules it is easy to add logging support in JUnit tests. The example test bellow shows how to add simple logging for each test method and the result of the test.



For more details about logging in JUnit tests see the blog post "JUnit 4 Test Logging Tips using SLF4J".

Example - JMockContext - JUnit 4.7 Rule Support

JMock now contains in the SVN trunk (http://svn.codehaus.org/jmock/trunk/jmock2) a class org.jmock.integration.junit4.JMockContext, the class is a JUnit 4.7 rule. With this new JUnit rule for JMock no JUnit runner is needed to use JMock in JUnit tests.

Here a small example test which use the new JMockContext rule:

import java.util.Observer;

import org.jmock.Expectations;
import org.jmock.auto.Mock;
import org.jmock.integration.junit4.JMockContext;
import org.junit.Rule;
import org.junit.Test;

// JUH @RunWith is not needed 
public class ExampleJMockContextTest {
    
    @Rule public JMockContext context = new JMockContext();
    
    @Mock Observer mockObserver;
    
    @Test public void expectNoException(){
        context.checking(new Expectations(){{
            oneOf(mockObserver).update(null, null);
        }});
        
        mockObserver.update(null, null);
    }
    
    @Test(expected=RuntimeException.class)  
    public void expectExceptionInSUT() throws Exception {
        
        context.checking(new Expectations(){{
            oneOf(mockObserver).update(null, null);
        }});
        
        mockObserver.update(null, null);
        
        throw new RuntimeException("Exception in SUT");
    }
    
    @Test(expected=RuntimeException.class) 
    public void expectExceptionInMockObjectTest(){
        
        context.checking(new Expectations(){{
            oneOf(mockObserver).update(null, null);
            will(throwException(new RuntimeException("Observer unavailable")));
        }});
        
        mockObserver.update(null, null);
    }
    
}

JUnit 4.8 is released

JUnit News

http://www.junit.org/node/581

and here a the release notes

http://kentbeck.github.com/junit/doc/ReleaseNotes4.8.html

Example JUnit Test With Test Steps

Some times in a tests we want to have separate test steps that means to ensure the run of more then one statement e.g. assert statements.

Here a simple JUnit test, this tests show how such tests steps can look in a JUnit 4.X tests:

public class StackTest {

 Stack stack;

 @Before public void createStack() {
  stack = new Stack();
 }

 @Test public void push() {
  new Steps() {
   
   @Step public void setupPushElementsInStack() {
    stack.push("Element One");
    stack.push("Element Three");
    stack.push("Element X");
   }
   
   @Step public void verifyStackSizeIsThree() {
    assertEquals(2, stack.size());
   }
   
   @Step public void verifyStackElementOne() {
    assertElementIs(0, "Element One");
   }
   
   @Step public void verifyStackElementTwo() {
    assertElementIs(1, "XY");
   }
   
  };
 }

 private void assertElementIs(int position, String expected) {
  String element = stack.get(position);
  assertEquals(expected, element);
 }
}

And here the test result:

And here the code of the Steps class:

public class Steps {

 private List testStepFailures = new ArrayList();
 
 {
  run();
 }
 
 public final void run() {
  Method[] methods = getClass().getMethods();
  for (Method method : methods) {
   try 
   {
    Step step = method.getAnnotation(Step.class);
    if(step != null){
     method.invoke(this);
    }
   } 
   catch (Exception e) 
   {
    TestFailure failure = new TestFailure();
    failure.method = method;
    failure.exception = e;
    testStepFailures.add(failure);
   } 
  }
  if(testStepFailures.size() > 0){
   throw new TestStepsFailureException(testStepFailures);
  }
 }

}

JUnit Rule for jMock instead of the JMock Runner

In JUnit 4.7 there is new feature called "Rules" see my previous posts...

The jMock framework comes with a JUnit 4.X runner for verifying the mock object  states after each test run. Now with rules all the stuff of the runner can be done in one rule. I have implemented an example rule based on the JMock runner (see also jMock JIRA http://jira.codehaus.org/browse/JMOCK-237), here is the code of my example implementation:

import org.jmock.Mockery;
import org.jmock.lib.AssertionErrorTranslator;
import org.junit.Test;
import org.junit.rules.MethodRule;
import org.junit.runners.model.FrameworkMethod;
import org.junit.runners.model.Statement;

public class JUnitRuleMockery extends Mockery implements MethodRule {

 public JUnitRuleMockery() {
    setExpectationErrorTranslator(AssertionErrorTranslator.INSTANCE);
 }
 
 public Statement apply(final Statement base, final FrameworkMethod method,
   Object target) {
  return new Statement() {
   @Override
   public void evaluate() throws Throwable {
    try {
     base.evaluate();
    } catch(Throwable exp) {
     if(!isExceptionExpectedByTestMethod(exp, method)) {
      throw exp;
     }
    }
    assertIsSatisfied();
   }
  };
 }

 protected boolean isExceptionExpectedByTestMethod(Throwable exp, FrameworkMethod method) {
  Test test = method.getAnnotation(Test.class);
  return test.expected().equals(exp.getClass());
 }

}

Example Test which use the rule:

import org.eclipse.osgi.framework.console.CommandInterpreter;
import org.jmock.Expectations;
import org.jmock.Mockery;
import org.jmock.integration.junit4.JUnit4Mockery;
import org.jmock.lib.legacy.ClassImposteriser;
import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;

// No jMock specific Runner needed only the default JUnit runner
public class FrameworkCommandProviderTest { 

@Rule
    public JUnitRuleMockery context = new JUnitRuleMockery() {{
       setImposteriser(ClassImposteriser.INSTANCE);
    }};
  
 InternalSystemBundle mockInternalSystemBundle;
 
 Framework mockFramework;
 
 CommandInterpreter mockCommandInterpreter;
 
 FrameworkCommandProvider mockFrameworkCommandProvider;
 
 FrameworkCommandProvider frameworkCommandProvider;
 
 @Before 
 public void setUp(){
  
  // Create mock object for system bundle
  mockInternalSystemBundle = context.mock(InternalSystemBundle.class);
  // Mocking the getContext Method, returns always null
  context.checking(new Expectations() {{
   allowing (mockInternalSystemBundle).getContext(); will(returnValue(null));
  }});
  
  // Create mock object for framework
  mockFramework = context.mock(Framework.class);
  // Set the mock object system bundle into the framework mock
  mockFramework.systemBundle = mockInternalSystemBundle;
  
  // Create mock object for command interpreter
  mockCommandInterpreter = context.mock(CommandInterpreter.class);
  
  // Create mock object for the SUT class, this mock will be used to mock methods in the SUT
  mockFrameworkCommandProvider = context.mock(FrameworkCommandProvider.class);
  
  // Create the SUT object of type FrameworkCommandProvider
  frameworkCommandProvider = new FrameworkCommandProvider(mockFramework) {
   
   /** 
    * Mock the SUT Method getBundleFromToken(...)
    * All method calls will delegate to a mock 
    * of the same type then the SUT object 
    */
   @Override
   protected AbstractBundle getBundleFromToken(CommandInterpreter intp, String token, boolean error) {
    // delegate the call to a mock object.
    return mockFrameworkCommandProvider.getBundleFromToken(intp, token, error);
   }
  };
  
 }
 
 @Test(expected=RuntimeException.class) 
 public void _bundle_OneArgumentsAndBundleNotFoundByToken() throws Exception { 
  final String firstToken = "myBundleName[1.0.0]";
  // Setup mock context
  context.checking(new Expectations() {{
   // one argument
   one (mockCommandInterpreter).nextArgument(); will(returnValue(firstToken));
   // next calles return null
   allowing (mockCommandInterpreter).nextArgument(); will(returnValue(null));
   
   // getBundleFromToken returns null
   one (mockFrameworkCommandProvider).getBundleFromToken(mockCommandInterpreter, firstToken, true);
   will(returnValue(null));
   
  }});
  
  // invoke the _bundle() method
  frameworkCommandProvider._bundle(mockCommandInterpreter);
  
  throw new RuntimeException();
 }

}

xUnit test implementation pattern for mocking methods in a SUT object

Simple test implementation pattern for mock a method in the SUT object. The pattern is interesting when  you would refactor a method in legacy system. Before you can start with the refactoring stuff you must write a test for the method with a high coverage (Refactoring details see Martin Fowlers great book "Refactoring - Improving the Design of Existing Code").

The idea is to test only the legacy method you would like to refactor (to get quick a high coverage),  there for we want mock all method calls which will be invoked from this method. Normal mocking frameworks like EasyMock or JMock for Java can create mock objects for interfaces and classes. This pattern shows how this tools can be used for mock an internal method in the SUT object.

The idea, overwrite all method which should be mocked in the SUT object and delegate the calls to a mock object of the same type as the SUT object.

Example setup code with JMock as mocking framework:
...

@Before 
public void setUp(){
   ...
   // Create mock object for the SUT class, this mock will be used to mock methods in the SUT
   mockFrameworkCommandProvider = context.mock(FrameworkCommandProvider.class);
  
   // Create the SUT object of type FrameworkCommandProvider
   frameworkCommandProvider = new FrameworkCommandProvider(mockFramework) {
   
    /** 
     * Mock the SUT Method getBundleFromToken(...)
     * All method calls will delegate to a mock 
     * of the same type then the SUT object 
     */
     @Override
     protected AbstractBundle getBundleFromToken(CommandInterpreter intp, String token, boolean error) {
         // delegate the call to a mock object.
         return mockFrameworkCommandProvider.getBundleFromToken(intp, token, error);
     }
   };
}
...

The code with a example test which expected a mock method call:
...

@Test 
public void _bundle_OneArgumentsAndBundleNotFoundByToken() throws Exception { 
 final String firstToken = "myBundleName[1.0.0]";
 // Setup mock context
 context.checking(new Expectations() {{
  // one argument
  one (mockCommandInterpreter).nextArgument(); will(returnValue(firstToken));
  // next calles return null
  allowing (mockCommandInterpreter).nextArgument(); will(returnValue(null));
   
  // getBundleFromToken returns null
  one (mockFrameworkCommandProvider).getBundleFromToken(mockCommandInterpreter, firstToken, true);
  will(returnValue(null));
 }});
 
 // invoke the _bundle() method
 frameworkCommandProvider._bundle(mockCommandInterpreter);
}
... 

Downsides:

• Much code for the test setup to overwrite all the methods which should be mocked in the SUT object
• Only protected, package and public methods can be mocked by this approach
• Private methods can't be mock with this pattern
• In the test code it's not clear which SUT methods are mocked and which not

Here the whole example test code:

package org.eclipse.osgi.framework.internal.core;

import static org.junit.Assert.*;

import org.eclipse.osgi.framework.console.CommandInterpreter;
import org.jmock.Expectations;
import org.jmock.Mockery;
import org.jmock.integration.junit4.JMock;
import org.jmock.integration.junit4.JUnit4Mockery;
import org.jmock.lib.legacy.ClassImposteriser;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;

@RunWith(JMock.class)
public class FrameworkCommandProviderTest {

 // Create mock context and activate mocking for java classes.
 Mockery context = new JUnit4Mockery() {{
     setImposteriser(ClassImposteriser.INSTANCE);
    }};
 
    InternalSystemBundle mockInternalSystemBundle;
 
 Framework mockFramework;
 
 CommandInterpreter mockCommandInterpreter;
 
 FrameworkCommandProvider mockFrameworkCommandProvider;
 
 FrameworkCommandProvider frameworkCommandProvider;
 
 @Before 
 public void setUp(){
  
  // Create mock object for system bundle
  mockInternalSystemBundle = context.mock(InternalSystemBundle.class);
  // Mocking the getContext Method, returns always null
  context.checking(new Expectations() {{
   allowing (mockInternalSystemBundle).getContext(); will(returnValue(null));
  }});
  
  // Create mock object for framework
  mockFramework = context.mock(Framework.class);
  // Set the mock object system bundle into the framework mock
  mockFramework.systemBundle = mockInternalSystemBundle;
  
  // Create mock object for command interpreter
  mockCommandInterpreter = context.mock(CommandInterpreter.class);
  
  // Create mock object for the SUT class, this mock will be used to mock methods in the SUT
  mockFrameworkCommandProvider = context.mock(FrameworkCommandProvider.class);
  
  // Create the SUT object of type FrameworkCommandProvider
  frameworkCommandProvider = new FrameworkCommandProvider(mockFramework) {
   
   /** 
    * Mock the SUT Method getBundleFromToken(...)
    * All method calls will delegate to a mock 
    * of the same type then the SUT object 
    */
   @Override
   protected AbstractBundle getBundleFromToken(CommandInterpreter intp, String token, boolean error) {
    // delegate the call to a mock object.
    return mockFrameworkCommandProvider.getBundleFromToken(intp, token, error);
   }
  };
  
 }
 
 @Test 
 public void _bundle_OneArgumentsAndBundleNotFoundByToken() throws Exception { 
  final String firstToken = "myBundleName[1.0.0]";
  // Setup mock context
  context.checking(new Expectations() {{
   // one argument
   one (mockCommandInterpreter).nextArgument(); will(returnValue(firstToken));
   // next calles return null
   allowing (mockCommandInterpreter).nextArgument(); will(returnValue(null));
   
   // getBundleFromToken returns null
   one (mockFrameworkCommandProvider).getBundleFromToken(mockCommandInterpreter, firstToken, true);
   will(returnValue(null));
  }});
  
  // invoke the _bundle() method
  frameworkCommandProvider._bundle(mockCommandInterpreter);
 }

}

Run the Example Test:

To run the test you need the equinox osgi implementation as SUT, you can get the equinox source from the eclipse CVS ("pserver:dev.eclipse.org/cvsroot/rt/org.eclipse.equinox/framework/bundles/org.eclipse.osgi/").

Links:

• JMock
• JUnit
• Eclipse Equinox