Sikuli – GUI Test Automation with Java Robot API and Images

The Sikuli project provides a simple tool for automate and test graphical user interfaces (GUI) using images (screenshots). The idea is to find an element on the screen by a screenshot and not by XPath or ID. The tool can be used for all GUIs also web applications. It’s a nice idea below you can see a simple example.

The SIKULI project is based on the Java VM and also provides an API for writing the automation in Java here a JUnit sample test.


For real world web testing it is not the right tool I think, because it depends on the style of the elements. When the button style changed you need a new screenshot of the button. But it is a nice tool for simple automation tasks. Have also a look at screen casts on the project homepage.

Links:
- Project Home - http://sikuli.org/
- How to use Sikuli Script in your JAVA programs - http://sikuli.org/docx/faq/030-java-dev.html

Example - JMockContext - JUnit 4.7 Rule Support

JMock now contains in the SVN trunk (http://svn.codehaus.org/jmock/trunk/jmock2) a class org.jmock.integration.junit4.JMockContext, the class is a JUnit 4.7 rule. With this new JUnit rule for JMock no JUnit runner is needed to use JMock in JUnit tests.

Here a small example test which use the new JMockContext rule:

import java.util.Observer;

import org.jmock.Expectations;
import org.jmock.auto.Mock;
import org.jmock.integration.junit4.JMockContext;
import org.junit.Rule;
import org.junit.Test;

// JUH @RunWith is not needed 
public class ExampleJMockContextTest {
    
    @Rule public JMockContext context = new JMockContext();
    
    @Mock Observer mockObserver;
    
    @Test public void expectNoException(){
        context.checking(new Expectations(){{
            oneOf(mockObserver).update(null, null);
        }});
        
        mockObserver.update(null, null);
    }
    
    @Test(expected=RuntimeException.class)  
    public void expectExceptionInSUT() throws Exception {
        
        context.checking(new Expectations(){{
            oneOf(mockObserver).update(null, null);
        }});
        
        mockObserver.update(null, null);
        
        throw new RuntimeException("Exception in SUT");
    }
    
    @Test(expected=RuntimeException.class) 
    public void expectExceptionInMockObjectTest(){
        
        context.checking(new Expectations(){{
            oneOf(mockObserver).update(null, null);
            will(throwException(new RuntimeException("Observer unavailable")));
        }});
        
        mockObserver.update(null, null);
    }
    
}

JUnit 4.8 is released

JUnit News

http://www.junit.org/node/581

and here a the release notes

http://kentbeck.github.com/junit/doc/ReleaseNotes4.8.html

Example JUnit Test With Test Steps

Some times in a tests we want to have separate test steps that means to ensure the run of more then one statement e.g. assert statements.

Here a simple JUnit test, this tests show how such tests steps can look in a JUnit 4.X tests:

public class StackTest {

 Stack stack;

 @Before public void createStack() {
  stack = new Stack();
 }

 @Test public void push() {
  new Steps() {
   
   @Step public void setupPushElementsInStack() {
    stack.push("Element One");
    stack.push("Element Three");
    stack.push("Element X");
   }
   
   @Step public void verifyStackSizeIsThree() {
    assertEquals(2, stack.size());
   }
   
   @Step public void verifyStackElementOne() {
    assertElementIs(0, "Element One");
   }
   
   @Step public void verifyStackElementTwo() {
    assertElementIs(1, "XY");
   }
   
  };
 }

 private void assertElementIs(int position, String expected) {
  String element = stack.get(position);
  assertEquals(expected, element);
 }
}

And here the test result:

And here the code of the Steps class:

public class Steps {

 private List testStepFailures = new ArrayList();
 
 {
  run();
 }
 
 public final void run() {
  Method[] methods = getClass().getMethods();
  for (Method method : methods) {
   try 
   {
    Step step = method.getAnnotation(Step.class);
    if(step != null){
     method.invoke(this);
    }
   } 
   catch (Exception e) 
   {
    TestFailure failure = new TestFailure();
    failure.method = method;
    failure.exception = e;
    testStepFailures.add(failure);
   } 
  }
  if(testStepFailures.size() > 0){
   throw new TestStepsFailureException(testStepFailures);
  }
 }

}

JUnit Rule for jMock instead of the JMock Runner

In JUnit 4.7 there is new feature called "Rules" see my previous posts...

The jMock framework comes with a JUnit 4.X runner for verifying the mock object  states after each test run. Now with rules all the stuff of the runner can be done in one rule. I have implemented an example rule based on the JMock runner (see also jMock JIRA http://jira.codehaus.org/browse/JMOCK-237), here is the code of my example implementation:

import org.jmock.Mockery;
import org.jmock.lib.AssertionErrorTranslator;
import org.junit.Test;
import org.junit.rules.MethodRule;
import org.junit.runners.model.FrameworkMethod;
import org.junit.runners.model.Statement;

public class JUnitRuleMockery extends Mockery implements MethodRule {

 public JUnitRuleMockery() {
    setExpectationErrorTranslator(AssertionErrorTranslator.INSTANCE);
 }
 
 public Statement apply(final Statement base, final FrameworkMethod method,
   Object target) {
  return new Statement() {
   @Override
   public void evaluate() throws Throwable {
    try {
     base.evaluate();
    } catch(Throwable exp) {
     if(!isExceptionExpectedByTestMethod(exp, method)) {
      throw exp;
     }
    }
    assertIsSatisfied();
   }
  };
 }

 protected boolean isExceptionExpectedByTestMethod(Throwable exp, FrameworkMethod method) {
  Test test = method.getAnnotation(Test.class);
  return test.expected().equals(exp.getClass());
 }

}

Example Test which use the rule:

import org.eclipse.osgi.framework.console.CommandInterpreter;
import org.jmock.Expectations;
import org.jmock.Mockery;
import org.jmock.integration.junit4.JUnit4Mockery;
import org.jmock.lib.legacy.ClassImposteriser;
import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;

// No jMock specific Runner needed only the default JUnit runner
public class FrameworkCommandProviderTest { 

@Rule
    public JUnitRuleMockery context = new JUnitRuleMockery() {{
       setImposteriser(ClassImposteriser.INSTANCE);
    }};
  
 InternalSystemBundle mockInternalSystemBundle;
 
 Framework mockFramework;
 
 CommandInterpreter mockCommandInterpreter;
 
 FrameworkCommandProvider mockFrameworkCommandProvider;
 
 FrameworkCommandProvider frameworkCommandProvider;
 
 @Before 
 public void setUp(){
  
  // Create mock object for system bundle
  mockInternalSystemBundle = context.mock(InternalSystemBundle.class);
  // Mocking the getContext Method, returns always null
  context.checking(new Expectations() {{
   allowing (mockInternalSystemBundle).getContext(); will(returnValue(null));
  }});
  
  // Create mock object for framework
  mockFramework = context.mock(Framework.class);
  // Set the mock object system bundle into the framework mock
  mockFramework.systemBundle = mockInternalSystemBundle;
  
  // Create mock object for command interpreter
  mockCommandInterpreter = context.mock(CommandInterpreter.class);
  
  // Create mock object for the SUT class, this mock will be used to mock methods in the SUT
  mockFrameworkCommandProvider = context.mock(FrameworkCommandProvider.class);
  
  // Create the SUT object of type FrameworkCommandProvider
  frameworkCommandProvider = new FrameworkCommandProvider(mockFramework) {
   
   /** 
    * Mock the SUT Method getBundleFromToken(...)
    * All method calls will delegate to a mock 
    * of the same type then the SUT object 
    */
   @Override
   protected AbstractBundle getBundleFromToken(CommandInterpreter intp, String token, boolean error) {
    // delegate the call to a mock object.
    return mockFrameworkCommandProvider.getBundleFromToken(intp, token, error);
   }
  };
  
 }
 
 @Test(expected=RuntimeException.class) 
 public void _bundle_OneArgumentsAndBundleNotFoundByToken() throws Exception { 
  final String firstToken = "myBundleName[1.0.0]";
  // Setup mock context
  context.checking(new Expectations() {{
   // one argument
   one (mockCommandInterpreter).nextArgument(); will(returnValue(firstToken));
   // next calles return null
   allowing (mockCommandInterpreter).nextArgument(); will(returnValue(null));
   
   // getBundleFromToken returns null
   one (mockFrameworkCommandProvider).getBundleFromToken(mockCommandInterpreter, firstToken, true);
   will(returnValue(null));
   
  }});
  
  // invoke the _bundle() method
  frameworkCommandProvider._bundle(mockCommandInterpreter);
  
  throw new RuntimeException();
 }

}

xUnit test implementation pattern for mocking methods in a SUT object

Simple test implementation pattern for mock a method in the SUT object. The pattern is interesting when  you would refactor a method in legacy system. Before you can start with the refactoring stuff you must write a test for the method with a high coverage (Refactoring details see Martin Fowlers great book "Refactoring - Improving the Design of Existing Code").

The idea is to test only the legacy method you would like to refactor (to get quick a high coverage),  there for we want mock all method calls which will be invoked from this method. Normal mocking frameworks like EasyMock or JMock for Java can create mock objects for interfaces and classes. This pattern shows how this tools can be used for mock an internal method in the SUT object.

The idea, overwrite all method which should be mocked in the SUT object and delegate the calls to a mock object of the same type as the SUT object.

Example setup code with JMock as mocking framework:
...

@Before 
public void setUp(){
   ...
   // Create mock object for the SUT class, this mock will be used to mock methods in the SUT
   mockFrameworkCommandProvider = context.mock(FrameworkCommandProvider.class);
  
   // Create the SUT object of type FrameworkCommandProvider
   frameworkCommandProvider = new FrameworkCommandProvider(mockFramework) {
   
    /** 
     * Mock the SUT Method getBundleFromToken(...)
     * All method calls will delegate to a mock 
     * of the same type then the SUT object 
     */
     @Override
     protected AbstractBundle getBundleFromToken(CommandInterpreter intp, String token, boolean error) {
         // delegate the call to a mock object.
         return mockFrameworkCommandProvider.getBundleFromToken(intp, token, error);
     }
   };
}
...

The code with a example test which expected a mock method call:
...

@Test 
public void _bundle_OneArgumentsAndBundleNotFoundByToken() throws Exception { 
 final String firstToken = "myBundleName[1.0.0]";
 // Setup mock context
 context.checking(new Expectations() {{
  // one argument
  one (mockCommandInterpreter).nextArgument(); will(returnValue(firstToken));
  // next calles return null
  allowing (mockCommandInterpreter).nextArgument(); will(returnValue(null));
   
  // getBundleFromToken returns null
  one (mockFrameworkCommandProvider).getBundleFromToken(mockCommandInterpreter, firstToken, true);
  will(returnValue(null));
 }});
 
 // invoke the _bundle() method
 frameworkCommandProvider._bundle(mockCommandInterpreter);
}
... 

Downsides:

• Much code for the test setup to overwrite all the methods which should be mocked in the SUT object
• Only protected, package and public methods can be mocked by this approach
• Private methods can't be mock with this pattern
• In the test code it's not clear which SUT methods are mocked and which not

Here the whole example test code:

package org.eclipse.osgi.framework.internal.core;

import static org.junit.Assert.*;

import org.eclipse.osgi.framework.console.CommandInterpreter;
import org.jmock.Expectations;
import org.jmock.Mockery;
import org.jmock.integration.junit4.JMock;
import org.jmock.integration.junit4.JUnit4Mockery;
import org.jmock.lib.legacy.ClassImposteriser;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;

@RunWith(JMock.class)
public class FrameworkCommandProviderTest {

 // Create mock context and activate mocking for java classes.
 Mockery context = new JUnit4Mockery() {{
     setImposteriser(ClassImposteriser.INSTANCE);
    }};
 
    InternalSystemBundle mockInternalSystemBundle;
 
 Framework mockFramework;
 
 CommandInterpreter mockCommandInterpreter;
 
 FrameworkCommandProvider mockFrameworkCommandProvider;
 
 FrameworkCommandProvider frameworkCommandProvider;
 
 @Before 
 public void setUp(){
  
  // Create mock object for system bundle
  mockInternalSystemBundle = context.mock(InternalSystemBundle.class);
  // Mocking the getContext Method, returns always null
  context.checking(new Expectations() {{
   allowing (mockInternalSystemBundle).getContext(); will(returnValue(null));
  }});
  
  // Create mock object for framework
  mockFramework = context.mock(Framework.class);
  // Set the mock object system bundle into the framework mock
  mockFramework.systemBundle = mockInternalSystemBundle;
  
  // Create mock object for command interpreter
  mockCommandInterpreter = context.mock(CommandInterpreter.class);
  
  // Create mock object for the SUT class, this mock will be used to mock methods in the SUT
  mockFrameworkCommandProvider = context.mock(FrameworkCommandProvider.class);
  
  // Create the SUT object of type FrameworkCommandProvider
  frameworkCommandProvider = new FrameworkCommandProvider(mockFramework) {
   
   /** 
    * Mock the SUT Method getBundleFromToken(...)
    * All method calls will delegate to a mock 
    * of the same type then the SUT object 
    */
   @Override
   protected AbstractBundle getBundleFromToken(CommandInterpreter intp, String token, boolean error) {
    // delegate the call to a mock object.
    return mockFrameworkCommandProvider.getBundleFromToken(intp, token, error);
   }
  };
  
 }
 
 @Test 
 public void _bundle_OneArgumentsAndBundleNotFoundByToken() throws Exception { 
  final String firstToken = "myBundleName[1.0.0]";
  // Setup mock context
  context.checking(new Expectations() {{
   // one argument
   one (mockCommandInterpreter).nextArgument(); will(returnValue(firstToken));
   // next calles return null
   allowing (mockCommandInterpreter).nextArgument(); will(returnValue(null));
   
   // getBundleFromToken returns null
   one (mockFrameworkCommandProvider).getBundleFromToken(mockCommandInterpreter, firstToken, true);
   will(returnValue(null));
  }});
  
  // invoke the _bundle() method
  frameworkCommandProvider._bundle(mockCommandInterpreter);
 }

}

Run the Example Test:

To run the test you need the equinox osgi implementation as SUT, you can get the equinox source from the eclipse CVS ("pserver:dev.eclipse.org/cvsroot/rt/org.eclipse.equinox/framework/bundles/org.eclipse.osgi/").

Links:

• JMock
• JUnit
• Eclipse Equinox