WebTestingExplorer Documentation Moving Forward

I started writing some documentation for WebTestingExplorer in the site Wiki. Because I would like people to start trying it out, I'm hoping that it will help them get started without necessarily having to pour over all the code and Javadocs. The core API documentation is (and always will be) in the Javadocs, but I will use the Wiki to explain some of the higher-level concepts and design philosophies. So if you get the chance, download, build, try it out, and, of course, let me know how it goes.

Suppressing Firefox favicon Requests from WebDriver

Right now, WebTestingExplorer only works with Firefox. There are a couple of reasons for that which I won't go into here. Anyway, we decided to wire our HTTP requests to go through a proxy by default. This allows us to look for specific status codes, presumably "undesirable" ones like the 500-series, or possibly 404's.

When we did this and pointed the tool at a specific URL, the first thing we noticed was Firefox firing off a bunch of requests to various locations on the target site fishing around for the favicon, most of which (kind of by definition) ended with a 404. I'm surprised that for as long as I have used Firefox attached to a proxy for testing and debugging, I had never taken note of this before. In this case, it's really annoying -- after all, one of the points of using the proxy in the first place was to verify that all requests came back with a valid response.

The first thing I considered was adding special-case code for all favicon requests in the handlers for the proxy. That seemed kind of lame, so the second thing I thought about was looking for a way to suppress the requests in the first place. It turns out that there are Firefox preferences for this -- browser.chrome.favicons and browser.chrome.site_icons -- as described here.

I have two main points left for this post. First, I wanted to show the code for how set these preferences in Selenium WebDriver when you build your FirefoxProfile:

final FirefoxProfile profile = new FirefoxProfile(); // The following preferences control Firefox's default behavior of sending // requests for favicon.ico, which results in lots of bogus 404's for sites // that don't have favicons.  profile.setPreference("browser.chrome.favicons", false); profile.setPreference("browser.chrome.site_icons", false);      FirefoxDriver driver = new FirefoxDriver(profile);

Second, I will pitch my opinion that in most automated testing scenarios, you probably ought to be suppressing favicon requests from Firefox, even if your site actually has one. A good way to do this is to save the preferences into a custom Firefox profile used for automated testing. If you don't already have one of those, you should. I will try to provide some tips on how to maintain and use one of those in a future post.

Introducing WebTestingExplorer

My friend and I recently started working on a new open-source project, WebTestingExplorer (hosted on code.google.com). We are setting out to do a high-quality, industrial-strength implementation of some of the ideas we worked on in graduate school at the University of Maryland, specifically applied to web application testing. The super-high-level idea is to incrementally "explore" the web user interface, simultaneously looking for bugs and generating test cases. Although the project is in a very early state, the current implementation can do a few useful things, such as:

• Detecting unexpected HTTP status codes and Javascript errors during exploration.
• Generating and replaying regression test cases, looking for differences in state.

A few tenets we have adopted are:

• Most everything regarding the exploration and test case replay process is configurable.
• Configuration is done primarily via code, not command-line arguments or XML files.
• The application-under-test is inherently "testable" (or we can make it so), such as having unique id's for important HTML elements, and hooks that enable reliable timing of actions.

If you are interested in automated test case generation and/or web application testing, feel free to start playing with it and sending feedback through the Google Code project. Although the primary vehicle for project communication will be the Project Wiki, I hope to blog here about some of the interesting issues we encounter. So until next time...

EasyMock Issue #2: Bad Argument equals()

One of the two canonical EasyMock failure messages (along with "Expectation failure on verify" is "Unexpected method call". This failure happens between replay() and verify() whenever EasyMock sees the class-under-test making a method call on a mock object for which the test did not set up an expectation. Generally, there are two possible problems to immediately look for:

1. A deficiency in the test, i.e. the behavior of the class-under-test is not correctly modeled during the expectation-setting phase of the test case.
2. A bug in the class-under-test, i.e. the test is expecting the right thing but the code isn't doing it.

(Incidentally, when it turns out to be #1, programmers tend to complain about how difficult it is to write and maintain mock-object tests, and when it is instead #2, they often fail to give mock-object-testing enough credit for detecting their bug.)

But there is a third possibility, which I would argue is neither a bug in the test nor in the class-under-test, but rather a bug in a completely separate, possibly innocuous-looking class that they both use. As usual, let's look at an example.

Here is the class-under-test, CustomerController for this post (slightly modified from the one I used in the previous two posts:

public class CustomerController {   private OrderService orderService;   public CustomerController(OrderService orderService) {     this.orderService = orderService;   }      public void postOrders(Order...orders) {     orderService.postOrders(Arrays.asList(orders));   } }

And the test for the postOrders() method, which uses a mock OrderService:

public class TestCustomerController {   @Test   public void testPostOrders() {     OrderService mockOrderService = EasyMock.createMock(OrderService.class);     CustomerController controller = new CustomerController(mockOrderService);     List<Order> expectedOrders = new ArrayList<Order>();     expectedOrders.add(new Order(1));          EasyMock.expect(mockOrderService.postOrders(expectedOrders)).andReturn(1);     EasyMock.replay(mockOrderService);     controller.postOrders(new Order(1));   } }

It is also important to note that we are posting Order instances:

public class Order {   private int orderId;   public Order(int orderId) {     this.orderId = orderId;   } }

When you run this, alas, you get an error:

java.lang.AssertionError:

Unexpected method call postOrders([Order@1e4cbc4]):

postOrders([Order@b2fd8f]): expected: 1, actual: 0

at org.easymock.internal.MockInvocationHandler.invoke(MockInvocationHandler.java:45)

at org.easymock.internal.ObjectMethodsFilter.invoke(ObjectMethodsFilter.java:73)

at org.easymock.internal.ClassProxyFactory$MockMethodInterceptor.intercept(ClassProxyFactory.java:92)

at OrderService$$EnhancerByCGLIB$$6d00fba4.postOrders()

at CustomerController.postOrders(CustomerController.java:11)

at TestCustomerController.testPostOrders(TestCustomerController.java:19)

The CustomerController code is quite trivial and correct in this case. The test is less trivial but also, well, at least ought to work. So what is going on here?

When EasyMock compares a mock method call to the expectation, it checks equality on each argument. In this case, that fails because the Order instance in the expectation does not equal the Order instance in the invocation. Note that Order is not overriding equals(), and since I set the test up to use different instances, the default instance-equality check fails. This example is pretty contrived. In reality, the more common case I've seen is that there actually is an equals() override, but somebody changes the class and breaks it, in which case the test is finding a real bug (albeit not in the actual class-under-test!).

There are two ways to fix this problem. First, I could override equals() (and hashCode(), of course) if it doesn't already exist on the class in question (the case with Order) or fix equals() if it were already there. That solution is probably more robust. The quick-and-dirty fix is to make the expectation and the invocation use the exact same instance(s). I will actually show the code for that:

public class TestCustomerController {   @Test   public void testPostOrders() {     OrderService mockOrderService = EasyMock.createMock(OrderService.class);     CustomerController controller = new CustomerController(mockOrderService);     List<Order> expectedOrders = new ArrayList<Order>();          // I will use this here AND below...     Order order = new Order(1);     expectedOrders.add(order);          EasyMock.expect(mockOrderService.postOrders(expectedOrders)).andReturn(1);     EasyMock.replay(mockOrderService);     controller.postOrders(order);   } }

Sorry if I've rambled on for too long about something that is actually quite obvious. But I have made and seen others make this type of error enough times that I can hopefully save someone some time. Next time, I will discuss an EasyMock limitation that might impact your interface design.

EasyMock Issue #1: Missing Behavior Definition

I am planning to do a series of posts (including examples) on issues that people (myself included) run into while using EasyMock. The first of these involves EasyMock throwing an IllegalStateException "missing behavior definition for the preceding method call". The executive summary is that this is caused by the omission of a return value specification (andReturn(...)). If that is all you need to know, thanks for coming. If you would like to see a specific example, keep reading.

Let's say we are writing unit tests for a CustomerController class that uses an OrderService class to post a Customer's Orders:

// OrderService.java public class OrderService {   public int postOrders(List<Order> orders) {     return orders.size();   } } // CustomerController.java public class CustomerController {   private OrderService orderService;   public CustomerController(OrderService orderService) {     this.orderService = orderService;   }      public void postOrders(Integer...orderIds) {     List<Order> orderList = new ArrayList<Order>();     for (int orderId : orderIds) {       orderList.add(new Order(orderId));     }     orderService.postOrders(orderList);   } }

In the real world, OrderService probably connects to external resources and therefore we want to mock it when creating a unit test for CustomerController. The interaction between CustomerController and OrderService consists of a single call to the OrderService.postOrders(List) method. Simple enough, here is our first attempt at writing this test:

public class TestCustomerController {   @Test   public void testPostOrders() {     OrderService mockOrderService = EasyMock.createMock(OrderService.class);     CustomerController controller = new CustomerController(mockOrderService);     List<Order> expectedOrders = new ArrayList<Order>();     expectedOrders.add(new Order(1));          // You may do the following assuming postOrders returns void:     mockOrderService.postOrders(expectedOrders);     EasyMock.expectLastCall();          // But unfortunately it doesn't.     EasyMock.replay(mockOrderService);     controller.postOrders(1);   } }

When you run this, it blows up with the following exception:

java.lang.IllegalStateException: missing behavior definition for the preceding method call postOrders([Order@82c01f])
 at org.easymock.internal.MocksControl.replay(MocksControl.java:174)
 at org.easymock.EasyMock.replay(EasyMock.java:1970)
 at TestCustomerController.testPostOrders(TestCustomerController.java:22)

Since I gave away the punch line in the opening paragraph of this post, you know that this happens because OrderService.postOrders() is not a void method; rather, it returns an int. I would argue that this is a pretty easy mistake to make, though, because the implementation of CustomerController does not care about or capture this return value. So unless we're paying pretty close attention to the OrderService interface, we might assume it returns void. And unfortunately, EasyMock requires that ALL non-void method calls on mocks have return values specified, even if the class under test will ignore it.

So the correct way to set up the OrderService.postOrders() expectation is as follows:

EasyMock.expect(mockOrderService.postOrders(expectedOrders)).andReturn(1);

In my next "EasyMock Issues" post, I will talk about a couple of things to look for when your expectations are mysteriously failing to match.