Validating WebDriver Tests Without the WebDriver API

One thing we have noticed working on WebTestingExplorer is just how excruciatingly slow WebDriver can be. Perhaps you have never noticed this because you only validate a few elements at the end of a WebDriver test. But in WebTestingExplorer use cases, we often need to examine hundreds and hundreds of WebElements and their attribute values after each action. And here, WebDriver's architecture where every call to the browser is a local RPC is a massive fail in terms of performance (and, I think, stability, although that is harder to pin down).

This is one of the (many) reasons I wish the testing practice was not standardizing on WebDriver: I feel like browser automation is far from a solved problem and would like to see what other smart people can continue to put forward. Nonetheless, we feel like we have to use it for WebTestingExplorer, both to provide synergy and integration between our tool and other testing efforts, and because there just aren't a lot of good alternatives under active development.

That said, we had been theorizing for awhile that we could improve life significantly if we started trying to implement WebTestingExplorer state checking and oracles on top of an alternative HTML-parsing library. A couple of weeks ago, I selected jsoup as the library and finally took a shot at this. The results were really incredible -- an order of magnitude or more speedup in many of our exploration and bug-checking scenarios, and fewer (read: zero) inexplicable, random exceptions flying out of WebDriver. Going forward, I hope to try and rework some of the WebTestingExplorer APIs to make this more natural.

But even if you're not using WebTestingExplorer, I think the overall approach is worth considering for any web unit/system/integration testing you might be doing. The idea is to command the browser using WebDriver as you normally do. Then, having completed that, grab the current DOM HTML of the page with WebDriver like this:

WebDriver driver; // ... JavascriptExecutor js = (JavascriptExecutor) driver; String domString = (String) js.executeScript("return document.documentElement.outerHTML;");

Done with WebDriver, feed that to the jsoup parser and enjoy its simple, fast API while writing your assertions. Yeah, you lose all the fancy element-is-displayed logic that WebDriver is constantly trying to perform, but hopefully (as in, you would think) it's not too hard to write assertions that don't care much about that. Enjoy!

Declaring Instances as Fields in Interfaces is a Testability Problem

Typically, when you see fields defined in Java interfaces, they are things you would normally think of as "constants" -- e.g. int's, Strings, maybe an enum or two. I had never really thought much about the fact that you can have instances of more complex classes as interface fields until I actually saw it in practice. Something like this:

public interface MyInterface {   // Singleton instance.   MyInterfaceImpl INSTANCE = new MyInterfaceImpl();      // Interface methods...   void doSomething(); }

Then you can use that field like a static variable, most likely as an old-school (pre-dependency-injection) singleton instance.

There are two problems with this as far as testability is concerned. First, you have the usual difficulty you get with static instances in unit tests, namely, that you cannot insert a mock, stub, or fake in all of the places where it is referenced. But if that is not enough to convince you to not use this pattern (hint: it should be), there is another problem that is arguably even more insidious.

Let's say that by some stroke of luck, you are trying to test a class that does not make static reference to the singleton instance and accepts the dependency properly. Now, when you go to create a mock or fake implementation of MyInterface for use in the test, Java still has to instantiate MyInterfaceImpl at classload-time. Depending on what all that involves, this may be expensive, unstable, or flat-out fail in the test environment, thereby thwarting your efforts to properly unit-test.

Obviously, the easiest way to solve this problem is to avoid it in the first place. But if you happen to run into it in a legacy system, you can start migrating away from it gradually by simply moving the instance to another class, or by using something like the value-holder pattern or initialization-on-demand holder idiom that moves the creation of the instance out of the classload sequence for the interface.

Where to Find cybervillainsCA.cer in Selenium

In the WebTestingExplorer documentation, we instruction users to create a Firefox profile (just for testing!!!) that trusts the cybervillainsCA.cer certificate normally bundled with Selenium distributions. The main reason we do this is to make it possible to inspect HTTPS traffic in a proxy and check for bugs in real-time as we explore the web application.

I've had several questions about how, exactly, to find cybervillainsCA.cer. The search results on this topic are unimpressive enough that I decided to write this quick blog post, hopefully attract some search engine love, and clear this up for everybody. So here you go:

1. Download selenium-server-standalone-x.xx.x.jar (as of this writing, at version 2.28.0).
2. Execute the following commands from a terminal from wherever you saved it:

1. mkdir selenium
2. cp selenium-server-standalone-*.jar selenium
3. cd selenium
4. jar xvf selenium-server-standalone-*.jar
5. ls sslSupport/cybervillains.cer

After step 4, you may get an error from jar. Ignore it. After step 5, you have verified that the file is there, and you can do whatever you need to with it. Enjoy!

Waiting for GWT RPCs in WebDriver Tests

When writing WebDriver tests for GWT applications, we often find ourselves writing WebDriverWait's to synchronize the tests with asynchronous GWT RPC's. These waits are often based on some DOM-observable result of the RPC completing -- for example, waiting until a label is changed by code in the onSuccess handler.

This has a couple of drawbacks. First, we would prefer to assert on the results of the RPC, not rely on them to control our tests. An assertion failure is preferable to a wait timeout when the onSuccess handler does something incorrectly. Second, if the behavior of RPC handling changes, our wait conditions will break and have to be maintained.

We would rather attach our wait condition to the RPC itself, rather than side-effects. Unfortunately, GWT's RPC mechanism does not build in anything for this, and we have to do it ourselves. In the remainder of this post, I will outline a simple (perhaps simplistic) way to do this.

The overall approach is to maintain a hidden field as the RPC goes out and comes back. In the .ui.xml:

<g:Hidden ui:field="ready" name="ready"/>

And the UiBinder declaration:

@UiField Hidden ready;

We'll create a helper method that updates this field based on whether we are "ready" or "busy" running an RPC:

public void setReady(boolean isReady) {
  if (isReady) {
    ready.setValue("ready");
  } else {
    ready.setValue("busy");
  }
}

Then our RPC call will follow this pattern:

public void onAction() {
  view.setReady(false);
  service.getData(new AsyncCallback<String>() {
    @Override
    public void onFailure(Throwable caught) {
      // Show failure message...
      setReady(true);
    }

    @Override
    public void onSuccess(String result) {
      // Show the result...
      setReady(true);
    }
  });  
}

Now in our WebDriver test, we can write a WebDriverWait like this:

WebDriverWait wait = new WebDriverWait(driver, 5000);
wait.until(ExpectedConditions.textToBePresentInElementValue(By.id("ready"), "ready");

There are some obvious drawbacks to this simplistic approach -- for example, it is not very extensible to an environment with multiple or batched RPC's. In a future post, I'll take on some of those challenges.

Manually Deleting the WebDriver Firefox Profile Directory

Some time ago, I starred a Selenium WebDriver bug that has a long discussion about how WebDriver leaves the Firefox profile directory behind in tmp after the profile goes away. Allegedly it was fixed at some point, but it still gets pinged from time to time, so who really knows.

Usually you will discover this when you are running a long set of tests that don't reuse the same Firefox, and then your hard drive fills up. This can be, well, a little frustrating. But on the long list of known WebDriver issues that never seem to really get fixed, this one is fairly minor, because it's simple to work around.

The comments in the bug suggest an @AfterClass method that uses Selenium's TemporaryFileSystem class to do the cleanup. This seems pretty satisfactory for most purposes. Nonetheless, depending on what else is going on behind the scenes, you may not want to clean out the entire Selenium-managed temporary directory in the middle of a test run.

I thought I would explain how we accomplish the same thing in WebTestingExplorer. We have a wrapper around the WebDriver instance that we use. It has a close() method that gets called between test cases. One of the things we do in that close() method is work around this bug:

    List<String> profileDirs = Lists.newArrayList("anonymous*webdriver-profile",
        "userprofile*copy", "seleniumSslSupport*");
    File tmpDir = new File(System.getProperty("java.io.tmpdir"));
    FilenameFilter profileDirsFilter = new WildcardFileFilter(profileDirs);
    File[] files = tmpDir.listFiles(profileDirsFilter);
    for (File profileDir : files) {
      LOGGER.info("Cleaning up tmp profile directory: " + profileDir.getAbsolutePath());
      try {
        FileUtils.deleteDirectory(profileDir);
      } catch (IOException e) {
        LOGGER.log(Level.WARNING,
            "Failed to delete tmp profile directory: " + profileDir.getAbsolutePath(), e);
      }
    }

As you can see, we fairly strategically delete the things that we found Selenium and Firefox can create each time a new FirefoxProfile/FirefoxDriver gets instantiated. Theoretically, if this bug ever gets fixed-fixed, we could take this code out, but actually, I don't see any reason not to just leave it forever. It's better than having the issue regress on us.