two ways to automate web testing

There are two ways to automate web testing.   The goal is to test the functionality of a web application.  One way is to write automation that drives a browser.   The second is to use a library that imitates a browser session and communicate directly with the server.

Tools that use the first method include open source applications such as Selenium, Watir, and Samie; and commercial products from HP/Mercury, IBM/Rational and Borland/Segue.  There are also two ways to drive the browser.  The way used by Watir, Samie/Pamie, and presumably by the commercial applications, is to use the automation APIs provided by the browser, typically Microsoft COM — which means IE.  Selenium uses a different method, driving the browser through Javascript.  It can use a proxy server to inject javascript code into the browser or a javascript library can be included on the server.

The other method is to cut the browser out of the equation.  Tools such as HTTPUnit, HTMLUnit, Canoo Web Test, WebDriver, and TestMaker use this method.   A similar method is used by other tools including JMeter, LibWWW, and curl.  The obvious disadvantage to using this method is browser compatibility issues.

The HTTPClient library (used by many of the above) is virtually a browser in it’s own right, with it’s own quirks.  It has support for cookies, Javascript, DOM, and HTTPS.  However, WebDriver doesn’t enable this by default.

Tools that use macros to drive a browser would fall into the first category.  Tools that have an “expect” like dialog would fall into the latter.

I usually advocate the first method, because nothing beats having a real browser excercise your application.

The often overlooked advantage of the second method is that you can more easily run browserless tests as integration and smoke tests.  Because it doesn’t have the overhead of the full browser, it is lightweight, and client independent.  It runs faster and can be more easily run concurrently (which makes it useful for performance testing.)  Browser timing issues (and crashes) are less of a problem.

I’d actually recommend type 2 tools for testing links, page flow, content (text/images), and principal functionality.  But if user interface testing is needed, I’d use type 1 tools. However, the truth is that UI and ajax timing related issues are much easier to find with manual testing.  I’d guess 3/4 of what automation buys you can be done with a browserless tool.

The advantage that browser-driven testing buys you is with recording tools.  However, the penalty is with stability and speed of your tests.

Advertisements

5 thoughts on “two ways to automate web testing

  1. Hi Aaron: Nice blog entry. Thanks for including TestMaker.

    You should know that TestMaker bridges the first and second categories of test. For instance, we ship with TestGen4Web (and soon Selenium) as plug-ins to Firefox. Playback there is through the browser. When you need to run the recorded test as a load and performance test or a service monitor then TestMaker runs the TestGen4Web test (or Selenese test) using HTMLUnit.

    Details are at http://selenium.pushtotest.com.

    Keep up the good blogging.

    -Frank

  2. Some nice points but sadly too much errors (for instance concerning the HTTPClient library which has absolute no support for JavaScript or DOM) and too much imprecisions (look a bit more carefully at WebDriver’s documentation). I disagree as well concerning recording tools because it is in no way limited to “browser-driven” testing and finally the usual “nothing beats having a real browser excercise your application” is far too short.

    @Frank: is a plug worth ignoring all issues of this blog entry? ;-)

  3. To clarify, HTMLUnit does provide javascript and SSL support, and it is based on HTTPClient, is it not?

    What tool do you work on, Marc? I’d like to know more about recording with non-browser driven tools.

  4. Ok, comment are working, it was probably my fault.

    Yes, HtmlUnit is based on HttpClient. JS support is provided by HtmlUnit whereas SSL support is provided directly by HTTPClient.

    I’m a bit surprised of your next post, should I have given you kudos for all inaccuracies of this post like Frank did?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s