Comments on: Functional/Acceptance Test Tools for Web Apps

By: Richard Lawrence

Richard Lawrence — Tue, 30 Nov 1999 00:00:00 +0000

Watir if you can introduce a new language. As @testobsessed said on Twitter, all test automation is programming, but Ruby is a good language to start with and Watir is a good intro to Ruby. If you have to stick with .NET, WatiN is a good tool. It does support dialogs, is faster than Selenium IME, and is not at all hard to use with IE. I’ve written about it quite a bit lately, and I’m in the middle of a series on my blog about patterns for maintainable WatiN tests.

By: Patrick Wilson-Welsh

Patrick Wilson-Welsh — Tue, 30 Nov 1999 00:00:00 +0000

I increasingly find that I cannot design a functional testing strategy, much less choose a tool, without knowing what the entire balance will be between the 3 layers of Mike Cohn’s testing triangle (bottom layer is healthy unit tests; mid layer is non-thru-the-GUI functional tests, end to end tests, etc; top layer is through-the-GUI tests).

To me, this will always end up being 3 different, simultaneous, carefully choreographed automated testing initiatives, coordinated between QA/Testers and programmers. It is a whole team problem. Together, the team answers the following kinds of questions:

– How few thru-the-GUI tests can we get by with, and what are their focus?
– How healthy are our unit/isolation/programmer test suites, and how are they improving?
– Where are we duplicating automated testing effort, and where can we avoid that?
– Where are the highest test maintenance costs, and how can we reduce them?
– Where are the biggest gaps in our automated testing, and how does that show up as defects, etc?
– How are we all learning automated testing together; what are we capable of now, and what can we stretch toward tomorrow?

Side note, to your original questions: I like several things about Selenium. Non-technical testers can record tests with it, and export them to any of several languages. The tests created thus are expensive to maintain, certainly, but we can tweak them programmatically using languages the local programmers likely already know. Some of these tests can remain thru-the-GUI tests, and some can be replaced with unit tests. In short, Selenium tools (IDE, RC, etc) encourage a conversation between programmers and testers about the entire automated test triangle, not just part of it.

By: Mark Levison

Mark Levison — Tue, 30 Nov 1999 00:00:00 +0000

Thanks Richard and Patrick. Interestingly you’ve outlined the two options that are currently going through my mind – Either we record tests with a goal to eventually creating a DSL or we do as much testing as possible through the slightly lower level (probably via FitNesse .NET) with GUI tests only to check the UI.

Its also heartening to hear that people have had good luck with both WatiN and Selenium. On Twitter the responses have been 2 for Watin and 3 for Selenium.

In other words all the choices seem at least good and it should up to the team to decided.

By: Lisa Crispin

Lisa Crispin — Tue, 30 Nov 1999 00:00:00 +0000

wrt Canoo WebTest, I don’t think it’s necessarily hard for non-programmers, but definitely not attractive to business people (I can’t get ‘em to use the FitNesse wiki, they’re sure not going to write XML). Personally I find it easy to specify tests in XML, esp if your editor color-codes it, but it’s not for the techno-phobic. Dunno if there might be a framework out there that makes it easy for a business person to enter tests that get turned into xml. I know you can drive WebTest scripts with data from a spreadsheet, it probably wouldn’t be hard to come up with a framework that suits.

WebTest has been one of my favorite open source test tools because the developer community is so active and helpful. You can even log a bug in their bug tracker and someone will fix it. It’s not too well-known here but widely used around the globe. By far the best ROI of any GUI test tool I’ve ever used. But it sounds like you want more of a behind-the-GUI tool.

I see other people recommended WatiN, I’m glad, I didn’t mention it ’cause I don’t know how it compares with Watir.

By: Ian Phillips

Ian Phillips — Tue, 30 Nov 1999 00:00:00 +0000

Canoo Webtest has a firefox addin for recording the xml script as you navigate around your application. Good way to get started but you will need to then enhance the script to make it more maintenable.

By: James O'Sullivan

James O'Sullivan — Tue, 30 Nov 1999 00:00:00 +0000

We are currently trialing Thoughworks Twist product for testing our ASP.NET based products. We’re pretty happy with it so far and its fairly easy for a QA person to generate tests using the built in Firefox based web recorder.

It will require a little bit developer support, but that’s mainly for refactoring the generated test code as the code-base gets larger. One thing to note is that all of the generated code is in Java.

By: Mark Levison

Mark Levison — Tue, 30 Nov 1999 00:00:00 +0000

Thanks to Lisa and Ian – we be investigating Canoo more closely now.

James – I know and like Twist but it costs $$$ and my client is unlikely to budget for that right this second.

By: Immo Huneke

Immo Huneke — Tue, 30 Nov 1999 00:00:00 +0000

I'm bookmarking this page as a quick guide to Web testing tools! One that doesn't appear in your list is Cucumber - I have not tried this myself yet, but it does attempt to address the requirement that test cases should be human-readable (and possibly writable) by non-programmers.

By: Mark Levison

Mark Levison — Tue, 30 Nov 1999 00:00:00 +0000

Immo – thanks for the comment and the compliment. I knew of Cucumber (and a few other Ruby tools). However they didn’t get mentioned because the client is a very conservative MS shop. Getting them to upgrade to VS2008, .NET 3.5 is a real effort. Getting them to install Ruby on their test rig would be an even bigger sell.

By: Alexandre Martins

Alexandre Martins — Tue, 30 Nov 1999 00:00:00 +0000

There’s also an acceptance test framework called JBehave (http://jbehave.org), a bit similar to Concordion but with no HTML involved. I wrote a post explanning how to integrate it with Selenium to write acceptance tests for web application. If you find it interesting take a look at http://blog.m.artins.net/acceptance-tests-with-jbehave-selenium-page-objects/

Cheers!