Functional/Acceptance Test Tools for Web Apps
I used to be rich client guy, but my current client is building web apps and so all of a sudden I'm a web app guy. Now I wish I’d paid more attention in the past. For my penance I promise to document whatever I learn on the forthcoming Agile Tools Functional Test Wiki (i.e. the wiki that we discussed last year at the AA-FTT meeting before A2008).
My client is building an ASP .NET application and we're trying to decide how to test it. The issues I have to deal with:
- The two QA people have no coding experience so any tool that requires significant coding will require more development support.
- The application will be run in IE on the IIS webserver. No other combinations need be tested.
- The application is really a document creation and management system. Once the documents have been created and approved they're submitted to an outside system.
My discoveries so far (via this list and the opensourcetesting.com) and some follow-up questions
Tools:
- Selenium seems to be written about the most although some users complain about slowness and less than intuitive api.
- Watin (now supports firefox) api seems to be well liked – but assertions have to be written in manually i.e. in code.
- Apodora – never heard of this tool before. From Open Source Testing: "Apodora is a framework/tool for automating functional testing of web applications. It provides the user with programmatic control of the web browser allowing them to interact directly with the browser's user interface. It uses a database backend in order to remember how to find your html elements. This also makes your scripts easier to maintain."
- FitNesse .NET – not entire convinced that the application is well suited to FitNesse style tests. But the team will make the decision. I really think I need to read chunks of Gojko's first book before passing any real judegement.
- Canoo WebTests – xml may be hard for customers and non programmers to write. Am I missing the point?
- Concordian – seems like a clever idea that will create a lot of work for developers.
Frameworks/IDEs
- Robot Framework – “Robot Framework is a generic keyword-driven test automation framework for acceptance level testing and acceptance test-driven development (ATDD). It has an easy-to-use tabular syntax for creating test cases and its testing capabilities can be extended by test libraries implemented either with Python or Java. Users can also create new keywords from existing ones using the same simple syntax that is used for creating test cases.”
- CubicTest – an Eclipse Based IDE for Selenium, promises “…It makes web tests faster and easier to write, and provides abstractions to make tests more robust and reusable.”
- StoryTestIQ – “STIQ is a mashup of Selenium and FitNesse. It is "wiki-ized" Selenium with widgets and features that make it easier to write and organize Selenium tests.”
Questions
Selenium
- Are the problems with Selenium slowness, need to insert delays after entering text and less than intitutive api real? (http://adamesterline.com/2007/04/23/watin-watir-and-selenium-reviewed/, http://www.testdrivendeveloper.com/2008/07/22/ComparingWatiNAndSeleniumForUITesting.aspx, http://hammett.castleproject.org/?p=120, ).
- Do you spend more time debugging the tests than finding real problems?
- Is Jay on the money is his positioning of Selenium: http://blog.jayfields.com/2008/07/immaturity-of-in-browser-testing.html “Selenium is best used by developers or testers when testing the most valuable (to the business) happy paths of a Javascript heavy web application that must function in several browsers”
Watin
- Does it support dialogs now?
- Does it suffer the same slowness problems as Selenium?
- Is it as hard to use with IE as Selenium can be?
- Is it compatible with Robot Framework?
If you’re using Selenium are you using any wrapper framework/IDE?
If you enjoyed this post, subscribe now to get free updates.
If you want to bring Mark into your organization for Training, Coaching or
Consulting please visit the corporate site: The Agile
Consortium.
Did you love this post?
Then please share it!
February 10, 2009 at 1:02 pm | Richard Lawrence
Watir if you can introduce a new language. As @testobsessed said on Twitter, all test automation is programming, but Ruby is a good language to start with and Watir is a good intro to Ruby. If you have to stick with .NET, WatiN is a good tool. It does support dialogs, is faster than Selenium IME, and is not at all hard to use with IE. I’ve written about it quite a bit lately, and I’m in the middle of a series on my blog about patterns for maintainable WatiN tests.
February 10, 2009 at 1:29 pm | Patrick Wilson-Welsh
I increasingly find that I cannot design a functional testing strategy, much less choose a tool, without knowing what the entire balance will be between the 3 layers of Mike Cohn’s testing triangle (bottom layer is healthy unit tests; mid layer is non-thru-the-GUI functional tests, end to end tests, etc; top layer is through-the-GUI tests).
To me, this will always end up being 3 different, simultaneous, carefully choreographed automated testing initiatives, coordinated between QA/Testers and programmers. It is a whole team problem. Together, the team answers the following kinds of questions:
– How few thru-the-GUI tests can we get by with, and what are their focus?
– How healthy are our unit/isolation/programmer test suites, and how are they improving?
– Where are we duplicating automated testing effort, and where can we avoid that?
– Where are the highest test maintenance costs, and how can we reduce them?
– Where are the biggest gaps in our automated testing, and how does that show up as defects, etc?
– How are we all learning automated testing together; what are we capable of now, and what can we stretch toward tomorrow?
Side note, to your original questions: I like several things about Selenium. Non-technical testers can record tests with it, and export them to any of several languages. The tests created thus are expensive to maintain, certainly, but we can tweak them programmatically using languages the local programmers likely already know. Some of these tests can remain thru-the-GUI tests, and some can be replaced with unit tests. In short, Selenium tools (IDE, RC, etc) encourage a conversation between programmers and testers about the entire automated test triangle, not just part of it.
February 10, 2009 at 2:06 pm | Mark Levison
Thanks Richard and Patrick. Interestingly you’ve outlined the two options that are currently going through my mind – Either we record tests with a goal to eventually creating a DSL or we do as much testing as possible through the slightly lower level (probably via FitNesse .NET) with GUI tests only to check the UI.
Its also heartening to hear that people have had good luck with both WatiN and Selenium. On Twitter the responses have been 2 for Watin and 3 for Selenium.
In other words all the choices seem at least good and it should up to the team to decided.
February 10, 2009 at 2:35 pm | Lisa Crispin
wrt Canoo WebTest, I don’t think it’s necessarily hard for non-programmers, but definitely not attractive to business people (I can’t get ‘em to use the FitNesse wiki, they’re sure not going to write XML). Personally I find it easy to specify tests in XML, esp if your editor color-codes it, but it’s not for the techno-phobic. Dunno if there might be a framework out there that makes it easy for a business person to enter tests that get turned into xml. I know you can drive WebTest scripts with data from a spreadsheet, it probably wouldn’t be hard to come up with a framework that suits.
WebTest has been one of my favorite open source test tools because the developer community is so active and helpful. You can even log a bug in their bug tracker and someone will fix it. It’s not too well-known here but widely used around the globe. By far the best ROI of any GUI test tool I’ve ever used. But it sounds like you want more of a behind-the-GUI tool.
I see other people recommended WatiN, I’m glad, I didn’t mention it ’cause I don’t know how it compares with Watir.
February 11, 2009 at 8:03 am | Ian Phillips
Canoo Webtest has a firefox addin for recording the xml script as you navigate around your application. Good way to get started but you will need to then enhance the script to make it more maintenable.
February 16, 2009 at 9:05 am | James O'Sullivan
We are currently trialing Thoughworks Twist product for testing our ASP.NET based products. We’re pretty happy with it so far and its fairly easy for a QA person to generate tests using the built in Firefox based web recorder.
It will require a little bit developer support, but that’s mainly for refactoring the generated test code as the code-base gets larger. One thing to note is that all of the generated code is in Java.
February 16, 2009 at 10:19 am | Mark Levison
Thanks to Lisa and Ian – we be investigating Canoo more closely now.
James – I know and like Twist but it costs $$$ and my client is unlikely to budget for that right this second.
February 17, 2009 at 10:35 am | Immo Huneke
I’m bookmarking this page as a quick guide to Web testing tools! One that doesn’t appear in your list is Cucumber – I have not tried this myself yet, but it does attempt to address the requirement that test cases should be human-readable (and possibly writable) by non-programmers.
February 17, 2009 at 10:44 am | Mark Levison
Immo – thanks for the comment and the compliment. I knew of Cucumber (and a few other Ruby tools). However they didn’t get mentioned because the client is a very conservative MS shop. Getting them to upgrade to VS2008, .NET 3.5 is a real effort. Getting them to install Ruby on their test rig would be an even bigger sell.
February 20, 2009 at 1:44 am | Alexandre Martins
There’s also an acceptance test framework called JBehave (http://jbehave.org), a bit similar to Concordion but with no HTML involved. I wrote a post explanning how to integrate it with Selenium to write acceptance tests for web application. If you find it interesting take a look at http://blog.m.artins.net/acceptance-tests-with-jbehave-selenium-page-objects/
Cheers!
March 8, 2009 at 4:43 am | Chris Sterling
Good day Mark,
I tend to believe that through the GUI testing should be minimized to testing acceptability and not internal behaviors. There are 3 prongs that I want to focus on:
1) GUI testing – automated tools such as Selenium (or the one we have worked on StoryTestIQ which is Selenium based) are used in this arena. I want to make sure the tool is able to generate small scripts of utility such as LoginAsAdmin and GoToShoppingCart so that scripts are not as brittle and failures can be tracked to a localized test component. Also, I want multiple perspectives of the tests so I can execute them in different batches depending upon which environment I am in. StoryTestIQ enables the use of multiple tables along with base Selenium functionality and also tagging of tests. Other tools do this, as well.
We still have found that exploratory testing of the GUI is important no matter the automation strategy at this level. We execute our automated tests at different speeds and in different browsers to see if there are any issues we notice as it goes by.
And then there is AJAX! I won’t go here but talk about a potential sink hole of debugging effort if you don’t have a good strategy of testability for JavaScript. Agghhh!
2) Functional testing – automated tools such as Fit*, RobotFramework, and Concordion tend to focus in this area. These tools act like another UI to the functionality and make sure the integrated components of your application work well together. I have not seen the amount of help in generating these tests as thought by some to occur but I still have found these tests give teams guidance on how to focus their efforts. These tests are going to take longer to execute so I still do not want too many of these tests but probably more than GUI since they will test behavior.
3) Unit testing – of course, a significantly greater number of tests will be defined at this level of testing hopefully.
Finding environments to support the testing across a browser matrix for your GUI tests is essential, IMO, and finding ways to grid out tests in categories 1 and 2 can help keep the duration of our testing feedback loop in check. I agree with Jeff Nielsen that executing the automated integration tests should take no longer than 10 minutes therefore we must find ways to optimize their throughput and structure as we go.
March 24, 2009 at 6:36 am | Keith
If you like Selenium but don’t want/need to program check out Fitnium
http://www.magneticreason.com/tools/fitnium/fitnium.html, an english language based DSL that allows you to create fitnesse tests in ‘almost’ plain english