I keep on seeing announcements for the next great Agile Task Tracking tool. I just saw one posted to Scrum Development where it’s author said: “I haven’t done much testing, so if you find a bug and want me to fix it let me know :-)”.
My reply: Congrats I’m sure you have an excellent application. I’m wondering if you see the irony – you’re posting to an Agile group and say that your app has hardly been tested? What is your definition of Done? Do you use TDD? At least Unit Testing? What was your approach to acceptance testing?
My Promise
Here is my promise to all of my future clients, I won’t show you an agile tool that wasn’t developed using Agile methods. From future tool suppliers I need to know:
- Your Definition of Done
- Whether you use TDD? Or at least Unit Testing?
- What do you do for Acceptance Testing?
- How often do you release?
- What did you learn in your last retrospective?
- ….
If you can’t answer these questions I’m not interested in your tool.
Update I mis-understood the author’s intent. He’s not releasing a tool, so much as testing the waters to see if there is any interest. For these purposes I think he did the right thing. I still stand behind the statement I would show a client a tool where the vendor couldn’t answer the above questions.
Related posts:
Very good idea Mark!
Nice :)
Hi Mark,
I’m Robert Dempsey, CEO and Founder of Atlantic Dominion Solutions. We developed Scrum’d: http://www.scrumd.com. A great post and an excellent challenge. To answer your questions:
# Your Definition of Done
We define done for each user story. Acceptance criteria includes the behavior a user can expect from the app when using it, the workflow for the feature, that it needs to be tested (and how it will be tested), any validations that occur, and if there is documentation for the feature that it is updated.
# Whether you use TDD? Or at least Unit Testing?
We do full TDD for all of our stories. We didn’t start out that way, however now that we have a full test suite we keep it that way. Also, if a bug is found, then a test needs to be written for that and then the code fixed.
# What do you do for Acceptance Testing?
We have staging servers set up for internal testing. We also have a few select customers that we let into our staging servers, and I show (in person) some of our current and potential clients the features to get their feedback as well. In addition, we use Get Satisfaction for our support site, and ask that commenters who suggested features that we implement to check out the release and see if it’s what they wanted.
# How often do you release?
This depends. We try to do monthly releases. Sometimes it longer, but typically it’s less. The release schedule really depends on the importance of features to our user community.
# What did you learn in your last retrospective?
You don’t always get it right the first time, however anything you do release needs to work correctly. Also, continuous integration is a huge help as your product gets bigger, and of course, testing is an absolute necessity. We would rather delay a release to ensure quality than release stuff that won’t work.
I like challenges, so…
Our DoD for stories:
- code meeting minimal requirements committed,
- unit tests passing,
- feature is working under FF, Chrome and Opera,
- feature is working without any significant failures under IE
- feature is tested and accepted by at least one team member that was not implementing the story
Our DoD for release:
- WAR and .exe distros built successfully,
- all unit test passing
- all functional bugs fixed
- all UI bugs fixed or small UI bugs (mostly for IE) at least scheduled to be fixed
- installation and upgrade documentation updated
- installation tested on Windows and Linux
- new features list updates at product web site
Do we use TDD? Well I would not say that becuase our coverage is not 100% which means some of our code is not unit tested (glue code), but yes we do unit testing with “test first” style for all domain and business logic code and we tend to shift into BDD style right now.
As our Product Owner is also one of active developers, for acceptance testing we tend to verify features within a team as we have strong and common vision of a product within the team (see also acceptance in our DoD for stories). It’s a low ceremony process (as all our processes in the team)
How often do we release? Every 2-3 months.
What have we learned from our last retrospective? That we need to shift into feature branches to be able to release small stories more often while the big ones are still in progress.
So what tool is it :-)? It’s tinyPM
Do you see any flaws or warning signs in what I’ve posted here? Go on, say it… we’re always happy to improve…
Regards,
Marcin Niebudek
Marcin – Interesting you clearly know what your doing. One small detail – I wouldn’t go after 100% test coverage. Instead I would ask where you find bugs in the glue code and let that guide your thinking. It seems like a small part of your, which I’ve not taken the time to play with (sorry) :-)
@Mark I intentionally wrote about TDD and coverage as many people think that writing unit test is doing TDD, which for me is not. As I understand TDD is that you simply get 100% coverage out of the box because you need to write tests first (no code without test first).
So for me of course 100% coverage is not a goal itself (we’re rather pragmatic on this), but in other hand not having 100% coverage is an evidence of not doing full TDD. If pursuing the full TDD is a good thing or not that is completely different question and I’m not going to start any religious war here :-)
Marcin – we’re in violent agreement. For me TDD is a design tool. 100% code coverage is neither realistic, desirable or useful. Code coverage is only useful as a hint about what you’ve missed. Nothing more.
Greg Wilson and Jordi Cabot did a survey on this last year: http://pyre.third-bit.com/blog/archives/2813.html. They found that most of the teams building agile project management portals were not using “formal” agile processes…
Rats, the auto-linkifier added the trailing period to the link. It should be http://pyre.third-bit.com/blog/archives/2813.html
Thanks for the pointer. The full blown article (http://www.cs.utoronto.ca/~gvwilson/articles/portals.html) says:
“Another noteworthy point was that almost all of the groups we interviewed acknowledged that they didn’t use any well-defined process themselves (not even the agile methods they preach). When asked, “Do you use XP [XP]?” or “Do you use Scrum?”, they invariably replied that their developers used a mix of best practices that didn’t strictly adhere to any published rulebook. None of the interviewees were defensive about this; all clearly believed that they had above-average developers who could be trusted to use pair programming, test-driven development, and whatever else was appropriate in context. This emphatically does not mean that their processes were chaotic: in all cases there was close and frequent coupling between development on one hand and requirements gathering and feature prioritization on the other. However, the day-to-day mechanics of actually producing high-quality code was trusted to developers and their consciences. It remains to be seen whether the users of the tools do follow specific agile methods or, as the tool developers’, they just use their own mix of agile practices.”
So I suspect the relevant tools vendors (Rally, VersionOne and Mingle) could all give good answers to these questions.
I don’t know about other tool vendors but we (Danube / ScrumWorks) work hard to keep our engineering practices as robust as possible. Like others have stated, no single metric is appropriate/complete, it’s more a philosophy of baking quality in. I personally hear this a lot and cringe: “Agile is cowboy coding, resulting in low quality products”. It’s really not. It’s actually the opposite.
Here’s a little run down of how we go about baking quality in (no way complete):
1) Start with strong defn of done at the PBI/story level regarding explicit/functional requirements. We don’t imply implementations per se, and leave that the UI design/team to sort out during the sprint.
2) We have a lengthy set of implicit done criterias in a wiki. It’s not ideal in that it’s not always visible. But if you’re on our team, you know about this list.
3) What’s on the implicit list? All code must be written using TDD mentality. Doesn’t mean the test is always first, but you write it from a testability standpoint. Code is unit tested to the extent reasonable/possible. We have a UI intensive app, so devs also write *integration* tests using Junit based testing systems (no record playback crap). Code reviews before anything is “done”. We use check-styles, must be documented, etc., etc.
4) All our tests are categorized by frequency of running. Unit tests run always, but integration tests take time. Fact is we have so many integration tests that running them all is a problem :) so we have to divide them up.
5) Continuous integration. We use TeamCity continuous to divide tests between multiple servers to run simultaneous to compress feedback loop. We run continuous on all platforms we support, non-stop.
6) Performance testing. This has been a more recent development, but we’re trying to get away from lengthy (ok, lengthy for us used to be 2 weeks, now it’s more like a few days) stabilization periods. Performance tests are run periodically during dev sprints.
7) Manual test. There is no substitute for some manual testing. We have to manage this carefully and keep it focused since we automate just about everything possible, including installer testing.
Oh, did I mention that we have only *one* QA resource in all? We release frequently to very large, unforgiving customers with a low incidence of defects nevertheless because we bake quality in to begin with, not after the fact. Our developers have a testing mentality.
Anyway, i could go on and on and I hope this isn’t taken as boastful; we’re lucky to have this environment. My point is actually that there’s a difference between saying you’re agile b/c you release fast with low quality, and actually taking agile principles to heart and baking quality in to start.
Victor – thanks for the comment – my list wasn’t meant to exhaustive or perfect, just a quick test to see if people are really Agile. Think of it as a Litmus test no more.
Hi, Mark:
I posted a detailed response to your questions on the Rally Engineering Blog here:
http://www.rallydev.com/engblog/2009/11/11/how-agile-is-rally/
Hi Mark,
the usage of Agile and Scrum at microTOOL to develop in-Step is described on our blog:
http://www.microtool.de/blog/post/How-We-Use-Agile-to-Develop-Our-Tools.aspx
Thanks for the Challenge, that was a good idea!
Have a nice weekend…
Take care,
Olaf Lewitz
Senior Consultant and Process Coach
microTOOL GmbH, Berlin
Great idea Mark, I wished I had come up with this.
I like it so much I want to add stuff ;-)
- Do you have a build server?
- How many tests do you have?
- Do you use a bug reporting tool?
Something that might be hard to ask for but that would really be good for me, is
-Where can I access reporting from your build server?
-Where can I access your bug reporting tool?
Hi Mark
Awesome idea!
I guess most of these tools grow as a personal side project. But even then we should try to follow best practices… at least when we release to the wild…