The results are encouraging, but largely unquantifiable. Would I feel any better buying this software over someone with a fully manual QA process or longer release cycles? I think that’s comfort you can only determine once you’ve used a package past a few upgrades.

One thing seems very obvious, with that level of process should be able to produce good metrics around the average defect rate per release? How many crashing, critical, annoyance bugs are they experiencing per release? How big is their customer base? How many new features are they releasing each iteration vs. bug fixes? Can we see your burn charts?

That would be *really* impressive to see and would really drive home the Agile Advantage message. Also, it would be a real differentiator for could even become the standard by which buy software. “Open” should be more than just the source code, but the metrics as well.

Hmm, dunno. Thinking out loud.

We’re fully transparent with the good, bad, and ugly on our support site for all of our products, including Scrum’d: http://getsatisfaction.com/atlanticdominionsolutions. Our defect count is fairly low per release due to automated testing and manual QA on the app. Nothing is perfect though, and we fit into that category. We also do many deploys as opposed to big quarterly releases. This helps us rapidly kill the bugs that we missed with our process.

We have close to 2K accounts on Scrum’d. Customers are from all over the world, and not necessarily doing software development, which was one of our requirements when we built the app – that anyone doing scrum could use the app.

In each sprint we typically complete between 3-8 stories, and then release every 1-2 sprints depending on how quickly we want to get the features to our customers. Sometimes we get excited and push updates asap.

I wonder if there would be useful object measures.

As a point of reference I’m currently using a 3rd party commercial library which is completely opaque to me. I see nothing of their process, bug counts or anything. The only thing I get is a release note list of features added and bugs fixed. I can tell at a glance the vertical this company is getting the majority of its revenues from since that’s where the bulk of the new features are going. If we were a bigger fish we could perhaps sway this. My point is, bug categorizations gets gamed all the time, not just to make the dev team look good [Cue the sound of fingers rubbing together]. And, even this rough list gives me a good insight into the libraries relative stability.

And thanks to the other commentors for the qualification of their processes.