Like clockwork every month, someone on one of the Agile Mailing lists asks a question about “comparing velocity between teams” or “benchmarking data on velocity.” Before we examine the problem, let’s recheck what velocity is—the amount of work completed by the team/time taken to complete it. The amount of work is usually measured in Story Points (a relative measure of size). So velocity is just the number of points the team completes in the average iteration/sprint.
The underlying point is that Agile/Scrum teams use relative estimation (e.g. is this story/feature bigger or smaller than our standard story?) vs. the traditional approach of absolute estimation. The problem with comparisons, benchmarking, or any other attempts to compare velocity is that my story points ≠ your story points, because different projects use different standard stories. They work in different problem domains, and they have different people.
If we could compare velocities across a team, what would happen? Estimation of inflation. Team Better hears that Team Good has just finished their release planning meeting and estimated that they will achieve 200 points before the next release in 6 iterations/sprints. Immediately, Team Better knows it needs to produce more points and so estimates 300 points. Now along comes Team Best. . . .
To my mind, the real value of Velocity and Release planning is giving the Product Owner a good idea of what can be achieved before the next release. For one client, our release planning told us that the list of required features was roughly double the amount of time in the project budget. For first three iterations, the Product Owner denied reality but eventually the team’s actual velocity made it clear that trade-offs were required.
What matters far more than comparing velocity is whether each of the teams arrived at a common understanding of their work. After each iteration, has the team delivered value? Does the Product Owner always have a good idea of how much value has already been delivered and how much they can expect to be delivered?
Anytime you’re about to spend time and money on something, ask if the customer would be prepared to pay for that.
BTW I smell an Agile FAQ coming out of this and other questions.
Mark Levison has been helping Scrum teams and organizations with Agile, Scrum and Kanban style approaches since 2001. From certified scrum master training to custom Agile courses, he has helped well over 8,000 individuals, earning him respect and top rated reviews as one of the pioneers within the industry, as well as a raft of certifications from the ScrumAlliance. Mark has been a speaker at various Agile Conferences for more than 20 years, and is a published Scrum author with eBooks as well as articles on InfoQ.com, ScrumAlliance.org an AgileAlliance.org.
Mark,
This is a good post. I half agree, and half disagree.
The part I agree with is about comparing velocity across teams, as long as we take velocity as some measure of how productive people are. As soon as we do that, people will start gaming the system. Not only that, they’ll start to feel pressured (or even paranoid), and morale will suffer.
Where I disagree isn’t so much as to claim you’ve said something wrong, but to advocate a different approach to the question of velocity that avoids this kind of comparison and gaming, while also providing a meaningful way to define velocity across teams.
I prefer to measure story size in absolute terms, not relative terms, and tend to use “Point” (never “Story Point”) as meaning a perfect person-day, or eight person-hours of work. This approach allows meaningful cross-team calculations, such as “how much work can five teams do.” It also prevents gaming and comparisons, since a Team’s velocity is simply the number of productive person-days available for a particular team in a particular Sprint.
This measure of velocity is not coupled to a Team’s productivity. Productivity isn’t something I see much value in trying to measure. All Team’s I’ve worked with have been eager to get things done, and measuring their productivity (even if possible) wouldn’t have provided value. As soon as you try to implement any measure of productivity, you run the risk of finding it appearing in annual reviews, and of people gaming the system. I’d rather not go there!
This approach has been adopted without difficulty by a dozen or so teams with which I’ve worked. I’m aware that some ScrumMasters say that their Teams can’t do estimation with absolute units. I believe them, but I haven’t encountered that problem personally.
Kevin Thompson, Ph.D., PMP, CSP
Senior Instructor & Consultant, cPrime
https://www.cprime.com/scrum
Kevin – thanks for taking the comment. Your idea is intriguing and makes velocity something you can compare across teams. But it suffers from the problem that Story Point estimation was invented to solve. Human beings it turns out are not good absolute estimators. Example: estimate the height and weight of the next 5 people you meet today. Now ask them and see how accurate you were. Likely you were not very accurate at all (there is some good science on this). When doing group estimation with Ideal Days (or even weeks) people will argue tooth and nail over the difference between 2 and 3 days. Since neither was likely right, it doesn’t matter. There are a bunch more reasons – but you just made me realize that I need another post on this subject. Thanks 🙂
Good and completely agreeable analysis IMHO.
Mark,
In my opinion, whether the technique is intentionally relative ala Scrum or attempts to be absolute like function points, both suffer from the imprecision of the information used as the basis of estimation.
However, the fact that Scrum is looking at a subset of the project complexity implemented over a few weeks means that the team is more likely to have an accurate idea of what is to be accomplished compared with an estimator using function points at the beginning of a waterfall project.
It would be an interesting experiment to have a function point specialist work with an agile team to convert user stories to Internal Logical Files/ External Interfaces / External Inputs / External Outputs / External Inquiries and estimate the function points for each sprint. Then compare the function points per developer over successive sprints with similar projects in the literature and with the velocity as calculated by the team.
John – its an interesting question – but I have to ask why would anyone spend the time and money on it? I always ask the question would the customer pay for it? Do they get value?
Like Kevin I use ideal days. We do this as we work in fixed price contracts (the majority of the time) and need to cost and quote the job upfront so being completely abstract with our story points just won’t work. As an aside we also use the cone of uncertainty to buffer ourselves from estimate deviations for the financials of the project as we take the risk of having to cost the job upfront.
To avoid the hang up over whether something is 1 or 2 days we play planning poker using the Fibonacci scale. This way it is easier for the team to decide whether something is 3 or 5 ideal days for example and after the conversation they will move one way or another.
When we look at the team we purely look at how many story points they can complete in a sprint. We don’t compare this across teams as I think this is dangerous, who is to say a team delivering 60 story points is any more or less productive than a team delivering 75. As has been said this will depend on the make up of the team and the work in hand.
We do however use these figures when performing release planning in conjunction with historic availability but this is only for planning and no other purpose.
Kevin I’m wondering if this is similar to what you do when you say “how much work can five teams do” If you do this however I guess the teams must all be working on the same project/product and have a similar skill set for the comparison to be true?
Cheers
Tom
https://theagilemindset.wordpress.com/
Mark – I agree that humans are not accurate estimators. That’s OK; the purpose of estimation in Sprint planning isn’t to be accurate, but to provide “good enough” estimates that the Team has a decent chance to implement the stories that their estimates say will fit in a Sprint.
Relative points are no more “accurate” than absolute points, so relative points have no intrinsic advantage in this area. For that matter, the idea of fitting relative points to a numeric scale (a la Planning Poker) has always seemed like a “neither fish nor fowl” way of looking at the world. It isn’t meaningful to map “bigger” and “smaller” to a Fibonacci or other numeric sequence. At a pragmatic level, this seems to work well enough for many Scrum Teams for them to get the job done, but this is more a testimony to the crudeness of estimation that Scrum can tolerate than it is anything else.
My conclusion, from reading and observation, is that Teams can use relative or absolute sizing for Stories, and succeed. This tells me that the choice of one versus another has more to do with local preferences than anything else, and that’s fine. What matters is the ability to deliver.
As for arguing “tooth or nail” over 2 versus 3 days, that is much a facilitation issue as anything else. People can always find something to argue over. If a situation like this arises when I’m wearing my ScrumMaster hat, I say,
“The split is even between the two cases. Three people say two, and three say three. This means everyone agrees that the Story can be done in three days. Is there any objection to going with three days?”
I don’t recall any case where we couldn’t agree and move on, and the answer has always been good enough.
Tom – Like you, my Teams estimate with Planning Poker, which starts with the Fibonacci sequence. That has always worked well for us. Also like you (I guess I’m a lot like you…) the only use for cross-team aggregates I see is purely for internal planning, at a very rough level.
Mark – Notwithstanding the interesting digression into estimation techniques, I strongly support your main point: Any attempt to measure (or worse, incent according to) the productivity of different Teams is fraught with peril. It will destroy morale and lead people to game the system.
Kevin Thompson, Ph.D., PMP, CSP
Senior Instructor and Consultant, cPrime
https://www.cprime.com/training
Mark – By the way, I’ve written an article on the mathematics of uncertainty at https://www.cprime.com/knowledge/articles/uncertainty.html. Using very simple math, it provides yet another explanation for why things take longer and cost more than we expect.
Cheers!
Kevin Thompson, Ph.D., PMP, CSP
Senior Instructor and Consultant, cPrime
https://www.cprime.com/training
We use a slightly different approach in that we associate tasks to each story and points to each task so from there we can calculate the story points. If you break it up into individual tasks (one trick is finding the righ balance between listing everything you need to do and getting lost in the detail and being too high level), you can begin to figure out how much effort each task will take. That allows you to find consistence in the way you create your tasks and thus your story points.
I agree these are all relative measures and it’s especially hard if you have multiple teams working on completely different projects. And I would never compare the two. They would each have their own set of benchmarks and KPIs. But I have found that getting it down to the task level helps a lot.
Long comment turned out to be a blog post:
https://jacobsingh.drupalgardens.com/content/velocity-tracking-agile
I’ve been using agile now for a couple years and it seems to me that those who are advocating absolute time estimation vs relative time estimation are missing an important point. Agile projects are divided into sprints which are meant to be a particular set amount of time which your team works well within. We use 3 weeks, which is probably a bit odd, but usually its 2 weeks or a month. When you have an idea of your velocity, you know how many stories you can put into a sprint, on average. My team’s velocity happens to be about 40. So, when the entire project has been estimated, we can get an absolute release date from our relative estimations by simply dividing the total story points by 40 to know the number of 3 week sprints we should need. If there’s a particular release date that is required, we can divide the time between into 3 week sprints to determine the maximum story points we could accomplish and reduce the feature set accordingly.
@Kevin and Tom,
Could I ask a clarifying question?
What is the makeup of your typical agile team? Do you mix developers and testers together, or do your team members perform any and all roles in the sprint.
What velocity would you give a team with 4 developers for a 2-week sprint? What velocity would you give a team of 4 developers, 3 testers and 1 technical writer for the same two weeks?
I have an interesting political challenge which I sort of rigged my self into since I moved from a mature agile organization to the other that just adopted agile.
We used to and still do display story points as delivery on commitment/promise and to track velocity. however few in execs have started to understand that as productivity and have asked how can we increase story points from 30 to 50 or above as then we know team is performing awesome and their productivity is increasing otherwise 30 or ballpark story points delivered reflects team is not improving. if someone could share nice articles which I can accompany with with my own examples that would be awesome. or any other strategies to answer this question.