Who killed Velocity?
During an update of our courses at the Amsterdam University of Applied Sciences, I was told that we would no longer be teaching about planning poker in Scrum projects. Apparently, measuring velocity was no longer part of the orthodoxy. I found this puzzling, as I had taken the rituals of estimating and measuring velocity to be part of the core processes of Scrum. Had I misunderstood the framework, or were there other forces at play?
Although velocity never was explicitly mentioned in the Scrum Guide, I did encounter this notion when I took the Scrum Master certification course in 2012. The course was given by Jeff Sutherland, one of the creators of Scrum, so there is little chance that he misinterpreted is own doctrine. When I took the course, it seemed to me that measuring velocity and reflecting on the team’s ability to deliver at fixed interval were key processes in the execution of Scrum.
So much so that when I later analysed Scrum through the lens of game design, I wrote:
“Each Sprint is like one game, where the goal of is to make the most accurate prediction of how many velocity points we will achieve. The trick is to predict an ambitious number of points, so that we beat our previous score, but not too ambitious either, because we want to be able to reach it.”
And:
“Scrum changes the goal from ‘delivering this massive project ‘ to ‘making the most accurate prediction of how much we can deliver in a limited time-period.'”
There could be only two options: either my analysis was deeply flawed, or something had changed in the world of Scrum.
“Help! He’s measuring my velocity!”
I put this puzzle on a shelf, until more recently a friend showed me this comedy video called “When the development team meets their first Scrum Master”:
In the video, one of the characters yells: “Help! Help! He’s measuring my velocity!”.
We couldn’t help but laugh at this sentence, recognising the implied suffering of management using velocity as a KPI in a financial forecast, or as a way to assess the performance of the team, or as another creative form of control that they might have invented, imbued by the power of metrics.
It then dawned on me that velocity was killed by Goodhart’s law.
Measures and targets
Goodhart’s law states that when a measure becomes a target, it ceases to be a good measure.
This phenomenon is well known by policy-makers in government. It describes a dynamic by which a measure can’t become a target while at the same time remaining a relevant measure. This is due to the effect of reflexivity: agents will put all their efforts into reaching the target, thus changing the reality that is being observed. It shifts the goal.
In the case of Scrum, what this means is that the measure of velocity becomes irrelevant as soon as it starts to be considered a relevant metric by observers outside the team. The team will play the game of trying to reach the “good” velocity, which will then change the nature of what is being observed.
Why does this phenomenon happen? Game design theory can give us some insights.
Conflicting goals
“Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.“
– Charles Goodhart
If we analyse the process through the lens of game design, we notice that there are two systems interacting. The problem comes from a confusion between these two systems which have two very distinct goals:
- The original goal of velocity is to act as a reflection tool. It is a simple feedback mechanism meant to be used within the Scrum team to create awareness about the work being produced, so that the team can identify impediments and adapt when needed. It works because everybody trusts that everybody is doing their best, and serves a discussion point when variations are spotted.
- The new goal of velocity is to compare an actual performance with an expected performance. It is a scoring system, and like all scoring systems its purpose is to motivate people to… score higher. It functions under the assumption that there is a higher level of velocity that could, and even should, be reached.
With this distinction we can see that the two system at play have fundamentally different goals, one being to increase self-awareness and the other to try and stimulate faster delivery.
However this doesn’t yet explain why velocity can work perfectly well as a reflection tool. How come the guessing-our-own-velocity game doesn’t collapse under Goodhart’s law?
Victory vs. achievement
The reason why velocity works as a reflection tool and not as a scoring system can be further explained by differentiating two types of winning known in game design: victories and achievements.
- Victory wins are won by overcoming formal challenges, meaning that there is a “check” moment where the game evaluates the result: victory or defeat.
- Achievement wins are more personal, subtle, and playful than victories. They occur during play rather than at the end of a specific segment, and they do not result in defeat when they are not won.
What is the win/loose moment in Scrum, where does the “check” occur?
The event that determines if a Sprint was a succes has been ritualised and formalised in the Scrum Guide and is called a Sprint Review. It is the only check that will ever be needed, as it is the moment where all stakeholders get together to review what was made, taking into account the complexity of the situation. Even then, the result is rarely a binary win/loose decision, but rather a discussion aiming at determining future adaptations to the product.
What about the guess-our-own-velocity game? Does it not matter at all?
We could argue that reaching the exact number of velocity points estimated in the Sprint goal is more like a achievement win and less like a victory win. It is great if we get it, but it is not what the main game is about.
For example, we could imagine an achievement such as “predict the correct velocity 100 times”, which would be a nice, playful way for the team to reward themselves for their increased self-awareness. It would however tell us nothing of value about the team’s progress towards the Product goal, as they might very well have delivered 100 increments that did not meet the client’s expectations.
Should we burry Velocity?
Now that we have outlined a few reasons for the misuse and subsequent disappearance of velocity, should we advocate for its reinstatement, or say our goodbyes?
In the case of our students, we are following the trends of the industry and advising them to estimate their user stories more loosely, often using three sizes (small, medium, large). This technique, sometimes referred to as “t-shirt sizes”, does have the advantage of not being a number, meaning it is harder to divert its goal for control purposes.
In the same way as no-one but Frodo could carry the Ring of Power and throw it into lava, most organisations have to recognise that they should steer away from Sprint goals and velocity, as they to do not possess the playfulness and lightness to measure velocity “as if it didn’t matter”. For the purest of hearts, planning poker is always an option.