Suboptimising Around Data

Reading time ~ 3 minutes

I originally wrote this article back in about 2009/2010 whilst working at a large corporation with a very strong measurement culture.

just some dataAs more teams and companies are adopting lean concepts and with the strong influence of the Lean Startup movement (which to reduce confusion is not the same as lean), this post feels relevant to finally publish publicly…

“Data is of course important in manufacturing, but I place the greatest emphasis on facts.” – Taiichi Ohno

There’s a great lean principle known as “Genchi Genbutsu” – “actual place, actual thing” Generally we interpret this as: “Go and See at the source”

When this critical pillar of lean thinking is eroded by a proxy we open ourselves to some painful problems.

Where organizations become too measurement focused, we risk our proxy for Genchi Genbutsu becoming data.

Lean and agile processes both rely on data but these are indicators only. Particularly in agile, there is a strong emphasis on data being used by the team for internal diagnostics – in fact very little agile material talks about data support for external management and there are good reasons for this.

Even where managers are fully aware that data are not the whole story, external or imposed measurement drives strange behaviors.

Talk to any individual or team that is measured on something that managers at one time or another thought was “reasonable” and chances are there will be a range of emotions from bemusement or cynicism to fear. All these responses will drive negative alterations in behavior. Occasionally there are good measures but they’re pretty rare

You’ll find individuals and teams that are working in the “expected” way will be absolutely fine. But their behavior will now be constrained by the metrics and their capacity to improve will be limited.

Those that are aware of their constraints (and are often limited in what they can actually influence to solve their problems) will at best sub-optimize around their own goals and at worst “game” the system in order to preserve themselves. This is a natural self-preservation response.

I’ve seen the most extreme example of this using game theory in simulated product management (in a session run by Rich Mironov back in 2009).

A team of 4 product managers were each given a personal $1M sales target, a set of delivery resources and a product. Performance against their target was measured on a quarterly basis. In the example, the game was deliberately rigged. It was impossible for all product managers to meet their personal targets with the limited resources they were given. However It was possible to achieve well above the total $4m target if product managers collaborated and in some cases were actually willing to sacrifice their own products, releases and resources in order to fund cycles on better-performing products.

Data may also only tell you how tools are being used. If a team is constantly inspecting and adapting, I would expect their tool usage to change. It may then not reflect expectations or worse, they may not be able to adapt for fear of damaging “the numbers”.

Here’s a great example of this from (of all places) a hair Salon… http://lssacademy.com/2007/08/05/bad-metrics-in-a-hair-salon/

For a closer to home example try this experiment…

If you’re not already doing so, start measuring story cycle time (from commencement of work to acceptance and ideally delivery).

Now try the following:

  1. Measure cycle times without giving feedback to the team. What are you observing in the data? What can you infer from this? What can’t you see?
  2. Continue measuring but start reporting the numbers to the team and discussing observations. Ask the team what they’re observing, what they can infer and what can’t they see. What would they change?
  3. Ask your teams if they’re willing to report their data to “management”. What responses do you get?
  4. If the teams are willing. Start reporting the data to management. Ask them what they’re observing and what they can infer? What Can’t they see? What would they change?
  5. Consider the level of trust in your organization. How would the experiment above change behavior if trust was significantly higher or lower than its current level?

Food for thought.

Leave a Reply

Your email address will not be published. Required fields are marked *