The Pitfalls of Measuring “Agility”

Share
Reading time ~7 minutes

This post expands on one of the experiences I mentioned in “Rapunzel’s Ivory Tower“.

I presented these lessons and the story at Agile Cambridge back in 2010.  It’s taken nearly 5 years to see the light of day in writing on here. I hope it’s not too late to make a difference.

I and my team hadn’t been in our roles long. We’d been given a challenge. Our executives wanted to know “which teams are agile and which aren’t” (see the Rapunzel post for more). We managed to re-educate them and gain acceptance of a more detailed measurement approach (they we were all Six Sigma certified – these people loved measurement) and I’d been furiously pulling the pieces together so that when we had the time to work face to face we could walk away with something production ready.

Verging on quitting my job I asked James Lewis from Thoughtworks for a pint at The Old Spring. I was building a measurement system that was asking the right questions but there was no way I could see a path through it that would prevent it being used to penalize and criticise hard-working teams. This was a vital assessment for the company. It defined clearly the roadmap we’d set out, took a baseline measure of where we were and allowed us and teams to determine where to focus.

My greatest frustration was that many of the areas teams would score badly were beyond their immediate control – yet I knew senior management would have little time to review anything but the numbers.

James’ question he left me with was:

“How do you make it safe for teams to send a ‘help’ message to management instead?”

I returned to my desk fuelled by a fresh pair of eyes and a pint of cider. I had it!

At the time many agility assessments had two major flaws.

1 – they only have a positive scale – they’re masking binary answers making them look qualitative but they’re not.

2 – They assume they’re right – authoritative, “we wrote the assessment, we know the right answers better than you…”

  • What if the scale went up to 11 (metaphorically) How could teams beat the (measurement) system.
  • And what if 0 wasn’t the lowest you could score. What would that mean?

The assessment was built using a combination of a simpler and smaller agility assessment provided to us by Rally plus the “Scrum Checklist” developed by Henrik Kniberg, the “Nokia Test“, the XP “rules” and my own specific experiences around lightweight design that weren’t captured by any of these. As we learned more, so we adapted the assessment to bake in our new knowledge.  This was 2009/2010, the agile field was moving really fast and we were adding new ideas weekly.

The results were inspired – a 220 question survey covering everything we knew. Radar charts, organizational heat maps, the works.

The final version of the assessment (version 27!) covered 12 categories with an average of about 18 questions to score in each category:

  1. Shared responsibility & accountability
  2. Requirements
  3. Collaboration & communication
  4. Planning, estimation & tracking
  5. Governance & Assurance
  6. Scrum Master
  7. Product Owner
  8. Build and Configuration Management
  9. Testing
  10. Use of tools (in particular Rally)
  11. Stakeholder trust, delivery & commitment
  12. Design

The most valuable part was the scale:

  • -3 We have Major systemic and/or organizational impediments preventing this (beyond the team’s control)
  • -2 We have impediments that require significant effort/resource/time to address before this will be possible (the team needs support to address)
  • -1 We have minor/moderate impediments that we need to resolve before this is possible (within the team’s control)
  • 0 We don’t consider or do this / this doesn’t happen (either deliberate or not)
  • 1 We sometimes achieve this
  • 2 We usually achieve this
  • 3 We always achieve this (*always*)
  • 4 We have a better alternative (provide details in comments)

a radar chart from excel showing each category from the assessment

The assessment was designed as a half-day shared learning experience. For any score less than 3 or 4, we would consider & discuss what should be done and when, what were the priorities, where did the team need support, what could teams drive themselves and what were the impediments. Teams could also highlight any items they disagreed with that should be explored.

Actions were classified as:

  • Important but requires management support / organizational change to achieve
  • Useful, low effort required but requires more change support than low hanging fruit
  • Potential “low hanging fruit”, easy wins, usually a change in practice or communication
  • Important but requires significant sustained effort and support to improve

As a coaching team we completed one entire round of assessments across 14 sites around the globe and many teams then continued to self-assess after the baseline activity.

Our executive team actually did get what they needed – a really clear view on the state of their worldwide agile transformation. It wasn’t what they’d originally asked for but through the journey we’d been able to educate then about the non-binary nature of “being agile”

But the cost, the delays, the iterative approach to developing the assessment, the cultural differences and the sheer scale of work involved weren’t sustainable. An assessment took anything from an hour to two days! We discovered that every question we asked was like a mini lesson in one more subtle aspect of agile.  Fortunately they got quicker after the teams had been through them once.

By the time we’d finished we’d started to see and learn more about the value in Kanban approaches and were applying our prior Lean experience and training rather than simply Scrum & XP + Culture. We’d have to face restructuring the assessment to accommodate even more new knowledge and realized this would never end. Surely that couldn’t be right.

Amongst the lessons from the assessments themselves, the cultural differences were probably my favourite.

  • Teams in the US took the assessment at face-value and good faith and gave an accurate representation of the state of play (I was expecting signs of the “hero” culture to come through but they didn’t materialize).
  • The teams in India were consistently getting higher marks without supporting evidence or outcomes.
  • Teams in England were cynical about the entire thing (the 2-day session was one of the first in England. Every question was turned into a debate).
  • The teams in Scotland consistently marked themselves badly on everything despite being some of our most experienced teams.

In hindsight this is probably a reflection on the level of actual knowledge & experience of each site.

Partway through the baseline assessments after a great conversation with one of the BA team in Cambridge (who sadly for us has since retired) we added another category – “trust”. His point was all the practices in the world were meaningless without mutual trust, reliability and respect.

It seemed obvious to us but for one particular site there was so much toxic politics between the business leadership and development that nobody could safely tackle that he had an entirely valid point. I can’t remember if we were ever brave enough to publish the trust results – somewhat telling perhaps? (Although the root cause “left to pursue other opportunities” in a political power struggle not long before I left).

Despite all this the baselining activities worked and we identified a common issue on almost all teams. Business engagement.

We were implementing Scrum & XP within a stage-gate process. Historically the gate at which work was handed over from the business to development was a one-way trip. Product managers would complete their requirements and them move on to market-facing activities and leave the team to deliver. If a team failed to deliver all their requirements it was historically “development’s fault” that the business’ numbers fell short. We were breaking down that wall and the increased accountability and interaction was loved by some and loathed by others.

We shifted our focus to the team/business relationship and eventually stopped doing the major assessments. We replaced them with a 10 question per-sprint stakeholder survey where every team member could anonymously provide input and the product managers view could be overlaid on a graph. This was simpler, focused and much more locally & immediately actionable. It highlighted disconnects in views and enabled collaborative resolution.

Here’s the 10 question survey.

Using a scale of -5 to +5 indicate how strongly you agree or disagree with each of the following statements  (where -5 is strongly disagree, 0 is neutral and +5 is strongly agree)

Sprint

  • The iteration had clear agreement on what would be delivered
  • The iteration delivered what was agreed
  • Accepted stories met the agreed definition of done
  • What was delivered is usable by the customer
  • I am proud of what was delivered

Release

  • I am confident that the project will successfully meet the release commitments
  • Technical debt is being kept to a minimum or is being reduced

General

  • Impediments that cannot be resolved by the team alone are addressed promptly
  • The team and product manager are working well together

If you’re ever inclined to do an “agile assessment” of any type, get a really good understanding of what questions you’re trying to answer and what problems you’re trying to solve. Try to avoid methodology bias, keep it simple and focused and make sure it’s serving the right people in the right ways.

Oh – if you’re after a copy of the assessment, I’m afraid it’s one of the few things I can’t share. Those that attended the Agile Cambridge workshop have a paper copy (and this was approved by the company I was at at the time) but I don’t have the rights to share the full assessment now I’m no longer there. I also feel quite strongly that this type of assessment can be used for bad things – it’s a dangerous tool in the wrong circumstances.

Thanks – as always – for reading.

Seeing the Value in Task Estimates

Share
Reading time ~5 minutes

a list of task estimate sizes with beta curves overlaidYou might be aware of the ongoing discussions around the #noEstimates movement right now. I have the luxury here of rarely needing to use estimates as commitments to management but I usually (not always) still ask my teams to estimate their tasks.

My consistently positive experiences so far mean I’m unlikely to stop any time soon.

3 weeks ago I joined a new team. I decided I wanted to get back into the commercial side of the business for a while so I’ve joined our Sales Operations team. (Think DevOps but for sales admin, systems, reporting, targeting & metrics).

Fortunately for me the current manager of the team who took the role on a month or so earlier is amazing. She has so much sales domain knowledge, an instinct for what’s going on and deeply understands what’s needed by our customers (the sales teams).

I’d been working with her informally for a while getting her up to speed on agile project management so by the time I joined the team already had a basic whiteboard in place, were having effective daily standups and were tracking tasks.

The big problem with an ops team is balancing strategic and tactical work. Right now the work is all tactical, urgent items come in daily at the cost of important but less urgent work.

We’re also facing capacity issues with the team and much of the work is all flowing to a single domain expert who’s due to go on leave for a few months this Summer – again a common problem in ops teams.

I observed the movement of tasks on the team board for a week to understand how things were running, spot what was flowing well and what was blocked. As I observed I noted challenges being faced and possible improvements to make. By the end of the week I started implementing a series of near-daily changes – My approach was very similar to that taken in “a year of whiteboard evolution“.

Since the start of April we’ve made 17 “tweaks” to the way the team works and have a backlog of nearly 30 more.

Last week we started adding estimates to tasks.

I trained the team on task estimation – it took less than 10 minutes to explain after one of our standups. The technical details on how I teach this are in my post on story points. But there’s more than just the technical aspect. (In fact the technicalities are secondary to be honest)

Here’s the human side of task estimation…

  • Tasks are estimated in what I describe as “day fragments” – essentially an effort hours equivalent of story points. These are periods of time “small enough to fit in your head”.
  • The distribution scale for task estimates I recommend is always the same. 0.5, 1, 2, 4, 8, 16, 24 hours. (the last 3 are 1, 2 and 3 days) – It’s rare to see a task with a “24” on it. This offers the same kind of declining precision we see with Fibonacci-based story point estimates.
  • For the level of accuracy & precision we’re after I recommend spending less than 30 seconds to provide an estimate for any task. (Usually more like 5-10)
  • If you can’t provide an estimate then you’re missing a task somewhere on understanding what’s needed.
  • Any task of size 8 or more is probably more than one task.
  • Simply having an estimate on a task makes it easier to start work on – especially if the estimate is small (this is one of the tactics in the Cracking Big Rocks card deck)
  • By having an estimate, you have a better idea of when you’ll be done based on other commitments and activities, this means you can manage expectations better.
  • The estimates don’t need to be accurate but the more often you estimate, the better you get at it.
  • When a task is “done”, we re-check the estimate but we only change the number if the result is wildly off. E.g. if a 1 day task takes just an hour or vice versa. And we only do this to learn, understand and improve, not to worry or blame.

So why is this worth doing?

Within a day we were already seeing improvements to our flow of work and after a week we had results to show for it.

  • The majority of tasks fell into the 0.5 or 1 hour buckets – a sign of lots of reactive small items.
  • Tasks with estimates of 8 hours or more (1 day’s effort) were consistently “stuck”.
  • We spotted many small tasks jumping the queue ahead of larger more important items despite not being urgent. (Because they were easier to deliver and well-understood)
  • Vague tasks that had been hanging around for weeks were pulled off of the board and replaced with a series more concrete smaller actions. (I didn’t even have to do any prompting)
  • Tasks that still couldn’t be estimated spawned 0.5 or 1 hour tasks to figure out what needed to be done.
  • Large blocked items started moving again.
  • Team members were more confident in what could be achieved and when.
  • We can start capacity planning and gathering data for defining service level agreements and planning more strategic work.

I’m not saying you have to estimate tasks but I strongly believe in the benefits they provide internally to a team.

If you’re not doing so already, try a little simple education with your teams and then run an experiment for a while. You can always stop if it’s not working for you.

 

 

A quick update – Janne Sinivirta pointed out that “none of the benefits seem to be from estimates, rather about task breakdown and understanding the tasks.”

He’s got a good point. This is a key thing for me about task estimation. It highlights quickly what you do & don’t understand. The value is at least partially in estimating, not estimates. (Much like the act of planning vs following a plan). Although by adding the estimates to tasks on the wall we could quickly see patterns in flow of tasks that were less clear before and act sooner.

As we move from tactical to strategic work I expect we’ll still need those numbers to help inform how much of our time we need to spend on reactive work. (In most teams I’ve worked in it’s historically about 20% but it’s looking like much more than that here so far).

Martin Burns also highlighted that understanding and breaking down tasks is where much of the work lies. The equivalent of that in this team is in recognising what needs investigation and discussion with users and what doesn’t and adding tasks for those items.

Ship Early – Why New Software Sucks

Share
Reading time ~4 minutes

I’ve been working in software product development companies for nearly 20 years.

Until 4 years ago I’d always been involved in “enterprise” software.

You know – the monolithic systems with great reporting capabilities that sell well to managers and (at least historically) poor UX for the real users. Those same products that promise the moon during slick demos, require 6-12 month sales cycles where complex pricing structures are worked through and year-long implementations are agreed.

I’ve helped set up demos, I’ve watched amazing presales engineers work all night to rewrite chunks of an application to demonstrate to a prospect that it’ll meet their unique requests.

And then during implementation you discover some of it just doesn’t work.

I implemented one company’s products internally. Dogfooding from the day V1.0 was released. In 6 months I raised over 50 showstopper bugs (almost all were found by their first real customer not long after us). On a visit to HQ in San Francisco, after a fair bit of wine at a great Italian restaurant in Burlingame (the House of Garlic if I remember correctly) I challenged the then VP of Development for that application on why they shipped blatantly unfinished software.

His answer was simple,  logical and for a young, inexperienced graduate analyst programmer my first window into the commercial realities of product development. He said;

“Market timing”

“If we released when the software was actually finished, we’d be beaten to market by our competitors.”

“It’s acceptable business practice because it takes 6 months to sell and we can’t start selling until the product is released and in our price books.”

“Even after the sale it takes months to implement so by the time users are ready to go live we’ve fixed all the major issues because you guys are dogfooding it for us”.

It made a whole lot of sense but it wasn’t something they ever told us on the project!

The thing is, everyone else is on the same bandwagon and the escalation games start rolling.

Products are brought to market earlier and earlier in their maturity with the knowledge that “nobody trusts a v1.0 version of a product anyway.” It continues today. Many large banks won’t touch an x.0 version of a product until the first major maintenance release is shipped.

My experience was the same. In most companies I worked for; when a new release of the DB platform shipped, we’d plan on adopting it sometime after the first 6 months out in the wild when we knew it was stable.

The game has moved on. It’s no longer just products that take a year to sell and implement so our exposure to early releases is increasing and in general so is the entire industry tolerance (there are obvious exceptions in safety-critical systems).

In my time at Oracle in the late ’90s I saw them try to change this world. They had a vision known as”Gray Iron”. It seemed brilliant – for business, development teams and customers. The idea was that based on a playbook of “best practice” business processes a customer could buy a server entirely preconfigured and working out of the box to support their entire financial, ERP and CRM set of processes. They could simplify the pain of users, ensure quality was high and radically reduce implementation times. Obviously this also offers a great potential competitive advantage.

Sadly it didn’t take off. Competitors were still playing the “ship early” escalation game so we had to play too – the poor system fed itself.

The surge of change brought about by Eric Ries’ book The Lean Startup has fanned the flames of this mentality. Ship early, get customer feedback and adjust.

The sad thing is this used to be only in the enterprise market but the world has turned. Consumer electronic devices now face the same market timing escalation issues as this article from December on Forbes highlights.

It’s now common for us to expect our phones to need a reboot or crash and even for consumer software to have significant glitches.

It’s great for the pace of innovation but spare a thought for the end users and customer support teams.

It’s a commercial reality we can’t avoid but let’s make sure we’re not sacrificing the end user experience. If you must use your customers as lab-rats, let them opt in or out of your experiments. Let them decide if they want “bleeding edge”, “new” or “stable” and honor their wishes.

If your product is even remotely valid you probably have enough early adopters willing to validate the bleeding edge without sacrificing laggards on the altar of possible futures.

What’s your “ship early” policy like?

Recovering The Ability to Self-Organize

Share
Reading time ~11 minutes

At the beginning of this year the team I’ve been supporting hit a wall. After a deeply troubled project last year they’d started researching a “new” feature in December with a plan to get rolling in January.

After the holiday break, a couple of weeks into January we had a near-mutiny and hit the reset button.

Things are now back on track and working in a radically different way but here’s the story.

Timeline

Early August – Halfway through a deeply troubled “rewrite” project. One of the paired project managers left and I was asked to support the remaining PM.

Mid August – Working with the remaining PM we visualised the entire scope of the UI on the team’s whiteboard and immediately excluded all dialogs and rarely used parts of the UI from the project. Essentially halving the scope of work

Late August – The second PM moved back to a development role elsewhere and I joined the team full-time. I started digging deeper into the scope and data. After reconciling the stories the team had on the wall with those in the electronic backlog we discovered as much additional scope as had been removed 2 weeks earlier – literally “hiding in the walls”.

Early September – Having reviewed the state of the project I talked to our engineering manager and product owner and said “this project is f***ed, do you want to kill it?”  The response was “No, we believe this is the most important and valuable thing we can be doing right now.”

Late September – After the first attempts at recovery still hadn’t achieved enough I re-asked the same question – the reply was the same

Mid-October – And again. All the while the team are still working away trying to deliver.

Late October – Our Product Manager (not the Product Owner) finally decided the project needed a reset. The forecasts had finally tipped him over the edge and the total cost wasn’t acceptable. (This coincided with our push to get an Alpha release into the hands of users for concrete feedback.)

Mid-November – We shipped an alpha release to validate our new UI technology and started building plans and options for the finished product release.

Mid-November – We sought input from the team on options on how to exit the project gracefully. (there’s an entire article on this experience waiting to be written)

End November – We killed the project and I started on building a plan to move on – leaving the team with a new manager and have them successfully working on a new release by the end of January

December – A general lull. Recovery time. We had Down-tools week, holiday period, some bug fixing and a couple of the team started investigating the next feature. My replacement joined the team and we started on a gentle 2-3 month handover.

January – When the team returned from holidays we decided to start on the “new” feature (actually finishing a feature started 2 years earlier and never finished). One of my team had the task of reviewing current behaviour and delivering a demo to the team. This was eye-opening. The scope that had been implied from PO conversations didn’t address the fact that the feature was barely usable.

Mid-January – Over the 3 weeks this new feature had started (as a “3 month” project) scope had ballooned again to include rewriting a fundamental aspect of how data would be stored.

Mid-January – Team mutiny (During a retrospective). The feedback was honest and painful but most importantly it was actionable and had cemented the worries I already had – things were broken again and we needed to hit the reset button a second time. Although he was already on board, this was not the time to hand over to my replacement.

End January – Reset time. Feedback in hand, we had to act fast. I redesigned the entire way work flowed into and through the team, gave some direct and difficult feedback to our new product manager, engaged him in the reset and briefed the team on changes.

Mid February – With the reset well under way and team morale already on the increase I could safely hand ownership over without feeling guilty about the state of the project.

But what went wrong?

There were a catalogue of problems raised by the team. Here’ the points the team raised…

  • We talk about “quick wins” but never do them
  • Poor control of scope for the last 2 major projects
  • A team of 10 with 4 managers is consistently underperforming another team of 3 with no management – why is that?
  • The project backlog is only technical tasks, not user stories
  • We still don’t know what the focus of the next product version is
  • We’re deeply worried about the quality of the next feature (based on current scope)
  • We’re not bought into the new feature work – we’re repeatedly told users don’t need it and it’s just a checkbox
  • We want to be more reactive to support requests
  • We’re not delivering quickly to users or responding to their feedback
  • We want to feel proud of the product and what we’re delivering
  • The work isn’t structured to be delivered iteratively
  • There’s a complete lack of user perspective on upcoming work
  • Despite 4 managers, we have no leadership.

Brutal – but completely valid

For the last 6-12 months we’d been entirely focused on helping the team deliver what the business had been asking for. This was a good team. Sure they’d had a lot of technical problems on the previous project but the new one should have been different.

Both projects started with a technical requirement – rewrite X, finish Y.

This built some odd behaviour into the team. They were at the mercy of the backlog. They couldn’t self-organize, they couldn’t deliver iteratively, they couldn’t manage scope, they couldn’t even decide the best solution for our users. Everything was pre-defined.

I hadn’t spotted the problem for months. I’d been so focused on delivery of what we’d been given that I failed to identify the biggest issue was the incoming work itself!

After killing this second project, the next feature on the list was called “Implement Z” – where Z was a half-written product (again).

I now have two “stop” phrases added to my list of dangerous projects;

  • “it’s just…”
  • “3 months”

These are gross generalisation phrases much like “finish X” or “Implement Z”, they’re the smell of unbounded requirements. Taking them at face-value and accepting work into the team is a recipe for failure.  Both failed features started out on the same two flawed premises but beyond that, take another look at the list of issues raised by the team.

We’d let the team down. We were managing poorly, not leading and the features were not user-facing. They might once have been but not by the time they reached the team.

How do things look now?

First we clarified who in the management team was responsible for what. Who was “in charge” from the business, who was “supporting” and who was “managing”.

Then we restructured our view and approach to incoming work into 4 balanced streams with clear constraints, policies and ownership for each.
This is probably best explained using the presentation I ran with the team…

We asked the team during the presentation who would be willing to take on the role of “feature lead” for upcoming features. Every hand in the room went up!

We’re just reaching the 2 month checkpoint and things are significantly better.

  • The fixes and achievable streams are generally running smoothly and shipping frequent product updates.
  • The team have learned what sizes of work and level of understanding do/don’t work well as “achievable” items.
  • The feature lead role and relationship with the business is successfully flushing out real user needs.
  • The analysis/discovery stream took longer than the team were hoping for but development of the first major feature off the back of the research work has just started.
  • The team completed their first user story mapping session to plan a genuinely iterative set of potential releases.
  • The first iteration of a new major feature is (optimistically) planned to finish in 2 sprints.
  • At any point after the initial release we could stop and move on to something else.
  • After the first release the team will have the ability to run UX sessions, gather feedback and adjust plans based on real user responses.
  • The team’s new project manager has been able to take his sabbatical and leave the running of the project to the team for 6 weeks! (with occasional check-ins and support from me)

Better still, Nick (my successor) has been gathering measurable feedback from the team since we started the reset. Here’s the data.

team_happiness

From Architectural Decoupling to a Complete Rewrite

For a lesson in what not to do, here’s the gory detail on the first project that was killed.

The project goal was to “make it significantly easier to change the UI in future” – delivering value to internal stakeholders in the form of efficiency and quality savings.

This was conceived as “3 month” project back in April but ran into trouble over the summer. The PMs involved in the start of the project had assembled enough project data to establish “This could take anything up to another 1-2 years to complete”.

The project scope had changed from architectural decoupling to a an internal technology refresh and from a technology refresh to a pilot for a new (unfinished) product style guide, a complete UI replacement, implementing a set of untried technologies with a team that had little or no experience of using those technologies and improving the user experience of the UI along the way.

Essentially they were going for a complete unbounded UI rewrite.

The initial idea was to introduce the new UI as an MVP alongside the old UI with limited functionality and support for end-to-end workflows.

Somewhere in the course of time the “end-to-end” part of this had been lost.

By the time I joined, the implementation approach had become replacing whole tabs of the UI, developing multiple tabs in parallel to improve and validate code reuse and completing these widget-by-widget rather than by workflow.

On the surface, technically this seemed sensible. It reduced rework and kept architectural quality high.

I made things worse by pointing out that an MVP wasn’t viable for a product whose broad set of features were in daily use by over 1,000 customers. Thus it became a “like for like” replacement project – of which so many end up doomed by a big bang approach and repeated delays as scope is discovered!

Shipping a new release with significantly cut-down functionality is a big gamble and usually only one you can take if you have something else of significant value to offer to the customer.

We didn’t.

In fact, the most commonly requested feature from our customers – a rewrite of some particularly nasty functionality to support additional use cases and improve existing ones – wasn’t being worked on. We’d actually decided to remove the poorer version of that feature and not replace it until a future release.

In hindsight and from the outside this all seemed a bit nuts.

Before I joined the team, in my oversight role across all projects in the company I was aware of a nagging worry about the state of the project but I couldn’t put my finger on why. The reports and whiteboards seemed “ok” or “meh” but not fundamentally bad.

This is now another warning sign to us when reviewing project status. Month after month of “ok” (not “good”) status across an entire series of checks is a smell (see “Yellow is the new Green” )

I repeatedly asked about any deadlines, due dates and what was going on and received reassurance that everything was under control but there weren’t any real deadlines so everything was still “OK”.

As “Head of Project Management” I have to admit it’s pretty embarrassing to not have tackled this wave of trouble. I and others in the leadership teams saw it coming but trusted the feedback we were seeing. We continued to treat everyone as knowing what needed to be done and safe to be able to do so (“S4″ in Situational Leadership terms) when actually they were in trouble and needed help. (“D2″ level).

Things came to a head at the beginning of August with the project on the rocks, one PM left and another wanted to go back to a development role.

Determined not to face a “Netscape project” I joined the team to get things back on an even keel. Until now, I’d never had a project I couldn’t salvage quickly – usually in under a couple of weeks. This one was definitely much more of a challenge.

Fortunately the team had backlog and historic size & velocity data so we at least had a baseline to work from (the outgoing PM pointed out that the estimates were wildly inconsistent and hard to trust).

Step 1 – We visualised the entire scope of the UI and discovered some 30+ dialogues and little-used screens that could be postponed. We took a scythe to the scope listed in the backlog based on this visualisation and managed to improve the forecast delivery dates.

Step 2 – I was unhappy that the partial backlog on the wall didn’t offer easy traceability to the full digital backlog so I tagged up all the stories and reconciled the wall with the backlog in a spreadsheet and discovered another 50% of scope hiding in the walls.

At this point the project had ballooned beyond all recognition of the original. We’d dropped into sunk costs and the approach to delivery had removed our ability to deliver iteratively.

Every sprint that passed uncovered more technical problems and more missed scope. Some major performance issues with the new technology tripped the team up for weeks and even after shutting the project down, taking what was delivered as an Alpha in November, salvaging the most complete parts and getting it shippable involved 3 more months of stabilising the technology – insanity!

Fortunately, the salvaged tech finally paid off. After releasing the main work, the team were able to deliver a series of small enhancements near weekly for the next month.

They’re through the worst now and I’m moving onto another team.

The team did a great job staying sane in spite of the craziness around them and both they and “management” have learned a lot . What we have in place now goes beyond preventing a repeat of all these problems – it resets the whole way they work to focus on customer needs, customer problems and customer value. It gives them space and freedom to self-organise whilst balancing the needs of the business when necessary.

I’m sad it took nearly 9 months to get here but glad we’re finally on the right track.

We’re also using the lessons from here to ensure other projects that threaten a rewrite focus on incrementally delivering end user value by only rewriting areas that directly improve a team’s ability to deliver on the next customer-facing changes and features.

Update – 24th April 2015:

I had a chat with one of the team who reviewed this for me. Other than some tweaks to the early timeline he mentioned that as of a couple of weeks ago the team have now actually deleted about 2/3 of the UI code that was developed but never finished. That’s a really brave move given the effort they all put if but they found it was holding them back in adding new features (using the new UI technology in smaller increments).

He also mentioned that the first new feature to run through the updated team analysis process is flying – a way better start than the last!

They’re is still a great team and have really turned the corner by the looks of things now.

1,000% improvement is a statistical outlier

Share
Reading time ~3 minutes

This cautionary tale came up a few times at Agile 2010 (yes I know this may be old news!) – including one of the keynote speeches.

“For a million dollars we’ll make you Agile.

Here’s a list of previous clients whom we helped achieve a 1,000% performance improvement. Lead times halved, profits doubled and everyone was AWESOME!.

Here’s some REAL DATA …”

A graph going "up and to the right"

Axis titles shamelessly acquired from a recent company presentation

What you’ve not been told is that those testimonies are statistical outliers!

These are the top 1-5% of companies that have successfully undergone a major transformation. (or the bar was set very low to start with) There are thousands of companies out there that don’t reach these lofty heights.

The journey is longer and harder than the marketing will ever tell you but that’s fine as long as you know what you’re investing in and why.

Your organization is unique. There are many factors about your organization that will make significant improvements hard to achieve and most of them will not be technical. The forces of resistance will be many and will be a mix of institutionalized and personal.

Let’s replay this with a more realistic conversation…

A consultant visits your site, and does a “free” one-day Agile Assessment of your teams.

“OK Boss. Right now you suck at developing software. Give us a million dollars, a year of your time and a willingness to drop productivity for a while and we can make you suck a bit less.”

Furthermore, they won’t actually be able to tell you just how much less you can suck and by when.

Why not?

Back to all those forces of resistance – how many of those can you really prod, assess and quantify in a day or even a week?

Every company, organization and site differs.

The investment may still be worthwhile but it’s time to manage expectations better. Those assessments should highlight where unexpected limitations lie. Maybe they could offer alternative higher priority areas to tackle (rather than up-selling scope).

If product development and software engineering was like cutting coke cans, there really would be a “one true right way” of producing software products.

Moreover. There wouldn’t be a thousand consultancies promising you the moon on a stick, there wouldn’t be conferences on improving the state of the art and there wouldn’t be hundreds of books full of great ideas on how to do/be agile or perform software engineering a little bit better.

In fact chances are there wouldn’t be a competitive software industry.

Or would there?

• Maybe there really is a one true way and the entire software, consulting, authoring and conference industry is in on the joke.

• Maybe all those leading lights on their skiing trip a decade ago came up with one of the world’s greatest “Long Cons”

• Oh and they invited 3M to the party and agreed to promote stickies in the 21st century in exchange for a marketing budget.

Probably not…

There are no “best practices”. Stop looking for them, stop asking consultants and Scrum Masters to implement them.

There are only “best known practices for your current state, knowledge and context”.

When your state, knowledge and context change, it’s time to look at what’s next and more importantly – what’s beyond your current focus – what have you missed or not considered yet?

What did you discard previously because there were constraints or issues preventing them working? (I learned a great term for this from Chris Matts &/or Dan North – I can’t remember which – “Idea Wombling” – seeking out great ideas that were previously discarded)

You may reach a point where your organizational immune system (politics and process) blocks progress.

Sometimes you’ll need outside help to see what’s “better” or learn new ways of working. That outside help can often be more effective at delivering hard truths than you can yourself. It’s worth investing in “straight-talk” from strangers sometimes.

Figure out what is holding larger improvements back (and where) and determine who you could pair with either externally or in a different part of the organization to make a real difference!