On Story Points and Distributions

Share

If you’ve been reading my posts for a long while, you might remember this curve in relation to fixing bugs. Today I’m resurrecting it for other reasons.right-skewed beta distribution First of all I’ll admit I used to be a bit of an estimation geek. I loved the subject. (really!)

Last summer I attended my first ever #noEstimates session at Agile 2014. What I heard made complete sense to me and there are a few teams here that have stopped estimating (they still measure and forecast throughput) but I still see value in trying to estimate. I’ve seen enough examples over the last decade where work is traded out because a team cannot estimate the effort of the previous.

I love Arlo Belshee’s concept of Naked Planning – in that if you remove all estimation information and focus purely on what the most valuable and important thing is, you’ll do the most valuable and important thing.

The trouble is, in many commercial product situations, it’s not actually that clear what the most valuable thing really is – and value generally has a trade-off of cost. (Or at least “how much am I willing to spend on this”).

I also like Chris Matt’s Options thinking alternative to estimates – “How much are you willing to spend to get a better understanding of this”. This ties in with a fundamental part of estimation theory in that there is a trade-off (and decreasing value) in spending additional time understanding a problem in order to better estimate the outcome.

Finally, whilst many teams I’ve worked with have used story point estimates for stories and hour estimates for tasks, we don’t go in blind. We have triangulation and reference points from prior work wherever possible.

I used to run half-day training courses on software estimation (and still see that course as valuable – primarily because the theory is transferrable). These days, if I have a team that have never had a real conversation about what estimates are, why do them and how I trim down that half-day content into about 15-30 minutes on the most important bits for story and task estimation.

I’ve been using the same explanations on estimation for about 7 years now and although some of my assertions on why we estimate are finally wavering, the information on how is still useful – and – as far as I know, nobody else has explained it this way.

Back in 2011 I wrote an article on Swimlane Sizing that was subsequently referenced and made popular by Alexey Krivitsky in his Scrum Simulation With Lego Bricks paper.

What I want to share today is how all this hangs together based on a very simple concept from 1950’s.

The PERT technique came from the Polaris missile project and in my very simple terms is essentially a collection of tools and techniques based around probability distribution.

When examining the probability of completing any task there’s a great rule of thumb to start with:

There’s a limit to how well things can go but no limit to how bad they could get!

With this thinking in mind, essentially the completion time and/or effort for a given task can be represented by a probability distribution. A right-skewed beta distribution.

Furthermore, when adding a series of these together, you have a collection of averages. (cue a #noEstimates discussion).

If you ask someone to estimate how long a bug will take to fix, without historic data you’ll get a “how long is a piece of string” type answer. Everyone remembers the big statistical outliers but if you strip these away, you can forecast quite accurately with about 95% confidence. (This is one of the foundations behind using data for service level agreements in Kanban).

Some items will take longer than average, some will be faster but based on those averages, you can get a “good enough” idea on durations.

With PERT, this beta distribution is simplified further to 3 points – “optimistic”, “most likely” and “pessimistic”. The math is simple but at least for today’s point, not important.

Here’s the bits I care about.

PERT summary

Now here’s a neat thing (and the point of this post).

With the introduction of story points, we’ve moved away from the amazing power of  3-point range estimation back to single points.

Once you’re in the realm of single point estimates, people start seeing a falsely implied precision. Unless you’ve actually had training on use of story points (many senior managers probably won’t have done) then you’ll start building all those same human inferences that used to occur with estimates like “It’ll take about 2 weeks”.

When we hear “2 weeks”, we leap to a precise assumption and start making commitments based on that. In range estimation we’d say It’ll take 1 to 4 weeks and in PERT we’d say, “Optimistically it’ll take a week, most likely 2 and pessimistically, 4. Therefore we’ll plan on 2 but offer a contingency of up to a further 2 weeks.”

(Of course in reality, wishful hearing means you might still end up with a 1 or 2 week commitment but hopefully the theory is making sense).

 

It’s also worth examining the size of the gaps between optimistic -> most likely and most-likely -> pessimistic (in particular, the latter of these 2). These offer a powerful window into the relative levels of risk and uncertainty.

By moving back to single point estimates – at least at the single story level, this starts feeling a lot like precise and accurate commitments**. Our knee-jerk reaction may then be to avoid providing estimates again.

But here’s the missing link…

Every story point estimate is in fact a REPRESENTATION of a range estimate!

We can take this thinking even further…

The greater a story point estimate, the less precise it is.

Obvious right? Let’s take one more step…

 If a story point estimate is a representation of a range then a larger number implies a WIDER range.

Let’s take that back to the swimlane sizing diagram and (crudely) overlay some beta distributions…

swimlane-distributions

Look carefully at how those ranges fall.

  • Some 2 point stories take as long as a 3 point story.
  • Some 5 point stories may be the size of 3 point stories, in rare cases, others may end up being 13 or even 20 points.
  • And some 13 point stories may go off the scale.

And that’s all entirely acceptable!

On average a 5-point story will take X amount of effort.

That’s enough to forecast and that’s enough to start building commitments and service level agreements around (should you need to).

So…

Time to start thinking about what distributions your story point estimates are.

If you’re getting wild variation, how might you capture some useful (but lightweight) data to help you, your team and your management understand and improve?

You might be in a fortunate place where you simply don’t need to produce estimates at all. That’s great. I’d assert there’s a lot to learn by simply trying to estimate even if you don’t use the results for anything but for the majority of us that need to do at least some sane forecasting, this thinking might just make estimation a bit safer and a bit more scientific again.

**As an aside, a while ago, the SCRUM guide was updated and replaced “commitment” with “forecast”. That’s a big change and tricky to retrain into those that saw SCRUM as the answer to their predictability problems. (Many managers needed guarantees!) For those of you facing continuing problems here, it’s worth gathering and reviewing story data so you can build service level agreements with known levels of confidence as an alternative.

200 Questions to help bridge the Product Owner divide

Share

A year or two before I first started writing publicly, I read an interesting, varied and quite long essay by Jeff Patton on the Product Owner role in Agile development (particularly Scrum)
http://www.agileproductdesign.com/blog/2009/product_owner_and_problem_shaped_hole.html

Whilst there are some areas I agreed with, the discussion on “heroics” was an area that left me cold. In my opinion, great software teams don’t have space for individual heroics and (in my opinion again) the sustainable pace ethos generally also plays down team heroics (but that’s a debate for another time).

There was a comment in Jeff’s article that resonated with me that I made a note of at the time. Years later it’s still highly relevant and I wanted to re-raise it this week in the context of recent challenges I’ve been facing.

“For the product you’re working on right now, how will your company benefit after the product has shipped? If you understand how, do others?”

Although this question was aimed at Product Owners it’s something I’d ask all development team members to really think hard about.

Now go further. Are you solving technical problems and a series of requirements to ship a product release or do you really understand the value of the release you’re working on, its reason for being developed and its impact on specific customers, the market or the company?

How many times have you or your team examined the reason for delivering a release with a specific set of features and understood their aim, questioned the backlog and vision and better still, properly understood what you’re trying to deliver in terms of solving the needs of the user?

When I say “understood”, I mean really understood – not just prettied up to follow one or another user story format.

Consider this an opportunity to pause for thought.

Back in 2013 I attended a Retrospective Facilitators Gathering and was introduced to an amazing concept over lunch by Willem Larsen that he described as “200 Questions”. He explained an idea that comes from animal tracking.

When tracking an animal you explore the environment in a way that an every day person doesn’t even consider. You look at every twig, leaf and blade of grass and ask questions of how it came to be in its given state (and why) in order to determine if you’re on the right path.

My translation of this to my own work is to try and develop a genuine and deep curiosity for every item you look at. (Even if you don’t have time to answer all your questions)

At the same lunch, we tried it with a salt cellar

How many people have handled it since it was made?

Who designed it?

What inspired the design?

What’s it made of?

Where did the salt come from?

What process did the salt go through?

(we lost count of how many questions we asked)

Applying this same healthy curiosity to incoming product features or requests is incredibly powerful.

You know things are wrong when every request coming in is essentially a specific solution being proposed.

And you know how translating feature requests back to user stories in anything except a shopping cart or finance trading application seems to be really poorly explained in the agile community?

Try this as an experiment for getting unstuck…

  • Set yourself a target of asking 200 questions about the next big item coming in to your team.
  • Try different “lenses” for asking the questions (much like 6 thinking hats)
    • The “User” lens
    • The “Investment” lens
    • The “Functional” lens
    • The “Market” lens
  • What questions logically follow from those first “level 0″ questions?
  • Decide which of those are important and valuable to actually answer in order to further your understanding towards your goal.
  • Work on answering them – collaboratively.

In no particular order (but with some logical grouping around lenses) and in the very rough form I’ve been actively using in my current team, here’s over 50 of the “level 0″ questions I’ve started experimenting with.

Feature Info

  • Name – what do we call it?
  • What is it? (2-3 sentence user-facing description *and* 2-3 sentence product description)

User info

  • Describe how a user would do this now? (without new code)
  • What types of users does this help?
  • How does it help them? (what does it enable, unblock, simplify etc.)
  • What is a user trying to actually achieve when they do this?
  • Why?
  • What constraints are they facing at that moment?
  • Who can we talk to about their goals and needs for this? (a selection of people willing to talk to us more)
  • How complex/accessible (to the user) is it?
  • How many/what proportion of users will benefit from this?
  • How frequently might we expect different types of users to use this?
  • How might this detract from the product for other users?

Investment/Product/goal alignment

  • How does this improve the product?
  • How big/expensive is it?
  • How much time do we want to invest in initial investigation?
  • If an initial investigation is inconclusive, what next?
  • How much are we willing to invest in this? (and how flexible is that decision)
  • Is there anything obviously more important/valuable/beneficial to do before this?
  • Would we be fools not to do this in the next 90 days? – why/why not?
  • If this were the only thing we did in the next 30-90 days, is this the right thing to do?
  • Why is this valuable to the team/company?
  • How does it align to current product/company goals?
  • How does it further our journey toward those goals?
  • Would you describe this as a tactical or strategic change?
  • Why?
  • Does this really fit within the product we have already?
  • (how) will it help differentiate the product from the competition?
  • Could this be a new product?
  • Are there any other products it may integrate with or impact? (ours or elsewhere)
  • Does this align more closely to another product than this one?
  • Do any of our competitors already do this? (and how)

Functional info

  • What solution options exist already?
  • Can you give an example of how that would work?
  • Under what situations might that not be suitable/successful?
  • What additional solution options can we think of
  • What’s the smallest simplest thing that could possibly work?
  • What’s “good enough” to meet customer needs?
  • Is there an 80/20 solution option for this?
  • How can we make this more discoverable, useful and valuable to users?
  • Is there an ingeniously simple solution for users?
  • What would it take to make a user say “wow!” (and is it worth the extra effort?)
  • What is the best solution in terms of cost/functionality/usability/quality balance
  • What performance and quality criteria do we have for this?
  • What is explicitly out of scope?
  • Are there any cross-cutting or performance risks?
  • Are there any other significant risks to be investigated?
  • What dependencies exist?
  • If we ship an incomplete solution, what is the follow-on plan and forecast cost for completion?

Market

  • What is the market demand for this?
  • Is there any market or customer sensitivity around timing for this?
  • What marketing is planned or expected for this feature (if any)?
  • How do we expect our competitors to respond to this?
  • For what we’re planning to deliver, what do we expect from our users on social media?
  • How will we launch this feature (alpha/beta/freq update/big bang etc)?
  • How will this impact our position in the market relative to our competitors?
  • What impact are we expecting on sales/renewals by announcing / releasing this feature?
  • What impact are we expecting on our reputation by announcing / releasing this feature?
  • How will we sell / price this?

It’s a lot like writing 10 minute test plans – so quick and so powerful.

Give 200 questions a try on your next big feature request.

Take It Off-Site (Part 1) – The Bad and The Ugly

Share

Offsite meetings have a checkered past. Back in 2006 a colleague shared this painful article with me.  To quote from the main article:

“…only about 10 percent of executives consider offsites truly valuable. Half said they aren’t worth the time or money.”

Over the last 7 years I’ve seen the exceptionally high value offsite meetings can bring but also the stress and pain that often leads up to them.

At my last employer, the entire global executive and senior management team would have a 6-monthly operations review offsite that ran for 3 days. We booked out significant portions of a large hotel in the US and fly the entire team together from over a dozen sites around the world. Over 3 days, the leaders from each site would present their statuses, finances, roadmaps, portfolios, visions, strategies and plans to a group of about 50 of us. We’d put them under the grill and review their work in depth both as bosses and peers.

We’d take time to focus on specific strategic issues that benefitted from face-to-face focused collaboraton. (When your management team are distributed around the world. A few intense days of co-location often delivers far more decisions and clarifications than 6 months of project work and conference calls.)

The general approach for these was: Prepare, Pitch, Review, Collaborate, Revise, Re-pitch.

Sounds pretty sensible? – It is…        … if you know what you’re in for. 

The offsite workouts were usually brutal but exceptionally valuable. By the end of the week everyone’s plans were in far better shape, we all understood what each other were doing, found areas for shared benefit and made a lot of difficult decsions.

Better still, we spent a few days working, eating and drinking together. 3 days of deliberate convergence in order to stem the behaviour issues associated with extreme divergence kepts us functioning as a successful management team.

Whilst the outcomes were great, they took their toll. The pain behind the scenes generally involved a bunch of us working 2 weeks of long evenings and weekends to pull all the data, presentations and spreadsheets together. ($100m of software projects contain a lot of moving parts & data) Often we’d still be revising our work during the start of sessions and whoever went first bore the brunt of painful lessons and questions whilst everyone else frantically updated their slide decks to accomodate the new knowledge.

And here’s the thing – the lines of questioning…

Our Senior Exec at that employer was an exceptionally sharp, experienced, bright guy. After working through multiple iterations and having joined the operations team on the organizing side of the offsites I saw what made him tick. He had a brain full of powerful questions that he’d learned from experience and was constantly adding to them. He knew what to ask and when that would lift the lid on just the right barrel to find a body (if there were bodies to be found).

As a spectator, sometimes it’s satisfying seeing the blood on the floor when someone you think is an ass-kisser takes a roasting. It’s not so fun when it’s your own team.

I even helped develop a set of “difficult questions” for our Head of Operations in the weekend prior to one of these offsites. They were great questions to ask. The thought and effort put into developing them was a lot like spending an hour writing test cases up-front. We learned quickly what we really needed and wanted to know (and why). We were confident we could get under the covers of even the most prettied-up project report.

But we could have made the whole thing so much less brutal.

Much like writing test plans, if we’d just put the questions out for participants to review in advance they’d have adjusted their research and reporting to accommodate them rather than playing project question battleship on the day.

So here’s an action for you.

Next time you’re reviewing a project or a piece of work, capture the questions you regularly ask.

Now…

Share them

Share them with the people you’ll be asking beforehand.

Now they can answer those questions sensibly first and you’ll all have time to dig into more valuable insights.

I have a few more posts planned around “critical questioning”, plenty around running offsites and I’ll be sharing where I’ve been for most of the last year so keep your eyes open for more soon.

 

A Year of Whiteboard Evolution

Share

Back in December last year I started supporting Red Gate’s .NET Developer Tools Division. As of this month, we’ve restructured the company and from next week, the old division will be no more (although the team are still in place in their new home).

When I joined the team things were going OK but they had the potential to be so much more so I paired up with Dom their project manager and we set to work.

The ANTS Performance Profiler 8.0 project was well under way already and the team had a basic Scrum-like process in place (without retrospectives), a simple whiteboard and a wall for sharing the “big picture”.

I spent the first week on the team simply getting to know everyone, how things worked and observing the board, the standups and the team activities.

We learned some time ago here at Red Gate that when you ask a team to talk you through their whiteboard, they tell the story of their overall process and how it works. Our whiteboards capture a huge amount about what we do and how we do it.

I attempted to document and capture at least some key parts of the journey we’ve have over the year in which we released over a dozen large product updates across our whole suite of tools. This post is picture heavy with quite limited narrative but I hope you’ll enjoy the process voyeurism :) If there’s any specifics you have questions about, please ask and I’ll expand.

Next time, I think I’ll get a fixed camera and take daily photos!

The end result? Multiple releases of all 5 of our .NET tools including a startup and quality overhaul for our 2 most popular products, support for a bunch of new database platforms, full VS2013 support (before VS2013 was publicly released), Windows 8 and 8.1 compatibility and a huge boost for the morale of the team.  See for yourself if you’re interested!

Of course this is just one aspect of what I’ve been up to. You might notice the time between photos over the summer grew a little. See my last post for more insights into what happens at Red Gate Towers.

Some Thoughts on Project “Closure”

Share

I’m currently working with the team of function heads at Red Gate developing a “skills map” for all our development roles. (I’m primarily covering Project Management).

Having “bought-in” skills inventories at previous companies and failed with months of time and effort wasted, we’ve taken our approach from the ground-up, knowing our role definitions are pretty unique to us and managed to get to “usable” for trials in less than 2 weeks! (We’ll be sharing what we did in the near future)

Having mapped out the set of skills and directions at a high level, we’re in the process of taking each skill and defining what the skill actually means to us, why it’s important, what does “good” look like and where to turn for more info or help. Each of us on the team is trying a couple out for starters and then planning to crowd-source the rest.

I’ve decided to start with “Project Closure” as a skill that’s not too simplistic, is really important to us and an area where through most of my career I’ve not often seen a “great job” done.

Before I progress, what I mean by Project Closure is everything required to get from an in-flight running happily along project to “done” – the ramping-down.  This is Not what PMI, PRINCE2, APM or most other standard PM definitions define as “Closure”.

This disconnect is why I decided to write this post. Most PM resources talk about closure being sign-off, completion of project documentation and lessons-learned meetings or retrospectives (and a few other process and stakeholder type things).

Having done a bit of hunting, it turns out there’s very little information available on “ramp-down”. It’s no surprise projects overrun with the lack of support and resources that exist around this area! “Traditional” project closure happens once everything’s already been delivered.

Forgive me if I’m being a bit ranty here but surely that’s after the horse has bolted, right?

What about all that effort of putting all those loose project tentacles back in the box first?

An Octopus in a treasure chest

How do we get from an “in-flight” project to “Done”.

My best analogy to this comes from the “String and Octopus Guide to Parenthood” that my first ever boss sent me a copy of over 15 years ago.

5. Dressing small children is not as easy as it seems: first buy an octopus and a string bag. Attempt to put the octopus into the string bag so that none of the arms hang out.

Time allowed for this – all morning.

Closing out projects well is hard and it takes  longer than you ever expect it to.

I’ve not finished writing up the full skill definition yet but I wanted to share where I am so far for anyone out there that wants at least a slight pointer in the right direction…

(Note, this is my definition based on the context I work in at the moment. It doesn’t conform to any certifiable standard so don’t scream if you answer your PMI exam questions on closure with stuff from this and get it wrong.)

A Short Definition

Project Closure (as a skill) is the ability to take a project from actively running to successful delivery and completion with a happy team, low defects, no panics, and in a timely fashion.

This is usually meeting a planned release date if there’s one set but for some teams can also mean getting to a point where we’re shipping production quality software with completed, working features on a frequent, regular basis and have the ability to move a team successfully off of a project stream (ready to work on another) and close it down in a satisfactory way for the team, customers, product management and the business if needed.

Why Is This Valuable?

Unless we ship valuable working software regularly we don’t keep customers happy and we find it harder to attract new customers. Working on products that don’t ship for long periods of time also has a pretty negative impact on morale for everyone involved – we love getting our stuff in the hands of customers and making them happy & successful!

We also risk building up large quantities of undelivered code or features (“inventory” waste in Lean terms), racking up undiscovered defects, increasing project risk and technical debt. The longer a project or feature is under active development, typically the longer it takes to shut down and ship.

Being able to ship and close down a project  is vital to our ability to balance investment across projects and products as it’s likely we’ll always have less teams than things we want to deliver and need to be flexible around team size, availability and commitments.

What Does “Mastery” Look Like Here?

Someone handling project closure really well will have a good handle on the state of development in relation to the remaining time left for a project to run and how long it’s been since we last shipped a successful release.

You’d probably expect them to be looking at what’s needed to close out the project from the point it’s about 2/3 complete. As an example – if you’re running a 3 month project, then most of the effort from the PM in the last month will be around closing things down.

They’ll be winding things down in a controlled manner with no surprises and no extreme rushes to the goal. Known bugs will be visible, regularly reviewed and going down, release testing will be planned, scheduled and resourced – chances are there will be a full-team release test towards the very end. Documentation will be all well under control, a UI freeze will be planned and completed. Features will be winding down or completed and bug fixing will be moving from high-risk breaking type bug fixes to showstoppers, visible, cosmetic or quality issues and small functional gaps for tying off.

They’ll have worked with our marketing team or product manager to ensure marketing is ready to go. Our sales and support teams will be being briefed on the new features, and we’ll have probably completed at least one bug hunt.

In the week or 2 before closure we’d expect to see a controlled code freeze and whilst the PM might be having to negotiate and plan what’s coming next they’ll still have a really tight rein on scope, be making decisions on any new incoming issues and setting clear priorities.

They’ll be encouraging the team to support each other across roles and swarming on areas that need a real push whilst ensuring that we don’t lose sight of the original project goals, customers’ needs and product quality.

 

That’s it – so far. If you have anything useful to add to this, please dive in and comment (mind the tentacles)