It’s been a month since I last posted about the work I’ve been doing around career development. Things are moving well and trials of the process described in the first 2 posts in this series have been really successful and way more popular than I expected. Everyone I’ve spoken to about it has been really excited and we’re now doing an expanded second round of trials with another 20 staff before adjusting and rolling out as a tool to all our development teams.
This is just a quick post to highlight a few tweaks and important points for anyone wanting to try this themselves…
So far everyone that’s tried it has found it really useful. My 2 favorite quotes so far…
“I never would have thought of some of the areas to action without this process”
“I’m better able to prioritize actions because of the process”
This will almost certainly work best for people with quite a strong sense of self-awareness and self-improvement already (as does any development plan) and remember you can’t forcethis on someone.
There’s bound to be cases where this process doesn’t work but we’ve not found one yet. If you try this yourself, we’d love to hear how it’s worked for you and remember, after trying it out once or twice, it’s worth tailoring to your own needs further.
Clarity – In a change to the original recommendation (working blind), we’ve found it’s more useful to share a clear agenda and state where we are in the process and why at each step so people understand what they’re meant to be thinking about and know what’s left to do.
Skill Levels – In addition to the “out of 10” ratings, we’re going to try developees/managers discussing where they feel they stand in terms of SLII (Situational Leadership) development level (e.g. D1-D4). This requires a bit more thought in understanding the differences but keeps our teams aligned with our broader efforts to use SLII when developing each other. We’ve not tried this bit yet (but will be doing so in the next couple of weeks) so it’ll be an experiment within an experiment – we’ll be seeking feedback and thoughts on how this best works.
Action Grid – Rather than defining all the actions in the action grid up-front; partially filling the set of actions at the beginning and then pulling more in after each step seems to work better for people. It’s important however that developees have filled something in each space to work from toward the end of the session before transferring things across to their plan grid. The idea with pushing for all these actions is to stretch our thinking a bit and gently encourage people out of their normal comfort zone.
Supporting the write-up – It’s still really important that the developee writes this up for themselves but format can be a bit challenging. Whilst we have a couple of people using Mural.ly and one other gave this a shot with excel, we’ve found keeping the visual management aspect afterward is important. Here’s a “blank” PowerPoint presentation with heading placeholders and post-it templates to help people along.
Pitfalls / Caution
Skills Categories – The skills category columns (Job/Role, Soft Skills, Domain/Industry) can be a bit tricky. We’re developing a role-based skills map for each role in development that participants can use as a cue (I’m planning to post more about how this is developing over the next month). Our suggestion here is to select a subset of these skills and traits to talk about in more depth rather than using them all.
Flow – As we’re looking at similar things from multiple angles to find patterns, there’s not always a logical progression through activities. This is deliberate but can sometimes feel a bit “bumpy”.
Time – This takes a lot of time and effort (a minimum of 2 hours depending on how “chatty” the participant is) you could consider breaking it up into 3 sub-meetings:
Skills session (including one small next step)
Actions brainstorm around common themes (using action grid)
(I’d still aim for running one straight-through session first time around as the energy from doing it all in one shot is fantastic)
Managing the session – Seriously avoid time-boxing each step. It’s better to reserve more time and let each part play out naturally.
Energy – This can be quite an exhausting session for both participants and we know one size won’t fit everyone so it’s worth setting expectations that this is hard work. I’d recommend running this when participants both have a lot of energy (e.g. start at ~10AM)
Follow-through – It’s reallyimportant that you follow up within the first 2 weeks to ensure the write-up is done and first actions are planned, (Momentum tails off really fast for development plans) and then remember to follow up regularly – e.g. on a monthly basis after that.
Longer-term – We expect people to want to return to this session in future but not to need to spend quite the same level of effort again unless they’re radically altering their role and direction. At the moment, my thinking is an in-depth refresh after about a year however you may feel like 6 months is more useful.
Following on from part 1 last week where I described the foundation concepts for a lean/agile development plan, here’s the detail…
So far I’ve done this twice with some of my team here and I’m in the process of introducing it across a couple of other teams and managers. As part of writing the details up, I’ve come up with some minor refinements and sequencing options that we didn’t think of at the time and added these in.
Before you start make sure you have a decent space to work in with no distractions and plenty of time. You’ll need about 2 hours but give yourself 2-3 so you have some slack. You don’t have to use it all and it’s surprising how good people feel when they think they’ve recovered some time back.
Grab a room, a pile of stickies and some markers. Did I mention you’ll need to use a lot of space? (We used 3 walls). In the cases I’ve run so far, I didn’t prepare anything in advance but it’s probably best to write up the headings beforehand – don’t show them until the participant is at the stage they’re needed though! (focus on 1 thing at a time – single piece flow!)
Step 0 – What are we trying to achieve?
An important point before we start. Are we working with someone who wants to improve themselves in their current role, address some known (or unknown) issues or who wants some kind of progression – a promotion, a sideways move, expansion of skills, increased responsibility, or something else?
Take 5 minutes to discuss the possible scope and direction of the session. Think about the level of change, disruption and ambition behind the improvement and what is or is not achievable. How long have they been in their current state, what sort or timescales are we looking at for the plan etc.
Step 1 – Looking Forward – “Over the next … … I will”
We’re going to use planning horizons and swim lanes to develop a timeline.
Build a large grid using stickies for each heading with vertical planning horizon columns (I suggest 1, 3, 6, 12 months) and 4 “achievement” (“I Will”) swim lanes; “Done”, “Improved At”, “Tried” and “Learned”.
It should look something like this:
Aim for the grid to have space for roughly 1-3 stickies per horizon/swim lane cell. – We’re using visual management and physical constraints to minimize over-commitment and set WIP limits. I’ll refer to this as the “plan grid” for the remainder of this post as we return to it a few times.
Now spend a few minutes adding stickies to the plan grid for the things you know you’re going to be doing. (don’t worry if there’s not many to start with we’ll be coming back to this later).
Here’s an example recently completed from one of my UX specialists at the start of the process…
This covers the baseline set of directions and commitments before we start adding to it.
If the person you’re working with has a good grasp of upcoming work for themselves, the team and product plans, you might find this fills quite well already. In the case above we’ve spent a lot of time talking about interests and direction in the past. We also have some pretty clear ideas on where we want to go with the product and technology. Visualizing this as a roadmap with a series of planning horizons and the “Do, Improve, Try, Learn” swim lanes for actions gives some initial structured thinking and timing to work from. We’ll circle back to this at the end of the session after exploring other areas of personal development and skills.
In this particular case, you can see we’ve actually filled quite a lot of space already. This doesn’t leave much room for new things unless we over-commit or trade-out other activities and that’s precisely the point – we’ve just baselined our commitment and workload.
Now let’s explore things further. We’ll take multiple approaches to exploring options. Some of these may duplicate information – that’s perfect, it’ll help us spot common threads later.
We’ve developed a grid of over 20 actions with a single placeholder for each action (borrowing a 5S technique from Lean). The set of actions is my chosen selection and works well so far. You may want to add or remove some of these depending on your context but the thinking behind these is to get participants to explore their available options beyond what might naturally come to mind.
Our aim is to place a single post-it/action under each heading. Eventually we’ll have every space completed with something sensible.
We’ve deliberately not looked at any personal skills, strengths, weaknesses or needs yet. This is simply thinking about what types of actions a person could employ to develop themselves.
Sequencing Options: The first couple of times I used this with my team we completed the grid in its entirety at the beginning. In hindsight, I recommend splitting the effort here to partially complete the list of actions at the beginning and then return to it after each of the next steps.
If you’re working with someone who needs more support in developing an understanding of their needs first, you may choose to start this step toward the end of the session after exploring strengths and skills further.
If you choose to use any of this process, please do experiment with the format and post your experiences and improvements here!
When we first used this action grid, we wrote “over the next 12 months I will/would like…” It worked well in our situation but it’s important to emphasize the difference between achievable and aspirational actions, so…
Minor Refinement: Once the grid is completed, run through it together and mark which actions are achievable – “I Will” – and which are aspirational -“I Would Like”. (using red/blue sticky dots or similar).
This encourages you to discuss and think through what will be possible and what may be a stretch (or even unrealistic). When looking at transferring actions to the plan grid, you can decide how much of a stretch you want to make in any given period based on workload and other existing commitments. The “I Would Like” actions may also require some additional support to be achieved.
3 – Self-Reflection – “I’m Happy As I Am With…”
So we’ve got a structure for a plan, we know what commitments already exist and we have a set of potential actions. Let’s take a step back and deliberately reflect on strengths and improvement areas.
As I mentioned in part 1, it’s really important not to go overboard on perceived “weaknesses” so here we break a person’s perception of what they do and how they work into 3 areas:
In order to reduce waste we want to emphasize improving strengths and bringing any gaps to a necessary minimum level only. At this point we focus mostly on the person side of things. How do they work and what do they do? (Not“what does their current or desired role need?” )
As a person or individual, what’s their self-perception around their day to day actions, activities and behaviors. Ask them to think about and describe what areas they see as real strengths – what’s their specialism or superpower if any, what do they really enjoy (and how competent are they at it?), what things do they feel they suck at (and is that a problem or not?). What are the remaining things that they just get on and do? How well do they do those?
Here’s an example of the sort of things we’re after…
Note how this is general activities that tie-back to what a person is doing on a day to day basis in their current context, not just the usual skill inventory type list. It’s about the things we perceive – what we do and how we do it. (we’ll look at the skills next)
If you decided to develop the action grid from step 2 in small steps, now’s the time to re-check to see if there’s some actions that could be added as a result of the strengths or improvement needs here.
Allow enough time for the participant to run out of things to add – don’t be afraid of a bit of silence or thinking time – just sit back and let them run until the ideas dry up.
4 – Skills Investigation
We’ve explored self-perception and started rolling on possible actions. Now we’re going to look more at role-based skills.
Depending on the scope and direction agreed in step 0, this may be covering either a current or desired role.
Have the developee list out what they think the skills they have/need for that role are, what additional industry or domain-related skills they have/need and what additional soft skills they have/need.
We want to steer participants into thinking of these skills themselves so if you think they’re missing anything, try to use coaching questions rather than direct addition of examples. Encourage thinking about their gaps or using examples. (E.g. “Assuming you’re about to kick off a new feature, what types of activities do you need to undertake? What skills do you need to engage the team?)
Once again, give this enough time to play out to a natural end.
Once you’ve got 3 lists of skills, have the participant rate themselves out of 10 at how good they think they are at each one of these skills. Get them to mark this consistently in one corner of each sticky. (They might guess what’s coming next but let this bit finish first).
Talk through what makes them say they’re at that level. Now get them to think about what their desired level for that skill would be – make sure they understand that “getting to 10” may not always be needed or even beneficial (remember their strengths and weaknesses). What would be right for them. Mark that in the opposite corner.
Now get them to mark the top 3 areas that they would particularly like to focus on.
For each of these focus skills, talk through what actions or activities they would need to do to get from their current state to move one point forward. E.g. If someone’s at a “4” today and eventually wants to get to a “7”, what would it take to get to “5”. Ask them to visualize what that result might look like and describe it to you.
Again if you’re partially completing the action grid as you go. Now’s the time to see if there’s some more actions to add or any existing ones that could be switched out or replaced.
6 – Common Threads & Themes
We’ve now tackled skills, behaviors and improvement ideas from multiple directions and explored (hopefully) some areas in quite a bit of depth. Looking across all the details on the walls, are there any patterns, common threads or themes?
What do these highlight?
Name these, write them out and put them next to the plan grid from step 1
This comes back to my love of patterns and naming. By giving something a name and focus you can assemble more of a mental model around what that thing is. This makes it easier to understand what is or isn’t in scope for a theme.
As with the skills, I recommend limiting the number of threads / areas of focus to 3 or less.
When I’m trying to get stuff done I have a small mental buffer and 3 is about my upper limit (2 works better). Any more than that and my brain starts thrashing. The same works here – trying to fit too much in your head will make things harder later.
7 – Actions – Pulling It All Together
Now let’s get back to the action and plan grids.
First of all, make sure we’ve got a single item in every space on the action grid. You might need to get some creative thinking in to fill them all but it’s worth the effort. When that grid is filled it gives us an unsorted long-term backlog of ideas for our developee. When someone completes a particular action type you can decide whether to add a replacement or to continue clearing down the grid before adding anything new.
Don’t forget to mark which are directly achievable – “I Will” – and which are aspirational – “I Would Like”.
For the aspirational items, take some time to discuss and understand what support or commitment will be needed to make them achievable.
Now have the participant copy a small number of the actions from the action grid and decide where to place them in to the 1/3/6/12 month plan in the plan grid. (don’t move the originals – you’ll need them)
Next, have them look at those top areas identified for improvement (from the Self-Reflection and Skills Investigation activities) and identify any further specific actions for each (or a subset if there’s too many)
For each selected action, they need to decide when they intend to undertake it and place it in the corresponding location on the plan.
If there’s not enough space on the plan grid, remember that physical space constraint gives us a WIP limit! What do they need to trade out to stop themselves over-committing?
Here’s an updated example from step 1. Note we didn’t add much beyond the first 1-3 months this time around as we expect to come back to this again when that planning horizon has cleared. We also had some really powerful conversations around their overall development and direction.
Don’t underestimate the value of the conversations themselves, not just the artifacts! For the participant in particular, those conversations may have crystallized a whole bunch of questions about their personal direction and goals for the future.
8 – Review – Is It Achievable?
Talk through the final plan grid, discuss timings and workload. Does the participant feel that the plan is achievable? Is there anything they might change? Are there any areas of risk or concern?
Take some time to talk through the actions grid (now a backlog). Are there any things there that might be time-sensitive or wholly unachievable or mis-aligned that they want to swap out?
9 – Ownership – Whose Plan Is It?
Here’s where many development plans fall down.
Chances are you can’t leave all that stuff on the walls so capture all the content on decent (readable) photos and then harvest the stickies in their respective groups.
Ensure the participant gets both the original stickies and a copy of the photos. Ask the participant to type up the results of the session and to share back – agree when they’ll have this completed – ideally within the next 24 hours. (It’s amazing how much you learn from typing up post-it notes after any session!)
Ensure their actions grid is converted to a backlog and the plan grid is written up as a plan that can both be updated and modified easily. (one of my UX team wrote theirs up using Mural.ly – this is particularly neat as they were able to mimic the format we used in the room very closely)
In discussing ownership of development plans we’ve identified that simply handing over ownership is not enough. The developee will still need support, coaching and mentoring to progress successfully. Don’t let them fail alone.
10 – Follow-Through
Once the write-up is complete (depending on the participant you may need to chase them to ensure this happens). Follow up quite frequently to start with. I recommend at least once or twice in the first 2 weeks, and then again at the end of the first month planning horizon.
Minor refinement: It may also be useful to extend the planning horizon backwards by a month or more during your review cycle. You can encourage the developee to add other things they’ve achieved and “bank” these.
We know that people naturally do things that aren’t strictly on a plan. Make a point of marking those achievements. This lets us review what we’ve achieved on a rolling basis and gives us a trailing indicator of progress and capacity.
Depending on how things are progressing and the level of interaction or support needed, agree on a follow-up cycle and set dates for a regular one on one catch-up session to talk through progress, changes, adjustments or resets. At the 3 or 6 month point, schedule a more detailed review and repeat session.
That’s it! (so far).
Edit: We’ve been so successful in our initial trial with a few staff in my division that we’re expanding the trial across the company. We’ve also learned a few important tweaks and traps along the way. See where we are now in part 3.
As part of a leadership program I attended with one of my previous employers back in 2007 I completed an in-depth 360 degree appraisal to learn more about my behaviors and emotional intelligence.
The assessment required input from 20+ people who knew me.
All direct reports
All immediate team peers
A selection of friends and colleagues across teams
My direct manager
My Manager’s manager
At least 2 Suppliers
At least 2 Customers
Every attendee of the residential programme was given their 360 degree feedback in a 1-1 session and walked through the results in detail. In a couple of alarming cases, the disconnect between how a person saw themselves and how their teams saw them actually required individual counselling.
In my case, I found that everyone but my direct manager was very closely aligned with my self-perception. Given the high standards my manager held for all his team I was happy with this result – he measured the team’s comparative performance against his own exactingly-high standards, it’s why we were both successful but continued to strive for improvement.
The walk-through of the 360 showed each of us what was considered an “acceptable” competence level for each category measured and what was considered “exceptional”.
This is a pretty powerful way of thinking about things. We may all want to be exceptional but I’ll highlight the lesson many people miss with an example…
Usain Bolt may be currently the worlds greatest sprinter but that doesn’t mean he’s the world’s greatest javelin thrower or even marathon runner.
Chances are he’s pretty good at other things – as a world-class professional athlete he’ll have a broad baseline of skills but he knows his strengths, invests in those and doesn’t waste his effort and training on the areas he’s just “ok” at.
He’s part of an Olympic team. In order for his country to be successful he’s surrounded by a high-calibre team with a broad set of skills, specializations and expertise. His team mates will be signficantly better than him in some areas.
By all means address your shortcomings that are actually problematic but recognize which skills you only need to be “OK” at, which you don’t need at all and where you need to excel. If you need excellence across the board, chances are you need a team, not an individual.
Look around you and build a team that complements each others skills, gaps, shortcomings and mediocrity.
Save your precious time & effort (eliminate waste!) to focus on your strengths, find someone else to complement your weaknesses instead.
At my current employer we already seek 360 feedback (but not at quite the depth of the EI appraisal) as part of our monthly staff 1-1s where we discuss performance, status, career direction and well-being. We don’t have annual performance appraisals but we do encourage staff to have personal development plans if possible (note – we only encourage them, they’re not mandatory).
Recently I’ve wanted to improve our uptake of development plans for the team but I know they’re often really hard to work. Let’s consider some of the issues I’ve seen in weak personal development plans over the years:
Despite there being a need for the person being developed to “own” the plan, the manager is the one to identify and initiate the process and then draw out the plans and actions. This leaves little ownership, motivation or inspiration for the developee.
Lack of ideas
When a developee first starts on a development plan, they think they “should” improve and move forward (it’s what businesses expect of us) but are unsure on personal direction and their needs. They may be starting with a blank slate.
Driven by identified shortcomings
The concept of a “PDP” is often a remedial measure. Whilst this may be necessary, they focus on getting developees “above the line”. They lack a means to shine and excel.
The “day job” gets in the way
Activities required to significantly level-up in an area may be disconnected from what’s required to simply deliver on a day-to-day basis. Balancing a day job with significant performance improvement is hard.
Behavioral improvements are hard to implement
If a developee isn’t a team player, how do you draw that out and get them to the point where they don’t revert back when the plan is over? Are there alternatives?
Over-ambitious development plans may feel great when first drawn up but energy and motivation dwindles fast when things are difficult to achieve.
The approach I’m exploring aims to tackle these using lessons from lean, agile and my personal experience from the EI 360. Here’s what I’m trying out…
I work with the developee in a one on one workshop-style format. We take over a room for a couple of hours and using a structured, visual, kinesthetic, reflective sequence of steps we gradually fill the walls with thoughts, ideas and actions.
We’ve developed a list of over 20 concrete “activity types” for generating ideas that work well for developing people in novel ways. By borrowing a 5S tactic – Seiton from Lean we create a placeholder space for each action. We know we’re “done” when each activity type has its single placeholder space filled.
Focus on Strengths
Here we aim to eliminate waste. Instead of looking at weaknesses, we explore 3 areas; “I think I’m good at…”, “I need to improve…”, “I’m happy as I am with…”.
Note the use of first person statements – this is about self reflection, not external direction.
In explaining this area we point out that items “I think I’m good at” are potential strengths that can be developed further. Items “I need to improve” should only be for weaknesses that actually need to be addressed within the context of their team, role and self development. Items “I’m happy as I am with” are areas that even if weak, do not need to be developed further.
Balancing the Day Job
In order to be successful, we need to ensure we have sustainable pace. We achieve this through setting a series of planning horizons and using visual management and physical constraints to define natural WIP limits.
Highlight the Spectrum of Skills
I work collaboratively with the developee to draw out the list the skills they have or may be required for their development needs. We explicitly call out Role-specific skills, additional broader skills and soft skills as 3 categories. We then use simple measurement & rapid feedback to establish both baseline and desired levels for each area.
After pulling the threads together and building a plan with sustainable pace, we sanity-check the plan. Do we feel what’s there is achievable in the next sprint (for example). We agree the review and follow-up cycle and adjust as we go.
In part 2 you can read though the detailed mechanics of how I’ve been doing this with some of my team, including some minor improvements as a direct result of writing up the details. I’ll post an update on how this is all going in future.
There’s a time & place for external consultants in organizational transformations. They can help you in unique ways but you have to pay attention and act.
Some years ago I had a game-changing conversation with an exceptional guy we’d hired in. Everyone loved his straight-talking, he had a way of talking common sense that resonated all the way from individuals on teams to our execs. We knew that one or two conversations would never be enough so I asked him to join us permanently – he was exactly what we were looking for.
His response (paraphrased)
“I’m flattered and I’d love to – but one of the great things about what I do is that I can tell you the painful things you don’t want to hear – knowing I’m right – without having to worry about losing my job and knowing my boss has my back.”
We managed to get him to speak to our local leader at the time about what we were trying to achieve.
“…you do know that the level of change you need here is going to have an impact on productivity for a while right? Are you ready for that?”
“…absolutely not, we have to complete our transformation with no impact on our current delivery commitments to the business”
If you invest in a consultant, It’s your responsibility (and risk) to act on their feedback. Be prepared for painful input and… please …do something with their advice – that’s what you paid them for in the first place right?
If there’s a risk or impact, weight it up, bring your stakeholders in, explain the situation and reset expectations. You are empowered to act but here’s the trick you’re missing! It doesn’t all have to be just on your shoulders.
In fact, you don’t have to have the difficult conversation yourself at all, you’ve just paid for someone to do it on your behalf. If they’re really that good, they’ll be more than capable and willing to do so and probably more credibly than you.
“Captain, we have a customer crisis, get your passport, we need you on the ground tomorrow.”
This would have been fine if it weren’t Thursday night, that I’d been pulling 14 hour days for about a month already and had promised to take my family out at the weekend to make up for it. – Sounds like something out of the goal, doesn’t it!
A couple of years after my first major lesson in application performance, now working for a different company; one of our key customers in South America was trying to get their staff through critical money laundering certifications before a fixed deadline. If this was missed, huge fines were on the cards.
Trouble-was, hardly anyone could get onto the system, it kept going down.
Our implementation team had taken our profile and scaling recommendations, added some contingency and specified the systems for the customer. The customer wasn’t happy with the cost so they’d gone for a cheaper processor and database option which made us uncomfortable but this was a huge deal for us. (It was an early version of SQL Server on Windows 2000 and they were trying to scale to nearly 10,000 users).
I arrived at the customer site. A really great, friendly bunch of people and one of the most bizarre office locations I’d ever been to. The office had a transient community of something like 1,000 staff with no other businesses anywhere nearby. Surrounding it was an entire cluster of lanchonetes that opened at lunchtime – solely to serve this building.
They’d taken the system offline for the weekend and were running their own testing using load runner. Having done scalability tests for customers before this seemed pretty simple. Ramp users up in waves to desired levels, test, ramp up more, test again, tune, retest etc. We recommended a transaction rate of about half a transaction per user per second as a “heavy” load for this type of application.
We reviewed the log files, it was dropping at about 1,000 users – way below where it should have been. I took a dig into their application servers, modified the JVM parameters for optimum heap size, garbage collection etc. – I’m pretty sure this will work.
Tests showed nothing abnormal. We’re looking good to move forward.
Let’s get the users back on…
10:30 AM Monday morning – The system goes down again.
We take a look at the logs.
Within the space of about a minute, nearly 5,000 users tried to simultaneously log into the system.
Back then, large-scale e-commerce event ticketing systems were not common. For an internal system like this, none of our team ever expected we’d have that kind of load just on login. Even 1,000 closely-packed logins seemed wildly unlikely.
What were they doing?
We talked to the managers about the users. We established that staff in all branches had 2 30 minute breaks per day in 2 patterns (half the staff at a time).
OK, so what…
This was the only time the staff were able to use their systems for this certification. The rest of the time they were serving customers!
These people’s breaks were precious, they had a deadline to meet for mandatory certification this was the only time they could do it.
Solution: Longer term we fixed the performance of the login routines so it’d never happen again but in order to achieve their short-term goals, the management team staggered break times for staff during the certification period. (Oh and I’d got the sequencing of one of the JVM hotspot parameters mixed up but that wasn’t so relevant).
1: When designing systems for performance & scalability (even internal ones), you need a good understanding of peak load based on real usage profiles – including extreme ones. This might seem obvious now but to us back then it wasn’t. We thought we knew how the system was going to be used.
2: Spend a day with the real users, not just the “customer”, Make a point of understanding their goals, and constraints. If you’ve not done so before, I guarantee you’ll learn something unexpected.
3: Sometimes the best solution isn’t technical. In this case, a simple work scheduling change saved a potential fortune to our customer in hardware & licenses.
4: Just as in my first performance lesson; if load is problematic in some areas, build safety margins into the system.
At 5PM one evening about 10 years ago I had a call from the head of application performance at the company I was working for…
“Captain, I just thought you’d like to know you’re currently responsible for the most expensive piece of SQL in the company – worldwide… “
It was using about 95% of the CPU resource on 2 Superdomes. This was a production showstopper; no going home tonight…
Now this was embarrassing. I and one of my team were responsible for a recent complete refactoring of a key component on one of our global systems. We’d tuned the hell out of this thing. Despite it trawling a decade’s worth of live data – millions of records, it was so efficient that it returned in under half-a second every time with minimum memory footprint, minimum disk I/O and minimum CPU load. We’ve profiled it like crazy under all kinds of data shapes and volumes. We’d taken production transaction volumes, built in contingency and ploughed it through those and it was just screaming away, no problems. In fact it was one of the best bits of refactoring & performance tuning I and my co-developer had ever done.
Something was seriously wrong. There was no disputing the logs.
As always with these major production issues, I and my development parter pulled an all-nighter on calls to various application groups in the US.
We found an anomaly.
One query was being called about 2,000,000 times a minute!
All our profiling, production comparisons and performance data had been based on a peak load of about 1,000 hits a minute. As a key component of a worldwide order system we knew it would be heavily used and put a reasonable load on the system but this was way beyond our most unrealistic expectations.
All our tuning had been based on a known forecast usage through our standard order management system. The calculation engine was typically called once or twice for every order placed worldwide. (We’d forecast about 500 orders per minute and easily had capacity for twice that volume)
Further investigation revealed the source.
Earlier that week, a new application from another team had been integrated into production.
There were rumors that the VPs of each team didn’t speak to each other. Either way, although we knew members of the other team, they were on the other side of the world and we didn’t collaborate very often.
This new application was dependent on the same calculation engine. We’d spent some time training their developers on how to interface to it and they were really pleased with the results they were seeing. Once our knowledge transfer was done, that was the end of the story as far as we were concerned.
What we didn’t know was that their initial effort were so successful that they had integrated it into their product as an auto-calculation system.
Every time a user tabbed from one field to the next on the order line details, it was performing a recalculation.
Now a typical order at this company contained about 100 order lines. And each order line contained ~20 fields. Our calculation engine was being put under nearly 2,000 times more load than had ever been expected!
Needless to say we had the team fix it fast.
They removed the “auto-price” feature and replaced it with a “recalculate” button on the shopping cart.
1: When you’re writing a system that will be integrated with multiple systems or uncontrolled third-parties, make sure there’s a mandatory part of the interface in place that requires the caller to be clearly identifiable and that your logging is user-friendly. Putting all blame aside, this immediately allows you to identify and isolate unique integration issues.
2: Don’t just train your users/customers how to use your system. Help review their approach and processes and once they’re up & running, get them to show you the results and walk you through what they did. Chances are they will try to do something exotic that you weren’t expecting and would make you squirm. Better to find it early and deal with it than allow it to become a production issue. Remember it’s up to you to be a role model in this collaboration.
3: Identify, set your performance constraints, expectations and limitations up-front. Consider even building them into the system in some configurable way and tell your user/customers what they are. In the days of denial-of-service attacks, secure coding requires us to put throttles into our systems to prevent them running away with resources. (anyone ever tried to buy Glastonbury tickets online?) Even in smaller internal systems, it’s worth having some idea on volume and usage. Performance defects are notoriously difficult to resolve, are usually showstoppers and often require major architectural rework – We were lucky this time!
4: Many defects are found when your software is used in ways that you and your requirements didn’t foresee. Often that flexibility is a major asset or differentiator but if it’s not, consider putting up safety rails to limit your system to intended usage only. If it “might be useful” to work another way in future, remember YAGNI – You aren’t gonna need it. If you feel you must have that potential flexibility, at least consider putting a lock on it that requires your customer or user to call up or understand and recognize the impact first.
Yes I do still dwell circa 1992 – You should see my car!
On the way out I recognised The Wonder Stuff’s new drummer – Fuzz Townshend(formerly of Pop Will Eat Itself). This guy has been a key contributor to some of my favourite music for over 20 years so I stepped up to congratulate him on a fantastic gig.
He seemed geninely shocked (and pleased) to be both recognized and thanked for his performance and took the time to discuss his thoughts on the likelihood of a PWEI reformation and on the whereabouts of their unreleased final album of material and was – like most mature rock stars – well worth taking the step to talk to.
It made me realize. Whilst most of the crowd had been stopping to have their photos taken with the guitarist, the recognition for the rhythm section that binds a band together is rarely the same.
On software development teams (and companies in general) this same pattern applies. How often have you seen someone tirelessly working behind the scenes, a complete hidden hero – perhaps only recognized by their own team – get passed by when the awards and recognition are handed out.
Take some time out this week to recognize your rhythm section.