Tag Archives: collaborate

You Are Not A Lathe Operator

Posted on by
Reply
Reading time ~ 3 minutes

This weekend my replacement copy of “The Goal” arrived. (My last copy escaped).

I wrote the following article in mid-2009 – some time before I read Goldratt’s “The Goal”. This was eventually published in 2010 internally at the company I was working with at the time. Today felt like a good day to refresh it and share more widely.

________________________________________

If you haven’t yet done so – read up on the theory of constraints. In fact I recommend that every manager in any size of organization obtain and read a copy of “The Goal” discuss it with your peers & teams and make copies available for your staff.

In Spring 2009 I had the good fortune to attend a training course in Florence, Italy – a hub for European manufacturing training for a large Oil & Gas company.

One of my working team during the week was in the process of implementing a lean production line for the local factory floor. Their layout and travel spaces were all mapped out but the reason for him attending the training was to cover the human aspects of such a radical change.
He had factory floor staff who had been performing the same expert role for 20-30 years or more. Now they were being asked to radically change their working practices and he was feeling the pain.

An operator producing bearings may traditionally be measured on their production volume (and quality).  Their single-minded goal is to maximise their own output. The more bearings produced, the better an operator they are. In some cases, particularly where waste is not well managed, volume may even trump quality.

In a production line, operators are contributing not to the production of bearings but to the production of complete working units as part of a team.
They have to look both up and down-stream and assess the state of flow through the entire team. If there’s a bottleneck in the line, they’re expected to adjust or stop their own activity to help out and “level the load” in order to maximize the performance and throughput of the entire assemblies they’re working on in the factory.

The challenge here was that some operators had learned over years of training and measurement to sub-optimize around their own roles and personal output even though the real unit of value was in the whole team’s output.

His challenge was not unique and for this particular company, lean manufacturing was already a major focus. The striking synergies between lean & agile meant I was able to share insights from my own experiences with him.

Now let’s take his example and consider some quotes often heard during software development.

  • “It’ll be far more efficient if I do all the coding now in one go.”
  • “Seeing as I’m already doing some surgery here it makes sense to add these few other things that I’m pretty sure we we’ll need”.

Often these are suffixed with “it’ll be ready in when it’s ready”.
I’ll raise my hand and admit to this behavior in the past – particularly when in the midst of a major refactoring exercise.
But…

  • What about everyone upstream and downstream of my code?
  • What’s the impact on the testers of me delivering 3 months worth of code over 2 days worth of code?
  • Do I have people waiting for work downstream?
  • If I changed my delivery practices would I get feedback sooner?
  • If I got feedback sooner would those bugs be less of a pain to context switch and fix?
  • Would the overall throughout of the team be greater?
  • If requirements change how much of my work would be wasted?

This is where agile and manufacturing meet.

Admittedly in manufacturing, the concept of a multi-skilled operator is far more prevalent than in software however the software industry equivalent is often described as the “Generalizing Specialist” – that is: whilst a member of the team may primarily be role X, they are able to add support and value in role Y – even if that’s not their job title.

This may be a simplification of reality but some skills are exportable or importable across roles and allow us to level the load by having others do the heavy lifting.

It is the responsibility of every member of a development team to consider the entire value-stream, the activities of all skills and roles that contribute to a working product increment and ask questions like:

  • “Am I working on our top priority item? – if not, should I contribute to it in some way?”
  • “Could we be delivering more finished work sooner if I helped out elsewhere?”
  • “I’d really appreciate someone covering these areas around the edges of what I’m working on”
  • “Here’s a problem that keeps slowing us down, let’s figure out how to remove the bottleneck.”

Think about your current role in a development team.

  • Are there opportunities for members of the team covering different roles to support each other better?
  • Is anyone over-producing in comparison to capacity of other areas and causing headaches downstream?
  • Could some conversations be had sooner, be much shorter and save pain?

Take some time out this week and consider how your teams and individuals could improve their cross-role collaboration, share some of the heavy lifting and deliver as a more cohesive unit?

Black Holes & Revelations

Posted on by
Reply
Reading time ~ 3 minutes

Have you ever had to deal with a black hole on your team?

“As predicted by general relativity, the presence of a large mass deforms spacetime in such a way that the paths taken by particles bend towards the mass. At the event horizon of a black hole, this deformation becomes so strong that there are no paths that lead away from the black hole” – Wikipedia

I’m not a physicist so here’s a simplified view that I can fit in my smaller brain:

Black holes are like huge “gravity traps” sucking in all energy from the surrounding area. Energy and mass are drawn toward the event horizon, sucked in and lost forever. The more they take in, the larger or denser they get.

Here’s some cool stuff I learned from Karl Schoemer a few years ago.

A team undergoing change can be coarsely divided into 3 behaviors: Design, Default and Defiant/Detractor.

• The “Design” population are your role models; your supporters & change agents – but be aware, some may have short attention spans or become zealots. This is up to 20% of your population.
• Those following the “Default” behavior will sit on the fence; “What.. …ever”, “it doesn’t apply to me”, “I’ll carry on as I am thank you” are all common “default” responses. Typically this is 70% of your population!
• “Defiant/Detractor” behavior exhibits extreme symptoms including shouting, arguments, tantrums, sabotage, threatening to leave and pulling everyone else down with them. Less extreme responses include focusing on the minutiae, public cynicism and endless debate without action. In many cases, whilst this may seem prevalent, often this is actually as little as 10% of your population!

Now let’s return to the Black Hole. In space, black holes are invisible – only their effects can be seen. In change management, we simply fail to recognize and identify them.

Human black holes must be understood and handled with extreme caution.

For those inexperienced with black holes, your instinct will be to try and defuse them. You must spot when you are feeding a metaphorical black hole, rewarding negative behavior by pouring your finite energy and resources in. Feeding black holes provides them additional credibility in front of their peers – their gravity trap grows ever-larger.

Lean values time… Eliminate waste! – Where are you wasting your energy?
If you removed the energy feeding a black hole would it eventually burn out?
In human change, detractors usually either get with the program or leave.

If you’ve read some of my prior articles you’ll know that whilst I appreciate good people; if your behavior and attitude isn’t up to scratch, all the technical prowess in the world is unlikely to make me want you on my team.

Some black holes may be an almost permanent rift in space. Work to minimize their impact and sphere of influence rather than offering more fuel. Consider using them as your “professional cynic” – your sounding board for the detractor response – but be aware this is a lot like playing dodgeball with a burning coal. It’s usually safer to move them away from the powder magazine instead.

Where could your wasted energy be better spent?
Simple! Use it to shift the center of gravity on your team away from the black hole.
Partner with your “design” members as a team and swing your population of defaulters toward your chosen direction. Some may be pulled toward or into the black hole but work on the overall gravity shift to bring the team around.

If you don’t have sufficient design weight to adjust the center of gravity right now, go digging for more – one person at a time if needed. At some point you will be able to tip the balance.

(Oh – a nod to Muse for inspiring the title of this post)

Express Your Real Motivations

Posted on by
3
Reading time ~ 2 minutes

Hidden agendas make you unpopular, especially those that are poorly concealed.

Everyone has an agenda. I prefer mine open and public. Some may try to take advantage of that but most will respect it…

Some time ago I attended a session with a team where there had been some communication challenges. The team’s normal very tight, cohesive ethos was fraying.

As a globally distributed coaching team they’d arranged to co-locate for a week to bash out a few things and generally get together before returning to their usual sites.

Dinner and a couple of beers into the first evening together and the team spirit had started to sparkle again – recognizing each other as friends, not just colleagues. The hint of political undercurrent was still gnawing at the edges of the smiles.

Into the second day and one of the very perceptive team members called a halt to proceedings.

My paraphrasing of the conversation…

“OK, time for a break. Before we go on, let’s catch up with each other for a bit… …why are you really doing this job, what’s your motivation – what’s your angle?”

This team knew each other well enough to already know the answers but actually calling them out publicly in front of each other was a new step in uncovering potential hidden conflict.

Because the team ran on trust and acted as a balanced cast (I’ll write about team casting in future); everyone acknowledged and accepted each others’ motivations knowing that despite being potentially sensitive they were honestly and openly given.

Even better, they discussed how each other could support those motivations.

The politics and tension were gone.

Earlier this week I watched someone with a blindingly obvious personal motivation attempt to leverage it in front of a smart bunch of people who were mostly there for related but different reasons. Rather than a public calling out, it was handled through amiable debate over a beer later but everyone in that following conversation recognized the unspoken calling out had been made and started trying to re-engage and collaborate.

Or at least I hope they did

In keeping with the spirit of this post I therefore share my own agenda…

  • I’m naturally creative and like to share
  • I seek personal but usually not financial reward
  • I want to be recognized for “good” things

I strive to make a positive difference by sharing my thoughts or observations and by participating in conversation. I seek personal reward through constructive intelligent feedback, good friends and good company.

Building a Case for Change

Posted on by
Reply
Reading time ~ 3 minutes

I recently led a short evening session discussing strategies for collective code ownership. We opened with a set of patterns and practices but the team highlighted something painfully obvious that I’d overlooked…

What’s the business value and driver behind collective code ownership. What’s in it for your product owners and customers?

So often we focus on the people barriers and technical challenges to collective code ownership without recognizing that the barriers may be organizational or financial.

This challenge happens when you attach yourself to “the right thing to do” and seek to progress without recognizing that the population you’re likely to be working with will fall across a spectrum from complete support, through “don’t care” to disagreement and dissent.

If you want to pursue a strategy for collective code ownership, step back and work through the following questions (some of these are hard – consider starting by brainstorming with a small supportive team):

  • If we just got on did it, would we be successful?
    • Yes – Great, you’re working with a team and business that “gets” this or trusts you to do the right thing. You could stop reading here, but you might want to skim this to see what ideas & thoughts are triggered.

    • No – OK, you’re part of the more normal population (for now) read on for some ideas. You might not need to cover all of the following (and if you do, it might slow you down)

Still reading? – Good!

The following apply to pretty much any change or investment program but answering them is surprisingly hard. Your context will probably differ to mine so I’ve not given all the answers today.

  • What are the top 1-3 benefits of pursuing this?
  • What are the top 1-3 costs of pursuing this?
  • Are there people outside the team that I’m going to need to “sell” to?
  • Are there people inside the team that I’ll need to sell to?
  • What are the risks, pitfalls and political or personal agendas?
  • What do we sacrifice – what’s the opportunity cost?
  • Who are my supporters and how can they help me?
  • Who’s on the fence, do they need to be involved and how can they help/hinder?
  • Who will fight against this and how big a problem is that really?
  • Do I ask for permission and support or press ahead and ask for forgiveness later?
  • Who will this impact and in what ways (positive and negative)?  **detail below

**How about those stakeholders?

For the set of stakeholder questions I suggest you’ll achieve better flow and find natural breaks if you cover positive and negative aspects for each role by “switching hats” as you work through rather than batching up all the positives first.

  • What’s in it for you and what’s the down side?
  • What’s in it for your peers and what’s the down side?
  • What’s in it for your boss and what’s the down side?
  • What’s in it for your team and what’s the down side?
  • What’s in it for your business and what’s the down side?
  • What’s in it for your users and what’s the down side?
  • What’s in it for your customers and what’s the down side?
  • What’s in it for a product manager/owner and what’s the down side?
  • What’s in it for a project/program manager and what’s the down side?
  • What’s in it for an individual team member and what’s the down side?

Wow! that’s a lot of questions.

As I said – you might not need to answer all of these and you really don’t want to have to write them all down but it’s worth spending a few minutes or hours considering these if you’re planning to invest your time, effort and credibility in something you believe in.

Beyond everything here, I also recommend a read of Fearless Change to help you on your way.

Patterns For Collective Code Ownership

Posted on by
Reply
Reading time ~ 5 minutes

Following the somewhat schizophrenic challenges of dealing with person issues on collective code ownership, here we focus on the practices and practical aspects. How do we technically achieve collective ownership within teams?

We’re using the “simple pattern” approach again. For each pattern we have a suitable “anchor name”, a brief description and nothing more. This should be plenty to get moving  but feel free to expand on these and provide feedback.

Here’s the basic set of patterns to consider:

Code Caretaker

Let’s make it a bit less personal, encourage the team to understand what the caretaker role for code entails. It’s not a single person’s code – in fact the company paid us to write it for them. Code does still need occasional care and feeding – Enter the “Caretaker”.  Be wary though that caretakers may become owners.

Apprenticeship

Have your expert spend time teaching & explaining. An apprentice learns by doing – the old-fashioned way under the tutelage of a master craftsman guiding them full-time. This is time and effort-intensive for the pair involved but (unless you have a personality or performance problem) is a sure-fire way to get the knowledge shared.

If you need to expand to multiple team members in parallel,  skip forward to feature lead & tour of duty.

Ping-Pong Pairing

Try pair programming with an expert and novice together. Develop tests, then code, swap places. One person writes tests, the other codes. Depending on the confidence of the learner, they may focus more on writing the tests and understanding the code rather than coding solutions. Unequal pairing is quite tricky so monitor this carefully, your expert will need support in how to share, teach & coach.

Bug Squishing

Best with large backlogs of low-severity defects to start with. Cluster them into functional areas. Set a time-box to “learn by fixing” – delivery is not measured on volume fixed but by level of understanding (demonstrated by a high first-time pass rate on peer reviews). If someone needs help they learn to ask (or try, then ask) rather than hand over half-cut. This approach works with individuals having peer review support and successfully scales to multiple individuals operating on different areas in parallel.

A Third Pair of Eyes (Secondary Peer Review)

With either an “initiate” (new learner) or an experienced developer working on the coding, have both a learner and an expert participate in peer reviews on changes coming through – to learn and provide a safety net respectively. The newer developer should be encouraged to provide input and ask questions but has another expert as “backstop” for anything they might miss.

Sightseeing / Guided Tour

Run a guided walk-through for the team – typically a short round-table session. Tours are either led by the current SME (subject matter expert) or by a new learner (initiate). In the case of a new learner, the SME may wish to review their understanding first before encouraging them to lead a walk-through themselves.

Often an expert may fail to identify and share pieces of important but implicit information (tacit knowledge) whilst someone new to the code may spot these and emphasize different items or ask questions that an expert would not. Consider having your initiate lead a session with the expert as a backstop.

Feature Leads

The feature lead approach is one of my favourites. I’ve successfully acted as a feature lead on many areas with teams of all levels of experience where I needed to ramp up a large group on a functional area together.

By introducing a larger team, your expert will not have the capacity to remain hands-on and support the team at the same time. They must lead the team in a hands-off approach by providing review and subject/domain expertise only. This also addresses the single point of failure to “new” single point of failure risk seen with “apprenticeship”.

This is also a potential approach when an expert or prior owner steps in too frequently to undo or overwrite rather than coach other members activities. By ensuring they don’t have the bandwidth to get too involved you may be able to encourage some backing-off but use this approach with care in these situations to avoid personal flare-ups.

Cold Turkey

For extreme cases where “you can’t possibly survive” without a single point of failure, try cold turkey. Force yourself to work without them.

I read an article some years ago, (unfortunately I can’t track it down now) where the author explained how when he joined a project as manager, the leaders told him he must have “Dave” on the project as nobody else knew what Dave did. He spent a week of sleepless nights trying to figure out how to get Dave off the project. After removing Dave from the project, the team were forced to learn the bottleneck area themselves and delivered successfully.

(Apologies to any “Dave”s I know – this isn’t about any of you)

Business Rule Extraction

Get your initiate to study the code and write a short document, wiki article or blog defining the business rules in the order or hierarchy they’re hit. This is usually reserved for absolute new starters to learn the basics of an area and show they’ve understood it without damaging anything. It is however also a good precursor to a test retrofit.

Test Retrofit

A step up from business rule extraction whilst delivering some real value to the team. Encourage your learner to write a series of small functional tests for a specific functional area by prodding it, working out how to talk to it, what it does and what we believe it should do. Save the passing tests and get the results checked, peer reviewed and approved.

Chances are you might find some bugs too – decide whether to fix them now or later depending on your safety margin with your initiate.

Debt Payment

After a successful test retrofit, you can start refactoring and unit testing. As refactoring is not without risk this is generally an approach for a more experienced developer but offers a sound way of prizing out functionality and learning key areas in small parts.

This is also a natural point to start fixing any bugs found during a test retrofit.

National Service (also known as Jury Duty)

Have a couple of staff on short rotation (2-4 weeks) covering support & maintenance even on areas they don’t know. This is generally reserved for experienced team members who can assimilate new areas quickly but a great “leveller” in cross-training your team in hot areas.

Risks are generally around decision-making, customer responsiveness, turnaround time and overall lowering of performance but these are all short-term. I’ve often used this approach on mature teams to cross-train into small knowledge gaps or newly acquired legacy functional areas.

My preferred approach is to stagger allocation so that you have one member on “primary” support whilst another provides “backup” each sprint. For the member that covered primary in sprint one, they’re on backup in sprint 2. (expand this to meet your required capacity).

3-6 months of a national service approach should be enough to cover the most critical functional gaps on your team based on customer/user demand

Tour of Duty

Have your staff work on longer term rotations (1-6 months) working on a functional area or feature as part of a feature team. This couples up with the “feature lead” approach.

Again, this is an approach that is very useful for mature agile teams that understand and support working as feature teams but can be introduced on less-experienced teams without too much pain.

The tour of duty can also work beyond an individual role. For example a developer taking a 3-6 month rotation through support or testing will provide greater empathy and understanding of tester and user needs than simply rotating through feature delivery. (My time spent providing customer support permanently changed my attitude toward cosmetic defects as a developer)

Adjacency

This takes a lot more planning and management than the other approaches described but is the most comprehensive.

From the areas each team member knows; identify where functional or technical adjacent areas lay and then use a subset of the approaches described here to build up skills in that adjacency.

Develop a knowledge growth plan for every team member sharing and leveraging their growth across areas with each other. Although this is an increase in coordination and planning, the value here is that your teams get to see the big picture on their growth and have clear direction.

All these patterns have a selection of merits and pitfalls. some won’t work in your situation and some may be more successful than others. There are undoubtedly more that could be applied.

Starting in your next sprint, try some of what’s provided here to developing shared ownership and knowledge transfer for your teams. Pick one or more patterns, figure out the impacts and give them a try.

(Coming soon “Building a Case for Collective Code Ownership” – when solving the technical side isn’t enough)

Your Baby Might Be Ugly

Posted on by
1
Reading time ~ 3 minutes

Everyone thinks their own babies are beautiful. Some probably are but there are plenty that really aren’t. In fact I remember one particularly interesting comment -“ugly people produce ugly babies”

This happens in code too.

Collective code ownership has many challenges. Here I focus on the person pitfalls.

Imagine you’ve just spent the last 3, 5, or even 10 years nurturing and grooming an insanely complex (and dare I say “clever”) functional area of a product. This has been your working heart and soul for years – apart from the occasional customer to distract you. It follows your own unique,  “perfect” coding style and conventions. You’re so damn good you don’t need unit tests, you instinctively know and understand every intricate tracery and element of logic in there.

Why then if it’s so perfect are you not willing to share?

This is where “Bad Captain” starts to kick in with me…

  • Do you believe that every other developer in the building is less capable than you?

(OK, sometimes it might be true but that’s very rare.)

– That’s fine, you can peer review and provide feedback if they don’t meet your standards right? It might be an emotional lurch to start with but seeing someone else succeed and recognize your greatness must surely be rewarding right? I’ll help you if you need.

  • It’ll always be quicker if you work on it. It’s just a waste of time and effort to train someone else up.

– Sure. Except if we had 3 or 4 people capable of working on it in parallel we might get everything that’s on the wish list for it done this year and you might be able to do that other interesting stuff that the rest of the team have been playing with. In fact, I’m willing to invest if it’s that important…

Bad Captain starts to twitch…

…and if it’s not, why do I have someone who believes they’re “one of the best” working on it – I want my “best” people on my priority tactical and strategic items.

  • You’ve been doing this as your own pet-project on the side and it’s not ready to share.

– If it’s important I want a team on it. I’ll happily lobby for some strategic investment. If it’s not as important as anything else we have committed right now I want you helping the rest of the team out…

Bad Captain…

– even if you’re good, you’re not “special”, don’t expect to be treated any differently than the rest of the team.

These first reasons are almost valid. But not good enough for me. The more of these I hear the more I start to twitch, the more terse I get and the more I start to frown and tense up.

Suddenly it’s too late! Captain Hyde takes the helm!

Here’s why Captain Hyde thinks you’re not sharing…

  1. You’re lazy, you’re only any good at this one area and don’t want to have the responsibility of teaching it or learning something new that you’re not so good at. – Time for you to learn a bit about intellectual humility.
  2. If you relinquish control of this code, you relinquish your seat of power, you’ll make yourself dispensable and that can’t possibly happen, you think you’re far too important for that. – I believe in good people, if you’re one of them I’ll fight to keep you. If there’s a squeeze, even the indispensable will get the push. In fact in my experience, the all-rounders are more likely to find new homes.
  3. Secretly in your heart you know it’s loaded with technical debt, years of warts strapped onto the sides, a catalogue of compromises and mud that doesn’t mirror the patterns of the outside world in the 21st century. – In fact I could probably find something open source that does the same thing. Quite frankly it’s an ugly baby that should have been put up for adoption or surgery years ago but your professional pride won’t admit it.

Captain Hyde can’t think of any good reason other than personal self-preservation or embarrassment that someone who is a single code owner would be unwilling for other people to work on and learn their code.

Maybe I’ve just been in management too long but I distinctly remember in my early coding years a call with a colleague in San Francisco where he said:

“Damn, you’re the only guy in the world that knows this stuff man”.

– at which point I felt that little clench inside that said. “I need to find and train my replacements up as soon as I can!”

A similar counterpart of mine in the US at the same time got thrown another pile of stock options every time he threatened to leave – that wasn’t my style as a developer and it’s not my style as a manager either.

Now despite all this ranting, there is another very real problem.

Developing collective code ownership from single points of failure on complex products is time-consuming, expensive and probably not what my business stakeholders want my team “wasting” their time on when we already have experts who are “perfectly capable of maintaining this themselves”.

It’s a fair point.

With a team who are willing to share but have no slack, if I want collective code ownership to succeed I’ve got to be able to market it properly.

The Joy of Peer Reviews (Part 1 – Code)

Posted on by
Reply
Reading time ~ 2 minutes

Pair programming replacing peer reviews is a myth in the same way that “agile projects have no documentation”.

From my experience peer reviews continue to hold a vital place in agile development and software craftsmanship. Unfortunately they are often misunderstood or misapplied.

“Humanizing Peer Reviews” by Karl Wiegers is the best primer I’ve read so far on peer reviews so I’m not going to duplicate Karl’s efforts – I strongly recommend a thorough read. In fact, print a copy and give one to every member of your delivery team to read and discuss.

Like everything else in modern software development, peer reviews are a collaborative team learning experience. Reviewing code properly means both the reviewer and reviewee walk away having learned something and improved their craft.

With code reviews; beyond reviewing for functional correctness, (the simplest, most obvious and potentially quickest part of a review) there’s a selection of considerations I expect reviewers to look for. (There are plenty more).

  1. No code without tests
  2. Good variable naming
  3. Correct use of classes and interfaces
  4. Small methods
  5. Adherence to standards and conventions
  6. Style consistency across the team
  7. Readability
  8. Good test coverage from small tests
  9. Test code follows the same quality standards as production code
  10. Tests describe expected behaviours
  11. All tests pass
  12. Evidence of some test preparation to consider boundary cases, failure modes and exceptions
  13. Clear exception handing, failure modes and explicit boundaries
  14. Sensible error messages
  15. Code smells
  16. Performance risks or tuning opportunities
  17. Security or other “ility” issues.
  18. Opportunities to learn new tricks
  19. New good practices & patterns
  20. Functional correctness

Once you have cultural acceptance on the breadth of technical peer reviews develop your own checklist that everyone supports and remember the goal of a review is to share improvement opportunities, not for lazy coders to have someone else find their bugs for them or for staff to step on each other.

Now…

If you think peer review of code is problematic for your teams, I guarantee that document peer reviews will be a whole lot worse! – see part 2.

Your Leaders Are Not Gods

Posted on by
Reply
Reading time ~ < 1 minute

It’s lonely at the top, but who makes it that way?

In Large companies there seems to be a myth – often perpetuated at the middle tier – that senior leaders are somehow “gods” that cannot be spoken to or at least not in the same way as mere mortals.

The business leaders that I’ve had the pleasure of talking to have been very smart, politically astute, personable, socially aware and most of all they care what people have to say. Admittedly they’re strapped for free time but they’re still human beings.

A casual conversation, sharing of thoughts and opinions or mail exchange should be possible at any level. In fact that no-nonsense, relaxed, open and honest communication is a breath of fresh air from the political games and data feeds faced most of the day.

In any organization that claims to be lean or agile, isolating communication with our leaders to single PowerPoint slides and 2 minute bursts of data defeats the entire point of a true lean corporate culture.

“Go see” also means listen, share, learn, coach, mentor, teach, act, support and most critically interact.

Most leaders understand this (they all started out somewhere) but you may have to cut through a layer of defense to get there and re-educate along the way.

When your leaders do go see, make sure they really see and understand. It’s not all a parade no matter how your local glitterati might want to make it one.

Remember no matter where you are in the food chain, a truly agile organization values individuals and interactions.

Freedom To Vent

Posted on by
Reply
Reading time ~ 2 minutes

Sometimes you just need to let things out.

When weasel politics come into play (instead of just the usual politics), my timer goes off. I know I should be more moderate but I don’t tolerate it, I choose not to play political games, they benefit nobody.  I see it, recognize it and want to call it out publicly.

My ability to be incredibly hard and direct or just downright 4-letter offensive is legendary (but rare). My colleagues coined the term “Bad Captain” for when the Tourettes really kicks in – but that part doesn’t happen in the office.

Sometimes it’s the right approach – I’ve seen public calling-out work wonders on particularly toxic characters but more often it’s a career-limiting and collaboration-damaging thing to do.

If you face an underlying need to flame that doesn’t go away – you know – that urge to write down everything that’s wrong, calling out someone’s personal shortcomings, copying the entire world (and their bosses) and hitting send – remember the following advice…

It’s a bit like peeing in your pants as a kid, it feels warm for a few seconds but gets cold and uncomfortable very fast.

Rather than allowing me to blow a fuse, a former boss and I developed an understanding.

He recognized my need and would allow me to spend that magical half–hour furiously and perfectly crafting the necessary barb-laden email knowing that writing it was critical to my corporate sanity.  He was even willing for me to hit send!

As long as it only went to him.

After throwing my mail-bomb at his inbox, he’d wait for me to grab a cup of tea, take a break and calm down and would then come over…

“I’ve read your mail. There’s some valid points in there. Do you want me to do anything with this or just hit delete?”

That simple response was everything I needed in order to release the pressure, ask for help, get a response and start getting the situation back under control.

If you have staff or team members with a relatively high level of professional passion,  provide them the freedom to vent in a safe environment but support them in learning how to control it themselves and when to pick up the phone.

Lessons in Application Performance (Part 2)

Posted on by
Reply
Reading time ~ 3 minutes

“Captain, we have a customer crisis, get your passport, we need you on the ground tomorrow.”

This would have been fine if it weren’t Thursday night, that I’d been pulling 14 hour days for about a month already and had promised to take my family out at the weekend to make up for it. – Sounds like something out of the goal, doesn’t it!

A couple of years after my first major lesson in application performance, now working for a different company; one of our key customers in South America was trying to get their staff through critical money laundering certifications before a fixed deadline. If this was missed, huge fines were on the cards.

Trouble-was, hardly anyone could get onto the system, it kept going down.

Our implementation team had taken our profile and scaling recommendations, added some contingency and specified the systems for the customer. The customer wasn’t happy with the cost so they’d gone for a cheaper processor and database option which made us uncomfortable but this was a huge deal for us. (It was an early version of SQL Server on Windows 2000 and they were trying to scale to nearly 10,000 users).

I arrived at the customer site. A really great, friendly bunch of people and one of the most bizarre office locations I’d ever been to. The office had a transient community of something like 1,000 staff with no other businesses anywhere nearby. Surrounding it was an entire cluster of lanchonetes that opened at lunchtime – solely to serve this building.

They’d taken the system offline for the weekend and were running their own testing using load runner. Having done scalability tests for customers before this seemed pretty simple. Ramp users up in waves to desired levels, test, ramp up more, test again, tune, retest etc. We recommended a transaction rate of about half a transaction per user per second as a “heavy” load for this type of application.

We reviewed the log files, it was dropping at about 1,000 users – way below where it should have been. I took a dig into their application servers, modified the JVM parameters for optimum heap size, garbage collection etc.  – I’m pretty sure this will work.

Tests showed nothing abnormal. We’re looking good to move forward.

Let’s get the users back on…

10:30 AM Monday morning – The system goes down again.

We take a look at the logs.

Within the space of about a minute, nearly 5,000 users tried to simultaneously log into the system.

Back then, large-scale e-commerce event ticketing systems were not common. For an internal system like this, none of our team ever expected we’d have that kind of load just on login. Even 1,000 closely-packed logins seemed wildly unlikely.

What were they doing?

We talked to the managers about the users. We established that staff in all branches had 2 30 minute breaks per day in 2 patterns (half the staff at a time).

OK, so what…

This was the only time the staff were able to use their systems for this certification. The rest of the time they were serving customers!

These people’s breaks were precious, they had a deadline to meet for mandatory certification this was the only time they could do it.

Solution:  Longer term we fixed the performance of the login routines so it’d never happen again but in order to achieve their short-term goals, the management team staggered break times for staff during the certification period.  (Oh and I’d got the sequencing of one of the JVM hotspot parameters mixed up but that wasn’t so relevant).

Problem solved!

Lessons:

1: When designing systems for performance & scalability (even internal ones), you need a good understanding of peak load based on real usage profiles – including extreme ones. This might seem obvious now but to us back then it wasn’t. We thought we knew how the system was going to be used.

2: Spend a day with the real users, not just the “customer”, Make a point of understanding their goals, and constraints. If you’ve not done so before, I guarantee you’ll learn something unexpected.

3: Sometimes the best solution isn’t technical. In this case, a simple work scheduling change saved a potential fortune to our customer in hardware & licenses.

4: Just as in my first performance lesson; if load is problematic in some areas, build safety margins into the system.