Hit Squads as a bridge to agility

If you've read some of my previous posts, you will know that I write mostly about software projects in the world of digital TV set top box (STB), broadcast headend systems, including internet TV, Video-On-Demand (VOD) and over-the-top (OTT) projects. There is a tremendous amount of software running in these components, end-to-end, from the STB device (which in itself is a complicated system), to the headend/backend server-side components.

The development teams are usually not under one roof, are less likely to come from single suppliers, with their own methods of working, their own release / test cycles. It is difficult, but not impossible, to establish a regular cadence for the overall delivery stream. Some teams may be following agile/scrum, without continuous delivery - and others prefer to work in a more staged, requirements-up-front / development / test / integration cycles.

There are cases however, where a large part of the development, test, integration & delivery teams are in-house, but segmented by classic functional organisation structures that result in silo-based mentality. I've seen this in a few places, especially when for example, PayTV operators take ownership for product development in-house. The application development team for instance may be following scrum/agile, other teams however, don't.

The situation is almost always the same in such projects: Driven by a Hard deadline. Typical development cycles until feature complete. Enter test/integration cycle (this is usually the first time you know the true status of all the key features and functional areas - expect trouble). You find out there's issues, you're not even close to being feature complete.

The deadline isn't going to move and you've eaten up your development schedule (you're now into the time to stabilise and in what should be surgical mode). Sequential processes with silo'ed teams are a hindrance - you need a quick way to uncover issues, resolve them quickly, providing quick feedback into the project stream. There's pockets of agile/scrum in some teams, but not everyone is convinced that is the way to go. You don't want to disrupt the teams completely yet at the same time promote a different style of working.

What can you use as a bridge-to-agility without getting into the whole methodology debate??

Enter the Hit Squad Team (or also known as Tiger Team) - a concept that I've used on more than one occasion. If managed well, the benefits of having cross-functional teams are obvious. The lessons you take from here are used in your next delivery project, likely to become the preferred choice of working. I've seen this transition take shape & became the norm, without having to religiously convert people to a new mindset - I've also learnt it aint that easy. As many more before me have warned, what worked for one organisation isn't necessarily going to work for others. Still though, if you're operating in a similar technology space or problem domain like STB development, it wouldn't hurt to try this out...

What's a Hit Squad then?

Project Premortems

I recently ran, what was probably a first in South Africa (?) (SA readers please reply if you've seen this implemented in other companies), and definitely a first for the company in question, a project premortem. I had asked around the various stakeholders which included people with decades of working experience, and no-one had known about this concept of "project premortems". Sure, people were familiar with project postmortems (or end-of-project reviews, more recently called "Retrospectives" in agile parlance) but this "premortem" concept was an enigma, as it was to me, before I came across the concept from a book (2014 release) I read earlier this year, Scaling up Excellence by Robert Sutton & Huggy Rao. Although this book's core messages are around scaling up organisations, a lot of the findings are equally applicable to the context of running large, complex projects. In my world, I manage large technical programs that involves many separate business units, clients & third-parties, with large development teams...The idea is now seeded in the company, people are talking about it, and it was well received by some CEOs, who have started using this concept in their high-level steering meetings (awesome)...

I am a huge fan of Sutton, having first come across his work in Good Boss, Bad Boss: How to Be the Best... and Learn from the Worst and The No Asshole Rule: Building a Civilised Workplace and Surviving One That Isn't. I also keep up with his blog at work matters, follow him on twitter & listen to podcasts whenever he talks at Stanford's ETL lectures, and he is also my personal connection on LinkedIn.

As a program manager I have run various planning scenario workshops in my time, the classic risk brainstorming sessions, best-worst-case scenarios, etc - but I had never labelled the event as a "project pre-mortem", the name resonates with so many people because of their encounters with the classic project management tool of "project post-mortems". Not only the naming resonated well, the concept and implementation of this activity is in harmony with Management 3.0 concepts and practices of inclusion & transparency in agile practices. I was looking forward to changing tact, to experiment and try this out on a bunch of people (old-timers and youngsters) who are relatively receptive to new ideas and ways-of-working. It wasn't a complete success especially finding time out from a busy project that's already in motion, people came to the workshop a little skeptical, felt out-of-their-comfort zone, but nevertheless respected the process and participated as best as they could.

What's a "premortem" then?
Before I describe my custom implementation of this premortem, quoting Scaling up Excellence, Chapter 8, "Look Back from the Future", pages 264-265:

...We close with an additional twist, a mind trick that goads and guides people to act on what they know and, in turn, amplifies their odds of success. We build on Nobel winner Daniel Kahneman's favorite approach for making better decisions. This may sound weird, but it's a form of imaginary time travel. It is called "the premortem". Kahneman credits psychologist Gary Klein with inventing the premortem technique and applying it to help many project teams avert real failures and the ugly postmortems that often follow.

A scaling premortem works something like this: when your team is on the verge of making and implementing a big decision, call a meeting and ask each member to imagine that is is, say, a year later. Split them into two groups. Have one group imagine that the effort was an unmitigated disaster. Have the other one pretend it was a roaring success. Ask each member to work independently and generate reasons, or better yet, write a story, about why the success or failure occurred.  Instruct them to be as detailed as possible and, as Klein emphasizes, to identify causes that they wouldn't usually mention "for fear of being impolitic". Next, have each person in the "failure" group read his or her list or story aloud, and record and collate the reasons. Repeat this process with the "success" group. Finally use the reasons from both groups to strengthen your scaling plan. If you uncover overwhelming and impassable roadblocks, then go back to the drawing board...

How I customised the premortem for our context
As highlighted earlier, I help co-ordinate and steer large-scale projects that involve a few teams from different business units (with their own CEOs/CTOs), that must come together to develop a technical platform (set top box) and product/brand features (which is not owned by the engineering team responsible for the STB) and various operational management entities. We have over 150 people working on the project, scattered across the business. The project has been ongoing for a few months already, we've had team-wide or component-wide risk sessions, retrospectives and work was well underway. With an imminent release ahead of us, I wanted to hold this premortem with the primary focus of surfacing any issues / concerns that have either fell through the cracks, or swept under the carpet (which is often the case), ensuring we identify accountable owners for maintaining control and steering of topics to result in a positive outcome.

Leave it all behind

I have a favourite T-shirt that has imprinted on the back, the words "Leave it all behind". I came across this shirt ten years ago, when I saw it, I just felt I should get it, the slogan profoundly resonated with me on so many levels, as if it was created just for me! It became my favourite Friday casual dress to work. Ten years on, and it still remains a favourite, so much so, that my wife went out and produced a replica for me as a father's day gift. A friend recently probed why this particular tee resonates with me so much...on what level, what/why does Leave it all Behind signify - so I tried to explain to him, and now I want to share this part of me, publicly on my blog, for all to see.

Yikes, isn't this risky thing to do? Why on earth expose myself like this?

Well, one of the reasons for starting this blog, was to push me to my limits, to the edge of my comfort zone, embracing the public of the internet, being inspired by the writings of Jeff Jarvis, people (friends, colleagues, potential recruiters, clients, etc.), to experiment in this new world...

So here goes a new "About me" post...

Work-Life
If you've read other posts about me, you will have learnt that I've worked hard to get where I am today, with an almost relentless passion to push myself to the limits, to never give up, hard work, determination & grit. Part of this desire lies with my tendency to leave my past life behind, do what it takes to break-away from the underprivileged, below middle-class, above-poverty line life that I grew up in. In doing so however, I realised that working too hard all the time isn't everything, that work too, should be just left behind:

• I try not to take work home in the evenings or weekends - I leave it all behind.
• Work can wait (I generally average around 10 work-hours a day), there are other important things to consider in my life: my family, my own personal space for spirituality, and my hobbies, interests (like reading, blogging, self-learning) and my desire to experiment with new ideas to look at starting-up my own business one day.
• When I'm at the office, I give it my all, my 150% attention to detail, follow-through, focused and try to be hyper-productive. I set myself goals for each day, aim to get everything done and dusted - and leave the office switching my mind off from work.
• I learnt this the hard way. I worked on a few death-march like projects when I used to work in the UK. Worked to the point of burn out on more than one occasion, the constant is that the work will always be there tomorrow. Leave it all behind, start fresh tomorrow.
• I try to switch off from work when I'm at home, it has become easier over the years. No more do I wake up at 3AM writing up emails, planning the days work, solving problems. Although sometimes I can't help myself when I'm really passionate about an idea, but most of the time, I leave it all behind and don't let work be the focus of my life...even now that I'm a consultant and don't have the comfort of a permanent pay cheque, holidays or insurance - I still have a good night's sleep.
• Don't let things at work affect me too personally
• I once was on a project that had SLAs in place for engineers to be on-site supporting the customer in the states (LA). I was looking forward to my two weeks stint as I'd never been to America before, and I was the senior engineer on the project anyway - it so happened that at the time of this event, my in-laws made their first trip to the UK. The timing of the US Visa application didn't make forward planing any easier, so I was left with a choice of staying back and fulfilling the rights of my wife and in-laws, or leaving them for two weeks and go to America. I chose the former, choosing to Leave it all behind at work. Yes, it did take a toll at the office, my commitment was questioned, at the end of the day, another engineer went in my place, engineers are a dime-a-dozen, parents & family are priceless.
• For over two years I ran a daily morning power meeting, 9AM-10AM, with really difficult managers, characters and personalities. I fell below most of them in the pecking order, yet I had to co-ordinate, assign them actions and get updates. Sometimes there were heated debates, sarcastic remarks (this was the UK of course), I would be left drained after the meeting...I learnt to shrug it off, Leave it all behind
• Even today, I run my meetings and workshops - people show up or not, pitch up complaining about not having enough time to do real work, etc - I'm not phased. My meetings have purpose, there's always a goal with expected outcomes, I give people advance notice, time to prepare. I'm human, I do take note of comments, value feedback and take criticism to self-reflect and improve things, but my point is, I don't let negative experiences in the work place affect me when I leave the office - I leave it all behind.
• Don't get too stuck in projects, there will always be another project down the line
• On more than one occasion, I took a break from core projects by transitioning just before final launch / go-live. 
• In one instance, I had done all the work leading up to the release, the last issues needed closing out, there wasn't much to do and I didn't want to wait till the end to see through maintenance phase, there was another job waiting for me. So I left just before launch.
• Another instance, the project had taken on a few more managers that I was transitioning and handing over work to, I felt that my personal life was suffering, so had requested a month's leave to go back home to South Africa, to see family, to re-energise, and also fix a couple relationship problems. It was a difficult choice to make, I gave a lot of energy to that project, and to leave the project just a couple months before go-live and handover the reigns to other project managers, just didn't feel right...but there were other pressing matters to sort out, so I left it all behind.
• There will be other projects, my contribution to the projects is shown by the legacy I leave behind.
• A good project manager knows when to handover the baton to others, and is willing to leave a project when the tipping point is within reach, when you're confident the worst is over, and the project is on a path to completing.
• Be prepared to switch, take the risk & leap-of-faith
• I left my home country, South Africa, because I saw a brighter future for me overseas, financially and professionally. I left my entire life behind, ventured into unknown territory, alone to see what will happen - I left it all behind
• I fell in love with Dublin, Ireland - I thought I was going to live and work there for the rest of my life. Sadly, the work-situation wasn't that great, I experienced my first lay-off, which was a great learning experience. I learnt never to get comfortable, be prepared to leave it all behind and start fresh
• I decided to switch from a senior embedded software engineer position to being a junior software engineer in enterprise / server applications space. I was on the road to promotion and being a senior/lead in embedded set top box projects, I left it all behind and started anew in a new team, new product space, new domain...and I never looked back. That experience rounded me in so many ways, to the point of working for the best self-organised, high-performing team I've ever worked with.
• I had the perfect job before deciding to leave the UK, I had reached my goal of working for blue-sky, start-up like projects, the advanced technologies labs team that looked ten years ahead. Free to dream and invent new things. The competition to land that job was great. I had left management all behind and returned to coding, working on accessibility projects, doing tangible value-add stuff. Two months into the job, another personal decision materialises: to return back to South Africa, after spending ten years building up a career (and a life: had my own house, excellent relations across the business spread across four countries, kids of school-going age, good friends) I left it all behind.
• Most recently, I made a switch into being an independent consultant - it wasn't easy leaving it all behind, as I was turning down a very good permanent senior position in the company, we had just completed a major product delivery, I was set to go places...but personally, company politicking-aside, I felt I could achieve more for myself, on a personal & professional level, so I left it all behind to start afresh as a consultant, opportunities abound in South Africa, especially with the skills-knowledge-competency gaps in the country.
• Choose my own work
• I would like to reach a point where I can cherry-pick the work I take on, aspiring towards working a 20 hour-or-less work-week, to focus more on things that add value to me, leaving the rest behind.

Personal / Spritual

Part of my growing up, I knew what it meant not to have things. It thought me self-control, discipline and a sense of awareness that life isn't all that easy - sometimes dreams, desires and fantasy must be left behind because of the facts-of-life & reality of the situation (such as apartheid). I left behind my desire to study medicine because I couldn't afford to, had no money. I left behind that college dream and was prepared to work-and-pave-my-way, until an opportunity presented itself...

In my teenage years and early adult-life, I came into contact with people on the Sufi path, focusing on inner reflection and perfecting one's self. To let go of the ego, see this world as just a temporary stage, to strive to being a better human being, letting go of the ego ("Nafs"). These experiences shaped the path I would tread, to be willing to leave it all behind when required. In my years 20-25, I would seek out contentment, meditating each evening - to reach a point of being satisfied with my lot, of having just enough - whilst at the same time seeking out that balance of surviving in the world of work, and maintaining a spiritual path.

There's a saying that's attributed to Jesus (we Muslims hold Jesus in high esteem, as a prophet and messenger of God), that goes like this:

Jesus, son of Mary (on whom be peace) said: "The world is a bridge, pass over it, but build no houses upon it. He who hopes for a day, may hope for eternity; but the world endures but an hour. Spend it in prayer, for the rest is unseen"

For a while that saying struck a chord with me, and may have delayed some of the grand ambitions I'd had earlier. Nowadays, I choose not to lead such a detached life, I'm aware of the temptations but accept I have to focus on achieving as much as I can in this world, at the same time not losing sight of the bigger picture, that this life is just temporary.

What is interesting is that Islam stresses the temporary nature of life, right from the moment of birth. When a child is born, the Azan/Iqamah (call to prayer) is read to the child as early as possible. When we die, the prayer service held, called the "Janaaza" prayer, omits the call to prayer. The reason is that the call to prayer was already given to you at the time birth, re-iterating the short nature of life: you enter this world to leave this world...be prepared to leave it all behind.

I use Leave it all Behind as the trigger when praying the Salaat (form of prayer that partly entails physical actions standing, bowing, prostrating - being mindful to God) that starts off by lifting both arms up, hands to the ears - the act of doing so signifies taking all the world and pushing it behind you, leaving everything behind to clear your mind and focus on the task at hand, of being fully present for prayer...

So spiritually, you can see where my personal biases come from - which has ingrained this personality trait.

Other personal experiences that re-enforced this message of Leave it all Behind is when I chose to leave my past life of being a bachelor, with many friends, including other women friends, past relationships behind - and focus on my new life, with my wife and kids. As difficult as it was to let go of past relationships, there is more value to my preserving and maintaining my own family life....

On the practical side of things, I also try to live lean-and-mean. I try not to hoard (my only guilty pleasure is a collection of printed books and journals that keep growing and growing) stuff - if I've not used a piece of clothing, furniture, gadget, etc in the last six months or so, I get rid of it. I keep the bare minimum required for me to survive, the less baggage I have, the better to Leave it all behind...

Core competencies/behaviours for System Integration Engineers & Release Managers

I have written in the past on system integration topics, you can click on "Integration" in the tag cloud... 

It is interesting how some of the topics keep recurring throughout ones working life, recently, I've had to observe, deal, control and to an extent mediate on issues that shouldn't really have happened - this is in the area of Software Integration & Release Management, which in short, is not an easy role to take on in any project, let alone one with a hard delivery expectation with the whole world watching...people become stressed, nervous, emotional with knee-jerk reactions, etc, etc.

A major emphasis of Release Management & Software Integration/Delivery is on managing stakeholder expectations, above and below. To do this one expects a certain level of core competencies, which I believe is around these core topics:

• Great judgement when it comes to soft, people skills (be weary of pissing people off, triggering emotive responses & causing undue panic)
• Clear, unambiguous communications: knowing what to communicate & when (filter out reports, allow time for things to settle)
• Measured, personally self-motivated & control (it isn't easy being under fire all the time)

I've sketched a picture of what this world looks like, it shows just the world of set-top-box integration, it gets more intense as you go through to the worlds of end-to-end integration, where there's multiple systems involved (not just the STB), multiple customers, and way more front-line fire you have to deal with:

The picture should say it all:

• Release Managers are on the front line when it comes to committing to a project delivery timeline. They front many customers, including some senior stakeholders. Release Managers work closely with System Integration (SI), in fact, should rely heavily on the output from SI
• System Integration not only front hard timeline commitments from the Release Managers, but also, have a direct line-of-sight to stakeholders. To be in SI, you need to have competencies not only on technical toolset, but a fair amount of personal qualities, including very clear communications style, to front queries from stakeholders.
• The Development vendors are usually shielded from a lot of the fire that SI face, although these vendors must deliver quickly and resolve the burning issues in a timely manner. Some projects allow for direct access to development teams by stakeholders, however, it is not always the best strategy. SI is usually the protector and conduit, often buffering the development teams. Release Managers still manage vendor planning though.

What do I expect, as a minimum, behaviours of a System Integration Engineer??

• DRIVEN - I think this sums it all!
• Self starter, ramps up in no more than two weeks delivering value
• Grit, rigour & diligent, hardcore problem-solver
• Not easily demotivated, although big enough to seek help when truly stuck
• "To be Blocked is to admit defeat!" kind-of-attitude
• Go the extra mile to solve problem
• Be able to take a high level problem and run with it, even if the problem is unconstrained & unbounded
• Passionate about problem solving and debugging
• Clear communicator at all levels
• Works well under pressure & stress
• Comfortable with client-facing interactions
• Doesn't take "if it ain't broken, don't fix it" too literally. Expect tools, scripts to be created to add value, instead of accepting status quo
• Must grasp source code / software design patterns quickly, even if it has to be done through a process of re-engineering
• Does not have to be told twice what to do
• Dependable, ability to work with multiple tasks concurrently often with competing priorities
• Ability to work with third parties, including managing & directing external parties
• Does not depend on too much guidance from team lead or project manager
• Thick skinned to pick up on nuances of politics and management dynamics 
• Does not panic, is in control of emotions, tackles problems in a clear, calm manner
• Fully committed to delivery expectations of project, striving to keep customers happy

What do I expect, as a minimum, from a Release Manager??

• All of the above traits from SI plus:
• SENSIBLE sums it up!
• Level headed.
• Keen eye for detail, but not to the extent of micromanaging teams
• Must understand the boundaries, limits of influence (sphere of influence)
• Communications must be unambiguous - do not invite panic!
• Must be able to sit in hot seat with calm composure
• Must intervene tactfully & gracefully in conflict scenarios within the team but also more importantly with third parties / vendors
• Must be familiar with people management - take time to get to know people on the team
• Very good listening - be sure to understand what technical people are reporting / expressing, filtering out noise
• Not to escalate unnecessarily
• Be prepared to say "I am not sure, but will find out"
• Be prepared to protect the teams - avoid dictating, be cognisant that people have personal lives, and may have baggage to deal with
• Seek out guidance when not sure, speak to peers & colleagues who may have experience to help you along the journey
• Must be capable of realistic strategic planning, but not in isolation of the team
• Should have a led a team of skilled engineers of 5-12-20 people in the past with at least 3-5 years experience as a team leader
• Should have been involved with development / integration with a realistically accepted time frame (at least 3 years)
• Should have software domain knowledge in terms of best practices, even if its from other similar software systems
• Must be able to lead and manage all types of people
• Must be able to plan using past experiences & learnings
• Must use knowledge & experience of own proven practices & processes to up skill team if processes are lacking or immature

KPIs/Metrics for Set Top Box Application Development teams

I have learnt some interesting insights into the life of consulting, especially around change management, organisational transformation, leading, influencing and inspiring mindset shifts. One of the challenges is meeting a team that is on a level of maturity that is screaming out for intervention, and having that self-control to contain myself from blurting "I told you guys this three years ago! And only now you're seeing the light! You need to listen more!".
As a coach, one has to be patient, and live the journey with the team, this is okay, I accept that. And yes, it is quite rewarding to see the team come off age, mature and eventually implement, performing, if not outperforming, your own expectations.
However, it is somewhat a little more challenging to have hard delivery timeline pressures thrust upon you, knowing that a team isn't prepared yet (not at the right maturity level, will need micro managing and lots of admin/management overhead), or have the building blocks in place to not only deliver, but to continue on, post-launch with a sustainable way of working...
So the journey has to be lived and walked through with the teams, even though you as an expert know the destination already, even if it maybe 3-5 years to get there...

The story has been repeated in my lifetime a few times already: New product, one customer, hackathon to deliver, deliver, then the struggle of maintenance & support, and the rush to support new products & customers...
Example:
We start with a fairly young application development team, responsible for Java development of a Set Top Box EPG / User Interface. We try out this new thing called Scrum and aim to operate using the Scrum/Agile framework, we lack the supporting engineering tools & processes to manage quality (no real time to focus on CI, automated unit tests, etc) - we deliver against the odds to make an impossible launch happen. We hoped we'd have time to settle, fix the mistakes and improve working processes in time for the next release, but the work continues to pile on. Not only do we have to deliver subsequent releases, but now have to support other products as well, with the same sized team. Management want to hear none of "Refactor, Rework, Technical Debt" - on top of that, management decide to implement Key Performance Indicators (KPIs) as a way of measuring productivity...

This story isn't new, I've seen this repeated a few times especially with STB product development. You start with what looks like an elegant design, over promise the capabilities, a real demanding customer comes along with an insane delivery target, the elegant design gets infected with hacks, the hacks turns into product, the product launches, customer is happy, expectations are now set in stone. The app is reused for other products, the customers increase. The team size remains the same. We are asked to deliver more with the same "resources" and deliver with improved quality. We will be measured by the quality of our component delivery. This, whilst all the time, running parallel streams often with simultaneous or overlapping component releases for one or more hardware products (Zappers, PVRs, etc.) - Sound familiar?? Probably not as unique to set top box software development right??

What do you as a development manager do? You just embarked on the road to Agile/Scrum. You have a massive backlog of technical debt. Your team isn't performing at the level or maturity it should. Management is pressing on some metrics from you that you must use to justify progress towards increased productivity...

Apply MVP to any project in life or work

Courtesy of Henrik Kniberg, brilliant illustration

When it comes to product development, or any other project, nothing shows it better than this picture:

In the case of set top box products, we usually start with basic STB, aspiring to the advanced:

- Always satisfy "watch TV"

- Single tuner SD zapper

- 1+ tuner video recorder

- n+ tuners PVR

- HD user interface

- + animation

- + 3D OpenGL ES graphics

- + connectivity: USB, Ethernet

- + video on demand

- + home network streaming

- + remote management & diagnostics

- + recommendations engines

The above list is your stock standard product roadmap. To get there we always  release the basic minimum viable product, interestingly the constant expectation throughout each iteration must preserve "can I watch TV??"!