Achieving continuous product delivery in Set Top Box (STB) software projects: Challenges & Expectations

The story all software product development teams face at some point in their journey, not necessarily unique to digital Set Top Box (STB) software development: So you entered the product development space, setup a small team that delivered the unimaginable - you got your product out in record time. Customers are pouring in, marketing & strategy are imposing an aggressive roadmap. You've set the pace, possibly surviving through possible death-march-like project deliveries, and now you're expected to do more: Scale! You need to scale your product development & operations to support an aggressive business roadmap, the competition is intense, you're charged with figuring out how to deliver releases to market as frequently as possible in one calendar year - so you set the target of a releasing every quarter, or rather, inherited a Product Roadmap that showed Market Launches every quarter…

You barely survived getting one release to market with quite a small team, how do you scale your operations to satisfy the business demands? What behaviours and habits do you change? What changes can you make to your current practices? You must've been using some form of Lean/Agile technique to meet the original aggressive delivery, is the process that was used before enough, can it scale to get you from where you are now, to where you want to be? What are the implications of achieving continuous product release flow, in an environment, that is typically unique to Set Top Box hardware & software development?

In this post I highlight just one possible framework that cuts through all the areas involved in product engineering of Set Top Box software releases. I will not go into detail behind the intricacies of this environment (search my blog to learn about that) - instead I will map out by using a few pictures that shows the scenarios around achieving continuous product releases to market.

The pay TV space is becoming highly competitive, especially with the likes of online / OTT (over-the-top) players like Hulu, Netflix, Amazon, etc - such that traditional operators are hard pressed to up their game from providing the clunky, almost archaic standard TV experience to a more slicker user experience, offering advanced features, integrated with traditional linear services with additional services aimed at stifling the modern competition. No longer can these traditional pay TV providers release new software once a year at least, users having become used to frequent updates with new features on their smart devices every few months, people are becoming more and more impatient with bugs that go unfixed for some time…Nowadays, it is expected that set top boxes are updated with new features as often as possible. 

With this in mind, how does one structure a product engineering department to cope with this demand? How should projects be managed? What do we expect from teams? What are the implications of reaching continuous delivery? Where does a typical agile/scrum team fit in? Is Scrum even a preferred method, or do we choose a hybrid model?

[This is still very much a work in progress, made public to get feedback/advice from other practitioners out there...]

Focusing on the soft, people side in coaching agile teams

In my day-to-day work I generally impart advice and guidance to all types of teams that play a part in the projects that I run, though it seems to be from the point of a "program manager parting some guidance on development / integration topics" - it is expected from the role, not strictly being measured by the progress I make with it, considered as free-and-impartial advice. I have very recently however, landed my first official gig as an Agile coach, taking on a group of fairly young people (split into two teams, but part as one group). I was at first a little edgy with this engagement to be honest, since my exposure to date involved middle managers & team leads, whereas this involves interacting with a fairly young, dynamic & fresh "Gen Y" bunch, and to top it off, are not involved in Software / Systems engineering, instead operate at the Business Intelligence / Customer Experience process area.

I'm hoping I can share this new journey as part of this blog, so here goes. I am, by the way, the third coach to be assigned to this team, so in my first two-hour session, I decided to focus on the classic retrospective: What have you learnt to date?, What would you like to learn? What's your goals? Lets create a backlog for this journey that we can measure progress over each week?? The audience was a mixed batch of people, some having been exposed to a years coaching already, others only a couple of months, having recently joined the team.

I also wanted to touch on the people, softer topics first as a measure to break the ice, get to know everyone, and see where it goes. It turns out though, that the soft topics took the entire session, people were quite engaged, quite a few topics, comments and innuendos surfaced that pointed to deeper people / team challenges so solve in the background, at the same time, thinking about how to promote process improvements with value stream mapping, etc. I had sat in on two previous coaching sessions where we created the team's process map, that still needs to be rationalised.

In the end, we left off with a few exercises for the team to go away and come back to the next retrospective with feedback around: Team Charter & Appreciation Agreements. I also ended by asking for direct feedback for my own self-learning, reflection & improvement.

Appreciation Agreements

One of the first things I do when running retrospectives and other workshops (planning, brainstorming, post-mortems, etc) is to set the stage for the session by introducing the working agreements, or appreciation agreements. This is necessary and vital to creating an atmosphere for collaboration, openness, trust and respect. Some of people in the team were already familiar with this topic, but expressed appreciation for doing the refresher since they "learnt about it before, had never followed through and consistently implemented it in practice". The stuff that came out of this conversation was enlightening, pointers parked as exercises for the team. I covered the following key points that seems to be a common starting point for setting the scene, pretty self-explanatory:

Use Root Cause Analysis (RCA) in Sprint Retrospectives to Improve Quality

A couple of years ago, I wrote an exhaustive piece about how software and systems projects need to focus on effective quality management across the board, covering in some fair amount of detail the various aspects to quality management, drawing on the experiences from a past project of mine as a case study.

In that paper, I closed by touching on how we, the management team, used as an intervention to improve quality of releases, by injecting Root Cause Analysis into the project stream, by embedding a Quality Manager as part of the team (~250 people, 20 odd development teams), whose role was to monitor various aspects of quality, covering escaped defects, quality of defect information, component code quality, build & release process quality, etc. This manager met with component teams on a weekly basis, instigating interventions such as code reviews, mandatory fields to input information into the defect tracking (for better metrics / data mining), proposing static analysis tools, and so on. The result of this intervention saw a marked improvement across the board, right from individual component quality to final system release quality. This was a full time role, the person was deeply technical, very experienced engineer, and had to have the soft skills to deal with multiple personalities and cultures. Because we were distributed across UK (2 sites), Israel, France, India (2 sites), we had in-country quality champions feeding into the quality manager.

In the above-mentioned project, for example, we ended up with something like this (at a point in the project, Release 19, not the end), the snapshot shows progress over 11 System Releases, capturing 33 weeks of data, where each release was time boxed to a three week cycle:

Starting off, in order to do a reasonable job of RCA, we had to trust the quality of data coming in. Our source was the quality of defect information captured in the defect tracking tool, Clearquest. The stacked bar graph on the top of this picture shows how we measured the quality of defect info: Dreadful, Poor, Partial & Good. The goal was to achieve Good, Reliable defect information - over time, through lots of communication and other interventions, we saw an improvement such that there was barely any track of Partial / Poor / Dreadful defects.  The radial graph shown in the bottom, measures the trend of root causes per release, and movement of causes from one release to another. At the end of Release 19, we still had some "Design" causes to get under control, but we'd seen a substantial improvement in the level of "Coding", "Architecture" & "Unknown" issues. For each of the causes, the team implemented interventions, as described here.

In this post, I drill down into a component development team that's only just starting with agile / scrum, to show how the team can adopt similar Root Cause Analysis (RCA) principles in managing the quality of their own component delivery, by leveraging off the Scrum Retrospective as a platform to implement an Inspect & Adapt quality improvement feedback loop. 

In search of low hanging fruit following on from high level retrospective

In my previous post I shared the experience of running a retrospective with a management team who are finding it quite a challenge to maintain a coherent team together that's needed to deliver on an aggressive product roadmap. 

The outcome of the initial sense-making workshop highlighted what in essence, is a longer term vision and strategy for the team, and could be used to establish the goals for this particular team (check out an earlier post on Agile goals). The workshop has indeed left the management team a little nervous about jumping straight in and tackling the BHAGs (Big Hairy Audacious Goals), and felt that we should instead start addressing potential low-hanging fruit first. Of course, this is a natural reaction to change, but not entirely unreasonable. Recall the scorecard assessment shown on the left.

To jump in an solve the "Efficiency of Team Structure" problem is bound to be quite disruptive, especially when the teams have just started experimenting with some internal improvements on their own. Added to that, the department structure that's currently in place, doesn't lend itself well to having fully cross-functional teams. Added to that, some department managers have yet to settle between themselves the expectations & services provided by each area. We can clearly see how going agile puts pressure on traditional segmentation of departments by role, e.g. Development vs Systems Integration vs QA, etc.

Personally I've been advocating for a single, unified, cross-functional team approach for some time, but have faced the classic resistance from managers because its new territory, not really been done before, alarm bells ringing. This hasn't been made easier by the surprising rapid increase in headcount and growth of all the departments, dealing with multiple projects and platforms, that the guys just didn't have time to sit back, and discuss overall strategy & direction for each service area.

We continued to plough ahead, this time, going back directly to the foot soldiers in all teams, where each manager was asked to hold internal team retrospectives, addressing these three questions:

• What are your current top, pain-in-the-ass issues, your most frustrating bug-bear?
• What do you think the Root Causes are?
• What would be a potential solution to the problem?

The topics again sparked some interesting conversations, some people also took feedback quite personally, people came up to me afterwards and shared their emotional frustrations. It is interesting how the different managers and their teams respectively, have different perceptions, expectations and understanding, yet people still tend to work be working together as a team, on the surface. The point of these retrospectives is to highlight these hidden topics, feelings & emotions are quite important. People need to feel comfortable sharing their experiences and opinions, without being intimidated by fellow peers, we also don't want peers to start defending themselves. Lets just get the issues out in the open, and start to process them in a systematic way.

The low-hanging fruit that came out from this: Improvements in Pre-Planning, Improving Control in SI (System Integration). Interestingly, these areas map nicely into the scorecard assessment that encapsulates the eventual goals, covering: Quality of Planning and Level of Control, no surprises there.

As the neutral facilitator, who's run a few of these workshops over the last year, the following is quite evident:

• Different perceptions & understanding what's required from an Agile Project, including misplaced assumptions
• Lack of clarity about the department's new structures, understanding the difference between Development, Delivery including quality
• Clash of expectations (where does agile fit in, when to switch to traditional SI model, etc.)
• Sentiment of lethargy or inaction - most of the improvement actions have been identified before, but no-one has had time, or had taken ownership of driving the improvement, no follow-through
• Bigger people, softer issues around team camaraderie

The journey continues…

Above & Below: Outputs from the session

High Level Agile Retrospectives: Sense making with the management team

In my previous post "Will the real product owner please stand up?", I sketched the scenario where a

development team is pressed to flex and adapt to product roadmaps that tend to be generally quite aggressive, where a roadmap is generally accepted as template for driving the delivery plan and so the expectation is set at the highest levels of business, which means the team must just get on and deal with it.

This scenario is typical of many software product teams, where the success of the initial product launch was due to a Herculean effort of the entire team, the product gets out on time; and that original pace of delivery then sets the scene & the tone - the precedent is set such that people have the expectation that the team can pull it off again and again.

This is all well and great since the team has inspired confidence that they are committed and can indeed deliver. Little of the legwork and thrashing that went on internally in the development team is known to the business stakeholders, the team would still love to commit to the roadmap, but are quite edgy about the reality and implications of doing so.

We are faced with the classic problem of scaling, the notion that "What got you here, won't get you there!" scenario unless something changes - but people are nervous about abrupt changes, because the teams are currently working through their own internal improvement plans that first need to be seeded for a period of time, reviewed & then evaluated, before committing to subsequent changes into the feedback loop. 

As an Enterprise Program Manager, it is my job to put a high-level plan together that shows the stages of delivering on the product's roadmap. I become comfortable with doing this planning as long as I have confidence in the development & execution processes of the impacted delivery team. Unless I have confidence that the teams understand what's expected, that they have a reasonable processes / template for execution, I will not put my name against a delivery plan, that I know is close to impossible to achieve. 

So I roll up my sleeves, don my consulting hat, and try to facilitate conversations with the development & delivery team. In this post, and the series that will follow, I share my journey with a management team that face interesting challenges with balancing the delivery expectations of the business on the one hand, and growing their young agile development team on the other hand, by applying systems thinking tools of sense-making, to better understand the challenges that await.

I work with the mid-senior managers only, expecting the respective managers to have held internal team conversations before-hand, bringing all their team's issues to the table with the rest of their fellow management peers. Despite the team adopting an Agile/Scrum mindset, it is quite interesting to note that the "one-team approach" is really quite difficult to achieve in practice.

I share the results of the first sense-making workshop, that was guided principally around answering these two questions:

• Given your current way of working, how much confidence do you have in achieving the roadmap?
• What ideas can you suggest that we should look into, that if we implemented now, could help us to get close to achieving the roadmap?

Interestingly enough, to my surprise, the answer to the first question was a resounding NO! The team had little confidence the roadmap could be delivered upon, not even a Yes from the product team!  The second question generated some interesting ideas that sparked some healthy dialogue. Upon reflection, it turns out that our problems aren't that unique, sharing quite a lot in common to all development stories you read about. More interesting is that the ideas were not new topics for the teams themselves, showing that how teams can get stuck-in-a-rut, dealing with the everyday pressures that take higher priority over implementing and driving changes. Despite having many internal retrospectives, the same topics were brought up, showing clearly a lack of championing and initiative from within the various teams. Without such initiatives emanating from within the teams, how can management not ignore this, and not take the top-down approach to managing??

Technique used in the workshop

I facilitated the retrospective workshop, playing the impartial neutral person. The people involved were a mix of middle & senior managers spanning product management, software development, systems integration, agile managers, business analysts and quality engineering.  The idea generation workshop lasted two hours, followed by an hour session the next day for sense making. 

I used the concept of "Rounds" where each person gets a chance to speak & offer comment without interruptions. The rule is that everyone must respect each other, no interruptions, and strictly no challenging views - the place must be a safe one where people can speak freely without being afraid of fellow peers or line managers (tricky since it was during performance review time!). Following the initial ideation workshop, the second session was around applying Systems Thinking tools to unpack the ideas.

Systems thinking is around gathering the data into themes, looking for common traits, allocating a main topic by describing the term as a variable. For example, topics touching the subject of "Quality" is grouped as a variable "Level of Quality". Once these variables are identifying, we look for patterns of influence: How does each variable impact the other?? We look for relationships between the variables, using what is called an "Inter relationship or Network" diagram. From that we generate a Scorecard that shows us the core driving variables. From the scorecard, we assess and estimate the amount of effort we're spending in each area. Once that is known, the ones with the least effort, are usually candidates to immediately improve on.

Index cards, stickies and posters are used throughout the workshop.