Process for Fast-Tracking Issues - The Daily Scrub

In this post I'd like to share a simple, yet effective process to managing the defect discovery phase, typical experienced during the early stabilisation phase of launching a STB product. The process does not only apply to STB projects, it can be used for Headend projects; or even general Software Projects that involve a number of components requiring system integration. Since the bulk of my experience comes from STB projects, the information presented here thus has a bias towards STB projects.

The material presented here is not that ground breaking, really. If you've been in the business of delivery DTV projects for a long time, then you probably have a tried and tested way of managing the system integration processes. However, if this is the first time you're involved in system integration, you are either a PayTV Operator taking more ownership of SI (System Integration), or you might be a professional services outfit and have won your first big project - If you fall into either of these categories, then I have no doubt that you'll find this useful, at best, trigger to review your currently planned strategy.

As an aspiring independent consultant, the Scrub Process is yet another addition to my PM Toolbox. Recently I had to present this process to a project team, who's processes were a bit old-fashioned and rather rigid, and slow. I needed a medium to express the concepts of gaining more efficiencies out of the SI Process, thus put together this presentation to communicate the idea, which I'll only briefly touch on here. For more details, please download the presentation.

Set-Top-Box (STB) BootLoader Management Template

The Set-Top-Box Bootloader is a crucial piece of software (actually firmware that is generally burnt into (ROM) Read Only Memory, usually One Time Programmable (OTP)) that performs not only the vital job of the classic bootstrap (Power on, boot, launch application) for the device but also is fundamental to applying software updates to the system.  Like in PCs, the bootstrap is the initial code that is loaded that generally performs a self-test to check if the device configuration is OK, and looks for an image to load that essentially boots the device into operation, ultimately loading the EPG application.

The Bootloader is generally implemented as very low-level code, with limited set of drivers to enact the basic hardware functionality, with the focus of occupying as small a footprint as possible. There are different types of Bootloaders implemented in STBs, generally known as "n-stage" Loaders. Typically you find a Singe-Stage Bootloader and Two-Stage Bootloaders. A Single-Stage Bootloader entails an all-encompassing loader, self-sufficient, not requiring the help of another bootstrap to continue the boot process. Whereas, a 2-Stage Bootloader, as the name suggests, includes a secondary stage, that kicks of a higher-level bootstrap to facilitate the loading of subsequent system components.

As a STB Project manager, there will be a occasions where the Bootloader requires special attention. This is especially true for projects involving brand-new hardware, different Middleware or Drivers. Generally though, the Bootloader is closely coupled with the Target Hardware: Change in Hardware implies Bootloader changes.  However, if implemented correctly, the software architecture for a Bootloader can be effectively designed like any other software stack, with generic components as a Middleware would, leaving a HAL/Hardware Abstraction Layer open to port for different Hardware devices, thus making the job of supporting Bootloader code more efficient. In some cases, Bootloader is a product in its own right, and many companies and consultants have made a business out of it.

STB Bootloaders could be provided by the CA Vendor, Middleware Vendor, or traditionally the STB Manufacturer, because of the code being so low-level and tightly linked to the physical hardware. In the early days when the traditional medium of delivering software to the STB was through the physical tuner cable, the role of the Bootloader was relatively simple. It gets complicated when we move to hybrid STBs, especially with an IP connection, as this opens up another channel for distributing software updates. IP Bootloaders in STBs is relatively new, and hasn't reached a state of maturity as say, the traditional Satellite STB Bootloader. Bootloader testing is extremely technical and complicated, the security, reliability and resiliency testing is really thorough. For example, when testing a Satellite STB, the Bootloader must deal with all kinds of Signal Quality conditions: Removing the signal feed in the middle of the download process, recovering signal, time-outs and general reliability: pulling out the power during the final stages of the write process, etc.

In an IP Bootloader, the same scenarios are considered, but at the TCP/IP networking level. First and foremost, the STB IP Loader must implement a robust IP networking stack. We had a problem in one of our projects where the Bootloader had hard-coded MAC addresses to Zero! We only picked this up late in the project as we entered the final testing. Such basic scenarios should not get past the first stage. Once we fixed that problem, we then hit stability problems: network jitter, packet loss & general robustness w.r.t. IP-connectivity. The Bootloader software component must be ready and verified well in advance of formal production of the hardware. Remember this component is burnt into the STBs at the time of manufacture. It has to be defect-free, any rework post production is going to be next to near impossible, or severely time constraining, manual reworking that will cost the PayTV operator tons of money. You screw up the Bootloader, you fired!

The above experience thus triggered me to instigate a more formal, rigorous process in managing the Bootloader Workstream, which is presented below.

Template for STB Programme Manager: Bootloader Workstream

If your STB project involves a hardware element then it's most likely to impact the Bootloader. It is essential to highlight this dependency in your project plan, even though you might not be directly responsible for the development & delivery of the said platform. If the project was initiated well, or your organisation is quite mature and have been delivering STB projects for many years, then this workstream happens almost like clockwork, automatically in the background, most likely by your company's STB Hardware / Drivers Porting team.

Nonetheless, defining the key roles and assigning owners to the management of the Bootloader delivery will help clarify and add structure to the overall programme:

• Product Owner - If you're a broadcaster or PayTV Operator, you need a good understanding of the feature-set and requirements of this component that meets the needs of your business objectives & operations. If you're a component supplier, you still need someone to own the product specification for this component.

• Technical Owner - Is someone with technical responsibility, generally an architect, who will translate the Product Owner's requirements into detailed technical requirements specification, that Bootloader vendors must implement.

• Component Owner - The party responsible for implementing, testing & delivery of the Bootloader component.

• System Integrator: Low-Level & System High Level. The responsiblity of the assigned STB System Integrator to prove the integration of the said software component.

• Quality Assurance: Security & Customer. Generally CA Vendors provide certificate of approval of the Bootloader, as it has to pass strict security criteria, remember the CA is responsible for generating millions in revenue (this is generally known has Low-Level code testing). The Bootloader therefore has to be secure to prevent intrusion, compromising the integrity of the STB, leading to piracy. The PayTV operator also performs ATP as the Bootloader must adhere to various business scenarios for software download, upgrade, resiliency & recovery modes. Operator testing is generally referred to as High-Level Code or Customer Code testing.

• Project Team. There must be a Project Manager assigned to manage this Workstream. There are generally many dependencies at the System Integration level, so this is best assigned to an SI Project Manager.

Simple Template for Bootloader Workstream

Download the Template!

Some other useful references around Bootloaders
S3's Whitepaper on STB Migration

Tutorial on the inner workings of a STB

Modeling the Climb to Launch for STB Delivery Projects

Continuing with my aim of sharing the tools I use (and come to value) in running my own projects, in this post I'd like to share a simple, yet powerful tool, that can be used for scenario planning a typical Set-Top-Box Product Launch, which I've come to term as "The Climb to Launch".

To understand what I'm about to present in this post, you should at least familiarize yourself with some preliminary info:

• Set-Top-Box (STB) Release Process
• Set-Top-Box Release Campaign Concepts
• Managing Quality in DTV Projects
• Architectural Components in a DTV System
• Typical Programme Structure for DTV Projects

The last mile of any DTV project is the STB Delivery Project. It is widely accepted practice to deliver the components of the system (a.k.a. Backend or Headend), the core infrastructure that provides essential services to the to STB, well in advance of initiating the formal Test/Integration/Release phase of the STB project. Some might argue this is all very well in theory, and in a complicated system when delivering a brand new feature like Video On-Demand, where the entire value-chain is impacted, one cannot avoid the big-bang Integration/Test approach. I maintain this is just a very lazy approach to programme management that should be avoided at all costs! But that's a topic for another post...

A STB Delivery Project Manager has to implement a plan taking into account all the inputs and resulting outputs. Although there is always variability in these projects, the variability reduces as we approach the end of your typical Development phase, and enter the Stabilisation phase of the project. 

We can therefore, quite easily quantify the planning variables involved in the STB Delivery Plan, and use these variables as inputs into producing an overall delivery plan, as I'll explain below.

The STB-Planning Model

Based on the Release Campaign Process outlined in earlier posts, the STB Delivery Plan needs to target specific milestones in the Climb to Launch. I call it the Climb to Launch because it is literally that: A tenuous climb by the entire project team, working through instabilities, performance issues and errors in functionality and user experience, that must all be fixed in order to Launch. The resulting Excel Tool implementing my model produces a Gantt View that takes the shape of Steps leading up to Launch Milestone, hence I've coined the term "Climb to Launch":

Output from Planning Model showing "The Climb to Launch"

Key Planning Variables for the STB Delivery Manager

Again, I'm not advocating that STB Delivery PM is easy and can be solved by using a simplified modeling tool in Excel! A STB PM adds value by having a keen appreciation for the various systems impacted, development & integration processes and most importantly building effective relationships with various teams and stakeholders. It is also important the Delivery PM is well equipped with relevant domain knowledge, can hold technical conversations with ease, as well as have the ability to prioritize issues, despite ambiguity, conflicts & uncertainty.

Planning is no less an important activity for a Delivery PM or even Programme Manager, but spending too much time maintaining a plan on paper, instead of active involvement in the shaping, driving and maintaining of focus - is another recipe for disaster. Having said that, at the high level though, essentially all STB projects follow typical patterns and involve the following key planning milestones or variables:

• Availability of the STB Target Hardware to the rest of the Core Development team
• There is only so much a development team can do on a typical reference development platform
• The sooner the target hardware is made available, the sooner SI can start stabilization
• Software components of STB must also be available, i.e. fully development, tested & certified - this is especially true for the STB Loader, less so for accompanying Drivers

• Availability of Target Hardware Drivers
• This is the foundation of the STB platform. It is generally expected practice that Drivers are delivered, as far as possible, independent from related Middleware components. As long as the interface specification is solid and stable, this development can happen in a parallel stream.
• STB SI team must verify and accept the Drivers are fit-for-purpose and can sustain ongoing Development & Integration

• STB User Interface Application Component Development Complete
• Applicable if there's any new EPG development happening, impacts STB SI timelines

• STB Middleware Component Development Complete
• Applicable if there's any new Middleware development happening, impacts STB SI timelines

• Number of Cycles for STB SI to Stabilize and recommend suitable build on Target Hardware
• Assuming your SI team operate in fixed time-boxed iterations, this variable is a useful timeline multiplier

• Headend Components Development Complete
• Generally, the real STB stabilization phase can only begin once a suitable Headend System is in place that supports the majority of use cases required from the STB product

• End-to-End SI Deployment of Headend to Live
• Milestone that is useful for tracking - useful for Programme Management, informative for STB Delivery PM

• Number of days to time box STB SI Bug Fix / Release Cycle
• Length of SI release / bug fix cycles impact the timeline

• Number of days STB SI require to create a full stack build for further testing
• Necessary and useful to capture so that you don't underestimate the work involved by SI team in the initial bring up of the Software Stack. This doesn't happen by magic, and consumes time. 

• Number of Cycles to reserve for reaching First Major Milestone (Functionally Complete)
• For initial planning, and based on past projects - this is a variable that determines how long the initial stabilization phase is expected to take to hit the first real milestone of being fully Functionally Complete and ready for Closed-User-Group Testing

• Number of Cycles to reserve for reaching subsequent milestones, leading up to Wider Field Trials
• As above, similar penciling-in a slot to reserve for External Home User Testing

• Number of Cycles to allocate to Closed-User-Group (CUG) Testing
• As above - how long do we run the CUG stage for?

• Number of Cycles to allocate to Wider Field Trials (External Testing)
• As above - how long do we run the External Home Testing for?

• Earliest Indication of Launch Build Candidate
• Based on all the above, what then becomes the outlook for Product Launch?

The Modeling Tool - Example Use

Based on the above planning variables, the PM can model different scenarios. Typically one caters for Best Case, Worst Case and Most Likely scenarios. There is also the Three-Point Estimate technique that some PMs use to better quantify the estimates. I use this technique as guidance, but generally don't go down into the detail of mapping the probabilities of the different outcomes.

Suppose you were given a Programme to manage (but wasn't involved from the beginning) and the project itself has changed along the way to the latest Scenario. It's January 2010, almost a quarter way through the project, the business decides to change its mind: Brand new STB Hardware, New User Experience EPG, new Middleware and also new Headend. You're told that you need to deliver the project in 10 months, or else...

Clearly if you've been involved in STB projects as long as I have, you'll know immediately this project is a pipe dream. How do you report back to the Stakeholders, in a diplomatic way, that all the Stakeholders must be smoking something, or in La-La land, that they've never ran a Product Launch from the ground up on this scale before, and without any planning or analysis - you reject the deadline outright?? No matter how good your gut is, or hunch-base, you won't be taken seriously, unless you have some proof you've applied your mind to the scenarios, and things just don't add up.

But you don't have the luxury of time on your side to get into a detailed planning / analysis phase, with work breakdown estimations, etc. You need to summarize the plan at a high level, working through the various scenarios - this is where a tool comes in handy.

Below is a snapshot from the tool:

The Modeling tool

• Step 1: Input the dates / values for variables for the Tasks/Milestones offering the three Estimates: Best Case, Most Likely, Worst Case
• Step 2: Click on one of the Buttons to produce the Resulting Plan
• Step 3: Select the Variables you want to Fine Tune (using the + / - buttons)
• Step4: Repeat / Settle on an Acceptable View to Share with your Stakeholders

In the example project, you can work backwards with your deadline, adjust all the variables such that the plan meets the deadline - and the review the reality of achieving the plan with your stakeholders. You then sit with the primary stakeholders, along with the respective Project Managers to work through estimates that are more realistic. Use that as the baseline plan going forward.

The tool can also be used to summarize the key milestones, by automatically producing a milestones table:

Milestones Table Automatically Generated

Download the Tool Now!

Click on the above link to download a copy of the tool, please ensure you have Excel / Office 2010 or higher installed. As I use a Mac, the tool is best viewed on a Mac, preferably a 32 inch HD display. I will "port to Windows" when I have the time. Remember to enable Macros.

Concluding Remarks

This is a tool I keep in my own PM Toolbox. I use it as guidance to not only plan any new STB project, but also as a way of communicating not only to Stakeholders, but the entire Project Team, of the various scenarios and milestones we need to keep track of in achieving a successful Product Launch, on time or to realistically meet expectations. I have really made this into a generic tool only this year (2012), but again, the concepts and milestones are indeed common and applicable to any STB delivery project. I am hoping this could help some of the new PMs entering the DTV space, improving their ramp-up times.

Lastly, if you do take time to download and experiment with the tool, or even start using this in your own work, please get in touch, offering feedback as appropriate!

ORITs: One Roof Integration Teams for DTV/STB Projects

In my previous post I introduced the concept of Release Campaigns that are a series of cycles executed to reach stability & maturity of the STB Product, which largely happens during the Integration / Stabilisation phase of the project's life cycle.

During these campaigns it is expected to uncover a number of bugs or defects, especially during the first release campaign, where the various system components are actually being stressed for the first time, end-to-end. Expect a reasonable avalanche of defects reported against STB functionality, stability & performance areas. The team that faces the brunt of these defects is usually the STB SI team, who need to investigate, reproduce the problems, and characterise the defect assigning to a relevant vendor to fix. This process is known as "triage" in some circles.

How does the project team manage dealing with these defects? How do we control the quality of the system ensuring the focus is maintained, that we're not spinning our wheels? How do we mitigate the project from further slippage, removing the burden on the SI team? How do we ensure we focus on the areas that are the pain points, the core features and functionality that add value, and affect the bottom-line: customer experience?

Even though the project is likely to have defined clear guidelines for acceptance criteria & defect severity & priorities as discussed in this post, the project will still have to be directed in a manner to maintain a sense of urgency to get the burning issues fixed.

Two areas are almost certain to happen on every STB/DTV delivery project:

• Business / Product owners have their own view of describing the product, as a separate list of feature areas, despite what is defined in all the product documentation
• STB SI team will become the bottleneck and on the critical path if the standard processes are maintained - the project will slip, unless some other practical solution is found

Enter the Hit-Squad or ORIT Teams

So we set-up dedicated teams to address key functional areas that cause us pain. Teams are focused on particular areas, with the sole remit of clearing away all problems. Typical areas that cause pain in a STB PVR product is shown below:

Common Areas to Focus Early during Stabilisation Phase of STB PVR

STB SI Team Bandwidth is Limited

The idea behind the one roof / hit squad is to get everyone in the same room, forming one team, focused and dedicated to closing down issues. Typically you have vendors supplying different components of the software stack, with their own processes & priorities - to expedite issues much faster, removing delays that come from communications, ping-pong of emails and phone calls - it's best to get the engineers on-site and deal with the issues face-to-face. This sounds like a no-brainer to those from the Agile or XP camp - but in reality some companies can be pretty rigid in their support service agreements. Hence it is critical this is driven quite hard by the customer, and pains should be taken to agree on the concept that projects will call for people (resources) to be available on-site during Integration/Stabilisation or even final Acceptance Testing.

Advantages of ORITs

• ORITS definitely add focus and bring a sense of urgency to the project, across the board. It gets the attention from senior management, and commitment from teams to resolving the issues. In essence, we have the full participation of the vendors.
• Communication-paths are reduced significantly by being under one-roof or in the same room. This is often time consuming, and generally is impacted by timezones and vendors being geographically dispersed.
• Ensures the right level of support is provided - i.e. all required technical experts are available to support, and easily accessible. You have their full attention, no distractions.
• Improved turnaround time for fixing issues - The ability to fix code on-site, in real time, providing engineering builds adds tremendous value and cuts a lot of the red tape associated with vendors build/release processes.
• Customer is kept happy - all stakeholders have confidence that there is focus, attention & drive. Having a dedicated owner for each area helps build confidence, maintains pace and momentum. Of course, interventions can be applied much sooner than later - no delay in management decisions.

Some Challenges with ORITs

• Almost certain to disrupt existing teams and structures. People will have to be seconded to other teams, virtual - people will generally be interacting with each other for the fist time, which means that these teams might not initially gel well.
• Requires focus, participation and attention from vendors. Generally vendors are hard pressed to support multiple project deliveries, taking out a key engineer and dedicated to one customer for several weeks, stresses the vendors other commitments.
• Engineers allocated to ORITs need to be sufficiently skilled and competent. We are looking not only for senior, hard-core technical experts but people who can deal with stress and pressures from senior management, as well as communication & reporting must be excellent. Very hard to find such calibre people.
• Logistics challenges: The premises for the one roof sessions must support the physical & personal needs of engineers (Equipment setup, kitchen, personal workspace, etc).

Concept of an ORIT Owner is Imperative

For ORITs to return value and actually improve the outcome of the project, you have to assign a strong technical leader overall to manage the various ORITs holistically, as well as strong technical leads per ORIT area, a.k.a. Owners.

• An Owner is essentially responsible & accountable for ensuring all critical issues for a particular area (e.g. Recordings) are resolved to closure, proven on-site, and tracked through completion in a formal release, all timely.
• This means not only investigating & triaging, but also working closely with - and driving the component vendor engineers & teams for fixes or patches including committing to target fix dates for final implementation. If progress is not going as expected, escalate to senior management as appropriate.
• Requires sound technical knowledge, domain experience & appreciation for problems, ideally having worked or experienced previous issues on past projects of similar nature.
• An owner has freedom to prioritise issues depending on severity without having to wait for approval from Product Owner
• An owner also takes responsibility for leading the team assigned, dealing with people as well as performance related issues

Does it work?

Like all other topics I've shared in my PM Toolbox, I wouldn't be sharing stuff that is fluffy and woolly - not implemented or experienced first-hand by me, directly. So, yes - ORITs do work, if managed correctly. It is no easy feat, requires mindsets & personalities that can maintain a steady pace of rigour, fortitude and relentless determination to get to the root cause of problems - seeking out the best practical solutions in the time allowing. I have both managed ORIT teams myself, observed other projects from afar, and been an engineer on-the-ground as Hit Squad member. It can work well, or can be a complete failure. There needs to be a unifying voice from the top, harmony across the entire project team, and complete focus and attention... It is not an easy or straightforward intervention...

A little more detail behind the mechanics

In the attached slide pack, I try to summarise the key points of this process, including notes on the ideal team sizes, reporting expectations including templates. 

Template for Set-Top-Box Release Campaigns

In this post, I'd like to share a concept I've coined as "STB SI Release Campaigns: The Climb to Launch" which is something a STB SI (Systems Integration) Project Manager, Development Manager or Delivery Manager can use to manage a process of stabilizing the product leading up to final Launch / Deployment campaign.

No matter what flavour your DTV project is, if the STB (Set-Top-Box) is impacted, either it's a new hardware, or new software, or existing hardware but brand new features, the last mile of any DTV launch product is the STB testing, acceptance and deployment. In general, it is widely accepted practice that the building blocks and fundamental backend or headend systems required to drive the STB end user experience, are in place well in advance of the STB. 

Sometimes this isn't always possible, especially when changes impact across the entire end-to-end system. In such cases, some prefer to do big-bang integration testing; however I much favor and prefer component-based testing using test harnesses and simulation; leaving the rest to integration testing. This is a topic for another post altogether...

So the last mile of the majority of DTV deployments is the STB system testing, including user experience & final acceptance testing. The team that are usually the custodians of the STB delivery is the STB System Integration team (STBSI). It is this team that carries the burden of producing a stable build, ironing out all the critical issues, and ultimately produce release candidates for recommending to launch. 

Typical Milestones in a STB Delivery Project
I've written previously about clearly defining objectives & goals to your STB project, so much so that it can be quantified & qualified through a measured process. To recap, a STB project will typically include the following Major Milestones:

1. Start of Closed-User-Group (CUG) Field Trials
• Headend is available in advance and operational on the live broadcast
• STB SI have created a build that is functionally complete to all intents & purpose (FC)
• We are in a position to release this build to enter formalized acceptance testing (ATP QA)
• We are ready to start the path to Release Candidates, the climb to Launch - i.e. iterative cycles that will be repeated to reach final Product Launch
2. Start Wider Field Trials
• Signifies the build is stable, nearing completion and is ready for external audience
• STB passes all certifications (HDMI, CA, Macrovision, etc.)
• STB Hardware passes all hardware testing (is the hardware fit-for-purpose, Consumer Acceptance, Safe, Green, etc.)
• We are getting closer to producing a final Production Build
3. Produce Launch Build
• Field Trials drawing to a close
• STB SI issues are clearing away, last few hurdles to pass through, but not critically blocking launch
• Finalised Release Candidate almost in hand (RC3)
4. Clean Run
• STB SI Produces Launch Build
• Final ATP QA expected to complete with no Showstoppers
• Final build sent out to Field Trials to verify critical issues
• Soft download done to select group of subscribers
• No critical issues found from the soft launch
5. Deploy
• Final Deployment build created by STB SI and provided to Launch Delivery Team
• Image is broadcast for software upgrade or launch of decoder STB to market (go public in retail stores)
• Process continues for some time, initially about 2 months
• Next release is being planned (new features & bug fixes)

For each of the above milestones, the project will have to define entry & exit criteria. See previous post or the attached presentation. This might seem simple and logical, but generally STB projects are often complicated by lots of legacy rules, history of business & project decisions, and often involve more than one target hardware STB/Decoder platform. In cases where there are more than one STB involved, typically one chooses a lead STB for launch, with a quick follower - it depends on business objectives of course.

Introducing the Release Campaign Concept
I use this concept to bring structure to a STB Delivery Project. Having clearly defined milestones, as well as realistic, clear & unambiguous goals/objectives for each milestones, naturally allows for a simple process that can be executed repeatedly until all the objectives are met. The key points:

• Structured, Repeatable & Easily Measurable
• Unambiguous & clear focus on defined Launch Criteria
• Sequence of repeatable activities to execute almost automatically to produce Release Candidate (RC) builds
• Planned up-front: Release Campaigns can target a specific milestone, and aim for a fixed duration: Example: I would like to achieve a CUG FC build within Eight (8) weeks of reaching code complete.
• The release cycle, production of the RC build for each milestone is Incremental
• Current practice is to use a cycle of 10 working days (2 work-weeks)
• Involves the buy-in & full participation of the whole team
• Project & Programme Managers
• Product Owners
• STB SI Team
• Component Vendors (Middleware, Drivers, Application)
• ATP QA Team
• SI QA Team
• Field Trials & Operational Support team

Associated with this is a simple process that can be used by Delivery Project Managers to plan, based on a set of available macro variables:

• Duration for SI to produce an incremental build (I use 10 days)
• Amount of time to allow vendors to fix Showstopper defects in pre-candidate builds (can vary between 1-10 days depending on service level agreements)
• Duration of the Release Campaign (How many SI cycles of build increments to allocate for specific milestone?)

Below is an illustration of the core variables & typical timeline milestones:

Overview of Time Milestones in a Release Campaign (RC)

The process is repeatable, one Release Campaign flowing to another, with feedback. It is a continuous process of defect fixing, incremental builds & continuous verification leading up to the production of the next candidate build for the next release campaign, as illustrated by the pictures below:

Flow of Release Campaigns

Example Flow from One Release Campaign to Another

As the quality of the builds improve over time (this is expected), the length of campaigns is expected to reduce over time. This assumes of course the project is in real delivery mode, that all feature development is complete (or the project implements a very strict stability regime) and that the fundamental foundations of the stack has reached an acceptable maturity point.

Release Campaign Model - Tool for the Project Manager
What the model allows the Delivery Project Manager then is to play with a few scenarios, especially the length of the release campaigns, adjusting for bug fixing and stabilization problems. The PM can then use a tool, that based on fine tuning a few variables, can come pretty close to producing realistic plans. The fine-tuning of course can be used to indicate and prove to your stakeholders the project still has a long way to go (which in general is quite true for almost all STB projects: they are usually too ambitious and poorly planned from inception):

• #Days STBSI need to create & sanitize a build. The STB SI team usually integrate a few components from third-parties, put a stack together and instrument basic sanity, smoke & reliability testing. This usually takes the order of 3-5 days depending on the complexity of the stack, as well as maturity & competence of the SI engineers.

• #SI cycles required to stabilize core software on final hardware. Generally there is some time required to get a software stack stable enough on the target hardware (regardless of this being a mature software stack on a previous hardware being ported to a new hardware; or a brand new stack on first time hardware)

• #Cycles to allow vendors to meet FC build CUG Criteria. How much time to allocate for the initial Release Campaign to produce a Functionally Complete build, according to the agreed acceptance criteria. This is generally the first time all components are delivered to STBSI as ready for functional integration, and is expected to take the most amount of time.

• #Cycles from Release Campaign X -> Release Campaign Y. Repeat until happy - the amount of RC attempts will of course vary according to the nature of your project. I tend to go with three major campaigns, you can have as many as you like.

• #Cycles to run & manage Field Trial Testing. As above, typically the field trial testing happens in parallel, and should not be on the critical path. The business might decide to let these test phases run for much longer though, thus ultimately impacting your launch date.

If you want more detail, please refer to the accompanying presentation.

You're doing it wrong - with Agile there is no need for a separate Stabilization Phase!

There are a lot of people jumping on the Agile/Scrum bandwagon, advocating its use in every software or systems delivery project. Whilst I'm quite a firm support of Agile/Scrum, I have to admit that adopting Agile/Scrum in keeping with its true essence, is next to impossible in the real world, especially when typical Digital TV Systems Projects involve players from a multitude of vendors, with the PayTV Operator being the end-customer, and the real Product Owner. Bearing in mind also, that these independent software vendors have their very own products to support, delivering to multiple customers - to implement a truly STB agile project requires a significant investment on the part of the Pay TV operator, and exceptional buy-in from the vendors; so realistically it is practically undoable -- unless the Pay TV operator either owns the project in-house, or has a Project Charter or Contract in place that clearly stipulates the main leader, driver, and overall planning is under the ownership of PayTV operator...more on this debate later...

So what happens in practice though, is that software vendors & systems integrators work in relative isolation, and are free to adopt whatever approach is required.  The commonly accepted best practice of STB Software Projects, if managed properly, should be a short-lived Integration, Stabilization & Bug-Fixing phase - solely focused on stabilizing the product leading up to launch. We are not saying that Stabilization phase appears toward the end-phase of the project and that component vendors are unable to release stable components prior to this phase: No, not at all. In reality, a STB project depends on a number of pieces of a the puzzle fitting together, and it's often that the first time this really happens is closer to last mile of Components-Integration (a.k.a. Systems Integration & Stabilization).

In complex projects that impact the end-to-end DTV value chain, where the Headend & Full STB Stack is brand-new, or fundamentally changed, then it's really difficult to avoid a separate stabilization phase. Components are expected to be stable & performant as part of their independent component testing & release processes, but the system ultimately only begins to be stressed when all components come together.

The typical, best practice of Systems Integration Management & Delivery is shown below:

Industry Best Practice of STB SI (Courtesy/Permission granted by S3)

The above model follows a structured approach to a STB delivery project, ideally SI (System Integration) is kicked off from stable foundations, then becomes a series of cycles that is executed until the system is fir for further Field Trials / Certification. Essentially these SI cycles are the Integration & Stabilization phase of the project. What also happens in reality is that Certifications/Field trials kick-off at a suitable SI cycle, in parallel and thus is not sequential - i.e. not Waterfall.

As much as I'd like to see a STB project implemented from start-to-finish using Agile/Scrum, I've been involved in many DTV Systems Projects to know it is extremely difficult and challenging, incurs a huge management & administration overhead, is costly & requires mammoth coordination, but not impossible! I've seen some projects come close to doing end-to-end development & integration, incrementally, adopting as much Agile principles as possible, but not really wearing the badge of pukka-Agile (see earlier posts).

In the real world, benchmarking and signing off a system that's going to generate millions of dollars in revenue, one cannot avoid controlling the acceptance of the product by implementing a fairly strict process for stabilization, testing & final acceptance.

Conclusion
I have shared a simple model for managing & planning STB software release schedules, that a STB Delivery Project Manager can use. This is not a theoretical model based on zero real world experience or case studies. I have seen this model used in the past, and though not publicly available, I've seen many good PMs use this technique naturally. I am using this model to manage some of the projects I'm currently running. I believe it is probably the first time somebody has ventured to share this model, and also the first time to provide a tool for free, that can be used to assist with planning & modeling release scenarios. I have worked with, and learnt from, brilliant managers who apply this planning model instinctively without depending on tools, they have excellent hunch-bases borne from delivering many real world projects - I'm grateful to have worked with such giants (BK, DD, NT, ST, JC, MK)...

Some hardcore engineers might retort by remarking the classic "How long is a piece of string?" argument. Sorry, us poor Project Managers need to have some way of measuring & predicting the output of a project, and therefore must rely not only on analytics (like Defect Trends/Prediction) but also tools based on insight, intuition, wisdom and lessons learnt from past projects. This tool is borne from this experience - I am confident in its use & value it could bring to your project. The recurring, iterative cycles of the model makes it easier to find the answer to the length of the string problem - we have to provide an expectation based on some reasoning, disclosing all the risks & uncertainties upfront of course...

If you found this info useful or would like to learn more, or bounce ideas, or even share your own experiences from your own PM toolbox, do drop me a line or two! :-)

Download the Free Planning Excel Tool!
If you've read this far, that's great! In my next post I'll share a powerful planning tool that is based on this release campaign model. You can use this tool to model Best Case, Most Likely (Realistic), Worst Case and the resulting 3-Point Estimate - to help you not only plan, but visualize the Climb to Launch, offering you many options to tweak your planning, stay tuned...