System Integration is King - Release Manager, Build Meister-Gatekeeper

I spent a lot of energy in the last week trying to (and succeeded in) convincing some senior managers about processes, that in my humble opinion, are tried-and-tested, well-established norms in the management of software: the art of Software Release Management - taking control of component releases, especially w.r.t. approving or rejecting bug fixes. It has been good practice in terms of establishing myself as a consultant-expert on all things related to Set-Top-Box Software Development & Integration. I spend a lot of my time promoting best practices and trying to win & influence people over, something I have to do to maintain focus and direction of the overall programme plan...

The point I want to make is that, in a STB delivery project, the System Integration (SI) team is really the GateKeeper, with full authority and accountability for producing a stable, functional release. This means that SI have every right to manage component releases coming in, especially with reviewing & approving bug-fixes. If SI are not happy with a component release, for example, the component team may have fixed more bugs than was required, or fixed far fewer bugs than was requested, or made any other changes that wasn't specifically requested by the SI team, then SI can quite easily reject the component's release.

I have written in the past on the following topics:

• Template to manage STBSI Release Campaigns
• Release Schedule template to manage Component Vendors
• Modeling the Climb to launch in STB Projects
• Process for fast tracking issues, reduce the backlog on SI
• Tracking Defects using Predictive Analysis
• Effective Defect Management Processes required in STB Delivery Projects

All of these areas become the focus of a STB project as we get closer and closer to launch mode. Recapping the template that most STB projects follow:

• Develop / Integrate Features >>> Functionally Complete >>> Closed User Group >>> Wider Field Trials >>> Launch Candidate 1 >>> ... >>> Launch Candidate X >>> Clean Run >>> Deployment Build >>> Launch

As the system matures to reach each of these milestones, the defects are expected to get less and less. Obviously no product gets launched with zero defects! All projects can do is ensure as best as they can that almost all high impact, high severity defects (a.k.a. Showstoppers or P0s) are dealt with to the best of their ability; and are prepared for the inevitable influx of new Showstoppers from in-field, real world usage of the product - it is the reality, unavoidable. Very rarely do products launch having met all original quality engineering requirements, after all, this is an entertainment device, not a life critical device - so we can relax a little on quality objectives! :-)

So how should you control releases leading up to launch?? It's simple really, not complicated, and really not rocket science...

The theme is essentially: Lock-down, create a bankable release, and incremental introduce fixes that improves both stability & functionality such that regressions are contained, and ultimately achieve fit-for-purpose release. The process is evolutionary and iterative -- but requires some investment from management & project administration, and quite an onus on SI to maintain the integrity of the releases.

At the early stages of development/integration, the project can afford to have open gates and accept as many defect fixes as possible. Running up to launch, say, if you are 2-3 months behind launch date, then the controls have to be tightened. A project cannot just accept on average over hundred (100) P0s Showstoppers from random areas of the product: this creates a lot of uncertainty, churn and misdirection. It makes it really difficult to zone in on a unit of stable functionality, core stability or solid features. Without a repeatable and automatic process to manage, track and control regression; accepting large amount of defects-fixed-per build is just plainly unsustainable! Product will never reach stability in the timeframe remaining; launch will not be in sight.

The only way to make launch within the timelines is to really enter "Lock-down" mode. Focus, Focus, Focus. Zoom in one major pain points, and target a realistic delivery plan to attain those targets. Control, Control, Control. Become the master of the defects - control all points of entry!

The defect backlog will have to reviewed daily. Key decision makers accountable for the product launch have to take responsibility for creating time to review and approve the defects that are absolutely critical to launch, and must absolutely be fixed. Priority is determined by the business owners, irrespective of the severity of the defect. Severity is still maintained for traceability back to the original defect submitter, usually QA - such that, in the case of a nasty event post-production, the QA teams have ammunition to use against the decision makers that chose to ignore the severity and lower the priority!

STB projects generally assign a Release Manager or in software development parlance, the classic Build Master takes the role of doing the following:

• Work closely with the Product Manager or Business Owner to agree and understand the blocking features / areas of functionality for Launch. These guys need to be in sync, on the same page, no ambiguity
• Manage & Control defect release schedule from Component vendors on a daily basis:
• On a weekly basis, set the tone, direction and plan of action for the week
• This includes list of defects for each component vendor that must be fixed for the week
• Appoint component build / verification masters that review and approve every component release
• Did you get only the specific defect fixes you requested? If not, push back hard
• If you got fixes you didn't request, reject the release
• Impress on the vendors to release every blocking defect fix on a patch-by-patch basis
• Ensure firm release control & configuration management processes are in place
• It is common practice for software developers to implement a defect fix on a branch
• Developers can support fixes on a patch-by-patch basis
• Developers should have ability to rollback versions as-and-when required
• Work with the vendors on multiple release plans simultaneously
• Fixes blocking launch
• Other fixes that can add value, low risk and ultimately improve launch quality
• Forward releases with post-launch features

Real-Life is Unavoidable!

I was involved in projects where on a daily basis, defects had to be reviewed and prioritized. On a weekly basis we had to sit, discuss and agree as well as debate with the customer what was required for launch, the theme of functionality/stability/performance to deliver each week, etc. We had to review our open defect counts and metrics since we were commercially obligated to deliver on quality targets. As over burdensome as this may sound, too much administration and management overhead, too much micromanagement of the vendors and suppliers, it is a fact-of-project life, it has to be done - and is unavoidable...crunch time.

Giving QA a Voice

When a project enters this phase, you will get an immediate push-back from the various QA teams. STB projects generally have many streams of QA: Component QA, STB QA, CE-QA, Field Trials QA, etc.  It is definitely QA's job to own the following:

• Testing the Product Specifications - Requirements-to-Test coverage map & report
• Exploratory Testing Feedback, including Edge-Cases
• Feedback based on past product testing experiences
• Own the defect logging in terms of Severity of the Issue
• Can recommend a Priority knowing that ultimate Priority assignment is controlled by the business owner

I absolutely support the view that all QA teams have a voice. It is perfectly natural to raise concerns when the project starts changing priorities of defects and changing direction. In light of this, the Business Owner really owns the risk of making such decisions - but QA should be allowed to provide their assessments and state their recommendation for the record. 

Doing this allows QA to remain empowered, as the project does provide support for, and provides a platform for QA to voice their concerns. But a line must be drawn somewhere, and the decision to proceed or stop will be made, taking into account such input. 

QA however, should support and follow the Business Owner in any event; instead of becoming an obstacle to the project.

The Old is really the new New

I am always amazed by how STB development projects, especially if the teams are new, think that the problems they face are new and unique to them! Not really, we are just developing software. A wealth of best practices exist for Release & Defect Management - this challenge is not new, and not specific or unique to STB Software development. Microsoft has had a long standing process for controlling component releases & build management - it was one of the first books to shed light on the subject of Triage/WAR meetings, quite peculiar coming from Microsoft! Open Source projects are also tightly controlled - there is always several levels of build / defect masters. In the run-up to a major release point, all control is assigned to the Release Manager or Build Master...some even call this Release Block as used by Google in their Chromium project. At the end of the day, STB projects will adopt some Release Management Process that suits the needs and culture of the specific project, component vendors need to just fall-in-line, and follow the direction of the lead Systems Integrator...


How I Influenced the Change & got my way in the end...
So it became clear that a project wasn't heading in the right direction - below is an excerpt of an email on how I went about setting the stage for the many debate/discussion sessions we had behind the scenes over two weeks, culminating in a change in strategy:

• Launch Candidate Backlog is NOT clear
  • We need to maintain and track on a daily basis the Launch Candidate (LC) Defect Backlog
  • Currently the goal is too broad to reach zero P0s
  • We need to define the set of defects that must be fixed for Launch Candidate that will drive the opening of Wider field trials
    • I don't expect Wider Field Trials to start with these defects still open

• Firmer control on contents of SI Releases/Build
  • This LC defect backlog must be managed daily – defects will be added / removed depending on severity of new issues
  • We should have a good view of the capacity of the development teams to close defects now
  • We don't accept any other fixes until we bank the launch candidate fixes
    • Too much noise – needs to be controlled
    • Unwarranted fixes will introduce regressions in other areas causing more work, results in lost focus
  • If a component is not assigned a launch candidate blocking defect, then that component can work on other defects on a branch
    • This means that a new component version might only contain just one blocking defect fix
    • The other fixes can follow or merge back to next release once confidence is gained from testing – no regression, etc.
  • SI must strictly control this launch candidate release and not allow unwanted defects in
    • SI team could work on multiple releases simultaneously
      • Launch Candidate builds
      • Other forward builds with other component versions
    • SI should push out frequent releases with launch candidate blockers for testing to Engineering Field Trials
    • Don't wait for end of two week release cycle to get a feel for all LC fixes
    • Get immediate feedback from Dev/SI field trials 
  • This is standard practice for managing software releases:
    • Release Manager / Build Master controls contents of Release
      • Nothing escapes or gets past the Build Master
    • BoM is published with specific set of defects that MUST be fixed
    • BoM is updated if new showstopper arrives and there is time to fix
      • If new Showstopper can't be fixed in time, then build master could reject build depending on severity of new issue
    • Component vendors implement every defect fix on a branch such that it can be easily rolled back if major regression occurs