Modeling Software Defect Predictions

In this post I will share yet another tool from my PM Toolbox: A simple method any Software Project Manager can use for predicting the likelihood of project completion, based on a model for defect prediction. This tool only scratches the surface of real software engineering defect management, but has nevertheless proved useful in my experience of managing a large software stack. For instance, this tool was borne from my direct experiences as a Development Owner for a large Software Stack for STB Middleware that comprised of more than eighty (80) software components, where each component was independently owned and subject to specific defect & software code quality requirements. I needed a tool to help me predict when the software was likely to be ready for launch based on the health of the open defects; as well as use it as evidence to motivate to senior management to implement recovery scenarios. It was quite useful (and relatively accurate within acceptable tolerances) as it offered perspective of a reality that more often than not, people underestimate the work required to get defects under control, the need for predicting project completion dates based of defect resolution rates, depending on the maturity of the team is often also misunderstood. 

I created the first instance of this tool back in 2008/2009, the first public version was shared in this post on Effective Defect & Quality Management. Since then, I've upgraded the tool to be more generic, and also significantly cut down on the number of software components. I still use the tool in my ongoing projects, and have recently convinced senior management in my current project to pay heed to defect predictions. 

Risk Management - Generic Risk Register DTV Projects

In this post, I share a generic Risk Register than can be tailored for almost any Digital TV Systems Project, especially projects around Set-Top-Box product development. The register can be used by Program & Project Managers from the customer (Pay TV Operator), vendor (Middleware / EPG / Drivers / CA) or Systems Integrator. It can be tailored to meet the specifics of a project.

You are free to download this tool, use it as a template for your own projects, as long as you retain some attribution to me as the original author. All my work on this site is licenced according to Creative Commons Share-Alike attribution.

Over the past decade of working with Digital TV projects, especially around Set-Top-Box projects, I've come to experience common themes that repeat themselves one project-after-another-after-another. Whilst it is true that no two projects are the same, I do believe that projects will ultimately be exposed to similar risks, issues & opportunities - besides, we can no longer say that Digital TV is a new and emerging field. We've been making Set-Top-Box (STB) hardware & developing STB software for just over two decades now, and as with all manufacturing & development processes, we have probably reached a point where some standard blueprints can be put together, forming templates that can be used to guide such projects going forward.

So it is the aim with this post, to share a Generic Risks Register, that is borne out of my own personal experiences - that I myself use as a template for managing Risk on every new project I embark on.

SI QA Sanity Tests debate...round one

Today, after 19 months of joining the company & about 18 months from getting involved with ongoing projects, we finally decided to have a discussion around the topic of "Sanity testing" since I had a tendency to always comment on why we continued with downstream testing even though "sanity" had failed, but I was always overridden that it was deemed acceptable for the current stage of the project, and since I wasn't directly involved in managing that project, so I let it be - the timing wasn't right yet...

But even today, on a project that I do have direct involvement in as the overall programme manager, having influenced & steered much of the development / integration / test process improvements to bring the project back on track again, there is still some confusion around what "sanity testing" really means. The technical director managing the launch, who's position in the corporate hierarchy is one level above myself (and the QA manager), has maintained a different view of Sanity Testing than I. But it seems I'm not alone in my view, having recently gained the support of the QA Manager as well as the Product Owner - so we decided it was time to meet and discuss this topic in an open forum, to try and reach an understanding, to avoid confusing the downward teams, and establish a strategy for possible improvements going forward...easier said than done!

One of the challenges I face almost daily is that of enabling change across the project team, sticking my nose into Development / Integration / Test departments with the aim of steering them in the right direction, to help increase the chances of the project delivery.  I do this, even though I myself, strictly speaking am positioned within the organisation as a "Programme Manager or Strategic Planner", wearing the hat of Project Management, as if applying PM pressure across the team isn't enough of a thorny subject already, I still go and push for process changes in Dev / Int / QA - am I a sucker for punishment or what?? But I can't help myself really, because my technical background, I've been through all the stages of software engineering as both engineer and manager, and have worked for the best companies in the industry, who, through the years evolved to using best practises -- so when I see things being done that deviates from what I consider the norm in the industry, I just can't help myself, but to intervene and provide recommendations because I can help short-circuit the learning curve and help the team avoid repeating expensive mistakes!

And so with the topic of QA Sanity, we've never respected the term, and continue, despite failing sanity, to proceed with testing an unstable product, in the hope of weeding out more failures:

• What does Sanity mean? 
• What do we do when Sanity fails?
• Do we forsake quality until later in the project?

Types of Digital TV Projects, Decision Factors & Typical Duration

In this post I will discuss the various types of Digital TV projects that I've come across over the last twelve (12) years. Although the DTV system is a complex one, and in theory, there are many different permutations and combinations of possible projects, we should not forget that on the part of the Pay TV Operator, it is quite an expensive affair, and therefore these guys are quite resistant to change. Initial costs in setting up the broadcast network must be made up in as short a time as possible (ROI in less than 5 years maybe), subscriber growth must be on the increasing trend, competitive threats kept at bay.

Depending on the market dynamics, some PayTV operators (especially Europe & North America) have strong competition, the market is quite open and those regions usually have a strong regulatory body that promotes a free market, competitiveness and most important of all, the right of consumers to choose. Whilst in other markets such as Africa and Middle East, the PayTV operators that were first to get in when the time was right, are usually market leaders, the dominant player and lack any strong competition.

Regulatory bodies in this region are either non-existent or immature in its abilities in governance and promoting a free market based on consumer choice (South Africa is a good example: not generally consumer driven, although recently this has seen a change with the introduction of the Consumer Protection Act but is still far from having an SA-equivalent of the UK's Ofcom for example ICASA doesn't come close). In the Asia/Pacific markets, there is a growing number of PayTV operators, where competition is rife, time-to-market more important over say, user experience or innovative features (I've seen this with my own eyes, for example - in India, there are some operators that have got products that look like they're built on eighties technology)...

I've set myself quite an ambitious task (as usual). The nature of DTV projects vary, depending on which side of the fence you're sitting on. Of course, all projects that generate revenue are driven by the PayTV operators themselves. If you're a software vendor, say Middleware or EPG Application, you will have a mix of customer delivery projects and internal R&D projects. Both projects have their own challenges, involve various factors that influence decision-making; and both can largely be estimated using general guidelines (that are usually based on performance of past projects).

In this post, I'll discuss projects from the PayTV Operator's point-of-view, in terms of typical use cases. In a future post I will touch on typical R&D projects and what drives primarily software vendors operating in the DTV product space.

Breakdown:

• Overall Goal
• High-Level Breakdown of Kinds of DTV Projects
• Factors that Influence DTV Projects
• Typical DTV Project Scenarios
• Related Reading Material

Process for Fast-Tracking Issues - The Daily Scrub

In this post I'd like to share a simple, yet effective process to managing the defect discovery phase, typical experienced during the early stabilisation phase of launching a STB product. The process does not only apply to STB projects, it can be used for Headend projects; or even general Software Projects that involve a number of components requiring system integration. Since the bulk of my experience comes from STB projects, the information presented here thus has a bias towards STB projects.

The material presented here is not that ground breaking, really. If you've been in the business of delivery DTV projects for a long time, then you probably have a tried and tested way of managing the system integration processes. However, if this is the first time you're involved in system integration, you are either a PayTV Operator taking more ownership of SI (System Integration), or you might be a professional services outfit and have won your first big project - If you fall into either of these categories, then I have no doubt that you'll find this useful, at best, trigger to review your currently planned strategy.

As an aspiring independent consultant, the Scrub Process is yet another addition to my PM Toolbox. Recently I had to present this process to a project team, who's processes were a bit old-fashioned and rather rigid, and slow. I needed a medium to express the concepts of gaining more efficiencies out of the SI Process, thus put together this presentation to communicate the idea, which I'll only briefly touch on here. For more details, please download the presentation.

Set-Top-Box (STB) BootLoader Management Template

The Set-Top-Box Bootloader is a crucial piece of software (actually firmware that is generally burnt into (ROM) Read Only Memory, usually One Time Programmable (OTP)) that performs not only the vital job of the classic bootstrap (Power on, boot, launch application) for the device but also is fundamental to applying software updates to the system.  Like in PCs, the bootstrap is the initial code that is loaded that generally performs a self-test to check if the device configuration is OK, and looks for an image to load that essentially boots the device into operation, ultimately loading the EPG application.

The Bootloader is generally implemented as very low-level code, with limited set of drivers to enact the basic hardware functionality, with the focus of occupying as small a footprint as possible. There are different types of Bootloaders implemented in STBs, generally known as "n-stage" Loaders. Typically you find a Singe-Stage Bootloader and Two-Stage Bootloaders. A Single-Stage Bootloader entails an all-encompassing loader, self-sufficient, not requiring the help of another bootstrap to continue the boot process. Whereas, a 2-Stage Bootloader, as the name suggests, includes a secondary stage, that kicks of a higher-level bootstrap to facilitate the loading of subsequent system components.

As a STB Project manager, there will be a occasions where the Bootloader requires special attention. This is especially true for projects involving brand-new hardware, different Middleware or Drivers. Generally though, the Bootloader is closely coupled with the Target Hardware: Change in Hardware implies Bootloader changes.  However, if implemented correctly, the software architecture for a Bootloader can be effectively designed like any other software stack, with generic components as a Middleware would, leaving a HAL/Hardware Abstraction Layer open to port for different Hardware devices, thus making the job of supporting Bootloader code more efficient. In some cases, Bootloader is a product in its own right, and many companies and consultants have made a business out of it.

STB Bootloaders could be provided by the CA Vendor, Middleware Vendor, or traditionally the STB Manufacturer, because of the code being so low-level and tightly linked to the physical hardware. In the early days when the traditional medium of delivering software to the STB was through the physical tuner cable, the role of the Bootloader was relatively simple. It gets complicated when we move to hybrid STBs, especially with an IP connection, as this opens up another channel for distributing software updates. IP Bootloaders in STBs is relatively new, and hasn't reached a state of maturity as say, the traditional Satellite STB Bootloader. Bootloader testing is extremely technical and complicated, the security, reliability and resiliency testing is really thorough. For example, when testing a Satellite STB, the Bootloader must deal with all kinds of Signal Quality conditions: Removing the signal feed in the middle of the download process, recovering signal, time-outs and general reliability: pulling out the power during the final stages of the write process, etc.

In an IP Bootloader, the same scenarios are considered, but at the TCP/IP networking level. First and foremost, the STB IP Loader must implement a robust IP networking stack. We had a problem in one of our projects where the Bootloader had hard-coded MAC addresses to Zero! We only picked this up late in the project as we entered the final testing. Such basic scenarios should not get past the first stage. Once we fixed that problem, we then hit stability problems: network jitter, packet loss & general robustness w.r.t. IP-connectivity. The Bootloader software component must be ready and verified well in advance of formal production of the hardware. Remember this component is burnt into the STBs at the time of manufacture. It has to be defect-free, any rework post production is going to be next to near impossible, or severely time constraining, manual reworking that will cost the PayTV operator tons of money. You screw up the Bootloader, you fired!

The above experience thus triggered me to instigate a more formal, rigorous process in managing the Bootloader Workstream, which is presented below.

Template for STB Programme Manager: Bootloader Workstream

If your STB project involves a hardware element then it's most likely to impact the Bootloader. It is essential to highlight this dependency in your project plan, even though you might not be directly responsible for the development & delivery of the said platform. If the project was initiated well, or your organisation is quite mature and have been delivering STB projects for many years, then this workstream happens almost like clockwork, automatically in the background, most likely by your company's STB Hardware / Drivers Porting team.

Nonetheless, defining the key roles and assigning owners to the management of the Bootloader delivery will help clarify and add structure to the overall programme:

• Product Owner - If you're a broadcaster or PayTV Operator, you need a good understanding of the feature-set and requirements of this component that meets the needs of your business objectives & operations. If you're a component supplier, you still need someone to own the product specification for this component.

• Technical Owner - Is someone with technical responsibility, generally an architect, who will translate the Product Owner's requirements into detailed technical requirements specification, that Bootloader vendors must implement.

• Component Owner - The party responsible for implementing, testing & delivery of the Bootloader component.

• System Integrator: Low-Level & System High Level. The responsiblity of the assigned STB System Integrator to prove the integration of the said software component.

• Quality Assurance: Security & Customer. Generally CA Vendors provide certificate of approval of the Bootloader, as it has to pass strict security criteria, remember the CA is responsible for generating millions in revenue (this is generally known has Low-Level code testing). The Bootloader therefore has to be secure to prevent intrusion, compromising the integrity of the STB, leading to piracy. The PayTV operator also performs ATP as the Bootloader must adhere to various business scenarios for software download, upgrade, resiliency & recovery modes. Operator testing is generally referred to as High-Level Code or Customer Code testing.

• Project Team. There must be a Project Manager assigned to manage this Workstream. There are generally many dependencies at the System Integration level, so this is best assigned to an SI Project Manager.

Simple Template for Bootloader Workstream

Download the Template!

Some other useful references around Bootloaders
S3's Whitepaper on STB Migration

Tutorial on the inner workings of a STB