Pragmatic Set-Top-Box QA Testing - Don't just trust Requirements-to-Test Coverage scripts

This post might be a little edgy - I think it's important to understand alternative perspectives around Set-Top-Box (STB) testing, how it is uniquely different to other forms of IT Testing; to understand the dynamics of STB testing that promotes agility and flexibility, whilst simultaneously understanding the risks associated with pragmatic testing.

The Headline:
DON'T BLINDLY TRUST REQUIREMENTS-TO-TEST COVERAGE MAPS

Last year, I wrote quite an in-depth paper on Effective Defect & Quality Management in typical DTV projects. It covered many topics, and touched briefly on the aspect of Project Metrics reporting. This post expands on the subject of QA Metrics tracking, focusing on how this reporting can help change the direction of the project, and instigate changes in focus to the overall QA efforts, particularly around Set-Top-Box QA testing. I advocate the project will change QA focus as stability is achieved with Requirement-to-Test coverage maps, to include more Exploratory, Risk-based & User-based testing.

Worlds of QA Testing in Digital TV Systems Projects

I have previously written about the Digital TV ecosystem with its complexities and challenges in defining the Architecture & Systems Integration spaces. In this post I will share some thoughts around the QA (Quality Assurance) / Testing problem space, as well as generally accepted strategies for solving these problems.

What really triggered this post (which is centred around E2E QA) was a recent conversation with a QA Manager who in passing commented that he's facing a challenge with the "End-to-End QA" (E2E QA) team, in that it takes up to six weeks to complete a full QA cycle. Now some might think this is expected, as E2E QA is the last mile of QA and should take as long as it takes. My response to this is that it depends...

It depends on which phase your project is: is it still in early development / integration testing - where components are being put together for the first time to test out a feature across the value chain? Or is the project well ahead in its iterations having already executed much of the E2E testing? It also depends on the scope and layers of testing the systems and sub-systems before it gets to E2E Testing. Has a version of the system already been promoted to live? If so, what are the deltas between the already live, deployed system compared to the current system under test?

[--- Side note:

This post is one of many to follow that will encompass some of the following topics:

• Worlds of testing / QA
• Is ATP / E2E testing the longest cycle than other system test cycles - or should it be the shortest?
• High level block diagrams of architecture / interfaces / system integration
• Visualisation of technical skills required for various aspects of testing skills required at various areas of the environment: Scale of technical - heat map
• E2E QA is highly technical and should be considered as part of Integration, not a separate QA function
• Which came first: the chicken or the egg? Headend first, then STB, or do both together - big bang?
• How to solve complex systems integration/test in DTV?
• What is end-to-end testing, what areas should you focus on?
• What is required from a Systems ATP?
• What kind of QA team structure is generally accepted practice? How does Agile fit in with this?
• Can E2E Testing be executed using Agile - iterate on features End-to-End?

Side note --- end ---]