Appreciating the Role of System Integration

Last year I started my series of posts around the different aspects of DTV system projects. Starting out by describing the a typical end-to-end system architecture and the importance of clearly identifying the roles and responsibilities of the architect team (see post: Architect Roles), followed by a write-up on considerations for setting up a programme organisation structure (see post: Managing a DTV Programme) and more recently, a brief overview of auditing the STB System Integration process (see post: SI/Dev Process Audits).

I still have a few more topics in the pipeline, however the aim of this post is to highlight the worlds of System Integration, as this can often be overlooked and open to interpretation, introduces dangerous assumptions and automatically puts the programme at risk.  With a large-scale DTV deployment, the programme must at the outset, clearly define the boundaries of system testing, and establish clear roles for the system integration effort.

This post is especially relevant to TV Operators deciding to become more involved with technical ownership of their value chain, as previously highlighted, more and more companies are taking the plunge into doing their own Software Development, and System Integration. This post is another humble attempt at raising awareness that such operators should proceed with caution, ensuring the full extent and gravity of the undertaking is understood, because System Integration is a significant investment in people (highly experienced technical experts required), equipment (investment in costly infrastructure) and time (starting from scratch, building this competency from the ground up is at least a 5 year maturity period).

Recall the end-to-end architecture originally present a few posts back:

As highlighted in previous posts, the world of Digital TV consists of Broadcasters / PayTV operators employing one of more component vendors or Technology Service Providers offering specialist services of Software development for Headend/STB components, Conditional Access Providers, Billing, Scheduling  & Play-out Systems and System Integration services.  This is made possible by basing the various technologies on open standards (MPEG/DVB/IET/IEEE) and thus allows for creating an end-to-end system consisting of a variety of components from different, independent vendors.  However, there are some TV operators who opt to use one supplier, often referred to as the Primary System Integrator, who are tasked with not only ensuring the end-to-end system is architected correctly, but quite often, the Primary System Integrator provides the bulk of the system components end-to-end (i.e. dual role of component developer supplier and integrator).

System Integration and Architecture are linked, they go hand-in-hand. They are not separate entities, although in theory the act of architecting is different to the act of integrating, but integration drives the architecture.  Looking at the above figure showing the different architectural streams, the following becomes clear:

• Solutions Architect is replaced with Solutions Integrator
• Headend Architect is replaced with Headend Integrator
• STB Architect is replaced with STB Integrator

This now exposes us to the world and levels of System Integration.  A Programme needs to define its System Integration strategy at the outset, success of the programme depends on a solid System Integration strategy.

If you're a broadcaster taking ownership of SI, you need to decide in what capacity you will be involved in:

• STB SI
• This is generally taking responsibility for integrating the various components of the STB stack (typically Platform Drivers, CA Component, Middleware, UI & Interactive Engines) providing a stable build (STB image).
• Interfacing with a broadcast/backend is part of this responsibility but STB SI assumes the necessary backend/headend infrastructure is in place
• Responsible for managing and driving through component deliveries in terms of quality expectations and to some extent project time-lines
• It is always useful to have access to component source code to aid in investigating and characterising bugs, but this isn't always possible as it has commercial implications with IPR, etc.

• Headend SI
• Within the Headend system itself, there are a number of component systems that must come together in implementing a specific macro-component features, like Conditional Access, Scheduling, On-Demand Catalogue/Playout, IPTV streaming.
• Generally though, headend SI is about integrating these headend components either individually or collectively, proving the interfaces are working as designed (i.e. verifying the input/output is as specified by the protocols, essentially obeying the Interface Control Definitions ICDs)
• Test tools might be used as test harnesses, or the STB itself can be used depending on the situation
• There is a fine line between Headend SI and End-to-End SI or Solution SI, which can often lead to confusion and false assumptions

• End-to-End SI or Solution Integration
• This is about testing the solution in a representative environment, as close to real life operations as possible
• It brings together proven systems at the component level, i.e. STB, Headend and other dependent infrastructure systems like Transport (Encoders, multiplexors, IP networking), Billing & Subscriber Management
• There is often a dependency or interface with systems that are currently live (live components are considered crown jewels, usually managed by a separate delivery/operations team).
• Failover, Recovery and Upgrade scenarios are performed as part of this exercise
• Requires investment in one or two end-to-end labs (Some broadcasters impose this on their major vendors to ensure appropriate end-to-end testing is setup locally at the vendor premises; and some vendors have end-to-end labs set-up on the operator's site)
• Often confusion around how this activity defers from System Delivery/Operations - think of this as pre-delivery, the last mile of the programme's development phase.
• Becomes really important to define an end-to-end SI strategy where the overall DTV value change is a conglomerate of components provided by different business units offering different services, e.g. a Broadcaster might have separate departments, sometimes independent units each offering a slice of the system (Broadcast, Interactive, VOD, Guide, Operations, R&D) - all these business units must come into alignment at the outset of planning the programme...

In a nutshell, the following diagram tries to illustrate the worlds of SI, the intersections shaded show the typical integration points required. The worlds are interconnected, and can't be expected to be tested and qualified in isolation of other systems (during development phase, unit/component testing is expected, but as soon as we have stable elements to make up the E2E system, then its time to kick-off your End-to-End SI activity). Typically, E2E testing must start well in advance of launch/go-live, typically 6-8 months of E2E testing is not unheard of...

Worlds of SI

The world of Delivery/Operations

The absolute last mile in completing a DTV Programme, is the actual launch - go live!  This is usually under the control of an Operational entity who's sole purpose is to ensure the smooth operational deployment of the systems on live - 99.9999% of reliability is expected, the transition from End-to-End SI to Live Deployment must have the highest probability of success.  In parallel to the E2E SI activity, there is usually a Delivery/Operations team that are mirroring the E2E SI effort, making preparations for launch.  This delivery team must be included in the Programme Schedule from the start, so they can make the necessary equipment procurements, network configurations and deployment schedules in advance to ensure the smooth transition.  Practically though, the delivery/operations team might re-use the same people from E2E SI, but it is important to ensure that everyone understands the milestones and phases of this activity.   Forward planning is crucial to avoid ambiguity and confusion down the line, which goes back to my point earlier that the boundaries and scope for each Testing/Integration activity are clearly understood.

As always, I'm interested in your comments and feedback.