White Paper Preview: Technologies for Internet TV

I am working together with a friend on a joint white paper covering popular technologies around Internet TV. We've been working with this on & off for a good few months, in between work, balancing family needs and what little spare time we have.  The idea is to share our skills & experiences through white papers, thus attracting offers for consulting...I'm releasing this very draft work in progress to act as a motivator for us to actually finish this white paper in 2012:

If you stumble across this post and are interested in this topic, please get in touch.

-----------------------------------------oOo-------------------------------------------------

General Overview of IPTV Technologies, covering Online Portals with media consumption via PC, Internet Streaming technologies, Client implementation (PC, STB, TabletPC), Open Source tools & Home Networking

Increasingly Telcos and Broadcast operators are looking into increasing their revenue generation stream by providing multiple mechanisms for customers to consume entertainment content over the Internet. These businesses usually decide that the achievement of this objective can be realised by having a web (or online on-demand) presence by creating a platform for delivering content over the Internet.

The road to success lies in solving a multitude of issues in a variety of domains including marketing, sales, legal, and technical. This white paper focuses on the technology choices and challenges for implementation, providing a discussion point around a high-level, simplistic solution intended to highlight the challenges these businesses face when creating the solutions.

So from the construct that a business decides to have a online presence and wishes to provide a entertainment video streaming solution to augment the bottom line the following issues needs to be resolved:

• Rights to Content for distribution
• Marketing the solution against other vendors
• Technical solution to implement the portal and distribution mechanism
• Support for customers

Rights to Content for distribution

This is key to the success of the project: If the telco is unwilling to acquire the right mix of content based on its targeted subscriber base the project is likely to fail.

The fundamental issue impacting this area is getting the content providers to allow distribution of the content over the Internet in the background of security concerns where it seems nothing is safe to hacking attacks on the Internet. Content providers look for assurance that their content will not be compromised and redistributed without them realising revenue.

This usually requires utilising a Digital Rights Management (DRM) solution into the product where the content provider feels satisfied that the protection mechanism is secure enough for them to use the solution.

Choosing a DRM solution is an an activity that involves thorough planning right at the outset from designing the architecture, as this is a fundamental component to the system architecture, from back-end distribution to consumption on the end-consumer device. A number of DRM-solutions exist in the marketplace, this paper will focus on the popular (not necessarily most secure) solutions including Microsoft Windows Media DRM, Apple Streaming DRM, NDS Secure VG DRM, …

Marketing of solution against other vendors

Since businesses need to differentiate in the competition space, focus is made on a variety of factors such as price, product features (integration with devices, social networks, widgets etc) and type of content (niche versus premium) amongst others.

Companies with existing Portal solutions (TODO: Give example references) generally adopt a common theme of marketing, specifically centred around on-demand, instant access to content, variety of content, consumer freedom to choose to watch “whatever you want, whenever you want, on any device, anywhere”. Of course this is music to the consumer’s ears but the proof of the pudding lies in the actual implementation - and to realise this marketing objective is to overcome a number of tough technical challenges as we’ll focus later in this paper.

Technical solution to implement the portal and distribution mechanism

The technical challenges in relation to the above two is a simpler problem to resolve and relates to creating the platform, infrastructure, know-how of a wide variety of issues that encompass the following areas:

• Online Portal
• Subscriber Management
• Payment System
• Content Management including Acquisition, Processing and Distribution
• Content Consumption by devices
• Support Systems
• Regional Infrastructure

Online Portal

This is the gateway into the offering by the business; the user experience greatly drives the success or failure of the system. The Online Portal must provide features for:

• authenticating the user;
• managing the user’s profile;
• billing information;
• social network integration;
• allow searching for content; and
• placement of  recommended content for the user based on their previous usage; and
• YouTube-style features to allow the user to follow trends and watch what other users of the system are watching, have recommended / commented upon etc. In essence this is the public face of the content management system which is integrated with the billing system and distribution system.

This is an intensive area requiring expertise in creating a very usable interface for all types of users. Along with this, the main feature is to allow consumption of the content and is typically based around using a Flash Player (TODO Reference to Flash) providing a compelling experience in comparison to the user’s TV.

There is a tremendous amount of integration work required, typically focused on:

• adaptation of the user experience across all web-browser platforms
• the content player that renders the content based on the format of the content
• what audio/video players are available
• what devices are required to be supported
• what features are to be supported such as the protocol for delivery of content
• support for degradation of video quality as the bandwidth available to the user fluctuates based on usage pattern in the user’s home;
• the bandwidth available to the user;
• the pathway between the source of the content on the Internet and the user’s consumption device.
• Additionally other features including the DRM technology dictates how the interaction occurs.

We shall aim to discuss each of the above points in further detail in future sections.

Subscriber Management

This domain needs to be well understood by the business model. It broadly encompasses the following areas:

• uniquely identifying the user, the device (including the limitations of distribution profiles) being used for consumption;
• business model restrictions based on subscription tier, promotions, discounts, billing information etc;
• the management system integrates with the online portal in addition to the content management system as well as the payment system to realise the revenue.

Payment System

This is typically a legacy or third party system that the operator utilises and integration of this to the online system needs to occur.

<TODO add more info>

Content Management including Acquisition, Processing and Distribution

This is a significant aspect of setting up the process flow and delivery chain for content consumption by the operator. Depending on the investment, an operator budgets for the solution that need to be defined encompassing the following:

• Setting up infrastructure to acquire content from diverse content providers
• Automating the ingest of content in to the system
• Preparing the content for consumption of the variety of devices the platform is intending to support, ensuring that the correct DRM aspects are catered for each type of end user device
• Managing the content through its life cycle which may entail marketing aspects such as promotions
• Delivering the content through a variety of delivery mechanisms requiring integration with third party vendors

Content Acquisition

This area involves a plethora of decisions that need to be made about the content and how it becomes available on the platform and ultimately how it gets delivered. Issues to resolve include:

• What format the content (audio, video) is provided in by the content provider?
• What metadata is required for presenting to the user on the online portal or during playback and how that is transformed from the input all the way to the point of consumption?
• Whether encoding services (software and or hardware) are required to transcode the content to the end device format?
• Whether features such as adaptive bitrate streaming is required?
• Whether support for live, restart, catch-up, on-demand playback of content is required?
• What third parties to select and what level of integration is required?
• What licensing issues resolve with content owners?
• What licensing is required for encoding to different device formats?

Content Processing

Once the content has been ingested into the platform, decisions around the following areas need to be resolved that include:

• What is the life-cycle of the content?
• What promotion mechanisms are required?
• What DRM(s) need to be applied and what business models will be applicable?
• How is the content encrypted (dictated by the DRM(s) in use)?
• How will the variety of DRM(s) work seamlessly as the user begins consuming content from a variety of devices?
• What extra metadata needs to be added to support different types of playback and interactivity during consumption?
• What support is required by the systems administrators to manage outages and issues with the process flow during deployment?

Content Distribution

Once the content is available for distribution it needs to be made available for consumption over a variety of mechanisms; issues that needs resolution include:

• Protocol for delivery on the network (RTP, RTSP, HTTP, RTPM)
• CDN (Content Delivery Network) support dependent on protocol choice
• Commissioning of hardware resources to serve content to devices
• Integration with DRM system(s)

Content Consumption by devices

Dependent on the number of devices being supported a large amount of effort is used in integrating the Content Player in to the OS (Operating System) framework of the end-device. This entails a variety of issues:

• Choice of devices (PCs, Macs, Tablets, Smartphones, Set-Top-Box, SmartTVs)
• Player Usage (such as Flash/Silverlight/QuickTime/Windows Media Player/Custom) of player using bespoke or open source software (typically involves providing plug-ins for web browsers on desktop (Windows/OSX/Linux) platforms and integrating to the mobile / tablet devices OS framework
• Integration and maintaining integrity of DRM(s) on devices
• Fine-tuning user experience of UI and play-back
• Supporting integration between devices via UPnP (Universal Plug & Play) and DLNA (Digital Network Living Alliance) to support download and play-back between devices

Support for customers

This area revolves around trouble-shooting, resolving issues due to plurality of devices and the versions of software they run and creating baselines for the operator to know what issues cause outages and rectifying them.

Infrastructure capacity

A successful provisioning an online content portal with capacity for multiple device streaming to the home relies on fundamental components of infrastructure support being in place. Not only does the business need to ensure that it can deal with system loading (hundreds of thousands of subscribers accessing the content simultaneously; protection from attackers intent on crippling the service through denial of service attacks) but more importantly the business would have carried out an analysis covering the following areas:

• Number of subscribers with online access, i.e. able to access the Internet?
• Intended delivery mechanism for streaming (xDSL, WiFi, 3G, etc)
• Solutions for supporting a number of different delivery profiles
• Regulatory compliance - what relationships or conditions are in place to allow/prevent delivery of content over third-party networks (e.g. not all ISPs will allow traffic through)
• Content Delivery Networks - is enough infrastructure in place to support the demand (Edge servers, etc)
• Reliability of the underlying delivery network?

Infrastructure and Regulatory concerns might not be an urgent topic to address if the business is operating in a developed country where access to Internet is now considered a basic utility, but for other parts of the world, especially the Indian/African countries, Internet adoption and underlying network infrastructure has not reached the maturity levels as say, compared with countries like the United Kingdom/Germany where there is a high penetration of cable supplying xDSL for instance.  In South Africa for example, xDSL is not the main player of Internet connectivity, instead 3G WiFi is the dominant player - and for such markets, the business must therefore have a solution that meets the variety of access mechanisms in front of the user.

<TODO Examplae Case Study>

We’ll now walk though a general architectural overview of the components in and end-to-end system, assuming the required infrastructure exists...

SI/Development Project/Process Audits...

Earlier this year I wrote about setting up the right Roles & Responsibilities in managing a typical DTV project, covering the Architecture & Project Team requirements.  In this post, I'd like to share my experiences with the Project Audits.

The scope of Project Audits is in itself a vast topic; mostly around the area of auditing the Project Management aspects of the project, along with ensuring effective Quality Management processes are in place, proper Project Governance is enforced, etc. The IEEE SWEBOK dedicates a chapter on Sofware Quality, of which Project Audits is considered as part of this knowledge-area, summarising the process as:

“The purpose of a software audit is to provide an independent evaluation of the conformance of software products and processes to applicable regulations, standards, guidelines, plans, and procedures” [IEEE1028-97]. The audit is a formally organized activity, with participants having specific roles, such as lead auditor, another auditor, a recorder, or an initiator, and includes a representative of the audited organization. The audit will identify instances of nonconformance and produce a report requiring the team to take corrective action.

While there may be many formal names for reviews and audits such as those identified in the standard (IEEE1028- 97), the important point is that they can occur on almost any product at any stage of the development or maintenance process.

What to audit in DTV Projects?
Again, with a bias towards the STB Software Development delivery project, audits are introduced to review two main areas:

• STB Development / Test Processes across the stack (Drivers, Middleware, Applications)
• STB System Integration Processes

If you're a product vendor supplying any of the STB components, you might want to review and assess your own development/test processes to gauge how well you're doing against industry best practices. It's also good for your portfolio when pitching for new business to potentional operators.

If you're a TV operator and dependent on product vendors supplying components to your value chain, you need to ensure that your vendors are indeed delivering quality components & systems. Typically (and this can vary with the type of contract in place), the Operator (customer) provides the Product/System requirements to Vendors; Vendors implement and test their components according to the spec; produce test results (self-assessments), making it difficult for the Operator to dig too much into the detail, to find out exactly how the results came about, or gauge the quality of the underlying development processes, typically until it's too late in the project life-cycle.  This can of course be remedied by building into the contracts the requirement for auditing the vendor's various processes, at any stage of the project life-cycle.

Some operators outsource the whole testing and verification effort to third-party test houses. These test houses, depending on the strength of the commercial contracts in place, could very well take complete ownership of the test & quality management, and enforce audits on encumbent component providers.

As noted in my previous posts, more and more Operators/Broadcasters are taking more ownership over the product development, testing and integration. Take for instance, the task of STB System Integration.  Whilst there are well known companies like S3 & Red Embedded, that specialise in STB SI, operators like Sky, DirecTV, Foxtel, Multichoice have dipped their feet into the waters; attempting to own STB SI activity internally, in-house. Incidentally, there are more players on the STB SI front, some STB manufacturers like Pace, UEC; as well as service houses in India like Wipro & Tata Elxsi. NDS Professional Services is also a well known player in the System Integration field.

In my rather short stint of 11 years with STB projects, the world of DTV is quite small; the popular companies of choice for SI, from my experience are:  S3, Red Embedded & NDS.

In my current company, being a DTV Operator/Broadcaster, we are running our own development & STB Integration projects. Because we're relatively new in this game, having built up a good team of people, making every effort to source the right people with the right skill-set (which is really challenging in South Africa), we'd been through a few rounds of Development & SI, using third-party vendors supplying Middleware & Drivers, we decided now was a good a time as any to perform a self-diagnostic, a review and audit of our current processes, specifically focussing on STB System Integration and Supplier Deliverables coming in. Although we have enough knowledge in the team to perform a self-assessment ourselves, and despite having learnt and corrected a lot of mistakes from past experience; we opted for a professional consultancy to do the audit, and chose S3.

Why we chose S3?
S3 despite being a relatively small company in Ireland, has been involved with TV technology for many years; and is well respected in the DTV community; for their specialist design services & system integration knowledge. They've leveraged the wealth of past project experiences in producing useful products like S3 StormTest & Decision Line, a STB automated testing system that has only in increased in popularity since its inception (which we're currently using by the way). The S3 Engage Portal is also another useful product to manage and co-ordinate a complicated DTV SI project.  S3 are also known to be frank & objective, offering feedback and analysis that is open, without hiding the nasty details. S3 were chosen as SI for some big players in Europe; and they're also being used in some other projects of ours.  Based on strength of previous relationships; despite the fact that I previously worked at S3! The world of DTV is small, the other companies mentioned above are also competent and capable.

What areas do you focus on for an SI audit?
Of course I can't share with you specifics of the S3 Audit process, but if you're interested in learning more please contact S3, or speak to this guy.
In a nutshell, S3 shares a model of SI that is well known to the industry:

• Incremental / Iterative approach, built on a strong foundation of domain expertise, using experienced System Integrators.  The foundation cornerstones that deliver an effective SI process, rely on the following:
• Strong Technical Expertise
• Strong Project Management
• Usage of Advanced Tools & Infrastructure to support the process
• Strong Quality Assurance Practices

S3 have been kind enough to allow me to publish a slide on their overview of SI process:

S3's Integration Model: SI is a continuous cycle once all Components are in a suitable state of completeness

Based on the above cornerstones, the audit focuses on 48 topics of concern, that largely fall into the following categories - again, the below is generally best practices as evoked by IEEE SWEBOK Chaper 11 on Software Quality which proves S3 appreciate Industry best practices:

• Project Management
• Co-ordination & Communications
• Requirements Management
• Defect Management
• Testing
• Quality Assurance
• Supplier Management
• Release Delivery
• Resourcing
• Software Architecture
• Integration & Configuration Managmeent
• Software Development

My Own List of Audit Criteria
Since this knowledge comes from my own past experience, I've no problem sharing information with with you. If you want more details, feel free to contact me, or leave a comment. The list below is something I put together in advance of the audit we've just undertaken.

If I were doing a STB audit, this is what I'd be interested in:

• Foundations of the STB Component (Middleware/UI) architecture design
• Design philosophy – apart from being MHP/Jave/MHEG/etc compliant, what is the design ethos behind the architecture, i.e. does it scale?
• Design patterns – what abstractions are used to ensure flexibility with changing requirements without major impact to design?
• Performance requirements – is the product being architected with performance considerations in mind?
• Security considerations – is security checks in place, to prevent buffer overruns, etc?
• Stability considerations – how do we ensure stability checks are enforced in design / implementation?
• Open Source components usage – if open source is being used, is there a process for choosing components, etc?
• Inter-component Interface design and control – assuming the middleware is a collection of components, how are component interfaces managed?
• Documentation – Requirements, Design – is there a documentation standard in place? Middleware design, component design, etc?
• Process for introducing change to architecture? New components to implement new features? – How are new components introduced to the system?
• API ICD Controls – how do we manage API changes??
• Is there an SDK?
• Quality of API Documentation? Is it usable, self-explanatory, auto generated??
• New APIs must be communicated to application developers
• Changes in APIs must be controlled
• APIs must be backwards compatible
• API Deprecation rules

• Testing the STB (Middleware/UI/Drivers) stack – what is the testing strategy employed?
• Depending on the architecture, assuming it’s component based architecture – do we have a continuous build and integration system in place essentially for regression testing:
• Component based testing – unit tests on the component, using mocks for other components, assuming interfaces are controlled
• Component group testing – where you collect related components together and test as unit, e.g. PVR engine can be tested in isolation
• Middleware stack testing – test the middleware as a whole, i.e. collection of components forming the MW
• Regression Suite – does one exist, how is it tested, and how often is it tested?
• Spread of STB hardware platforms used for testing
• Use of a Simulator
• How often do we check for regression – daily, weekly, just before release?
• How is new feature development controlled – i.e. new features to be delivered without causing regression in previous working functionality
• Is there a concept of a middleware integration team?
• Middleware development is separate to middleware integration, separate teams
• Development teams do component development, and component unit testing
• Middleware integration team takes development components and integrate – test new features against last stable middleware as well as test new functionality
• Middleware integration team consists of technical people who can code – best practise is to use a middeware test harness that exercises the middleware, not just the external Application APIs, but speak directly to component APIs themselves
• Is there a separate team owning the typical STB Hardware Abstraction Layer / Driver Interface?
• Who defines the HAL spec / interface?
• Change control procedures in place to support HAL spec?
• Testing of the HAL?
• Is there a test harness that validates new platform ports?
• Is there a separate Platform Integration team?

• Supporting multiple customers and build profiles
• How does the architecture support the needs of different customer profiles: Satellite / DTT / Cable / IP ?
• For the same customer with different project requirements, how is the codebase managed?
• Build, Release, Configuration Management and Bbranching philosophy?

• Multi-site development
• How is multi-site development managed?
• What configuration management system is used? Does it support multi-site development well?
• Are off-shore teams used as support arms, or do actual development?
• So off-shore team have any ownership over components?
• How is communication managed with off-shore teams??
• Planning releases
• How are new releases planned?
• How is work assigned to development teams?
• What is the usual process for planning / starting new development?
• How much time is spent designing new features?
• Is testing included as part of the initial design review?
• Are new features delivered back to main product code tree in isolation or in big chunks??
• How much time is allocated to testing on reference platform?
• How much time is allocated to testing on customer production platforms?
• How is the backlog prioritized?
• How long is the development cycle?
• What is the exit criteria for development?
• How long is the test cycle?
• What is the exit criteria for testing?
• Is Middleware integration part of planning?
• Is full-stack integration part of planning?
• How are defects planned?
• Does the plan cater for stabilization period for new features?
• Who decides on the branching strategy nearing up to release?
• Are new tests identified as part of the planning process?

• Defect Management
• Is there a defect management process in place?
• How are defects managed and factored into the development process?
• How often is root cause analysis performed?
• How is quality being managed and tracked internally?
• What defect tracking tool is being used?
• What metrics can be generated to gauge the quality of all components in the stack??
• Average submission rate
• Closure rate
• Time it takes to close the average Showstopper
• Most suspect components
• Most complex components

• Development practises – industry standard coding practices should be in place
• Coding standard
• Architecture rules – e.g. system calls to avoid, avoid using strcpy, etc?
• Is CUnit test framework or any other suitable framework used for native component testing?
• Is JUnit being used?
• Do component unit tests map to component requirements?
• Do middleware tests map to system requirements or use cases?
• Are any of the following code quality metrics being looked at:
• Zero compiler warnings
• Zero MISRA warnings for native C components
• Which Java best practises being used?
• Code Coverage:
• Line coverage tools
• Branch Coverage
• Code complexity
• Function points
• Other static analysis tools like Klocwork
• Development organisational structure
• Size of development teams
• Structure of development teams – is it based on architecture components, i.e. notion of component teams, with component owners?
• If using multisite development, what is the line management structure – is the team a matrix, with virtual management teams?
• Do you have separate groups for:
• Development: Separate UI development, Separate Middleware team?
• Integration – is there a concept of middleware integration team?
• Platform – is there a concept of a team specialising in low level HAL/Kernel/Drivers/Platform
• What are the defined roles and responsibilities:
• Product Architect  - the person owning the product architecture, maintains the integrity of the philosophy, keeping everyone in check?
• Middleware Component Architect – do you have people owning specific components in the middleware, ensuring the interfaces are respected. For e.g. PVR engine usually is a separate architect?
• Lead Developer / Component Owner – someone responsible for delivering component to spec, owns the internal design and implementation of the component?
• Engineers – what is the mix of people and skillsets across the organisation

Managing a Digital TV Programme: Consideration for Project/Organisational Structures...

Following on from my recent post on the role and placement of the "architect" in a DTV systems project; in the same vein I'd like to expound on the role of the project team in the same end-to-end system delivery, touching on overall organisational structure, considering the concepts of:

• End-to-End Product Owner
• Feature Product Owner
• Component Product Owner
• End-to-End Programme Manager
• Headend Programme Manager
• STB Programme Manager
• Headend Component Project Manager
• STB Component Project Manager
• Headend Scrum Master / Tech Lead
• STB Scrum Master / Tech Lead

The collection of some or all these roles will make up your DTV Project team. These roles might each be performed by a person each, or we might have someone performing multiple roles.  Defining these structures up-front at the start of the project is considered Project Management 101, but you'll be surprised there are real world projects operating today where time, effort and preparation hasn't been invested in defining these organisations structures, and are operating in a quasi-chaotic state. I've been on projects where people run like headless chickens, not knowing who's responsible for what, where the boundaries lay in terms of roles and responsibilities, where due diligence isn't practiced through the overall programme: architecture, development, integration, testing & delivery...

The question of whether these roles are managed by a specific organisation unit like your classic Project Office or whether these roles come together naturally through different business units and operate as a virtual project team -- is outside the scope of this discussion.  My aim with this post is to highlight from my experiences what seems to work for DTV projects involving a launch of a new platform like VOD, or replacing old Middleware with new; again with more of a bias towards STB software, although the Headend will not be so dissimilar.  In essence, this can be drilled down to generic component product development at the micro-level, and at the macro-level these are rolled up in the end-to-end system as previously highlighted here for architecture, and the figure below that shows largely how a project team can be structured around the system.

It doesn't really matter about the internal business organisational structure, what matters is that the project has a defined structure and that these roles exist; the roles and responsibilities are well defined and clear, ownership and accountability is understood by all parties, that everyone is connected and on the same page. Of course you'd think this is Product/Project Management 101, that this is a no-brainer -- not so. 

In my experience, companies don't always have this done right, yet you might argue that products are still successfully launched, projects executed and closed, etc. Indeed, but IMHO such projects generally rely on a few individuals that go above and beyond the call of duty, the proactive, driven by a sense of urgency to get things done; filling in the gaps in process & roles -- that companies have grown to depend on.  Whilst this is great and generally seen as heroic efforts, the fire-fighting and critical moments can be saved if a little time is spent up-front by defining & clarifying the project structure.

Again, in the context of DTV projects - the structure depends on the viewpoint.  In general, if you're a broadcaster / PayTV operator (BSkyB, DirecTV, Multichoice, etc), you will have to implement a detailed project team as there is a much wider impact on the business (development, lab, testing, validation, going live, operations, support, customer service) than compared to one of your vendors (NDS, OpenTV, Irdeto, etc). If you're a Professional Services outfit acting as Systems Integrator and ultimately responsible for the delivery of the project, then you might have to mirror the same project structure of your customer.

Let us look at a picture to see where these roles can be placed. Notice this map is exactly the same layout as the architects post:

Figure 1: Overview of DTV System highlighting Project Roles

If you're an Operator focused on content delivery only, then...

You generally hire a System Integrator to manage the design and end-to-end delivery of the system on your behalf. These operators make a conscious decision not to be technical experts and thus rely on preferred partners to manage new development and operations.  The operator would still have someone who's responsible for product ownership, which could be someone from Marketing/Sales. There would also be a Programme or Project manager assigned to manage the project.

This is typical of first-time operators entering the market from cold, there are some professional services companies like NDS, who can manage not only the end-to-end systems development or customisation (generally this is off-the-shelf to enable shortest time to market, followed by product updates post launch) but also able to commission a broadcast centre and operations control from scratch...

So essentially this is an operator that outsource most of the Delivery Project Management - the organisational structure will pretty much be the same as Figure 1.

If you're a mature Operator with an established product and adding new features, but without technical ownership, then...

This scenario again relies on the Operator leaning on preferred partners for providing the technical solution, possibly spanning multiple vendors involved in development of components, system integration, delivery and deployment as well as operational support.  There would be a Product Owner / Manager that defines the high level requirements for the product's feature set. There could also be a Project Office team that takes ownership for the overall co-ordination effort from start to finish, or this could be outsourced to a professional services company to manage the whole project.

Some Operators have separate business entities owning features of the overall value chain, so the primary product spans multiple product owners. It becomes more challenging to manage this distribution of product owners especially when trying to manage an consistent brand and user experience, and coherent presence in the market. But companies are doing this today, it works to an extent, the key being effective communication is pivotal to successful product launches.

Generally though, in this scenario, the Operator has established strong relationships with its vendors, and projects may follow a typical template plan that would be driven by a Project Office team.

You might find the Operator would enlist a Project Board to manage the project's priorities, resources and budgets - that a Product Owner/Manager and Programme Manager reports and takes direction from.

Again, the project organisational breakdown will follow Figure 1.

If you're a mature Operator and responsible for developing and delivering new features, with technical ownership, then...

This can be very complicated and challenging if this is the first time the Operator is responsible for technical ownership. As touched on in my previous post, more and more Operators are taking more ownership in managing DTV projects, around the area of STB & Headend Systems Integration, EPG Development, Interactive Applications development, and some Operators are even developing their own Middleware.

In this scenario, it then becomes fundamental that the organisation starts on the correct footing, ensuring the foundations are cemented correctly, processes and practices firmly put in place as early as possible -- not doing so will definitely result in chaos, which I personally have come to label a "Project Fiasco".

So the roles I'd expect to see an organisational structure clearly defining the Architecture roles, Development, Integration and Testing; as well as a well defined Project Team.

I would expect to see a variant of Figures 1, 2 & 3 for the organisational structure.  There will be interaction with vendors. The relationships, roles & responsibilities must be explicitly communicated....

If you're a Product Vendor that delivers components to Operators, then...

Product Vendors are generally companies that provide software components for STB or Headend components that enable the Operator to function.  As a Product Vendor, the challenge is to internal manage your development organisation to maintain a consistent product line offering features that can be easily customized, tailored or profiled in or out to meet the requirements of your customer, the operator. Typically vendors manage multiple customers. Some customers more important than others - the customer usually generating the most of amount of revenue is assumed the lead customer, if not, the customer that in fact drives forward your product development and innovation.

Vendors typically have a Customer Delivery Manager assigned as an anchor support point, interface to the customer. This Delivery Manager ensures the customer's requirements are accepted by the product development team, and ensures the resulting plans are in accordance with the customer's expectations.  The vendor may choose to use a Product Steering Group that ultimately makes the decisions on customer priorities. Depending on the extent of the products being supplied by the vendor, I would expect to see an organisational structure not so dissimilar to Figure 3.

Brief Definitions

• Product Owner / Product Manager
• Operator - responsible for defining the user experience, high level requirements of the features for the product. For example, Operators looking at VOD will have a Product Owner owning the feature set, along with priorities. The Product Owner takes direction from the Project Board or even the business in understanding the market priorities and urgency of the business w.r.t. launch targets. The Product Owner / Manager will use the services of the architects and project team to get the product specified, designed, implemented and ultimately delivered.  The extent of ownership is around product specification only...The Product Owner/Manager gets actively involved in prioritizing defects leading up to the launch phase of the project, setting priorities, accepting waivers or concessions, etc.

• Vendor - as a vendor, the Product Owner / Manager is different to the role taken on by an Operator.  As a vendor, the Product Owner is concerned with satisfying as many customers as possible, without much deviation or customer-specific features; and will work hard to maintain a consistent offering. The Product Owner generally consists of a Product Team that centrally manage the product backlog. Depending on the size of the projects and demands from customers, this could be one person, or a team of people. If it's a Headend component with specific technical requirements (generally Headend component is a translator - data in, data out component) that implements a protocol, then this could be one person.  However, if this product is a STB Middleware that must sustain multiple customers, and potentially support tens of millions of boxes, then you'd need a separate team managing the product. This Product Management team will own the roadmap for the project, control the work assigned to customer projects, and ensures a coherent release schedule.  The team will also ensure the product specification and requirements are kept-up-to-date supporting the generic product as well as the customer's profile of the product. This Product team will consist of project managers driving through the product roadmap.

• In some cases the term Project Owner / Key Accounts Owner / Customer Delivery Manager is used which basically is the person who's neck is on the chopping block - i.e. the person accountable for the project delivery, who has seniority to make judgement calls to steer the project along in his/her preferred direction, etc - basically the ultimate person in charge, the "X says so" person...

• Programme Manager - this can be generalised both from Operator and Vendor side - and is typically the person responsible for the high level planning of the various work streams that make up the ultimate delivery.  There can be Programme Managers for each system, primarily Headend & STB, as well as managing the overall End-to-End delivery plan.  The Programme Manager works side-by-side with Product Owner in understanding business priorities, but the Programme Manager has autonomy in defining the plan and driving through the execution of that plan.  Programme Managers lean on their respective project managers to take responsibility for the detailed planning that delivers on the high level milestones.

• Project Manager - is responsible for planning, managing and driving through the deliverable of the assigned component/project. Reporting and taking instruction from the Programme Manager, the Project Manager handles the day-to-day issues/planning/tracking, escalating to next level (Programme Manager) as required.

• Scrum Master - can be a Project Manager or Team Leader employing the philosophy of Scrum/Agile to get the component/project team to deliver according to the milestone plan put forward by the Programme Manager and Project Manager. 

What a Product Owner / Manager is not responsible for...

When not acting in the capacity of the Project Owner, i.e. the main person accountable for the success of the delivery, then the Product Owner / Manager should only be focused on managing the Product requirements and Feature set, setting expectations and aligning priorities.  A Product Owner has no influence on the resource structure of the team, the scheduling of the development....that's what the Project Team, including the Programme Manager is there for - the Project Team provides a service to the organisation...Depending on the size/complexity of the deliverable, in DTV projects, they are large enough to warrant a clear separation of boundaries between Project Team & Product Team....Having people stomp on each others toes is not conducive to productivity...

We are an Agile Company, we don't like heavy Project Organisation...

Sure, depends on what kind of company you are. If you're in Application Development writing EPG UIs or Interactive applications, then of course, you can be light on the ground.

If you're in the Middleware business and big enough to be mentioned in the same sentence with NDS, OpenTV, etc -- and you don't have these structures in place, then you're at a competitive disadvantage. I have worked on small and large projects alike, the lack of a clearly defined and sensible Project/Organisation structure does not work...you might reach there eventually through luck, a lot of hard work and unnecessary fire-fighting...

For Operators taking the plunge at taking more technical ownership in your projects, my advice to you is "Do your homework". Spend time up-front investigating, planning and settling on a processes that has been known to work, i.e. proven - by getting real consultants in to advise...By real consultants, I mean not the people who know the theory but have never managed a real life project from Start-to-Finish, not someone who's never worked in the DTV industry before...Invest in the right people up-front, employing or head-hunting candidates who have first-hand experience and proven track records, and pay heed to the advice henceforth...

Typical Project Org Structure for a typical Operator

Below is a simple model of how an Operator might define an Organisational Structure of a DTV project...

Figure 2: A possible Org/Project Structure from the Operator's viewpoint

Typical Org Structure for a Vendor supplying components...

Below is a complicated model of a Vendor who's main business is STB development. Some vendors specialising in components in specific components will own a slice of this structure...

Figure 3: Org/Project Structure from a STB Software Vendor's Viewpoint

PDFs of Diagrams

Figure 1

Figure 2

Figure 3

Your Feedback

Please note my posts are always based on my first-hand experiences from past projects, and is a reflection of reality instead of theory. I am always interested in feedback, please feel free to comment or rate this post!

Architect Roles in Digital TV Projects

The subject of IT architecture is vast and multi-faceted; often open to interpretation by organisations in the same and different businesses alike; especially the definition and expectations of the "architect" role in projects in supporting the technical needs, administration and day-to-day operational management. A number of people attempt to explain what architecture means, from professional institutions like IEEE & TOGAF that provide a framework for describing the various architect roles for IT projects to attempts by wikipedia.

The rationale behind my post is to highlight how the role of architect manifests in companies specialising in digital TV systems, be it on the broadcast systems engineering front, or product developer offering systems and software services. It has been my experience that even with companies involved in the same industry, the use and understanding of the role "architect" varies, and is open to interpretation, often leading to confusion when you're mixed up in customer-vendor service relationships in managing projects. Some companies do attempt at having an organisational structure supporting technical architects, whilst other companies try to hire anyone with a technical background and assign the role "architect". But the aim of this post is NOT to describe the traits and skill-sets of architect VS coder; but to focus on the functional roles that an increasing number of DTV projects are now demanding from architects.

If you're a PayTV operator like BSkyB, DirecTV, Multichoice, etc - you need to think about the level of technical expertise required for your business:

• how much technical systems knowledge do you need to own?
• how much do you depend on external vendors to manage this on your behalf (e.g. the design of your end-to-end system)?
• do you have to produce & own your very own technical specifications for the entire value chain; or just key parts (e.g. EPG Spec, Middleware Spec, UI Spec)?
• or are you comfortable relying on your preferred system partners?
• do you need to have technical people on your side to prevent vendors from pulling the wool over your eyes?
• do you need technical owners for each product/sub-system of your deployment?
• do you want to drive or your suppliers/vendors or be driven (locked-in) by your suppliers solutions?
• do you opt for generic products to configure & customise yourself, or do you impose a custom requirements to serve your purpose, to meet your business needs, according to your environment?

This list goes on & on... There are an increasing number of PayTV operators that are moving away from the traditional vendor-supplier-user model, to taking more ownership and control over their products. Some operators have set-up internal development houses to manage Headend & STB UI/Middleware development; whilst other operators are taking over STB manufacturing in an attempt to control the end-to-end value chain.  In such cases, it therefore becomes even more relevant to have a solid technical product team, connected from end-to-end, with a thorough understanding of complete system, delivery and value chain of products, from high level features, requirements and design to detailed technical operations of the product --- this is not just my opinion, some companies have already embarked on this path (BSkyB, DirecTV, Foxtel).

If you're in the business of providing systems & software services to the above-mentioned operators, the likes of NDS, OpenTV, Irdeto and other Professional Services companies, then you probably have well defined roles for architects, and probably have an organisational structure of an architects group, supporting your many projects, with a group of chief architects as overseers for your product deployments.  If not, then please contact me as I'm interested in learning about what others in the industry are doing here :-)

I've been mulling over writing about DTV architects for a while now, but what really triggered this humble attempt stems from a recent experience on one of the projects I'm managing: I'm sure most of you are familiar with "It's not my problem, I'm an application architect not a systems architect" - you look for a systems architect only to find that one doesn't exist yet!

Anyway, to take this discussion forward - I consider the end-to-end layout of a typical Digital TV system. I'll present the different flavours of architect roles, and conclude laying out the minimum expectation of the roles that must exist to fulfill your typical DTV product deployment, with a bias towards the STB-side than the Headend.

This is NOT a theoretical or academic exercise. I'm describing real world experiences from past projects I've worked on with NDS/BSkyB/DirecTV/Foxtel/OpenTV, although the major influence is from NDS who, in my opinion, do an excellent job at executing complex DTV projects; and have an org structure not so dissimilar to what I'm presenting here.

I am really interested in your comments and feedback, what is the rest of the industry doing in this area?? Feel free to comment directly or drop me an email.

Consider this picture of a traditional end-to-end Digital TV system:

Figure 1: Components of DTV System Value Chain (Download the PDF)

Disclaimer: I'm no architect, this is Version 1, just my understanding and interpretation only. Architects do the best they can under the circumstances....there is a level of pragmatism involved and no one should be under the pretense that the solutions created are perfect, they are what they are i.e. a solution out of a continuum of many possible solutions for the problem which solves a business need at that point in time for a price the organisation can afford under the assumptions of the perceived known knowns and anticipating the future usage for that solution at that time.

A typical end-to-end DTV system is complicated; there's certainly nothing simple to it. At the highest macro level, one could summarise a DTV system as being split fundamentally into three areas (or in architect-speak, subsystems):

• Transport/Signalling: This is basically the physical layer, the profile of the underlying delivery mechanism. It can be abstracted and treated as a separate system as the products that sit on top of this layer still provide the same application behaviour, regardless of medium. I've separated it out because here you'll find physical hardware/infrastructure systems that prepare the data for the transmission across the appropriate delivery mediums (Satellite, Cable, IP, Terrestrial, Mobile)
• Producer: Headend/Backend: Systems that generate, secures & distributes content and information over the delivery medium (satellite, cable, terrestrial, mobile, IP)
• Consumer: Devices - typically the STBs receiving the transmission, decode and presenting the info and a suitable user experience to the subscriber at home

Within each area, the system is broken down further and further down to individual components performing a specific task. In reality, you will find that this end-to-end system is a carefully orchestrated integration of a number of different subsystems; offering a mix of different hardware and software components that are supplied by different companies. This is especially true for the back-end, integrating third party components such as the Conditional Access (CA) system, Billing & Subscriber Management, Scheduling, Information Data Generators, Content Catalogue Servers, Play-out systems, Encoders & Multiplexers.    At the consumer device end, the STB is a system in itself, also broken up into components or subsystems; and also not provided by one single company: Typically the STB consists of an Application Engine / User Interface component (the EPG), Middleware (the TV operating system) and Platform Components (Drivers, Hardware, Kernel). And these core STB components themselves are broken down into sub-components.

So your end-to-end DTV system is complicated & highly component-based. These sub-components, components, sub-systems, systems and ultimately the end-to-end solution must be co-ordinated and controlled -usually all these systems obey strict rules and contracts as defined by Interface Control Documents - Architecture 101...

So where do the different architect roles fit in then?

At the macro level, an end-to-end system could be managed by a team of architects specialising in these broad roles, be it at the Headend or STB domain:

• End-to-End (E2E) Systems Solutions Architect
• This is the top of the hierarchy, the solutions architect has a birds-eye view of the end-to-end system. Generally working directly with the the business (marketing/sales) in understanding the business and market driving factors in creating a solution for a specific need or product. 
• The E2E solutions architect is someone who can work with any technology domain, has a deep appreciation for all the subsystems including third-party external components, and will work with the relevant system architects in defining the high-level solution. This results in producing a grand, high-level design of the system components or building blocks, specifies the requirements and interfaces expected from each subsystem, laying the ground rules for system architects to take forward, adapt as necessary to apply the design in a real-world deployment.  
• The solutions architect is a domain expert, expected to take existing deployments and adapt, expand and enhance as necessary to deliver new product requirements.  This is generally the first stage of the projects design and scoping phase.  
• In some projects the solutions architect goes right into the detail of enhancing the MPEG/DVB signalling specification; and some times even creating a new end-to-end signalling protocol altogether. 
• This is a highly collaborative role, involving a lot of communication back-and-forth with sales/marketing, product development and integrators.  This could be one person, or a team of solutions architects taking responsibility for certain slices of the solution, depending on the scale of the project and problem being solved.
• Some people use the traditional IT term "Enterprise Architect" - but in all the projects I've worked with, the DTV industry tends to use "Solutions Architect". 
• In a nutshell, the solutions architect uses existing subsystems to deliver a solution to a problem and identifying shortcomings/gaps for attention Systems/Product Architects

• End-to-End Product Systems Architect
• Depending on the project and product being delivered, the solutions architect usually takes an existing solution and adapts in to the needs of the new product. In doing so, the solutions architect will touch on many components or subsystems that are impacted by this change, thus highlighting the areas that make up the new product.
• One might therefore assign specific Product Architects to take responsibility and own the components making up the product's value chain. 
• Primarily concerned with interactions between subsystems where Headend and STB is subsystem.
• The Product Architect assumes responsibility for the product solution and works hand-in-hand with the solutions architect in defining the solution that best meets the needs of the business.
• Solutions architects typically get involved at the initiation stage of the project and leave to solve the next challenge - they rarely see a project through to completion.  Product Architects however are expected to stay till the very end, supporting and making all architectural decisions to get the product from development through launch.
• This is still a high level responsibility, product architects work closely together with respective system products architects: Headend & STB.
• The product architect drives through the respective product specifications, covering functional and non-functional requirements (performance, reliability, resilience, etc).
• This role should not be confused with Product Owner or Product Manager; which are more management roles focussed on product features, specifications and high level requirements, which feed the work of the Product Architect.  Depending on the size of the organization and skill-sets available, these roles could be merged.
• Some people tend to refer to Product Architects as Technical Owners.

• System Architect - The system architect spans Headend & STB subsystems:
• STB System Architect / STB Product Architect
• Responsible for the end-to-end stack of the STB covering platform hardware, kernel, driver layer, middleware and EPG - Responsible for the overall STB product, taking ownership for the full product.
• Has a deep understanding of the key areas of the STB to the point of specifying the interfaces for all the above components
• Collaborates with respective EPG & Middleware architects in designing the overall STB solution
• Is responsible for defining the Functional & Non-Functional Requirements of the system: e.g. monitor/stipulate memory usage and own performance
• Helping the product team with requirements, and that includes backlog, supporting the initial planning phases and producing high level estimate
• The gate-keeper for analysing change requests
• Assisting in analysing, reviewing & prioritizing defects

• Headend System Architect / Headend Product Architect
• Whist the STB is a fairly closed system, the Headend on the other hand is an  integration of distributed systems. These systems are all interconnected, the Headend product architect is concerned with the interactions, interoperability and reliability of the integration.
• Works closely with other architects (subsystem/component architects) in defining the product specification. Since the Headend is meant to provide generic services and is a high-risk operation, Headend systems are generally well-thought out in advance, go live months before the STB is ready for deployment. The product architect plays a big part in driving the integration of these systems at the high level.
• Product architects generally own slices of the system, e.g. VOD
• All these slices, i.e. product architects work closely with the Solutions Architect for Headend, sometimes referred to as Chief Architect

• Component Architect - STB
• Application Architect - UI / EPG
• Essentially designs the UI / EPG as a component. Responsible for the internal architecture of this component
• Collaborates with Middleware Architects as necessary e.g. influencing APIs and interfaces
• Works with the STB Product Owner to understand requirements from high-level to detail
• Works closely with user experience specification team to ensure the UI demands are realistic, respecting the boundaries of the architecture of the UI's internal components
• Supports the product team in defect analysis, review and prioritization
• Implement STB system architect's Functional & Non-Functional requirements
• Assist the programme management team in high-level planning and estimates

• Middleware Architect
• Middleware is a system in itself, consisting of multiple components. Without drilling down into too much detail, the Middleware Architect ensures the component satisfies all requirements by providing services to applications using it
• Generally, the STB Architect is in fact the Middleware architect; using Middleware sub-component architects for detail. I worked on a Middleware product that had a centralised STB architect driving a team of component architects, where the Middleware was split into 80+ components...
• Defines APIs both internal and external to the Middleware
• Works with the STB Product Owner to understand requirements from high-level to detail
• Supports the product team in defect analysis, review and prioritization
• Implement STB system architect's Functional & Non-Functional requirements
• Assist the programme management team in high-level planning and estimates

• Platform Architect - Kernel/Drivers
• Responsible for the design and specification of the low-level drivers and kernel interfaces, typically the industry refer to this as Hardware Abstraction Layer (HAL) or Common Driver Interface (CDI)
• In general STB hardware specifications, once a platform launches, undergoes minor updates over time - the interfaces are generally future-proof, and new interfaces try to be backwards compatible, so the Platform Architect is usually required at the start of the project to define the major hardware features and enhancements...

Example scenarios of where this hierarchy is applicable?

Of course, projects don't always have to adopt the complete structure. Real world example projects applicable to this model include: Introducing Hybrid-Satellite/IP features offering Progressive Download (PDL), PushVOD, Broadcast Downlod, PullVOD/IP, Recommendations, Integrated Search, Application Stores, Catch-up, Network PVR, DRM - all these products impact the end-to-end value chain and needs a carefully orchestrated architecture and project execution team.

Why I think projects need to have clearly assigned architect roles?

Projects without a well defined technical team are bound to face difficulty and will most likely fail.

Projects need to have access to a technical team, there must be ownership and accountability for the different areas of the system. More importantly, the product must be connected end-to-end, the value-chain must offer a coherent design solution. Some broadcasters operate separate business units that not well connected. Just like a project team is set-up to manage the project delivery, it is vital to set-up your technical solutions team, assigning people to support the product and project from start to completion.

Take for example, a common project for DTV companies nowadays of changing your STB supplier, be it changing EPG UI or the STB Middleware. For simplicity, assume there are no changes required in the HeadEnd, we're just want to swap out old for new.  At the very least, the project should have the following identified as key roles:

• End-to-End Systems Architect
• STB Architect (doubling as Product Architect)
• EPG UI Architect

At the end of the day, we need to deliver a product. We need to go live and make some money. Having clearly defined roles for architects on your project team will:

• provide clarity, removes confusion over roles, responsibilities & expectations
• highlight gaps in your organisation - for e.g. you might have an EPG architect, but missing a STB architect - this is only going to cause problems down the line for integration.
• provides interfaces and eases communication channels
• promotes a coherent system - without this, chaos looms on the horizon

If you're a broadcaster and lack the skill-set and competencies for these roles, then you should demand this from your vendors. 

I'm an operator, do I really need to go into that level of detail?

Again it depends on how you answered the opening questions, fundamentally how much do you want to be in control?? BSkyB & DirecTV for example, have  STB Middleware & EPG architects; as well as STB & Headend System architects.  BSkyB produce their own Middleware design specification, in addition to requirements. BSkyB are very involved with end-to-end solution design, often taking weeks to review and comment on design proposals. As an operator, BSkyB have a strong technical team who understand the technology implementation of their entire value chain and can influence architectural discussions and decisions from both Headend & STB-side.  It is your product after-all, so why not control it as much as you can? At the macro level, it's down to trusting your suppliers and risk mitigation. Suppose your vendors disappear, do you know enough to support, enhance and maintain your products in their absence??

Useful Reference Material on the Subject

Who needs an architect anyway?

Architecture Definitions

Software Engineering Institute on Architecture definitions

How should an IT Architecture team be organized within a large company?

The Role and Task of the System Architect

The Role of the architect

Difference between coder and architect

The Role of the Agile Architect

Principles of the Agile Architect

Useful book on System Architecting

MIT Coursework on Architecture

MIT Coursework on Role of Architect