Monday, 20 February 2012

Subscription-based Computing (Raised March 2007)


As mentioned in a previous post, another aim of this blog is to share my ideas...I have finally mustered up some courage and guts to start the process of showcasing some of the ideas I've had in the past, as well as ideas currently germinating -- opening up to public scrutiny, in the hope that people leave constructive feedback, or get in touch to jointly pursue ideas going forward -- ultimate dream is to latch onto an idea promising start-up possibilities.  This is partly due to Jeff Jarvis's influence as I'm currently reading his recent book Public Parts, having first read What Would Google Do which inspired me to create this blog in the first place!

I am also wary that my ideas at the time was raised whilst working for a company that is quite secretive and security-conscious. But I'm taking the risk in so far as only sharing my ideas that were raised that lead to nothing, zero, zilch: topic was raised, discussed and not taken forward. Hence I'm confident there are no existing IP-infringements on what I'm about to disclose now as well as in future posts.  After-all, these in fact were my own ideas that were submitted for review to an ideas committee but nothing really materialised mostly because the ideas were not relevant to the core business at the time... I had personally kept a record of all my ideas submitted for reasons such as this post...

Why am I still taking the risk then??  Well, I have a bee in my bonnet really - but I'm not stupid, I've edited our company-sensitive info.  I know that deep down inside of my I've got the entrepreneurial spirit, the mind of a thinker, a rebel and a desire to take risks, disruptive -- although I've not really taken most of my ideas past the initial stages, i.e. move from Idea to concept, to prototype (I have actually pushed one through to completion though, which you'll get to know of on a future post, "Talking TV"). You might think me crazy, but the more I dabble with ideas, the more I read about start-ups and entrepreneurship, the more closer I feel I am to latching on to something for real...it's a feeling I can't shake off, call it instinct, time will tell - I'm only 34, there's years in my yet to be inspirational...

I also want to prove to people out there, especially companies seeking like-minded-people such as myself, that my ideas are not just nuts, because there are companies that are indeed making similar products.  So unlike some folks who say "well I wish I would've thought of that", I am saying that "Dude, I did have the very same idea!" and can show you the proof as well...I want to be noticed (Google, please find me). I want to work for Tech Giants, the likes of Google / Apple / Microsoft - that's no secret, I would love to go to Silicon Valley / Redmond one day to experience the excitement of managing a start-up or working with a start-up...what's wrong with dreaming hey??

You don't have to believe me - it's the truth - that's about it.  I want to share the ideas I had in the past that are actually becoming real now, to not only prove to you that I do have innovative ideas but also to strengthen my case for offering professional technical consultancy...

Here goes:  I'm kicking off my first public idea by provoking some possible contention.  I had the notion of a WebOS, replacing the classic PC OS with an all-in-one-simple-device back in 2007 before Google or HP mentioned ChromeOS or HP/Palm's WebOS.  Of course those giants might've have seeded the ideas around the same time as me, but according to Wikipedia, Google first started work on ChromeOS in early 2009; HP also took forward Palm's WebOS around the same time in 2009.  So I keep telling myself I was two years ahead of these guys....

My idea went further in that Service Providers will have provided boxed, branded-units that basically just worked, was always online and met the basic technical/PC needs of the average PC consumer...Basically I predicted exactly this as portrayed by Google themselves (see video below).   Personally, I still think there is mileage in pursuing this going forward - most probably make a business on the back of ChromeOS - why? Because people still need to customize their applications and environment, ChromeOS doesn't yet offer you stress-free, seamless subscription-based computing. My bet is people will still want something that works out-of-the box, first time, without fuss (no wiring, no wireless contracts - it just works)....

                                                                      oOo                         

Below is a thread of discussions of how the idea unfolded and was left dangling in the air:
                                                                         oOo   


Subscription-Based Computing, Idea was raised on21 March 2007

I am wondering why a subscription-based model to PC computing hasn't taken off, or perhaps has it even been considered?
I'm questioning the need for having a PC in households. The majority of people are not techies, and will never be techies. People want things that just work, without needing any special knowledge, etc. With these assumptions, the requirements are simple:
1. I want to connect to the internet (and I don't care how it happens - it should just work)
2. I want to send email (and I don't want to know anything about POP, SMTP, User name, password, etc)
3. I want to write a document (and I don't care about integrated features)
4. I want to maintain a simple spreadsheet to manage my accounts
5. I want to be able to share photos (and I don't care about drivers, photo software, etc. My service provider can sort this out)
6. I want to watch TV (anything I fancy and no fixed schedule please)
7. I want to play games
8. I don't want to worry about Firewall protection, antivirus, anti-spyware, malware,etc.
9. I want access to my data anywhere, any time.
10. I just want my pc to work out-of-the-box, hassle-free.
And the most interesting part is: I am willing to pay a subscription to a service provider (BSkyB in 30 years time maybe?) I just don't care about the cheapest broadband service, the best AV, the best hardware, I don't want to manage OS upgrades, etc, etc - I want my service provider to do all of this for me - hassle-free - computing!
A PC is supposed to be general-purpose - but is it really that general? What do the majority of users do with their PCs? It would be interesting to learn if any surveys are available in this area...
Can such a system be realised? The consumer just subscribes for the services, and the service provider just serves? Can this semi-closed system survive? Will people be interested in this?
What I'm questioning is the need for people to buy PCs and for people to have the know-how to configure and operate them. The majority of people just want to use the internet, send email, view pictures and watch movies.
So, just as we have a subscription model for mobile phones, digital tv, etc - could the future provide a similar service for computers? For example, I purchase a subscription from Sky. A SkyPC is delivered, it may need installation as usual. But once installed, I have all my basic computing needs satisfied.
This device is still a PC, but a Sky-branded one - not a STB.
The device could be so dumb, it wouldn't even have a hard drive - a simple client connecting to a terminal. The terminal serves the needs of all clients. Ignore the physical limitations of this idea right now, and assume the infrastructure is present in 30-50 years time....
If this can fly, then the winner will truly dominate house-holds... 
Threads of discussion:


_____________________________________________
Sorry for bringing up this old thread again – but I was wondering if we were continuing to do anything in this area? Vodacom SA has released an offering that is not dissimilar to my original idea: http://www.linkbook.co.za/


Basically looking capitalising on the cloud, web apps, and web OS, offering a tightly controlled interface (as we do for EPGs) but still providing a portal to the internet (monitored), offering basic necessities of a browser, messenger, skype, etc. Everything is plug and play, the user basically doesn’t have to be technical.


Have we looked into the idea of being a Data Service Aggregator/Mediator? Why are devices exposing yet another interface for connectivity – 3g/4g sim cards? What if there was a mediator that deals with the likes of Virgin, O2, T-mobile, Orange, etc. I purchase a huge data bundle from all these providers. Consumers sign up with me for data services, they don’t care about the underlying platform offering, they just want to be online. I have a setup, just as current mobile operators do with their management systems, but my system intelligently handles handover of the data service from one operator to another, example – when a user is in an area known for poor coverage from Orange, but excellent coverage by Vodafone – because of my relationship with these operators, the user will seamlessly switch from one operator to the other, guaranteeing a sustained level of quality.
With this relationship and system in place, I can offer always on connectivity to subscribers without them worrying about where they get the data from. This could be extended to hot-spots as well, my system could negotiate deals automatically with hotspot providers without consumers having to worry about local payments, and configuring the connectivity…


_____________________________________________
This is interesting news indeed. I've tried to get the gist of the EasyNeuf (using Babelfish translation - why didn't the French colonize South Africa so I could've learnt French instead :-)


I am interested to learn more about that company; and if that project has had much success. Looking at the press releases, there seems to be some recent publications.


Although the idea of having 3 versions of the interfaces, taking the user from beginner to intermediate and then to advanced level sounds good (as it is empowering the users) - that's not the approach I had in mind originally - as I was targeting the segment of people who just want a box to do the basic stuff - non-technical users: parents, grandparents, or even younger middle-aged individuals who just couldn't be bothered with all the fancy applications - but just want to use their email, surf the net, shop online, socialise, do a bit of budgeting, shopping lists, downloading content, playing movies, listening to music, access to TV, etc, etc. That would be the entry level. The moment we grant freedom to advanced level - then people would want to experiment, tweak, configure, enhance, etc. That shouldn't be allowed - get a PC to do that :-)


Besides, if after paying a subscription of 40 euros (in addition to a 150 euro down payment) - after 2-3 years, the amount spent on subscriptions may well have amounted to the cost of a new laptop, without having to hassle with subscriptions. Having said that, the value add must be so good that consumers stick to their subscriptions because of their peace of mind, ease of use, flexibility, security, maintenance-free and mobility - that can be offered by such a device.


Yes, such as system presents a plethora of technical challenges: network infrastructure (routing, wi-fi, adsl, 3g, etc), Operating System (local OS or thin client), packaged applications, interoperability, etc, etc.


But I share Ronnie's view in that I just cannot see this not happening in the medium to long term future; and quite certainly a disruptive technology.


_____________________________________________
I will point at a french stuff that is exactly what Muhammad is talking about: it was announced last year by Neuf Telecom, and has been launched since then.


They call it ‘’easy neuf’’ and came with a surprising microwave-ish form factor !


Below is a related article in French: http://www.clubic.com/actualite-38705-easy-gate-box-routeur-pc.html


The commercial site is  http://www.easyneuf.fr/


It is based on Intel M600 + 852GM, 512 M RAM, 512 M Flash, linux 2.6.17, firefox, Gimp image soft, multimedia player and a small productivity suite (spreadsheet, text …).


No internal HDD (to be extended with external USB2 disc or memory stick)


Note that the product offers 3 level of UI: easy, ergo and expert (see picture above).


I have no idea where they are today and if they are successful. The price starts at 149 euros for the central unit (oven). See the ‘’tarif’’ (pricing) section on the commercial site.


Funny, even though I share the feeling that there is still something to dig here.


____________________________________________
Yes  - of course! The more comments the better…I really think subscription-based computing is not that impossible - especially when remote storage is becoming cheaper, web technologies providing not only online office apps, but video editing tools as well…there will come a time when the need for a monolithic local OS will disappear - and everything will be managed remotely and distributed…And assuming the problem of getting TV over the internet is solved, you not only have your personalised mobile "PC" but also access to TV…The best part is, no pains of security alerts, upgrades, etc…complete trust in your service provider! Who knows, we may travel full circle going back to the days of thin terminals….
_____________________________________________
Thanks again for replying. Perhaps the word "PC" is conveying the wrong meaning - I envision a closed device (could be a laptop, could be your STB Gateway, or STB itself) with a wireless keyboard (so you could use your HD tele) or indeed - a laptop-like device.


The device can only be tweaked and configured by the service provider, like [ServiceProvider]. Call it the "[ServiceProvider] Explorer" box - that meets all your internet or computer needs…


Think about it - how many people really use their PCs to their full potential? There should be some research to back this up: Mostly browse the internet, check email, online shopping, a bit of Word processing, and a bit of spreadsheets. No more than that. So a box that is always on, does not need tweaking, can interconnect with other devices (printers, cameras, etc) - and provides not only the security for online shopping - but also peace of mind because of [ServiceProvider]'s reputation with parental controls ( XXX's business)…


[ServiceProvider] could even have their own secure payment system ( XXX to provide the technology of course)
[ServiceProvider] would have to have their own OS (Linux or Fusion) - ( XXX to provide of course)…
[ServiceProvider] will need headend control -  XXX to provide of course… :-))


What a stir this would cause to Microsoft and other OEMS :-))


_____________________________________________ 
I'm sure you're probably getting loads of emails for review of ideas - I just thought I share this one more idea with you - and if you could provide your comments/feedback/gut-feel - that'll be great :-)


At a very high level: Provide box-standard, packaged solution - of a personal computer - providing:
- always on access to the internet (email, chat, browser, etc)
- office applications
- security (viruses, malware, etc)
- parental controls (filtered view of the internet) - using some sort of entitlements
- unlimited remote storage
- free maintenance, support, upgrades, etc
- usual share photos, content, etc - USB, plug and play, etc…
[Nick>]  Don't you think that people like HP, Dell and others already aspire to this?


Picture this: [ServiceProvider] offering the [ServiceProvider]PC. Engineer comes home, and installs the box. You don't need to worry about broadband tariffs, ISPS, Viruses, Software, Phone line, etc, etc. You have a box standard, all the tools one needs to meet the basic computing needs…offered by service provider.
[Nick>]  I agree completely with your proposition, and I think this is where [ServiceProvider] would like to go. The reason that they might shy away from it just now is a) most PCs are very user configurable, and this can lead to some very nasty support call headaches. With PCs (and even Macs), there isn't a viable business in supporting home users.


The closest I have seen anyone come to this is the One Laptop per Child project (OLPC). If we could reproduce PC functionality with STB levels of reliability (you may laugh, but they are much more reliable in practice), then you might get close to your vision.


It works for TV, mobile phones - why can't this model work for the PCs.
[Nick>]  Because its very easy to mess them up as a user.


The majority of people just want something that works - not technical gurus - they don't want to control their pc environment - it's a headache really…
[Nick>]  So how about something which is more like a PC latop then, or more like an embedded device, and heavily cost reduced. Maybe very carbon efficient too.


What about parental controls for the internet?? Surely  XXX can offer something in this space?
[Nick>]  Looks like a good idea


I've blogged about this here: http://XXX


What do you think?
[Nick>]  We need to think about what the functionality as a user would really need to be, to provide something that would need little or no user support.


Cheers,
Muhammad


Tuesday, 14 February 2012

My Professional Software Engineering Background...Part 2

Latest update 2022 - CLICK HERE to read my reflections on the 22+ years of working, then read below for details of achievements.


Okay, as promised in an earlier part one post, I will attempt to describe my work experience:

                                                                               oOo


    Public mentions of projects I helped deliver:
    DStv launches 4K Decoder and OTT Streama Device

    Tuesday, 7 February 2012

    White Paper Preview: Technologies for Internet TV



    I am working together with a friend on a joint white paper covering popular technologies around Internet TV. We've been working with this on & off for a good few months, in between work, balancing family needs and what little spare time we have.  The idea is to share our skills & experiences through white papers, thus attracting offers for consulting...I'm releasing this very draft work in progress to act as a motivator for us to actually finish this white paper in 2012:

    If you stumble across this post and are interested in this topic, please get in touch.

    -----------------------------------------oOo-------------------------------------------------

    General Overview of IPTV Technologies, covering Online Portals with media consumption via PC, Internet Streaming technologies, Client implementation (PC, STB, TabletPC), Open Source tools & Home Networking


    Increasingly Telcos and Broadcast operators are looking into increasing their revenue generation stream by providing multiple mechanisms for customers to consume entertainment content over the Internet. These businesses usually decide that the achievement of this objective can be realised by having a web (or online on-demand) presence by creating a platform for delivering content over the Internet.

    The road to success lies in solving a multitude of issues in a variety of domains including marketing, sales, legal, and technical. This white paper focuses on the technology choices and challenges for implementation, providing a discussion point around a high-level, simplistic solution intended to highlight the challenges these businesses face when creating the solutions.

    So from the construct that a business decides to have a online presence and wishes to provide a entertainment video streaming solution to augment the bottom line the following issues needs to be resolved:
    • Rights to Content for distribution
    • Marketing the solution against other vendors
    • Technical solution to implement the portal and distribution mechanism
    • Support for customers
    Rights to Content for distribution
    This is key to the success of the project: If the telco is unwilling to acquire the right mix of content based on its targeted subscriber base the project is likely to fail.
    The fundamental issue impacting this area is getting the content providers to allow distribution of the content over the Internet in the background of security concerns where it seems nothing is safe to hacking attacks on the Internet. Content providers look for assurance that their content will not be compromised and redistributed without them realising revenue.

    This usually requires utilising a Digital Rights Management (DRM) solution into the product where the content provider feels satisfied that the protection mechanism is secure enough for them to use the solution.

    Choosing a DRM solution is an an activity that involves thorough planning right at the outset from designing the architecture, as this is a fundamental component to the system architecture, from back-end distribution to consumption on the end-consumer device. A number of DRM-solutions exist in the marketplace, this paper will focus on the popular (not necessarily most secure) solutions including Microsoft Windows Media DRM, Apple Streaming DRM, NDS Secure VG DRM, …

    Marketing of solution against other vendors

    Since businesses need to differentiate in the competition space, focus is made on a variety of factors such as price, product features (integration with devices, social networks, widgets etc) and type of content (niche versus premium) amongst others.
    Companies with existing Portal solutions (TODO: Give example references) generally adopt a common theme of marketing, specifically centred around on-demand, instant access to content, variety of content, consumer freedom to choose to watch “whatever you want, whenever you want, on any device, anywhere”. Of course this is music to the consumer’s ears but the proof of the pudding lies in the actual implementation - and to realise this marketing objective is to overcome a number of tough technical challenges as we’ll focus later in this paper.

    Technical solution to implement the portal and distribution mechanism

    The technical challenges in relation to the above two is a simpler problem to resolve and relates to creating the platform, infrastructure, know-how of a wide variety of issues that encompass the following areas:
    • Online Portal
    • Subscriber Management
    • Payment System
    • Content Management including Acquisition, Processing and Distribution
    • Content Consumption by devices
    • Support Systems
    • Regional Infrastructure

    Online Portal

    This is the gateway into the offering by the business; the user experience greatly drives the success or failure of the system. The Online Portal must provide features for:
    • authenticating the user;
    • managing the user’s profile;
    • billing information;
    • social network integration;
    • allow searching for content; and
    • placement of  recommended content for the user based on their previous usage; and
    • YouTube-style features to allow the user to follow trends and watch what other users of the system are watching, have recommended / commented upon etc. In essence this is the public face of the content management system which is integrated with the billing system and distribution system.


    This is an intensive area requiring expertise in creating a very usable interface for all types of users. Along with this, the main feature is to allow consumption of the content and is typically based around using a Flash Player (TODO Reference to Flash) providing a compelling experience in comparison to the user’s TV.
    There is a tremendous amount of integration work required, typically focused on:
    • adaptation of the user experience across all web-browser platforms
    • the content player that renders the content based on the format of the content
    • what audio/video players are available
    • what devices are required to be supported
    • what features are to be supported such as the protocol for delivery of content
    • support for degradation of video quality as the bandwidth available to the user fluctuates based on usage pattern in the user’s home;
    • the bandwidth available to the user;
    • the pathway between the source of the content on the Internet and the user’s consumption device.
    • Additionally other features including the DRM technology dictates how the interaction occurs.


    We shall aim to discuss each of the above points in further detail in future sections.

    Subscriber Management

    This domain needs to be well understood by the business model. It broadly encompasses the following areas:
    • uniquely identifying the user, the device (including the limitations of distribution profiles) being used for consumption;
    • business model restrictions based on subscription tier, promotions, discounts, billing information etc;
    • the management system integrates with the online portal in addition to the content management system as well as the payment system to realise the revenue.

    Payment System
    This is typically a legacy or third party system that the operator utilises and integration of this to the online system needs to occur.
    <TODO add more info>

    Content Management including Acquisition, Processing and Distribution
    This is a significant aspect of setting up the process flow and delivery chain for content consumption by the operator. Depending on the investment, an operator budgets for the solution that need to be defined encompassing the following:
    • Setting up infrastructure to acquire content from diverse content providers
    • Automating the ingest of content in to the system
    • Preparing the content for consumption of the variety of devices the platform is intending to support, ensuring that the correct DRM aspects are catered for each type of end user device
    • Managing the content through its life cycle which may entail marketing aspects such as promotions
    • Delivering the content through a variety of delivery mechanisms requiring integration with third party vendors


    Content Acquisition

    This area involves a plethora of decisions that need to be made about the content and how it becomes available on the platform and ultimately how it gets delivered. Issues to resolve include:
    • What format the content (audio, video) is provided in by the content provider?
    • What metadata is required for presenting to the user on the online portal or during playback and how that is transformed from the input all the way to the point of consumption?
    • Whether encoding services (software and or hardware) are required to transcode the content to the end device format?
    • Whether features such as adaptive bitrate streaming is required?
    • Whether support for live, restart, catch-up, on-demand playback of content is required?
    • What third parties to select and what level of integration is required?
    • What licensing issues resolve with content owners?
    • What licensing is required for encoding to different device formats?


    Content Processing

    Once the content has been ingested into the platform, decisions around the following areas need to be resolved that include:
    • What is the life-cycle of the content?
    • What promotion mechanisms are required?
    • What DRM(s) need to be applied and what business models will be applicable?
    • How is the content encrypted (dictated by the DRM(s) in use)?
    • How will the variety of DRM(s) work seamlessly as the user begins consuming content from a variety of devices?
    • What extra metadata needs to be added to support different types of playback and interactivity during consumption?
    • What support is required by the systems administrators to manage outages and issues with the process flow during deployment?


    Content Distribution

    Once the content is available for distribution it needs to be made available for consumption over a variety of mechanisms; issues that needs resolution include:
    • Protocol for delivery on the network (RTP, RTSP, HTTP, RTPM)
    • CDN (Content Delivery Network) support dependent on protocol choice
    • Commissioning of hardware resources to serve content to devices
    • Integration with DRM system(s)



    Content Consumption by devices

    Dependent on the number of devices being supported a large amount of effort is used in integrating the Content Player in to the OS (Operating System) framework of the end-device. This entails a variety of issues:
    • Choice of devices (PCs, Macs, Tablets, Smartphones, Set-Top-Box, SmartTVs)
    • Player Usage (such as Flash/Silverlight/QuickTime/Windows Media Player/Custom) of player using bespoke or open source software (typically involves providing plug-ins for web browsers on desktop (Windows/OSX/Linux) platforms and integrating to the mobile / tablet devices OS framework
    • Integration and maintaining integrity of DRM(s) on devices
    • Fine-tuning user experience of UI and play-back
    • Supporting integration between devices via UPnP (Universal Plug & Play) and DLNA (Digital Network Living Alliance) to support download and play-back between devices


    Support for customers

    This area revolves around trouble-shooting, resolving issues due to plurality of devices and the versions of software they run and creating baselines for the operator to know what issues cause outages and rectifying them.

    Infrastructure capacity

    A successful provisioning an online content portal with capacity for multiple device streaming to the home relies on fundamental components of infrastructure support being in place. Not only does the business need to ensure that it can deal with system loading (hundreds of thousands of subscribers accessing the content simultaneously; protection from attackers intent on crippling the service through denial of service attacks) but more importantly the business would have carried out an analysis covering the following areas:
    • Number of subscribers with online access, i.e. able to access the Internet?
    • Intended delivery mechanism for streaming (xDSL, WiFi, 3G, etc)
    • Solutions for supporting a number of different delivery profiles
    • Regulatory compliance - what relationships or conditions are in place to allow/prevent delivery of content over third-party networks (e.g. not all ISPs will allow traffic through)
    • Content Delivery Networks - is enough infrastructure in place to support the demand (Edge servers, etc)
    • Reliability of the underlying delivery network?

    Infrastructure and Regulatory concerns might not be an urgent topic to address if the business is operating in a developed country where access to Internet is now considered a basic utility, but for other parts of the world, especially the Indian/African countries, Internet adoption and underlying network infrastructure has not reached the maturity levels as say, compared with countries like the United Kingdom/Germany where there is a high penetration of cable supplying xDSL for instance.  In South Africa for example, xDSL is not the main player of Internet connectivity, instead 3G WiFi is the dominant player - and for such markets, the business must therefore have a solution that meets the variety of access mechanisms in front of the user.

    <TODO Examplae Case Study>

    We’ll now walk though a general architectural overview of the components in and end-to-end system, assuming the required infrastructure exists...

    Friday, 20 January 2012

    Appreciating the Role of System Integration


    Last year I started my series of posts around the different aspects of DTV system projects. Starting out by describing the a typical end-to-end system architecture and the importance of clearly identifying the roles and responsibilities of the architect team (see post: Architect Roles), followed by a write-up on considerations for setting up a programme organisation structure (see post: Managing a DTV Programme) and more recently, a brief overview of auditing the STB System Integration process (see post: SI/Dev Process Audits).

    I still have a few more topics in the pipeline, however the aim of this post is to highlight the worlds of System Integration, as this can often be overlooked and open to interpretation, introduces dangerous assumptions and automatically puts the programme at risk.  With a large-scale DTV deployment, the programme must at the outset, clearly define the boundaries of system testing, and establish clear roles for the system integration effort.

    This post is especially relevant to TV Operators deciding to become more involved with technical ownership of their value chain, as previously highlighted, more and more companies are taking the plunge into doing their own Software Development, and System Integration. This post is another humble attempt at raising awareness that such operators should proceed with caution, ensuring the full extent and gravity of the undertaking is understood, because System Integration is a significant investment in people (highly experienced technical experts required), equipment (investment in costly infrastructure) and time (starting from scratch, building this competency from the ground up is at least a 5 year maturity period).

    Recall the end-to-end architecture originally present a few posts back:
    As highlighted in previous posts, the world of Digital TV consists of Broadcasters / PayTV operators employing one of more component vendors or Technology Service Providers offering specialist services of Software development for Headend/STB components, Conditional Access Providers, Billing, Scheduling  & Play-out Systems and System Integration services.  This is made possible by basing the various technologies on open standards (MPEG/DVB/IET/IEEE) and thus allows for creating an end-to-end system consisting of a variety of components from different, independent vendors.  However, there are some TV operators who opt to use one supplier, often referred to as the Primary System Integrator, who are tasked with not only ensuring the end-to-end system is architected correctly, but quite often, the Primary System Integrator provides the bulk of the system components end-to-end (i.e. dual role of component developer supplier and integrator).

    System Integration and Architecture are linked, they go hand-in-hand. They are not separate entities, although in theory the act of architecting is different to the act of integrating, but integration drives the architecture.  Looking at the above figure showing the different architectural streams, the following becomes clear:
    • Solutions Architect is replaced with Solutions Integrator
    • Headend Architect is replaced with Headend Integrator
    • STB Architect is replaced with STB Integrator

    This now exposes us to the world and levels of System Integration.  A Programme needs to define its System Integration strategy at the outset, success of the programme depends on a solid System Integration strategy.

    If you're a broadcaster taking ownership of SI, you need to decide in what capacity you will be involved in:
    • STB SI
      • This is generally taking responsibility for integrating the various components of the STB stack (typically Platform Drivers, CA Component, Middleware, UI & Interactive Engines) providing a stable build (STB image).
      • Interfacing with a broadcast/backend is part of this responsibility but STB SI assumes the necessary backend/headend infrastructure is in place
      • Responsible for managing and driving through component deliveries in terms of quality expectations and to some extent project time-lines
      • It is always useful to have access to component source code to aid in investigating and characterising bugs, but this isn't always possible as it has commercial implications with IPR, etc.
    • Headend SI
      • Within the Headend system itself, there are a number of component systems that must come together in implementing a specific macro-component features, like Conditional Access, Scheduling, On-Demand Catalogue/Playout, IPTV streaming.
      • Generally though, headend SI is about integrating these headend components either individually or collectively, proving the interfaces are working as designed (i.e. verifying the input/output is as specified by the protocols, essentially obeying the Interface Control Definitions ICDs)
      • Test tools might be used as test harnesses, or the STB itself can be used depending on the situation
      • There is a fine line between Headend SI and End-to-End SI or Solution SI, which can often lead to confusion and false assumptions
    • End-to-End SI or Solution Integration
      • This is about testing the solution in a representative environment, as close to real life operations as possible
      • It brings together proven systems at the component level, i.e. STB, Headend and other dependent infrastructure systems like Transport (Encoders, multiplexors, IP networking), Billing & Subscriber Management
      • There is often a dependency or interface with systems that are currently live (live components are considered crown jewels, usually managed by a separate delivery/operations team).
      • Failover, Recovery and Upgrade scenarios are performed as part of this exercise
      • Requires investment in one or two end-to-end labs (Some broadcasters impose this on their major vendors to ensure appropriate end-to-end testing is setup locally at the vendor premises; and some vendors have end-to-end labs set-up on the operator's site)
      • Often confusion around how this activity defers from System Delivery/Operations - think of this as pre-delivery, the last mile of the programme's development phase.
      • Becomes really important to define an end-to-end SI strategy where the overall DTV value change is a conglomerate of components provided by different business units offering different services, e.g. a Broadcaster might have separate departments, sometimes independent units each offering a slice of the system (Broadcast, Interactive, VOD, Guide, Operations, R&D) - all these business units must come into alignment at the outset of planning the programme...
    In a nutshell, the following diagram tries to illustrate the worlds of SI, the intersections shaded show the typical integration points required. The worlds are interconnected, and can't be expected to be tested and qualified in isolation of other systems (during development phase, unit/component testing is expected, but as soon as we have stable elements to make up the E2E system, then its time to kick-off your End-to-End SI activity). Typically, E2E testing must start well in advance of launch/go-live, typically 6-8 months of E2E testing is not unheard of...
    Worlds of SI


    The world of Delivery/Operations
    The absolute last mile in completing a DTV Programme, is the actual launch - go live!  This is usually under the control of an Operational entity who's sole purpose is to ensure the smooth operational deployment of the systems on live - 99.9999% of reliability is expected, the transition from End-to-End SI to Live Deployment must have the highest probability of success.  In parallel to the E2E SI activity, there is usually a Delivery/Operations team that are mirroring the E2E SI effort, making preparations for launch.  This delivery team must be included in the Programme Schedule from the start, so they can make the necessary equipment procurements, network configurations and deployment schedules in advance to ensure the smooth transition.  Practically though, the delivery/operations team might re-use the same people from E2E SI, but it is important to ensure that everyone understands the milestones and phases of this activity.   Forward planning is crucial to avoid ambiguity and confusion down the line, which goes back to my point earlier that the boundaries and scope for each Testing/Integration activity are clearly understood.

    As always, I'm interested in your comments and feedback. 



    Thursday, 15 December 2011

    SI/Development Project/Process Audits...


    Earlier this year I wrote about setting up the right Roles & Responsibilities in managing a typical DTV project, covering the Architecture & Project Team requirements.  In this post, I'd like to share my experiences with the Project Audits.

    The scope of Project Audits is in itself a vast topic; mostly around the area of auditing the Project Management aspects of the project, along with ensuring effective Quality Management processes are in place, proper Project Governance is enforced, etc. The IEEE SWEBOK dedicates a chapter on Sofware Quality, of which Project Audits is considered as part of this knowledge-area, summarising the process as:
    “The purpose of a software audit is to provide an independent evaluation of the conformance of software products and processes to applicable regulations, standards, guidelines, plans, and procedures” [IEEE1028-97]. The audit is a formally organized activity, with participants having specific roles, such as lead auditor, another auditor, a recorder, or an initiator, and includes a representative of the audited organization. The audit will identify instances of nonconformance and produce a report requiring the team to take corrective action.

    While there may be many formal names for reviews and audits such as those identified in the standard (IEEE1028- 97), the important point is that they can occur on almost any product at any stage of the development or maintenance process.

    What to audit in DTV Projects?
    Again, with a bias towards the STB Software Development delivery project, audits are introduced to review two main areas:
    • STB Development / Test Processes across the stack (Drivers, Middleware, Applications)
    • STB System Integration Processes
    If you're a product vendor supplying any of the STB components, you might want to review and assess your own development/test processes to gauge how well you're doing against industry best practices. It's also good for your portfolio when pitching for new business to potentional operators.

    If you're a TV operator and dependent on product vendors supplying components to your value chain, you need to ensure that your vendors are indeed delivering quality components & systems. Typically (and this can vary with the type of contract in place), the Operator (customer) provides the Product/System requirements to Vendors; Vendors implement and test their components according to the spec; produce test results (self-assessments), making it difficult for the Operator to dig too much into the detail, to find out exactly how the results came about, or gauge the quality of the underlying development processes, typically until it's too late in the project life-cycle.  This can of course be remedied by building into the contracts the requirement for auditing the vendor's various processes, at any stage of the project life-cycle.

    Some operators outsource the whole testing and verification effort to third-party test houses. These test houses, depending on the strength of the commercial contracts in place, could very well take complete ownership of the test & quality management, and enforce audits on encumbent component providers.

    As noted in my previous posts, more and more Operators/Broadcasters are taking more ownership over the product development, testing and integration. Take for instance, the task of STB System Integration.  Whilst there are well known companies like S3 & Red Embedded, that specialise in STB SI, operators like Sky, DirecTV, Foxtel, Multichoice have dipped their feet into the waters; attempting to own STB SI activity internally, in-house. Incidentally, there are more players on the STB SI front, some STB manufacturers like Pace, UEC; as well as service houses in India like Wipro & Tata Elxsi. NDS Professional Services is also a well known player in the System Integration field.

    In my rather short stint of 11 years with STB projects, the world of DTV is quite small; the popular companies of choice for SI, from my experience are:  S3, Red Embedded & NDS.

    In my current company, being a DTV Operator/Broadcaster, we are running our own development & STB Integration projects. Because we're relatively new in this game, having built up a good team of people, making every effort to source the right people with the right skill-set (which is really challenging in South Africa), we'd been through a few rounds of Development & SI, using third-party vendors supplying Middleware & Drivers, we decided now was a good a time as any to perform a self-diagnostic, a review and audit of our current processes, specifically focussing on STB System Integration and Supplier Deliverables coming in. Although we have enough knowledge in the team to perform a self-assessment ourselves, and despite having learnt and corrected a lot of mistakes from past experience; we opted for a professional consultancy to do the audit, and chose S3.

    Why we chose S3?
    S3 despite being a relatively small company in Ireland, has been involved with TV technology for many years; and is well respected in the DTV community; for their specialist design services & system integration knowledge. They've leveraged the wealth of past project experiences in producing useful products like S3 StormTest & Decision Line, a STB automated testing system that has only in increased in popularity since its inception (which we're currently using by the way). The S3 Engage Portal is also another useful product to manage and co-ordinate a complicated DTV SI project.  S3 are also known to be frank & objective, offering feedback and analysis that is open, without hiding the nasty details. S3 were chosen as SI for some big players in Europe; and they're also being used in some other projects of ours.  Based on strength of previous relationships; despite the fact that I previously worked at S3! The world of DTV is small, the other companies mentioned above are also competent and capable.

    What areas do you focus on for an SI audit?
    Of course I can't share with you specifics of the S3 Audit process, but if you're interested in learning more please contact S3, or speak to this guy.
    In a nutshell, S3 shares a model of SI that is well known to the industry:
    • Incremental / Iterative approach, built on a strong foundation of domain expertise, using experienced System Integrators.  The foundation cornerstones that deliver an effective SI process, rely on the following:
      • Strong Technical Expertise
      • Strong Project Management
      • Usage of Advanced Tools & Infrastructure to support the process
      • Strong Quality Assurance Practices
    S3 have been kind enough to allow me to publish a slide on their overview of SI process:
    S3's Integration Model: SI is a continuous cycle once all Components are in a suitable state of completeness

    Based on the above cornerstones, the audit focuses on 48 topics of concern, that largely fall into the following categories - again, the below is generally best practices as evoked by IEEE SWEBOK Chaper 11 on Software Quality which proves S3 appreciate Industry best practices:
    • Project Management
    • Co-ordination & Communications
    • Requirements Management
    • Defect Management
    • Testing
    • Quality Assurance
    • Supplier Management
    • Release Delivery
    • Resourcing
    • Software Architecture
    • Integration & Configuration Managmeent
    • Software Development

    My Own List of Audit Criteria
    Since this knowledge comes from my own past experience, I've no problem sharing information with with you. If you want more details, feel free to contact me, or leave a comment. The list below is something I put together in advance of the audit we've just undertaken.

    If I were doing a STB audit, this is what I'd be interested in:
    • Foundations of the STB Component (Middleware/UI) architecture design
      • Design philosophy – apart from being MHP/Jave/MHEG/etc compliant, what is the design ethos behind the architecture, i.e. does it scale?
      • Design patterns – what abstractions are used to ensure flexibility with changing requirements without major impact to design?
      • Performance requirements – is the product being architected with performance considerations in mind?
      • Security considerations – is security checks in place, to prevent buffer overruns, etc?
      • Stability considerations – how do we ensure stability checks are enforced in design / implementation?
      • Open Source components usage – if open source is being used, is there a process for choosing components, etc?
      • Inter-component Interface design and control – assuming the middleware is a collection of components, how are component interfaces managed?
      • Documentation – Requirements, Design – is there a documentation standard in place? Middleware design, component design, etc?
      • Process for introducing change to architecture? New components to implement new features? – How are new components introduced to the system?
      • API ICD Controls – how do we manage API changes??
        • Is there an SDK?
        • Quality of API Documentation? Is it usable, self-explanatory, auto generated??
        • New APIs must be communicated to application developers
        • Changes in APIs must be controlled
        • APIs must be backwards compatible
        • API Deprecation rules

    • Testing the STB (Middleware/UI/Drivers) stack – what is the testing strategy employed?
      • Depending on the architecture, assuming it’s component based architecture – do we have a continuous build and integration system in place essentially for regression testing:
        • Component based testing – unit tests on the component, using mocks for other components, assuming interfaces are controlled
        • Component group testing – where you collect related components together and test as unit, e.g. PVR engine can be tested in isolation
        • Middleware stack testing – test the middleware as a whole, i.e. collection of components forming the MW
      • Regression Suite – does one exist, how is it tested, and how often is it tested?
      • Spread of STB hardware platforms used for testing
      • Use of a Simulator
      • How often do we check for regression – daily, weekly, just before release?
      • How is new feature development controlled – i.e. new features to be delivered without causing regression in previous working functionality
      • Is there a concept of a middleware integration team?
      • Middleware development is separate to middleware integration, separate teams
      • Development teams do component development, and component unit testing
      • Middleware integration team takes development components and integrate – test new features against last stable middleware as well as test new functionality
      • Middleware integration team consists of technical people who can code – best practise is to use a middeware test harness that exercises the middleware, not just the external Application APIs, but speak directly to component APIs themselves
      • Is there a separate team owning the typical STB Hardware Abstraction Layer / Driver Interface?
      • Who defines the HAL spec / interface?
      • Change control procedures in place to support HAL spec?
      • Testing of the HAL?
        • Is there a test harness that validates new platform ports?
        • Is there a separate Platform Integration team?

    • Supporting multiple customers and build profiles
      • How does the architecture support the needs of different customer profiles: Satellite / DTT / Cable / IP ?
      • For the same customer with different project requirements, how is the codebase managed?
      • Build, Release, Configuration Management and Bbranching philosophy?

    • Multi-site development
      • How is multi-site development managed?
      • What configuration management system is used? Does it support multi-site development well?
      • Are off-shore teams used as support arms, or do actual development?
      • So off-shore team have any ownership over components?
      • How is communication managed with off-shore teams??
    • Planning releases
      • How are new releases planned?
      • How is work assigned to development teams?
      • What is the usual process for planning / starting new development?
      • How much time is spent designing new features?
      • Is testing included as part of the initial design review?
      • Are new features delivered back to main product code tree in isolation or in big chunks??
      • How much time is allocated to testing on reference platform?
      • How much time is allocated to testing on customer production platforms?
      • How is the backlog prioritized?
      • How long is the development cycle?
      • What is the exit criteria for development?
      • How long is the test cycle?
      • What is the exit criteria for testing?
      • Is Middleware integration part of planning?
      • Is full-stack integration part of planning?
      • How are defects planned?
      • Does the plan cater for stabilization period for new features?
      • Who decides on the branching strategy nearing up to release?
      • Are new tests identified as part of the planning process?

    • Defect Management
      • Is there a defect management process in place?
      • How are defects managed and factored into the development process?
      • How often is root cause analysis performed?
      • How is quality being managed and tracked internally?
      • What defect tracking tool is being used?
      • What metrics can be generated to gauge the quality of all components in the stack??
        • Average submission rate
        • Closure rate
        • Time it takes to close the average Showstopper
        • Most suspect components
        • Most complex components

    • Development practises – industry standard coding practices should be in place
      • Coding standard
      • Architecture rules – e.g. system calls to avoid, avoid using strcpy, etc?
      • Is CUnit test framework or any other suitable framework used for native component testing?
      • Is JUnit being used?
      • Do component unit tests map to component requirements?
      • Do middleware tests map to system requirements or use cases?
      • Are any of the following code quality metrics being looked at:
        • Zero compiler warnings
        • Zero MISRA warnings for native C components
        • Which Java best practises being used?
        • Code Coverage:
          • Line coverage tools
          • Branch Coverage
        • Code complexity
        • Function points
        • Other static analysis tools like Klocwork
    • Development organisational structure
      • Size of development teams
      • Structure of development teams – is it based on architecture components, i.e. notion of component teams, with component owners?
      • If using multisite development, what is the line management structure – is the team a matrix, with virtual management teams?
      • Do you have separate groups for:
        • Development: Separate UI development, Separate Middleware team?
        • Integration – is there a concept of middleware integration team?
        • Platform – is there a concept of a team specialising in low level HAL/Kernel/Drivers/Platform
      • What are the defined roles and responsibilities:
        • Product Architect  - the person owning the product architecture, maintains the integrity of the philosophy, keeping everyone in check?
        • Middleware Component Architect – do you have people owning specific components in the middleware, ensuring the interfaces are respected. For e.g. PVR engine usually is a separate architect?
        • Lead Developer / Component Owner – someone responsible for delivering component to spec, owns the internal design and implementation of the component?
        • Engineers – what is the mix of people and skillsets across the organisation