A blast from the past: my experience building a large-scale tech platform

In the years 2003-2011, I worked for a pure technology service provider, NDS (acquired by Cisco in 2012, then later became Synamedia) which was considered at the time, the world leader in end-to-end digital TV software systems. I was fortunate enough to experience as an engineer every major area of platform development for this complex ecosystem; and then later as a software manager, I would own the software delivery for a core piece of the software stack known as "middleware", for NDS's primary anchor customer BSkyB/Sky Darwin and then later would own the full stack delivery of NDS's flagship Mediahighway Fusion/Unity product. This experience would mark my entry into very complex large-scale technology delivery initiatives, which even to this day, thirteen years later, as I work with the world's largest cloud provider, Amazon AWS, in building out its enterprise cloud support systems (AWS Support Center / Technical contact systems), Fusion still takes the prize for the most intense professional experience, learning and growth, technical complexity, risk and high-stakes projects. So yeah, I find myself having to dig deep into my memory to recall this work experience because it's funny that 13 years on, I'm encountering the same topics of engineering management even though it is supposed to be a different domain, turns out "software is just software"!

NDS had captured almost every top-tier PayTV operator around the globe at the time: Sky, DirecTV, UPC, Sky Italia, Sky Deutschland, Foxtel, Sky LA, Yes, Bharti, etc. NDS was prominently known for its conditional access product, a video content protection system call NDS Videoguard, however, NDS offered more than just security and offered customers a fully vertically integrated ecosystem (think "Apple" ecosystem for PayTV customers). Whilst digital TV was built on open standards and interoperability, most customers limited their integration points. So when they opted for NDS as their security provider, they also had the option of integrating all other services - from broadcast backend services in the headend to consumer device hardware development and software service integration with chipset vendors. The consumer device software was known as TV Middleware. At the time, the main players were NDS Mediahighway, OpenTV & TiVo. NDS was known for convincing customers to migrate to NDS Mediahighway, its technology migration programs were demanding, complex and executed flawlessly. As an engineer, I contributed software to replace TiVo, an overnight win for 40 million devices. Later as a software delivery manager for the Sky Darwin migration project, we would replace OpenTV software almost obliterating its presence from Sky, save for a few ancient, ageing hardware profiles.

NDS, with an increasing number of customers using its security, middleware and application services, couldn't afford to scale out with engineering teams for each custom build. A platform strategy was needed, consolidating the best of software from across the globe (US, UK, India, Israel, France) into a new shared technology stack, that offered flexible customisation and tailoring for any type of customer profile (Tier-1 customers like Sky for advanced applications to Tier-3/4 customers in territories just starting off with basic digital TV), using a shared engineering resource pool - and extensible configuration engine for producing tailored custom releases. So was borne, NDS Mediahighway Fusion.

The flagship customer for Fusion was Sky, which went live in 2010, replacing up to ten variants of its consumer device software services, with new Fusion components and Sky's own custom-developed consumer application "EPG" known then as the "Orchid EPG". Fusion provided an SDK/API for customers to develop their own primary applications, along with an interactive HTML engine, that allowed PayTV operators to add additional mini apps to their devices, like games and weather apps. With Sky being the anchor customer, Fusion had proved itself in the market and thus was ready to onboard new customers like Sky Italia, UPC, Foxtel, Yes, etc. Post Darwin launch, I took the lead for building the new platform vision, called Fusion Snowflake EPG through project Sunrise - birthing the platform that would create customer, tailorable configurations for any customer, maximising reuse and minimising customisation but allowing for a selection of custom user experiences.

Why am I claiming Fusion as large-scale (even in 2023, 13 years later)?

I write this in 2023, after spending 2.5 years with Amazon AWS. I am part of the group that build AWS Support Center and related Contact Center services. We are a team of under 100 people, deemed large- scale and building complex systems. Yet, if I have to be brutally honest with myself, I'm mildly impressed by my exposure to date, because my current work pails in comparison to my work on Fusion, 13 years ago. Yes I know it's a different domain, a different paradigm and culture of Amazon's 2-Pizza team model for software product ownership (which I actually find quite cool)...still I'm finding it hard to rationalise my move to AWS almost 2.5 years on, have I gone too far backwards? Am I living too much in the past & not ready to view things from a new perspective? What am I not seeing? (Topics for another post). So whilst I've defintely adapted my mental models since joining Amazon, yet I really can't ignore some software engineering truths which is the reason for my bringing up the past now. 

In 2012, I wrote the first story about Fusion, introducing the term LSSDP I coined to mean Large Scale Software Development Project. I also dived deep, writing lengthy white papers about the product and engineering management processes:

• How we "scaled agile" to adapt to our unique global challenges
• How to design a global org-structure to deliver a large-scale project in a scaled agile way
• What mechanisms are needed to deliver a large-scale technology initiative in an agile way
• How we executed and raised the engineering & product management bar

Fast forward to 2023, now using my Amazon AWS experience as a lens for defining a large-scale initiative and indirectly checking engineering manager role guidelines for large-scale:

• Business Impact - Fusion started off with a $75 million investment and later a joint-venture with the flagship customer, Sky. The entire company pivoted to focus on Fusion as its next-generation software platform, with up to 3000 engineers world-wide working on multiple streams, some strategic foundational streams kicked off at least 2 years before the mainstream program. In my role as software delivery owner for Sky Darwin project, it was critical the project delivered successfully, flawlessly - as it involved migrating software in 10 million people's homes (their living room TVs) seamlessly with no rollback. To the end customer (the person sitting at home watching TV), they would notice very little change to their experience. Overall, Fusion software components delivered to multiple middleware stacks, at the time of 2011 when I departed NDS, our software was running in excess of 60 million people's homes daily, globally.

• Scope and Size - Fusion introduced a new paradigm of the TV software ecosystem, end-to-end, including broadcast headend components as well as embedded software architecture. The stack was open, based on a Linix/Posix and a complete departure from the initial decade of TV software operating systems. This was before the advent of Android TV or fully open source middleware. Fusion's product backlog captured over 2000 epics in the form of work packages, cutting across multiple customer needs, in parallel. The scope included all layers of the device software stack: Chipset drivers, hardware absraction layer, Linux kernel, Linux abstraction, Middleware services, Application SDK/APIs, multiple frontend application engine proxys for C / C++ / Java / HTML / Flash applications. Take a look at the software architecture diagram - it is multi-layered, multiple service teams. Another point on scope, we managed initiatives or epics in the form of work pacakages (WPs), that could impact up to 25 service teams in one WP, see here.

Product Plan visuals - concepts & examples from real-world programs

I recently wrote about my role as project leader for the original DStv Explora consumer device launched in 50 territories across the African continent from 2012-2014. In this post, I will share some visual tools I used to communicate the planning and release strategy. Suffice it to say, I am a big fan of visual planning tools over detailed text narratives any day. There is power in visualizing the plan, on a single piece of paper that beats reading pages of text.

The launch is when the work actually starts

Here's a sample of a post-launch plan that mashes big-picture milestones for executives whilst providing enough detail to software delivery and integration owners. With this single piece of paper, managers can use this schedule as their primary map to navigate their work plans.

Visualizing an end-to-end technology program on one page

Building a new consumer device such as a digital TV set-top-box, from the ground up, end-to-end is a large-scale program with many moving parts. The challenge is how to show as much high-level and low-level detail as possible, starting with output milestones and cascading to detailed team expectations like agile sprints. I can't claim to have authored this view from scratch since I borrowed concepts from my previous projects and other program managers I looked up to, when I worked with Sky/NDS in the UK. 

The timeline below is a snapshot from the early days of Explora planning, where I was the primary plan owner and designer.

Below is a view with extra commentary showing business leaders the hotspots with the plan and calling to action for workstream owners:

For CEOs, I created much-simplified views since they weren't interested in the agile sprints:

A sample project charter for launching a consumer device end-to-end

In a previous post, I wrote about the importance of the project charter and the various forms it can take. In this post, I will go deeper and share a rather detailed sample of a project charter that I authored for a real-world project that involved launching a new original equipment manufacturer (OEM) consumer device, a digital TV set-top box, called the "DStv Explora" end-to-end. My role was overall program manager. My task was basically - to fix everything and get the project on track to deliver, owning the entire plan, directing and steering multiple businesses, project offices and engineering organisations. My stakeholders were C-suite from at least five firms. The project cost in the region of R2 billion ($200m). 

By end-to-end, I mean all workstreams covering technology innovation & development, satellite infrastructure, device hardware engineering, device software engineering, infrastructure systems & software (including configuring satellites in space), application software development, software testing, end-to-end security, consumer field trials and go-to-market (finance, supply chain, marketing, communications, sales, customer support) launching in 50 countries in Africa, with catering for the unique rules & regulations per country, bespoke marketing & comms plans and at times supporting a different look-and-feel brand per country. I insisted on project charters for each go-to-market stream for significant territories like South Africa, Nigeria, Kenya & Ghana. The structure for the business at that time was rather loose, even though there were centralised project offices co-ordinating major launches, we had to partner with in-country business owners for launch planning. I don't share these business charters here, as they're go-to-market workstreams one can find online (or ask chatGPT).

The Explora project enabled me to make a significant impact on the company, the people and teams I worked with and most importantly, opened up my eyes to my potential as a prominent leader, boosting my confidence. Prior to the Explora, I'd just relocated from the UK, having worked with NDS for the last ten years, building and releasing a variety of TV software services, working in engineering and answerable to a number of customer and account delivery managers, taking instruction from bigger program managers and owning a few core technical workstreams. With Explora, my position was elevated to the highest level, giving me the opportunity I had long sought after, and that is to learn everything there was to know about running and operating a PayTV business. For Explora, I not only enjoyed a birds-eye view of the bigger picture - actually I created the bigger picture for everyone to follow. I was responsible for ensuring engineering teams are set-up for delivery success end-to-end, instigating and driving changes needed to ensure project success. My voice was heard. My opinions were listened to. My advice was heeded. I was granted autonomy once I'd earned the trust of all stakeholders involved (a post for another day). 

As far as I can tell, many of the engineering disciplines I introduced for Explora still remain in place today, 10 years later.  Here are some anecdotes as public endorsements on LinkedIn (so I'm not exaggerating my resume here, check out my LI profile recommendations for additional evidence):

Here's an email from Phil on the interventions I introduced in the last stretch of Explora launch:

Hi Muhammad

This one week cycle you have introduced on Explora is a stroke of genius. We have a lot to thank you for on this project, as you have saved our collective asses several times. I for one, really appreciate the quality and quantity of effort you put into supporting us. When we make it on the 1st August you should be able to look back on this project with a great deal of satisfaction. MCA is not the easiest place to bring order to, but you can't fault the guys on their commitment to making things happen:)

Have a great weekend my friend!
Regards Phil

Here's a video about the DStv Explora from 2013/14:

The project charter for Explora set the high bar for modelling all future consumer device launches going forward. Following Explora, I would manage the launch of the DStv HD decoder, followed by software updates launching new features like Catch Up Plus for streaming over the internet, incuding remote recordings. I left the consumer device division in 2015, to spend the next year running the program for launching Showmax, a new streaming video platform business, end-to-end from zero to launch in 8 months. With these successful launches behind me, I'd developed enough credibility in the market that further opened up opportunities for consulting, and later on, took me one step closer to reaching my original ambition of having a seat at the business table, the C-suite round table. Once I'd experienced that view, I decided to seek a new venture, risking starting over again (a future post, stay tuned).

Sample Project Charter

[Disclaimer: Please note I write about my past work and have permission to share my work experiences through my blog as was part of my contract with employers over the years. I've waited more than 10 years to share this particular work experience, the technology & business have moved on since then, such that this sharing is rather informational and can be seen as training material for engineering/delivery managers.]

A product roadmap visual depicting a single tech platform journey

I spent the first decade of my software engineering life building technology stacks for digital TV businesses for the likes of DirecTV, Sky, Multichoice DStv, Liberty, US Cable, etc - working at that time, for what was the world's leading Digital TV Technology Services company, NDS. We were in the business of selling full-stack embedded software (like Android / iOS SDK) tailored for set-top boxes (STBs), along with the backend infrastructure services needed for digitising, encrypting and transmitting TV signals over-the-[air/cable/internet] to these consumer device STBs, so people can basically watch TV. We would sell the tech stack along with a suite of TV applications that could essentially be tailored for any type of customer need, including changing different look-and-feel frontend/user experiences, configurable features like live recording or basic watching without recording, on-demand or internet streaming -- all without having to run multiple versions of software codebase per customer. In modern software parlance, we built multi-tenanted technology stacks in a way.

In 2010, we embarked on a vision to harmonize in creating the Nextgen version of the platform - taking the best of all customer engineering projects and core platform enhancements, and creating the next-generation stack, to scale to as many video entertainment providers across the world. We called this initiative Project Sunrise, symbolizing a new dawn for the next-generation experience built on Fusion Mediahighway Advanced offering a fully customizable Snowflake Unity UI experience. 

Here's a short clip from 2011 on Snowflake:

Back then in 2010, I'd just come off delivering what was the biggest migration program in the history of the company, launching a new service for Sky - and thereafter landed another client build for UPC Horizon Gateway STB. In addition, we had 5+ other customers all lined up for new tenants! It was going to be a busy next few years indeed. 

Our approach to building this technology stack included foresight from the very beginning. We were intentional about using a single stack, configurable architecture end-to-end, including customizable applications for custom experiences - to avoid rework and duplication, but most importantly quick delivery turnaround times. Our customers also benefited from leveraging features and capabilities they didn't have to pay for, because some other customers would have already funded the development anyway :-) 

Technical Program & Product Management - Visualising the Roadmap

As our company was primarily a technology engineering company and a high-growth start-up, resources were constrained such that people took on multiple roles. I led the Sunrise project covering technical product and program management. I was responsible for creating the roadmap, backlog and overall sequencing, coordinating with multiple customer-delivery streams, along with the main core platform engineering deliveries - building the next-generation stack. The engineering activities were a mix of software integrations and application customizations through configuration, building out the default flagship application, that would come "out-of-the-box" for selling to prospective customers. The sales team would close the sale by signing off on the profiling customizations and configurations - and the Sunrise factory would eventually produce a release for the customer. This pattern of template-driven, profile-based software configuration approach was not new to us, but the technologies we used had changed over time (see the slide deck at the end). Project Sunrise was never a fully funded initiative though, so we had to partner with customer project teams and core platform engineering, being scrappy and inventive - but still ensuring we have a reference stack available, at all times, for new sales.

How did I communicate the Roadmap then?

I had a ton of detail to manage, spanning multiple customer requirements backlogs - working with teams across the globe, managing a unified backlog, understanding the features and gaps, prioritizing features for the base profile & then owning a delivery plan (which I'll expand upon in future posts). The one mechanism that earned the trust of senior leadership was a visualization I produced, that showed on a single piece of paper how all the streams fit together. Once the executives saw the roadmap, they then had an easy mental model to understand the complex pieces and stages of convergence - that went into building out the NextGen Sunrise platform. 

I decided to write this blog piece today, 13+ years later because, it so happens, I find myself now again responsible for building a Nextgen product (V2), with V1 (single tenant) that is currently supporting existing customers with an active roadmap - and V2 targeting multiple-tenants onboarding new customers with their own specific configs/capabilities - and my team are considering ways of communicating the plan!

Check this out:

The above roadmap accomplishes the goal of showing the interplay between Customer engineering deliveries (above the Sunrise line) and simultaneously showing the core platform engineering feature deliveries (below the Sunrise line) - all contributing to the holistic platform called Sunrise. So customers mutually benefit from the platform core and they themselves benefit from the internal platform development. All of these deliveries are contained within a timeline that serves as the roadmap. 

As I revisited this picture, thirteen years later, I can still appreciate the value of a picture like this - a powerful visualization that will beat any detailed text narrative IMHO.

And here's our scrappy Project Kick-Off Charter

Oh, and here is the initial MVP we demoed on Sunrise: