A blast from the past: my experience building a large-scale tech platform

In the years 2003-2011, I worked for a pure technology service provider, NDS (acquired by Cisco in 2012, then later became Synamedia) which was considered at the time, the world leader in end-to-end digital TV software systems. I was fortunate enough to experience as an engineer every major area of platform development for this complex ecosystem; and then later as a software manager, I would own the software delivery for a core piece of the software stack known as "middleware", for NDS's primary anchor customer BSkyB/Sky Darwin and then later would own the full stack delivery of NDS's flagship Mediahighway Fusion/Unity product. This experience would mark my entry into very complex large-scale technology delivery initiatives, which even to this day, thirteen years later, as I work with the world's largest cloud provider, Amazon AWS, in building out its enterprise cloud support systems (AWS Support Center / Technical contact systems), Fusion still takes the prize for the most intense professional experience, learning and growth, technical complexity, risk and high-stakes projects. So yeah, I find myself having to dig deep into my memory to recall this work experience because it's funny that 13 years on, I'm encountering the same topics of engineering management even though it is supposed to be a different domain, turns out "software is just software"!

NDS had captured almost every top-tier PayTV operator around the globe at the time: Sky, DirecTV, UPC, Sky Italia, Sky Deutschland, Foxtel, Sky LA, Yes, Bharti, etc. NDS was prominently known for its conditional access product, a video content protection system call NDS Videoguard, however, NDS offered more than just security and offered customers a fully vertically integrated ecosystem (think "Apple" ecosystem for PayTV customers). Whilst digital TV was built on open standards and interoperability, most customers limited their integration points. So when they opted for NDS as their security provider, they also had the option of integrating all other services - from broadcast backend services in the headend to consumer device hardware development and software service integration with chipset vendors. The consumer device software was known as TV Middleware. At the time, the main players were NDS Mediahighway, OpenTV & TiVo. NDS was known for convincing customers to migrate to NDS Mediahighway, its technology migration programs were demanding, complex and executed flawlessly. As an engineer, I contributed software to replace TiVo, an overnight win for 40 million devices. Later as a software delivery manager for the Sky Darwin migration project, we would replace OpenTV software almost obliterating its presence from Sky, save for a few ancient, ageing hardware profiles.

NDS, with an increasing number of customers using its security, middleware and application services, couldn't afford to scale out with engineering teams for each custom build. A platform strategy was needed, consolidating the best of software from across the globe (US, UK, India, Israel, France) into a new shared technology stack, that offered flexible customisation and tailoring for any type of customer profile (Tier-1 customers like Sky for advanced applications to Tier-3/4 customers in territories just starting off with basic digital TV), using a shared engineering resource pool - and extensible configuration engine for producing tailored custom releases. So was borne, NDS Mediahighway Fusion.

The flagship customer for Fusion was Sky, which went live in 2010, replacing up to ten variants of its consumer device software services, with new Fusion components and Sky's own custom-developed consumer application "EPG" known then as the "Orchid EPG". Fusion provided an SDK/API for customers to develop their own primary applications, along with an interactive HTML engine, that allowed PayTV operators to add additional mini apps to their devices, like games and weather apps. With Sky being the anchor customer, Fusion had proved itself in the market and thus was ready to onboard new customers like Sky Italia, UPC, Foxtel, Yes, etc. Post Darwin launch, I took the lead for building the new platform vision, called Fusion Snowflake EPG through project Sunrise - birthing the platform that would create customer, tailorable configurations for any customer, maximising reuse and minimising customisation but allowing for a selection of custom user experiences.

Why am I claiming Fusion as large-scale (even in 2023, 13 years later)?

I write this in 2023, after spending 2.5 years with Amazon AWS. I am part of the group that build AWS Support Center and related Contact Center services. We are a team of under 100 people, deemed large- scale and building complex systems. Yet, if I have to be brutally honest with myself, I'm mildly impressed by my exposure to date, because my current work pails in comparison to my work on Fusion, 13 years ago. Yes I know it's a different domain, a different paradigm and culture of Amazon's 2-Pizza team model for software product ownership (which I actually find quite cool)...still I'm finding it hard to rationalise my move to AWS almost 2.5 years on, have I gone too far backwards? Am I living too much in the past & not ready to view things from a new perspective? What am I not seeing? (Topics for another post). So whilst I've defintely adapted my mental models since joining Amazon, yet I really can't ignore some software engineering truths which is the reason for my bringing up the past now. 

In 2012, I wrote the first story about Fusion, introducing the term LSSDP I coined to mean Large Scale Software Development Project. I also dived deep, writing lengthy white papers about the product and engineering management processes:

• How we "scaled agile" to adapt to our unique global challenges
• How to design a global org-structure to deliver a large-scale project in a scaled agile way
• What mechanisms are needed to deliver a large-scale technology initiative in an agile way
• How we executed and raised the engineering & product management bar

Fast forward to 2023, now using my Amazon AWS experience as a lens for defining a large-scale initiative and indirectly checking engineering manager role guidelines for large-scale:

• Business Impact - Fusion started off with a $75 million investment and later a joint-venture with the flagship customer, Sky. The entire company pivoted to focus on Fusion as its next-generation software platform, with up to 3000 engineers world-wide working on multiple streams, some strategic foundational streams kicked off at least 2 years before the mainstream program. In my role as software delivery owner for Sky Darwin project, it was critical the project delivered successfully, flawlessly - as it involved migrating software in 10 million people's homes (their living room TVs) seamlessly with no rollback. To the end customer (the person sitting at home watching TV), they would notice very little change to their experience. Overall, Fusion software components delivered to multiple middleware stacks, at the time of 2011 when I departed NDS, our software was running in excess of 60 million people's homes daily, globally.

• Scope and Size - Fusion introduced a new paradigm of the TV software ecosystem, end-to-end, including broadcast headend components as well as embedded software architecture. The stack was open, based on a Linix/Posix and a complete departure from the initial decade of TV software operating systems. This was before the advent of Android TV or fully open source middleware. Fusion's product backlog captured over 2000 epics in the form of work packages, cutting across multiple customer needs, in parallel. The scope included all layers of the device software stack: Chipset drivers, hardware absraction layer, Linux kernel, Linux abstraction, Middleware services, Application SDK/APIs, multiple frontend application engine proxys for C / C++ / Java / HTML / Flash applications. Take a look at the software architecture diagram - it is multi-layered, multiple service teams. Another point on scope, we managed initiatives or epics in the form of work pacakages (WPs), that could impact up to 25 service teams in one WP, see here.

Product Plan visuals - concepts & examples from real-world programs

I recently wrote about my role as project leader for the original DStv Explora consumer device launched in 50 territories across the African continent from 2012-2014. In this post, I will share some visual tools I used to communicate the planning and release strategy. Suffice it to say, I am a big fan of visual planning tools over detailed text narratives any day. There is power in visualizing the plan, on a single piece of paper that beats reading pages of text.

The launch is when the work actually starts

Here's a sample of a post-launch plan that mashes big-picture milestones for executives whilst providing enough detail to software delivery and integration owners. With this single piece of paper, managers can use this schedule as their primary map to navigate their work plans.

Visualizing an end-to-end technology program on one page

Building a new consumer device such as a digital TV set-top-box, from the ground up, end-to-end is a large-scale program with many moving parts. The challenge is how to show as much high-level and low-level detail as possible, starting with output milestones and cascading to detailed team expectations like agile sprints. I can't claim to have authored this view from scratch since I borrowed concepts from my previous projects and other program managers I looked up to, when I worked with Sky/NDS in the UK. 

The timeline below is a snapshot from the early days of Explora planning, where I was the primary plan owner and designer.

Below is a view with extra commentary showing business leaders the hotspots with the plan and calling to action for workstream owners:

For CEOs, I created much-simplified views since they weren't interested in the agile sprints:

A sample project charter for launching a consumer device end-to-end

In a previous post, I wrote about the importance of the project charter and the various forms it can take. In this post, I will go deeper and share a rather detailed sample of a project charter that I authored for a real-world project that involved launching a new original equipment manufacturer (OEM) consumer device, a digital TV set-top box, called the "DStv Explora" end-to-end. My role was overall program manager. My task was basically - to fix everything and get the project on track to deliver, owning the entire plan, directing and steering multiple businesses, project offices and engineering organisations. My stakeholders were C-suite from at least five firms. The project cost in the region of R2 billion ($200m). 

By end-to-end, I mean all workstreams covering technology innovation & development, satellite infrastructure, device hardware engineering, device software engineering, infrastructure systems & software (including configuring satellites in space), application software development, software testing, end-to-end security, consumer field trials and go-to-market (finance, supply chain, marketing, communications, sales, customer support) launching in 50 countries in Africa, with catering for the unique rules & regulations per country, bespoke marketing & comms plans and at times supporting a different look-and-feel brand per country. I insisted on project charters for each go-to-market stream for significant territories like South Africa, Nigeria, Kenya & Ghana. The structure for the business at that time was rather loose, even though there were centralised project offices co-ordinating major launches, we had to partner with in-country business owners for launch planning. I don't share these business charters here, as they're go-to-market workstreams one can find online (or ask chatGPT).

The Explora project enabled me to make a significant impact on the company, the people and teams I worked with and most importantly, opened up my eyes to my potential as a prominent leader, boosting my confidence. Prior to the Explora, I'd just relocated from the UK, having worked with NDS for the last ten years, building and releasing a variety of TV software services, working in engineering and answerable to a number of customer and account delivery managers, taking instruction from bigger program managers and owning a few core technical workstreams. With Explora, my position was elevated to the highest level, giving me the opportunity I had long sought after, and that is to learn everything there was to know about running and operating a PayTV business. For Explora, I not only enjoyed a birds-eye view of the bigger picture - actually I created the bigger picture for everyone to follow. I was responsible for ensuring engineering teams are set-up for delivery success end-to-end, instigating and driving changes needed to ensure project success. My voice was heard. My opinions were listened to. My advice was heeded. I was granted autonomy once I'd earned the trust of all stakeholders involved (a post for another day). 

As far as I can tell, many of the engineering disciplines I introduced for Explora still remain in place today, 10 years later.  Here are some anecdotes as public endorsements on LinkedIn (so I'm not exaggerating my resume here, check out my LI profile recommendations for additional evidence):

Here's an email from Phil on the interventions I introduced in the last stretch of Explora launch:

Hi Muhammad

This one week cycle you have introduced on Explora is a stroke of genius. We have a lot to thank you for on this project, as you have saved our collective asses several times. I for one, really appreciate the quality and quantity of effort you put into supporting us. When we make it on the 1st August you should be able to look back on this project with a great deal of satisfaction. MCA is not the easiest place to bring order to, but you can't fault the guys on their commitment to making things happen:)

Have a great weekend my friend!
Regards Phil

Here's a video about the DStv Explora from 2013/14:

The project charter for Explora set the high bar for modelling all future consumer device launches going forward. Following Explora, I would manage the launch of the DStv HD decoder, followed by software updates launching new features like Catch Up Plus for streaming over the internet, incuding remote recordings. I left the consumer device division in 2015, to spend the next year running the program for launching Showmax, a new streaming video platform business, end-to-end from zero to launch in 8 months. With these successful launches behind me, I'd developed enough credibility in the market that further opened up opportunities for consulting, and later on, took me one step closer to reaching my original ambition of having a seat at the business table, the C-suite round table. Once I'd experienced that view, I decided to seek a new venture, risking starting over again (a future post, stay tuned).

Sample Project Charter

[Disclaimer: Please note I write about my past work and have permission to share my work experiences through my blog as was part of my contract with employers over the years. I've waited more than 10 years to share this particular work experience, the technology & business have moved on since then, such that this sharing is rather informational and can be seen as training material for engineering/delivery managers.]

A product roadmap visual depicting a single tech platform journey

I spent the first decade of my software engineering life building technology stacks for digital TV businesses for the likes of DirecTV, Sky, Multichoice DStv, Liberty, US Cable, etc - working at that time, for what was the world's leading Digital TV Technology Services company, NDS. We were in the business of selling full-stack embedded software (like Android / iOS SDK) tailored for set-top boxes (STBs), along with the backend infrastructure services needed for digitising, encrypting and transmitting TV signals over-the-[air/cable/internet] to these consumer device STBs, so people can basically watch TV. We would sell the tech stack along with a suite of TV applications that could essentially be tailored for any type of customer need, including changing different look-and-feel frontend/user experiences, configurable features like live recording or basic watching without recording, on-demand or internet streaming -- all without having to run multiple versions of software codebase per customer. In modern software parlance, we built multi-tenanted technology stacks in a way.

In 2010, we embarked on a vision to harmonize in creating the Nextgen version of the platform - taking the best of all customer engineering projects and core platform enhancements, and creating the next-generation stack, to scale to as many video entertainment providers across the world. We called this initiative Project Sunrise, symbolizing a new dawn for the next-generation experience built on Fusion Mediahighway Advanced offering a fully customizable Snowflake Unity UI experience. 

Here's a short clip from 2011 on Snowflake:

Back then in 2010, I'd just come off delivering what was the biggest migration program in the history of the company, launching a new service for Sky - and thereafter landed another client build for UPC Horizon Gateway STB. In addition, we had 5+ other customers all lined up for new tenants! It was going to be a busy next few years indeed. 

Our approach to building this technology stack included foresight from the very beginning. We were intentional about using a single stack, configurable architecture end-to-end, including customizable applications for custom experiences - to avoid rework and duplication, but most importantly quick delivery turnaround times. Our customers also benefited from leveraging features and capabilities they didn't have to pay for, because some other customers would have already funded the development anyway :-) 

Technical Program & Product Management - Visualising the Roadmap

As our company was primarily a technology engineering company and a high-growth start-up, resources were constrained such that people took on multiple roles. I led the Sunrise project covering technical product and program management. I was responsible for creating the roadmap, backlog and overall sequencing, coordinating with multiple customer-delivery streams, along with the main core platform engineering deliveries - building the next-generation stack. The engineering activities were a mix of software integrations and application customizations through configuration, building out the default flagship application, that would come "out-of-the-box" for selling to prospective customers. The sales team would close the sale by signing off on the profiling customizations and configurations - and the Sunrise factory would eventually produce a release for the customer. This pattern of template-driven, profile-based software configuration approach was not new to us, but the technologies we used had changed over time (see the slide deck at the end). Project Sunrise was never a fully funded initiative though, so we had to partner with customer project teams and core platform engineering, being scrappy and inventive - but still ensuring we have a reference stack available, at all times, for new sales.

How did I communicate the Roadmap then?

I had a ton of detail to manage, spanning multiple customer requirements backlogs - working with teams across the globe, managing a unified backlog, understanding the features and gaps, prioritizing features for the base profile & then owning a delivery plan (which I'll expand upon in future posts). The one mechanism that earned the trust of senior leadership was a visualization I produced, that showed on a single piece of paper how all the streams fit together. Once the executives saw the roadmap, they then had an easy mental model to understand the complex pieces and stages of convergence - that went into building out the NextGen Sunrise platform. 

I decided to write this blog piece today, 13+ years later because, it so happens, I find myself now again responsible for building a Nextgen product (V2), with V1 (single tenant) that is currently supporting existing customers with an active roadmap - and V2 targeting multiple-tenants onboarding new customers with their own specific configs/capabilities - and my team are considering ways of communicating the plan!

Check this out:

The above roadmap accomplishes the goal of showing the interplay between Customer engineering deliveries (above the Sunrise line) and simultaneously showing the core platform engineering feature deliveries (below the Sunrise line) - all contributing to the holistic platform called Sunrise. So customers mutually benefit from the platform core and they themselves benefit from the internal platform development. All of these deliveries are contained within a timeline that serves as the roadmap. 

As I revisited this picture, thirteen years later, I can still appreciate the value of a picture like this - a powerful visualization that will beat any detailed text narrative IMHO.

And here's our scrappy Project Kick-Off Charter

Oh, and here is the initial MVP we demoed on Sunrise:

 

Why I never ran a program without a Project Charter

Lessons on large-scale delivery program management ...

I continue to dig into my past artefacts to showcase my work portfolio. I'm using a multi-pronged approach here: 1\ Showcase my work to prospective employers; 2\ Openly share my work so that others (people I coach, my colleagues and boss, etc.) can benefit; and 3\ Act as my own living knowledge repository.  

I spent a decade climbing up the project management ladder, in the same way I climbed up the software engineering ladder (from junior engineer to principal engineer) - I first started project managing small software product development (2-4 teams with 10 services), then scaled up to large middleware services (20+ teams, 50+ services) as lead delivery owner, then up to full stack systems integration (full stack of all major components: kernel, middleware, integration layers, applications), then program managed a full go-to-market product launch scaling out to including Tech, Business (Finance, Marketing, Supply Chain) & Operations (Customer Care, Retention, Content, Legal, Regulatory) - as senior program manager. I also owned the full plan of starting a business from scratch to launch (a video streaming company) in 8 months. I did a stint in management consulting, running the top 5 business projects for a $3 billion run-rate business, which some companies might call Tier 0/1 initiatives - where I co-ordinated these large-scale programs, as Chief Program Director - delivering through multiple business lines, multiple project management offices and multiple product and engineering teams. In a sense, I served as the CxO program manager, advisor and delivery owner.  

It is with this experience and knowledge, that I dare to share about my work experiences - and I'm not making these things up - you can check my LinkedIn recommendations page for proof.

During my tenure as the lead program director mentioned above, I often found myself picking up and repairing distressed programs - and along the way, I'd help improve team processes and coach the management teams as well. I also ran new business & technology initiatives from scratch, start-to-finish-then-handover. So with this diverse experience, I developed a simple method that helped me navigate both types of program scenarios: either resetting or starting from scratch, the simple, powerful mechanism of a Project Charter document. To this day, I'm surprised to see many program & project managers failing to use the Project Charter in the way it was meant to be used (clarifying the essence), and often find less-experienced, newly minted PMP/Prince2 certified professionals, doing it "by the book". My approach to project charters went much deeper than that...

So what do I mean by using the Project Charter in "clarifying the essence" then?

A seasoned, experienced project leader, chief program director, end-to-end project manager, senior technical program manager, etc - call the roles what you like - in my view, is not about just putting a plan together, working backwards from a deadline or target completion date. No, I believe as senior program leaders must apply their minds to appreciate the bigger picture and create a program structure that becomes the north star in guiding and leading multiple delivery teams. I never started a program without first establishing my project charter, which at the top level, focuses on the following:

• Start by understanding the why. Why is this program needed? Why is it important?
• Move on to understanding the who. Who are the sponsors, stakeholders and teams impacted? Who will be working on the program? "First who, then what"
• A program manager must be sufficiently well-versed with all the roles expected from the program, and work hard to secure the roles needed. Yes, this means the program manager must escalate to get the people needed for the program (on the bus, as well as off the bus). A responsible program manager would raise all these risks & concerns up-front, before officially kicking off the program.
• Clarify the what, including calling out what's missing - Set up the mental model for the program. What is this program about? What is it not about? What's in scope? What's not in scope? What workstreams make up the program? How do all pieces come together?
• Agree, Align, Action - The 3 As of project execution involve agreement on the deliverable, alignment of all parties involved which includes acceptance of their workstreams and ultimately agreeing on the action plan to execute.

Project charters don't necessarily have to be communicated in a written document, a slide deck is more than adequate to communicate the essence. Depending on the business environment and culture (for example, some business cultures prefer slide decks over detailed documents to save reading time, whilst others like Amazon, insist on detailed text narratives). So a seasoned project leader must adapt their style to suit the particular business need & culture of the teams.

In this post, I'm sharing a version of a Project Charter as a slide deck. In a future post, I will share a detailed 50+ page document project charter that involved the launch of a consumer electronics device, the program covered a mix of engineering, business and operations workstreams.

Example Program: Transform Digital Self-Service of a $3 billion run-rate business

I was called in to help reset and kickstart an overarching cross-cutting program to improve a selection of key metrics that would result in increased usage of digital self-service channels, improving customer satisfaction and overall reducing operational costs. This program covered the full value chain delivering the service: 3rd party technology vendors developing phone "mobi" apps using USSD, iOS/Android self-service app, Website, Payments, a hardware kiosk station, set-top box interactive application, integration with internal & 3rd party CRM/Billing systems - and resulting business workflows: finance, customer care & banking channels. Technology teams were spread between the CTO/CIO lines (3 IT pillars), and business teams reported separately to the CEO. The program also served the needs of Group Strategy, Risk & Regulatory. Bringing all these things together requires a steady hand, a tactful negotiator, a strategic and business mindset as well as a strong technology leader. This is why I enjoyed such challenges as these programs were never boring, limited to only tech/engineering.

Enough said, let the slides talk and let me know what you think in the comments!