Agile Architecture - Roadworks Analogy

I've had some interesting, somewhat frustrating conversations recently, around the roles, responsibilities and expectations of a Systems Architect. Refer to a previous post on how I broke down the various architecture roles for Digital TV Systems Projects.

What triggered the recent discussions is around understanding how architecture fits in with the Agile process. Before people who think themselves Agile Aficionados (Agileficionados) singing praise that architecture can be done just-in-time, please remember the context which I refer to: Embedded Set-Top-Box or Backend Systems Engineering.

We are not in the business of writing web applications, smart phone apps, tablets, or any desktop application, or even a web site -- where these environments are usually proven, solid and stable that fosters rapid application development cycles.

Digital TV projects, especially Set-Top-Box projects are a little more complicated than iterating through web page developments, switching versions easily through redirecting URLs, eliciting real world customer feedback for alpha/beta sites or apps, etc. STB projects generally involved multiple vendors or software suppliers, each with its own development processes. Testing is often done in a limited lab environment, whilst real world customer feedback demands an end-to-end engineering system to be deployed in a live broadcasting environment (which is currently in active use, generating revenues for the PayTV operator, so engineers better be damn sure they're not going to disrupt business operations, etc).

So some STB project teams are indeed embracing Agile (or at least trying to be Agile in the context of their component-worlds). I have written about the difficulty of doing Agile in these projects that I'm not going to repeat here. How then does one deal with the concept of architecture in Agile? Do we stick to Just-in-time, just enough or do we embrace a little structure and discipline, in keeping with Mile-wide, Inch-thick view of the future, but not waste time with too futuristic features??

For Digital TV projects, I prefer the mile-wide, inch thick approach, where we're looking at least a year ahead into the future, to develop new features and enhancing the roadmap. This is especially true for long lead features, like Recommendations, Targeted Advertising, Home Networking, or even designing the next evolution of your Middleware or EPG. And I believe that Architecture, can still be done incrementally, in an Agile-manner, although with a little discipline called foresight and pragmatic proactivity. To support this picture, I created the Paving-the-Road analogy:

What it takes to cut the mustard in STB Systems Integration

I have had this topic sitting on my backlog since I first activated this blog, now is a good time to start this topic off since recently it has become quite a topic of conversation in the workplace. I will refine this post over time, however, the first installation is just to share my thoughts around the role of STB Systems Integration, its importance and touch on the traits/skillsets as pre-requisites for the role, i.e. What I lookout for in people when hiring a Systems Integration Engineer.

First of all, one needs to understand the context of being a Systems Integrator. In the context of Set-Top-Box software, there are a couple variations to the concept of integration. The core building blocks for this software stack are generally split into the following components: Hardware, Drivers/Operating System, Middleware, Conditional Access Client, EPG Application, Virtual Machine Engine and Interactive Applications. In terms of variations of SI, it essentially falls into two categories, not so different to "White Box" & "Black Box" concepts used in testing - however, it does have a direct impact on the level of influence & amount of topic-specific knowledge the SI team has or is empowered with, thus having a direct impact on the outcome of the project.

Ultimately, the PayTV Operator owns the full stack that makes up the product. The components are often provided by multiple software vendors. Assembling these components into a workable stack is left up to the task of a Systems Integrator (SI). SI acts as the mediator, the aggregator, the arbitrator - assembler of  components, puts the build together, runs smoke & sanity tests, including full functional testing, investigates defects and assigns such to offending components. SI controls the flow of releases, plans the release schedules and is ultimately accountable for generating the launch software, thus getting it to deployment / operational space.

In a previous post on SI is King, I stressed the importance of SI. I still maintain that the role of SI is in fact the most important role in a STB project, SI has high visibility and high impact, the gateway to success and the gatekeeper for maintaining the integrity of the software. Without a strong SI team, your project is really dead-in-the-water, project will thrash for months on end and will likely result in a project doomed for failure.

In terms of growth potential for engineers, I personally believe, through my own hard-won experience as an engineer myself, having been involved in various engineering roles; as well as from my experience in managing projects across several development & integration teams, that Systems Integration (SI) is the ultimate place to be in, in terms of career progression and should be a natural development path for engineers seeking the next level from software development...

The effectiveness of the SI team is governed by the teams ability to execute on expert technical knowledge thus implementing best practices, streamlining processes, operating as an efficient well-oiled machine. More importantly, the strength of SI is determined by the calibre of its contingent Engineers, the strength of its workforce.

In this post, I will share my thoughts around the following areas:

• Basic Qualifying Criteria
• Technical Skills & Competencies
• Behaviors: Personal / People skills, including Work Ethic
• SI Manager Expectations
• My own sample of Interview Questions for SI Manager Role

It doesn't really matter on the context of Black Box or White Box, however, I do feel that Black Box SI is often counter-productive and is really at odds with the intent of true SI, but in some projects or commercial agreements, this is sometimes unavoidable. SI engineers who shine in Black Box SI would probably excel even more if they had access to source code, build tools, etc. In the whole however, when it comes to requirements for SI engineers, they should be equipped to work equally well in any context.

Programme / Systems Readiness Heat Maps

In the past I've written about the make up of the Digital TV (DTV) ecosystem or "value chain" making it quite clear that the system in itself is a complicated mix of different systems, often provided by more than one vendor. To recap, you can follow-up on these posts:

• DTV End-to-End Architecture
• Programme Organisation for DTV Programs
• Worlds of System Integration

It is not very common for DTV projects to impact the end-to-end value chain, where implementing a new feature or major product launch touches upon just about every element of the system, but it does happen. It is complicated, but it can be managed. It takes an investment in effort, diligence, rigour, sense-of-strong-will, and patience to work with the different teams, to coherently come together in delivering the overall plan.

In this post, I share what I've come to find quite a useful tool: the Programme/Systems Readiness Heat Map.

It is well-known that human beings think best in pictures. As the well known works of Edward Tufte teach us, that visualisations are a powerful way of communication. A visualisation, if done correctly or smartly, can accurately reflect or communicate the state of "things" or "tells a story" so that almost anyone can just, from looking at the picture, get the message.

So, as a Programme or Systems Projects Manager, dealing with complicated technology components, and time lines are highly parallelled and intertwined, when it comes close to the final delivery stages of the project, what is the best way for you to communicate the overall status of readiness of the system?

Sure, one can use a series of PowerPoint slides, representing the status of each Work Package, focusing on the key delivery criteria for each component - this is usually what most PMs do anyway in reporting overall Project Progress via status reports. Your SteerCo have very little patience to wade through 30 slides of PowerPoint tables...

So, what the Heat Map does is simple: On a single piece of paper (okay, maybe the size is A3!), the heat map shows the state of the entire System / Program, focusing on the key criteria for each component that determines the fit-for-purpose state that your SteerCo & Executive can use as input into making the decisions for approving final deployment: Go Live. Essentially the Heat Map is represented as an n x m matrix, with n number of components to track, m the number of metrics/criteria that must be met to guarantee successful implementation to deploy. Each entry is given a value or an associated Heat Colour (Red, Amber, Green), that when all entries are filled in, the reader can quickly ascertain the state of readiness (are we hot or cool or in between?)

I will talk you through a generic template that I've used in my own programs...

PayTV Interactive Applications is dying slowly, but surely

Interactive PayTV is dying slowly, or was it dead on arrival? Imagine these scenarios:

• OK TV, Have any of my favourite shows been recorded lately?
• OK TV, Find me something nice to watch - I am feeling good
• OK TV, I've had a crap day at work today, put something on that'll make me unwind and relax
• OK TV, What are my friends watching now?
• OK TV, What show is trending on the social scene?
• OK TV, What's selling hot on Box Office today? Show me the trailer, skip the ads please.
• OK TV, Let me see what's happening on my hangout?
• OK TV, Give me the top five recommendations today
• OK TV, What's the latest news headlines?
• OK TV, What's happening with Syria today? Connect me to any Live News Stream?
• OK TV, What's making waves on YouTube recently?
• OK TV, Tell me the latest stats on the Formula 1 (Cricket, Football) scene
• OK TV, What's the weather like at my mom's place?
• OK TV, Any weather warnings I should worry about?
• OK TV, Are there any live sports events happening in my area? Which is the closest?
• OK TV, I'm in the mood for Star Trek - make it so!
• etc

Get the picture? This is what interactivity supposed to be all about! TV is a passive medium, lean back, relax & watch TV. I don't want to read wades and wades of text on the screen! I've had a long day behind my PC, been coding all day or working with spreadsheets, the last thing I want is to navigate through some tiresome and clunky menu tree to find the information I want, then I must be forced to get up from my couch because the font & text is so small that I need to stand at least 3 feet away to read the screen! More work, more effort for me - I don't want to be interacting with the remote control to do stuff - I want to interact with my TV, not control my TV!

Yet, most of the PayTV operators have maintained to still support the original concept of Interactive TV - the famous Red Button, then the infamous loading time for the application, then interact with an eighties-style user interface. Whilst there are ways to workaround and improve the classic "Please wait...application is loading..." time, the original concepts have not changed that much.  There's a lot of chatter around Interactive TV 3.0 or second screen, etc - about the next wave of interactivity, that people would rather interact with a second-screen device (smartphone / tablet / etc), but this is not what my post is about.

Mostly, this post is part rant, and part explanation about the current predicament PayTV Operators find themselves. I've already a solution in my head that I think is the way forward, but it is probably disruptive to keep even the most open, objective PayTV operator running away at lightning speed! Just take a look at this absolutely cool demo shown recently at GoogleIO:

This is the future! I think Google is on the right track - Years ago, going back to 2005/2006, I was pushing my company to explore Voice & Text-To-Speech integration into Set-Top-Boxes. When we looked at Recommendations back in 2004/2005, we considered voice -- but hats off to Google, they've done it with Search, and I'm sure the next wave is enhancing their Interactive TV offering.

My elevator pitch: Times are changing! PayTV Operators must stick to their core business: Content provision, distribution, content protection & core information services. Partner with third parties to bring true interactivity to the user experience & platform. Focus more on Open Systems, don't control the whole world / ecosystem. Create strategic partnerships with Google (or other), creating a bridge between a closed PayTV world & the Open Social Apps world... 

This post is organised as follows:

• Brief story about Interactive TV and Typical Architecture Model
• Application Stores Overview - What PayTV Operators need to consider
• My view of a possible Future Platform / Architecture

System Integration is King - Release Manager, Build Meister-Gatekeeper

I spent a lot of energy in the last week trying to (and succeeded in) convincing some senior managers about processes, that in my humble opinion, are tried-and-tested, well-established norms in the management of software: the art of Software Release Management - taking control of component releases, especially w.r.t. approving or rejecting bug fixes. It has been good practice in terms of establishing myself as a consultant-expert on all things related to Set-Top-Box Software Development & Integration. I spend a lot of my time promoting best practices and trying to win & influence people over, something I have to do to maintain focus and direction of the overall programme plan...

The point I want to make is that, in a STB delivery project, the System Integration (SI) team is really the GateKeeper, with full authority and accountability for producing a stable, functional release. This means that SI have every right to manage component releases coming in, especially with reviewing & approving bug-fixes. If SI are not happy with a component release, for example, the component team may have fixed more bugs than was required, or fixed far fewer bugs than was requested, or made any other changes that wasn't specifically requested by the SI team, then SI can quite easily reject the component's release.

I have written in the past on the following topics:

• Template to manage STBSI Release Campaigns
• Release Schedule template to manage Component Vendors
• Modeling the Climb to launch in STB Projects
• Process for fast tracking issues, reduce the backlog on SI
• Tracking Defects using Predictive Analysis
• Effective Defect Management Processes required in STB Delivery Projects

All of these areas become the focus of a STB project as we get closer and closer to launch mode. Recapping the template that most STB projects follow:

• Develop / Integrate Features >>> Functionally Complete >>> Closed User Group >>> Wider Field Trials >>> Launch Candidate 1 >>> ... >>> Launch Candidate X >>> Clean Run >>> Deployment Build >>> Launch

As the system matures to reach each of these milestones, the defects are expected to get less and less. Obviously no product gets launched with zero defects! All projects can do is ensure as best as they can that almost all high impact, high severity defects (a.k.a. Showstoppers or P0s) are dealt with to the best of their ability; and are prepared for the inevitable influx of new Showstoppers from in-field, real world usage of the product - it is the reality, unavoidable. Very rarely do products launch having met all original quality engineering requirements, after all, this is an entertainment device, not a life critical device - so we can relax a little on quality objectives! :-)

So how should you control releases leading up to launch?? It's simple really, not complicated, and really not rocket science...

So you think writing a Set-Top-Box Middleware is easy? Think again!

So you came across this technology called Digital TV and stumbled across the various (open) standards (MPEG, DVB, MHP, etc) and thought it cool for you to write your own software stack? Or you've been writing Set Top Box Software for other customers (namely PayTV operators) and decided you have enough knowledge to venture out on your own and sell your own software stack? Or you're a

PayTV operator, sick and tired of being locked into expensive and time-consuming Middleware vendors that you decide it is time to write your own software stack, be in control of your destiny and deliver cool products to market faster, cheaper and on-time?
Or you figured there is a quick and easy way of making some money, since just about every country in the world is going to switch off analog TV and turn to digital, where you can provide a very cheap and cost-effective software stack?

If you've answered yes to at least one of these questions, then you should be asking yourself: Just how serious are you about Middleware?  Just how serious are you about the EPG User Experience?

Just how serious are you about competing in this marketplace? Is there any room for you to compete in this arena? Where  do you see your software stack in the marketplace: low-tier, middle-tier or top tier? Who are your competitors? Are you big enough, flexible-enough, savvy-enough, innovative-enough or unique-enough to displace your competitors? Or do you even wish to displace your competitors? What relationships do you have with the big technology players? What is the compelling value proposition that you're offering that your competitor's aren't? Do you stick to one large customer that can finance your operation? Do you seek out other customers to monetize on your product's offering? Do you keep a common product that can service multiple customers at relatively low cost? Do you feel the market is big enough to support many divergent offerings? Are you a small fish in a small pond, big fish in a small pond, or are you swimming with the sharks and you just don't know it yet? Do you rely on hope that everything goes well, or have mountains of cash so as not to notice you're burning fast, but don't realise it? Have you been sold vaporware, demoware, software that's far from being fit-for-purpose for real world mass production use?? Etc, Etc, Etc...

I have met developers, architects, software managers and product owners who think that writing a STB Middleware stack is easy, a piece of cake, all you need is just four really good developers...to which I smile and think in my mind, oh how naive they are! Yes, for sure, the software architecture can make sense on paper, heck you can even hack a proof-of-concept together in about three months or less that proves a simple Zapper or PVR stack. But that's just it, it's a hack, a Proof-of-Concept (POC) - nothing close to getting a real world product into the marketplace.

My rambling in this post covers the following:

• Typical Challenges for New Middleware Players
• Securing support from Chipset Vendors
• Choosing the Application / Interactive Engine / API for EPG
• Integrating with Conditional Access Vendors
• Competitive Costing / Pricing Model
• Overall Middleware Strategy
• Advice for PayTV Operators
• Conclusions - Middleware/EPG business is hard work
• Last Point: If I entered the Middleware/EPG Market today, what would I do?