Agile Architecture - Roadworks Analogy

I've had some interesting, somewhat frustrating conversations recently, around the roles, responsibilities and expectations of a Systems Architect. Refer to a previous post on how I broke down the various architecture roles for Digital TV Systems Projects.

What triggered the recent discussions is around understanding how architecture fits in with the Agile process. Before people who think themselves Agile Aficionados (Agileficionados) singing praise that architecture can be done just-in-time, please remember the context which I refer to: Embedded Set-Top-Box or Backend Systems Engineering.

We are not in the business of writing web applications, smart phone apps, tablets, or any desktop application, or even a web site -- where these environments are usually proven, solid and stable that fosters rapid application development cycles.

Digital TV projects, especially Set-Top-Box projects are a little more complicated than iterating through web page developments, switching versions easily through redirecting URLs, eliciting real world customer feedback for alpha/beta sites or apps, etc. STB projects generally involved multiple vendors or software suppliers, each with its own development processes. Testing is often done in a limited lab environment, whilst real world customer feedback demands an end-to-end engineering system to be deployed in a live broadcasting environment (which is currently in active use, generating revenues for the PayTV operator, so engineers better be damn sure they're not going to disrupt business operations, etc).

So some STB project teams are indeed embracing Agile (or at least trying to be Agile in the context of their component-worlds). I have written about the difficulty of doing Agile in these projects that I'm not going to repeat here. How then does one deal with the concept of architecture in Agile? Do we stick to Just-in-time, just enough or do we embrace a little structure and discipline, in keeping with Mile-wide, Inch-thick view of the future, but not waste time with too futuristic features??

For Digital TV projects, I prefer the mile-wide, inch thick approach, where we're looking at least a year ahead into the future, to develop new features and enhancing the roadmap. This is especially true for long lead features, like Recommendations, Targeted Advertising, Home Networking, or even designing the next evolution of your Middleware or EPG. And I believe that Architecture, can still be done incrementally, in an Agile-manner, although with a little discipline called foresight and pragmatic proactivity. To support this picture, I created the Paving-the-Road analogy:

What it takes to cut the mustard in STB Systems Integration

I have had this topic sitting on my backlog since I first activated this blog, now is a good time to start this topic off since recently it has become quite a topic of conversation in the workplace. I will refine this post over time, however, the first installation is just to share my thoughts around the role of STB Systems Integration, its importance and touch on the traits/skillsets as pre-requisites for the role, i.e. What I lookout for in people when hiring a Systems Integration Engineer.

First of all, one needs to understand the context of being a Systems Integrator. In the context of Set-Top-Box software, there are a couple variations to the concept of integration. The core building blocks for this software stack are generally split into the following components: Hardware, Drivers/Operating System, Middleware, Conditional Access Client, EPG Application, Virtual Machine Engine and Interactive Applications. In terms of variations of SI, it essentially falls into two categories, not so different to "White Box" & "Black Box" concepts used in testing - however, it does have a direct impact on the level of influence & amount of topic-specific knowledge the SI team has or is empowered with, thus having a direct impact on the outcome of the project.

Ultimately, the PayTV Operator owns the full stack that makes up the product. The components are often provided by multiple software vendors. Assembling these components into a workable stack is left up to the task of a Systems Integrator (SI). SI acts as the mediator, the aggregator, the arbitrator - assembler of  components, puts the build together, runs smoke & sanity tests, including full functional testing, investigates defects and assigns such to offending components. SI controls the flow of releases, plans the release schedules and is ultimately accountable for generating the launch software, thus getting it to deployment / operational space.

In a previous post on SI is King, I stressed the importance of SI. I still maintain that the role of SI is in fact the most important role in a STB project, SI has high visibility and high impact, the gateway to success and the gatekeeper for maintaining the integrity of the software. Without a strong SI team, your project is really dead-in-the-water, project will thrash for months on end and will likely result in a project doomed for failure.

In terms of growth potential for engineers, I personally believe, through my own hard-won experience as an engineer myself, having been involved in various engineering roles; as well as from my experience in managing projects across several development & integration teams, that Systems Integration (SI) is the ultimate place to be in, in terms of career progression and should be a natural development path for engineers seeking the next level from software development...

The effectiveness of the SI team is governed by the teams ability to execute on expert technical knowledge thus implementing best practices, streamlining processes, operating as an efficient well-oiled machine. More importantly, the strength of SI is determined by the calibre of its contingent Engineers, the strength of its workforce.

In this post, I will share my thoughts around the following areas:

• Basic Qualifying Criteria
• Technical Skills & Competencies
• Behaviors: Personal / People skills, including Work Ethic
• SI Manager Expectations
• My own sample of Interview Questions for SI Manager Role

It doesn't really matter on the context of Black Box or White Box, however, I do feel that Black Box SI is often counter-productive and is really at odds with the intent of true SI, but in some projects or commercial agreements, this is sometimes unavoidable. SI engineers who shine in Black Box SI would probably excel even more if they had access to source code, build tools, etc. In the whole however, when it comes to requirements for SI engineers, they should be equipped to work equally well in any context.

Programme / Systems Readiness Heat Maps

In the past I've written about the make up of the Digital TV (DTV) ecosystem or "value chain" making it quite clear that the system in itself is a complicated mix of different systems, often provided by more than one vendor. To recap, you can follow-up on these posts:

• DTV End-to-End Architecture
• Programme Organisation for DTV Programs
• Worlds of System Integration

It is not very common for DTV projects to impact the end-to-end value chain, where implementing a new feature or major product launch touches upon just about every element of the system, but it does happen. It is complicated, but it can be managed. It takes an investment in effort, diligence, rigour, sense-of-strong-will, and patience to work with the different teams, to coherently come together in delivering the overall plan.

In this post, I share what I've come to find quite a useful tool: the Programme/Systems Readiness Heat Map.

It is well-known that human beings think best in pictures. As the well known works of Edward Tufte teach us, that visualisations are a powerful way of communication. A visualisation, if done correctly or smartly, can accurately reflect or communicate the state of "things" or "tells a story" so that almost anyone can just, from looking at the picture, get the message.

So, as a Programme or Systems Projects Manager, dealing with complicated technology components, and time lines are highly parallelled and intertwined, when it comes close to the final delivery stages of the project, what is the best way for you to communicate the overall status of readiness of the system?

Sure, one can use a series of PowerPoint slides, representing the status of each Work Package, focusing on the key delivery criteria for each component - this is usually what most PMs do anyway in reporting overall Project Progress via status reports. Your SteerCo have very little patience to wade through 30 slides of PowerPoint tables...

So, what the Heat Map does is simple: On a single piece of paper (okay, maybe the size is A3!), the heat map shows the state of the entire System / Program, focusing on the key criteria for each component that determines the fit-for-purpose state that your SteerCo & Executive can use as input into making the decisions for approving final deployment: Go Live. Essentially the Heat Map is represented as an n x m matrix, with n number of components to track, m the number of metrics/criteria that must be met to guarantee successful implementation to deploy. Each entry is given a value or an associated Heat Colour (Red, Amber, Green), that when all entries are filled in, the reader can quickly ascertain the state of readiness (are we hot or cool or in between?)

I will talk you through a generic template that I've used in my own programs...