Agile Architecture - Roadworks Analogy

I've had some interesting, somewhat frustrating conversations recently, around the roles, responsibilities and expectations of a Systems Architect. Refer to a previous post on how I broke down the various architecture roles for Digital TV Systems Projects.

What triggered the recent discussions is around understanding how architecture fits in with the Agile process. Before people who think themselves Agile Aficionados (Agileficionados) singing praise that architecture can be done just-in-time, please remember the context which I refer to: Embedded Set-Top-Box or Backend Systems Engineering.

We are not in the business of writing web applications, smart phone apps, tablets, or any desktop application, or even a web site -- where these environments are usually proven, solid and stable that fosters rapid application development cycles.

Digital TV projects, especially Set-Top-Box projects are a little more complicated than iterating through web page developments, switching versions easily through redirecting URLs, eliciting real world customer feedback for alpha/beta sites or apps, etc. STB projects generally involved multiple vendors or software suppliers, each with its own development processes. Testing is often done in a limited lab environment, whilst real world customer feedback demands an end-to-end engineering system to be deployed in a live broadcasting environment (which is currently in active use, generating revenues for the PayTV operator, so engineers better be damn sure they're not going to disrupt business operations, etc).

So some STB project teams are indeed embracing Agile (or at least trying to be Agile in the context of their component-worlds). I have written about the difficulty of doing Agile in these projects that I'm not going to repeat here. How then does one deal with the concept of architecture in Agile? Do we stick to Just-in-time, just enough or do we embrace a little structure and discipline, in keeping with Mile-wide, Inch-thick view of the future, but not waste time with too futuristic features??

For Digital TV projects, I prefer the mile-wide, inch thick approach, where we're looking at least a year ahead into the future, to develop new features and enhancing the roadmap. This is especially true for long lead features, like Recommendations, Targeted Advertising, Home Networking, or even designing the next evolution of your Middleware or EPG. And I believe that Architecture, can still be done incrementally, in an Agile-manner, although with a little discipline called foresight and pragmatic proactivity. To support this picture, I created the Paving-the-Road analogy:

Paving or Building a Road Analogy
I'm no civil engineer and have no clue on the actual processes, but have been on enough roads to get an idea of what goes on. When paving a road between points A & B (start and stop points respectively), one of the first actions would be to get a feel for the lay of the land.
A survey is done to assess the pathway, identify likely obstacles that could be problematic, look at alternate scenarios and go about mapping out a strategy. As part of this, some testing is probably done close to problem areas, further getting a grip on the lay of the land.

Once a reasonable strategy is in place, we then start with preparing a rough path, i.e. carving out the rough path. In the case of a road upgrade, generally an alternate path is created for people to use whilst the road is under construction. Ultimately though there's a rough path cleared away that proves the A-to-B layout, so that construction can begin.

Constructing the road happens in stages, mile-by-mile, the road surface is incrementally prepared until point-B is reached. All the while, people can use alternate access to get them effectively from A-to-B.

Sometimes, the fundamental foundation structures are put-in-place in advance of the final road surface, especially true in the case of bridges, etc. These structures appear often in the early stages of the project. Care is also taken to not be overly disruptive on existing commuting traffic.

I doubt very much that construction engineers take the just-in-time, just-enough approach, that is, foregoing the initial lay of the land, and just starting the paving/tarring on bare ground, working towards B,
with the view of tackling obstacles (hoping there won't be any), as-and-when, and responding with agility on the next set of actions...Frankly, this road is more likely to end up in disaster...

The road still gets developed incrementally, in batches and one mile-at-a time. The velocity or speed of development is still measurable, confidence gained from the initial groundwork provides promise of successful completion.

In pretty much the same way, we can address the topic of Systems Architecture in Digital TV, whether it relates to Set-Top-Box Architecture or Headend Systems Equipment:

• Lay of the Land: Architects need to consider the big picture, a realistic long term view of the end-goal. Lay out the framework for the system, proving key building blocks early (often iterating through a few concepts), prove the core design/architecture by realising one or two core
  
  scenarios or use cases. In much the same way as construction engineers deal with obstacles, providing the initial rough terrain for roadworks to begin.
• The Systems Architect must provide this rough terrain, even though the full details are impossible to flesh out - but still be confident with just-enough information, that proves the sustainability and scalability of the core architectural design will deliver the features and functionality required.
• Architecture provides the rough paving, that engineers will use to build upon, adding layers and layers to the rough surface, thus producing a fine road surface.
• Repeating this incrementally, as the road is incrementally paved, will eventually build up to prove the overall system completeness.

This is just my view, I have not yet seen beautiful software systems created in any other fashion, i.e. without an initial investment in upfront planning, design verifications through thought experiments and proof-of-concepts, and without iterating to a point of first proving the skeleton architecture is solid to build upon. 

I believe Systems Architecture can still be approached in an Agile fashion using this construction analogy, that sufficient documentation is required up-front, that makes for valuable reference material in subsequent phases of the project. Just saying that architecture can be done just-in-time, through conversation with minimal documentation, is IMHO, a poor Systems Engineering practice.

Granted that we do have complicated Software Systems - so does the construction world. Take a look at the London Tube Map, this is a complicated system of tracks with interconnections, many stations and over several layers of underground floors. Some serious planning was surely required. In a complex Software system, the interconnections are the many component interfaces that need to be managed, layers of floors refer to the multiple layers of software component functions -- doing this system just-in-time is probably not the best approach!