On E2E architect role in projects

I have shared my thoughts in the past on the subject of architects in the domain of digital TV software systems which can be found here. In this post I will share further thoughts about the specific role of an E2E (End-to-End) architect from a program/project management perspective, including my expectations of the role as well as the associated challenges I've faced with this concept, especially where the role either does not exist, or is not clearly defined in cases where companies don't have a clear enough understanding or have not yet set up a formal architecture competency.

My Background & My possible Anchor Bias

In my first decade of working experience with software and systems engineering projects, I was fortunate to work with world class technology service providers (S3 in Ireland & NDS/Cisco in UK). This experience helped shaped the way I would view future technology projects, including product & project management, software engineering, integration and delivery functions. It's the kind of experience that never leaves you, especially when you've witnessed first hand, one successful delivery after another both as an engineer and manager. So I'd come to see the world in a particular light, taking with me the style of project execution where ever I went, and assumed that all companies naturally followed similar concepts. And the cool thing about this experience was a well grounded appreciation for software & systems engineering design principles - principles which should not be confused with implementation methods, as what is a hot topic of debate these days, that of Agile V Waterfall, that has been the bulk of my recent challenges in projects in the last six years.

On returning back home to South Africa* still working in Digital Media TV domain, I soon realised that things are a little different to say the least (absence of architect roles, working with business analysts where I'd been used to analysis being a function of architecture, no defined role of systems integration, no method of continuous delivery & lack of solid post-launch monitoring & operations support systems). My first major program, involved a large-scale overhaul of the end-to-end broadcast system, including a video-on-demand component and a new set top box software stack. In order for the program to have a decent chance of success, I not only had to restructure & reshape the program, but also had to introduce functional roles that for me, were always obvious: an architecture team with a specific role of E2E architect, E2E systems integration, test and delivery teams. With a lot of hard work of convincing senior management these roles were necessary for project delivery, we managed to implement the structures I proposed that ultimately helped in the successful delivery of the project.

Strangely enough, as I took on other engagements in large programs in the same group of companies, I found myself repeating the same. It could be because I have a bias that is so strong and deep that does not allow me to see any other way, even though I always try to adapt my style to the culture and teams currently at my disposal, yet I still find myself coming back to these principles because its been proven and tested, that I can't really find any real fault in - possibly it is because these are indeed engineering principles that have stood the test of time, especially this concept of an End-to-End Solution Architect role. As the saying goes, never leave home without it, if I'm running a large program of work, that involves many systems & component suppliers, I never run a program without having an E2E architect attached to my programs, as step one. Once that is established (often as a result of canvassing support which takes up some time and energy), I move on to structuring the role of E2E systems integration, which in turn drives the processes and behaviours expected from component engineering and test teams. The program timeline then shows a simple picture of the high-level milestones involved, where the bulk of the program management and execution is a function of coordinating delivery plans with respective project delivery teams (who maintain the detailed project plans) as well as managing stakeholder engagement (which is a vital component of running large programs). This hasn't been smooth sailing all the time, as I'll try to share some of the challenges that frequently come up, later on in this post.

*Disclaimer: Although my writing is about an experience with primarily one industry domain in South Africa (Digital PayTV), I'm mindful about not passing a broad value judgement against all large corporates with a software competency. At first I thought these challenges of projects and roles like BAs for example was limited to just this environment, but the more I researched by skimming through job specs, attending networking events, technical conferences, meetups, training courses and meeting people from other corporates, I later realised the patterns run across all the major corporates like financial services, banking and the big telcos, that more often than not, the structures in place mirror more traditional-IT governance than hardcore technical software development structures that I was used to, in working in a pure technology services environment. So I'm reasonably confident there is no broad value judgement being applied here, that these experiences are likely to resonate with large companies operating outside PayTV.

Typical Landscape

The picture below aims to illustrate a typical landscape of this ecosystem. This could be separate entities run independently, or even group under a technology division, with different functional expertise, that in reality are still perceived as separate silos of responsibility:

This picture speaks to a typical reality of technology service providers in a digital TV value chain. Each silo represents a set of core technology expertise that exist in separate functional lines either under a technology division, or could exist as separate lines of business altogether. 

Each silo may have its own set of speciality with architects and development teams. Each unit may also likely to have a very unique culture: people, skill-set and ways of working. Each silo may also have its own operating model: how work gets prioritised and injected, build, test and release procedures. These units might also have their own terminology, a lexicon / vocabulary for their specific domains. They may also not be entirely self-sufficient, relying on external third-party component suppliers that form part of their system. These third-party suppliers are just like another silo, each with their own operating model, from architects to development teams, to ways of working as stipulated by their underlying contractual agreements.

These systems & teams usually come together in delivering a product or service that directly impacts a customer (end user). Typically launching a major new product (or platform, say for example a new internet product like a streaming app for video on demand) will impact at least all these systems. To do so, a project is set up as a program delivery initiative that will call for the co-ordination of all systems coming together in delivering the new product or feature to market. So a program team would usually be setup to manage this execution and delivery. Typically a program manager would be appointed to manage the full end-to-end delivery, which is not only the technology arm, but also the wider business streams for delivering product to market (marketing, customer care, sales, training, retention, communications, legal & regulatory). Sometimes, the overall program manager is supported by an end-to-end technical program manager, but this is not always the case. More often than not, the overall program manager assumes co-ordination of the technology ecosystem as well. For now though, I use program manager to either mean end-to-end technical program manager or overall program manager interchangeably.

The need for an End-to-End Solutions Architect

The picture below illustrates the positioning for a dedicated role of E2E Solution Architect to a project or program of work:

Just as you would appoint an end-to-end project / program manager to the role of maintaining the coherency of the overall delivery project, so too should you assign an end-to-end solution architect role, to maintain the integrity and coherency of the overall technical solution design. This is the crux of my proposition. Related to architecture is a close relative called end-to-end systems integrator role, I've touched on systems integration topic on a previous post called Worlds of System Integration, so will not discuss integration in this post. Related to end-to-end integration is testing, in my past experience, integration implied testing or QA, but as I later learnt on coming to back to South Africa, most companies still see QA as a separate activity, so I have another post that talks to the Worlds of QA testing.

Why?

Because it makes running a project so much simpler! 

When I work with an E2E architect, I use it as a vital supporting pillar of the project. The E2E architect will take care to understand the business or product requirements from the business owner, works with all the respective technology domain experts (usually the assigned system architects) on mapping out a solution architecture, defining the new technology dictionary and model (if needed), highlights the system building blocks and supporting components that would be impacted, agrees on high level technical requirements from such systems, as well as important integration points, technology risk assessment, along with unearthing required non-functional requirements (performance, stability, redundancy, etc.) that would be needed for a complete solution. All of this is kept high-level, with just enough information to allow the project team to shape and scope the project, as well as prepare the technical teams of design constraints to think about. All of this is done in the early stages of shaping and scoping a project, has proven in my experience to be absolutely vital in a successful project outcome.

To be assigned an E2E Solution Architect to a project is no small feat. The expectations are quite high indeed. The role not only requires a sound grasp of technical concepts, but also mandates a personality that is comfortable with extreme forms of collaboration, managing and dealing with diverse stakeholders, personalities and egos (don't forget egos - tech guys can be a difficult bunch of people with huge egos), a strong suite of leadership skills, self-discipline, self motivation and the ability to work with ambiguity, a responsible attitude to self-management of tasks (without depending on a project manager to co-ordinate every meeting for example). Added to that, the E2E Architect must have excellent communication skills, demonstrating both in written and verbal form, the ability not only to handle technical communications, but also be the person to translate to business stakeholders in a simple, non-technical manner.

I rely on this architect to also grasp technical and architectural issues that might come up in delivery, and be able to provide solutions and advise on ideas and proposals to manage any impediments on the overall solution provisioning.

Why can't you just feed the business requirements to each division?

The short answer is the risk of lack of coherency of an overarching, end-to-end solution design. The absence of an E2E architect acting as the gatekeeper for maintaining overall integrity and coherency of the business requirements and related solution design, runs the risk of each division designing and implementing fairly independent solutions, leading to fragmentation, increased integration and last-minute rework to get a coherent solution delivered. This also leads to a system implemented without foresight, lacking the vision or with the end in mind. An E2E architect would fill this void, save a lot of time and energy in putting all the pieces of the puzzle together. 

Though not impossible to complete a project without the E2E architect, it is often through a lot of last minute co-ordination efforts of a project management team, or some efforts of a few technical heroes coming together, taking collective ownership of the problem. Generally these units have other work and projects going on concurrently, where focus on a single project is difficult to achieve. If the environment of very open collaboration and team work, where there aren't really any silo-mentality, then yeah, it's possible. But the reality is that in most organisations, silos exist, collaboration is not the norm, and the overhead of a project management team coordinating these activities is too high, let alone the implied technical knowledge of these project managers to make this a worthwhile activity.

This reminds me of the story about Christopher Wren, the architect responsible for rebuilding St. Paul's Cathedral after the Great Fire of London. Wren was walking the length of the partially rebuilt cathedral when he asked three bricklayers what they were doing. The first bricklayer responded, "I'm working." The second said, "I'm building a wall.". The third paused, looked up, and then said, "I'm building a cathedral to the Almighty."**

How many teams fully appreciate the work they do in the context of the end-to-end ecosystem? It is not expected every team have this kind of vision, it would be great if they could all see the big picture and be visionary, but in reality they're not - and there's nothing wrong with that. The instructive part of Wren's story is that he didn't come up with a sense of purpose himself and pound the vision into everyone's head. Each bricklayer cared about something different, even though all three were working on the same thing. Wren's role was to listen, to recognise the significance of what he heard, and to create working conditions that allowed everyone to find meaning in their own way.

In the same way that Wren appreciated the vision, the E2E architect plays the same, working with, collaborating, listening and negotiating with individual teams. Feeding a business requirements spec to each technical division independently, without a golden thread joining everything together, is not the most effective way of maintaining coherency.

[Next section: Why a solution architect and not a technical project manager instead?...]

Why a solution architect and not a technical project manager instead?

There are indeed cases where a technical project manager, assuming one who has the requisite technical experience of being a developer, architect, integrator of a few technology domains, could very well play the role of solution architect, if the organisation allows for it, can afford to do it and the environment is pretty fluid or flat. This person must have had a track record of solution architecture of course, and has the respect of the technical community, having gained the trust and credibility to do justice to the role.  However, as a project manager, trying to balance both technical and project management capability is not an easy thing to accomplish. In reality both functions are pretty demanding and time consuming, trying to find the balance and remain impartial in either role can be difficult to say the least. The job of the solution architecture is to collaboratively emerge a solution design that would solve the problem at hand, working with known technical constraints. Sometimes, more often than not, it's difficult to push the project management hat aside and not think about delivery timelines. 

In my roughly twenty years of software and systems engineering profession, I've not seen anyone handle both disciplines without compromising on one or the other. Generally, solution architecture remains a discipline of architecture, working closely with a technical project management team.

Project management in itself is exhausting. Mapping out a network of tasks, keeping an eye on the risks and issues, looking ahead into the future, managing all kinds of stakeholders is a full time job, often needing just enough high level technical knowledge to keep things in motion, relying on a strong team of architects to maintain the true integrity of the solution design.

Remember I'm talking about large-scale projects here, not a project that involves just one system, but multiple of systems. I've seen some project managers moonlight as both architect and project manager, but these were just limited to one system entity.

However, it has also been my experience (and I've been guilty of this myself and thankfully have overcome this) is the tendency of project managers, people who were once quite technical and left technical for project management, is to assume they can still lead technical solution design workshops, and offer their opinions on architecture. To this I say, Stop! You are no longer technical, your role is not a solution architect anymore, sure review the outputs for quality, but don't get carried away by offering your own solutions, business rules or process design. Leave it to the experts...

What are the core deliverables for an E2E Architect?

As a minimum, the supporting deliverables an E2E architect must provide the project management team are the following. Remember it is about spending just enough time upfront in analysing the problem domain and emerging a high-level solution design that speaks to solving the current problem and designing a realistic solution with the future in mind. It is not about detailed system & component design. The deliverables are time-bound, aligned to the start-up shaping & scoping milestone of a large project:

• High-Level technical translation of the business or product requirements - say a high level technical specification or brief. It does not need to be a lengthy document, but brief enough to capture the core product use cases and related supporting technical or system use cases
• High-Level Architectural view of the solution landscape showing all systems, components and interactions needed. This is usually the form of an end-to-end block diagram, showing systems and components from the various technology providers. Colour-coded blocks usually show the different entities responsible for the solution. This helps the project planning team identify all role players involved.
• High-level requirements of each major system / component. For each major system use case, what are the associated requirements (brief description, not necessarily too detailed), just enough detail to show relationships and components impacted. These requirements usually take the form of a simple table, listing the system/component, product use case, system use case and associated behaviour/requirement from the component.
• Updated models (domain model) if required, showing the new lexicon or dictionary and glossary of new terms the new solution being designed for.
• High-level interface definitions, focusing on core interactions needed to maintain the integrity of the system. This could be sample XML, interface definitions, new protocol descriptions (in cases where transport signalling is extended, headers and payload data to be interpreted, etc.). Expectations in terms of behaviour, like synchronous or asynchronous calls, polling, messaging behaviour, etc.
• Leading an end-to-end Proof-of-Concept (POC). In instances where the initiative is completely brand new, or there is no close industry equivalent (that is, no other industry player has invented it already), or even if there is an industry equivalent but the existing technology ecosystem is divergent from the established norms, that some investigative work is needed to prove the concept, then the E2E architect must lead this POC team and feedback outcomes in the form of proposals, scenarios, etc - to be fed into the project planning team.

In producing these deliverables, the E2E architect must collaborate with his/her peers: fellow system architects, component architects, technical leads, as much collaboration and design workshops needed to uncover a coherent, fluid end-to-end solution design. The architect is not expected to know everything, sometimes there is rapid learning involved, enough learning to be able to understand how impacted systems work. The E2E architect is not expected to know in detail the nuts and bolts of every system, that's what we have system architects for. 

From a project management perspective, it's simpler and more efficient to have a single technical owner on the project, assigned the role of E2E architect, to represent the various different technical teams. 

Simple rule of thumb, one project manager, one solution architect. It's clean, no room for interpretation, and simpler to manage. Suffice to say, there is a very close relationship expected to be built between the project manager and the architect.

What are some common challenges in introducing E2E Architect role?

As I highlighted in the beginning, this role was always assumed by me as a natural role in any large project, but I later came to appreciate some of the challenges in advocating for this role, here are just a few topics to be aware of. Most importantly, it's down to role clarification.

• Tension between Functional Role & Project Role of Architects
• When the situation is as shown in the first picture showing distinct functional silos, each with their own architects within the group, where there is a clear absence of a role called E2E solution architects, one is bound to run into people and emotional challenges in understanding how a project role differs from a functional job title of existing architects.
• Misunderstanding and misinterpretation of what the role means. Immediate assumptions take the form of "So do my architects now report to this new E2E Architect? Why should my architect report to him? Who chose this guy as end-to-end architect? What are his qualifications? This guy doesn't have a clue on what or how we do things around here? Who's this guy and why is he asking us architecture questions?. So who's going to do my people's appraisals? What does HR have to say about this? Is HR involved since this is a change to the job spec?"
• This usually stems from a lack of understanding of why the role is needed, and the importance it serves in the context of a project.
• Sometimes senior management agree with the concept, and go to the extent of assigning an architect (from another division) and it comes as a surprise to rest of the technical teams.
• This can be solved by better up-front canvassing of all technical stakeholders before an assignment is made. Once there's alignment and agreement, it's the responsibility of senior management to convey the message to all impacted teams.
• It is usually difficult for the project management team to be the conveyors of this message, especially with organisations where the PMO (Project Management Office), and project management teams are not a recognised discipline or part of the culture, such as a project-driven organisation. 
• Where there isn't a separate group of architects belonging to a distinct "Solution Architects" group, or where the architects are dispersed across the organisation without a unifying architect's group, designating a solution architect usually takes the form of answering the question "Which systems and components are most impacted by the new business / product requirement? The one with the most impact, usually dubs a system architect from that division as the E2E solution architect."
• So some serious care and thought must be given in the assignment, otherwise the role will probably fail and delay the project, leaving the project manager to pull all the pieces together.

• Confusion over the role & expectation of business or systems analysts & architects? 
• Admittedly the experiences that led to my anchor bias of solution architects speaks to my first decade of working with technology solution providers that never really involved a BA (Business Analyst) role - if they existed, I had no interactions with them, but as far as I can remember, we did not employ business analysts. External customers may have transmitted their high level briefs that were the product of a BA exercise. I also worked with internal product development, but we had no formal BA role writing business requirements. Product managers would create a brief, a couple of pages or a PowerPoint presentation. The rest was up to the architects. I always viewed requirements analysis and elicitation a function of architecture. So in my new role entering South Africa, I came to realise companies still relied on BAs for doing business and technical analysis - a separate role, completely separate to architecture. But the more I observed, the more I realised the lines between architecture and analysis can get blurred and leads to all sorts of confusion:
• Where does business requirements start and end?
• How much technical detail should a business requirements document (BRS) cover, if any?
• What documents should the product managers create? 
• Why isn't the BRS created by a product manager?
• When does the architect take over?
• Does the architect assume the BRS is gospel and cannot be questioned?
• What access does an architect have to the owner of the business requirement?
• Should a BRS contain technical component impact assessment along with a high-level picture?
• If so, what then is thee use for an architect?
• Often the project management team is left to deal with these ambiguities and grey areas, has to manage the underlying tension and resulting conflict. If the organisation hasn't taken time to clarify the purpose of the role, the assignment of a BA, an architect, and responsibilities thereof, this extra work falls on the project manager to iron out - again exacerbating the functional problem of lack of clearly defined roles.
• My default expectation is to just have an architect fulfill the role of requirements elicitation straight from the source (business or product manager), but when that is not an option, then make sure the boundaries are understood loud and clear.

• Each silo has different ways of working or operating models
• Some technical divisions might have an entrenched way of working that does not support the concept of E2E Solutions Architecture. This is generally the case where this division functioned largely in isolation, typical of IT business systems, where it was fairly okay to receive BRS requirements that spoke specifically to its systems, without too much interdependence on interconnected systems. With this model, they would lead and create interfaces with other systems and components, that these other components must just accept as the way to do things, "it's the API, so use it". This approach does not work well with integrated, connected systems coming together to deliver a coherent user experience. Nowadays everything is connected, a simple frontend application maybe designed in a way that offers the best user experience possible, but unable to do so, because the backend APIs were implemented in isolation, using a closed-view of the BRS requirements without appreciation for the full end-to-end experience.
• So the challenge is when kicking off a project that calls for an E2E architect, to work with these stakeholders and agree to a slightly different way of working. Sometimes this is very hard to do, because a lot of canvassing needs to happen to get people to accept this new project's expectation.
• The biggest change is the need for some upfront collaboration with the respective architecture teams. The E2E architect role spends the majority of its time in the project start-up phase: shaping and scoping / design. Some divisions expect to be given a brief, leave it for their own architect teams to spend a few weeks (assuming there's capacity and the resource planning schedules this time) analysing, then come back. This often conflicts with the project's expectations of closing the architecture design, but not impossible to plan for, once all stakeholders are aligned and there's general agreement and acceptance of the role.

•  The Agile Debate - we don't do upfront design thinking
• All I can say here is that teams must accept their current reality of the full organisational context in which they operate. 
• Going back to the opening picture: it shows a reality of where a project needs to deliver on an outcome that relies on a coordinated work effort from several disparate and independent technology providers. These providers might be different companies, or under one overarching technology division, with segments of speciality - that is a unit. Each unit has their own way of working. Some could be using a Waterfall approach, whilst others adopting an Agile method - Scrum, Kanban or whatever flavour of agile.
• Reality check: The organisation is not agile.
• Reality check: A project needs to be delivered.
• Reality check: All differences needs to be set aside for the greater goal of the company: deliver the project.
• Reality check: No individual unit is going to convince a big organisation with different culture-sets to adopt another unit's way of working, i.e. convert to agile. Agile transformation is a journey in and off itself.
• In the case where a unit has adopted an agile method AND there still exists a distinct architecture group (some agile purists would shun the very existence of a separate architecture team), the typical challenges that come up are:
• We don't do any formal architecture documentation, this only dis-empowers the development team
• We need to engage with the development team and let the team come up with the architectural impact
• We don't want to specify APIs because this will make the developers feel side-stepped that we're making all the decisions for them
• This is not how agile works, we can't contribute to a solution design until we understand the user stories, get them prioritised and inject them into sprint planning where the design would unfold
• These are all plausible reasons that must be respected, and if the effort was not an end-to-end initiative and just restricted to the team responsible, then sure - no problem with that. But this is not the case for large programs. A large program is going to impact many teams, it's going to cost the business in time, money and resources that must be budgeted to some degree (it does not have to be a high degree of certainty, just good-enough) - how does the E2E architect deal with meeting the project expectations, and getting the support from the respective teams??
• The project manager again, has to respect the culture and the stakeholders, canvassing support for this new role of E2E Solution Architect on the project. Once the need is accepted and agreed by these agile stakeholders, then you work together to integrate this new expectation into the team. Some solutions would be to allow for E2E Architect support in every iteration / sprint, agree on schedules that synchronise with planning workshops, etc. Remember the E2E architect is a collaborative role, in line with agile principles of interactions with individuals & customer collaboration. The E2E Architect is an indirect customer associated with a project. Working with the architects and development teams in the same session, an E2E architect can achieve the outputs expected. 
• The team will retain their freedom to design internal system & component architecture as they see fit, as long as the external interactions and interfaces are agreed upon upfront realising the world is much larger than themselves, that development streams will run in parallel and often in times ahead of others - design contracts must therefore be agreed and committed to. The job of the E2E architect is to resolve these contract debates early, and signal the project that teams are ready, aligned and work can begin.
• See next sections for additional thoughts of agile challenges. Also see my post on Agile Good Hype and Ugly.

Is there a solution for organisations instead of repeating each project with the same challenges?

Personally, insisting on upfront solutions design architecture is something I will probably never change, especially when given the task of running a large program of work. It is rooted in sound software and systems engineering, that offers the bedrock of principles that should not be ignored, or if ignored will be asking for trouble and run the risk of derailing the project and causing a number software development problems. I have come to appreciate this style of working having witnessed first-hand large complex projects deliver successfully on the back of a carefully thought-out solutions architecture design process.

If after trying a few projects in this manner, and the success is proven, then it will naturally lead to this concept being entrenched into the functional organisation. This might be through growing the architecture competency within the organisation, establishing new roles for solution architects. 

Increasingly organisations are wrestling with integrating agile concepts, often confusing sound software engineering practices with the philosophy of agile principles as shared in the agile manifesto, I've written in the past that people tend to fall in love with these principles alone, forgoing the necessary diligence needed in implementing complex systems, end-to-end. I'm not sure why this is the case, maybe it's to do with the proliferation of modern apps development (and the young developers born at the turn of this century), context does matter - the Digital TV Systems environment involves software spanning the spectrum of services: highly-reliable fault tolerant services to apps development. All these systems come together to deliver an experience, and methods of delivering these systems are not the same. There is a growing movement in the industry to promote software engineering principles, a framework called SEMAT Software Engineering Methods and Tools is an attempt to bridge this gap, that shows that software & systems engineering principles are not incongruent to the methods of agile delivery. The framework supports many agile methods, including Scrum, but the principles remain firm. 

Having an E2E Solutions Architect is not a departure from core systems engineering - it is needed, and every project should have one.

---
** Credit for this story goes to Kim Scott, from her book Radical Candor