On E2E architect role in projects

I have shared my thoughts in the past on the subject of architects in the domain of digital TV software systems which can be found here. In this post I will share further thoughts about the specific role of an E2E (End-to-End) architect from a program/project management perspective, including my expectations of the role as well as the associated challenges I've faced with this concept, especially where the role either does not exist, or is not clearly defined in cases where companies don't have a clear enough understanding or have not yet set up a formal architecture competency.

My Background & My possible Anchor Bias

In my first decade of working experience with software and systems engineering projects, I was fortunate to work with world class technology service providers (S3 in Ireland & NDS/Cisco in UK). This experience helped shaped the way I would view future technology projects, including product & project management, software engineering, integration and delivery functions. It's the kind of experience that never leaves you, especially when you've witnessed first hand, one successful delivery after another both as an engineer and manager. So I'd come to see the world in a particular light, taking with me the style of project execution where ever I went, and assumed that all companies naturally followed similar concepts. And the cool thing about this experience was a well grounded appreciation for software & systems engineering design principles - principles which should not be confused with implementation methods, as what is a hot topic of debate these days, that of Agile V Waterfall, that has been the bulk of my recent challenges in projects in the last six years.

On returning back home to South Africa* still working in Digital Media TV domain, I soon realised that things are a little different to say the least (absence of architect roles, working with business analysts where I'd been used to analysis being a function of architecture, no defined role of systems integration, no method of continuous delivery & lack of solid post-launch monitoring & operations support systems). My first major program, involved a large-scale overhaul of the end-to-end broadcast system, including a video-on-demand component and a new set top box software stack. In order for the program to have a decent chance of success, I not only had to restructure & reshape the program, but also had to introduce functional roles that for me, were always obvious: an architecture team with a specific role of E2E architect, E2E systems integration, test and delivery teams. With a lot of hard work of convincing senior management these roles were necessary for project delivery, we managed to implement the structures I proposed that ultimately helped in the successful delivery of the project.

Strangely enough, as I took on other engagements in large programs in the same group of companies, I found myself repeating the same. It could be because I have a bias that is so strong and deep that does not allow me to see any other way, even though I always try to adapt my style to the culture and teams currently at my disposal, yet I still find myself coming back to these principles because its been proven and tested, that I can't really find any real fault in - possibly it is because these are indeed engineering principles that have stood the test of time, especially this concept of an End-to-End Solution Architect role. As the saying goes, never leave home without it, if I'm running a large program of work, that involves many systems & component suppliers, I never run a program without having an E2E architect attached to my programs, as step one. Once that is established (often as a result of canvassing support which takes up some time and energy), I move on to structuring the role of E2E systems integration, which in turn drives the processes and behaviours expected from component engineering and test teams. The program timeline then shows a simple picture of the high-level milestones involved, where the bulk of the program management and execution is a function of coordinating delivery plans with respective project delivery teams (who maintain the detailed project plans) as well as managing stakeholder engagement (which is a vital component of running large programs). This hasn't been smooth sailing all the time, as I'll try to share some of the challenges that frequently come up, later on in this post.

*Disclaimer: Although my writing is about an experience with primarily one industry domain in South Africa (Digital PayTV), I'm mindful about not passing a broad value judgement against all large corporates with a software competency. At first I thought these challenges of projects and roles like BAs for example was limited to just this environment, but the more I researched by skimming through job specs, attending networking events, technical conferences, meetups, training courses and meeting people from other corporates, I later realised the patterns run across all the major corporates like financial services, banking and the big telcos, that more often than not, the structures in place mirror more traditional-IT governance than hardcore technical software development structures that I was used to, in working in a pure technology services environment. So I'm reasonably confident there is no broad value judgement being applied here, that these experiences are likely to resonate with large companies operating outside PayTV.

Typical Landscape

The picture below aims to illustrate a typical landscape of this ecosystem. This could be separate entities run independently, or even group under a technology division, with different functional expertise, that in reality are still perceived as separate silos of responsibility:

This picture speaks to a typical reality of technology service providers in a digital TV value chain. Each silo represents a set of core technology expertise that exist in separate functional lines either under a technology division, or could exist as separate lines of business altogether. 

Each silo may have its own set of speciality with architects and development teams. Each unit may also likely to have a very unique culture: people, skill-set and ways of working. Each silo may also have its own operating model: how work gets prioritised and injected, build, test and release procedures. These units might also have their own terminology, a lexicon / vocabulary for their specific domains. They may also not be entirely self-sufficient, relying on external third-party component suppliers that form part of their system. These third-party suppliers are just like another silo, each with their own operating model, from architects to development teams, to ways of working as stipulated by their underlying contractual agreements.

These systems & teams usually come together in delivering a product or service that directly impacts a customer (end user). Typically launching a major new product (or platform, say for example a new internet product like a streaming app for video on demand) will impact at least all these systems. To do so, a project is set up as a program delivery initiative that will call for the co-ordination of all systems coming together in delivering the new product or feature to market. So a program team would usually be setup to manage this execution and delivery. Typically a program manager would be appointed to manage the full end-to-end delivery, which is not only the technology arm, but also the wider business streams for delivering product to market (marketing, customer care, sales, training, retention, communications, legal & regulatory). Sometimes, the overall program manager is supported by an end-to-end technical program manager, but this is not always the case. More often than not, the overall program manager assumes co-ordination of the technology ecosystem as well. For now though, I use program manager to either mean end-to-end technical program manager or overall program manager interchangeably.

The need for an End-to-End Solutions Architect

The picture below illustrates the positioning for a dedicated role of E2E Solution Architect to a project or program of work:

Just as you would appoint an end-to-end project / program manager to the role of maintaining the coherency of the overall delivery project, so too should you assign an end-to-end solution architect role, to maintain the integrity and coherency of the overall technical solution design. This is the crux of my proposition. Related to architecture is a close relative called end-to-end systems integrator role, I've touched on systems integration topic on a previous post called Worlds of System Integration, so will not discuss integration in this post. Related to end-to-end integration is testing, in my past experience, integration implied testing or QA, but as I later learnt on coming to back to South Africa, most companies still see QA as a separate activity, so I have another post that talks to the Worlds of QA testing.

Why?

Because it makes running a project so much simpler! 

When I work with an E2E architect, I use it as a vital supporting pillar of the project. The E2E architect will take care to understand the business or product requirements from the business owner, works with all the respective technology domain experts (usually the assigned system architects) on mapping out a solution architecture, defining the new technology dictionary and model (if needed), highlights the system building blocks and supporting components that would be impacted, agrees on high level technical requirements from such systems, as well as important integration points, technology risk assessment, along with unearthing required non-functional requirements (performance, stability, redundancy, etc.) that would be needed for a complete solution. All of this is kept high-level, with just enough information to allow the project team to shape and scope the project, as well as prepare the technical teams of design constraints to think about. All of this is done in the early stages of shaping and scoping a project, has proven in my experience to be absolutely vital in a successful project outcome.

To be assigned an E2E Solution Architect to a project is no small feat. The expectations are quite high indeed. The role not only requires a sound grasp of technical concepts, but also mandates a personality that is comfortable with extreme forms of collaboration, managing and dealing with diverse stakeholders, personalities and egos (don't forget egos - tech guys can be a difficult bunch of people with huge egos), a strong suite of leadership skills, self-discipline, self motivation and the ability to work with ambiguity, a responsible attitude to self-management of tasks (without depending on a project manager to co-ordinate every meeting for example). Added to that, the E2E Architect must have excellent communication skills, demonstrating both in written and verbal form, the ability not only to handle technical communications, but also be the person to translate to business stakeholders in a simple, non-technical manner.

I rely on this architect to also grasp technical and architectural issues that might come up in delivery, and be able to provide solutions and advise on ideas and proposals to manage any impediments on the overall solution provisioning.

Why can't you just feed the business requirements to each division?

The short answer is the risk of lack of coherency of an overarching, end-to-end solution design. The absence of an E2E architect acting as the gatekeeper for maintaining overall integrity and coherency of the business requirements and related solution design, runs the risk of each division designing and implementing fairly independent solutions, leading to fragmentation, increased integration and last-minute rework to get a coherent solution delivered. This also leads to a system implemented without foresight, lacking the vision or with the end in mind. An E2E architect would fill this void, save a lot of time and energy in putting all the pieces of the puzzle together. 

Though not impossible to complete a project without the E2E architect, it is often through a lot of last minute co-ordination efforts of a project management team, or some efforts of a few technical heroes coming together, taking collective ownership of the problem. Generally these units have other work and projects going on concurrently, where focus on a single project is difficult to achieve. If the environment of very open collaboration and team work, where there aren't really any silo-mentality, then yeah, it's possible. But the reality is that in most organisations, silos exist, collaboration is not the norm, and the overhead of a project management team coordinating these activities is too high, let alone the implied technical knowledge of these project managers to make this a worthwhile activity.

This reminds me of the story about Christopher Wren, the architect responsible for rebuilding St. Paul's Cathedral after the Great Fire of London. Wren was walking the length of the partially rebuilt cathedral when he asked three bricklayers what they were doing. The first bricklayer responded, "I'm working." The second said, "I'm building a wall.". The third paused, looked up, and then said, "I'm building a cathedral to the Almighty."**

How many teams fully appreciate the work they do in the context of the end-to-end ecosystem? It is not expected every team have this kind of vision, it would be great if they could all see the big picture and be visionary, but in reality they're not - and there's nothing wrong with that. The instructive part of Wren's story is that he didn't come up with a sense of purpose himself and pound the vision into everyone's head. Each bricklayer cared about something different, even though all three were working on the same thing. Wren's role was to listen, to recognise the significance of what he heard, and to create working conditions that allowed everyone to find meaning in their own way.

In the same way that Wren appreciated the vision, the E2E architect plays the same, working with, collaborating, listening and negotiating with individual teams. Feeding a business requirements spec to each technical division independently, without a golden thread joining everything together, is not the most effective way of maintaining coherency.

[Next section: Why a solution architect and not a technical project manager instead?...]

The Project Team as a Relay Team

The analogy of a relay race team has been used a few times in project management. It is a great analogy for project teams because it reflects the passing of the baton, from one player to the next, after each player completing a set distance, just as with projects consisting of multiple teams dependent on each other for the ultimate success of completing the project. The passing of the baton, the execution and seamless delivery from handover-to-pickup-and-sprint more often than not (assuming the runners are the same) will account for a win or a lose the race moment.

What triggered me to write this post is not the relay race applied to a project team responsible for execution (typical critical chain analogy of passing information to teams waiting, managing the interfaces so that it's clear, etc.), but rather I wanted to focus more on the actual PMO (project management office) or project management team itself (typically the group of project managers assigned to the project). Where there is a large endeavour, a project of more than 12 months up to say 36 months, those long running projects that have a team of project managers responsible for overall delivery, then with this context too, I believe the relay race analogy is quite apt and relevant. It becomes a relay race for project managers: the baton is passed on to a new project manager, with each passing of the baton, the project receives a burst of new energy, enthusiasm, keeping the project alive. For long-term projects, I believe it is vital to renew the energy by bringing in new people often replenishes the project, breathes new life and momentum...as long as you choose wisely and get the timing just right.

If you're leading a PMO (Project Management Office), you will have at your disposal program managers, project managers & project administrators - all of varying degrees of skills, experiences, competencies, strengths and weaknesses.  Similarly a program manager of a large project, will have have access to a similar compliment of people, making up the project team. Whether you're leading a PMO or a running a program, as the senior project professional, you assume the role of project leader. As a project leader, you create your project management team and are responsible for the life of the team.

Project management teams are no different to other teams. They also go through the stages of development: Forming, Storming, Norming and Performing. They too have their own challenges, just like any other team would. I believe that as a project leader, it is important to keep abreast of your project management team's competencies, you need to be able to gauge the energy of the team and tell when it's time to pass on the baton. Even as a project leader yourself, you might have to pass on the baton to another project leader to see the project through to completion. And I don't see this passing of the baton as a reflection of failure on the part of the manager, it is just a fact of reality. A true leader will know when to move on instead of causing harm to his followers. A project manager should know when its time to pass the baton on to another project manager for the best interest of the project.

On long-term projects, keeping the team in tact can be challenging. Depending how far along the project is in execution, you run the risk of the work becoming routine, rote and uninteresting. Once things become mechanical, which is not a bad thing for the project, but for the human being serving the role as project manager, it might start to get tedious. Of course, I'm not passing a value judgement on all project managers, everyone is different - some people love staying the course, from start-to-finish, must-see-it-deliver mindset, whilst others might want to leave just as the corner is being turned with the project close-to-completion, and all downhill from there.

Examples of Relay Race for Project Management Teams

Looking at the classic PMBOK stages for project life-cycle: Initiation, Planning, Execution, Monitoring & Control and Closure, or, looking at a more broader stages of project leadership: Shaping & Scoping, Start-up, Delivery & Closure - what if your project managers mastered a particular stage of the life-cycle? 

With a specialist PM per stage of life-cycle, you play to people's strengths. As much as people might want you to believe that a project manager should be versatile and be proficient with all stages of the project life cycle (heck, that's why there's certification right?) - it has been my experience that this is not the case.

There are some people that are really good at starting projects but not finishing. Some are great at big picture strategy, scoping, shaping and starting-up, but lose interest when it comes to execution and delivery. Whilst some people are great at execution, monitoring and control, but lack the depth, breadth and wisdom of big picture envisioning to dream up a decent project charter, engage in meaningful dialogue with stakeholders and visualising the strategy. Some people are great at detailed mapping of project plans in Microsoft Project, detailed status reporting and tracking but fail to see the big picture. Some people have so much energy to close out the last phase of delivery, but find themselves at a loss when starting a project on a blank piece of paper. Some people experience fatigue mid-way through the project, others are sprinters who love the stress of the last mile.

Yet, when all these people form part of a bigger project team (the project relay team), magic happens. Each PM plays to his/her strength during the project life-cycle, handing over as needed, just-in-time before the project becomes a burden to complete...the project heartbeat is maintained and can live to completion.

So what am I saying then, really?

In the past I have handed over my projects as if I was handing over the baton in a relay race. I saw my passing the baton to another project manager as a vital component to the success of the project. A runner can not run the relay race alone, so in the same vain, for large projects that demand a level of endurance of project management execution, you must set yourself up with a project management team that would be capable of passing on the baton...I have in the past, introduced new project managers into my team specifically to introduce "new blood" and new energy as way to prevent project fatigue (it works)...

Past Experience

I once worked on a very large project that lasted about five years, with me joining the project in the second year. The project was one of the best projects I've worked on, it shaped me and thought me the hidden gems of project management that no PMBOK book or certification course will teach you. We were a team of five project managers based in UK, with peer PMs placed in France, India & Israel. Overall the full project team was between 350-500 people. We had a customer that was ruthless and very demanding, the customer also had a complement of 5 PMs in the UK.  

Ship of State Metaphor

I recently completed Jim Benson's book "Why Limit WIP: We are Drowning in Work". It's a simple, straightforward read, in the form of a story around typical life pressures in a corporate running multiple projects, all with the same sense of urgency, but teams not actually shipping any product (that is, completing any work product and releasing to market).

Jim introduces a metaphor called the Ship of State, as a way to help the team's product and project managers navigate through the uncertainty, thus allowing the team freedom to decide (empowered) what is safe and within their control to make decisions and others where the team still need to act, but still keep the stakeholders informed of their thinking. The scene is around product features for roadmap, which I thought quite powerful and useful - with a much wider application than just roadmapping features.

So I created this little poster that captures the essence of the metaphor. Does it makes sense?
Think about what your ship of state would look like (personal or professional)....

Ship of State

Signs of Stress every Project Professional should know

When you're working on projects you will experience many challenges: from working on multiple, often conflicting and ambiguous priorities, multiple streams of work (tasks, work packages, etc.) as well as interacting with many different types of (difficult) people, from different cultures, behaviours to different languages spoken. Your customers (stakeholders, client, sponsors) will vary in quality, more often than not applying pressure for their work getting done (because more than likely its linked to some kind of personal objective / business reward). This pressure is usually passed down to the project team members assigned. With the focus on delivery & timeline pressures, the project manager is often forced to multitask, expecting the same from project team members who often have the challenge of juggling between between project and non-project work, multi-project work (generally people are assigned to more than one project at a time) and not forgetting people's own personal/life (family, health, etc.) challenges thrown into the mix.

All of this can become too much, and as a project professional (project manager), who's primary responsibility (in my humble opinion) is to ensure his people are led, directed, guided, coached through the implementation phase of the project thus meeting expectations of the customer and coming away intact, that it behoves the project manager to be aware of the common signs and symptoms of stress. After all, we work with human beings, not machines that are oblivious to mood swings, emotional problems and simple pleasures - so being mindful of your team members state-of-mind, as well as your own -- and taking time to truly pause and reflect, adjusting your behaviour as a project manager can go a long way to not only improving your work relationships, but also can help with the the successful outcome of your project implementation.

The American Institute of Stress (AIS) stated that the term "stress" was coined by Dr. Hans Seyle in 1936, who defined it as a "the non-specific response of the body to any demand for change". He sometimes defined it as "the rate of wear and tear on the body".

Here are some fifty common signs and symptoms of stress from the AIS. Look out for these in your own life as well as the people you work with.

• Frequent headaches, jaw clenching or pain
• Gritting, grinding teeth
• Stuttering or stammering
• Tremors, trembling of lips, hands
• Neck ache, back pain, muscle spasms
• Light-headedness, faintness, dizziness
• Ringing, buzzing or "popping" sounds
• Frequent blushing, sweating
• Cold or sweaty hands, feet
• Dry mouth, problems swallowing
• Frequent colds, infections, herpes sores
• Rashes, itching, hives, goose bumps
• Unexplained or frequent allergy attacks
• Heartburn, stomach pain, nausea
• Excess belching, flatulence
• Constipation, diarrhoea
• Difficulty breathing, sighing
• Sudden attacks of panic
• Chest pain, palpitations
• Frequent urination
• Poor sexual desire or performance
• Excess anxiety, worry, guilt, nervousness
• Increased anger, frustration, hostility
• Depression, frequent or wild mood swings
• Increased or decreased appetite

• Insomnia, nightmares, disturbing dreams
• Difficulty concentrating, racing thoughts
• Trouble learning new information
• Forgetfulness, disorganisation, confusion
• Difficulty in making decisions
• Feeling overloaded or overwhelmed
• Frequent crying spells or suicidal thoughts
• Feelings of loneliness or worthlesness
• Little interest in appearance, punctuality
• Nervous habits, fidgeting, foot-tapping
• Increased frustration, irritability, edginess
• Overreaction to petty annoyances
• Increased number of minor accidents
• Obsessive or compulsive behaviour
• Reduced work efficiency or productivity
• Lies or excuses to cover up poor work
• Rapid or mumbled speech
• Excessive defensiveness or suspiciousness
• Problems in communication or sharing
• Social withdrawal and isolation
• Constant tiredness, weakness, fatigue
• Frequent use of over-the-counter drugs
• Weight gain or loss without diet
• Increased smoking, alcohol or drug use
• Excessive gambling or impulse buying

Check out this Human Function Curve by Nixon in 1979, that illustrates the generally accepted view of the effect of stress on worker productivity. It is entirely personal and shows that each person has their own individual limits for stress - there are good stresses and bad ones. Zero stress = Zero performance - interesting.

Human performance peaks at modest amounts of stress. (Source: Nixon, p: Practitioner 1979)

Credits / Reference Material
I came across this topic from this book, Chapter 2 on The Human Behaviour Problem as Root Cause: Multitasking from Critical Chain Project Management by Leach

The Immutable Laws of Project Management

I came across this classic list a while back and was reminded recently about it in Critical Chain Project Management by Leach, so I thought it useful to store as reference (yes, it's already online, but I actually enjoy writing things down as a way to reinforcing my memory).

I can personally identify with all these laws in my history of working on projects, how does your experience compare :-)

The Immutable Laws of Project Management

LAW 1

No major project ever completes on time, within budget, with the same staff that started it, and the project does not do what it supposed to do. It is highly unlikely that yours will be the first.

Corollary 1: The benefits will be smaller than initially estimated, if they made estimates at all.

Corollary 2: The system finally installed will be late, and will not do what it is supposed to do.

Corollary 3: It will cost more but will be technically successful.

LAW 2

One advantage of fuzzy project objectives is that they let you avoid embarrassment in estimating the corresponding costs.

LAW 3

The effort required correcting a project that is off course increases geometrically with time.

Corollary 1: The longer you wait the harder it gets.

Corollary 2: If you wait until the project is completed, it is too late.

Corollary 3: Do it now regardless of the embarrassment.

LAW 4

Everyone else understands the project purpose statement you wrote differently.

Corollary 1: If you explain the purpose so clearly that no one could possibly misunderstand, someone will.

Corollary 2: If you do something that you are sure will meet everyone's approval, someone will not like it.

LAW 5

Measurable benefits are real. Intangible benefits are not measurable, thus intangible benefits are not real.

Corollary 1: Intangible benefits are real if you can prove that they are real.

LAW 6

Anyone who can work effectively on a project part-time certainly does not have enough to do now.

Corollary 1: If a boss will not give a worker a full-time job, you shouldn't either.

Corollary 2: If the project participant has a time conflict, the work given by the full-time boss will not suffer.

LAW 7

The greater the project's technical complexity, the less you need a technician to manage it.

Corollary 1: Get the best manager you can. The manager will get the technicians.

Corollary 2: The reverse of corollary 1 is almost never true.

LAW 8

A carelessly planned project will take three times longer to complete than expected. A carefully planned project will take twice as long.

Corollary 1: If nothing can possibly go wrong, it will anyway.

LAW 9

When the project is going well, something will go wrong.

Corollary 1: When things cannot get any worse, they will.

Corollary 2: When things appear to be going better, you have overlooked something.

LAW 10

Project teams detest weekly progress reporting because it so vividly manifests their lack of progress.

LAW 11

Projects progress rapidly until they are 90 percent complete. Then they remain 90 percent complete forever.

LAW 12

If project content is allowed to change freely, the rate of change will exceed the rate of progress.

LAW 13

If the user does not believe in the system, a parallel system will be developed. Neither system will work very well.

LAW 14

Benefits achieved are a function of the thoroughness of the post-audit check.

Corollary 1: The prospect of an independent post-audit provides the project team with a powerful incentive to deliver a good system on schedule within budget.

LAW 15

No law is immutable.

My library on Project Management

Here is a live list of all the project management books I own in my personal library. I have read every one from cover to cover. Each one has helped me in some way or the other in mastering the art and skill in project management, that I now find myself less interested in the detail of project management, but rather more interested in the project leadership aspects of the domain: setting strategy, starting-up and shaping projects, forget date-drive tracking and focus on building commitment instead, building relationships, program report visualisation (one page project manager reports, Toyota A3, etc.) & simplification (cut-down meetings, reduce waste, manage work-in-process), running & facilitating workshops (brainstorming, scoping, discovery, pre-mortems, retrospectives), organisational change & transformation, playing coach, guide, mentor and advisor to PMOs & Execs.

This list is in no particular order but towards the bottom end are the most recently read books. Not sure where all the icons disappeared to, but the links should work.