Architect Roles in Digital TV Projects

The subject of IT architecture is vast and multi-faceted; often open to interpretation by organisations in the same and different businesses alike; especially the definition and expectations of the "architect" role in projects in supporting the technical needs, administration and day-to-day operational management. A number of people attempt to explain what architecture means, from professional institutions like IEEE & TOGAF that provide a framework for describing the various architect roles for IT projects to attempts by wikipedia.

The rationale behind my post is to highlight how the role of architect manifests in companies specialising in digital TV systems, be it on the broadcast systems engineering front, or product developer offering systems and software services. It has been my experience that even with companies involved in the same industry, the use and understanding of the role "architect" varies, and is open to interpretation, often leading to confusion when you're mixed up in customer-vendor service relationships in managing projects. Some companies do attempt at having an organisational structure supporting technical architects, whilst other companies try to hire anyone with a technical background and assign the role "architect". But the aim of this post is NOT to describe the traits and skill-sets of architect VS coder; but to focus on the functional roles that an increasing number of DTV projects are now demanding from architects.

If you're a PayTV operator like BSkyB, DirecTV, Multichoice, etc - you need to think about the level of technical expertise required for your business:

• how much technical systems knowledge do you need to own?
• how much do you depend on external vendors to manage this on your behalf (e.g. the design of your end-to-end system)?
• do you have to produce & own your very own technical specifications for the entire value chain; or just key parts (e.g. EPG Spec, Middleware Spec, UI Spec)?
• or are you comfortable relying on your preferred system partners?
• do you need to have technical people on your side to prevent vendors from pulling the wool over your eyes?
• do you need technical owners for each product/sub-system of your deployment?
• do you want to drive or your suppliers/vendors or be driven (locked-in) by your suppliers solutions?
• do you opt for generic products to configure & customise yourself, or do you impose a custom requirements to serve your purpose, to meet your business needs, according to your environment?

This list goes on & on... There are an increasing number of PayTV operators that are moving away from the traditional vendor-supplier-user model, to taking more ownership and control over their products. Some operators have set-up internal development houses to manage Headend & STB UI/Middleware development; whilst other operators are taking over STB manufacturing in an attempt to control the end-to-end value chain.  In such cases, it therefore becomes even more relevant to have a solid technical product team, connected from end-to-end, with a thorough understanding of complete system, delivery and value chain of products, from high level features, requirements and design to detailed technical operations of the product --- this is not just my opinion, some companies have already embarked on this path (BSkyB, DirecTV, Foxtel).

If you're in the business of providing systems & software services to the above-mentioned operators, the likes of NDS, OpenTV, Irdeto and other Professional Services companies, then you probably have well defined roles for architects, and probably have an organisational structure of an architects group, supporting your many projects, with a group of chief architects as overseers for your product deployments.  If not, then please contact me as I'm interested in learning about what others in the industry are doing here :-)

I've been mulling over writing about DTV architects for a while now, but what really triggered this humble attempt stems from a recent experience on one of the projects I'm managing: I'm sure most of you are familiar with "It's not my problem, I'm an application architect not a systems architect" - you look for a systems architect only to find that one doesn't exist yet!

Anyway, to take this discussion forward - I consider the end-to-end layout of a typical Digital TV system. I'll present the different flavours of architect roles, and conclude laying out the minimum expectation of the roles that must exist to fulfill your typical DTV product deployment, with a bias towards the STB-side than the Headend.

This is NOT a theoretical or academic exercise. I'm describing real world experiences from past projects I've worked on with NDS/BSkyB/DirecTV/Foxtel/OpenTV, although the major influence is from NDS who, in my opinion, do an excellent job at executing complex DTV projects; and have an org structure not so dissimilar to what I'm presenting here.

I am really interested in your comments and feedback, what is the rest of the industry doing in this area?? Feel free to comment directly or drop me an email.

Consider this picture of a traditional end-to-end Digital TV system:

Figure 1: Components of DTV System Value Chain (Download the PDF)

Disclaimer: I'm no architect, this is Version 1, just my understanding and interpretation only. Architects do the best they can under the circumstances....there is a level of pragmatism involved and no one should be under the pretense that the solutions created are perfect, they are what they are i.e. a solution out of a continuum of many possible solutions for the problem which solves a business need at that point in time for a price the organisation can afford under the assumptions of the perceived known knowns and anticipating the future usage for that solution at that time.

A typical end-to-end DTV system is complicated; there's certainly nothing simple to it. At the highest macro level, one could summarise a DTV system as being split fundamentally into three areas (or in architect-speak, subsystems):

• Transport/Signalling: This is basically the physical layer, the profile of the underlying delivery mechanism. It can be abstracted and treated as a separate system as the products that sit on top of this layer still provide the same application behaviour, regardless of medium. I've separated it out because here you'll find physical hardware/infrastructure systems that prepare the data for the transmission across the appropriate delivery mediums (Satellite, Cable, IP, Terrestrial, Mobile)
• Producer: Headend/Backend: Systems that generate, secures & distributes content and information over the delivery medium (satellite, cable, terrestrial, mobile, IP)
• Consumer: Devices - typically the STBs receiving the transmission, decode and presenting the info and a suitable user experience to the subscriber at home

Within each area, the system is broken down further and further down to individual components performing a specific task. In reality, you will find that this end-to-end system is a carefully orchestrated integration of a number of different subsystems; offering a mix of different hardware and software components that are supplied by different companies. This is especially true for the back-end, integrating third party components such as the Conditional Access (CA) system, Billing & Subscriber Management, Scheduling, Information Data Generators, Content Catalogue Servers, Play-out systems, Encoders & Multiplexers.    At the consumer device end, the STB is a system in itself, also broken up into components or subsystems; and also not provided by one single company: Typically the STB consists of an Application Engine / User Interface component (the EPG), Middleware (the TV operating system) and Platform Components (Drivers, Hardware, Kernel). And these core STB components themselves are broken down into sub-components.

So your end-to-end DTV system is complicated & highly component-based. These sub-components, components, sub-systems, systems and ultimately the end-to-end solution must be co-ordinated and controlled -usually all these systems obey strict rules and contracts as defined by Interface Control Documents - Architecture 101...

So where do the different architect roles fit in then?

At the macro level, an end-to-end system could be managed by a team of architects specialising in these broad roles, be it at the Headend or STB domain:

• End-to-End (E2E) Systems Solutions Architect
• This is the top of the hierarchy, the solutions architect has a birds-eye view of the end-to-end system. Generally working directly with the the business (marketing/sales) in understanding the business and market driving factors in creating a solution for a specific need or product. 
• The E2E solutions architect is someone who can work with any technology domain, has a deep appreciation for all the subsystems including third-party external components, and will work with the relevant system architects in defining the high-level solution. This results in producing a grand, high-level design of the system components or building blocks, specifies the requirements and interfaces expected from each subsystem, laying the ground rules for system architects to take forward, adapt as necessary to apply the design in a real-world deployment.  
• The solutions architect is a domain expert, expected to take existing deployments and adapt, expand and enhance as necessary to deliver new product requirements.  This is generally the first stage of the projects design and scoping phase.  
• In some projects the solutions architect goes right into the detail of enhancing the MPEG/DVB signalling specification; and some times even creating a new end-to-end signalling protocol altogether. 
• This is a highly collaborative role, involving a lot of communication back-and-forth with sales/marketing, product development and integrators.  This could be one person, or a team of solutions architects taking responsibility for certain slices of the solution, depending on the scale of the project and problem being solved.
• Some people use the traditional IT term "Enterprise Architect" - but in all the projects I've worked with, the DTV industry tends to use "Solutions Architect". 
• In a nutshell, the solutions architect uses existing subsystems to deliver a solution to a problem and identifying shortcomings/gaps for attention Systems/Product Architects

• End-to-End Product Systems Architect
• Depending on the project and product being delivered, the solutions architect usually takes an existing solution and adapts in to the needs of the new product. In doing so, the solutions architect will touch on many components or subsystems that are impacted by this change, thus highlighting the areas that make up the new product.
• One might therefore assign specific Product Architects to take responsibility and own the components making up the product's value chain. 
• Primarily concerned with interactions between subsystems where Headend and STB is subsystem.
• The Product Architect assumes responsibility for the product solution and works hand-in-hand with the solutions architect in defining the solution that best meets the needs of the business.
• Solutions architects typically get involved at the initiation stage of the project and leave to solve the next challenge - they rarely see a project through to completion.  Product Architects however are expected to stay till the very end, supporting and making all architectural decisions to get the product from development through launch.
• This is still a high level responsibility, product architects work closely together with respective system products architects: Headend & STB.
• The product architect drives through the respective product specifications, covering functional and non-functional requirements (performance, reliability, resilience, etc).
• This role should not be confused with Product Owner or Product Manager; which are more management roles focussed on product features, specifications and high level requirements, which feed the work of the Product Architect.  Depending on the size of the organization and skill-sets available, these roles could be merged.
• Some people tend to refer to Product Architects as Technical Owners.

• System Architect - The system architect spans Headend & STB subsystems:
• STB System Architect / STB Product Architect
• Responsible for the end-to-end stack of the STB covering platform hardware, kernel, driver layer, middleware and EPG - Responsible for the overall STB product, taking ownership for the full product.
• Has a deep understanding of the key areas of the STB to the point of specifying the interfaces for all the above components
• Collaborates with respective EPG & Middleware architects in designing the overall STB solution
• Is responsible for defining the Functional & Non-Functional Requirements of the system: e.g. monitor/stipulate memory usage and own performance
• Helping the product team with requirements, and that includes backlog, supporting the initial planning phases and producing high level estimate
• The gate-keeper for analysing change requests
• Assisting in analysing, reviewing & prioritizing defects

• Headend System Architect / Headend Product Architect
• Whist the STB is a fairly closed system, the Headend on the other hand is an  integration of distributed systems. These systems are all interconnected, the Headend product architect is concerned with the interactions, interoperability and reliability of the integration.
• Works closely with other architects (subsystem/component architects) in defining the product specification. Since the Headend is meant to provide generic services and is a high-risk operation, Headend systems are generally well-thought out in advance, go live months before the STB is ready for deployment. The product architect plays a big part in driving the integration of these systems at the high level.
• Product architects generally own slices of the system, e.g. VOD
• All these slices, i.e. product architects work closely with the Solutions Architect for Headend, sometimes referred to as Chief Architect

• Component Architect - STB
• Application Architect - UI / EPG
• Essentially designs the UI / EPG as a component. Responsible for the internal architecture of this component
• Collaborates with Middleware Architects as necessary e.g. influencing APIs and interfaces
• Works with the STB Product Owner to understand requirements from high-level to detail
• Works closely with user experience specification team to ensure the UI demands are realistic, respecting the boundaries of the architecture of the UI's internal components
• Supports the product team in defect analysis, review and prioritization
• Implement STB system architect's Functional & Non-Functional requirements
• Assist the programme management team in high-level planning and estimates

• Middleware Architect
• Middleware is a system in itself, consisting of multiple components. Without drilling down into too much detail, the Middleware Architect ensures the component satisfies all requirements by providing services to applications using it
• Generally, the STB Architect is in fact the Middleware architect; using Middleware sub-component architects for detail. I worked on a Middleware product that had a centralised STB architect driving a team of component architects, where the Middleware was split into 80+ components...
• Defines APIs both internal and external to the Middleware
• Works with the STB Product Owner to understand requirements from high-level to detail
• Supports the product team in defect analysis, review and prioritization
• Implement STB system architect's Functional & Non-Functional requirements
• Assist the programme management team in high-level planning and estimates

• Platform Architect - Kernel/Drivers
• Responsible for the design and specification of the low-level drivers and kernel interfaces, typically the industry refer to this as Hardware Abstraction Layer (HAL) or Common Driver Interface (CDI)
• In general STB hardware specifications, once a platform launches, undergoes minor updates over time - the interfaces are generally future-proof, and new interfaces try to be backwards compatible, so the Platform Architect is usually required at the start of the project to define the major hardware features and enhancements...

Example scenarios of where this hierarchy is applicable?

Of course, projects don't always have to adopt the complete structure. Real world example projects applicable to this model include: Introducing Hybrid-Satellite/IP features offering Progressive Download (PDL), PushVOD, Broadcast Downlod, PullVOD/IP, Recommendations, Integrated Search, Application Stores, Catch-up, Network PVR, DRM - all these products impact the end-to-end value chain and needs a carefully orchestrated architecture and project execution team.

Why I think projects need to have clearly assigned architect roles?

Projects without a well defined technical team are bound to face difficulty and will most likely fail.

Projects need to have access to a technical team, there must be ownership and accountability for the different areas of the system. More importantly, the product must be connected end-to-end, the value-chain must offer a coherent design solution. Some broadcasters operate separate business units that not well connected. Just like a project team is set-up to manage the project delivery, it is vital to set-up your technical solutions team, assigning people to support the product and project from start to completion.

Take for example, a common project for DTV companies nowadays of changing your STB supplier, be it changing EPG UI or the STB Middleware. For simplicity, assume there are no changes required in the HeadEnd, we're just want to swap out old for new.  At the very least, the project should have the following identified as key roles:

• End-to-End Systems Architect
• STB Architect (doubling as Product Architect)
• EPG UI Architect

At the end of the day, we need to deliver a product. We need to go live and make some money. Having clearly defined roles for architects on your project team will:

• provide clarity, removes confusion over roles, responsibilities & expectations
• highlight gaps in your organisation - for e.g. you might have an EPG architect, but missing a STB architect - this is only going to cause problems down the line for integration.
• provides interfaces and eases communication channels
• promotes a coherent system - without this, chaos looms on the horizon

If you're a broadcaster and lack the skill-set and competencies for these roles, then you should demand this from your vendors. 

I'm an operator, do I really need to go into that level of detail?

Again it depends on how you answered the opening questions, fundamentally how much do you want to be in control?? BSkyB & DirecTV for example, have  STB Middleware & EPG architects; as well as STB & Headend System architects.  BSkyB produce their own Middleware design specification, in addition to requirements. BSkyB are very involved with end-to-end solution design, often taking weeks to review and comment on design proposals. As an operator, BSkyB have a strong technical team who understand the technology implementation of their entire value chain and can influence architectural discussions and decisions from both Headend & STB-side.  It is your product after-all, so why not control it as much as you can? At the macro level, it's down to trusting your suppliers and risk mitigation. Suppose your vendors disappear, do you know enough to support, enhance and maintain your products in their absence??

Useful Reference Material on the Subject

Who needs an architect anyway?

Architecture Definitions

Software Engineering Institute on Architecture definitions

How should an IT Architecture team be organized within a large company?

The Role and Task of the System Architect

The Role of the architect

Difference between coder and architect

The Role of the Agile Architect

Principles of the Agile Architect

Useful book on System Architecting

MIT Coursework on Architecture

MIT Coursework on Role of Architect

Jumbo Universal Remote Control - missed a trick!

This weekend I played tech support to my father-in-law, aged 70 - he was at war with his remote controls. His DSTV remote wasn't working properly, the power button was not doing anything, so he couldn't switch off his TV.  He'd gone out and bought not one, but three jumbo universal remote controls from the local warehouse that sells almost anything, mostly knock-offs made in China.

So he shows me this Jumbo Universal Remote:

At first glance, this remote looked interesting. Very nice, large buttons, good contract. Very nice for the in-laws. Alas, this remote wasn't suitable for my father's-in-law set up.

But here's the really annoying part. The user manual for this remote is a teeny tiny booklet, using a font-size that even I had trouble reading!

I am passionate about accessibility as you'll see from a future post on Talking TVs, a personal project of mine. This lack of regard for the users really annoys me. How can you create a large jumbo remote control and ship a really small user manual with it - I mean what's the point? Are you playing a joke? Are companies so insensitive to the needs of their users??  There is a large and growing user base of people who need accessible products, this market should not be ignored. New products must cater for this group from day one, at the design phase -- accessibility is rarely difficult to add-on post production. Most products that attempt to add accessibility afterwards, and not making it inherently native to the product, fail to deliver...unlike Apple products. Apple, despite their modern and ultra slick products, are surprisingly accessible and being used by many with vision problems (blind/partially sighted)...this is not just my opinion, ask Stevie Wonder for his opinion :-)

At first, I thought the remote must have been a cheap Chinese clone. But no, go to JumboRemoteDotCom and download the user manual and see for yourself!  That's not the only thing - take a minute to think about the users of this remote control, such as my father-in-law: The instructions are virtually impossible to comprehend...sigh.

There are indeed TV companies out there that care about accessibility, and go to lengths spending time and money on R&D to produce decent accessible remote controls. Take BSkyB in the UK for example, who provide accessible remote controls free of charge to their subscribers needing assistance.  More recently in South Africa, DSTV have also produced a large remote control, although not free for subscribers due to a somewhat different business model and market to BSkyB. This company from US also have some interesting products on offer.

DSTV Jumbo Remote:

BSkyB Remote:

Pressure versus Urgency - is there a difference?

Last week I experienced an interesting interaction with the lead architect on my team, who'd been away for several days on a conference; and come back to find things not to his liking - the quality of the deliveries was not up to standard, people were not adhering to the architecture -- his take was that the team are under enormous pressure from me to get things done and delivered at the end of the sprint.

This caught me by surprise at first - how odd, I've not even started applying any pressure...But I've grown used to the different kinds of people on the project team; and hence didn't react negatively or emotionally.  My reaction:

"I appreciate your concern and understand where you're coming from. I think there's been a misunderstanding, perhaps due to my not making myself clear. There is a difference between Pressure and Urgency.  Yes, there's a sense of urgency around us finishing work according to plan because people are depending on work being created, but at no point should we compromise quality  - there was no pressure to take short-cuts; the architecture rules must be adhered to, no working off a branch, everything must be done on the main product, adhering to all quality standards..."

Having satisfied this architect, who left away happy that we were on the same page, I was still left with a sense of uncertainty myself. The team have probably got the concepts mixed up, although to be fair, there is rather a vague line between the two terms Pressure & Urgency -- but big difference with how these terms manifest themselves through action.  In the spirit of collaboration I then fired off an email to the team highlighting the difference in the context of our project and business needs, and then settled the matter at the retrospective. It turned out, as so often is the case with these things, the architect misunderstood what the team had been attempting, working on a branch only to prove concepts and theories that necessitate rapid development at the expense of some the architectural bottlenecks, i.e. we willingly broke the architecture to prove a few points....

Nevertheless, I still wonder if I may have unconsciously contributed to applying artificial pressure. I will explain the nature of my current project in a future post - Suffice to say my role is blurred, call it "Pseudo-Scum-Master-trying-to-balance-Hard-Delivery-Project-Manager-Development Manager-with-a-slight-touch-of-Product-and-Programme-Manager". I am seriously not blowing my own trumpet here, we are an evolving project team where the (unwritten) roles&responsibilities that were supposedly assumed well defined is in reality quite blurry - no surprise to software development projects, and especially not surprising when its an organisation's first attempt at truly in-house software development...

Definition of Pressure according to Free Dictionary

The ones that stand out for me are:

• the act of pressing
• the condition of being pressed
• the application of continuous force by one body on another that it is touching; compression
• a compelling or constraining influence, such as a moral force, on the mind or will: pressure to conform
• urgent claim or demand: under the pressure of business
• to force, as by overpowering influence or persuasion.
  an urgent claim or demand or series of urgent claims or demands to work under pressure

Definition of Urgency

• the quality or condition of being urgent; pressing importance: the urgency of the call for help; pleading with urgency.
• a pressing necessity
• all hands and the cook A state of emergency which of necessity becomes everyone’s top priority. This early American cowboy expression described the precarious state of affairs in which the herds were wild and all available persons were needed to bring the situation under control. Under normal circumstances, cowboys tended herds and cooks fed the cowhands; however, an emergency required that everyone chip in, temporarily ignoring differences of rank or task.
• the state of being urgent; an earnest and insistent necessity

Agile lends itself well to Urgency

The very nature of Agile/Scrum with its fixed sprint cycles, the two-week time-box for example can be in itself quite pressurising. There is a natural sense of urgency to ensure that enough detailed planning is done up-front, the work is well understood and the team hit the ground running on day one of the sprint. In Ken Schwaber's Agile Project management with Scrum, he talks about how important it is to do detailed planning, that agile doesn't ignore sensible process; and the importance of delivering something at the end of the sprint.

Incidentally, the same architect who talked to me about pressure, at my interview took pride in saying that the team would do whatever it takes to finish the sprint  according to plan - work overtime, come in on weekends, etc - what gives? According to Dave@PracticalAgility, projects with urgent goals are prime agile candidates, but he also cautions not rushing along to do things quickly, and the importance of taking a pause - which coincidentally is what I've proposed to senior management just yesterday (only saw his post today as I googled on this subject)...

Is there really a difference between Urgency & Pressure?

Yes, there is. It's not just my take on it - Others outside of software face similar challenges. Take for example the Pressure VS Urgency from Real Estate, although software projects are a little more peculiar.

People Under Pressure Don't Think Faster!

Tom DeMarco dedicates chapter 15 "Think Fast" of The Deadline to the effects of pressure. The Deadline is an interesting tale as it covers almost every experience one encounters in software projects, especially the unrealistic and nonsensical demands from senior management placed on their underlings, with the poor project manager acting as the buffer. "Think Fast" is about the main stakeholder changing direction and enforcing his might just because he can, and this sets the team down the road of discussing & modelling the effects of pressure. It concludes by asking the ever-knowing Oracle:

Why does the effect of pressure on programmers max out after only 6% productivity gain?

Oracle replies:

PEOPLE UNDER PRESSURE DON'T THINK FASTER -- Tim Lister

DeMarco then summarises the effects of pressure (Page 199, The Deadline):

• People under pressure don't think any faster
• Extended overtime is a productivity-reduction tactic
• Short bursts of pressure and even overtime may be a useful tactic as they focus people and increase the sense that the work is important, but extended pressure is always a mistake
• Perhaps managers make so much use of pressure because they don't know what else to do, or are daunted by how difficult the alternatives are
• Terrible suspicion: The real reason for use of pressure and overtime may be to make everyone look better when the project fails

The same message is discussed in the more serious book, outside of the novel context, in DeMarco's "Slack", Chapter 7, titled "The Cost of Pressure". Hopefully you can see this snapshot, please get a copy of this book.

Essentially confirming what is now well known: Pressure has limited capacity to reduce delivery time, maybe 10 or 15 percent at the most. Excessive pressure weakens performance. The model is divided into 3 regions:

• Region 1: workers respond to increasing pressure, eliminating waste, staying late, focus on critical path - falsely suggesting improved delivery
• Region 2: workers are getting tired, feeling pressure from home, doing other stuff "undertime" - no motivation, unwilling to sustain, resigning to ignorance
• Region 3: workers are polishing up CVs and beginning to look elsewhere

Closing Summary - Software projects can't hide from Pressure/Urgency

I have in the past worked on many a demanding project and I've always challenged management's decisions of increasing the pressure points, always quick to point out that adding pressure doesn't make people work any faster, it'll take as long as it takes, people will leave the project, etc. I must admit though, that I've come to learn there is indeed a difference between urgency and pressure; but it takes some tact to get it right. When people are motivated by an urgent need - urgent needs can be good motivators - people surprise themselves - problems are solved in novel ways, there is more focus and concentration. There is a natural pressure of course, but not the kind of pressure from a demanding boss expecting results now now now - if the urgency is communicated well, it brings out the best in people. If it's just imposed pressure, people turn to acid and can't wait to leave the project.

Applied in bursts, pressure/urgency can be a great tool - ultimately though, you're working through people - so ensure people are rewarded and appreciated for their efforts!

I have personally been on projects that may have seemed like Death Marches, but appreciated the sense of urgency to get things done, the reward was self-gratification is seeing the product deployed in tens of millions of people's homes, or providing crucial services to improving the company's bottom-line. Yes, some  projects could definitely been managed differently to avoid burnout, but sometimes, in the world of business this is an unfortunate reality; and especially in today's globalised economy - the hard truth of the matter is, engineers are a dime a dozen..

It is still important though to make sure your team understands the context of the business, the phase of the project and have a sense of awareness of the demands being placed on them, giving them the right and freedom to express concerns of pressure impacting the deliveries.

Practical Retrospective on Agile Retrospectives...

If you read my previous posts on Agile, you'll know that I'm not claiming to be an Agile expert, nor a pukka Agile practitioner that abhors classic project management principles, evangelising the merits of Agile/Scrum to everyone, etc - but I do believe in the core principles of the method and am applying as many of these principles in my day-to-day activities in my current company, where I'm a part-time Scrum Master.

I'd like to share my experience with running Retrospectives. Reading about the topic is one thing, going on a training course or seminar is definitely enlightening, but practising the thing in real world scenario is another experience all together. It is especially challenging when the team are used to doing it a certain way, have certain misconceptions or are not familiar with the goal of the process.

Take my current team for example - we're a team of 20 developers & testers, with a handful of business analysts, graphics artists that make up the product team. We're working on an advanced user interface for STB EPGs. It's no typical project because 18 months down the line the requirements for the UI completely changed [maybe it is your typical S/W project :-)] and we're dealing with a lot of uncertainty.  I'll discuss the nature of this challenge in a future post, and how different streams of Agile can be applied to remedy the situation.

Focussing on the retrospectives for now, you need to understand the nature of the team. A Scrum Master is not just a buffer, is not just someone who can facilitate and remove impediments, is not just someone who knows and applies Scrum processes properly. A Scrum Master must also be in-tune with detecting the make-up of the team, understanding the nature of each individual, and apply different tactics for different people. Whatever management courses teaches you about knowing the characteristics of your people, their temperaments, strength deployment index & motivational values -- all of this must be noted and applied in the daily management of Scrum teams.

It is especially relevant when it comes to running a retrospective.  In my team for example, we've got two development units. One unit comprises a team of contractors from India, being led by an Indian who's been living in South Africa for the last seven years. Most of the guys are from the same region Kerala, and gel quite well as a unit.  The second unit is dominated by white South Africans, mostly Afrikaners & a native SA Indian. The former unit, is quite reserved and look for direction from the team leader. Generally the team leader speaks on behalf of the team - this is typical culture of the Indian way of working.  The latter unit however, is a very loud, outspoken team - people are not afraid of talking (mostly out of turn), and venting their issues.  In the midst of each team, are representatives from the Product team, consisting of very experienced analysts (who've never used Agile before, as well as some really new people who've not worked with STBs before). The product team supply the work to the development units, and thus are part of both teams retrospectives. So we two separate retrospectives (we also do two, now going on three, separate stand-ups, planning meetings & planning reviews).

We run two week sprints, we've just finishing Sprint 23. Up until Sprint 21, the retrospectives was all about - "What are the issues, How can we fix them...". In Sprint 22, we introduced the "What's worked well, What's not worked so well & should stop,  What can we do better/You like to add".  I kept the format of the previous retrospective, but introduced a the retrospective as a game. Essentially my message to the team was that Sprints are personal experiences for each member, it's an individual experience - so we want to hear from each team member on his/her take of the Sprint...

The rules of the game:

• Each member in the team must express an opinion - must voice something.  (The Indian team are normally reserved, and leave the talking to the team leader). Enforcing this rule, then compels people to participate
• Don't be afraid to say anything - we will not judge you, interrupt you or challenge your response (Some members are really loud, physically imposing, sometimes intimidating and rash when speaking that can be misunderstood by most people as being directed personal jabs, people are shy or insecure about saying something that makes them look stupid) - So remove all of that out, and create an atmosphere of safety
• No speaking out-of-turn - Wait for your turn to speak please. Stop loud people in their tracks, they must observe the rules
• Give everyone a chance to have his/her say. You can only comment on what topic at a time - e.g. first round looks at "What is working well" - repeat as required until you run out of suggestions, then start the next topic
• Do not try to solve problems in detail in the retrospective - we identify the top 10 burning issues that must be resolved in the next sprint, assign owners to actions
• Keep-to-time: Strong time-keeping, do not stray off-topic. Force people to stick to the topics
• Allow a few minutes of silence time for people to collect their thoughts and write their suggestions/concerns on a post-it (We usually provide food during the retrospective as we run this over lunch period - so it's a good ice breaker, eat and think about what you're going to say)

The result??

It was amazing - the Indian team participated fully. We heard each member speak, some for the first time. People observed the rules as best as they could, and enjoyed contributing. So quite positive for this team. The second unit shared similar response, although it was much easier to get this team to speak (it's usually hard to get them to shut-up!) - but with the rules in place, even the loud-mouths had to hold their tongue, wait for their turn, etc. We also didn't end up trashing other people's ideas - and we tried to stick to the time limit. 

A lot of good stuff came out of the retrospective - I hope we can carry this forward.  To conclude, Retrospectives can be fun, but some work is required upfront to ensure that not only you as the Scrum Master gets the results you need, but also the team should leave the session having felt they've made valuable contributions, listened and paid cognisance to other people's problems and most importantly, trust the outputs of the session will create value going forward....

Agile Checklist from the Trenches

I've been working on projects for a number of years now. Whilst I'm not claiming to be a certified Scrum Master (which personally I don't think you really need to get certification), if you go about managing projects with a sense of common sense and realism, adhering to core principles, not only adopting a process but ensuring consistent implementation of the processes -- then you're bound to be successful.

Most of my experience with Agile/Scrum comes from running multi-site projects, with teams situated in different parts of the world, with shared product ownership. One can maintain the spirit of Scrum in such projects, but it requires some management and oversight...compared to the classic small person teams of 10-30 people located under one roof, in the same building, on the same floor. Physical location is important. It is really quite interesting the effects of geographical location of development teams plays on the success of the project, the dynamics of the team, and the overall end-goal of product quality. It's not as straightforward as this, just Google for "multi-site software development". But still, on principle, managing a small local team of mixed individuals versus a global team, incurs less management overhead but share underlying Scrum philosophies.

This post is a brain dump of my lessons learnt and observations so far with Scrum/Agile, the headings comes from an Agile Refresher workshop I attended last year at my previous company but the explanations are all my own opinion.

Hopefully you apply some of these learnings to of your own Scrum projects:

Best Practices - multi region projects

1. Team Setup

• No one man teams
• Ensure all people are given equal opportunity to choose tasks, even though there might be an expert in the team that can do most, if not all the jobs on his/her own. Avoid allocating work to specific technical experts, embrace knowledge sharing. Use the experts to help with user stories and work distribution
• Consider and assign people with multi site collaboration skills
• Working in a multi-site team requires to have people who are skilled at soft skills, a natural talent for bringing people together and engaging in meaningful discussions despite cultural differences.
• Consider if you have to duplicate roles on sites, e.g. have a proxy Product Owner (PO) or Development Owner (DO)
• If the local and off-site teams are equally sized requiring similar management overheads, consider adding proxies on the remote site - as this enables some central control and synchrony. Challenge with this of course is for the local & remote POs to communicate and understand each other well

2. Align Expectations

• Set clear vision and get team buy-in
• It's important to set clear goals for the project, and directions for all the teams. Propose, instead of direct how the project will be structured, involve the team in fleshing out the organisational structure since the team members know best what their strength's and weaknesses are - bring all of this to light at the start of the project creates an atmosphere of inclusion, better chances of getting team buy-in, because the team will have contributed to defining the vision and be determined from day one to actively contribute to the success of the project
• Define roles and responsibilities
• This is important to all development processes, not just multi-regional.  Clear roles and responsibilities must be defined so that people understand what's expected of them. In local and remote teams, this thus avoids duplication of work, people are less-likely to step on each other's toes, or choose work without first understanding the process of getting new work assigned to the backlog. In the case of strong minded technical people who are confident and self-motivated, try to give some responsibility and autonomy to these guys for example, being the proxy PO or DO. 
• Define decision process
• This is quite important in keeping with defining the roles and responsibilities. Decisions need to be made quickly, it's about reacting to the current need, wearing the hat of pragmatism. Beware of strong minded and loud personalities, ensure that everyone buys-in to the decision process - be clear as to where the buck stops! 
• Define success criteria (for a  multi regional collaboration)
• In the end we not only want the project to be successful in terms of meeting its objectives, we want the team, the people that led to completing the project, finish the project feeling a sense of accomplishment, having learnt something and for the team to grow from the experience. 

3. Communication

• Set-up Communication Structure
• Face-to-Face
• This is always preferred where practical. Certainly with local on-site teams, F2F should be encouraged. Discourage email and phone calls especially when the team is co-located. Be sure though to limit the number of meetings - meetings must have a purpose and add value to the project.
• Video Conference (weekly)
• The next best thing to face-to-face is Video conference - especially with remote teams. Ensure you have the right tools that enable a please VC experience.
• Email Distribution Group
• Setting up an email distribution group is useful for quick, directed communication. Beware of unnecessary email though.
• Reporting (team internal, group, external stakeholders, etc)
• Agree on the reporting mechanisms up-front, with clear owners for reporting. Keep the local Scrum management fluid, less admin heavy and try to make public whatever the team is up to. Do not spend too much time generating reports that no one will read.
• Iteration planning
• This might come as a surprise to most people new to Scrum, but you have to have a clearly defined iteration planning process, including pre-planning, analysis and design, etc - ensure the planning sessions are well communicated in advance, keeping an up-to-date backlog that is publicly accessible is also good practice.
• Demos
• Demos are key to communicating to stakeholders the progress of the project. Ensure regular demos are held, made publicly available - communicate via email, wiki or blog - but get the message out to your target audience!
• Shape Communication Culture
• No matter what school of project management you come from, there is an underlying problem in all: Communication, communication, communication. Instil this culture by:
• Provide feedback
• Continuously seek out opportunities to provide feedback at team level, or even at individual level.
• Create Information flow (using email group and other means)
• Don't hide information - open and transparency helps as long as you're confident the team can deal with this information, otherwise it has to be limited information flow
• Make sure everyone has the necessary information, be inclusive
• Everyone one the team must have the information to help them get the job done - there should be no secrecy
• Be open about language issues - create a culture where it's OK to ask for repetitions
• This is important because for most remote teams, their first language is most likely to differ from the core team on location. Cut-short whatever cultural blame-games are going on, instil openness and be frank that there is the language barrier - the team must respect this difference and be patient to repeat discussions or instructions if required.

4. Agile/Scrum

• Create common understanding of what Agile means for your project, including Retrospectives every iteration, Shared Backlogs, Iteration Demos, etc.
• It is important the entire team understand what Agile/Scrum means and how the project is going to adopt those principles. Get all misunderstandings corrected and clarified at the outset, for e.g. the myth that documentation is not required, etc, etc.

5. Team Building

• Early Face-to-Face meeting - cross pollination
• As pointed out above, face-to-face is personal, adds human touch - helps with getting to know one another. This should be encouraged
• Build Trust
• Generally managers (POs/DOs) take a long time to build trust - ensure that when people come to you to solve problems/impediments, that you act immediately. This will instil confidence in the team that management is listening and can be counted on to solve issues. Protect the team if things are not going well - managers are a shield to the team.
• Create Team Spirit (we want the team to succeed)
• As covered above, encourage peer dialogue, pair programming, etc. Promote team exercises, challenge team in other ways to possibly solve problems not related to work - all of this will get people to gel together, building team spirit
• Be Respectful
• Especially with multi-site teams with cultural differences, respect is important. This can be learnt by promoting cultural awareness. Where the POs/DOs are senior technical people themselves, allow yourself to step away from the problems and let the problems be solved by the team
• Empower - give people responsibilities equally across regions
• Where there is interest from people who have shown competence and can be trusted, then continue to add more responsibilities. Where teams are separated geographically and the skills are equally matched ensure that work is evenly distributed. Where there is a skills gap, try to bridge that gap with doing knowledge share sessions, etc - promote cross functional knowledge, avoid domain experts.
• Recognise Success/Failure equally
• Remember we're all in this to learn. We're working with people - projects will come and go, but the people will most likely remain the same.  Reward successes, and see failures as just delayed successes. Failure is just one step to success, to wisdom - try to understand the reason for the failure and then help the team to overcome this.  Make sure the team adopts a "no-blame, no name-and-shame" attitude.

6. Management

• Project Management (PO & DO) to follow team closely
• Multi-site development teams require some close management, not micro-management - but POs/DOs/PMs must ensure there isn't any hidden festering issues. All the while the management team must adopt the stuff described in the above points
• Be aware of the culture you have in the project/team
• Management team must be aware of the culture of the project, and have realistic expectations
• Intervene when experiencing problems, e.g. by doing Force Field Analysis / Root Cause analysis with the team
• Where required, if you spot the team going down the wrong path - then step in and offer assistance. Don't dictate, be tactful in approaching the problem - ensure you get to the root cause
• Exchange ideas/best practices between projects, e.g. knowledge-bridge
• Helping other projects out as a result of learnings from your project is a nice way to network, promote knowledge-sharing and instil values in the company of sharing knowledge rather than having competing teams
• Inspect and Adapt
• As always, no process is perfect. Neither is a process static. You have to continuously reflect on your processes, team, behaviours, outputs, etc and be willing to change direction, adapt your processes and be open with failure as well...