ORITs: One Roof Integration Teams for DTV/STB Projects

In my previous post I introduced the concept of Release Campaigns that are a series of cycles executed to reach stability & maturity of the STB Product, which largely happens during the Integration / Stabilisation phase of the project's life cycle.

During these campaigns it is expected to uncover a number of bugs or defects, especially during the first release campaign, where the various system components are actually being stressed for the first time, end-to-end. Expect a reasonable avalanche of defects reported against STB functionality, stability & performance areas. The team that faces the brunt of these defects is usually the STB SI team, who need to investigate, reproduce the problems, and characterise the defect assigning to a relevant vendor to fix. This process is known as "triage" in some circles.

How does the project team manage dealing with these defects? How do we control the quality of the system ensuring the focus is maintained, that we're not spinning our wheels? How do we mitigate the project from further slippage, removing the burden on the SI team? How do we ensure we focus on the areas that are the pain points, the core features and functionality that add value, and affect the bottom-line: customer experience?

Even though the project is likely to have defined clear guidelines for acceptance criteria & defect severity & priorities as discussed in this post, the project will still have to be directed in a manner to maintain a sense of urgency to get the burning issues fixed.

Two areas are almost certain to happen on every STB/DTV delivery project:

• Business / Product owners have their own view of describing the product, as a separate list of feature areas, despite what is defined in all the product documentation
• STB SI team will become the bottleneck and on the critical path if the standard processes are maintained - the project will slip, unless some other practical solution is found

Enter the Hit-Squad or ORIT Teams

So we set-up dedicated teams to address key functional areas that cause us pain. Teams are focused on particular areas, with the sole remit of clearing away all problems. Typical areas that cause pain in a STB PVR product is shown below:

Common Areas to Focus Early during Stabilisation Phase of STB PVR

STB SI Team Bandwidth is Limited

The idea behind the one roof / hit squad is to get everyone in the same room, forming one team, focused and dedicated to closing down issues. Typically you have vendors supplying different components of the software stack, with their own processes & priorities - to expedite issues much faster, removing delays that come from communications, ping-pong of emails and phone calls - it's best to get the engineers on-site and deal with the issues face-to-face. This sounds like a no-brainer to those from the Agile or XP camp - but in reality some companies can be pretty rigid in their support service agreements. Hence it is critical this is driven quite hard by the customer, and pains should be taken to agree on the concept that projects will call for people (resources) to be available on-site during Integration/Stabilisation or even final Acceptance Testing.

Advantages of ORITs

• ORITS definitely add focus and bring a sense of urgency to the project, across the board. It gets the attention from senior management, and commitment from teams to resolving the issues. In essence, we have the full participation of the vendors.
• Communication-paths are reduced significantly by being under one-roof or in the same room. This is often time consuming, and generally is impacted by timezones and vendors being geographically dispersed.
• Ensures the right level of support is provided - i.e. all required technical experts are available to support, and easily accessible. You have their full attention, no distractions.
• Improved turnaround time for fixing issues - The ability to fix code on-site, in real time, providing engineering builds adds tremendous value and cuts a lot of the red tape associated with vendors build/release processes.
• Customer is kept happy - all stakeholders have confidence that there is focus, attention & drive. Having a dedicated owner for each area helps build confidence, maintains pace and momentum. Of course, interventions can be applied much sooner than later - no delay in management decisions.

Some Challenges with ORITs

• Almost certain to disrupt existing teams and structures. People will have to be seconded to other teams, virtual - people will generally be interacting with each other for the fist time, which means that these teams might not initially gel well.
• Requires focus, participation and attention from vendors. Generally vendors are hard pressed to support multiple project deliveries, taking out a key engineer and dedicated to one customer for several weeks, stresses the vendors other commitments.
• Engineers allocated to ORITs need to be sufficiently skilled and competent. We are looking not only for senior, hard-core technical experts but people who can deal with stress and pressures from senior management, as well as communication & reporting must be excellent. Very hard to find such calibre people.
• Logistics challenges: The premises for the one roof sessions must support the physical & personal needs of engineers (Equipment setup, kitchen, personal workspace, etc).

Concept of an ORIT Owner is Imperative

For ORITs to return value and actually improve the outcome of the project, you have to assign a strong technical leader overall to manage the various ORITs holistically, as well as strong technical leads per ORIT area, a.k.a. Owners.

• An Owner is essentially responsible & accountable for ensuring all critical issues for a particular area (e.g. Recordings) are resolved to closure, proven on-site, and tracked through completion in a formal release, all timely.
• This means not only investigating & triaging, but also working closely with - and driving the component vendor engineers & teams for fixes or patches including committing to target fix dates for final implementation. If progress is not going as expected, escalate to senior management as appropriate.
• Requires sound technical knowledge, domain experience & appreciation for problems, ideally having worked or experienced previous issues on past projects of similar nature.
• An owner has freedom to prioritise issues depending on severity without having to wait for approval from Product Owner
• An owner also takes responsibility for leading the team assigned, dealing with people as well as performance related issues

Does it work?

Like all other topics I've shared in my PM Toolbox, I wouldn't be sharing stuff that is fluffy and woolly - not implemented or experienced first-hand by me, directly. So, yes - ORITs do work, if managed correctly. It is no easy feat, requires mindsets & personalities that can maintain a steady pace of rigour, fortitude and relentless determination to get to the root cause of problems - seeking out the best practical solutions in the time allowing. I have both managed ORIT teams myself, observed other projects from afar, and been an engineer on-the-ground as Hit Squad member. It can work well, or can be a complete failure. There needs to be a unifying voice from the top, harmony across the entire project team, and complete focus and attention... It is not an easy or straightforward intervention...

A little more detail behind the mechanics

In the attached slide pack, I try to summarise the key points of this process, including notes on the ideal team sizes, reporting expectations including templates.