Managing Large-Scale Projects using Agile, Part 1 - What is a LSSDP anyway?

What is a LSSDP anyway?

In my previous post I introduced the subject of LSSDP: Large-Scale Software Development Projects.
In this post I'll try to describe what I've come to understand as a Large-Scale Software Development Project (LSSDP) based on my real-world experience of successfully delivering such projects.  There are many buzzwords flying around, recapping my own attempt to capture headlines on my CV:  Experience with large-scale, geographically globally distributed-development projects spanning 200+ people, using Agile principles, multi-project

• Large-Scale Software Development Projects
• Global Software Development Projects
• Multisite Software Development Projects
• Globally Distributed Software Projects
• Geographically Distributed Software Projects

What do I mean by Large-Scale?

I consider a project as being "Large-Scale" where the project in question has the following ingredients:

• Significant financial investment in the project spanning tens of millions of dollars
• Technology  being produced by the engineering team is of a complex nature
• The engineering team (developers/testers/integrators), product and project management teams span in excess of 200 people
• Exceedingly high Expectations placed on the project team to ensure a successful outcome
• Strong implementation of processes & methodologies to co-ordinate and steer the project along
• Globally Distributed teams (where co-location impossible)
• Commercial/Proprietary

What do I mean by Globally Distributed?

Let's start with defining "Distributed" by using its opposite, which in software terms, we usually use "Centralised" - single point of control. This isn't strictly applicable here, we are instead focused on the physical locality of the the project's team members, i.e. the Locality of the team.  Locality in my opinion, has many variations, at the highest level, we sometimes say a team is co-located when all the team members are physically present in the same region/area (e.g. Office park, Campus R&D building, third-floor, west-wing) - but that is quite specific. Some might say that a team is co-located if they're all present in the same office park, for instance, your company might have a campus with many buildings on-site, just being on campus can be interpreted as locality local, co-located.  But I prefer to depart from this view and rather choose to say that a distributed team is one where the team are not all housed together and are separated by one or more arbitrary barriers.

How deep does one go? If my team are not all in the same building, on the same floor, then my team is distributed: Same building, different floor. Different buildings same campus park. Same country different province. Same province different campuses.   Get my drift?  As soon as there's some barrier that's going to impact communications, co-ordination, team building and coherency -- we then have a distributed team.

We have a Globally Distributed Project where the team consists of participants located from other geographic areas: Same continent, different countries. Different Continents, Different Countries - i.e. a Geographically Dispersed Team.

A Quick Note on Open Source Projects

Before anyone jumps up and down and wax lyrical on how the big-name open source projects are all large-scale and distributed and that we should learn from this community, etc. etc -- I accept, respect and admire the open source movement myself; but as we all know, the corporate world is an altogether different machine. Distributed projects can be run along the same principles as the open source movement, in the context of the corporate shell.  Delivery large-scale, commercial programmes come with a lot of risk to the business (which is not applicable to open source projects), and as such these projects have to be run a little differently.... All I would like to do is share my exposure in running such projects, not really advocating one methodology over another (although when you dig deeper into how decisions are made in open source projects, there is usually a single entity, some call it the BDFL that eventually calls the shots, much like a commercial programme!)...

Case Study: Developing Set-Top-Box Software, Large-Scale, Globally Distributed Project

The remaining posts are all based from a past project, which I can can truly say was one of the best project experiences in my career-to-date, that allowed me to grow in more ways than I can imagine.  It is no secret that NDS is on a mission to becoming the world's leading Middleware provider with its MediaHighway Portfolio.  Like all successful software product companies, there is always the challenge of supporting your existing customers using legacy technology, and trying to work on cutting-edge innovative new products & features.  NDS was no exception, over the 10-25 years of providing Middleware to many different customers, satisfying the various needs and demands of these customers, the products started to fragment and reach their maturity. With development centres situated across the world (US, UK, France, Israel, India, Korea) each supporting a slice of the product (Basic, Evolution, Advanced), each centre wishing to innovate and move the product forwards in their own way. To cut a long story short, the company decided to use the vast knowledge-base and experience from its global technical team to produce the next generation, advanced Middleware platform, which was known at one stage to the outside world as "MediaHighway Advanced", but this often changes whichever way the marketing wind is blowing. Internally we  (and for a short while, the press) referred to this product as MediaHighway Fusion, I'll stick to "Fusion" for the remaining posts.

Fusion was not just about new technology and new architecture. Yes, it was borne from learning the hard lessons from its' predecessors (Core4, Evolution, Pantalk, XTV), taking the best design principles, merging with modern Linux & general operating system architectures, changing some models up-side-down, for example the concept of CDI (Common-Driver-Interface) that departs from the classic HAL (Hardware Abstraction Layer)-STB manufacturer tight coupling....(future post will discuss STB architecture)....Fusion was more about improving response times to deliver new features to market: Agility, Flexibility and Continuous Delivery. Faster time to market, can we deliver new features in three week sprints, i.e. sustain product increments every three weeks??

So what made Fusion a Large-Scale Project anyway??  The core Fusion architecture and principles were conceived by a small team of technical experts. The first major milestone was the production of the CDI specification, which is a 1500+ pages of technical specifications aimed at Chipset Vendors to comply with the underlying broad platform infrastructure, the low-level interface definitions. That was the ultimate starting point as it laid the foundation for services supplied to the upper layers of the software stack.  The Middleware architecture was component based, consisting in excess of 100 components.  The components were to be written and owned by development teams, located throughout the world: Two sites in UK, France, Israel & India (Note that language and cultural dynamics!). The CDI team located in Korea.  The architecture was very complex, and was owned by a team of 20-30 architects, with a team of 3 chief architects controlling the overall decision making process. We referred to those 3 elite techno gurus as the Trinity. So IMHO, we have all the ingredients for a LSSDP...In fact, I think the method of Agile implemented, although not purist in its form, was very disciplined that proved itself well in managing this LSSDP project...

Even in the R&D phase, this internal product development project was considered large-scale: ticking boxes of complex architecture and geographically dispersed teams.  On landing the first customer win, Fusion switched focus to becoming both a development and delivery project (the Middleware was not complete, but we had to submit to project urgency). The customer (BSkyB) being quite bullish and demanding in its launch targets, resulted in an overdrive focus.  The Project teams grew and expanded, new component teams were created, methodologies were introduced to control multiregion planning, management, development and integration. In a relatively short period of time the core development team scaled from 50 to 250 people. Two years later, there would be between 300-700 people all engaged on project deliveries (for major customers) all on the Fusion platform (that was still maturing).

In the next post I'll go deeper into the organizational structures - but for now, lets summarise how Fusion meets the requirements for LSSDP:
- Complex Architecture: 100+ components, new technologies, strict architecture rules
- Trinity of Chief Architects - Geographically Distributed Globally
- Group of System Architects (20-30+) - Geographically Distributed Globally
- Development & Integration Team (250+) - Geographically Distributed Globally
- Project Management Team (10+) - Geographically Distributed Globally
- Product Management Team (5+) - Geographically Distributed Globally
- Financial Investment (Tools, Infrastructure, People) - Circa $100 million or more
- Delivery Pressures - Customer Launch delivery in 2-3 years

Do you agree this is an LSSDP?? Read Part 2: Organisational Challenges of LSSDP Agile Projects...

Managing Large-Scale Projects using Agile, Part 0 - Introduction...

A few years back, I had updated my Resume (CV) and as part of the headline included the text "experience with large-scale, geographically globally distributed-development projects spanning 200+ people, using Agile principles, multi-project" and was rightly challenged by a colleague as to exactly what message those words were actually trying to convey?  Of course, CVs supposed to contain all the attention-seeking buzzwords, but it really did sound and read like a mouthful, most people would not understand what I hoped it would mean, it sounded too arty-farty and dressed up - advice was "Please remove it, it makes no sense" - and this was coming from this guy who I hold in great esteem, one of the best software system architects I know (and worked with closely); and so removed that sentence from existence...until now.

In the next series of posts I aim to describe my experiences from a very large-scale software development project, that was not only based on agile principles in general but also adopted a highly structured & rigorous approach as well. Whilst I was not part of the original group that conceived and instigated the processes & framework foundations, I joined the team to take over the mainstream project management (relieving the founding team to concentrate on product management and future customers) to help get the first major product release delivered. This was at a stage when the founding management team had already experimented with various agile strategies twelve months into the project, having made numerous mistakes and having learnt from those mistakes, were just transitioning to their third attempt, which actually ended up proving itself as the final methodology; and was used for the subsequent two years with only minor, natural evolutionary updates to accommodate new needs and demands as the project evolved through development into delivery & final release phases. However, the underlying fundamental key principles were solidly embedded and continues to work reasonably well to this day IMHO. So I ended up being an executioner of the process for a period of just over two years, and then joined the product management team in promoting the methodology for new projects, providing teaching, training & coaching to other projects as well as introducing new tools and concepts, until I figured I'd seen enough and moved back into mainstream software development.

However, in those solid 3+ years of working on that programme, I had learnt quite a bit about the theory of Agile versus appreciating and adopting much of spirit of Agile, taking into account organisational & project maturity, structures and processes, theory vs practice, resistance vs acceptance, people vs management, processes & controls vs fluidity & flexibility, not forgetting customer focus vs ideal product management (customer always comes first).  So now I'd like to communicate the guiding principles that made and continues to make the above-mentioned endeavour (could this become a model for other companies perhaps?), which the industry refers to as "Large-scale Software Development", work in the real world:  where a truly geographically dispersed, distributed multidisciplinary team of developers, architects, integrators and testers, project & programme management teams are all synchronised on a common development & delivery methodology having its roots in Agile -- are actually sustaining high profile, time-critical, multimillion dollar projects of an exceptionally high quality and complex nature, including the challenges of using just one single codebase to support multiple independent customers -- a case study in Distributed Development using Disciplined Agile Methodologies for a Digital TV Set-Top-Box Software Project.

The underlying reason behind this post however, like all my previous posts, is a focus on Digital TV projects specifically around Set-Top-Box UI EPG & Middleware development initiatives, where increasingly organisations are embarking on implementing Agile end-to-end, even on my current project which I've inherited, Agile had been chosen as the development methodology.  Don't get me wrong, I am all for agile, I respect its philosophy; as someone who loves autonomy myself, I wouldn't have it any other way.  But nevertheless, as I've written in the past, Agile can go very wrong if being implemented for the very first time.  This is why (and this is not just my opinion, see this report), Agile should not be chosen on a mainstream project that is the company's crown jewels, instead, the recommendation has generally been to start with a pilot programme that you can afford to throw away, fail and start over, expect to fail many times before getting the ingredients just right....but I digress...getting back to the purpose behind the next series of posts:

I have seen a model of Agile (Disciplined Agile Delivery) implemented successfully in the real world on a large-scale development & delivery project - and would like to share this experience with others as I think it would be useful & helpful, at the very least can be used as input material if you're considering adopting Agile in your own DTV projects.

I've broken down the concepts into several themes/parts - I'll update with links to each post as each is published:

• Part 0: Introduction: this post - Objectives
• Part 1: The Stage: What is a Large-Scale Software Development Project (LSSDP) anyway?
• Part 2: Organisational Structures for Large-Scale Software Projects
• Part 3: Implementing Large-Scale Agile/Scrum Model - Product Management Methodology
• Part 4: Implementing Large-Scale Development, Integration & Delivery Model
• Part 5: Implementing a Large-Scale Architecture Model 
• Part 6: Implementing Large-scale Continuous Integration (including Config Management)
• Part 7: Evolution of the Model - Dealing with Growth, Scaling the backlog, etc.
• Part 8: The challenges with Continuous Delivery
• Part 9: Conclusions

And finally, if you have read (or end up reading) all the parts of this topic; and would like to pursue this discussion further with me one-on-one, I am always available to discuss possible consulting opportunities - so please do get in touch!