When Agile goes wrong...

When Agile Software Development goes wrong?? Really, Agile/Scrum is the best thing that's happened to the softwared development fraternity, how can it ever go wrong?? It's the latest trend (not exactly, it was the latest straight after XP eXtreme Programming, but the current latest trend is a mixture of Agile and Lean development or Kanban, so much so that some people are calling it "Scrumban").

Let me point out that I am a huge promoter of Agile/Scrum, have seen it in action from small project teams (less than 12) to large-scale projects with 200+ people distributed across the world - Agile/Scrum does work, it's basically common-sense pragmatic practices, with the aim of getting the product development right early on, involving early requirements feedback with regular customer interaction, continuous and incremental testing, etc, etc.  You can find a lot of reading material online or many books published on the topic.

I feel that some people are too quick to jump on the Agile/Scrum bandwagon, after reading a book or going on a quick training course covering the essentials of the methodology.  This is where the danger comes in - when newbies fresh into the concept after being exposed to the classic style of very well-structured, co-ordinated and controlled project management practise of software development -- are keen on dropping all concepts done under what is now generally labelled as classic "Waterfall" development, and whenever challenged with understanding or improving the project processes, people are just too eager to show the "Waterfall" trump card: "That's Waterfall, we're Agile now, we don't do it that way..." - and basically this drives me nuts sometimes because people don't fully appreciate what these terms mean, quick to dispense with the "Agile" vs "Waterfall" cards, especially when these remarks come from people who really cannot call themselves seasoned experts, or seasoned developers that have consistently released real-world products into the market, not only experiencing the development phase of the product life-cycle, but also the ongoing maintenance, support and upgrading efforts that result from a successful product launch...

Let me come back to topic. So how does Agile go wrong??

Firstly, as stated above - newbies start the process, without much thought or planning, or without grasping the inner core of what Agile/Scrum really means.  They basically fall in love with the Agile manifesto, and because of being heavily managed on previous projects, adopt a mindset that is close to the euphoric 1960s where everyone is basically free to do what they want, respect and love the team, allow the team to self-organise and self manage...

Lets look at the Agile Manifesto again, as I'm going to touch each point in context of my real world experience, especially with Set-Top-Box software development:

Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation

Responding to change over following a plan

What else can cause Agile to go wrong?? 

Essentially this boils down to being in love with team freedom and dynamics -- and forgetting that even with Scrum, there is some process overhead.  The fundamental driving force behind driving a project with Scrum, is the Product Backlog.  Not only do you need to have a Product Backlog, you need to have prioritized Backlog, owned by a Product Owner, executed by the development team through the guidance of a Scrum Master (usually this is someone who's come from leading a development team as team leader, or an old-school project manager looking at managing software projects with a fresh eye).  The backlog is not a list of requirements, the backlog is a list of work packages, use cases or stories that describe a piece, area or feature of the product that can be done in a short amount of time (i.e. a Sprint) that can be developed, tested and demonstrated at the end of the Sprint.  The backlog is not a ToDo list of atomic requirements for burning down...

Any intro text on Agile, as in Agile Project Management with Scrum (Microsoft Professional) expresses the importance of maintaining an up-to-date, prioritized backlog, owned by the product owner who is supported by the Scrum Master - a missing backlog is a fundamental flaw in a team's adoption of Scrum/Agile...

What else can cause Agile to go wrong??

So we covered over indulgence of the manifesto; missing product backlog for Product and Sprint planning.  The next thing that goes wrong is not enough pre-planning the Spints: What are we doing now, and what do we plan to do next??  Despite the flexibility of Agile, sprints must execute fluidly, systematically, almost like clockwork. The sprint process should be almost natural to all on the project team.  Poor planning up-front in identifying the possible stories to focus on the next sprint, doing the due diligence to ensure that for the upcoming work the team will be as free from impediments as possible.  Planning is not just choosing a bunch of requirements that people like to do, it's about sitting with the Product Owner and agreeing what is desirable for the next Sprint.  Poor planning and not identifying the work packages before the sprint, that is, not doing pre-preparation is likely to result in blockages, waste and inefficiencies during the sprint, resulting in less than optimal user story completion, injecting obscure data into your metrics gathering for predicting throughput/velocity of the team...

What else can cause Agile to go wrong??

The size of the teams.  Put 30 developers & testers in a team, and controlling that is going to be challenging, especially when a)No real structure to team; b)No backlog; c)No Pre-planning.  It gets worse when the team is further segmented into different teams, Team A, Team B, Team C - all get to choose which "requirements" to work on for the next sprint. All being managed by one or two Scrum Masters -- this brings with it communication challenges obviously, but also poor team dynamics. Instead of having people with cross-functional knowledge of the stack, instead we get people specialising in certain areas, with a natural tendency of reluctance or insecurity to work in a different area of the product.

Is that all I've got?? No - there is one more thing that can trip you up: using the wrong metric to measure performance of the team.  With the absence of a prioritized backlog of user stories, some projects use Requirements Implemented as a measure of progress - so much so, that when reporting progress, we report on X/Y requirements done, not what is more important that being features completed.  Some projects even predict the number of people requirement, and use "One requirement per developer per day" and base their argument that over time, this value averages out and can be trusted.  Thus, number of requirements completed is measured for team output, and with a fragmented team, this measurement is per team, not per project.

I am of course talking from experience. In fact, I am experienced this Agile mentality in my current job, thankfully with the help and assistance from another scrum master, between the two of us, we have slowly been instigating changes, turning things around such that we now have a Prioritised backlog, we now focussing on use cases and stories, we're now measuring throughput in terms of features, and we're now doing some pre-planning. It's not perfect but it is a start, hopefully in the right direction....it's not yet smooth sailing though, because management have pulled the rug from right under our feet by changing focus to implementing a brand new UI. Good thing we're doing agile and we can deal with uncertainty....BUT....

BUT Agile is not easily applied to Set-Top-Box development projects, especially when the aim is to launch a new generation EPG, Middleware and new hardware.  Changing requirements almost two years into the project whilst still expect the team to still deliver to original time-scales is woefully optimistic to say the lease, but management have also misunderstood Agile, and flaunt the term willy nilly, assuming that the team is flexible and can cope with change!

Why is a STB project a difficult Agile Project when you've fallen in love with the Agile Manifesto?

A STB project involving new hardware, new middleware and new EPG is NOT like writing a web site or a smartphone application.  You don't have the luxury of setting up test websites, beta sites to ask for feedback, neither can you release and retract a release because everything is on the server for such web apps. STB projects are intensive, expensive, involve a lot of third party vendors, and thus realistically implementing core Agile principles from the outset is going to be difficult. I say difficult, but not entirely impossible -- however, it comes with a lot of management and administration overhead to indeed run a true Agile STB project.

Of course, assuming one has all the components like Hardware, Drivers, Operating System, Middleware to a mature state of completion, then you can definitely run an EPG development project using Agile/Scrum, assuming of course, you have a clear picture of the end goal. A clear picture of the end goal, doesn't that defeat the purpose of using Agile/Scrum then?? Yes, in some ways - but remember, we're aiming for time to market here - if we keep changing our minds, launching a product is not going to happen to plan, especially when the underlying implementation might need to be extended, changed or remodelled to fit the changing requirements of the EPG application.  I will expand on EPG design in a future post...

If the EPG & Middleware are under development, then you cannot escape notion of pre-planning: work out what the dependencies are in advance of each sprint. Ensure that Middleware development leads EPG development, ensure there is a path to the teams for communicating and resolving issues, etc...

If the Graphics Design, EPG & Middleware are under development, then you add another layer of sprinting for the graphics team, such that the graphics team leads firstly the middleware, then EPG.

If you're in the business of documenting every requirement for each layer of the stack, then you'll have to either specify the requirements up front, but because the graphics design and user experience is open to interpretation, requirements cannot truly be closed or published, unless the team have been through an iteration or two of proving and testing.

Lets look at each point in the Agile Manifesto

Individuals and interactions over processes and tools

Yes, this is all well and good when the team is small, but when you have a team that is more than ten, say thirty individuals who are a mixture of full-time and contractors from India, not local talent - then you will need to have processes for communication, planning and tools for enabling collaboration.  EPG development does not end at the point when development is complete, there is a strenuous batch of user, hardware and security acceptance testing that must take place to get to launch candidate. Changes that come into the product needs to be controlled and managed, hence there needs to be a change request process in place. You cannot just have a whiteboard with post-it notes posted to keep track of development, you need an audit history to refer back to when it comes to maintaining and enhancing the product in future.  Within the local development team however, to get the job done within the sprint, then the team is free to do what they want or need to do to get the job done and show something visible at the end of the Sprint.

One also has to be careful with the amount of freedom given to the team. A self-organising, self managing team does not happen overnight. Of course, management must provide enough autonomy and leeway to make this happen, but if the individuals within the team do not have the talent or competencies to self-organise, take initiative on their own without having to follow direction, that they understand the big picture and are in-tune with the overall goal of the company and project -- then you will find it difficult to have a self-functioning team.  Instead they will look for direction from a senior person, be it the team lead or architect.

On the subject of architecture, the architecture and design patterns that underpin the software stack must have a central owner, the one responsible for the technical integrity and coherency of the architecture.  Yes, the architect must accept enhancements and open to criticism, but the architecture must be documented in sufficient detail to allow people to make a valuable contribution.

A design review process must be in place for the EPG - at the start of any new work, the concepts must be dicussed and a design review held. Important architectural changes must be documented as this is fundamental to the future maintenance of the EPG.

The make-up of the team is important - Contractors are there to just get the job done. Don't expect them to gel well, form a self-organising team and help the scrum masters in getting the backlog burned down on their own initiative. The more work contractors have to do, the longer they stay in a job.

I don't think all of the old-school of development is dead, that's why Fred Brook's The Mythical Man-Month: Essays on Software Engineering, Anniversary Edition (2nd Edition)

, especially the essay on The Surgical Team and other parts of the book dealing with people issues is still very much relevant today.

Working software over comprehensive documentation

Yes, we all want that. An EPG needs to be sufficiently documented not only in terms of architecture and design, but also on user manuals that enable independent third-party testing.  Comprehensive where necessary in terms of describing the requirements from third-party component vendors, less detail when the team is local and in control of the development. At the end of the day, a working product is needed as we can all agree, it makes common sense - but for the maintenance and enhancement teams down the road, sufficient documentation is required to enable the longevity of the product.  It becomes especially important if the development team consists of a bunch of contractors not loyal to the company or product.

Customer collaboration over contract negotiation 

If you own the EPG in-house, i.e. you're the customer - then of course, collaboration must happen regularly.  But even in places where this is the case, the company is fragmented into different departments, each with their own expectations of the end product. Sometimes, as happened in my project, a year and half down the line, people took note of the product's deficiencies to make a U-turn and almost start-over again.  Regular feedback must take place.  However, in the case of a third-party supplier, then contracts must be negotiated as part of any good project, it is project management best practise after-all. You have to know what you're signing into or signing up for, and take necessary measures to ensure your IP is protected, and you make a nice profit in the end.  In this day an age though, and in the market of Digital TV, the players are small, some are even part of the same parent company - so naturally contract negotiation takes a back seat...

Responding to change over following a plan

There must always be a plan, not the traditional Gantt chart project management plan of tasks and dates and deadlines, tracking every minor slippage etc, etc. There must be a plan in principle with key milestones in place, with checks and balances - ensuring the development parties are fully aware of the constraints being placed upon them - and for them to work with this certainty as a hard-deadline, and dealing with the uncertainties within that - so deal with change, but be cognisant of the fact you are delivering code in the end.

At some point in the business of EPG development, change must stop - there is so many user experiences one can think of - if you continue to respond to such changes, you will not ship your product.  Your plan must cope for some change of course....