Non Functional Requirements and Sluggishness

There is a lot of talk about whether writing code, or creating software, is really an art, a craft, or is it a highly disciplined, software engineering discipline. This post is not about this debate, although I do wish to make my stance that great software is just as much it's a craft as is a software / systems engineering discipline. One should not discount the analysis, rigour that should go into the design of any software system, be it relatively simple 4th/5th tier high level applications built on very easy frameworks (e.g. WebApps, Smartphone & Tablet frameworks), or very low-level, core, native software components (e.g. OS kernel, drivers, engine services).

Take the Set Top Box (STB) for example, a consumer device that is usually the central and focal point of the living room, people expect the thing to just work, to perform as well as any gadget in the household, being predictable, reliable and always delivering a decent user experience.

How then does one go about delivering performance in a STB software stack? Is it the job of the PayTV operator to specify in detail the NFR (Non-Functional Requirements)? Or is it the job of the software vendors (Middleware / Drivers) to have their own internal product performance specifications and benchmark their components against their competitors? Does the PayTV Operator benchmark their vendors based on the performance of their legacy decoders in the market, for example: this new product must be faster than all infield decoders by a factor of 20%. Or is their a collaboration between the technical owners (i.e. architects) to reach mutually agreed targets?

Take the problem of Sluggishness. According to good old Google,

sluggish

ˈslʌɡɪʃ/

adjective

1. slow-moving or inactive.

   "a sluggish stream"

   synonyms:

   inactive, quiet, slow, slow-moving, slack, flat, depressed, stagnant,static "the sluggish global economy"

   • lacking energy or alertness.

     "Alex woke late feeling tired and sluggish"

     synonyms:

     lethargic, listless, lacking in energy, unenergetic, lifeless, inert,inactive, slow, torpid, dull, languid, apathetic, passive, unresponsive,weary, tired, fatigued, sleepy, half asleep, drowsy, heavy-eyed,enervated, somnolent
   • slow to respond or make progress.

     "the car had been sluggish all morning"

When we're field testing STBs, over long periods of time (or sometimes after not-so-long-usage), people & customers report:

• The box got very sluggish, needed to reboot
• The STB isn't responsive to remote presses, takes 4-5 seconds to react 
• Search takes too long and I can't do anything else but wait for search to complete
• Channel change is slow compared to my other decoder
• When using the program guide or navigating through the menus, the box gets stuck for a few seconds as if it's processing some other background activity - it's very irritating to my experience

This feedback, whilst extremely useful to the product team, is very hard for engineers to characterise without having access to system logs, to trace through the events that led up to the slowdown in the performance that resulted in degraded user experience.

In my view, based on my own experience of writing STB applications and managing complex systems projects, I believe that unless we quantify in fair amount of detail what sluggishness means, then it's a cop out for passing on the buck to other parties: either the product owners didn't take time to functionally spec out the requirements, or the application is using API services that it doesn't have control of...

In the remaining part of this post, I will touch on an example of a previous project of how we handled the problem of sluggishness, and this was through a process of rigorous Non-Functional Requirements focused on Performance Specifications...

To Ringfence or not to Ringfence?

I have been running systems & software engineering projects for some years now, the topic of "ringencing" teams seems to be quite a recurring one. It often comes up during the early stages of startup/initiation, or later into the project when stakeholders get quite nervous and edgy when the project begins to show signs of being in distress (i.e. the launch or completion trajectory doesn't look so good). One of the natural instincts for such management teams is to tighten up control, for example to ringfence all people with central command control within one single delivery team.

So what does it mean to ringfence anyway?

According to Google, enter "define ringfence" gives you:

ring fence

noun

noun: ringfence

1. 1.

   a fence completely enclosing a farm or piece of land.

   • an effective or comprehensive barrier.

     "he did not say when the merger would be completed or when the ring fence would be abolished"

verb

verb: ringfence

1. 1.

   enclose (a piece of land) with a ring fence.
2. 2.

   BRITISH

   guarantee that (funds allocated for a particular purpose) will not be spent on anything else.

   "the government failed to ring-fence the money provided to schools"

I have been on both camps: first as an engineer being assigned to support a project/customer; and later as the manager driving ringfencing agreements with vendors.  As a software engineer, I didn't fully appreciate being assigned to a customer "You will work on only Customer A and nothing else. If you don't have work to do, speak to the customer project manager. We are getting paid exclusively to support this customer, so when you need to fill in your timesheets, we need to know in detail the tasks being billed for. Any item not having direct relevance or is not a direct requirement for the customer will not be billed for". In the product services development business (which is where I spent the bulk of my software engineering career in), this isn't strange at all. Teams get assigned to customer, we work at the behest of the customer, taking work requests in via the customer delivery manager. It does get a bit more challenging though, when your product on offer is actually designed in such a way to be generic with some customisation, such that you only need just one core development team to satisfy, simultaneously, the requests from multiple customers (at the same time), in which case, a ringfenced team makes less sense...but hey, we're being paid to do the work, without the customer, our product is not going to sell, so if customers wants a ringfenced team, we'll give 'em ringfenced teams ;-)

Later on in my career, I started managing software product development, as a team of product development managers, we were often faced with the requirements from customers to guarantee "resources" to their projects. So during the initial start-up, planning & contractual negotiations, we would roughly estimate the project's backlog, and the impact on resource headcount, guaranteeing the customer will have X number of heads ringfenced for the entire duration of the project. The customer would have first right of refusal if and when a situation arose that we'd need to support new customers, or share development work amongst other projects, at the risk of reusing some of the people originally ringfenced and committed.

As a team responsible for product development, the whole point of having a common core product, is that we can easily reuse the same product, with minimal customisation required to deliver on multiple customer projects. Take for example, the business of set-top-box (STB) middleware or the STB application that we sold to many a Pay-TV operator. Using a common product development team, one team would support multiple customers, via one product release. In the case of STB application, if designed correctly, the major differences lay in the look and feel, and custom business logic workflows, the engine components were essentially the same. So we, as a product development team, could with reasonable confidence, guarantee to customers A/B/C, that from a development perspective, the need for a ringfenced team is really undesirable, causing us too much pain - which I will go into later.

However, on the other side of the fence, more recently, I now sit as the customer! When donning the hat of the customer, I am just more comfortable when I have better control, ideally all control! If I'm guaranteed a ringfenced team, the "resources" are totally under my control, focused and dedicated to my delivery. After-all I am paying the vendor to deliver, and I want total control of the delivery team. The better control I have, the better chances of delivery....plain and simple, accepting the challenges being placed on the development teams. What helps is that although I can be the hard customer, I do have a pretty good idea that the development vendors can overcome their perceived challenges and difficulties with the ringfenced approach, because, I myself was in the trenches and made it work. So I often find myself explaining component-architecture and single trunk-codebase management, multiple release streams management, to the technical teams.

And really, when you pause for a moment, and think about the big picture - you just have to appreciate both sides of the argument. The customer wants more control as this will help him/her sleep better at night, the development vendors would like to be left alone, trusted to deliver their product, as they know best how to run their development streams. Trust, however, has to be earned, and I as the customer, need to have a very good relationship with the vendor, as well as the vendor must have proven itself on more than one occasion, that they can come to the party and deliver...If there's been some turbulence in the past with non-delivery, you can be sure that I, as the customer will insist on the ringfenced team (the voice of the software engineer inside me head is silenced by reality of business needs)...In the end though, Delivery always wins!!

Let us look at an illustration the essentially captures how I approach this subject of ringfenced teams:

The Trust Curve as a tool to start relations

I have recently taken up an engagement that involves a project rescue intervention as the overall integration program manager responsible to bring in various technical teams and components represented from different business units (each with their own priorities & project deliverables), that when integrated together delivers greater business value as a whole (than each component servicing its own business process in isolation) - an awesome yet complicated system. The stakeholders have set high expectations, hoping to bring in some structure and alignment to the teams by setting up a program stream that would centrally command and control the deliveries - taking on the classic Steerco approach.

So I needed to setup a formal kick-off workshop, get everyone in a room together to flesh out as much as possible (review where each independent project stream is, what's to do, architecture, design, integration backlogs, etc.) thus culminating in a unified program plan. I had one problem though: I sensed a lot of tension and nervousness between the teams, there was quite a bit of history around expectations not being met, poor communications, etc. I just did not get the feeling that everyone trusted each other, there was a lot of suspicion going on. And now with this management intervention of having a lead integration program manager, people had their reservations. For me as this integration program manager, I needed to start the stream off on a good note, establishing a level of trust (that was apparently nonexistent), and get the core team leads gelling together to form the core delivery team. I anticipated a lot of tension that could surface in the main kickoff workshop, so I decided to run a retrospective prior to the official workshop, as a way to pre-empt the emotional discussions that, if left unchecked, would eventually derail the formal planning workshop.

In doing so, I started to look at retrospective tools that could help me approach the subject looking up (Getting Value Out of Agile Retrospectives - A Toolbox of Retrospective Exercises by Linders/Goncalves and Agile Retrospectives by Derby/Larsen). I also sought advice & counsel from a good friend, colleague and agile-coach-cum-mentor of mine: Farid@Crossbolt - explained my challenges, and in ten minutes, he provided excellent suggestions around team building activities as the first prize, second prize is address the elephant-in-the-room directly through a retro, where we could talk around a simple curve, call it the "Trust Curve". I settled on the latter suggestion (my original approach) of doing the retro since we just couldn't afford any time/money for the team building activity.

I have since then shown the Trust Curve to a few people, and have received some really good feedback, and hence this blog post. I want to share this simple, yet powerful picture that can be used as a starting point in setting the stage in building new relations, in an open & mature manner:

Trust Curve

The above picture tries to show the different positions of trust, that you could use to frame the conversation. Engineers understand pictures, especially a curve - that at first glance, appears so simple, yet profoundly powerful if you allow yourself to be open to critical reflection. The picture tries to show a journey of how trust will develop through a course of a relationship.

Managing software project teams through transition: Roles & Responsibilities

In this post I talk about a recent engagement with a client around settling on a framework to understand the roles & responsibilities in the development organisation. It was an activity that lasted the tune of just over 12 months to reach a point where everyone impacted could agree with the outcomes. It was really quite a journey with many individuals across the board from senior managers to engineers. The end result was a framework, based on the RACI model, that not only showed the RACI matrix, but also output high-level job descriptions for each key role, that acted as handy input to job-specs that could be tied in with people's objectives, in terms of their personal development plan for performance appraisals management.

The context that triggers this activity is in my view, quite common for any young entity that has just started on the path to software product development (the project team effectively shapes up the future direction & structure for the organisation) - i.e. their first real stab at doing in-house product development. Usually what happens is the project grows & ramps-up as required (building various teams required to get the delivery done), the project, in effect shapes up the future structure of the organisation. I've been in this story a few times already in my career: Work on a next generation concept, build a team, deliver, then re-shape the organisation to focus on this product as the mainstream focus, transforming the project structure into an effective organisational functional model...

In this particular scenario, the situation was around a new unit that was setup to deliver their first major product deployment (Set Top Box (STB) software - completely developed locally in-house), and as part of the experience, the team operated with a fair amount of autonomy for several months (experimenting with agile/scrum) finding their feet, until the business ran out of patience sensing that if things continued as is, the promise made to the business wasn't going to materialise (delayed), and so thus intervened by "management directive", almost pulling the rug from right underneath the team. The project switched focus to a highly-driven delivery mindset, where the team freedom was curtailed to a point that all technical decisions were made through a single entity, called the "Launch Director", who's only focus is to do-what-it-takes to get the product out-to-market, running with complete freedom to cut-and-chop processes (in the guise of efficiency improvements) to get the job done.

In retrospect, this was the right call made by senior leadership - a gap was identified around the lack of a key resource accountable for aligning the technical delivery, a strong driving force was required to reign in the stray streams to get them all pulling in the right direction. Going forward however, we kept the role under "Delivery Owner" to be considered for large-scale new project initiatives.

So the team came eventually out in the end, a little scarred, battle-hardened, with the product delivered to market as an extremely huge success, which would not have been possible had the launch director intervention not been implemented.

Our challenge was how to move forward, with the launch director moving on, leaving this product development team to put the pieces together and continue...

How does the team pick up where it left off? Before the switch to delivery mode, the teams were just entering a rhythm, they knew who the players were, the key project managers, product owners, technical leads, etc. Once the launch director kicked-in, most of those roles faded into the background, people overshadowed, disempowered, meant to just follow the lead, taking direction from management. Coping with this change upset the trajectory the teams were hoping to achieve with their agile/scrum intentions (whilst those sensitive to agile principles can indeed sympathise with the team, it has been my experience that even with best intentions of managers adopting agile teams, there comes a time when a delivery mindset, driven by senior stakeholders, takes priority, delivery must happen!).

So how do you fill the void, dismantle and regroup, redistribute the roles & responsibilities to a point that can sustain the teams going forward in keeping with this mindset of agile/scrum values, and ensure that when the next major product release needs to go-to-market, that they don't fall into the same operational, escalated emergency mode of working: i.e. crisis-delivery mode?? How do you prevent the house of cards from toppling down?

There's no easy answer really, apart from having loads of conversations, being patient with the rebuilding process, a lot of co-ordinating and re-enforcing repeated messaging around respecting boundaries, etc, etc. A lot of feedback loops, discussions do help, but people expect more - eventually one has to reach a point of some certainty, by taking time out to flesh out in writing, the roles / responsibilities / expectations from all.

Suffice to say that this is really a journey through transition - I've taken this particular organisation to a point that enough of a framework is in place to provide the necessary guard rails...

Note: As a consultant I share my work experiences through my writings, whilst respecting my clients, I take time to present generic descriptions that has applicability to general software management experiences. I am by no means passing a value judgement on my client, rather sharing the outcome of the experience that could be helpful to people faced with similar challenges...

The point of this post is to share the learnings of this experience, and not undermine the specifics of the project, this isn't a rant post at all. As a matter-of-fact, the result of establishing this RACI matrix and unpacking the team's roles and responsibilities, has led to the team achieving a major milestone release, without needing the previous management intervention: Between the product, development, test, integration streams, and empowerment of the Release Manager role - this team has come together quite nicely... I've actually been asked to help facilitate this RACI process with other departments within the business unit.

P.S. If you're new to this site, I write about Set Top Box software development topics. Please search through my blog for background reading on the nature & context of digital TV software projects...

Take a chance, leap, put yourself out there

I came across this graphic that resonated with me on so many levels. It also epitomizes myself to an extent, that the poster itself makes good for another "About Me" post...