Applying Google's Test Analytics ACC model to Generic Set Top Box Product

Early this year, I recently completed, what I felt was the best software book I've read in a really long time: How Google Tests Software by Whittaker, James A., Arbon, Jason, Carollo, Jeff ( 2012 ) (HGTS).

Although a couple of years old now, and even though the main driver James Whittaker, has left Google and gone back to Microsoft, this book is really a jewel that belongs in the annals of great Software Testing books, really.

The book is not filled with theory and academic nonsense - it speaks directly at practitioners of all types and levels involved in Software Engineering. Straight-forward, realistic breakdown of how things go down in software product teams, the challenges with fitting in Test/QA streams (actually there's no such thing as fitting in QA/Test, it should always be there, but the reality is that developers don't always get it), balancing the needs of business in terms of delivering on time against meeting the needs of the customer (will real customers use the product, is it valuable), etc, etc.

Please get your hands on this book, it will open your eyes to real-world, massively scaling software systems & teams. Honestly, it felt quite refreshing to learn Google's mindset is similar to mine, as I'd been promoting similar testing styles for much of my management career, being a firm advocate that developers should do as much testing upfront as possible, with Test/QA supplementing development by providing test tools & frameworks, as well as employing a QA Program Manager (known as Test Engineer in Google) to oversee and co-ordinate the overall test strategy for a product. I have written about QA/Testing topics in the past, the ideas there don't stray too far from the core message contained in HGTS.

The book shares a wealth of insider information on the challenges that Google faced with its product development & management as the company scaled in growth in both its use of its products as well as the number of people employed, working across continents, explosive growth and large development centres across the world. It is filled with interviews with influential Googlers that give some insight into the challenges and solutions adopted. You will also find information on the internal organisational structures that Google implements in its product development teams, along with some useful references on Open Source Tools that have been borne out of Google's own internal Testing Renaissance, now shared with the rest of the world for free, thanks Google for sharing!

One such tool is Google Test Analytics ACC (Attribute, Component, Capability) analysis model - which is the topic I want to focus on. In the Set-Top-Box projects that I run, I look to the QA Manager/Tech Leads to know the product inside-out, to develop an overall Test Strategy that outlines the ways and methods of testing that will achieve our Go-To-Market in a realistic & efficient manner (and not adopt process-for-process-sake). I generally look toward usability and risk-based test coverage, seeking out a heat map that shows me in broad strokes, the high level breakdown of the products feature set (i.e. what is it about the product that makes it so compelling to sell), what the product is made of (i.e. the building blocks of the product), and what the product does (i.e. the capabilities or core feature properties). We do this generally in Microsoft Excel spreadsheet, manually importing test data from clunky test repositories. What I look out for, as a classic Program Manager, the RAG (Red, Amber, Green) status that tells me the fifty thousand foot view, what the overall quality of this product is, and how far away we're from having a healthy launch-candidate system.

It turns out the Google do pretty much the same thing, but are way more analytical about it - they call it ACC, according to the book written in 2011/12 "ACC has passed its early adopter's phase and is being exported to other companies and enjoying the attentoin to tool developers who automate it under the "Google Test Analytics" label".

So I've created my own generic Set-Top-Box project based on Google's ACC model, aiming to share this with like minded people working on STB projects. It is not complete, but offers the basic building blocks to fully apply ACC in a real project. I think this will be a useful learning for anyone involved in QA/Testing role. 

My generic-STB-Project is currently hosted on the Google Test Analytics site, along with a complete ACC profile. It took me about four hours to do the full detail (spread over a few days of 10 minute tea breaks, between meetings), and about one hour to get the first draft of the High Level Test Plan with at least one ACC triple filled in. The book challenges you to get to a High Level Test plan in just 10 minutes, which I actually I think that is quite doable for STB projects as well!! 

In about four hours, which is about half-one working day, I was able to create a Test Matrix of 254 core high level test scenarios (note I'm really not a QA/Test expert so imagine what a full time QA/Test Engineer could knock up in one day?) that looks like this:

Capabilities & Risk Overview of a Generic Set Top Box

Google currently use this model in their high level test plans for various products, like ChromiumOS, Chrome Browser & Google+. It was developed by pulling the best processes from the various Google test teams and product owners, implementation pioneered by the author's Whittaker, Arbon & Carollo. Taken from the book. Test planning really stops at the point you know what tests you need to write (it is about the high level cases, not the detailed test scripting). Knowing what to test kicks off the detailed test case writing, and this is wherer ACC comes in (quoting snippet from Chapter 3, The Test Engineer):

ACC accomplishes this by guiding the planner through three views of a product, corresponding to 1) adjectives and adverbs that describe the product's purpose and goals, 2) nouns that identifiy the various parts and features of the product, and 3) verbs that indicate what the product actually does.

• A is for "Attribute"
• Attributes are the adjectives of the system. They are the qualities and characteristics that promote the product and distinguish it from the competition. Attributes are the reasons people would choose to use the product over a competitor.
• C is for "Component"
•  Components are the building blocks that together constitute the system in question. They are the core components and chunks of code that make the software what it is.
• C is for "Capability"
• Capabilities are the actions the system performs at the command of the user. They are the responses to input, answers to queries, and activities accomplished on behalf of the user.

Read more to find out how/what my ACC model for Generic Set-Top-Box looks like (first draft!)...

Sticky Metaphors for Software Projects: Bankable Builds, Surgical Mode

In this post I share an experience from a recent project where I influenced the senior management team to change strategy & tactics, by moving away from the standard Development->Integration->QA Team 1->QA Team n (which lasted anywhere between 2-4 weeks) approach to a more controlled Agile, Iterative/Lean approach of weekly Build & Test cycles, to a build-every-three-days, in order to make the deadline (that was fixed). We did this entirely manually, where our Agile processes were still coming of age, and had little in terms of real-world continuous integration. What I wanted to share really, is the way in which I influenced and got people to change their mindset, by using simple metaphors that were actually quite powerful, and can be used to sell similar stories:

• Bankable Builds - Reusing concepts from The Weakest Link
• Surgical Mode - Taking a page from medicine - Surgery is about precision, don't do more than what's required

It is interesting that these concepts really stuck with the entire department, so much so that it's become part of the common vocabulary - almost every project now talks about Bankable Builds & Surgical mode!

The post is presented as follows:

• Story & Background
• Bankable Build Metaphor
• Surgical Mode Metaphor
• Sample Snippets from Management Feedback
• Recommendations I set at the kick-off

Note: Like any high-profile project with intense management focus, the decision to make these changes came from senior management, which put a spanner in the works with respect to granting our "Agile" team autonomy & empowerment. Although a little resistance was felt at the lower levels, the teams nevertheless remained committed to do-what-it-takes to ship the product. The teams and vendors did share their concerns in the retrospective feedback that the one-week release cycle was quite problematic to the point of chaotic with large overheads.

The facts can't be disputed though, that our agility was indeed flawed from the get-go by not having a solid continuous build, integration & regression test environment (lacking automation) in place. It was a struggle, with a lot of manual overhead, but it was a necessary intervention, that got us to launch on time. Not having done so, would have meant us missing the target deadline, the project was too high-stakes to ignore this intervention.

Going forward however, the principles of bankable builds is still a solid one, one that is well known to teams with mature software processes, I can't emphasise the value of investing in up-front automation with continuous build, daily regression testing & automated component / system tests will help make life so much easier for software development teams.

Invest time and effort getting the foundations set up at the expense of churning out code, continuing to do so will build a huge backlog for QA down the line, teams ending up spinning-their-wheels in putting out fires that would've been avoided, had the right discipline been set in place.

Custom Voice Recoder App: Instant Recorda

OK, before I get labelled for being lazy and not searching enough on the iOS Appstore or Google Play store, I have searched but couldn't find the specific features I need from the various Voice Recorders out there. Perhaps you might have come across a suitable app, so please get in touch with me!

Basic Idea / Elevator Pitch
The idea was borne from the many commutes to work and back, listening to the car radio, or reading some big billboards that caught my attention. For me, when listening to adverts on the radio, or some plugs during an radio interview, I would try to make a mental note (especially some new websites to check out), or when checking out some billboard, try to remember the site or key terms to follow-up for later in the day or whenever I have some free time.

What happens often though, by the time my journey is over, I've forgotten most of the stuff I wanted to follow-up on! What was that URL again??

So I thought wouldn't it be nice that whenever I came across something interesting to follow-up, there would be something that I just trigger, or tag for follow-up, it will record the last audio stream of whatever it heard: If the radio was playing and I gave the signal like "Tag", then the last ten seconds would be recorded. If I saw a billboard whilst driving, then I speak it out, and say "Tag" and presto, it gets recorded. At the end of the day, I go back an review all the things I tagged (no need for me to forget anymore). You could also have some way of triggering the device to record, hit the screen or button, or something - the trigger will just instruct the device to save the audio buffer into this playlist.

Then I thought about it a bit more, realised this could even be useful in the work-place as well, during meetings or keynotes, just take stuff for following up (audio stream).So I thought there should be an app for this: Call it instant Recorda :-)

Basically the app listens in the background, constantly recording the audio stream, waits for the trigger, and saves the last ten seconds. The file is tagged by date & time, and could be stored in playlist of some sort. And the opportunities grow from there… I have discussed this idea with colleagues and friends, they seemed quite positive about it, so hopefully there is some merit in pursuing this further.

Idea is shared Public on Trello

I've started a draft backlog of the work to do, sharing in public with the rest of the world. If any app developer comes across this, please get in touch!Check it out here: https://trello.com/b/fEv5GaWt/instant-recorda-app

Architecture Interview Questions

I've written about architecture topics in the past, especially relating to the role of "Architect" in Digital TV Systems & Software projects. Though I'm no architect myself, although I could very well have become an architect if I chose to maintain on the purely technical path; I still have to be across the technical disciplines at the high level, when it comes to program or project managing large-scale technical projects. 

When I start or join a new project, one of the first things (apart from familiarizing myself with the product spec) I do is to seek out the architects, looking for any technical information in the form of design,

(document/wiki/etc) that scopes the high level system design or architecture that we're trying to implement & deliver. 

Failing that, having found no clearly defined architect role, I would go about setting up this role as a matter of urgency. No architect means no technical ownership, means flaky interfaces, chaotic system integration points, and a very bumpy project ride lays ahead!

In the past I have worked very closely with many great architects, both as an engineer as well as manager, that I've learnt from this experience enough to know what to look for when seeking out architects. If a project requires it, I do get involved with the interviewing architects, system integration managers, project managers and other roles. 

I thought I'd share a list of some of the questions I generally tend to ask when interviewing for the role of Architect in a Software/Systems project - hope you find it useful. If you read my other posts on architecture, you will get to understand I have a more holistic expectations from architects, rather than just answering this list of ~100 questions!

Agile team goals - straw man...

Over the past few years, going back since 2004, when I first started working in an agile-like fashion to present day, I've seen companies and teams transform their development practices, maintaining a strong desire to continuously improve from past experiences. I've seen such transformations driven top-down (where senior management saw inefficiencies with existing processes, accepted a step change was needed to survive growth challenges and thus set about a strategy to implement new processes). I've also experienced the change coming from within the team themselves, experimenting at first (in the background), proving the value added to not only the local team, but to the wider organisation as a whole (although it wasn't easy, it is generally harder to influence bottom-up).

In order to derive some success from these transformational initiatives, there is usually a person who takes on the role of catalyst or driver, maintaining a clear vision of the goal and desired outcome - keeping people together, constantly reinforcing the message, always constant reminding. Generally this person is energetic, evangelic and carries enough of an aura for people to just believe & follow, the mark of a true leader. This is also made possible by having a team that has been through the mill, have battle-scars to show for it, and are really sincere in adopting the new strategy.

But what do you do, when you have a relatively young development organisation, cobbled together from various parts of the world, all differing skill levels & competencies, and when the leadership is lacking (incoherent) such that no one in the department has ever delivered a project successfully using Agile/Scrum before, have never really seen the development, people & cultural transformation required to realise the change from start-finish, and not have had the patience & privilege to live the agile journey of a few years??

What do you do when a team of aspiring young engineers that are just starting out on delivering formidable, real world-class products for the first time, who have been used to the heavy-handed old school project management styles of yesteryear -- suddenly fall in love with the premise of Agile -- but unfortunately lack the depth and breadth of understanding the real challenges of software engineering (product development, not proof-of-concepts)??

In the business of Set-Top-Box software product development, this is the software that provides your TV services like Personal Video Recording, Video On Demand, Linear TV, Electronic Program Guide, etc. A mixture of hardware design, low-level drivers, middleware operating system & high level user interface. Technologies are Java, Flash, HTML5, C/C++, embedded environment. This space is becoming highly competitive, how do such software providers remain competitive??

The answer generally touted is "We're adopting Lean/Agile/Scrum - we deliver fast, work with a rapidly changing market, flexible & practical" principles for product delivery -- but in reality ain't that easy - Agile adoption is a journey of at least three years, including time for experimentation & failure - it doesn't happen overnight. Some foundations need to be in place, principles that become the substrate for the future - a team without these foundations is operating in chaos, or as they say "Paying lip service to Agile"...

In this post I provide a straw man around goals that a development team could strive for in their quest for Agile. Just as they teach you in business school, that you need to have a plan guided by a mission, vision, strategy & plan of action -- so too can a development team ground themselves around goals that are SMART (Specific, Measurable, Attainable, Realistic, Timely). Moreover, if a team can latch onto some Real, Tangible, Concrete, Credible messages, these can be quite powerful in galvanising the teams around a singular purpose:

• Straw-man goals to drive Agile Development
• Sample Software Development Team Charter