QA Manager Job Spec

As a Software & Systems Engineering Management Consultant, I spend a lot of my time helping senior management sort out their department processes, run audits around international benchmarking, teach and coach topics that I'm quite familiar with, help with crafting roles & responsibilities for project & department structures, help set up implementation strategies, and remain on board to see these recommendations & changes through to completion...

Recently, I helped a client rationalise its Quality Assurance (QA) Engineering organisation. I have written about QA/Testing in the past, most of those posts revolve around the experiences from this particular client. You can check out these two resources for my background & knowledge of the Quality Engineering QA/QC disciplines:

• All QA blog posts on this site
• My most read QA white paper 

One of the triggers that drove this client to rethink its QA structure was some work that I did last year, around benchmarking what other organisations do in similar projects, and the outcome from that investigation identified an overload in the QA space, with lots of teams doing similar activities, resulting in duplication. The ratio of QA-to-Developers seemed unnaturally high. I personally knew this from the very beginning, but had to learn the hard way of taking a company through its own journey of self-discovery, that whilst an expert might see flaws and holes on day one, the customer is usually so set in its ways-of-working, that they are blinded by the inefficiencies because the projects do deliver anyway.

Anyway, I helped identify a need for a QA Manager, and helped create a job specification around this - which is what I'd like to share with you, in case you are looking for something similar. In my view, this is still a journey, the QA manager maybe an interim step, as an improvement to embedding some quality disciplines within the organisation, my end goal however, is to lead this client to a Lean Mindset, which takes the best of Agile/Lean processes, and drives quality engineering principles & disciplines right up the value chain, to the source - which is around architecture, design & development...we're not yet ready for that yet, so a QA Manager intervention is as good-a-starting-point-as-any....

P.S. My source of info for drawing up this job spec was the IEEE SWEBOK (I once studied the IEEE CSDP although didn't want to write the exam because I'm currently against any form of certifications, although I'm still a member of IEEE & ACM) as well as my own previous job experience...

 

JOB SPECIFICATION:: QUALITY ASSURANCE MANAGER

ENGINEERING: DECODER QUALITY

 

CONTEXT

[THE COMPANY] has rationalized its various Testing roles in the department and has introduced a new role focused on driving Software Quality Assurance Practice across the business. The reporting department line (CEQA) is overall responsible for applying Software Quality Management (SQM) that applies to all perspectives of decoder software processes, products and resources. SQM defines processes, process owners, and requirements for those processes, measurements of the process and its outputs, and feedback channels.

 

SQM processes consist of many activities. Specifically, with respect to [THE COMPANY], this is broken down into the following areas:

• Software / System Quality Assurance
• Software & Systems Testing (Quality Control / QC)
• Automation Tools & Methods
• Real-world home testing (Field trials)

    

Collectively, the above areas are separate lines reporting to an overall Head: Quality. These SQM streams however are interconnected and share the underlying foundation of quality principles, which all these groups need to adhere.

This role is about a Quality Assurance Manager position, which is based on three pillars:

• Measurement & Metrics
• Standards, Reviews & Audits
• Project-based role as QA Program Manager / QA Director

SOFTWARE QUALITY ASSURANCE (SQA) VISION

SQA processes provide assurance that the software products and processes in the project life cycle conform to their specified requirements by planning, enacting, and performing a set of activities to provide adequate confidence that quality is being built into the software at all levels of the software development lifecycle (SDLC).

The role of SQA with respect to process is to ensure that planned processes are appropriate and later implemented according to plan, and that relevant measurement processes are provided to the organization.

The SQA plan defines the means that will be used to ensure that software developed for a specific product satisfies the user's requirements and is of the highest quality possible within project & business constraints.

SQA takes into account the culture of the development teams, identifies processes, standards and practices, and conventions governing the project, highlighting how they will be checked and monitored to ensure adequacy, compliance, and ultimately the decision that the product is fit-for-purpose from a QA standpoint (checks and balances).

SQA also identifies measures, statistical techniques, quality metrics including procedures for problem reporting and corrective action, techniques & methods including tools, training, reporting & relevant documentation.

SQA also contributes to defining acceptance criteria, as well as reporting and management activities which are critical to software quality.

SQA will also contain a review and audit facility that may involve, but not be limited to the following areas: Management reviews, Technical reviews, Inspections, Walk-throughs & Audits. This can only be implemented once the department has reached a level of maturity and stability of processes (i.e. standards adherence) that have proven to deliver business value, such that it becomes the charter & practice of all projects going forward, and hence warrants a review & audit activity.

WHAT ARE WE LOOKING FOR IN A QUALITY ASSURANCE MANAGER?

I was once an ESTJ

I am continuing to dig through my records of psychometric & personality test reports, this one goes back to when I was a Software Engineer of three years, I'd just lost my job through redundancy, Ireland was going through some pretty bad times, and I was faced with a lot of uncertainty: do I return back home to South Africa, find a job in Ireland, or try my hand in the UK?? 

As part of the redundancy package, included a one week intense course on Career Planning, CV/Resume writing, Video Interviews and Personality testing. At the time, this would have been my second experience with personality tests, the first was right after university, where I showed all signs of an analytical, thinking, careful engineer. In the two years since, I had left my home country, settled in a brand new world, got to meet new cultures, and see a different side of software engineering - that it changed my outlook of life and work quite a bit.

I strongly feel that a person is on a path to discovery, despite being born with certain traits, that life and experiences will change a person's outlook (although we're told that our real core tenets are what we born with, and fall back to these by instinct or default), thus I have some reservations with these personality tests, because I just don't like being placed in a box. I prefer to create my own path, and adapt to the situations as needed. However, these tests are interesting, and useful to trigger one to reflect...

So this goes back to 2002, thirteen years ago, when I did the Myers-Briggs profile questionnaire, here's the results of that time:

In 2002, I showed signs of ESTJ

So looking back almost thirteen years ago, it's interesting to note that I remained a Software Engineer for another four years, before breaking out into leadership roles as development project manager, senior project manager, and more recently senior program & technical manager.

However, I am no longer that traditional and conservative, as I'm constantly learning and adapting my processes, style and working tool-set, although I do sometimes fall-back to my own traditions that are known to work, deliver value...

I am still quite the organisational, control freak...everything has it's place. I'm so glad that it's actually a Lean discipline to keep the workspace organised, clean & efficient :-)

Culture: Working with the Israelis

I'm in the process of rounding up all my reports on my self from various psychometric tools & frameworks that I had to use in the course of my professional life, and share them openly on this blog. I did make a start some time ago with these two posts from a few years ago:

• BBC Interview Psychometric Report
• Multichoice Interview using CPP (report still to share)

This week, I experienced another framework called the "Enneagram" which I also will aim to share on this blog.

As I dug through my old records, I found one particular course that brought back some fond memories: Working with the Israelis :-) 

Working with Israelis, a One Pager Cheat Sheet

So I worked with a lot of Israelis, all levels: engineers-to-directors, and it was quite exciting and a thrill, personally & professionally. I really didn't let any personal/religious biases get in the way of working professionally, and actually developed some really good relationships that I'm happy to call them my friends...where there is code, there are no barriers....all you need is code :-)

So if you ever confronted with working with an Israeli, here is a handy table to help get you to understanding the experience:

Cheat Sheet

Cross-Cultural Awareness is becoming increasingly important in a Connected World

In the years 2003-2011, I worked with a company that was truly global. We had software teams all around the world: North America, Canada, United Kingdom, Paris, Denmark, Australia, Israel, India, Korea & China. The company went on a global training drive to address the issue of working with the various cultures. The company partnered with the folks from TMC CulturalNavigator,  an online system that we could use to learn about the people from all parts of the world, however, specifically tailored for the regions we operated in. 

Each person in the company had to complete the online questionnaire (like all these psychometric questions) to gauge the type of person you are. The results were mostly online for people to access, as it was useful information to have to hand especially when you're meeting a colleague for the first time, or have a high-profile meeting to attend and you want to get a sense of the people, etc, etc.

I applied the tools quite often, found it extremely useful and handy to have around...

In addition to the online tool, we had coaches come in and talk around this system, which is called Cultural Orientations Index - I will write about my own results from 2007 in a follow-up post, the COI contains the following dimensions:

I will expand on this in follow-up posts...

Street vendors and Talent houses

WARNING: This is probably one of those posts that just might not work, but anyway, this has been playing on my mind today that I had to write about it, so here goes!

The sell

If you're living in South Africa, or you spent some time here, you would have definitely come across our resourceful street vendors that hang around at major intersection traffic lights (robots), offering some really tempting fruits on sale. It is convenient, on-the-go, and price-competitive, reasonably safe, and most of the time, the stuff on sale looks really good. So to save yourself some time from stopping at a green grocer, you grab what's on offer, haggle a bit, pay and drive off. Seller and buyer are happy.

Sometimes though, and this is something that's becoming increasingly popular, is your street guy propping up his boxes, making it seem you're getting a good deal, the box has got the best samples on top, and tucked behind those good ones, are the not-so-good, slightly older fruit. Sometimes, the box has been pushed from the bottom to give the perception of volume, but sadly you're getting much less than you expected. In some bad cases, you get deceived by finding some really old rotting stuff (not cool)...

Hidden gems

A thought struck me today, that this is similar to the shows I've seen from software service providers, especially the ones that outsource, off-shore resources. They put on a good show, showcase their best talent at the initial meet-and-greet, CV/Resumes ripe for the taking - you sign the contract, agree on a medium-to-long-term support contract, thinking you're not only getting value-for-money, but also getting ripe fruit for the taking!

Alas, after a few months into the engagement, you realise that the good fruit was just a show, that the wider team is a box of dull tasting fruit, that you were sold more packaging than what you bargained for. Bodies are thrown at your project, giving the perception of a strong workforce, but you're unable to see any real value, delivery happens at snail-pace. Just what kind of fruit did you end up buying?? 

You're left with a sour taste in your mouth, you make a mental note not to be fooled by the alluring front of such street vendors. You will do your homework next time, ask the right questions, and continuously monitor the progress and commitment that your outsourced vendor had initially promised.

Too often, development managers are faced with just seeing the contract through, working with a mediocre team, fronted by a good tech lead if you're lucky...and kicking yourself for not taking extra time out to seek out to the rightfully established reputable greengrocer, who takes the time to source just the right quality fresh produce you really wanted, even if you have to pay a premium or wait in long queues!

Be ware of up-and-coming Talent houses. Take time to develop the relationships, and make sure you look past those lovely shining front-facing, mouth watering fruit....

As I've become my own independent consultancy, I am always mindful of overselling my expertise, mindful about putting on a show & being seen for a fraud, and take pains to delight my clients with work (and talent) that I can be proud to associate my brand with...

Non Functional Requirements and Sluggishness

There is a lot of talk about whether writing code, or creating software, is really an art, a craft, or is it a highly disciplined, software engineering discipline. This post is not about this debate, although I do wish to make my stance that great software is just as much it's a craft as is a software / systems engineering discipline. One should not discount the analysis, rigour that should go into the design of any software system, be it relatively simple 4th/5th tier high level applications built on very easy frameworks (e.g. WebApps, Smartphone & Tablet frameworks), or very low-level, core, native software components (e.g. OS kernel, drivers, engine services).

Take the Set Top Box (STB) for example, a consumer device that is usually the central and focal point of the living room, people expect the thing to just work, to perform as well as any gadget in the household, being predictable, reliable and always delivering a decent user experience.

How then does one go about delivering performance in a STB software stack? Is it the job of the PayTV operator to specify in detail the NFR (Non-Functional Requirements)? Or is it the job of the software vendors (Middleware / Drivers) to have their own internal product performance specifications and benchmark their components against their competitors? Does the PayTV Operator benchmark their vendors based on the performance of their legacy decoders in the market, for example: this new product must be faster than all infield decoders by a factor of 20%. Or is their a collaboration between the technical owners (i.e. architects) to reach mutually agreed targets?

Take the problem of Sluggishness. According to good old Google,

sluggish

ˈslʌɡɪʃ/

adjective

1. slow-moving or inactive.

   "a sluggish stream"

   synonyms:

   inactive, quiet, slow, slow-moving, slack, flat, depressed, stagnant,static "the sluggish global economy"

   • lacking energy or alertness.

     "Alex woke late feeling tired and sluggish"

     synonyms:

     lethargic, listless, lacking in energy, unenergetic, lifeless, inert,inactive, slow, torpid, dull, languid, apathetic, passive, unresponsive,weary, tired, fatigued, sleepy, half asleep, drowsy, heavy-eyed,enervated, somnolent
   • slow to respond or make progress.

     "the car had been sluggish all morning"

When we're field testing STBs, over long periods of time (or sometimes after not-so-long-usage), people & customers report:

• The box got very sluggish, needed to reboot
• The STB isn't responsive to remote presses, takes 4-5 seconds to react 
• Search takes too long and I can't do anything else but wait for search to complete
• Channel change is slow compared to my other decoder
• When using the program guide or navigating through the menus, the box gets stuck for a few seconds as if it's processing some other background activity - it's very irritating to my experience

This feedback, whilst extremely useful to the product team, is very hard for engineers to characterise without having access to system logs, to trace through the events that led up to the slowdown in the performance that resulted in degraded user experience.

In my view, based on my own experience of writing STB applications and managing complex systems projects, I believe that unless we quantify in fair amount of detail what sluggishness means, then it's a cop out for passing on the buck to other parties: either the product owners didn't take time to functionally spec out the requirements, or the application is using API services that it doesn't have control of...

In the remaining part of this post, I will touch on an example of a previous project of how we handled the problem of sluggishness, and this was through a process of rigorous Non-Functional Requirements focused on Performance Specifications...

To Ringfence or not to Ringfence?

I have been running systems & software engineering projects for some years now, the topic of "ringencing" teams seems to be quite a recurring one. It often comes up during the early stages of startup/initiation, or later into the project when stakeholders get quite nervous and edgy when the project begins to show signs of being in distress (i.e. the launch or completion trajectory doesn't look so good). One of the natural instincts for such management teams is to tighten up control, for example to ringfence all people with central command control within one single delivery team.

So what does it mean to ringfence anyway?

According to Google, enter "define ringfence" gives you:

ring fence

noun

noun: ringfence

1. 1.

   a fence completely enclosing a farm or piece of land.

   • an effective or comprehensive barrier.

     "he did not say when the merger would be completed or when the ring fence would be abolished"

verb

verb: ringfence

1. 1.

   enclose (a piece of land) with a ring fence.
2. 2.

   BRITISH

   guarantee that (funds allocated for a particular purpose) will not be spent on anything else.

   "the government failed to ring-fence the money provided to schools"

I have been on both camps: first as an engineer being assigned to support a project/customer; and later as the manager driving ringfencing agreements with vendors.  As a software engineer, I didn't fully appreciate being assigned to a customer "You will work on only Customer A and nothing else. If you don't have work to do, speak to the customer project manager. We are getting paid exclusively to support this customer, so when you need to fill in your timesheets, we need to know in detail the tasks being billed for. Any item not having direct relevance or is not a direct requirement for the customer will not be billed for". In the product services development business (which is where I spent the bulk of my software engineering career in), this isn't strange at all. Teams get assigned to customer, we work at the behest of the customer, taking work requests in via the customer delivery manager. It does get a bit more challenging though, when your product on offer is actually designed in such a way to be generic with some customisation, such that you only need just one core development team to satisfy, simultaneously, the requests from multiple customers (at the same time), in which case, a ringfenced team makes less sense...but hey, we're being paid to do the work, without the customer, our product is not going to sell, so if customers wants a ringfenced team, we'll give 'em ringfenced teams ;-)

Later on in my career, I started managing software product development, as a team of product development managers, we were often faced with the requirements from customers to guarantee "resources" to their projects. So during the initial start-up, planning & contractual negotiations, we would roughly estimate the project's backlog, and the impact on resource headcount, guaranteeing the customer will have X number of heads ringfenced for the entire duration of the project. The customer would have first right of refusal if and when a situation arose that we'd need to support new customers, or share development work amongst other projects, at the risk of reusing some of the people originally ringfenced and committed.

As a team responsible for product development, the whole point of having a common core product, is that we can easily reuse the same product, with minimal customisation required to deliver on multiple customer projects. Take for example, the business of set-top-box (STB) middleware or the STB application that we sold to many a Pay-TV operator. Using a common product development team, one team would support multiple customers, via one product release. In the case of STB application, if designed correctly, the major differences lay in the look and feel, and custom business logic workflows, the engine components were essentially the same. So we, as a product development team, could with reasonable confidence, guarantee to customers A/B/C, that from a development perspective, the need for a ringfenced team is really undesirable, causing us too much pain - which I will go into later.

However, on the other side of the fence, more recently, I now sit as the customer! When donning the hat of the customer, I am just more comfortable when I have better control, ideally all control! If I'm guaranteed a ringfenced team, the "resources" are totally under my control, focused and dedicated to my delivery. After-all I am paying the vendor to deliver, and I want total control of the delivery team. The better control I have, the better chances of delivery....plain and simple, accepting the challenges being placed on the development teams. What helps is that although I can be the hard customer, I do have a pretty good idea that the development vendors can overcome their perceived challenges and difficulties with the ringfenced approach, because, I myself was in the trenches and made it work. So I often find myself explaining component-architecture and single trunk-codebase management, multiple release streams management, to the technical teams.

And really, when you pause for a moment, and think about the big picture - you just have to appreciate both sides of the argument. The customer wants more control as this will help him/her sleep better at night, the development vendors would like to be left alone, trusted to deliver their product, as they know best how to run their development streams. Trust, however, has to be earned, and I as the customer, need to have a very good relationship with the vendor, as well as the vendor must have proven itself on more than one occasion, that they can come to the party and deliver...If there's been some turbulence in the past with non-delivery, you can be sure that I, as the customer will insist on the ringfenced team (the voice of the software engineer inside me head is silenced by reality of business needs)...In the end though, Delivery always wins!!

Let us look at an illustration the essentially captures how I approach this subject of ringfenced teams: