BBC Psychometrics

Continuing with my psychometrics journey in self-reflection...

In 2010 I put myself on the job market in the UK before deciding to leave the life I built up there from ten years, and just return back home to South Africa. So I took a stab at applying to the BBC. At the time, the BBC was moving much of its organisation from London to the north Manchester, Salford Quays - they had hundreds of positions available. I actually sat in one session going through five different positions. 

I know now, that's not a good way to go about the job search, it had been a long time since I'd interviewed, I came off the biggest project delivery, and was confident in my chameleon-like abilities of taking on multi-disciplinary roles.

Anyway, the BBC had at the time, a long and intense filtering process, starting with email and telephonic conversations, a formal write-up to respond to scenario & experience-project-based questions, and an online assessment exercise, that lasts between two-five hours, with all sorts of profiling: pattern recognition, maths, scenario-based questions, video simulation, the works. Once you past that hurdle, your results are reviewed, and then if you're lucky, you get called in for the interview.

I tried to follow-up with BBC on their specific tools-set they used for their profiling. This was their response at the time:

As requested, please find attached feedback following your application for a role at BBC North. This has been generated automatically based on how you have responded to the questions as part of the online assessment. The assessment and feedback has been created using software developed by a company called Previsor (www.previsor.co.uk), who specialise in selection tools.

A key part of how we’re recruiting for BBC North is looking at the behaviours of applicants and whether they match the ones that we feel are important in our employees. Our online assessment is therefore based on finding out about your preferred working style and how well this matches our requirements.

We hope the feedback is useful to you. 

This company, Previsor merged with SHL and are still in the business of providing organisations with talent assessment tools.

The report that was generated for me, had this to say:

This report is designed to give you information about your relative strengths and weaknesses on the competencies known to be important for success in this type of job. In addition, the report provides valuable on-the-job tips and suggestions to help you excel in the workplace.
The score that you receive describes how your responses compared against our database of responses consisting of your peers. The assessment that you have taken has been scientifically validated by up to 30 years of statistical data collection and analysis. People who score higher on the dimensions tend to perform better on the job in the key areas outlined in the report.

What annoyed me was that there wasn't any indication of the relative errors in the report, that the results are based on theories, instead it assumed a validity of correctness that I found a little unsettling, because it assumed to "know me", since some of the findings reported were totally off, as you shall see below (see Flexibility & Resilience commentary).

In summary, I did fit into BBC's style, as the report concluded:

Situational Judgement

This is a measure of the degree of alignment between one's approach to resolving work-related issues and the expectations of BBC. This is characterised by responding to work-related scenarios in a way that promotes original thinking, connecting with audiences and communities, being open to new ideas, taking responsibility for decisions and actions, and displaying a can-do attitude when faced with obstacles.

Your responses to the work scenarios demonstrated a high level of match with the judgements and behaviours looked for by the BBC. The BBC values and promotes original thinking, connecting with audiences and communities, being open to ideas and partnerships, taking responsibility, and having a can-do attitude.

I was placed in their talent pool for Senior/Technology-Manager roles.

See detailed report below...

UK versus Israel Cultural Orientations Index

I wrote earlier about working with the Israelis and a useful framework called the COI / Cultural Orientations Indicator. 

I found this interesting comparison of the two profiles for the company that I worked for at the time, UK region versus Israel region. Remember we did this in an effort to understand the teams and people a bit better, ultimately aiming to improve collaboration.

In the globally connected world of today, how can we ignore such cultural differences?? Any company having international clients & relationships, or based in multiple countries, should take time to invest in applying the COI...

UK vs Israel across the COI dimensions

My 2010 MVS & SDI (Motivational Value System & Strength Deployment Inventory)

Continuing with my journey into psychometric profiling...

In July 2010, I attended a training course with my team of fellow project & program managers, on the Strength Deployment Inventory & Motivational Value Systems, based on the work of Elias H. Porter, coursework from Personal Strengths Publishing. Checkout the reference material at the end of the post.

The essence is around improving relationships and managing conflicts:

My Results

• Strength - the SDI helps people identify their personal strengths in relating to others under two conditions: 1) when everything is going well, and 2) when they are faced with conflict.

• Deployment - means to move strategically or take a position for effective action. The SDI suggests ways that one's personal strengths may be used to improve relationships with others.

• Inventory - the SDI is not a test where judgements and "right" or "wrong" answers are graded. It is an inventory for taking stock of motivational values (the basis for how you feel and act in different situations). It is a self-discovery tool.

We were a team of nine project and program managers based in two sites in UK (Southampton & Staines), collectively we managed a portfolio in excess of £50 million pounds (close to R1 billion+ South African Rand), with our teams extending across to France (5 PMs), Israel (6 PMs), India (2) - roughly across the globe we had about 25 project & program managers involved in technical program & product management, for a Set Top Box software stack (middleware & applications), interacting with a development team in excess of 350 people worldwide, including 20+ development managers, ~30 system architects, 3+ Chipset vendors, 5+ STB device manufacturers & customers such as BSkyB, UPC, Sky Italia, Foxtel, Sky Deutchland, Yes, Get, Tata Sky...

The core product team was run from the UK, where I was based. I started off with owning the development & delivery of the product to one primary UK customer, then moved on to coordinating and managing the product release schedule for multiple customers. So I was part of the R&D Technical Product team, my customers would be the customer-facing delivery & account managers (who spoke directly to the clients).  Our UK PM team itself was split between locations, we'd meet regularly for PM forums, we weren't a fully well-formed team (hence also the course to find out about your colleagues), however we didn't really need to be (how many management teams are really self-organising and fully collaborative hey?) since we were each consumed with specific areas of responsibility within the product-space, albeit we all shared the same strong delivery mindset...

I would interact with hundreds of people across the globe, different levels of seniority, departments and domains. Often faced with multiple customers, competing project priorities, and hard-to-please-clients in terms of timelines, quality, etc. - and an engineering team (system architects, UI/UX designers, software developers, integrators & testers) scattered across the globe, challenged to maintain clear communications of priorities, direction, not to mention language and cultural challenges as well.

Working in this environment, I not only had to maintain a sense of myself (as a person, individual, professional), but also have an appreciation for the relationships I would have to foster to get the job done. Projects don't deliver because of a project plan, because of a PM constantly checking up on the status, no...it's the people that deliver projects (and mind you, I have paid my school fees in this area!).  

According to Donnie MacNicol, who ran our training (article published in April-May 09 Construction Journal titled "Colourful Relationships"):

PMs are often at the sharp end of projects, needing to deliver even when multiple technical challenges exist and relationships are under strain. It is critical that PMs develop strong and sustainable relationships to allow them to influence others. This will require them to understand: a) the impact they have on the feelings of others (b) what makes others 'tick' and how they react in certain circumstances (c) what makes themselves tick.

The SDI is a self-development tool based on Relationship Awareness Theory.... underlying assumption of SDI 'all human beings need to interact with others in a way that makes them feel good about themselves'. The SDI looks at our motivations in good times as well as when face with conflict.

It is important to understand a person's intention and motivations as this allows us to relate more effectively to them and their actions...

The SDI distinguishes our underlying motivations by introducing four main Motivational Value Systems (MVS), which describe 'how we seek to be valued by ourselves, others and in all life situations'. Our MVS means we seek to be recognised for being:

• of genuine help to others - BLUE
• focused on achieving results - RED
• self-reliant and orderly - GREEN
• part of an effective group - HUB (which is what I came out as)

Four Motivational Value Systems (I was a HUB)

 Our behaviour may vary due to circumstances and the environment, but our MVS will remain as an anchor when things are going well... Relationship awareness theory defines conflict as a reaction to a perceived threat to self-worth so, typically people are willing to go into conflict about things that are important to them. This allows you to identify their conflict 'triggers...

...we approach conflict with a predictable sequence of motivational changes and related behaviours. Initially we tend to focus on the problem, the other person and ourselves. If the conflict is not resolved, then our behaviour changes and we will tend to focus on on the problem and ourselves. If still not resolved, then at the 3rd stage the person would focus only on self-preservation. Conflict is rarely resolve at this stage.

So our training entailed getting to know ourselves, as well as our fellow team members. We each created our SDI/MVS profile, had individual and group-sessions to talk about it.

My results - What is a HUB then??

Example case study of Project Cultural Challenges

In my previous post I talked about the experience I had with learning about other cultures. I found a case study that we used as part of the role play testing, check it out below. Read it carefully, mull over it, reflect...I often work on large-scale projects interacting with a variety of people from different backgrounds and cultures, what I've come to learn (the hard way) is don't settle to your default way-of-working, always be cognisant that you're dealing with people, take pains to ensure your message is getting across in a way that will be well received (not perceived) by the intended recipients....

Case Study: Working with Cultures

My Cultural Orientations Indicator Profile from 2007

I'm continuing with my journey of self reflection by digging through my archives of Psychometric reports, this one goes back the year 2007, where I was operating in the technical space as Senior Software Engineer and moonlighting as a Technical Projects Manager as well.

At that stage of my professional development, I had experienced core software development in embedded systems as well as server-side systems programming. I'd seen the launch of three major Set Top Box projects, worked with a high-performing, self-managing team in the space of VOD / IPTV

streaming products, and was also managing a product portfolio of broadcast headend components as Project Manager.

And at the same time, I was also working on greenhouse, new innovation projects such as the Talking TV.  By then I was interacting with people at all levels of seniority, managers and engineers, and interacting with customer account & delivery managers. I had come across people & customers from various cultures, and had come to appreciate the pace / performance expectations of high-intensity-beat-the-competition-like projects.

So the company I worked with at the time, was a truly global player, and thus felt it necessary that all staff were trained and equipped with some knowledge of working across different cultures. They employed this system "Cultural Navigator" that implemented the "Cultural Orientations Index". Everyone had to register their profile on this system, answer a set of questions, resulting in a personal profile of one's preferences, and operating styles. Training was provided by TMC, where we spent a day or two learning about this assessment tool, and working through some role plays.

This was quite a valuable and powerful learning experience for me. I had used the tool quite often in preparing for interactions, be it for meetings, telecons, or face-to-face intros at new countries. Since almost every employee's profile was available online (company intranet) to compare your profile against others (even a country-by-country difference), as well as having access to to country-specific information (public holidays, type of food, gift preferences, shared interests, etc, etc.)...
Check out the video at the end of this post.

We live in a highly connected and globalised world. In the software business, it is almost impossible to NOT have a team made up of varying cultures. In this day-and-age, you cannot ignore this reality, and to plead ignorance is just not on.

Every company that has cross-country relationships, should implement the Cultural Navigator, it is extremely valuable IMHO!

So, would you like to see my full profile (downloadable) in all its glory :-) ??

QA Manager Job Spec

As a Software & Systems Engineering Management Consultant, I spend a lot of my time helping senior management sort out their department processes, run audits around international benchmarking, teach and coach topics that I'm quite familiar with, help with crafting roles & responsibilities for project & department structures, help set up implementation strategies, and remain on board to see these recommendations & changes through to completion...

Recently, I helped a client rationalise its Quality Assurance (QA) Engineering organisation. I have written about QA/Testing in the past, most of those posts revolve around the experiences from this particular client. You can check out these two resources for my background & knowledge of the Quality Engineering QA/QC disciplines:

• All QA blog posts on this site
• My most read QA white paper 

One of the triggers that drove this client to rethink its QA structure was some work that I did last year, around benchmarking what other organisations do in similar projects, and the outcome from that investigation identified an overload in the QA space, with lots of teams doing similar activities, resulting in duplication. The ratio of QA-to-Developers seemed unnaturally high. I personally knew this from the very beginning, but had to learn the hard way of taking a company through its own journey of self-discovery, that whilst an expert might see flaws and holes on day one, the customer is usually so set in its ways-of-working, that they are blinded by the inefficiencies because the projects do deliver anyway.

Anyway, I helped identify a need for a QA Manager, and helped create a job specification around this - which is what I'd like to share with you, in case you are looking for something similar. In my view, this is still a journey, the QA manager maybe an interim step, as an improvement to embedding some quality disciplines within the organisation, my end goal however, is to lead this client to a Lean Mindset, which takes the best of Agile/Lean processes, and drives quality engineering principles & disciplines right up the value chain, to the source - which is around architecture, design & development...we're not yet ready for that yet, so a QA Manager intervention is as good-a-starting-point-as-any....

P.S. My source of info for drawing up this job spec was the IEEE SWEBOK (I once studied the IEEE CSDP although didn't want to write the exam because I'm currently against any form of certifications, although I'm still a member of IEEE & ACM) as well as my own previous job experience...

 

JOB SPECIFICATION:: QUALITY ASSURANCE MANAGER

ENGINEERING: DECODER QUALITY

 

CONTEXT

[THE COMPANY] has rationalized its various Testing roles in the department and has introduced a new role focused on driving Software Quality Assurance Practice across the business. The reporting department line (CEQA) is overall responsible for applying Software Quality Management (SQM) that applies to all perspectives of decoder software processes, products and resources. SQM defines processes, process owners, and requirements for those processes, measurements of the process and its outputs, and feedback channels.

 

SQM processes consist of many activities. Specifically, with respect to [THE COMPANY], this is broken down into the following areas:

• Software / System Quality Assurance
• Software & Systems Testing (Quality Control / QC)
• Automation Tools & Methods
• Real-world home testing (Field trials)

    

Collectively, the above areas are separate lines reporting to an overall Head: Quality. These SQM streams however are interconnected and share the underlying foundation of quality principles, which all these groups need to adhere.

This role is about a Quality Assurance Manager position, which is based on three pillars:

• Measurement & Metrics
• Standards, Reviews & Audits
• Project-based role as QA Program Manager / QA Director

SOFTWARE QUALITY ASSURANCE (SQA) VISION

SQA processes provide assurance that the software products and processes in the project life cycle conform to their specified requirements by planning, enacting, and performing a set of activities to provide adequate confidence that quality is being built into the software at all levels of the software development lifecycle (SDLC).

The role of SQA with respect to process is to ensure that planned processes are appropriate and later implemented according to plan, and that relevant measurement processes are provided to the organization.

The SQA plan defines the means that will be used to ensure that software developed for a specific product satisfies the user's requirements and is of the highest quality possible within project & business constraints.

SQA takes into account the culture of the development teams, identifies processes, standards and practices, and conventions governing the project, highlighting how they will be checked and monitored to ensure adequacy, compliance, and ultimately the decision that the product is fit-for-purpose from a QA standpoint (checks and balances).

SQA also identifies measures, statistical techniques, quality metrics including procedures for problem reporting and corrective action, techniques & methods including tools, training, reporting & relevant documentation.

SQA also contributes to defining acceptance criteria, as well as reporting and management activities which are critical to software quality.

SQA will also contain a review and audit facility that may involve, but not be limited to the following areas: Management reviews, Technical reviews, Inspections, Walk-throughs & Audits. This can only be implemented once the department has reached a level of maturity and stability of processes (i.e. standards adherence) that have proven to deliver business value, such that it becomes the charter & practice of all projects going forward, and hence warrants a review & audit activity.

WHAT ARE WE LOOKING FOR IN A QUALITY ASSURANCE MANAGER?