Mo Khan's Outlet!: CTO

Showing posts with label CTO. Show all posts

Friday 10 March 2023

A neat tool to bridge the gap between Product & Engineering: Combined Backlog

I came across this picture as I was browsing my consulting data archives. In one of my engagements, there was growing tension and trust issues between product and engineering teams. This is not unusual when these disciplines sit across different lines of business, or do not fall under a single leader. Often there is not enough transparency and sharing, or not enough mutual cooperation going on.

Engineering/IT generally are not great at sharing the true nature of their technical challenges, architectural issues or simply, the state of technical debt. Even if there isn't much technical debt, engineering leaders and CTOs are responsible for owning a technology roadmap, usually the platform strategy roadmap which creates enough runway for future innovation. The situation is that the work needed to service both the Technology as well as the Product roadmaps, usually needs to be done by the same group of people - "same resources" - running into capacity conflicts, that drives escalation problems. Focus too much on technical streams, you starve the business / product / customer delivery. Focus on too much product features, you run the risk of crippling the technology stack through lack of innovation, foundational architecture thus creating too much technical debt. Healthy tension should exist - but - hope is not a good strategy. More often than not, introducing a simple mechanism of co-operative planning is a good way of building trust through disciplined execution.

As way to bridge this gap, a common solution is to meet half-way, creating a single backlog that combines both Product & Technical backlog items into a single shared Combined Master Backlog. Along with this, create simple mechanisms for managing the backlog with a regular cadence, and as a team, agree on some common terminology, e.g. understanding the status of a backlog item.

Sunday 5 March 2023

How I switched from Program Manager to CTO

During the first quarter of 2017, I was wrapping up a massive digital transformation project "VOD Wars" in its second year of running and nearing completion. A program that I led from inception in the capacity of Chief Program Director (for Multichoice Group) as some classic corporates call this role, Amazon has a similar role called the Single Threaded Leader (STL).

I was responsible for co-ordinating multiple programs and workstreams that cross-cut multiple businesses. Each business had their their own heads of technology, engineering, program & project offices, operations & support functions. Each business also had their their our business goals & KPIs and in addition, had to support group-wide goals (Group-wide goals are akin to Amazon's S-Team goals). Whilst my primary stakeholders were CEOs of the various business units, I had to not only keep my eye on the high level (managing upwards and indirectly influencing across businesses that I had no positional hierarchy at all (as I was doing my own thing as an independent senior management consultant), I also had to stay close to the technical engineering, operations, support, marketing and customer experience details on the ground. The program was a great opportunity for me since I'd always wanted to experience every piece of the Video Entertainment value chain puzzle as much as possible.

Prior to VOD Wars, I'd further strengthened my exposure to all domains of the media Video Entertainment business, by being the chief program director for launching a new video streaming business "Showmax" to Africa, in 2015. This too, was a wonderful experience, launching a business start-up from zero to launch in under ten months, co-ordinating every business (legal, finance, marketing, strategy), technology (buying a new tech stack, integrating new offshore development team, building & customising product features, integrating payment vendors, 3rd party integrators, etc.) and operations (content workflows, infrastructure & customer support).

And before Showmax, I'd been leading major group-wide initiatives for advanced and internet-connected devices (I was lead end-to-end program manager for DStv Explora, for a small stint delved with the then nascent DStv Mobile), and before that, spent my time as technical program & product manager for advanced set top box middleware software, NDS Mediahighway/Videoguard platforms.

In fact, the last time I was engineering-focused, strictly speaking was in 2010 - when I'd taken up the role of Principal Engineer after inventing a Speaking TV/EPG - a role change, after being involved in project, program and product management before then. And the years 2011-2013, when I'd helped transform the consumer devices division of Multichoice, to use modern software engineering methods of planning, product development and end-to-end systems integration, in getting them to launch their first version of DStv Explora.

Being a rock-star program manager consultant in a niche industry in Africa, did come with its perks! I billed by the hour, and was mostly in control of my time. I had back then in 2014-2017, experimented with 4-day work-weeks, I took personal time off (PTO) for long periods of time (sometimes 2 months unpaid or more). It was around this time that I had started working on my RAGE model for personal development. As I dived deeper into my professional self-reflections, the following realisations about my aspirations started to really gnaw at me:

I was getting bored of being a program manager, I felt there was no challenge left and I was no learning anything new any more. I'd been reading, studying, applying and mastering the many forms of project & program management since 2008 and by 2017, I think I'd arrived and was feeling satisfied with my craft, as an expert program manager, a project leader at the top of PM hierarchy as explained here.
I felt I reached my goals of understanding how to run a full blown video entertainment business as I had experienced by then, every single aspect of business, technology & operations of the Pay-TV value chain.
I had my program management work mechanics down to an art form: I had a repeatable process, had built templates for structuring program charters, communicating progress, etc. There wasn't much more I could learn from the mechanisms needed in program management and was operating at the highest level of project leadership. A lot came naturally to me, operating on instinct most of the time.
I felt could run a Project Management Office (PMO) with my eyes closed. I'd started mentoring and coaching other project & program managers but I was not interested in specialising in PMOs.
I learnt the secrets of engaging and managing high-powered senior executives, I was confident in discussions, meetings, presentations and contractual negotiations.
I was not sure I could continue being a consultant without having skin-in-the-game, or having a seat-at-the-table.
I had failed to land other consulting engagements outside the scope of Video / Media - so my "business" AS3 (Africa Systems & Software Services) was a one-man show, tied to one big corporate without hope of branching out of video - so why remain a consultant when I could have a seat at the table if I wanted to?
As a consultant, I'd developed my own prime directives of knowing when to offer advice, opinion or put a proposal together. Consultants serve a purpose, they can lead through indirect influence but also need to remain humble and fully aware, that they don't really have any clout or say in strategic decision making. Something, if I'm completely honest with myself, I wanted to influence directly, I wanted my ideas to be heard, I wanted to be directly responsible for change, and influence strategy and change the status quo, if given the chance. I could sit on the sides and offer advice and witness slower pace of change, or get in the ring, get my hands dirty and experience true ownership, accountability and responsibility. I yearned for an opportunity to experience being a senior executive, responsible for a big organisation.
I felt I'd drifted too far from the technology domain - and needed to get back to the core. After all, I built software myself in the early days, and have degrees in Engineering and a masters in Computer Science. I wanted to get closer to the tech teams building modern apps, internet scale.
I wondered if a Program Manager could switch back to being a Technology Leader - looking at people around me in executive roles, I felt I had more than the requisite experience and technical know-how to adapt and do the job.
I needed to experience what it meant to be a manager with direct responsibility for people and bottom-line P&L. No more assisting from the sides.

With those reflections in mind, I explored new opportunities in the tech space, landing first an Interim CTO/GM (fractional CTO) role, still as a consultant for the first year which then later converted into a permanent role as Head of Technology/CTO for DStv Digital Media, Online Video Platforms, which then later became Multichoice Connected Video. I would spend the next 3.5 years intensely immersed in the role of CTO, learning new skills and soaking the experience all in: VUCA, big organisation of 150+ people, handling large budgets to the tune of R1.5 billion, growing people, making decisions with the seat at the table, navigating and surviving corporate politics and most importantly, getting my hands dirty with technology development again, turning around a distressed platform and reviving an engineering team...I decided to leave again as soon as I felt it was time for another change...however, I'd like to think I'd left the place looking better than I first found it. I'm going to share some of my experiences of this journey as and when I feel inspired to do so, like today.

Getting to apply for a CTO role was nerve-wrecking at first, even though the job advertised was a General Manager - Technology & Platforms. I felt like an imposter, full of doubt - classic imposter syndrome. I was vocally self critical so much so that I analysed and critiqued the original role guidelines advertised and provided a deep dive of my self assessment with regards to the role's expectations, benchmarking myself against what I assumed was the high bar for the role. I wanted the hiring manager to be fully aware of who I was, what my experiences were, and what my aspirations were as well before going into the interview.

Looking back, I was only able to gain this trust based on my previous work as a program manager with a reputation for getting the job done, delivering results by earning trust, that the executives entrusted me with the role.

One executive who is now the CEO of a large business, once stopped me in the hall-way and said "Mo, I don't know how you do it - but you have a remarkable way of making the complex look so simple!"

Below is the original self-assessment I shared with the hiring manager for the role. I remain truly grateful to him for the opportunity and trust afforded to me. Here is the doc with my self-assessment. I'm sharing this with you because I believe it could be useful for your own professional review. When applying for a new role, even if it's outside your immediate comfort zone, it helps to take the time to analyse, commenting and rationalising how your present or past experiences can help you delivering in the new role. You don't necessarily have to lay all your cards open to the hiring manager as I did (because I'd already built up the trust relationships over previous years, so people knew me as a program manager but they weren't necessarily aware of my tech background). The prevailing advice you'd find in most career counseling is understand your current reality (be brutally honest about your current reality whilst being positive about your future aspirations) and look for ways your past skills and experiences can speak to the gaps in the role you seek. Every job spec asks for more than what's realistic, so don't give up without at first trying. Every new role comes with an opportunity to learn and acquire new skills, why else does one seek to level up, if not to learn?

Tuesday 17 August 2021

On Cloud Transformation, CTO reflections on scaling tech & people - Part 1 - Intro

In a series of posts this year, I plan to write on how I led a transformation of a technology platform and engineering team - and delivered results in scaling to 10X+ growth on KPIs such as user-and-device growth, user engagement, enhanced personalisation & content discovery, reduced platform instability by increasing availability from 97 to 99%; created a 20X+ reduction in core operating costs (saving R100m+) and simultaneously built a scalable leadership team to take over. All in 3.5 years.

This draws on my professional work experience from March 2017 - October 2020, when I spent my time as CTO (Chief Technology Officer) of an OVP (Online Video Platform) for Africa's largest VE (Video Entertainment) provider. In this short period my team delivered end-to-end transformation (not just software development) that set up the IT/Engineering to scale for future growth. I also left a scalable leadership pipeline in place which allowed me to comfortably transition to my next role outside of video systems by leaving a technology roadmap delivery plan & sustainable processes in place for at least another 2 years. Since my departure, I remain in contact with the team who continue not only to thank me for the roadmap but also for the opportunities I helped create to grow their own careers as leaders, who are themselves on track to become CTOs & CIOs as well.

Context

The business grew increasingly concerned about their online video platform's ability to scale to increased forecasted demand anticipated as more customers switched from traditional broadcast satellite-TV viewing to on-demand, on-the-go-viewing through streaming video over the internet. With incumbents like Netflix & Amazon Prime Video and others entering the African territory, we also needed a reliable internet-ready TV product that customers have come to take for granted. Until then, the video platform which was built largely in-house by a local engineering team, had an active user base that consisted mainly of early adopters, an internal start-up project, fledgling at best.

This platform and product was still in its infancy, not-yet-ready for exponential internet growth, which could happen at any time. As such, this team operated on a shoestring budget of a constrained start-up for a number of years. The engineering team was also spread quite thin, working on multiple, incoherent projects, product and services not specifically focused on internet video. Rather, we were the "online people" that did everything from hosting websites, various content management systems, wrote media apps and ran operations for digital marketing sites across African continent. In short, I owned a digital IT shop that was a multi-armed, multi-headed hydra that needed taming. Such constraints whilst bred out of necessity, unfortunately ignored the bigger picture, long-term strategic investments needed in the platform to scale for future growth were largely ignored because of budget constraints. The platform barely supported its early adopters in so far as providing consistent availability was never guaranteed or reliable. Customer satisfaction scores were very low, below 4 (<40%). Net Promoter Scores (NPS) almost non-existent. Outages due to platform stability was the norm, with on-call load on support engineers increasing as number of users began to increase.

The business, unawares of the true state of the platform, had nonetheless planned increased marketing and awareness campaigns for its internet streaming product. Large events like the FIFA Soccer World Cup (2018), Olympics (2018) Cricket World Cup (2019), Rugby World Cup (2019) sparked much concern about the platform's ability to scale for increased load. Other events like UEFA, Premiere League, Game of Thrones and other popular video content expected to bring increased traffic to the platform. Apart from primary content drivers, the fear of not making a noise on the streaming side was high - we needed to take this fledgling product & platform and make it mainstream. Marketing increased. Along came a decent technology budget assigned to me to help turnaround & deliver a recovery. I could not pass up this opportunity to test my skills in technology, engineering, strategy, delivery and leadership...what a journey it was!

The ask: build and scale an video streaming (live broadcast and video-on-demand like Amazon Prime Video or Hulu) platform to work across 50+ countries on the African continent, with localisation. Build, stabilize, replace, buy, partner - do what is necessary, but we don't have time-to-wait a year for new R&D or migration, as we're going to make a noise in marketing, so the platform better be available. At the same time, build complementary services for "internet connected set top boxes" in addition to pure online play.

What makes OTT Streaming platforms different to DTH/DTT STB Broadcast?

In the early days of this blog, I shared in deep detail the technical and project landscape of PayTV systems for the traditional broadcast headend and set-top-box (STB) software platforms and projects as subject matter expert in this area.

In this post, I will provide a high level overview of the core differences and challenges between broadcast DTH/SetTopBox systems versus OTT platforms. OTT has become a hot topic, with the nascent rise of OTT (Over-the-Top) internet streaming platforms that many a traditional PayTV operator is now transitioning to, in light of the growing competition for eyeballs & cord-cutters driven by the likes of primarily SVOD (Subscription Video On Demand) players like Netflix, Amazon & the like. The challenge though, is that these PayTV operators tend to trip up by bringing PayTV/DTH thinking into the OTT-world, which is why I felt the need to write about this, because I have, and still am currently experiencing these challenges. Suffice to say, it becomes very tricky to disagree can call out a high level broadcast TV executive's assertions (expecting like-for-like DTH & OTT experience) as not completely accurate when it comes to OTT engineering...

The main aim of this post is thus targeted at senior executives who come from a broadcast world, and may not fully grasp the nuances of an online streaming world, specifically pointing out the spectrum of control of the end-to-end platform that PayTV operators generally enjoyed much comfort in that world, in contrast to the new world of online, where it becomes impractical to expect the level of control previously enjoyed - basically you can't control all the components of the internet, unless you invest in owning a large part of the network, which some operators have attempted & haven't really succeeded!! Simply put, the internet was not designed for Live TV streaming unlike traditional broadcast TV - the internet is somewhat chaotic & messy to say the least. In addition to this level of infrastructure control & lack thereof, an OTT platform is largely software systems designed for internet-scale (i.e. massively distributed server-side backend components) which requires a different skillset and experiences traditionally enjoyed in the closed ecosystem of broadcast & set top box device software development teams.

Another point of this post responds to the standpoint from traditional PayTV executives who may have spent a large part of their work in building broadcast systems with an engineering mindset anchored to a past era that is no longer relevant in the OTT world. Expressing opinions such as "engineering systems either work or they don't, it's either a 1 or 0, no in between states..with an expectation of 100% always-on availability..." in my view is somewhat misinformed & borders on belittling the engineering challenges faced in the online world. Whist every responsible engineer understands such demands and expectations of any system, a key point to remember is that broadcast TV has benefited from many years of refinement, standardisation, systems & processes specialised for the domain of TV - and when it comes to the internet - well, it was never really designed for video consumption, the speed of innovation has outpaced the development of standards & so is a much more messy world compared to traditional broadcast TV (satellite/terrestrial).

Disclaimer: This post is not about downplaying the broadcast engineering world, in fact, a lot of OTT operations highly depend on the upstream capabilities from PayTV operator's existing broadcast engineering infrastructure. Broadcast engineering systems are a marvel of engineering in its own respects, systems are highly complicated & superior engineering. OTT systems, and the point of my post, is to show the extension of the value chain, highlighting the new software & systems engineering challenges that present itself in the OTT world, that will be new concepts for existing broadcast engineering teams to learn & appreciate.

Key points for TL;DR folks (especially the PayTV Exec CXOs)

It's not a simple thing to compare OTT platforms wearing hat of a traditional DTH broadcast guy, it is not simply a like-for-like comparison.
Don't be naive and pass value-judgements just because you were an engineer in the broadcast TV world (Headend or Set Top Box) does not necessarily make you qualified to understand OTT engineering domain. You can't bring broadcast/STB-thinking-mindset to the world of OTT.
PayTV operators have historically enjoyed a high level of end-to-end control, not so in OTT world. The internet is outside the PayTV operator's control.
There are far more components in the value chain of OTT than broadcast (backend, frontend and device software stack is more complicated in OTT than STB software).
High level of device & platform fragmentation in OTT that a traditional PayTV operator must consider.
The internet was never designed for TV consumption & has innovated at a much faster pace than traditional broadcast TV, the internet is a messy place, broadcast TV is somewhat cleaner and more determinstic.
OTT engineering teams have different challenges compared to traditional broadcast & set top box engineering.
PayTV operators need to plan carefully the convergence (DTH & OTT) of engineering teams and be prepared for a "clash of worlds".
Set Top Box software systems enjoy far more security control & protection than its OTT sibling.
Peak events & load challenges are unique to OTT platforms compared to broadcast DTH STB (scaling challenges don't really exist in DTH/STB).
Redundancy, Fast Rollbacks, Failover & Disaster Recovery are easier to achieve in OTT than in broadcast DTH/STB systems.

Open letter to Business & Tech Leaders

Dear Business & Technology Owners...

Have you ever stopped to reflect on why is there always tension and conflict between business stakeholders and technology providers? Instead of there being a "healthy tension" between the two partners (assuming you take the terms "partnership" seriously and not just paying lip service to the term), the relationship is often clouded with doubt, suspicion & mistrust on all sides?

I've been in this technology software business for all of my professional life (~20 years), spending the first half working on technology product & platform development (company run by engineers & not accountants), and the latter half on the other (dark?) side - the corporate business - with in-house technology / IT "service" divisions, where business (accountants mostly non-technical) people run the show - an environment where there is often just a complete lack of alignment, understanding & appreciation for what goes on in both worlds: technology-and-business-worlds alike.

[Bias disclaimer: Yes, I do I have a bias - I believe if a company thinks itself to be a technology-driven business, then I expect the CEO to at least have an engineering/science background, or the CTO/CIO must have written code & built solid (industrial-strength) products in a previous lifetime. If a CEO is not technical then at least develop a keen appreciation for the engineering challenges and then trust your technology owners to do the responsible thing & you move out the way....]

At a recent ExCo of which I'm just an observer - this lack of alignment reared it's ugly head again when "the business" could not understand how a six-month's projects wish-list from their side, translated into a two-and-a-half-year implementation estimate from the technology team, making a poor attempt at explaining, the classic "IT demand & capacity planning" challenges (what I still find quite surprising in today's time, a topic for another day, let's just say I think "IT Demand Management Process" is not relevant today, and I propose that this is just "old-school-IT" & that its days are numbered).

What followed then from business was the typical:

"Why don't add another 20 development teams to make the six months timeline?"
"Why can't we have CFTs (cross-functional-teams) like the other divisions have?"
"Are you sure you have the right engineers? Why is the system so brittle? Why must it take so long? Should we take our business elsewhere?"
"Why can't we have bespoke technical teams attached to each business owner?"

What also makes this quite a difficult conversation is the CULTURE - where technology teams have historically been at the receiving end as the source of all business problems (the cause of business frustration), the underdogs, just taking a beating from business, without any confidence to push back & have constructive conversations. It's a shame really that a large number of technology teams just have to "submit" and "do as you're told" by business...surely it can't go on like this??

So this is what I've reflected so far - and it calls on both Technology & Business leaders to come together to the table, for me it's simply about ALIGNMENT. In order to align, one needs to really LISTEN. I mean REALLY LISTEN and PARTICIPATE in dialogue, after-all, this is what PARTNERSHIP supposed to be about, right??

Dear Technology Leader...

I think a large part of the issue is technology leaders not doing enough to communicate and simplify the reality of their world. Assuming you're not building a greenfields stack, instead you're building on existing systems (not legacy, but still probably reaching end-of-life), more needs to be done to manage expectations. I would avoid too much abstraction & side-stepping: be open and transparent about everything, no hiding...here's some searching questions:

Have you stopped yourself to reflect on why there is such a disconnect or lack of understanding? What are you doing wrong? Is there a fundamental issue of principle?
Have you taken your customers through the system architecture and current challenges?
Have you communicated your assumptions clearly? For example - does the business understand maybe you're trying to build a common platform/product stack that can support multiple business requirements (e.g. in a multi-country scenario) in one-go?

Is this assumption valid? What if the businesses are quite divergent in terms of customer requirements?
Why is it important to provide a common technology platform anyway? For who's benefit?
Would it not be easier to branch the code & platform to serve individual businesses? Risk duplication but gain in business agility?
Who cares if the code is forked anyway, has this option ever been tabled?

How can you explain the nuances of common stack software development to non-technical people?

Does business understand how you've structured your delivery teams?

For example, you might be running with component teams, specialisation instead of generalisation and not ready for cross-functional full-stack teams yet?
Are you clear where your bottlenecks are?
Is your software engineering principles solid - i.e. can it support multiple release streams simultaneously?

Have you shown to business the complexities of what it means to spawn up multiple teams (i.e. this rather stupid notion of adding more people to get the job done sooner)?

Have you established a way-of-working between business & technology that aims to at least address some of the challenges, and set forth an ultimate vision / aspiration of what ideal could look like?
Are you doing enough to manage expectations upwards and thus protect your teams?

Why do you have a need to insert "Business Relationship Managers (BRMs)? Are your technology teams not equipped in dealing with business stakeholders directly? Why do technology units decide that an intermediary BRM would help solve relationship problems?

Are you empowered enough to challenge priorities & requirements if they don't make sense to you?
Have you communicated that systems generally have to refresh every 3-5 years? Have you considered presenting a technology roadmap showing all the technical initiatives planned whilst simultaneously expected to keep the business lights on (business-as-usual) and still stay ahead of the curve to keep your stack technically relevant?

Dear Business Leader...

Just as there may be some gaps with your technology leader counterparts, you should consider what gaps are on your side. For starters, have you really invested time to LISTEN to your technology partners?

Have you done enough to convey your business vision, strategy & goals in a clear manner that informs your technology partners of the role they play & your dependence on them for your overall success?
Have you resisted the urge to bark orders without first trying to listen?
Do you have an understanding of how your technology teams have built the system & components? Perhaps you had a role to play in this - i.e. ending up in the current state because of business owners?
Did you perhaps grow up with the industrial mindset (factory methods) & expect that software systems follow similar patterns "it's just a production line outputting widgets"?
Do you understand what it means to write common software code and deploy common software & infrastructure to serve the needs of multiple businesses (with its own set of rules)?
Do you have a clear picture in your mind of the distinctions and challenges of software teams: Component teams versus cross-functional teams?
How do you as business owners come together to align on overall business requirements (assuming you're in multi-country set-up) that shares a common platform?

Did you even agree with technology that a common platform is the way to go?

Are you doing enough to review your own project's backlog and business requirements - by priority?
Do you accept that your technology team would appreciate from focus - i.e. do one thing of value at a time?
Are you doing enough to make sense of your own business priorities?
Have you read "The Mythical Man-Month"? Do you actually believe that "adding more people to a software project will get it done sooner?" - Are you not being naive, have you stopped yourself to reflect that maybe you're just not getting it??
What are you doing to get closer to understanding your technology provider's challenges?

Do you use the IT/technology systems on a daily basis?
Have you considered sitting closer to your technology team?

What is your role in the partnership? Is it too one-sided (i.e. your side)? Do you get immense joy at squeezing your technology providers, adding pressures and artificial deadlines to make your case?
Why do you feel you need an intermediary between yourself & technology - and insist on this concept of a BRM? Are you not interested to learn more about your technology stack or develop closer relationships with the people building your technology?

Attention Business-and-Technology Leaders - Don't look for that Silver Bullet...you'll have to discover this for yourselves, there's nothing off-the-shelf to plug-and-play for you...

You might have attempted every fad at improving process improvement, this "agile" topic has become so misused that both parties can't really tell if agile is working for you or not - and you're quick to blame the process, or attempt to try a new way, and even get professional consultants in to help...but...as I've experienced agile in a variety of forms in the past ten+ years, I still come back to the essence - and I really feel that people have misunderstood the core. This core that I refer to, is the founding principles espoused in the Agile Manifesto. Simple statements but deeply profound, and it all starts there. I feel this essence is often just skimmed over in favour of an execution process which has led to the downfall of many IT/Business relationships, which is why we keep coming back to the rather archaic IT/Business management frameworks that calls for IT Demand/Capacity Management and Business Relationship Managers...

Go back, breakdown the silos, no pedestals or ivory towers allowed in today's age, re-align & commit to participating in respectful, sincere dialogue, meeting each other half-way. Identify the root of the pain-points, agree on sensible ways to manage classic "capacity challenges" which simply implies some focused management of priorities by business people...if after you've tried to re-align, and still there's an expectation of infinite capacity from IT delivery...then...maybe there's just no hope left...

Yours sincerely,

A weathered-and-battered-technology-leader-looking-for-hope!

Thursday 15 August 2019

Whiners never win

Whiners never win!

What simple yet powerful, three words?

These are words my electrical engineering professor used on me back in my second year, when I challenged my paper being wrongly graded. I remember waiting a long time at his office just to see him, the prof just gave me one minute "Mr. Khan, here's some advice for the rest of your life - whiners never win". That was it, and then he left, my paper unchanged.

This stuck with me as I completed my degree, started my first job, took a chance one year after graduating by emigrating to two countries, marriage, parenthood, etc. I keep coming back to this simple message "whiners never win" and so just get on with life or work...

Yet, coming back to South Africa, now 8 years and counting, I was surprised to learn how much whining happens in the workplace! It is emotionally draining. Almost everyone has a complaint, not happy with this or that, unable to separate personal emotions from being professional at the workplace. Just a whole lot of whining going on. I wonder if it's just the context of the nation, just whining, starting with the commute into work listening to radio stations (like 702) which I've stopped completely. I actually don't listen to such news anymore, it starts the day of on the wrong note, stepping into the office with negativity and stress. Don't forget the stress of driving on SA roads...

I was also surprised how people tend to bring their home issues into workplace as well, affecting how they show up, perform, etc. My mindfulness, listening and empathy skills have seen an accelerated growth since 2011...

Still, eight years on, I still see it a lot of whining in the workplace...come on folks, stop whining.

Whining never wins.

Bring your best self to work, show up and get through the struggle...

Stop whining. "Whining never wins"...

Don't get me wrong, I've got loads of empathy...but at some point, people need to step up and just stop whining as whiners never win!

Sunday 11 August 2019

On using metaphors for Tech Ops

I like finding metaphors from other worlds outside of technology or engineering that can help simplify concepts for business people. Using such comparisons "is like a..." can actually be quite a powerful way of connecting or contextualising, especially when these people (business stakeholders mostly) are not interested in the technical details at all. All these folks care about, when there's an operational issue impacting business/customers, are: "What's the issue? Why did it happen? When will it be fixed? I need 15 minute updates please. Not interested in detail. Just want to know when it will be resolved. And don't forget Root Cause Analysis RCA document & Consequence Management!" When I get these remarks, I'm like...left dumbfounded...and decided to find another way of communicating to manage expectations better...

Enter...
#draintheswamp
#ICU

#draintheswamp

Despite the term being popularised by Trump last year, which was incidentally around the same time I introduced the term - in the software engineering community, "draining the swamp" is used to describe some refactoring and possibly restructuring of code and architecture, to address key bottlenecks, instabilities and optimisations. This could also mean revisiting original design & implementation decisions. Some people in software might put this under the bucket of "paying down technical debt" whilst others might just put this down as a natural part of the software evolution.

I used this term to prepare business people on the impact of the changes we planned to make. This was in response to a business ask of anticipating the biggest load of the tech platform stack - will the system cope with a "Black Friday-like event"? The simple answer was NO, not in its current form, that we would need to make some aggressive changes to the platform - like scaling out to multiple datacentres. The stack was built as a monolith (multiple services & components, some micro services, running on traditional VM infrastructure, not containerised, etc.) and so the best chances of scaling for load was to move the stack to run off multiple data centres, where we could scale up on each datacentre as well, but still had bottlenecks with share database cluster off one primary datacentre.

Suffice to say, this stuff goes above the heads of the business people. So I said, it's like draining the swamp. When that happens, we are bound to find some nasty things lying beneath the surface, things we don't see today, and can only see when we start draining the swamp. When this happens, and we hit some boulders or some unforeseen monsters of the swamp, we are bound to experience downtime and an outage or two. Don't panic, this is expected, given our working constraints, that is, make the changes on live production environment.

#draintheswamp went down pretty well with business. They got it. Left us alone. They trusted us! They managed the customer & business impact when we needed help, and our tech teams worked heroic hours to get the stack running off multiple datacentres, just in time for the biggest event of the business for the year (2018), our user numbers reached record highs, and the platform did not fall over! So round one of #draintheswamp was done, but the swamp was not completely drained out...

Until earlier this year (2019), when we kicked-off another round of #draintheswamp to start the migration to cloud stacks, starting with containerisation...which led the platform to suffer a series of outages at the most inappropriate moments...customers were pissed, business impact teams were on the back-foot, social media was killing us, the changes for #draintheswamp was starting to kill the platform...when I resorted to introducing another metaphor: #ICU. Folks, the tech stack is in some severe TLC, we are instigating #ICU mode. Life support is initiated, vitals are not great, but patient is surviving, and needs high level of critical care...

Technical Operations / Site Reliability Engineering is not so different to running a hospital's ER Emergency Department...at any time, despite monitoring vital signs of existing patients (system components of tech stack), or doing day-to-day operational management of the ward (infra maintenance), an event could occur that can just spike, like a natural disaster, accident or terrorist attack (hardware failure, critical component dies, database crash unanticipated, etc.)...ER staff have to triage quickly (Tech Ops also have to triage), make life/death calls (Tech Ops when to call a P1 incident & inform business), decide on severity of the injury (requires immediate attention, operate now, or can wait??)...the same holds true with bringing back a technical platform from death to living healthy operations...so why wouldn't we want to reuse medical terms?? I think it makes perfect sense!

#ICU

The gist of this metaphor is simple: ICU/CCU means Intensive/Critical Care Unit. When a patient is in ICU, it is serious stuff, urgent priority, focused attention to monitoring, diagnostics & multiple treatment options, using whatever means necessary to enable a positive outcome for the patient.

The typical flow to recovery to health looks for these transitions:

Start in ICU/CCU, remain there until a period of time where interventions applied & life-support is no longer necessary, then...
Transfer to High-Care facility, remaining close to ICU, but stable enough to warrant reduced focus and attention as compared to being in ICU, but be prepared for surprises, so best be close to ICU ward and not far away...wait until doctors give the green light to move on to...
Normal / General Ward...the last stay before checking out to go home. Vitals and all other required checks all pass before discharged for home...

Although being in #ICU is rather stressful for everyone, there is a high sense of urgency, and the pressure to respond to business is quite intense (especially when Risk/Governance demands "Consequence Management"), we can draw additional parallels from medical ICU, like:

Remaining calm, address the topics / unknowns in logical manner, using tools of diagnosis you've trained for (Technical tools like Five Whys, etc.)
Running tests, take blood samples, etc. (Tech/Data forensics, logging, test hypotheses, etc.)
Call on other medical experts to bounce diagnosis / brainstorm (Involve as many tech experts to offer new perspectives)
Implement treatment plan according to hypothesis, wait for new results (Fix something, wait, analyse result, before making additional changes).
Communicate clearly to patient's stakeholders (Communicate to business stakeholders transparently without hiding or being defensive...come clean).
Pray :-)

Unpacking the medical terms...

According to Wikipedia, an ICU:

An intensive care unit (ICU), also known as an intensive therapy unit or intensive treatment unit (ITU) or critical care unit (CCU), is a special department of a hospital or health care facility that provides intensive treatment medicine.

Intensive care units cater to patients with severe or life-threatening illnesses and injuries, which require constant care, close supervision from life support equipment and medication in order to ensure normal bodily functions. They are staffed by highly trained physicians, nurses and respiratory therapists who specialize in caring for critically ill patients. ICUs are also distinguished from general hospital wards by a higher staff-to-patient ratio and to access to advanced medical resources and equipment that is not routinely available elsewhere.

Patients may be referred directly from an emergency department if required, or from a ward if they rapidly deteriorate, or immediately after surgery if the surgery is very invasive and the patient is at high risk of complications

When a technical platform goes into #ICU, we do the following:

We're on red alert - life threatening to business (customer experience is tanking)
Stop all other development work, or reduce planned work as much as possible by pulling in all the people we need to help (speaks to higher-staff-to-patient ratio)
Pull out all the stops, bring in experts, tool-up with advanced resources and equipment
Communicate daily to all business stakeholders
Perform multiple surgeries if needed (hotfixes, patches, etc.)
Run multiple diagnostics in parallel (think Dr. House)

According to Wikipedia, a High Care/Dependency Unit:

A high-dependency unit is an area in a hospital, usually located close to the intensive care unit, where patients can be cared for more extensively than on a normal ward, but not to the point of intensive care. It is appropriate for patients who have had major surgery and for those with single-organ failure. Many of these units were set up in the 1990s when hospitals found that a proportion of patients was requiring a level of care that could not be delivered in a normal ward setting.^[1] This is thought to be associated with a reduction in mortality.^[1] Patients may be admitted to an HDU bed because they are at risk of requiring intensive care admission, or as a step-down between intensive care and ward-based care.

According to HealthTalk, the last point to recovery is General/Normal Ward:

People are transferred from the intensive care unit to a general ward when medical staff decide that they no longer need such close observation and one-to-one care. For many people, this move is an important step in their progress from being critically ill to recovering..

'Nuff said, IMHO the similitude mentioned above should be more than self explanatory ;-)