Wednesday, 4 December 2019

What makes OTT Streaming platforms different to DTH/DTT STB Broadcast?


In the early days of this blog, I shared in deep detail the technical and project landscape of PayTV systems for the traditional broadcast headend and set-top-box (STB) software platforms and projects as subject matter expert in this area.

In this post, I will provide a very high level overview of the core differences and challenges between broadcast DTH/SetTopBox systems versus OTT platforms. OTT has become a hot topic, with the nascent rise of OTT (Over-the-Top) internet streaming platforms that many a traditional PayTV operator is now transitioning to, in light of the growing competition for eyeballs & cord-cutters driven by the likes of primarily SVOD (Subscription Video On Demand) players like Netflix, Amazon & the like. The challenge though, is that these PayTV operators tend to trip up by bringing PayTV/DTH thinking into the OTT-world, hence my post.

The main aim of this post is targeted at senior executives who come from a broadcast world, and may not fully grasp the nuances of an online streaming world, specifically pointing out the spectrum of control of the end-to-end platform that PayTV operators generally enjoyed much comfort in that world, in contrast to the new world of online, where it becomes impractical to expect the level of control previously enjoyed - basically you can't control all the components of the internet, unless you invest in owning a large part of the network, which some operators have attempted & haven't really succeeded!! Simply put, the internet was not designed for Live TV streaming unlike traditional broadcast TV - the internet is somewhat chaotic & messy to say the least. In addition to this level of infrastructure control & lack thereof, an OTT platform is largely software systems designed for internet-scale (i.e. massively distributed server-side backend components) which requires a different skillset and experiences traditionally enjoyed in the closed ecosystem of broadcast & set top box device software development teams.

Another point of this post responds to the standpoint from traditional PayTV executives who may have spent a large part of their work in building broadcast systems with an engineering mindset anchored to a past era that is no longer relevant in the OTT world. Expressing opinions such as "engineering systems either work or they don't, it's either a 1 or 0, no in between states..with an expectation of 100% always-on availability..." in my view is somewhat misinformed & borders on belittling the engineering challenges faced in the online world. Whist every responsible engineer understands such demands and expectations of any system, a key point to remember is that broadcast TV has benefited from many years of refinement, standardisation, systems & processes specialised for the domain of TV - and when it comes to the internet - well, it was never really designed for video consumption, the speed of innovation has outpaced the development of standards & so is a much more messy world compared to traditional broadcast TV (satellite/terrestrial).

Disclaimer: This post is not about downplaying the broadcast engineering world, in fact, a lot of OTT operations highly depend on the upstream capabilities from PayTV operator's existing broadcast engineering infrastructure. Broadcast engineering systems are a marvel of engineering in its own respects, systems are highly complicated & superior engineering. OTT systems, and the point of my post, is to show the extension of the value chain, highlighting the new software & systems engineering challenges that present itself in the OTT world, that will be new concepts for existing broadcast engineering teams to learn & appreciate.

Key points for TL;DR folks (especially the PayTV Exec CXOs)

  • It's not a simple thing to compare OTT platforms wearing hat of a traditional DTH broadcast guy, it  is not simply a like-for-like comparison.
  • Don't be naive and pass value-judgements just because you were an engineer in the broadcast TV world (Headend or Set Top Box) does not necessarily make you qualified to understand OTT engineering domain. You can't bring broadcast/STB-thinking-mindset to the world of OTT.
  • PayTV operators have historically enjoyed a high level of end-to-end control, not so in OTT world.
  • There are far more components in the value chain of OTT than broadcast (backend, frontend and device software stack is more complicated in OTT than STB software).
  • High level of device & platform fragmentation in OTT that a traditional PayTV operator must consider.
  • The internet was never designed for TV consumption & has innovated at a much faster pace than traditional broadcast TV, the internet is a messy place, broadcast TV is somewhat cleaner and more determinstic.
  • OTT engineering teams have different challenges compared to traditional broadcast & set top box engineering.
  • PayTV operators need to plan carefully the convergence (DTH & OTT) of engineering teams and be prepared for a "clash of worlds".
  • Set Top Box software systems enjoy far more security control & protection than its OTT sibling.
  • Peak events & load challenges are unique to OTT platforms compared to broadcast DTH STB (scaling challenges don't really exist in DTH/STB).
  • Redundancy, Fast Rollbacks, Failover & Disaster Recovery are easier to achieve in OTT than in broadcast DTH/STB systems.

Why do I write about this stuff anyway?

<#humbleBrag If you're looking for an DTV technology expert, happy to have a chat for CTO roles!>
I've been working in engineering Digital TV software systems all my professional life of twenty years. Starting in the early 2000s, where there were just a handful of technology platform providers available (compared to today's world in online streaming where almost every Joe provides a streaming platform off-the-shelf), I have seen almost every transition in this space. I started with writing set-top-box applications built on middleware SDKs (then OpenTV & NDS MediaHighway platforms), to writing middleware & operating systems code for these embedded devices, called Set Top Boxes (STBs). I branched out into broadcast headend software systems that provided the necessary signalling that is core to powering STB applications, like the classic EPG (Electronic Program Guide). I studied and mastered the various IEEE MPEG & ISO/DVB standards in detail, as the middleware software had to implement the protocols (transport stream specifications) from scratch so the STB could interpret the data. I once implemented from scratch, STB client-side for decoding the full specification for signalling terrestrial networks (UK-D Book), which included a driver hardware abstraction layer, graphics library, application engine management and signalling.
I later branched out further to join the IPTV wave, building real-time streaming engines that converted these satellite broadcast streams into IP packets (entailed transforming MPEG packets, signalling, wrapping over IP packetisation - converting an MPTS to SPTS, adding transcoding & encryption then playout of IP), doing live streaming encryption on-the-fly in real-time as well as media file-based offline-encryption packaged VOD (Video On Demand) assets for consumption over the network. Back then we built all these systems from the ground up on standard hardware, running on standard servers (it was truly pioneering stuff). The network streaming was primarily over UDP multicast. We had to build everything, from the control plane managing the clusters with seamless failover, full management console for administrating the IP headend, etc. We built our own transcoding engines, taking in the MPEG source and converting to other formats - all before the time that internet streaming started to become mainstream. This meant going down to the detail of coding the MPEG signalling standards down to the bits & bytes of the video containers, timestamping & metadata signalling. We built these distributed computing platforms from scratch, there weren't any widely available frameworks for abstracting the network & infrastructure as we have today. CDNs were just being built around the same time keeping pace with the ever changing landscape of web technologies, back then IPTV systems took the form of traditional "headends" that sat in telco/broadcast data centres, Live TV was mostly UDP multicast, with VOD files being stored close to the edges.  The consuming devices at the time were still STBs, and we were just starting on the web browsers as the client. I can truly say I've experienced just about the full journey of these systems, apart from writing any security-side code for conditional access & digital rights management (and also the distribution satellite systems was not my thing, I focused on building software). I then transitioned into engineering management roles that was all about software delivery of these value chains, and became quite an expert at it. In the last few years, my focus has been online, which is really the future of TV. There was actually a period in between for Mobile-TV broadcaast, DVB-H which tried to precede OTT, some PayTV operators gambled this would take off, in hindsight, that time and investment could have been better spent learning & developing their OTT platform instead.  In my opinion the classic broadcast TV systems using unconnected STBs is dead, time to move on. This technology is now commodity, PayTV operators shouldn't invest too much in this world, they should rather sweat their assets and drive all operating costs down to be lean & efficient, stop manufacturing bespoke custom STBs & applications IMHO. The days of closed proprietary STB middleware software is also dead, time to openly embrace more modern stacks, that come very close to seamless integration with the online apps world.
</#humbleBrag>

How TV is PayTV changing?

Today, the world is different and changing all the time. With the rise of the browser, "streaming over the top" has advanced at a much faster pace than the traditional broadcast world. There are so many topics I can talk about, which might become future posts. Take for example how software patterns for distributed systems had to evolve for internet web scale. A mistake often made in the early development of online-equivalent of traditional TV streaming, is one where the software stack for the streaming world emulated that of a STB, expecting real-time processing, synchronous calls to APIs, etc. Depending where you started, you might have a lot of legacy code to clean-up & undo design patterns that worked previously, but can't scale sufficiently to deal with exponential growth of online users. Some PayTV operators may have anticipated this online streaming by building their own online streaming platform, maybe 5-7 years back, resulting in some legacy already. Today, with the rise of cloud providers, and the constant comparison with the god of streaming (Netflix), PayTV executives expect the same level of performance & reliability from their in-house engineering team. Deciding whether your legacy online platform has reached the end of its shelf life, whether to continue to the restructure journey (e.g. migration to cloud), or hit refresh, starting from scratch build, or use off-the-shelf platforms, is a topic that has kept me awake many a sleepless night. A topic for another day. Another topic that makes traditional PayTV executives a little uneasy, is that the engineering, development, delivery & operational management for OTT platforms are much more dynamic & agile in nature, the fact that apps can be patched and upgraded quickly, that deployments to production happens almost daily, challenging traditional governance and change control procedures that demand varying levels of sign-off & approval, that developers can access production environment, etc. are topics IMHO that every senior executive from the old-world of PayTV, needs to come to terms with - i.e. don't expect your digital OTT engineers to behave the same way as your broadcast engineers! Anyway, these topics are best left for future posts...

Right, now that's off my chest, let's look at the big topics that speak to differences in traditional broadcast satellite TV platforms compared to OTT platforms:

Sunday, 8 September 2019

Open letter to Business & Tech Leaders


Dear Business & Technology Owners...

Have you ever stopped to reflect on why is there always tension and conflict between business stakeholders and technology providers? Instead of there being a "healthy tension" between the two partners (assuming you take the terms "partnership" seriously and not just paying lip service to the term), the relationship is often clouded with doubt, suspicion & mistrust on all sides?

I've been in this technology software business for all of my professional life (~20 years), spending the first half working on technology product & platform development (company run by engineers & not accountants), and the latter half on the other (dark?) side - the corporate business - with in-house technology / IT "service" divisions, where business (accountants mostly non-technical) people run the show - an environment where there is often just a complete lack of alignment, understanding & appreciation for what goes on in both worlds: technology-and-business-worlds alike.

[Bias disclaimer: Yes, I do I have a bias - I believe if a company thinks itself to be a technology-driven business, then I expect the CEO to at least have an engineering/science background, or the CTO/CIO must have written code & built solid (industrial-strength) products in a previous lifetime. If a CEO is not technical then at least develop a keen appreciation for the engineering challenges and then trust your technology owners to do the responsible thing & you move out the way....]

At a recent ExCo of which I'm just an observer - this lack of alignment reared it's ugly head again when "the business" could not understand how a six-month's projects wish-list from their side, translated into a two-and-a-half-year implementation estimate from the technology team, making a poor attempt at explaining, the classic "IT demand & capacity planning" challenges (what I still find quite surprising in today's time, a topic for another day, let's just say I think "IT Demand Management Process" is not relevant today, and I propose that this is just "old-school-IT" & that its days are numbered).  

What followed then from business was the typical:
  • "Why don't add another 20 development teams to make the six months timeline?"
  • "Why can't we have CFTs (cross-functional-teams) like the other divisions have?" 
  • "Are you sure you have the right engineers? Why is the system so brittle? Why must it take so long? Should we take our business elsewhere?"
  • "Why can't we have bespoke technical teams attached to each business owner?"
What also makes this quite a difficult conversation is the CULTURE - where technology teams have historically been at the receiving end as the source of all business problems (the cause of business frustration), the underdogs, just taking a beating from business, without any confidence to push back & have constructive conversations. It's a shame really that a large number of technology teams just have to "submit" and "do as you're told" by business...surely it can't go on like this??

So this is what I've reflected so far - and it calls on both Technology & Business leaders to come together to the table, for me it's simply about ALIGNMENT. In order to align, one needs to really LISTEN. I mean REALLY LISTEN and PARTICIPATE in dialogue, after-all, this is what PARTNERSHIP supposed to be about, right??

Dear Technology Leader...
I think a large part of the issue is technology leaders not doing enough to communicate and simplify the reality of their world. Assuming you're not building a greenfields stack, instead you're building on existing systems (not legacy, but still probably reaching end-of-life), more needs to be done to manage expectations. I would avoid too much abstraction & side-stepping: be open and transparent about everything, no hiding...here's some searching questions:
  • Have you stopped yourself to reflect on why there is such a disconnect or lack of understanding? What are you doing wrong? Is there a fundamental issue of principle?
  • Have you taken your customers through the system architecture and current challenges?
  • Have you communicated your assumptions clearly? For example - does the business understand maybe you're trying to build a common platform/product stack that can support multiple business requirements (e.g. in a multi-country scenario) in one-go?
    • Is this assumption valid? What if the businesses are quite divergent in terms of customer requirements?
    • Why is it important to provide a common technology platform anyway? For who's benefit?
    • Would it not be easier to branch the code & platform to serve individual businesses? Risk duplication but gain in business agility? 
    • Who cares if the code is forked anyway, has this option ever been tabled?
  • How can you explain the nuances of common stack software development to non-technical people?
    • Does business understand how you've structured your delivery teams?
      • For example, you might be running with component teams, specialisation instead of generalisation and not ready for cross-functional full-stack teams yet?
      • Are you clear where your bottlenecks are?
      • Is your software engineering principles solid - i.e. can it support multiple release streams simultaneously?
    • Have you shown to business the complexities of what it means to spawn up multiple teams (i.e. this rather stupid notion of adding more people to get the job done sooner)?
  • Have you established a way-of-working between business & technology that aims to at least address some of the challenges, and set forth an ultimate vision / aspiration of what ideal could look like?
  • Are you doing enough to manage expectations upwards and thus protect your teams?
    • Why do you have a need to insert "Business Relationship Managers (BRMs)? Are your technology teams not equipped in dealing with business stakeholders directly? Why do technology units decide that an intermediary BRM would help solve relationship problems?
  • Are you empowered enough to challenge priorities & requirements if they don't make sense to you?
  • Have you communicated that systems generally have to refresh every 3-5 years? Have you considered presenting a technology roadmap showing all the technical initiatives planned whilst simultaneously expected to keep the business lights on (business-as-usual) and still stay ahead of the curve to keep your stack technically relevant?

Dear Business Leader...
Just as there may be some gaps with your technology leader counterparts, you should consider what gaps are on your side. For starters, have you really invested time to LISTEN to your technology partners?
  • Have you done enough to convey your business vision, strategy & goals in a clear manner that informs your technology partners of the role they play & your dependence on them for your overall success?
  • Have you resisted the urge to bark orders without first trying to listen?
  • Do you have an understanding of how your technology teams have built the system & components? Perhaps you had a role to play in this - i.e. ending up in the current state because of business owners?
  • Did you perhaps grow up with the industrial mindset (factory methods) & expect that software systems follow similar patterns "it's just a production line outputting widgets"?
  • Do you understand what it means to write common software code and deploy common  software & infrastructure to serve the needs of multiple businesses (with its own set of rules)?
  • Do you have a clear picture in your mind of the distinctions and challenges of software teams: Component teams versus cross-functional teams?
  • How do you as business owners come together to align on overall business requirements (assuming you're in multi-country set-up) that shares a common platform?
    • Did you even agree with technology that a common platform is the way to go?
  • Are you doing enough to review your own project's backlog and business requirements - by priority?
  • Do you accept that your technology team would appreciate from focus - i.e. do one thing of value at a time?
  • Are you doing enough to make sense of your own business priorities?
  • Have you read "The Mythical Man-Month"? Do you actually believe that "adding more people to a software project will get it done sooner?" - Are you not being naive, have you stopped yourself to reflect that maybe you're just not getting it??
  • What are you doing to get closer to understanding your technology provider's challenges?
    • Do you use the IT/technology systems on a daily basis?
    • Have you considered sitting closer to your technology team?
  • What is your role in the partnership? Is it too one-sided (i.e. your side)? Do you get immense joy at squeezing your technology providers, adding pressures and artificial deadlines to make your case?
  • Why do you feel you need an intermediary between yourself & technology - and insist on this concept of a BRM? Are you not interested to learn more about your technology stack or develop closer relationships with the people building your technology? 
Attention Business-and-Technology Leaders - Don't look for that Silver Bullet...you'll have to discover this for yourselves, there's nothing off-the-shelf to plug-and-play for you...
You might have attempted every fad at improving process improvement, this "agile" topic has become so misused that both parties can't really tell if agile is working for you or not - and you're quick to blame the process, or attempt to try a new way, and even get professional consultants in to help...but...as I've experienced agile in a variety of forms in the past ten+ years, I still come back to the essence - and I really feel that people have misunderstood the core. This core that I refer to, is the founding principles espoused in the Agile Manifesto. Simple statements but deeply profound, and it all starts there. I feel this essence is often just skimmed over in favour of an execution process which has led to the downfall of many IT/Business relationships, which is why we keep coming back to the rather archaic IT/Business management frameworks that calls for IT Demand/Capacity Management and Business Relationship Managers...

Go back, breakdown the silos, no pedestals or ivory towers allowed in today's age, re-align & commit to participating in respectful, sincere dialogue, meeting each other half-way. Identify the root of the pain-points, agree on sensible ways to manage classic "capacity challenges" which simply implies some focused management of priorities by business people...if after you've tried to re-align, and still there's an expectation of infinite capacity from IT delivery...then...maybe there's just no hope left...

Yours sincerely,
A weathered-and-battered-technology-leader-looking-for-hope!

Thursday, 15 August 2019

Whiners never win

Whiners never win!
What simple yet powerful three words?
These are words my electrical engineering professor used on me back in my second year, when I challenged my paper being wrongly graded.  I remember waiting a long time at his office just to see him, the prof just gave me one minute "Mr. Khan, here's some advice for the rest of your life - whiners never win". That was it, and then he left, my paper unchanged.

This stuck with me as I completed my degree, started my first job, took a chance one year after graduating by emigrating to two countries, marriage, parenthood, etc. I keep coming back to this simple message "whiners never win" and so just get on with life or work...

Yet, coming back to South Africa, now 8 years and counting, I was surprised to learn how much whining happens in the workplace! It is emotionally draining. Almost everyone has a complaint, not happy with this or that, unable to separate personal emotions from being professional at the workplace. Just a whole lot of whining going on. I wonder if it's just the context of the nation, just whining, starting with the commute into work listening to radio stations (like 702) which I've stopped completely. I actually don't listen to such news anymore, it starts the day of on the wrong note, stepping into the office with negativity and stress. Don't forget the stress of driving on SA roads...

I was also surprised how people tend to bring their home issues into workplace as well, affecting how they show up, perform, etc. My mindfulness, listening and empathy skills have seen an accelerated growth since 2011...

Still, eight years on, I still see it a lot of whining in the workplace...come on folks, stop whining. Whining never wins.
Bring your best self to work, show up and get through the struggle...

Stop whining. "Whining never wins"...

Don't get me.wrong, I've got loads of empathy...but at some point, people need to step up and just stop whining as whiners never win!

Sunday, 11 August 2019

On using metaphors for Tech Ops


I like finding metaphors from other worlds outside of technology or engineering that can help simplify concepts for business people. Using such comparisons "is like a..." can actually be quite a powerful way of connecting or contextualising, especially when these people (business stakeholders mostly) are not interested in the technical details at all. All these folks care about, when there's an operational issue impacting business/customers, are: "What's the issue? Why did it happen? When will it be fixed? I need 15 minute updates please. Not interested in detail. Just want to know when it will be resolved. And don't forget Root Cause Analysis RCA document & Consequence Management!When I get these remarks,  I'm like...left dumbfounded...and decided to find another way of communicating to manage expectations better...

Enter...
#draintheswamp
#ICU

#draintheswamp

Despite the term being popularised by Trump last year, which was incidentally around the same time I introduced the term - in the software engineering community, "draining the swamp" is used to describe some refactoring and possibly restructuring of code and architecture, to address key bottlenecks, instabilities and optimisations. This could also mean revisiting original design & implementation decisions. Some people in software might put this under the bucket of "paying down technical debt" whilst others might just put this down as a natural part of the software evolution.

I used this term to prepare business people on the impact of the changes we planned to make. This was in response to a business ask of anticipating the biggest load of the tech platform stack - will the system cope with a "Black Friday-like event"? The simple answer was NO, not in its current form, that we would need to make some aggressive changes to the platform - like scaling out to multiple datacentres. The stack was built as a monolith (multiple services & components, some micro services, running on traditional VM infrastructure, not containerised, etc.) and so the best chances of scaling for load was to move the stack to run off multiple data centres, where we could scale up on each datacentre as well, but still had bottlenecks with share database cluster off one primary datacentre.

Suffice to say, this stuff goes above the heads of the business people. So I said, it's like draining the swamp. When that happens, we are bound to find some nasty things lying beneath the surface, things we don't see today, and can only see when we start draining the swamp. When this happens, and we hit some boulders or some unforeseen monsters of the swamp, we are bound to experience downtime and an outage or two. Don't panic, this is expected, given our working constraints, that is, make the changes on live production environment.

#draintheswamp went down pretty well with business. They got it. Left us alone. They trusted us! They managed the customer & business impact when we needed help, and our tech teams worked heroic hours to get the stack running off multiple datacentres, just in time for the biggest event of the business for the year (2018), our user numbers reached record highs, and the platform did not fall over! So round one of #draintheswamp was done, but the swamp was not completely drained out...

Until earlier this year (2019), when we kicked-off another round of #draintheswamp to start the migration to cloud stacks, starting with containerisation...which led the platform to suffer a series of outages at the most inappropriate moments...customers were pissed, business impact teams were on the back-foot, social media was killing us, the changes for #draintheswamp was starting to kill the platform...when I resorted to introducing another metaphor: #ICU. Folks, the tech stack is in some severe TLC, we are instigating #ICU mode. Life support is initiated, vitals are not great, but patient is surviving, and needs high level of critical care...

Technical Operations / Site Reliability Engineering is not so different to running a hospital's ER Emergency Department...at any time, despite monitoring vital signs of existing patients (system components of tech stack), or doing day-to-day operational management of the ward (infra maintenance), an event could occur that can just spike, like a natural disaster, accident or terrorist attack (hardware failure, critical component dies, database crash unanticipated, etc.)...ER staff have to triage quickly (Tech Ops also have to triage), make life/death calls (Tech Ops when to call a P1 incident & inform business), decide on severity of the injury (requires immediate attention, operate now, or can wait??)...the same holds true with bringing back a technical platform from death to living healthy operations...so why wouldn't we want to reuse medical terms?? I think it makes perfect sense!


#ICU

The gist of this metaphor is simple: ICU/CCU means Intensive/Critical Care Unit. When a patient is in ICU, it is serious stuff, urgent priority, focused attention to monitoring, diagnostics & multiple treatment options, using whatever means necessary to enable a positive outcome for the patient.

The typical flow to recovery to health looks for these transitions:

  • Start in ICU/CCU, remain there until a period of time where interventions applied & life-support is no longer necessary, then...
  • Transfer to High-Care facility, remaining close to ICU, but stable enough to warrant reduced focus and attention as compared to being in ICU, but be prepared for surprises, so best be close to ICU ward and not far away...wait until doctors give the green light to move on to...
  • Normal / General Ward...the last stay before checking out to go home. Vitals and all other required checks all pass before discharged for home...
Although being in #ICU is rather stressful for everyone, there is a high sense of urgency, and the pressure to respond to business is quite intense (especially when Risk/Governance demands "Consequence Management"), we can draw additional parallels from medical ICU, like:
  • Remaining calm, address the topics / unknowns in logical manner, using tools of diagnosis you've trained for (Technical tools like Five Whys, etc.)
  • Running tests, take blood samples, etc. (Tech/Data forensics, logging, test hypotheses, etc.)
  • Call on other medical experts to bounce diagnosis / brainstorm (Involve as many tech experts to offer new perspectives)
  • Implement treatment plan according to hypothesis, wait for new results (Fix something, wait, analyse result, before making additional changes).
  • Communicate clearly to patient's stakeholders (Communicate to business stakeholders transparently without hiding or being defensive...come clean).
  • Pray :-)

Unpacking the medical terms...

According to Wikipedia, an ICU:
An intensive care unit (ICU), also known as an intensive therapy unit or intensive treatment unit (ITU) or critical care unit (CCU), is a special department of a hospital or health care facility that provides intensive treatment medicine.
Intensive care units cater to patients with severe or life-threatening illnesses and injuries, which require constant care, close supervision from life support equipment and medication in order to ensure normal bodily functions. They are staffed by highly trained physiciansnurses and respiratory therapists who specialize in caring for critically ill patients. ICUs are also distinguished from general hospital wards by a higher staff-to-patient ratio and to access to advanced medical resources and equipment that is not routinely available elsewhere.
Patients may be referred directly from an emergency department if required, or from a ward if they rapidly deteriorate, or immediately after surgery if the surgery is very invasive and the patient is at high risk of complications
When a technical platform goes into #ICU, we do the following:

  • We're on red alert - life threatening to business (customer experience is tanking)
  • Stop all other development work, or reduce planned work as much as possible by pulling in all the people we need to help (speaks to higher-staff-to-patient ratio)
  • Pull out all the stops, bring in experts, tool-up with advanced resources and equipment
  • Communicate daily to all business stakeholders
  • Perform multiple surgeries if needed (hotfixes, patches, etc.)
  • Run multiple diagnostics in parallel (think Dr. House)

According to Wikipedia, a High Care/Dependency Unit:
high-dependency unit is an area in a hospital, usually located close to the intensive care unit, where patients can be cared for more extensively than on a normal ward, but not to the point of intensive care. It is appropriate for patients who have had major surgery and for those with single-organ failure. Many of these units were set up in the 1990s when hospitals found that a proportion of patients was requiring a level of care that could not be delivered in a normal ward setting.[1] This is thought to be associated with a reduction in mortality.[1] Patients may be admitted to an HDU bed because they are at risk of requiring intensive care admission, or as a step-down between intensive care and ward-based care.
According to HealthTalk, the last point to recovery is General/Normal Ward:
People are transferred from the intensive care unit to a general ward when medical staff decide that they no longer need such close observation and one-to-one care. For many people, this move is an important step in their progress from being critically ill to recovering..
'Nuff said, IMHO the similitude mentioned above should be more than self explanatory ;-)

Tuesday, 6 August 2019

On company values


"Disagree and commit"
"Embrace the elephant"
"Don't be a d!ck"
"Listen. Challenge. Commit. A good leader has the humility to listen, the confidence to challenge, the wisdom to know when to stop arguing and commit."
"We use data to drive decision making. Data-driven-decisions."

These are are just some of the catchy phrases that some "modern" workplaces aspire to implement as company culture....great words, but easier said than done in practise IMHO. There's this somewhat unreasonable expectation that human beings can just implement this stuff - but we all know too well, humans are emotional and predictably irrational...

If you're a leader...
You're bound to issue instructions, ideas or directives that are not going to sit well with your teams. You have a duty to listen to feedback, alternative points of view, and use these as inputs into your thought process and ultimately make the call. It's not about paying lip service and say "I've listened" without truly listening...what if you're wrong, and you might even be the HIPPO in the room? Even if your decision doesn't change, and you set the directive, you're ultimately responsible and accountable for the outcome.
What if the directive appears to be wrong? Do you stick it out till the end because of your ego and risk losing credibility? Some might say a real leader will step up, and own up saying "Hey, I was wrong guys, but we gained a lot of learning from this."
Still, as a leader, you then trust (expect) your team to follow-through no matter what...but is this even as simple as it seems? Probably not...how do you behave when you find out that there's still resistance? Do you flip your lid? What kind of a leader does that make you? Some might argue that being decisive, forthright, firm and tough on naysayers are great attributes for a leader...."the buck stops here!".

If you're on the receiving end of the instruction / strategic directive and you disagree...
You've said your piece, provided feedback, you may have even fundamentally disagreed with the decision...and now you're faced with understanding yourself: Can you let go, can you commit, even though you violently disagreed? Are you serious about the best outcome for the team regardless of your personal opinion? How will you stop yourself from unconsciously falling back into dissent-mode?
As a team player, your leader expects you do so. But have you prepared yourself to work towards that outcome?
How do you stop yourself from sounding like a stuck record?
My humble advice: stop being this guy. If you can't let go, then you're limiting your own growth.
BUT...ask yourself this: What am I willing to walk away from?
If the directive conflicts fundamentally with your core professional or personal value system, then what do you do?
My view: if it gets to that level of personal dilemma, and you're so sure of your value system, then leave, quit the company altogether, or change teams...it depends on how serious you are about this value system of yours.
If the conflict is not even near enough to compromising your value system, then ask yourself how can you adapt your own behaviour & approach, how can you help to solving the problem, and how can you help influence the outcome? How can you show your leader you're committed, no matter what?

This isn't always easy...the emotional forces at play can be intense, pulling you back...but once committed, your leader is expecting following through and be a team player.

Both Leader and Follower need to have a keen handle on self-awareness, "Know thyself"...some call this "mindfulness"...

Sunday, 3 March 2019

On choices, decisions, initiative, career planning: lessons, looking back


Image Source
At a recent town hall with my technology division (100+ people: frontend & backend engineers, platforms & infrastructure systems engineers, system architects, agile specialists, AI/ML engineers, Agile PMO & technical ops monitors), I opened up by giving my perspective on expected behaviours & responsibilities of the individual, especially when it comes to career development & growth aspirations.

The message was about taking a personal ownership for one's own growth, rather than leaving it up to the company or one's manager (a message that surfaced a few times in recent OfficeVibe feedback) - that the responsibility largely lies with the individual. Yes, managers/leaders are there to support you & guide you along the way, but only you know what defines you as a person, so don't leave it up to others to determine a path for you...

Reflecting on my own journey, it all comes down to understanding your current reality, weighing the choices on the table, defining your aspirations, taking initiative,  processing & reflecting through assessing the outcome of the initiatives, finding great people to learn from, tracking your trajectory on the path to growth. The path is not always clear, sometimes adjustments need to be made, sometimes a little backtracking is needed to enable the next leap forward - still, it all comes down to one's own personal ownership & level of commitment to controlling one's own future.

I thought I'd share my own timeline as guidance for people who might be stuck. Interestingly enough, although I only recently started formally implementing my own management framework around life/work planning by way of my RAGE model, that I was actually instinctively using this decision-making model all the time.

My timeline table shows the major periods in my career, commenting on reality of the situation at the time, choices I faced, decision made & eventual outcome. I think anyone who's considering what to do next with their career plan should do a similar exercise for their own sense-making.

My Perspectives

  1. Gain a keen appreciation for your current situational reality and take responsibility for it. Yes, reality sucks sometimes, but you got to play the hand you're dealt, don't let that get you down.  It is possible to change your reality. I made hard choices based on my reality of being caught in a low-income family, living through Apartheid.
  2. Discover your key motivations and use them as your guiding compass, some call it your "value system". I believed in myself and my ability to make things happen. If I felt my knowledge was lacking, I would learn & close the gaps myself. Don't assume you know everything, there are tons of smart people out there. I got a huge awakening when I went overseas, so much so that I had to learn software engineering & computer science all over again.
  3. Don't go seeking hand-outs or help, but if people or companies do extend their generosity, don't naively turn them down. There will be good people helping you along the way. After trying many avenues of financial aid/scholarships without success, I thought help would never come my way, but it eventually did.
  4. Don't bog yourself down with "If Only", or "What If" - this creates negativity & unnecessary anxieties. Move on, look forward. Sure, reflect on the past, learn from it, but never let it hold you back. You're in control of shaping your own reality. I chose a path that was the most practical, I switched jobs just as I was going to be promoted, I left big projects just as they were about to land, I left a start-up thinking I had a job lined up (but it didn't happen), I left what others would say is madness (left the stability of UK to return to volatile SA). Leaving UK was very difficult for me from a professional experience sacrifice, but I never allowed doubt and negativity to bog me down.
  5. If you want a good measure of your skills or experience an alternate reality, leave your country & work overseas. Even though the world has gone smaller through globalisation, that even in South Africa we do work with international teams, I still think getting overseas exposure is one of the best things one can do. Living and working in different countries exposes you to a different world of experiences. If you're under the age of 35, then you should try it. It doesn't have to mean relocating or emigrating, it could be a temporary secondment for a year or two. I was fortunate to experience working with many cultures across the globe on some really big projects. I learnt so much in a short space of time, it took my engineering skills to another level. As for Planning, Management & Execution principles, in my opinion, the UK ethic is world-class.
  6. Become comfortable with uncertainty & embrace the unknown, even it means leaving your home town/country for another one. You get this only through experience, and having been through at least one transition into the unknown. I've seen a few - it's not so bad, you must embrace your fear of uncertainty.
  7. Develop a learning & growth mindset - in any new role, work hard to learn as much as you can, by reading, studying, latching on to people as mentors, read other people's code through open source projects, etc. I became expert in a few areas: MPEG/DVB protocol spec & implementation, C & C++ coding, Voice synthesis & Text-to-Speech, Project, Agile Program Management & Execution, Professional services consulting and more recently Leadership skills. This doesn't come easy: I read a ton, implement the tools, learn from experienced, the wise, still remaining open to new experiences, no matter how edgy they might make me feel.
  8. Be ready to start-over again more than once - switch roles, domains or industries, sometimes what might seem to be a step or two backwards, actually turns out to be better than hoped. I started over at least 5 times in my career of 20 years. If you're in software, sure you can specialise (and there's nothing wrong with that) but you must then become expert at what you do. To grow in software, my view is to learn as many tools as possible, switch every two years. One of the best ways to do this is side projects, open source communities. Don't wait for your company to reserve hackathon sprints, or 20% time - take ownership. I taught myself text-to-speech synthesis on my own, developed POCs in my spare time and proved to company the potential innovation. If I had not taken initiative, I probably would not have landed the ultimate technical role I dreamt of.
  9. Don't pass the responsibility for your career on to someone else (your manager or company), rather you should have a view of your own map, your end goal. Your company or leaders can help with options available, guide on the gaps you need to fill - and it's even better when the company has a decent career ladder in place. Never pass the buck on and make excuses that your company / manager does not care, that you don't have enough syncs 1:1s or feedback sessions with your manager. You need to take ownership, period. In the companies I worked for, we at least had a decent career-ladder in place, showing all the upwards, sideways opportunities available. I made it clear to my leaders at the time, that my ultimate goal was to become a "Jack of all trades, but Master of SOME" T/PI-Shaped skills. They knew that when I'd enter a new role, I would learn, produce outcomes and then move on, thankfully, I had very good leaders that did not stand in my way. If you feel obstructed by your leader / team / company, first dig deep within yourself to reflect on whether your own behaviours need improving, and if you're still convinced it's not you, then leave, change your environment, change your circumstances.
  10. Don't get complacent or too confident your role is secure, retrenchments & redundancies are a reality, business-is-business. I got my first taste of layoffs when I was still a junior software engineer, naively thinking I was in a good place by virtue of being part of a cool new product team, and owning some key components. Since that first-and-only layoff (in 20 years), I developed my "spider senses" - and decided that it would always be me that decides whether I stay or leave, not a market event or the company.
  11. Do not grow an entitled mindset, or have unrealistic expectations from your employer. Say you studied hard and earned additional paper qualifications: MSc, MBA, PMP, etc. Don't expect the company to automatically increase your salary or grant you a promotion. Say your own personal life changes, you get married or have children, so you have more responsibility at home. Why should you expect your company to give you an increase, if the work you're doing hasn't materially changed, or your output is still the same, and you're still working at the level the role expects?? At the end of the day, it's up to you to manage your personal circumstances, it's not the company responsibility now to just automatically reward you or make your life easier financially - NO - you have to work at it. If you gained new qualifications, you need to show an interest in contributing your newly acquired knowledge, showing value. If you're seeking a promotion, you must show you've actively contributed covering much of the roles/output of the next role you're seeking. Companies don't owe you anything - so don't come up with unreasonable expectations or feel entitled. It's all about your output, meritocracy  is the only thing that matters.
  12. Becoming your own boss, running your own consultancy is hard work, be prepared to fail in this area. Although branching out on your own can be enormously liberating & exciting, unless you have a large network to tap into, moving from one consulting engagement to another, building up clientele & a pipeline of work, growing your team - whilst a lot of people have made this a successful venture - it is actually quite hard to do. I was fortunate to secure the company I left as my major client, although, the client only wanted to work with me, so in effect, I never really left! Without a strong network, the going was tough trying to break into other clients, even after doing some pro bono consulting work. You must invest a lot of time & energy, unpaid hours to build your own consultancy, something I didn't do, which showed I wasn't really fully invested in this venture. So I shut my company down, and was pulled back in by the strong gravitational forces of the big company. I learnt a great deal, became a better salesman, and became confident in interacting with C-Level executives. Consulting however, is a sure way to make extra money than being a permanent employee, but it comes with its own set of risks.
  13. Show gratitude. Whilst you might think that you're in control and the result of your successes are due to your own hard work, sweat and tears, never become arrogant and ignore that other forces helped you get this far. Take time to seriously reflect on this, and you will soon identify people or events that helped, and when these surface - be thankful & show your gratitude, develop humility. Although I was brought up in low-income household, I never once felt not having a complete home, or solid upbringing on life skills - if anything - this actually shaped my personal motivational value system. I never regretted or blamed my parents, I had a good childhood, was taught responsibility & key life skills. I've acknowledged people, friends and family that shaped my reality, the leaders in the various companies I worked with, were supportive, friendly and encouraging - I learnt so much from them, and still continue to learn from them today. Sometimes, when you're in the thick of the day-to-day job, you might not like what your manager says or does (over controlling, micromanaging, lecturing), and it's only when you leave, you realise the wisdom and lessons being taught. It's not always about you, and if you're leading teams, show appreciation for your teams as well. Your success is a result of your team's output. As you develop into senior roles, your visible output might become less-and-less, but you're still working hard through people, in the background. Never think you're the sole reason behind success - there's so much more that goes on, that we're often blindsided - don't get blindsided, actively seek out your blind spots.
  14. Be patient. Patience is linked to gratitude. Be patient with your role, allow enough time to learn the essence of the domain. Once you're comfortable & confident in your appreciation of the essence or core principles and you've remained long enough in the role to complete 1-2 major pieces of work / projects, then allow yourself the opportunity to move on. But don't rush things through, learning needs time to soak. Personally I'm always doubtful when I come across people's CVs hopping from one permanent role to the next in less than 18 months (this is because the major initiatives usually run for at least 18 months). From my experience, this (9-15 months) is just not enough time to do justice or have made a serious contribution in terms of outcomes (unless the gig was to rescue, recover or revive a project as a major intervention, or consulting gig). For me, it's been roughly a minimum of 18 months provided I felt confident in my results. I just about completed my engineer-in-training role after university when I took a big chance, although successful, in retrospect, I had big gaps to close anyway. The more higher one climbs the ladder, the more patient one needs to be, which means 18 months could grow to 24-36 months minimum. Currently I'm resisting the urge to switch, knowing that I still have another year to go before I can claim to have truly owned the role, so patience becomes a necessity.
  15. It's not always about the money or job title - neither does "years in role" contribute to "seniority". Although I might risk passing a value judgement on other people here, what I found is that money should not be a driving motivation, if you've set your sights on a learning and growth mindset. Sure, you can hop from one job/company to the next every 12 months or so, on each move your salary jumps - but if you're effectively not learning new skills or growing, is it worth it? Job titles are also relative, what's in a name after all? What matters is what you can do, and the value you bring to the table. What I found also trips people up is this complex of "seniority" based on "I've worked so many years in this role, hence I deserve a promotion as a senior xyz". In my journey, I sacrificed salary growth for knowledge, experience and a wide/deep toolbox of skills/capabilities. Later when things became challenging financially, other opportunities opened up that boosted my income, which wouldn't have been possible if I'd not honed by capabilities & demonstrated value as a result of experience. I once had a manager who, in his previous company was a VP of Engineering only to become a development manager in his next role - two steps back. I'd asked him why he made the move, this was when I learnt that work/life isn't just about the title. He restarted two levels down and in a short space of time, was back to being a Director of Engineering. I had a team leader once who was performing in the role of manager, but was so humble and patient that having the title did not bother him much. I have seen engineers who effectively remain doing similar activities for years, expecting a promotion by virtue of being doing the same job, even though the competencies haven't grown (e.g. influencing group/country/global teams, taking ownership, showing initiative) - so if you want a promotion, you need to earn it! I've also seen people who are content being a software engineer (with no aspirations of seniority/leadership), but who are expert at what they do, adding so much value, that the company provides enough incentives to keep the person happy (at times a brilliant software engineer could earn a higher salary, have indirect influence greater than his/her manager). At the end it does come down to personal motivations, and when the time does come around to being a financial/happiness constraint, then don't expect the company to help you - it's time for you to change!

My Timeline (Click here to see full table)