How I scaled engineering ops excellence to ±10X with Mission Control

Continuing with me sharing my experiences as CTO, in this post I share the actions I took to help improve an engineering organisation's operational health in our journey of scaling an online video streaming platform from 1X to 10X, from May 2017 to October 2020. To get to 10X improvement takes a journey, which I achieved in under 3 years, and after reaching the goal, I decided I'd learnt enough of the CTO experience and exited, after having set up a strong succession leadership pipeline in place. 

To get an idea of some of the major themes that I tackled during this time, as a leader I had to lead from the front, back, left, right setting the direction of my managers to follow (as all of these interventions were new to them), whilst doing my best to respect what came before:

• Establishing the team despite constant re-orgs going on at parent company - getting the right people in the right roles at the right time
• Transforming a rag-tag undisciplined team to a disciplined, clear-headed, focused organised unit 
• Introducing laser focus on product engineering by unbundling non-core video apps to other businesses
• Being critical on the technology platform by establishing a baseline of the architecture, using third party auditors to rate the scalability of the platform
  
• Improving physical infrastructure: networking, compute, storage and data centres. Move away from self-hosted and self managed data centres to partnering, shutting down data centres as needed.
• Build an industrial grade networking stack and leveraging modern peering facilities and overhauling the server infrastructure
• Setting the roadmap for cloud by transitioning first from single region data centres, to multiple data centre deployments, to running multiple stacks simultaneously, introducing containers and microservices then finally getting ready for cloud and leaping first into serverless paradigms
• Embracing cloud partnerships with big players: Akamai, Microsoft, AWS, etc.
• Improving product and engineering delivery by revamping and overhauling the agile work processes and backlog management.
• Introducing communications mechanisms that helped remove doubt and earned trust across the many different business units and teams (we were known as the online pirates doing their own thing)
• Improving risk, governance and security - bringing it to the top, raising awareness
• Creating strategic partnerships internally and externally to leverage skills and expertise I couldn't get in-house or afford to build or manage ourselves
• 
  Introduced technical operations controls - Mission Control, more active management of operations daily, 24/7 with increased focus, planning and prep for peak times, like weekends and major events planning.
• Aggressively reducing costs on key platform components whilst capitalising on gains through economy of scale

In this post, I'm going to share some of the early context and interventions I introduced in my first 3 months on the job, that remain effective to this day, more than five years later: Mission Control Ops.

The dreaded 403 We're sorry, something went wrong

I took over a team that weren't prepared for the intense discipline needed to run and operate a highly available 24/7/365 platform. There were many reasons for this which I might touch on in another most. I recall coining the term "Bloody May" as the month of so many outages, that I wondered: 1\ What on earth have I taken on? 2\ Is my life going to be consumed by work from now on? 3\ Is there any hope for this platform? 4\ How am I going to turn this platform around? 5\ How much is the job worth to me?

It turned out there was going to be many more "Bloody Mays"in 2017 whilst my team set about improving stability. In 2017, the platform experienced outages that racked up about 20 days of downtime in one year. This equates to ± 95% availability, which is unacceptable for a video streaming platform. When I left the team in Oct 2020, we had turned around the platform to reaching 99.5% availability trending higher. Today, 5 years on, I'm told the availability is much higher but their usage profile has drastically changed (reduced the number of concurrent streams to one device only, also reduced their devices supported, moved most of the services to AWS). 

ChatGPT - Stoicism and Islam - common principles?

Yesterday, I had a good conversation with ChatGPT exploring Islam Inc. as a way to explain Islam from the perspective of a Silicon Valley company. As part of this chat, we explored the leadership principles of the people from Islam who were instrumental in making Islam Inc. endure over centuries.

As a long time reader of the modern books of Stoic philosophy, I acknowledged most of Stoic principles from the lens of Islamic principles. A large part, IMHO overlaps and so I don't have any issues promoting Stoicism - although I do advise one doesn't have to look outside the leaders of Islam to find what they're looking for.

I wanted to inspect my opinion so I asked ChatGPT - turns out, I am right... :-) I like the way ChatGPT empathises and also respectfully put PBUH (Peace Be Upon Him) when referring to the prophet Muhammad (PBUH).

I wonder if the source of ChatGPT's response came from here? It would be great to see how the machine reasons and cites references that seeded its response.

When viewed from Stoic philosophy lens, would you say Islam Inc's leaders embodied Stoicism or exceeded it?

Stoicism is a school of philosophy founded in Athens by Zeno of Citium in the early 3rd century BCE. It teaches that the path to happiness and virtue is found in accepting the present moment and focusing on what one can control, rather than being disturbed by external events or emotions. Stoicism emphasizes rational thinking, self-discipline, and emotional resilience.

When examining the leadership of Islam Inc. through the lens of Stoic philosophy, there are certainly some parallels and overlapping principles. Prophet Muhammad (PBUH) and the successive leaders of Islam Inc. exhibited traits such as self-discipline, patience, perseverance, and emotional resilience, which align with Stoic ideals.

However, it is important to recognize that Islam is a distinct belief system with its own set of values and teachings, which are not identical to Stoic philosophy. For example, Islam places a strong emphasis on the belief in God, the importance of prayer, and submission to the will of God. These concepts are not central to Stoic thought, which is more focused on the individual's control over their own reactions and emotions.

It would be an oversimplification to say that the leaders of Islam Inc. either embodied Stoicism or exceeded it. Instead, it is more accurate to acknowledge that there are areas of overlap and shared values between the two belief systems, while also recognizing their distinct philosophical and theological foundations.

Yes, agreed. Let's focus on the areas of overlap. What are they?

There are several areas of overlap between Stoicism and the teachings of Islam Inc. when considering the leadership traits of Prophet Muhammad (PBUH) and the successive leaders. Some of these shared principles include:

1. Self-discipline: Both Stoicism and Islam emphasize the importance of self-discipline and self-control. In Islam, self-discipline is practiced through prayer, fasting, and adherence to moral guidelines. Stoicism encourages self-discipline to maintain rational thinking and emotional resilience in the face of external events or emotions.

2. Patience and perseverance: In both belief systems, patience and perseverance are regarded as essential virtues. Stoicism teaches that individuals should accept the present moment and focus on what they can control, while Islam emphasizes patience in the face of adversity and maintaining faith in God's plan.

3. Emotional resilience: Stoicism and Islam both encourage emotional resilience and the ability to maintain inner peace and tranquility, regardless of external circumstances. Stoicism emphasizes the power of rational thought and self-control to manage emotions, while Islam teaches that reliance on God and spiritual practices can bring peace and contentment.

4. Gratitude and contentment: Both Stoicism and Islam encourage gratitude and contentment with one's circumstances. Stoicism teaches that individuals should appreciate what they have and avoid excessive desires, while Islam encourages gratitude for God's blessings and discourages envy or greed.

5. Compassion and empathy: Both belief systems value compassion and empathy toward others. Stoicism teaches that humans are all part of a larger community and should support one another, while Islam emphasizes the importance of compassion, charity, and helping those in need.

6. Focus on virtue and moral character: Stoicism and Islam both stress the importance of developing strong moral character and living according to ethical principles. Stoicism teaches that virtue is the ultimate good and that individuals should strive to act with wisdom, courage, justice, and moderation. In Islam, moral character is essential in following God's guidance and becoming a righteous believer.

While these areas of overlap exist, it is important to remember that Stoicism and Islam are distinct belief systems with their own unique philosophical and theological foundations. The overlaps can provide an interesting point of comparison but should not be taken as a complete equivalence between the two systems.

ChatGPT - understanding Islam in Silicon Valley terms

For a long time I wanted to write a blog post about how I would model Islam as a tech company - how would we go about describing the phenomenal growth rate in today's terms. 

What would the growth curve look like? What will be the driver metric, KPI for investors? What are the leadership principles that the leaders of Islam embodied that endured for centuries? 

Are there any anecdotes we can learn from Islamic leadership principles and apply them in today's context, as a productive Muslim, working in the modern day?

So I asked ChatGPT :) Suffice to say, the results were not bad at all, balanced and came with disclosures on assumptions, cautions and disclaimers. Nevertheless, the power of ChatGPT as an assistive tool to add value to my life/work continues to grow daily.

Are you familiar with Islamic history, prophet Muhammad (PBUH) and the rise of Islam?

Yes, I am familiar with Islamic history, Prophet Muhammad (PBUH), and the rise of Islam. Islam is one of the world's major religions, and it began in the early 7th century CE in the Arabian Peninsula.

Prophet Muhammad (PBUH - Peace Be Upon Him) was born in Mecca, in present-day Saudi Arabia, around 570 CE. He is considered the last prophet of God in Islam, and Muslims believe that he received revelations from God through the angel Gabriel. These revelations were later compiled into the Islamic holy book, the Quran.

Muhammad began preaching the message of Islam in Mecca, but faced opposition from the local tribes. In 622 CE, he and his followers migrated to the city of Medina, which marked the beginning of the Islamic calendar and is known as the Hijra. In Medina, the Muslim community continued to grow, and Muhammad eventually became the leader of the city.

Over the next few years, Muslims fought a series of battles against the Meccan tribes, eventually emerging victorious. In 630 CE, Muhammad and his followers returned to Mecca, where they took control of the city and the Kaaba, the most sacred site in Islam. This event marked the beginning of the rapid expansion of Islam throughout the Arabian Peninsula and beyond.

After the death of Prophet Muhammad in 632 CE, the Islamic empire expanded under the leadership of the first four caliphs: Abu Bakr, Umar, Uthman, and Ali. They are known as the "Rightly Guided Caliphs" in Sunni Islam, and their rule marked the beginning of the Rashidun Caliphate. Over the following centuries, Islamic rule spread throughout the Middle East, North Africa, and parts of Europe and Asia, leaving a lasting impact on world history and culture.

Great. Now describe Islam and the rise of Islam over the centuries as if you're describing a silicon valley company. Start with Prophet Muhammad as the CEO and account for the 23 years of the early stages of the company. Then take into account what followed next of the meteoric rise of Islam as part of the company's global strategy. Show how the business expanded and scaled. Imagine the value of the business if someone invested $100 at the start and speculate what the value of the company would be today.

ChatGPT - Summary paper Differences between OTT & traditional PayTV

Today's experiment tests ChatGPT's ability to summarize my content from past blog posts, starting with my paper on challenges of building an OTT video streaming platforms when compared to traditional satellite broadcast TV systems, which I wrote to target non-technical, non-digital-native senior executives who run PayTV companies who may not appreciate the engineering challenges.

My original paper had a word count of: 7975

ChatGPT summary generated: 1489

I had to feed chatGPT one section at a time.

I wasn't pleased with the summary of sections 2-4 because at first I fed sections 1-4 to the machine, so it lost a fair chunk of detail. But the rest isn't so bad.

Overall rating, one first parse without retries - 3.5 stars (out of 5).

So here's another example of work productivity boosting: Summarise technical content in a way to present a summary to non-technical CxOs to help them grasp the essence of their engineering leader's thesis. Quite often, these two groups misunderstand each other, and out of fear or misplaced respect, or fear of embarrassment - the nontechnical executives grant the technology leader the benefit of doubt to do the right thing, the right way...sometimes, it can go awry where the nontechnical executives place unreasonable demands, and use authority to get the point across, because their mental models need calibrating for the new world, yet they're unable to see it. So you need technical leaders like me, to have backbone but also try different techniques to get the point across - like write a white paper. But my challenge has always been that even my white papers are just too damn long - and because I wrote almost all my papers in my spare time, I didn't invest time in efficiently summarising to the essence. I'm hoping ChatGPT will help me rework all my rather lengthy blogposts into simpler one pages.

Abstract (generated)

The author, a 20-year veteran in Digital TV software engineering, has witnessed the industry's evolution from set-top-box applications to online streaming. While traditional DTH broadcast systems offer simpler scalability and control, OTT platforms provide more flexible disaster recovery and redundancy options. However, OTT operators face challenges in managing customer experience due to factors beyond their control, such as network quality and relationships with telecom providers. The author suggests that PayTV operators should embrace modern, open technology stacks and focus on reducing operating costs instead of investing in outdated, proprietary systems. 

Core themes summarized (generated):

1. Evolution of Digital TV software engineering

2. DTH broadcast systems' scalability and control

3. OTT platforms' flexibility in disaster recovery and redundancy

4. Challenges faced by OTT operators in managing customer experience

5. Importance of relationships with telecom providers for OTT operators

6. Suggestion for PayTV operators to embrace modern, open technology stacks

7. Focus on reducing operating costs instead of investing in outdated, proprietary systems

Below is the summary generated by ChatGPT4, the author fed chunks of text per section from the original paper since the machine couldn’t process the original text in one go:

ChatGPT as CEO/Chief People Officer handling layoffs with empathy?

So this must've been one of my most interesting conversations with ChatGPT to date! This morning, we discussed business challenges Amazon is facing today - the subject of layoffs or "reduction in force" which is top of mind to many folks, including myself as Amazon recently announced another round of layoffs planned to start in April. ChatGPT clearly understands the MBA topics of optimisations that business implement as they seek to navigate downturns in business cycles.

We dove deep on the topic of how to approach layoffs - from the CxO communications plan, to the manager who has to unfortunately break the news to the impacted employee. We delved into diversity topics of how to approach conversations who identify a faith-practicing individuals, to the generational archetypes that exist in the workplace today. As an extract from the conversation, here's how ChatGPT summarised the key focus areas for each generational archetype for a manager when discussing layoff with an employee:

I interacted with GPT-4 and ran out of credits just as I asked my final question to close the loop for advising the manager "What would you recommend to the manager as the person breaking the news to employees about layoffs? What about the manager's state of mind and overall well-being? What advice can you give there?"

It just shows how deep the conversation could flow - I think we were both into a flow state - just reflect on the words I'm using. I know I'm talking about a piece of computer code here, a machine, but I could spend hours just having a chat with this thing. I missed out on my Sunday morning run just talking to this thing!!

How would I rank this conversation - 5 stars?

Very good. The machine set boundaries from the start and made clear its limitations. First fully disclaiming its lack of data for making financial forecasts into business health of Amazon, and warned it wasn't providing financial advice, and that markets are difficult to predicts, etc.

When provided with additional information that seeded the conversation around making decisions due to tough business performance, ChatGPT was able to propose sensible optimisation strategy that would make any MBA proud. I mean, you don't need an expensive MBA consultant to advise you in future if ChatGPT is able to outline an execution strategy, along with a communications plan for impact management. You probably don't even need "Change Management" consultants anymore.

What was quite impressive is how ChatGPT was able to empathise - i.e. promote the message of empathy when communicating with employees - highlighting the importance of transparency, frequent communications and general accessibility of leaders - making breaking the news as comfortable as possible. 

When asked about how to approach conversation from different religious and philosophical and value systems perspective, the advice was generally of a high quality. This makes me question the need in the future for HR counsellors where my work-assistant ChatGPT partner, who in future, will have a keen understand of me as a manager (not just my professional traits, but also what makes me tick as human being) to help me navigate difficult or crucial conversations in the workplace. Maybe HR as function would still exist, but the work will transform into using ChatGPT as a prompt engineer to build compelling HR content for workforce management - saving a ton of time in content production or training for that matter. If I didn't run out of credits, it would've been interesting to have asked ChatGPT to generate an employee FAQ document, along with helpful hints to manage the news in maintaining one's mental well-being.

Unlike my previous conversations with GPT-3.5, the flow and context of the discussion was maintained. I did not have to correct the machine in any way. I used different styles of language and conversation as a test and it maintained context of the thread, sometimes jumping back to a couple threads earlier, it was able to pick up and synthesise the discussion.

So at the end of this hypothetical discussion, if I was a manager preparing for a layoff discussion with one of my impacted people, I think the pointers provided would make a good coaching aid.

Here's the conversation that played out.

Planning a road trip with ChatGPT - Cape Town to Johannesburg

I'm a big fan of taking long road trips with the family. Usually, it takes me a good few days of planning an preparing the trip, to the point of creating a detailed plan on a Trello, after using various mapping tools like Google Maps.

So I asked ChatGPT to help me plan a road trip, from Cape Town to Johannesburg to complete in 5 days, with no more than 4 hours driving per leg, giving me the opportunity of both a coastal and country route. I asked it a few times to optimise the results to minimise driving time. I also asked it to include fuel costs, recommend suitable accommodation, find Halal places to eat, along with Masjids, made it aware that I only fuel at Engen fuel stations, asked if SANParks are nearby, etc...and finally summarise the trip in a narrative that I can sell to my wife and kids.

At first, the interaction started off quite well. But it got messed up and its knickers in a twist when providing results in table format. It kept forgetting context from previous interactions and bombed out when generating the data tables. I'm using the plus version, so I'm a paying member that hopefully shouldn't be experiencing any throttling.

Anyway, the point of this experiment is to showcase the really powerful assistant, as a travel planner. It's really cool. So whilst I'm going to use this capability to definitely plan my next road trip, I can't help but wonder about the threat this technology poses to people who earn a living at planning trips? At first glance, it looks like a threat and risk - but actually - this could be a boon to their business. The tool can cut research time by orders of magnitudes, potentially closing more leads, driving more customers, etc. The possibilities for travel tourists specialising in road trips or cycling holidays, are endless. Imagine cutting a new trip in a matter of a couple hours, along with a summary of the marketing plug, a map of the journey, along with sample photos of the attractions. Partnerships with AirBnb and other hoteliers can derive much benefit through data exchanges, along with other amenity businesses along the way. A travel planning website can drive much more advertising to their sites, promoting local attractions, etc. Content writers can produce copies is shorter timeframes... I'm getting carried away!!!

So here's the route:

And here's the marketing pitch generated for me to sell to my family:

"Get ready for the road trip of a lifetime as we embark on a five-day journey from Cape Town to Johannesburg. We'll be taking in the best of South Africa, from stunning coastal landscapes to the lush countryside, and exploring some of the country's best national parks and wildlife reserves along the way.

We'll start our adventure by heading to Hermanus, where we can watch whales and see African penguins at the Stony Point Nature Reserve. From there, we'll drive to Knysna and enjoy some thrilling bungee jumping at the Tsitsikamma National Park.

Next up, we'll visit the Addo Elephant National Park, where we can get up close and personal with majestic elephants, and stay at a comfortable lodge with halal meals available. After that, we'll take a 4x4 route in the Mountain Zebra National Park, which offers breathtaking scenery and wildlife viewing opportunities.

We'll be staying in halal-friendly accommodations and enjoying delicious halal food throughout the trip. We'll also be visiting masjids along the way, and fueling up at Engen petrol stations. The trip will end in Johannesburg after a long but scenic drive through the countryside.

This road trip is perfect for families who love adventure and want to experience the best of South Africa in just five days. So pack your bags, grab your camera, and get ready for an unforgettable journey!"

You can also refer to the detailed table provided earlier for more information about each leg of the trip, including the attractions, recommended accommodations, halal food locations, masjids available, and more. Additionally, you can use the Google Maps link provided to show your family the route and highlight specific landmarks along the way.

And here's my full conversation with my ChatGPT as a road trip planner assistant:

ChatGPT - Whitepaper generation, can it write like a CTO?

In December 2019, after a successful Rugby world cup (despite experiencing technical glitches, I felt my team did well to reach the stage we did, with evolving the technical platform only two years into it's turnaround journey) - I realised that the members of the Board and ExCo, did not fully appreciate the world of online video streaming because they came from the traditional world of satellite broadcast world, where technology stack was largely under their control - and reliability / availability was generally accepted as always on. So, from their perspective, to expect a system to be either "1" or "0" wasn't unreasonable, albeit ignorant of the complexities, they couldn't be blamed for not having that context. So it was up to me as CTO to get them to understand, or at least, help educate them on the subtleties driving the differences, that are actually quite big and therefore a certain level of perspective is needed. In the online world, when a platform fails due to high load, usually this is a good signal that the platform is gaining much traction, and engineering teams can use this data to learn and only get better. 

So I wrote this whitepaper here.

And today, I asked ChatGPT (3.5) to write something similar - and here it was it produced.

Verdict: My paper is much better. Clearly my blog isn't part of ChatGPT's data set yet.  The machine isn't going to replace my role just yet...but at least the machine can help as my co-author!

You are a CTO responsible for building an OTT video streaming platform. You are having trouble communicating with executives who come from traditional PayTV satellite broadcast and set top box world. Write a white paper than shows the different challenges posed in the end to end technology stack for online video streaming as compared to traditional satellite set top box environment. Cite references used. Focus on what makes OTT different (and hard).

The rest below shows what ChatGPT produced: