Tuesday, 21 March 2023

Sharing my writing example exercise from Amazon's interview process

Here is what I submitted to Amazon, as part of their interview process for an L7 Senior Engineering Manager role, in 2020. Depending on the role you're interviewing for, you will get a writing exercise - the one I chose was on Innovation - here's the ask:

Innovation
What is the most inventive or innovative thing you've done? It doesn't have to be something that's patented. It could be a process change, product idea, a new metric or customer facing interface – something that was your idea. It cannot be anything your current or previous employer would deem confidential information. Please provide us with context to understand the invention/innovation. What problem were you seeking to solve? Why was it important? What was the result? Why or how did it make a difference and change things?
Writing Guidelines
  1. Write in the style you would use to write a business whitepaper or essay and do not use bullet points, graphics, tables, charts or flow charts.
  2. Do not include any confidential or proprietary information from current/past employers.
  3. Remember as you write that the reader may not be familiar with specific technical terminology, corporate cultures, and scenarios.  Use language and descriptions in your response that enable readers to fully understand the situation.
  4. Please limit your response to 1-2 pages (no more than 8000 characters).

So since I was experimenting going back to being close to technical engineering, I decided to go deep into my past as an engineer, when I invented the Talking EPG:

More than ten years later, a talking interface finally made it to general availability:


Here's the document: Two pages in length, keeping to the written guidelines.


Monday, 20 March 2023

How I managed a technology budget ($100m) with 5+ CFOs, 3+ CEOs

In a previous post, I shared how I took on a challenging role as CTO for an online video streaming platform, with a business at the time that was going through rapid changes in leadership every year. In my 3.5 years stint, I would see my direct manager (CEO) change at least 4 times, their own ExCo board members change at least 2 times, and at least another 5 changes in CFOs and senior finance managers. Whilst these changes happened, my job was to still keep the technology platform running and my engineering team delivering on the product roadmap. 

Another spanner in the works was the ambiguity around having two engineering teams, led by two separate CTOs, building online video platforms, Delta and Sierra - a change that happened a year into my tenure after just landing the full-time CTO role (I'd stopped consulting, only to find myself in further uncertainty with a business merger and potentially more rounds of org-design changes). Delta was under my ownership, serving traditional TV customers with value-added internet services to access Live TV & Video on Demand (VOD) streaming. Sierra, was a pure internet, deep catalogue subscription video on demand service (like Netflix), built and run by a parallel engineering team, under different CTO, in another country, outside of South Africa.

So, with these many changes in executive leadership, at the top-level financial review - executives would naturally inspect why the business seems to be duplicating engineering efforts at building online video platforms, even though the two products serve a different set of customers, emerged at different timelines, using different set of technologies - but they both appear to "stream video" at the high level. So why is there duplication? Where are the costs going? How do we streamline engineering costs?

Enter the Financial Model & Technology Budget Commentary Document

The first order of business for me, being tasked with a turnaround challenge - was to review the existing cost components, identify the main cost drivers and derive a perpetual financial model based on the core business driver of growing monthly active users.

Starting in 2017, neither the technology team nor the finance managers could tell with a high level of confidence what the true costs of running the technology platform were, and ultimately what the cost of running a technology platform translated to a cost per user value, and ultimately how much from the profit margin this cost was eating into. My platform provided value-added-services for free, existing users were homed to a primary TV subscription package (satellite TV using a dish and decoder / Set Top Box (STB)) as main profile. Additional viewing profiles for online consumption through mobile phones, smart TVs, game consoles and web browsers, were offered at zero cost. Additionally, my platform served internet connected STBs with larger video catalogues for streaming on demand, this too, at zero cost to the subscriber.

Financial Model (brilliant, if I say so myself!)

So I created a detailed financial model that was able to show how to forecast a largely fixed cost investment, to operate in a variable cost domain. The largest cost components for video streaming is the infrastructure cost for the networking pipe (internet transit) and content delivery network (CDN) data costs. Since video consumes large amounts of data, these costs are covered by the streaming operator. CFOs prefer fixed cost accounting than variable costs because it helps manage their cashflows and forecasts better. In an on-demand video world, where customers come and go, or viewing behaviour varies by seasonality or during peak events, the experience is more bursty than constant. This is why video providers focus heavily on content personalisation and recommendation systems to drive stickiness and increase engagement. The more users watching video, the more data is consumed, the costs of data increases. The longer users remain on the platform, the more users are engaged and the more video data is consumed. All these point to increasing costs - but - at some point, the costs per user eventually decreases because the number of users increase, the cost per user eventually decreases. I created a financial model that showed how we could manage variable costs through a fixed cost model, by buying data wholesale upfront, using a minimum commit model with overflow-at-zero-cost into next year for unused data, and also negotiated year-on-year cost savings, since cost of data decreases year on year.

Using my financial model, I was able to show significant cost savings to the tune of R80m ($5-7m) in two years. I also, single-handedly negotiated significant costs savings with our primary transit links and CDN providers, as a result of my financial model forecasting. In year one, I negotiated a 1233% increase in CDN bandwidth and secured 71% decrease in out-of-band data costs, 50% decrease for inband per GB data costs - keeping my overall costs relatively flat to track a fixed expense. In year two, I negotiated additional cuts, reducing in-band data costs by a further 63%, out-of-band by 48%. In year three, I negotiated further deal reductions because of increased competition in both the transit-and-cdn space, landing on a further 20% reduction on inband and out-of-band costs. Over this period, I secured enough guaranteed capacity in the network that exceeded future growth targets, but kept most of the projected fixed costs inline. As a result, when I decided to leave the company, in my 3 years tenure, I drove internet costs down by 84% on a cost per user metric and 96%reduction on costs per GB per user metric.  Additionally, working with the same transit and CDN partners, I secured more than 7X improvement on overall internet throughput, reaching close to 1 terabits/sec on network capacity, which for the African continent, is not bad going indeed! In terms of storage, from 2017, I increased the data commitment by 2000% cumulatively year-on-year.

Sunday, 19 March 2023

An example of resource planning a 100 person technology team

Here's an example of how an engineering leader can go about managing headcount and resource planning to deliver on both the technology roadmap as well as business goals. For context, this is a resource plan for a 100-person size engineering organisation who had end-to-end responsibility for building and managing a live-streaming and video-on-demand online video platform "Delta". 

I was the CTO responsible for managing everything, pretty much a CIO in my own right, overseeing: 1\ Physical Infrastructure services: Datacentres, Networking, Compute, Storage; 2\ Internet backbone transit links; 3\ Cloud integrations and peering; 4\ Telecoms integrations; 5\ Software development & testing; 6\ Video broadcast & streaming infrastructure; 7\ CDN management; 8\ Personalisation & Content Recommender systems development; 9\ Agile Program Office; 10\ Technical Operations Command Centre offering 24/7/365 first/second/3rd line support; 11\ Enterprise & Solutions Architecture; 12\ Security, Anti-piracy, Risk & Governance; 13\ Platform Intelligence Dashboards & Analytics; 14\ Vendors.

Whilst I had my own technology strategy to deliver for the underlying platform to serve the scaling needs of the business, as we were aggressively targeting a growth phase - my teams had to also continue to build out product features and enhancements for all devices we supported (sometimes the apps varying by user experience depending on the device itself, e.g. Smart TV versus iOS vs Web app), we also had to deliver new business product offerings and deliver across group-wide projects and initiatives as shared goals.

With these challenges, an engineering leader must have a firm handle on the "resource" allocation. Sorry, agile folks - I myself cringe at the mere mention of "resources" - but heck, since joining Amazon, I was amazed of how natural the term "resources" is used by management. There are two camps, people who belong to "I am NOT a resource" and "I am a resource"!!

Anyway, I digress. As a leader, you have to keep an eye on your workforce. Based on on your existing headcount allocated to you, how do you best organise your people and plan around the various delivery demands placed on your team? How do you show your stakeholders where your resources are allocated? How do you motivate for additional headcount or escalate a change of priorities if you don't have the data that accompanies your story? How do you show that although you might have 100 people under your org, the capacity for doing project work is only 80 or less, because you need to account for management, team leads, shared service work and other internal projects? See below pivot summary table:

As tedious as it might be, an engineering leader has to do some "management" work - no matter how boring it might feel. This stuff is important. The finance folks need this information. The business owners need to have a view of what impact their projects have on your teams. 

A resource allocation plan can be a powerful tool - my advice - use it often, maintain it and let it become a key mechanism for you to have productive, collaborative conversations with your stakeholders.

Yes, I maintained my own list, accounting for each person in the org. 
No, I don't always support the notion that people are fungible resources, can be chopped and changed or "allocated" using numbers on a spreadsheet. I also don't believe the fractional numbers help either, but they are meant to be rough allocations to help with accounting - which I'm afraid is unavoidable - part of any software managers job really.

Below picture shows what goes into headcount planning / forecasting for the year, trying to keep the headcount to 100 people due to the hiring freeze. Just imagine, a 100-person size team responsible for building the full infrastructure and consumer applications for a streaming video platform that served the whole African continent (50+ countries)! Amazon has the Frugality leadership principle - we were frugal on people but wise on partnerships, using another of Amazon's LP - Invent & Simplify.



As an experienced engineering leader, my advice to software managers: If you're not maintaining a resource allocation plan, you're missing a vital tool of software management. Be careful!!

Saturday, 18 March 2023

Visualising a technology roadmap and year plan on just one page

Here's something that other engineering leaders might find useful: Showing all the work under one's ownership, a technology roadmap on a single page that nicely prints to A3 paper.

In this example, I capture everything that was important to me as the leader responsible for what was at the time, one of Africa's largest online video streaming platform (Live TV + VOD). I was responsible for the full platform end-to-end:
  • Physical Platform Infrastructure and Networking - Data Centres (Compute, Storage, Networking)
  • Cloud Services commercials & workloads
  • Software Engineering  - multi-platform, multi-device software development & testing
  • Enterprise & Solution Architects
  • Video Streaming Hardware & Software infrastructure integration & management
  • Recommendations & Content Discovery Engines - AI/ML scientists and engineering
  • Agile Program Management Office - Agile Specialists
  • Technical Operations & Integrations - Command Center & Mission Control 24/7/365 support
  • Security, Anti-piracy, Risk & Governance streams
How do I show on one page, the focus areas for the above teams in terms of our year plan, showcasing:
  • The themes that categorise the work so non-technical customers & stakeholders can understand
  • The cadence of releases for product feature delivery for product & marketing teams
  • The key KPIs our work drives - growth targets for monthly active users
  • The key events happening during the year that would put a strain on the platform load/stability
  • Show what the tech team will be producing month-on-month
  • The owners and points-of-contact for each work stream
So if you're a CTO or Head of Engineering responsible for an overarching technical platform, and you're not visualising your work in your own roadmap view, then IMHO, you're missing a trick. I used Excel to craft my roadmap because I was being scrappy - but it is possible to create such views if you invest in a solid product/program planning tool.

Click to enlarge: An example of a technology leader's roadmap / year plan

Friday, 17 March 2023

An example of a CTO's operating year's business plan

I'm continuing my series of posts sharing my work portfolio from my past as a technology executive (CTO) when I was directly responsible for a large engineering team (150+ people excluding partners). I approached this role as the next big challenge in my career aspirations. I gave myself a timebox of 3 years, after that, I'd intended venturing on to something else outside of the video world. I'm grateful for the experience as I left with learning all types of executive-management skills for managing business and technology operations. I also accomplished praiseworthy results in what would seem a short amount of time, in just 3 years. This shows how much intensity and density can be packed into a 3 year work experience. I know I would probably not see that much excitement and high-stakes challenges again for some time, even as I write this post in 2023, five years later, whilst working with Amazon's AWS...

Every CTO/Engineering Leader must write an Operating Plan

If you're a technology executive and you're not owning your very own technology operations business plan, in writing and not through high-level powerpoint slide decks, then you're missing a trick and you're missing out on a powerful asset and artefact, that IMHO, is essential for navigating your technology journey. Even before my time with Amazon, I used to write a lot, despite my employer not being a writing-documents oriented company, especially at executive leadership level. Personally, I needed a way to make sense of things, apply critical thinking and process my thoughts into a credible strategy for executing. I also needed to take my senior managers on my journey, lead by setting an example of thought-leadership through writing (because writing documents was unfamiliar to some managers). I also wanted to reach the software engineers on the floor, doing the day-to-day work, help them see the big picture and grasp how the work they do matters to the end goal. 

Another benefit of sharing your technology strategy in written form is to promote alignment and information sharing across many groups of stakeholders - customers, internal partners and external third-party primary partners. As a CTO for a specific business segment (online video), I had my own company's leaders as my primary stakeholders (CEOs, fellow C-Suite top team peers like Customer Acquisition, Retention, Marketing, Sales, Ops, Product, etc.), I also had to support other business' shared goals. In addition, I was part of a wider group structure of the parent group company, inheriting goals from the Group-CTO, Group-CIO, Group-CDO and Group-CISO domains. So the ever familiar problem of having multiple demands placed on a technology organisation. 

As a leader then, you need a mechanism for alignment on your execution plan for the year - this is usually an organisation's yearly operating plan. The aim is to show how you, as the engineering leader are accounting for all the priority goals you're taking on from your various customers and show how the work you plan to deliver, relates to achieving those goals and objectives. Additionally, the specific technology initiatives that you are taking accountability for yourself (like tech platform modernisation, addressing technical debt, etc.) remain nevertheless important for your org to deliver results and meet stakeholder expectations.

Below is one version of a year plan that I authored and used as a constant reminder for checking in and reflections on progress. Let me know if it helped you :-)

[Disclaimer: I write about my past work experiences, this post dating back to 2017-2020 referring to entities that no long exist today (in 2023). Previous mentions of such entities are widely in the public domain through news media outlets, press briefings, launch announcements, etc. I take time to ensure that nothing I share exposes commercially sensitive material. My intent is to share my learnings and experiences to teach and mentor people on a similar path to me; with the side benefit of showcasing my professional work portfolio to current and future prospective employers & head hunters, through my writing].