Showing posts with label Professional. Show all posts
Showing posts with label Professional. Show all posts

Saturday, 14 September 2024

How I used ChatGPT o1-preview to build a rough interactive software engineering resource forecasting model

Seed Prompt for tool (click here to play with it): I am a software engineering manager. A large part of my job is understanding the engineering resource utilization for my engineering team. I manage multiple engineering teams of varying sizes from 5 people to 20 people in a team. When estimating software projects, understanding availability of productive engineers is important for planning purposes. Help me build an interactive resource forecasting tool, that can be fully self-contained within browser app, embedded in a google blogspot page. The tool must include inputs for: 1\ Funded team size; 2\ Current Available engineers (i.e. actual current size of team). The tool must account for the time it takes for hiring new engineers, ramp-up time, learning, etc. before they can be deemed as productive for committing to project work. The tool must show a table the resource plan over 52 weeks in the year, and include a visual graph to show how the team size grows to meet funded size. Research the best methods in software engineering management, and create the tool that can help software managers have a good model for resource utilization. The tool can store information in the local user browser storage as needed. Remember, I need the full code to be fully self-contained so I can run the tool from within a google blogspot page.

Prompt for this article: Great I think we have good baseline version to start with. Now using your PhD capabilities, write up an accompanying blog post that details the model, the thinking behind it, and call out the limitations using a critical eye, suggesting improvements for the next version.

Prompt for Image using Dall-E: "An isometric illustration of a diverse engineering team collaborating in a modern office space. They are gathered around a large table filled with charts, graphs, and laptops, discussing and planning. Behind them, a wall-sized calendar and a whiteboard display timelines, Gantt charts, and upward-trending graphs. Include elements like gears and clock icons subtly integrated into the design to symbolize engineering and time management. Use a professional, clean art style with vibrant yet muted colors to convey a serious but engaging atmosphere.

Engineering Resource Forecasting: A Comprehensive Model for Team Planning

By ChatGPT o1-Preview (took about 4 hours to get the tool to version 1)

Effective resource planning is crucial for engineering teams aiming to meet project deadlines, optimize productivity, and manage budgets efficiently. The Engineering Resource Forecasting Tool is designed to help managers and team leads forecast their engineering resources over a year, considering various real-world factors such as hiring rates, attrition, ramp-up times, efficiency, leave days, and public holidays.

In this blog post, we'll delve into the model behind the tool, explore the thinking that shaped its development, critically analyze its limitations, and suggest improvements for future iterations.


Understanding the Model

1. Objectives of the Tool

The primary goals of the Engineering Resource Forecasting Tool are to:

  • Forecast Team Size: Predict how the engineering team will grow or shrink over time.
  • Calculate Effective Capacity: Determine the actual productive capacity of the team, considering factors that reduce availability.
  • Assist in Planning: Provide insights for hiring strategies, resource allocation, and project scheduling.

2. Key Components of the Model

The model incorporates several essential elements:

  • Funded Team Size: The maximum number of engineers that the budget allows.
  • Current Available Engineers: The starting point for the forecasting, representing the existing team.
  • Hiring Rate and Time: The rate at which new engineers are hired and the average time it takes for them to join.
  • Ramp-Up Time: The period new hires require to reach full productivity.
  • Attrition Rate: The expected percentage of engineers who will leave the team annually.
  • Efficiency Factor: The proportion of time engineers spend on productive work.
  • Leave Days and Public Holidays: Time when engineers are not available due to vacations and holidays.

3. Model Calculations

a. Effective Engineers

The number of effective engineers is calculated weekly by adjusting the total number of fully ramped-up engineers for efficiency and availability:

Effective Engineers = Total Ramped-Up Engineers × Efficiency Factor × (Weekly Available Days / 5)

b. Weekly Available Days

Engineers aren't available all five working days each week due to leave and public holidays:

Weekly Available Days = 5 - ((Annual Leave Days + Public Holidays) / 52)

c. Attrition Modeling

Attrition reduces the team size over time:

  • Weekly Attrition Rate:

Weekly Attrition Rate = Annual Attrition Rate / 52

  • Cumulative Attrition: The total number of engineers who have left the team up to a given week.

d. Hiring and Ramp-Up Process

  • Hiring Pipeline: Represents engineers who have accepted offers but haven't started yet.
  • Ramping Engineers: New hires who are in the process of ramping up to full productivity.

e. SDE-Weeks and SDE-Days

  • SDE-Weeks: Sum of effective engineer contributions per week within a month.
  • SDE-Days: Sum of effective engineer contributions per day within a month.

These metrics provide a more granular view of the team's productive capacity.


Thinking Behind the Model

1. Realistic Representation

The model aims to mirror real-world scenarios by considering:

  • Time Delays: Hiring and ramp-up times introduce delays between initiating a hire and gaining full productivity.
  • Non-Productive Time: Leave, holidays, and non-productive activities reduce actual working time.
  • Attrition Effects: Engineers may leave the team, impacting capacity and necessitating replacement hires.

2. Balancing Simplicity and Complexity

While the model incorporates multiple factors, it maintains usability by:

  • Using Average Rates: Distributes leave, holidays, and attrition evenly over the year for simplicity.
  • Assuming Linear Processes: Hiring and ramp-up are modeled as linear processes without accounting for fluctuations.

3. Providing Actionable Insights

By visualizing the forecast through charts and tables, the tool helps managers:

  • Identify Hiring Needs: Determine the hiring rate required to meet team size goals.
  • Plan for Attrition: Anticipate capacity reductions due to attrition.
  • Adjust Strategies: Modify inputs to explore different scenarios and their outcomes.

Limitations and Critical Analysis

Despite its usefulness, the model has several limitations:

1. Uniform Distribution Assumptions

  • Leave and Holidays: The model assumes that leave days and public holidays are evenly distributed throughout the year. In reality, leave may cluster around certain periods (e.g., summer vacations, year-end holidays), leading to temporary drops in capacity.
  • Attrition Timing: Attrition is modeled as a steady rate, but actual departures may occur unpredictably and in clusters.

2. Lack of Granular Ramp-Up Modeling

  • Binary Productivity: The model assumes engineers are either ramping up or fully productive after the ramp-up period. It doesn't account for gradual increases in productivity during ramp-up.
  • Individual Variations: All new hires are treated identically in terms of ramp-up time and efficiency, ignoring individual differences.

3. Hiring Constraints and Real-World Challenges

  • Hiring Rate Feasibility: The model calculates the hiring rate needed to meet targets but doesn't consider the practical challenges of achieving that rate, such as market conditions or recruitment resource limitations.
  • Candidate Pipeline Variability: It doesn't account for fluctuations in candidate availability or acceptance rates.

4. Simplified Efficiency Factor

  • Static Efficiency: The efficiency factor is constant over time, not accounting for potential improvements or declines due to training, process changes, or team dynamics.
  • Team Composition: It doesn't differentiate between engineers with different skill levels or specializations.

5. Absence of Project Dependencies

  • No Task Modeling: The model focuses on resource capacity without linking it to specific project tasks or timelines.
  • Overtime and Workload Peaks: It doesn't account for periods when engineers might work overtime or when workload exceeds capacity.

6. Economic and External Factors

  • Budget Changes: The funded team size is assumed constant, ignoring potential budget cuts or increases.
  • External Events: Unforeseen events like economic downturns, pandemics, or natural disasters are not modeled.

Suggestions for Improvement

To enhance the model's accuracy and applicability, future versions could consider:

1. Incorporating Seasonal Variations

  • Variable Leave and Holiday Distribution: Model leave and public holidays based on historical patterns to reflect periods of low and high availability.
  • Attrition Peaks: Adjust attrition rates to account for known periods of higher turnover.

2. Enhanced Ramp-Up Modeling

  • Gradual Productivity Increase: Implement a ramp-up curve where productivity increases incrementally over the ramp-up period.
  • Individualized Ramp-Up: Allow different ramp-up times for different roles or experience levels.

3. Realistic Hiring Constraints

  • Recruitment Capacity Limits: Introduce maximum hiring rates based on recruitment team capacity.
  • Candidate Pipeline Dynamics: Factor in time-to-fill variations, candidate drop-off rates, and competition in the job market.

4. Dynamic Efficiency Factor

  • Efficiency Over Time: Allow the efficiency factor to change over time, reflecting process improvements or team fatigue.
  • Role-Based Efficiency: Differentiate efficiency factors for various roles or seniority levels.

5. Integration with Project Planning

  • Task-Level Planning: Link resource capacity to project tasks and milestones to assess whether capacity meets project demands.
  • Risk Modeling: Incorporate risks and uncertainties that could impact timelines and capacity.

6. Economic Scenario Planning

  • Budget Flexibility: Allow for adjustments in the funded team size based on potential budget changes.
  • External Shocks: Introduce scenarios for external events that could affect hiring, attrition, or productivity.

7. User Input Flexibility

  • Custom Time Frames: Enable users to adjust the time frame of the forecast beyond a single year.
  • Data Import: Allow users to import historical data to better tailor the model to their organization's patterns.

Conclusion

The Engineering Resource Forecasting Tool provides a solid foundation for planning and visualizing engineering team capacity over time. By accounting for critical factors like hiring, ramp-up, attrition, efficiency, leave, and holidays, it offers valuable insights for managers.

However, like any model, it simplifies complex real-world dynamics to remain practical and user-friendly. Recognizing its limitations is essential for interpreting the results accurately and making informed decisions. By incorporating suggested improvements, future versions can offer even more robust and precise forecasting, aiding organizations in navigating the challenges of resource planning.


About the Author

[Your Name] holds a PhD in [Your Field] and has extensive experience in project management, resource planning, and organizational development. Passionate about leveraging data-driven models to solve practical problems, [Your Name] contributes to advancing tools and methodologies that enhance operational efficiency.


Note: The Engineering Resource Forecasting Tool is intended as a planning aid and should be used in conjunction with professional judgment and other planning tools. Always consider your organization's specific circumstances when making staffing decisions.

Sunday, 1 September 2024

The Jenga of life

Jenga metaphor #thismightnotwork

It's been ages since I last shared a metaphor on this blog. This one hit me during my morning walk today, 8th August 2024. As I was trudging through the early morning mist, my brain made an odd connection: life is kinda like a game of Jenga.

You know Jenga, right? That game where you're trying to build the tallest tower possible by yanking out blocks and stacking them on top. It's all fun and games until someone knocks the whole thing over. But today, I saw it differently.

The Baseline: Your Value System

Picture this: you start with a perfect Jenga tower. All the blocks neatly stacked, everything solid and stable. That's you. That's your life, your values, everything you stand for. It's rock solid, built on all the principles you've picked up over the years.

The Dangerous Dance of Compromise

Now, here's where it gets interesting. Every time you play Jenga, you're taking a risk, right? You pull out a block, the tower gets taller, but it's wobbling more. In life, it's kinda the same deal:

  • Maybe you tell a little white lie. No big deal, right? But they start to add up...
  • Or you cut corners on a business deal. Just this once, you tell yourself.
  • Perhaps you're racking up some debt. One credit card becomes two, becomes three...
  • Or you let slide something that goes against your values. It's just a small thing, a papercut. But man, those papercuts sting after a while.

Every time you look the other way, every time you compromise, you're pulling out another block. Your tower might be getting taller, but it's getting shakier too.

The Inevitable Collapse

We all know how Jenga ends - with a crash and blocks all over the floor. Real life isn't always that dramatic, but the consequences can be way worse. All those little compromises? They add up:

  • Those white lies might snowball into you being known as "that guy who can't be trusted".
  • That shady business deal? It might come back to bite you in the butt, big time.
  • And that debt? Before you know it, you're drowning in it.
  • As for those "minor" infractions? They chip away at your self-respect, bit by bit.

The Lesson: Handle with Care

So what's the point of all this Jenga talk? It's pretty simple: be careful with your values, folks. Every choice you make, everything you do, it's like moving a block in your personal Jenga tower. Some moves might seem like no big deal at the time, but they all add up to how stable (or unstable) your life is.

This doesn't mean you should never take risks or change your mind about stuff. Heck, growing as a person often means shaking things up a bit. But it does mean being careful about the choices we make and thinking about where they might lead us down the road.

Building a Stable Tower

So, how do we use this Jenga metaphor to build a life that doesn't topple over at the slightest breeze?

  1. Know your foundation: Take some time to think about what really matters to you. What are the blocks at the bottom of your tower?
  2. Think before you pull: Before you do something that goes against your values, ask yourself if it's really worth it. Is that short-term gain worth making your whole tower wobble?
  3. Shore up your base: Work on making your core values stronger. Read books, hang out with people who inspire you, practice what you preach in the little day-to-day stuff.
  4. It's okay to rebuild: If you realize you've compromised too much, don't be afraid to start over. Sometimes you gotta go back to basics to build something that'll last.
  5. Play the long game: Remember, life isn't a race to build the tallest tower as fast as you can. It's about building something that'll stand up to whatever life throws at it.

As we're all fumbling our way through this game called life, let's keep this Jenga thing in mind. Every decision, every compromise, every value we ignore - they're all blocks in our tower. Handle 'em carefully, think about your moves, and try to build a life that stands tall not just in height, but in being rock-solid and true to who you are.

So, what blocks are you moving around in your life's Jenga tower? And the million-dollar question: how stable is your foundation? Something to chew on, isn't it?

Sunday, 14 July 2024

What are you willing to walk away from?

Table of Contents

Introduction

These are loaded questions of course - there's so much that goes on in one's life, there are many dimensions to life - so we need to be more specific. When I talk about walking away, it's about the career and professional dimension - one's work, job, company, vocation, etc. I found that asking myself this question surfaced at almost all stages of my career. Whether I was walking away from my first choice of degree to study, or walking away from three job offers for my first engineering job, or leaving a role before being promoted, or as recent as 2020 when I decided to walk away from C-level back to mid/senior-management, "What am I willing to walk away from?" has been my resonant, background question driving my self-reflections. Earlier this month, call it a moment of inspiration, these words came to my mind:
Sometimes it is okay to experiment with new experiences long enough to realise the experience is not worth the reward you originally imagined, and that’s perfectly okay

The Concept of Walking Away

Being willing to walk away from your work / company / profession / business is not easy. Being prepared is even harder and more important. Asking the question at regular intervals can be somewhat uncomfortable and stressful, so timing is also important. I previously shared my experiences in software engineering, where I advised being careful to jump ship too early, and also included that as learning in my failure resume. IMHO, one needs to create some stability in the role for at least 18-24 months before moving on - I still stand by this. But that's not to say, frequent inspection of your current role and desired trajectory should be put off until 18-24 months. It's good to inspect, but don't do anything rash or hasty. Know what you stand for, have a plan or at least have a clear picture of your desired future state, being intentional about it. Answering the question becomes quite obvious when your realize the current track you're on is actually deviating from the track you intended - and if your intentions are still valid, then being willing to walk away is a very relevant question and important to resolve.

Insights from 25 Years of Professional Experience

Each career move has been a strategic decision to broaden my skill set, take on new challenges, and align my work with my evolving professional goals. This diverse experience has made me a more well-rounded professional, able to bring unique perspectives to each new role.

The career "ladder" is often more complex than it appears. Embracing a non-linear or "squiggly" career path can lead to a rich diversity of experiences. It's important to focus on personal growth and skill development rather than just titles or compensation.

Taking a step in a different direction isn't necessarily a setback. It can lead to growth in unexpected ways, though it's important to be prepared for the challenges that come with change. Every new path is an opportunity to learn and evolve professionally.

While I've made personal choices about the industries I work in, I respect that every professional must make their own decisions based on their individual values and circumstances.

While my current role may not have the same title or compensation as my previous position, it offers unique learning opportunities and the chance to contribute to groundbreaking projects, which I find incredibly fulfilling. I've found that a balanced approach to work often leads to increased productivity and creativity. My experience with alternative work schedules has taught me the value of focused, efficient work time balanced with adequate rest and personal development.

The pace of the tech industry can be relentless, but it's important to find a sustainable rhythm. It's okay to pause, reassess, and adjust your course as needed. Your career should serve your life goals, not the other way around.

Personal Experiences with Walking Away

2020: From CTO to Senior Manager

In 2020, I made a significant career transition, moving from a CTO position in an industry where I had spent 20 years building expertise to a senior management role in the cloud computing sector. This decision was driven by a desire for new challenges and growth opportunities. I made the decision to prioritize new experiences and growth opportunities over short-term financial gains, recognizing that long-term career satisfaction was more valuable to me at this stage.

The transition involved relocating to a new city with my family and starting fresh in a larger organization. It was a chance to step back from the complexities of C-level responsibilities and re-engage with the technical aspects of the work that had initially drawn me to the tech industry.

While the C-level role offered valuable experiences, I found myself seeking different challenges and a work environment more aligned with my evolving personal and professional values. I was excited by the opportunity to work on cutting-edge computing technologies, which aligned more closely with my personal interests and career goals at that time.

2019: From Consulting to CTO

In 2019, I transitioned from consulting to a CTO role, seeing it as an opportunity for professional growth and increased responsibility. While consulting offered flexibility and valuable experiences, I was eager to have a more direct impact on a company's strategic decisions and operations.

The move to a CTO position presented new challenges and learning opportunities. It meant exchanging the diverse project exposure of consulting for deep engagement with one organization's goals and challenges. This transition allowed me to apply my broad experience to a focused leadership role, accountable for significant business outcomes.

Although the consulting role offered certain advantages, including a flexible schedule and competitive compensation, I was drawn to the CTO position by the prospect of shaping company strategy and directly influencing outcomes. This move aligned with my career goal of expanding my leadership skills and taking on greater responsibility in driving business success.

2013: Senior Manager to Freelance Consulting

In 2013, after successfully leading a senior management role where I contributed to significant business and engineering advancements, I recognized an opportunity for broader professional growth. My experience in Satellite TV and embedded software applications had provided me with deep domain expertise, but I was eager to expand my knowledge into emerging areas like mobile and cloud streaming technologies.

Sensing that my professional growth potential in my current role was plateauing, I made the strategic decision to transition into freelance consulting. This move allowed me to diversify my experience and gain exposure to a wider range of business challenges and technologies.

The shift to consulting opened up numerous opportunities. Over the next five years, I engaged in various high-impact projects, including launching a video streaming business in Africa and leading major corporate-wide programs. This experience provided invaluable exposure to C-level decision-making processes and broadened my understanding of different aspects of the media and technology industries.

Consulting also offered the flexibility to structure my work schedule in a way that optimized my productivity and allowed for continuous learning. I found that this balance enhanced my ability to deliver high-quality results for clients while also pursuing personal development.

As my consulting career progressed, I began to seek new challenges that would allow me to have a more direct, long-term impact on an organization's strategy and operations. This desire for deeper engagement and accountability in business outcomes ultimately led me to consider transitioning back to a full-time leadership role.

2011: Returning to South Africa

In 2011, after building a successful career in the UK, progressing from junior software engineer to Principal Engineer and exploring roles in technical program, project, and product management, I made a significant life decision. I chose to return to South Africa, prioritizing family considerations and seeking a different quality of life.

This transition presented both challenges and opportunities. It required me to adapt my career path to a new professional landscape. I embraced this change, starting in a Scrum Master role which allowed me to apply my technical knowledge in a different capacity and gain new perspectives on software development processes.

The following years demonstrated the value of adaptability and continuous learning in one's career. Over a seven-year period, I progressed through various roles of increasing responsibility, including Program Director and Business Owner, eventually reaching a CTO position in 2017. Each role provided unique learning experiences and opportunities to broaden my skill set.

In 2020, I made another strategic career move, transitioning to a senior engineering management role. This shift allowed me to re-engage more directly with technical aspects of the industry while maintaining a strategic perspective. As of 2024, my role continues to evolve, encompassing responsibilities similar to my previous CTO position, albeit in a larger organizational context.

This journey has reinforced my belief in the importance of remaining flexible and open to diverse experiences throughout one's career. Each transition has contributed to my professional growth, providing valuable insights and skills that inform my current work and future aspirations.

Understanding the Stage of Life You're In

In reflecting on these life stages, I'm reminded of Daniel Levinson's groundbreaking work, "The Seasons of a Man's Life." Levinson proposed that our adult lives unfold in a series of stable periods and transitions, each lasting about 5-7 years. He identified key transition points – around 30, 40, and 50 – where we often reassess our life structures and sometimes make dramatic changes. This framework resonates deeply with my own experience of walking away at different points in my career. Levinson's concept of the 'dream' – our vision for our life – and how it evolves over time, has been particularly relevant. I've found that my willingness to walk away has often coincided with these transition periods, as my 'dream' shifted and I realized my current path no longer aligned with my evolving vision for my life and career. Understanding these natural cycles of stability and change has helped me be more intentional about when and why I choose to walk away, recognizing that these decisions are part of a larger, lifelong process of growth and self-discovery.

You know, as I've navigated through my career, I've realized that the decisions to walk away - and the consequences of those decisions - have been heavily influenced by the stage of life I was in. It's not just about the job or the industry; it's about where you are in your personal journey, including your family situation.

Let me break it down based on my own experiences:

The Twenties: Exploration and Early Family Formation

In my twenties, I was all about exploration, but I was also laying the foundation for my family life. Fresh out of university, I was eager to try everything professionally. Walking away was easier then - I had fewer responsibilities, no mortgage. I could take risks, like leaving South Africa altogether in 2001 for an overseas experience.

But here's the twist - by the time I was 32, I had three kids. Suddenly, my decisions weren't just about me anymore. The stakes were getting higher, and I had to start thinking about stability alongside growth.

Advice for this stage: Take calculated risks, but start thinking long-term. If you're starting a family early like I did, begin balancing your career exploration with the need for stability.

The Thirties: Establishing, Advancing, and Juggling

As I entered my thirties, things really started to shift. I was climbing the career ladder, moving from junior roles to more senior positions. But I was also deep in the trenches of raising young children. Walking away became a lot more complicated.

I had to consider things like long-term career progression and financial stability, all while making sure I had enough time and energy for my family. This was when I made the move from engineering to project management, seeking that seat at the customer's table, but also looking for roles that could provide better work-life balance.

My priorities started to change dramatically. It wasn't just about chasing the next big career move anymore. I had to think about school districts, family time, and building a stable home environment. Every career decision was filtered through the lens of "How will this affect my family?"

Advice for this stage: Focus on building expertise and leadership skills, but don't sacrifice family time. Look for roles that offer flexibility. Your decisions to walk away should be strategic, aimed at positioning yourself for bigger roles in the future while also considering your family's needs.

The Forties: Balancing, Redefining, and Contemplating the Future

Now in my forties - I'm closing in on 47 - I've found that walking away takes on a whole new meaning. It's less about climbing the ladder and more about finding fulfillment and balance. When I left my CTO position in 2020, it wasn't for a bigger title or more money - quite the opposite. It was about aligning my work with my values and finding new challenges, but also about being the kind of parent and partner I wanted to be.

At this stage, I'm juggling career aspirations with family responsibilities, personal health, and long-term financial planning. The decision to walk away from a high-paying, prestigious role wasn't easy, but it was necessary for my personal growth and well-being, and ultimately, for my family's well-being too.

I've realized that my career decisions don't just affect me - they set an example for my kids. Do I want them to see me endlessly chasing titles and money, or do I want to show them that it's okay to prioritize personal growth and family time?

As I approach my fifties, I'm starting to think about slowing down. The high-tech world of software engineering can be relentless, always pushing for the next big thing, the next innovation. It's exciting, but it's also exhausting. I'm beginning to ask myself: Do I need to keep up this pace? Is it time to find a role that allows me to contribute my experience without the constant pressure to be at the cutting edge?

Advice for this stage: Don't be afraid to redefine success. Walking away might mean taking a step back in title or salary for better work-life balance or a more fulfilling role. Remember, you're not just building a career, you're building a life and a family legacy. And it's okay to start thinking about how you want your career to look as you enter your fifties and beyond.

The Fifties and Beyond: Legacy, Mentorship, and Finding Your Pace

As I stand on the cusp of my fifties, this next stage is becoming less abstract and more of a reality to plan for. I imagine that walking away in your fifties and beyond becomes less about personal advancement and more about leaving a legacy, mentoring the next generation, and finding a sustainable pace that allows you to continue contributing without burning out.

In the high-tech world, turning 50 doesn't mean you're obsolete - far from it. But it might mean changing how you engage with the industry. Perhaps it's time to transition from being the one always implementing the latest tech to being the wise voice that guides strategy and mentors the younger engineers.

Advice for this stage: Consider how you can use your wealth of experience to give back, both professionally and to your family. Walking away might mean transitioning to advisory roles, teaching, or even starting your own venture that allows you to pass on your wisdom. Don't be afraid to set boundaries and find a pace that works for you - you've earned it.

The Importance of Reassessment

Here's the thing I've learned: no matter what stage you're in, it's crucial to regularly reassess where you are and where you want to be. Life stages aren't fixed - they're fluid. Major life events like marriage, having children, or dealing with health issues can shift your priorities overnight.

I've made it a habit to do a "life stage check" every few years. I ask myself:

  1. What are my current priorities, both professionally and for my family?
  2. Does my work align with these priorities?
  3. What would I regret not doing if I stay on this path?
  4. Am I still growing, or am I just comfortable?
  5. How are my career decisions affecting my family life?

These questions have guided my decisions to walk away, even when it seemed counterintuitive to others. Remember, your career should serve your life and your family, not the other way around.

Understanding your life stage isn't about fitting into a predetermined box. It's about recognizing your current needs, responsibilities, and aspirations, including those of your family. It's about being honest with yourself about what matters most right now.

The Stockdale Paradox and Confronting Reality

Throughout my career transitions, I've found myself often applying a principle known as the Stockdale Paradox, even before I knew it had a name. The "Stockdale Paradox" is a concept popularized by Jim Collins in his book "Good to Great". It's about maintaining an unwavering faith that you will prevail in the end, regardless of the difficulties, AND at the same time, confronting the most brutal facts of your current reality, whatever they might be.

In the context of walking away from established positions or industries, this paradox becomes incredibly relevant. When I left my CTO role to start over in a new industry, I had to balance two seemingly contradictory ideas:

  1. The unwavering belief that this move would lead to long-term growth and new opportunities (the faith part).
  2. The brutal acknowledgment that I was stepping into a role where I had less influence, less domain expertise, and even less compensation (confronting the current reality).

This paradox teaches us to confront the brutal facts of our current reality. It's not about being pessimistic, but about being realistically optimistic. When considering walking away from a job or industry, it's crucial to:

  1. Acknowledge the challenges: Recognize the difficulties you'll face in a new role or industry. For me, this meant accepting that I'd be starting from scratch in terms of industry knowledge and connections.
  2. Assess your resources: Take stock of what you bring to the table. Your skills, experience, and adaptability are your assets in navigating change.
  3. Plan for the worst while hoping for the best: Have contingency plans. When I moved, I made sure I had enough savings to cushion any unforeseen difficulties. I continue to maintain a healthy runway of savings, in the event of me deciding to take a break and pause for while (sabbatical) before considering my next pivot.
  4. Maintain faith in your decision: Despite the challenges, hold onto the belief that your decision to walk away will lead to growth and new opportunities in the long run.

I believe in reusing these business techniques to guide my own personal and professional development. Without taking stock of my current reality and having a vision of future self, how will I know how to act when it the time comes to walk away? I've shared other posts on self-awareness you can reference.

Applying the Stockdale Paradox has helped me make tough decisions with both eyes open. It's allowed me to take calculated risks, knowing that while the path might be difficult, I have the resilience and faith to see it through to a positive outcome.

Knowing Your Value System: When to Walk Away on Principle

Throughout my career, I've learned that one of the most important factors in deciding whether to walk away from a job or opportunity is how well it aligns with your personal value system. It's not just about the money, the title, or even the work itself – it's about being able to look yourself in the mirror each morning and feel good about what you do.

Defining Your Red Lines

We all have our own set of values and ethical boundaries. Throughout my career, I've encountered opportunities that, while potentially lucrative, didn't align with my personal values. These situations taught me the importance of having clear ethical guidelines. It's crucial to understand what types of work or industries align with your principles and which ones don't.

It's worth noting that these boundaries can evolve over time as we grow and our perspectives change. For example, earlier in my career, I was hesitant about working in finance, but I've since become open to opportunities in fintech, particularly in areas like ethical banking technologies. The key is to regularly reassess your values and ensure your work continues to align with them as both you and the industry landscape change.

The Tech Ethics Dilemma

In today's world, ethical lines in the tech industry are becoming increasingly complex. Many large tech companies, while driving innovation, also engage in business with various government and military organizations. This has led some engineers to question the potential societal impact of their work. I, too, continually reflect on the indirect effects of my work, considering the challenges faced by technology platform providers in maintaining neutrality, promoting fair and safe use of tech, and balancing the desire for revenue growth with the increasing responsibilities that come with success and scale.

I've observed colleagues grappling with the realization that their code could potentially be used in ways they hadn't anticipated or might not agree with. It's a challenging position to be in, and it underscores the importance of being aware not just of what you're working on, but also how it might be used down the line. Ultimately, the decision to engage with or step away from certain projects or companies is a personal one, shaped by individual values and convictions.

Practical Steps for Aligning Work with Values

  1. Define Your Values: Take time to explicitly write down what matters most to you. Is it environmental sustainability? Social justice? Privacy rights? Having a clear understanding of your values makes it easier to evaluate opportunities.
  2. Research Potential Employers: Before accepting a job, dig deep into the company's clients, partnerships, and overall mission. Don't just rely on their marketing materials – look for independent reports and employee reviews.
  3. Ask Tough Questions: During interviews, don't be afraid to ask about the company's ethical policies and how they handle potential conflicts. Their response (or lack thereof) can be very telling.
  4. Stay Informed: Once you're in a role, stay aware of your company's evolving practices and clientele. What was once aligned with your values might shift over time.
  5. Be Prepared to Walk Away: If you find your work increasingly conflicting with your values, be prepared to make the tough decision to leave. Your integrity is worth more than any paycheck.

The Bigger Picture

Remember, when you walk away from something that conflicts with your values, you're not just making a decision for yourself. You're setting an example for others in the industry. You're showing that it's possible to have a successful career without compromising your ethics.

Moreover, as more professionals make these stands, companies are forced to reevaluate their practices. We've seen this happening in recent years, with tech workers organizing and demanding more ethical practices from their employers.

Knowing your value system and being willing to walk away when those values are compromised isn't just about feeling good about yourself – though that's important too. It's about actively shaping the kind of industry and world we want to work in. Trust me, it's not always easy. There have been times when I've had to turn down opportunities that looked amazing on paper because something just didn't sit right with me ethically. But you know what? I've never regretted those decisions.

Here's what I want you to take away from this: Really think about your values. Dig deep. What are your non-negotiables? What kind of impact do you want your work to have on the world? And here's the kicker – what are you willing to walk away from to stay true to those values? If you're unsure of where to start, why not look at a framework I created for myself, called the RAGE model? This framework can help you define your value system across all the dimensions of your life, once you've mapped out your current reality against your desired aspirations, you can then reflect on decisions to make with respect to walking away.

It's a tough question, I know. But it's one that's served me well throughout my career. And I bet if you start asking yourself this regularly, you'll find it guiding you towards work that's not just profitable, but truly fulfilling.

Assessing Your Plan: When to Stay the Course and When to Pivot Again

So, you've made the leap. You've walked away from a comfortable position, embarked on a new journey, and set yourself a timeline - say, 3-4 years to achieve your goals in this new role or industry. But what happens when you're in the thick of it, and things aren't quite going as planned? How do you know whether to push through or whether it's time to walk away again?

This is a question I've wrestled with multiple times in my career, and let me tell you, it's not an easy one to answer.

The Case for Staying the Course

We've all heard the advice: "Stay the course." "Push through the difficulties." "Good things come to those who wait." And there's truth to these adages. When I left my CTO position to start over in a new industry, there were moments of doubt, times when I questioned my decision. The learning curve was steep, the pay was less, and the influence I once had seemed a distant memory.

But I had set myself a goal, a timeline. I knew that meaningful change and growth often require time and persistence. In those moments of doubt, I reminded myself of the reasons I made the change in the first place - the desire for new challenges, the need to align my work with my values, the long-term vision I had for my career.

Staying the course allows for:

  1. Deep learning and skill development
  2. Building meaningful relationships and networks
  3. Seeing projects through to completion
  4. Potential for unexpected opportunities to arise

The Argument for Pivoting Again

On the flip side, there's an argument to be made for knowing when to cut your losses. As the saying goes, "Don't cling to a mistake just because you spent a lot of time making it."

I've been in situations where, despite my best efforts and intentions, it became clear that the role or company wasn't the right fit. Maybe the company's values didn't align with mine as I had hoped. Perhaps the industry wasn't evolving in the direction I had anticipated. Or maybe my personal circumstances changed, altering my priorities and what I needed from my career.

In these cases, the cost of staying - in terms of personal wellbeing, career progression, or missed opportunities - might outweigh the benefits of persevering.

Reasons to consider pivoting again:

  1. Misalignment with personal values or goals
  2. Lack of growth or learning opportunities
  3. Negative impact on physical health, mental health or work-life balance
  4. Better opportunities arise that align more closely with your long-term vision

Finding the Balance: Reflective Assessment

So how do you decide? In my experience, it comes down to regular, honest reflection. Here are some questions I ask myself:

  1. Am I still growing and learning, even if it's not in the ways I initially expected?
  2. Does this role/company still align with my values and long-term career goals?
  3. Can I envision a path to my desired end-state from here?
  4. Am I still excited about the potential outcomes, even if the journey is tough?
  5. What would be the cost (personal, professional, financial) of walking away now vs. staying longer?

Remember, it's not about giving up at the first sign of difficulty. It's about honestly assessing whether the current path is still the best route to your ultimate destination.

The Time Factor

One argument that often comes up is the idea that "you can't make up for lost time." And while that's technically true, I've found that you can often make up for lost ground. Skills, networks, and experiences gained in one area can often be leveraged in unexpected ways in the future.

When I stepped down from my CTO role, I initially felt like I was moving backwards. But the perspective I gained, the new skills I developed, and the diverse experiences I continue to accumulate are actually invaluable in ways I couldn't have predicted - despite the ups and downs I continue to face daily in my journey of change.

My Personal Approach

Here's what I've learned to do:

  1. Set clear goals and timelines when making a change, but be flexible.
  2. Regularly reassess (I do this quarterly) - celebrate progress, acknowledge challenges.
  3. Stay connected to my network and keep an eye on the broader industry landscape.
  4. Trust my gut - often, we know deep down when something isn't right, even if we can't immediately articulate why.
  5. Be willing to admit when something isn't working and pivot again if necessary.

Remember, there's no universal right answer. What matters is making conscious, reflective decisions rather than passively drifting or stubbornly sticking to a plan that's no longer serving you.

Your career is a long journey. Sometimes, the bravest thing you can do is to stay and push through the difficulties. Other times, the wisest choice is to recognize that it's time to walk away again. The key is to stay true to yourself, your values, and your long-term vision.

Closing: The Courage to Walk Away and the Wisdom to Reassess

As I've explored throughout this post, the question "What are you willing to walk away from?" is far more complex than it might initially appear. It's a question that has followed me through every stage of my career, from my early days as a fresh graduate to my current position approaching 50 in the relentless world of tech.

I've shared the importance of understanding your life stage, recognizing that what you're willing to walk away from at 25 might be very different from what you'd consider leaving at 45. I've explored the value of diverse experiences, the deceptive nature of the career "ladder," and the importance of aligning your work with your personal values.

The Stockdale Paradox is a good concept that shows the delicate balance of maintaining unwavering faith in our decisions while confronting the brutal facts of our current reality. This paradox becomes particularly relevant when we find ourselves in the midst of a career transition, questioning whether to stay the course or pivot again.

And that's where the discussion on assessing your plan comes in. There's no one-size-fits-all answer to whether you should push through difficulties or walk away again. It requires regular, honest reflection, a clear understanding of your long-term goals, and the courage to make tough decisions.

Throughout all of this, I've emphasized the importance of knowing your value system. In a world where tech giants are increasingly blurring ethical lines, being clear about what you stand for - and what you won't stand for - is becoming more crucial than ever.

So, as you contemplate your own career journey, remember:

  1. Regular reassessment is key. Your goals, values, and circumstances will evolve over time. Make sure your career path is evolving with them.
  2. Don't be afraid of the squiggly career. Diverse experiences can be your greatest asset in an ever-changing job market.
  3. Align your work with your values. No paycheck is worth compromising your integrity.
  4. Prepare for the possibility of walking away. Having a financial runway and a strong network can give you the freedom to make bold moves when necessary.
  5. Trust your instincts, but also seek out diverse perspectives. The resources we've shared - books, podcasts, and articles - can provide valuable insights as you navigate your path.
  6. Remember, you can't make up for lost time, but you can often make up for lost ground. The skills and experiences you gain, even from "detours," can be invaluable in unexpected ways.

In the end, the willingness to walk away - whether from a job, a career path, or even a long-held belief system about what success or happiness looks like - is about having the courage to prioritize growth, fulfillment, and alignment with your values over comfort, convention or what others might say.

It is also not about impulsively quitting every time things get tough! It's about being intentional with your choices, extremely clear about your values and priorities, and being brave enough to make changes when necessary. It's about creating a career that serves your life, not a life that's enslaved to your career.

So, I'll ask you one last time: What are you willing to walk away from? And perhaps more importantly, what future are you walking towards? The journey might be squiggly, it might be challenging, but with clarity of purpose and the courage to make tough choices, it can also be incredibly rewarding.

Remember, in the story of your career, you're not just the protagonist - you're also the author. Don't be afraid to write some bold plot twists!

Further Reading and Resources

Books

  1. The Happiness of Pursuit by Chris Guillebeau - This book explores how pursuing our passions can lead to more fulfilling careers and lives.
  2. Designing Your Life by Bill Burnett and Dave Evans - Offers practical exercises to help you design a career and life that aligns with your values and goals.
  3. Good to Great by Jim Collins - Where I first learned about the Stockdale Paradox. It's primarily about business, but the principles apply well to personal career decisions.
  4. Range: Why Generalists Triumph in a Specialized World by David Epstein - This book challenged my thinking about the value of diverse experiences in our careers.
  5. The Squiggly Career by Helen Tupper and Sarah Ellis - This book is all about embracing the non-linear career path, which resonates strongly with my own experiences. It offers practical advice on how to thrive in the modern workplace where traditional career ladders are becoming obsolete.
  6. Ikigai: The Japanese Secret to a Long and Happy Life by Héctor García and Francesc Miralles - This book explores the Japanese concept of finding purpose in life, which can be incredibly valuable when making career decisions. It's helped me think about how to align my work with what truly fulfills me.
  7. Start with Why: How Great Leaders Inspire Everyone to Take Action by Simon Sinek - Sinek's exploration of purpose-driven leadership has implications far beyond just business. It's a great resource for anyone looking to understand their own motivations and how to make decisions that align with their core purpose.
  8. What I Wish I Knew When I Was 20 by Tina Seelig - This book offers valuable life lessons and unconventional advice that can be particularly helpful when making big career decisions or considering walking away from a comfortable position.
  9. How Will You Measure Your Life? by Clayton M. Christensen - Christensen provides a framework for finding meaning and happiness in life that goes beyond career success. It's a great read for anyone grappling with big life decisions.
  10. Solve for Happy: Engineer Your Path to Joy by Mo Gawdat - Gawdat's approach to happiness, informed by his background as a tech executive, offers a unique perspective on finding fulfillment both in and outside of your career.
  11. The Barakah Effect by Mohammed Faris - This book illuminates the vital role of good intentions, the importance of seeking Allah’s help, the power of gratitude, and the virtue of patience. Learn how to live a life centred on serving your Lord, cultivate a gardener’s mindset that combines ambition with contentment, navigate challenging family dynamics with grace, and leave a legacy that outlives your earthly existence. Learn how to be more with less… with The Barakah Effect.

Podcasts

  1. "How I Built This" with Guy Raz - Interviews with entrepreneurs often reveal moments of walking away and pivoting in their careers.
  2. "WorkLife with Adam Grant" - Often touches on topics related to career decisions and work-life alignment.
  3. "Pivot" with Jenny Blake - Specifically focused on career changes and transitions.
  4. "The Tim Ferriss Show" - While not exclusively about careers, Tim often interviews successful people about their decision-making processes and life philosophies.
  5. "Career Tools" - Offers practical advice on various aspects of career management and development.
  6. "Squiggly Careers" - A podcast by the authors of "The Squiggly Career" book, discussing how to thrive in the modern workplace.
  7. "Finding Mastery" with Dr. Michael Gervais - Explores the psychology of high performance, which can be insightful for making big career decisions.
  8. "The Mindset Mentor" with Rob Dial - Offers motivation and strategies for personal growth, including career development.
  9. "Happen to Your Career" - Focuses on helping people find and do work they love, often featuring stories of career changes.
  10. "Jocko Podcast" with Jocko Willink - While primarily about leadership, Jocko often discusses discipline and decision-making in ways that are relevant to career choices.
  11. "Slo Mo: A Podcast with Mo Gawdat" - Mo Gawdat, former Chief Business Officer at Google X and author of "Solve for Happy," discusses happiness, decision-making, and finding purpose in life and work. His unique perspective as a tech executive turned happiness researcher offers valuable insights for anyone contemplating major life or career changes.

Wednesday, 24 April 2024

Advice for Technology Executives eyeing out AWS L7 Snr Manager role

As I write this post, my phonetool (a cool internal service where employees can lookup people) says I've been working at Amazon for 3 years, 3 months and 13 days (it actually feels much longer than that!!). 
 
I thought it might be a good idea to provide some advice for people who are considering applying for L7 Senior Software Engineering Manager roles at Amazon, even though I am just a sample of one. Amazon is a huge, gigantic, complex entity - so by no means is my experience, advice is representative of the company...rather I'm sharing my very own personal experience, aimed mostly at South African professionals who might be considering interviewing for senior management roles -- and are curious to learn from others who walked the path before. We are so lucky to have a global high-tech giant like Amazon operating in South Africa, offering engineering opportunities. I was even contemplating relocating back to the UK in 2020, but decided to stick it out in this beautiful country, try Amazon and live & work from Cape Town...No other global hi-tech company, apart from Amazon, offers software engineering roles in South Africa as far as I'm aware.
 
If you're considering leaving a C-level or Executive Head position behind, coming from a South African corporate, and are now eyeing out senior engineering manager roles in Amazon/AWS locally or internationally, then this post might just be of use to you. 

DISCLAIMER: I'm sharing my own personal experiences and recommendations in the hope my story could help others make an informed decision. Yes, I'm currently employed at AWS as I write this, and so far enjoyed my journey of transition, pretty settled in now as an Amazonian, despite a bit of a rough start of complete change/disruption to both professional and life streams.

My experience going into AWS L7 Senior Manager Role

Before Amazon, from May 2017 to October 2020, I was the single threaded owner (STO aka CTO) for a video streaming platform serving 50+ countries in Africa. Reporting to me were a group comprising executive head of departments and senior managers responsible for: 
 
1\ Software Delivery & Testing (Backend + Apps running on Smart TVs, Set Top Boxes, Browser Web Apps (Chrome, IE, Firefox, Safari), iOS and Android mobile phones and tablets, Apple TVs, Android TVs. In this org, reporting to the senior manager were: 3x Software Dev Managers (Frontend + Backend - 10 dev teams), 1x Test/QA Manager.

2\ Platform, Networking & Video Infrastructure, DevOps: Data Centres in Johannesburg, Cape Town, Durban, Nigeria, London, Ghana. AWS & Azure cloud. CDN infrastructure. Networking backbone, internet transit infrastructure. Devops. Application Web hosting.

3\ Site Reliability Engineering and 24/7/36 Platform/Application Operations monitoring, 2nd and 3rd line technical support team.

4\ Platform Intelligence Health Dashboards, Alerting & Analytics, Content Discovery, Search and Recommendations. An AI/ML team responsible for building video and content discovery services.This team was also responsible for building Infra/Platform/App Health Analytics consoles.

5\ Enterprise & Solution Architecture. A group of architects cutting across software architecture, video streaming, encoding and media processing platform architects. Live streaming video headends, CDN architecture.

6\ Agile Program and Project Management Office. A group of cross-functional project management experts, including program managers, release managers, scrum masters and general program management.

7\ Cyber Security, Piracy and Governance - risk management, legal, compliance and regulatory ownership. Overall IT/Security and Compliance.

I owned all the above (I designed the org structure around cloud modernization), reporting directly to CEO and was part of a the Top Leadership team. I had direct P&L responsibilities in the order of billions of rands, having authority to manage commercial relationships with technology vendors, suppliers, etc. signing licensing agreements, SOPs, reviewing RFPs,etc. I serviced business, technology and customer operations (marketing, content, acquisition, HR tech, customer support). Under my leadership, the platform scaled  to 10X scale, paved the way for a cloud-first future and created NextGen AI streams for personalized content discovery. I had delivered significant cost-savings, created a cloud-first transformation strategy and clear roadmap and built a strong leadership succession plan before leaving. I had a team of 200+ people, which extended to partner contractors & professional services. I decided to leave and was attracted to Amazon because of my exposure to AWS as an enterprise customer, I wanted to learn about what happens in the engine room of AWS.

Between 2013 and May 2017, before becoming a CTO, I was a freelance senior management consultant, taking on program director contracts where I worked with C-level stakeholders as my primary customer, running very large enterprise programs that cut-across multiple lines of business. Effectively having a virtual seat at the C-suite table, but without skin-in-the-game, helping executives deliver their top business, technology and product transformational goals. For these programs, I directed and executed large-scale program deliveries across the African continent, with project team sizes from 350-1000 people, most of my programs were tagged "billion rand programs" costing "a million rand a day". I enjoyed working 4-day weeks, took my time off according to my schedule, and worked just over 10 months a year, during that time period. 

What made me leave executive roles behind and take a risk with a mid-senior management position? In this post, I shared a deep reflection that motivated for a change. I had set my target on a new domain, i.e. cloud services, an opportunity presented itself with AWS, I applied - and the rest is history. I had made a one-way door decision, there was no looking back - but I did have a bumpy start, which in retrospect, if I had access to these pointers beforehand, who knows...

Tuesday, 29 August 2023

Be the leader you wish you had

BE THE LEADER YOU WISH YOU HAD

I use this saying often in my 1:1s with my directs and in my private coaching sessions. It is a powerful way to make one pause for a moment, reflect, adjust to the discomfort, then embrace the excitement of a new energy that is created.

Adopting this mindset has transformed me from standard "manager" to empathetic "leader". Reading Seth Godin's "The Song of Significance" reinforced my instinctual leadership practices. 

Quoting from "13. Let's Get Real or Let's Not Play", Seth says this:

<quote> No one goes to the gym to willingly get punched in the face by the senior vice president of boxing. But some folks eagerly pay for a sparring partner when it's time to get better.  The difference is obvious, but we've forgotten to say it out aloud.  No grades, no check marks, no badges. I'm not in charge of you, and I'm not manipulating you. I'm simply establishing the conditions for you to get to where you said you wanted to go.  You tell me where you're going and what you need. You make promises about your commitment and skills development.  I'll show up to illuminate, question, answer, spar with, and challenge you. I'll make sure you're part of a team of people who are ready to care as much as you do. We can get real. Or let's not play. </quote>
This is not some leadership mumbo jumbo. Some time ago, I developed a model for personal development that borrowed concepts from agile product management by way of user stories (search RAGE tag on this blog). I then used the same methods in the way I work with my direct reports. HR people might call this "contracting with the employee" but I take it further. I get real. It's not about objectives, KPIs & deliver results. I put myself on the line. I reach out. And so when it comes to performance reviews, my reviews are a two-way conversation. My direct also evaluates Mo's performance - because as a leader, I believe leaders mirror & contribute to the performance of their direct reports. 

What's my mechanism then?

I ask each person to write a user story in this format:

In order for me, [Name] to do [XYZ] (e.g. my job | grow | be inspired | learn | etc.) I need my manager (Mo) to support me by doing [....insert your wish-list here] so that I can ....

So I start the year with level setting on our contracts together, and in our 1:1s, we check-in and inspect, comment, re-calibrate, adjust.  

Guess what? 

This mechanism might seem simple but it's quite challenging for people. Usually, it's the first time they're experiencing a manager doing it this way. There's hook both ways. Often, it takes a few iterations to get the user stories crafted in way that is mutually relatable and agreeable. My mechanism goes beyond the standard business SMART goal setting. I make it human. Real. Personal. For me, this is my song of significance.

Here's some real-world examples in play, from senior managers that report into me - See how doing so puts me, Mo, on the hook?

* In order for me to do my job, I need my manager (Mo) to support me by throwing me in the deep end and exposing me to as much as possible so that I can quickly learn and understand this business

* In order for me to do be inspired, I need my manager (Mo) to support me by leading by example so that I can learn from his vast experience

* In order for me to do grow, I need my manager (Mo) to support me by pushing me out my comfort zone so that I can grow in all directions.

* In order for me to do my job. I need my manager (Mo) to support me by throwing me in the deep end and exposing me to as much as possible so that I can quickly learn and understand this business

* In order for me to grow my skillset, I need my manager to support me in blocking out time on my calendar so I can complete the ‘make great hiring decisions’ course (5hrs)

* In order for me to get promoted to L7, I need my manager to support me by identifying key opportunities so that I can start building a roadmap of promotional milestones

Wednesday, 14 June 2023

A blast from the past: my experience building a large-scale tech platform

In the years 2003-2011, I worked for a pure technology service provider, NDS (acquired by Cisco in 2012, then later became Synamedia) which was considered at the time, the world leader in end-to-end digital TV software systems. I was fortunate enough to experience as an engineer every major area of platform development for this complex ecosystem; and then later as a software manager, I would own the software delivery for a core piece of the software stack known as "middleware", for NDS's primary anchor customer BSkyB/Sky Darwin and then later would own the full stack delivery of NDS's flagship Mediahighway Fusion/Unity product. This experience would mark my entry into very complex large-scale technology delivery initiatives, which even to this day, thirteen years later, as I work with the world's largest cloud provider, Amazon AWS, in building out its enterprise cloud support systems (AWS Support Center / Technical contact systems), Fusion still takes the prize for the most intense professional experience, learning and growth, technical complexity, risk and high-stakes projects. So yeah, I find myself having to dig deep into my memory to recall this work experience because it's funny that 13 years on, I'm encountering the same topics of engineering management even though it is supposed to be a different domain, turns out "software is just software"!

NDS had captured almost every top-tier PayTV operator around the globe at the time: Sky, DirecTV, UPC, Sky Italia, Sky Deutschland, Foxtel, Sky LA, Yes, Bharti, etc. NDS was prominently known for its conditional access product, a video content protection system call NDS Videoguard, however, NDS offered more than just security and offered customers a fully vertically integrated ecosystem (think "Apple" ecosystem for PayTV customers). Whilst digital TV was built on open standards and interoperability, most customers limited their integration points. So when they opted for NDS as their security provider, they also had the option of integrating all other services - from broadcast backend services in the headend to consumer device hardware development and software service integration with chipset vendors. The consumer device software was known as TV Middleware. At the time, the main players were NDS Mediahighway, OpenTV & TiVo. NDS was known for convincing customers to migrate to NDS Mediahighway, its technology migration programs were demanding, complex and executed flawlessly. As an engineer, I contributed software to replace TiVo, an overnight win for 40 million devices. Later as a software delivery manager for the Sky Darwin migration project, we would replace OpenTV software almost obliterating its presence from Sky, save for a few ancient, ageing hardware profiles.

NDS, with an increasing number of customers using its security, middleware and application services, couldn't afford to scale out with engineering teams for each custom build. A platform strategy was needed, consolidating the best of software from across the globe (US, UK, India, Israel, France) into a new shared technology stack, that offered flexible customisation and tailoring for any type of customer profile (Tier-1 customers like Sky for advanced applications to Tier-3/4 customers in territories just starting off with basic digital TV), using a shared engineering resource pool - and extensible configuration engine for producing tailored custom releases. So was borne, NDS Mediahighway Fusion.

The flagship customer for Fusion was Sky, which went live in 2010, replacing up to ten variants of its consumer device software services, with new Fusion components and Sky's own custom-developed consumer application "EPG" known then as the "Orchid EPG". Fusion provided an SDK/API for customers to develop their own primary applications, along with an interactive HTML engine, that allowed PayTV operators to add additional mini apps to their devices, like games and weather apps. With Sky being the anchor customer, Fusion had proved itself in the market and thus was ready to onboard new customers like Sky Italia, UPC, Foxtel, Yes, etc. Post Darwin launch, I took the lead for building the new platform vision, called Fusion Snowflake EPG through project Sunrise - birthing the platform that would create customer, tailorable configurations for any customer, maximising reuse and minimising customisation but allowing for a selection of custom user experiences.

Why am I claiming Fusion as large-scale (even in 2023, 13 years later)?

I write this in 2023, after spending 2.5 years with Amazon AWS. I am part of the group that build AWS Support Center and related Contact Center services. We are a team of under 100 people, deemed large- scale and building complex systems. Yet, if I have to be brutally honest with myself, I'm mildly impressed by my exposure to date, because my current work pails in comparison to my work on Fusion, 13 years ago. Yes I know it's a different domain, a different paradigm and culture of Amazon's 2-Pizza team model for software product ownership (which I actually find quite cool)...still I'm finding it hard to rationalise my move to AWS almost 2.5 years on, have I gone too far backwards? Am I living too much in the past & not ready to view things from a new perspective? What am I not seeing? (Topics for another post). So whilst I've defintely adapted my mental models since joining Amazon, yet I really can't ignore some software engineering truths which is the reason for my bringing up the past now. 

In 2012, I wrote the first story about Fusion, introducing the term LSSDP I coined to mean Large Scale Software Development Project. I also dived deep, writing lengthy white papers about the product and engineering management processes:
Fast forward to 2023, now using my Amazon AWS experience as a lens for defining a large-scale initiative and indirectly checking engineering manager role guidelines for large-scale:
  • Business Impact - Fusion started off with a $75 million investment and later a joint-venture with the flagship customer, Sky. The entire company pivoted to focus on Fusion as its next-generation software platform, with up to 3000 engineers world-wide working on multiple streams, some strategic foundational streams kicked off at least 2 years before the mainstream program. In my role as software delivery owner for Sky Darwin project, it was critical the project delivered successfully, flawlessly - as it involved migrating software in 10 million people's homes (their living room TVs) seamlessly with no rollback. To the end customer (the person sitting at home watching TV), they would notice very little change to their experience. Overall, Fusion software components delivered to multiple middleware stacks, at the time of 2011 when I departed NDS, our software was running in excess of 60 million people's homes daily, globally.
  • Scope and Size - Fusion introduced a new paradigm of the TV software ecosystem, end-to-end, including broadcast headend components as well as embedded software architecture. The stack was open, based on a Linix/Posix and a complete departure from the initial decade of TV software operating systems. This was before the advent of Android TV or fully open source middleware. Fusion's product backlog captured over 2000 epics in the form of work packages, cutting across multiple customer needs, in parallel. The scope included all layers of the device software stack: Chipset drivers, hardware absraction layer, Linux kernel, Linux abstraction, Middleware services, Application SDK/APIs, multiple frontend application engine proxys for C / C++ / Java / HTML / Flash applications. Take a look at the software architecture diagram - it is multi-layered, multiple service teams. Another point on scope, we managed initiatives or epics in the form of work pacakages (WPs), that could impact up to 25 service teams in one WP, see here.