Writing a Failure Resume

This week I completed Chapter 5 of Tina Seelig’s "What I Wish I Knew When I Was 20: A Crash Course on Making Your Place in the World”; and was left thinking about a particular topic that I found quite interesting: writing a “Failure Resume”.  Other people have already blogged about this in the past (ETL, Tina's own blog, mistake bank) sharing similar impressions as I. But what I’d like to do here, is present my very own failure resume just as Tina has done in her book, but adding a little more detail: in addition to summarising the failure, I will include the lesson learnt and future objectives for improvements.

I find Tina’s book a fascinating read, it’s as if the book was especially written for me. I can really relate to Tina’s advice, how I wish I could’ve attended a proper institution like Stanford. I try my best to follow Tina’s ETL series (Entrepreneurial Thought Leaders) program, how generous of them to share this valuable information with the world, for free!!  Whenever I read books on entrepreneurs, I get really excited. I have this feeling in my inner core that this is me, this is something I can do, this is someone I definitely want to be - but haven’t yet taken that leap of faith yet, although I can feel it’s just around the corner.  I plan to share my past ideas with you in another post, just in case you think I’m a nutter, a wanna-be, just a dreamer getting high on books ;-)

So back to this “Failure Resume” and what is so interesting about it??  Most of us concentrate on all the successes and overlook the past failures that probably influenced us more than we’d like to admit ourselves, and could possibly have accounted for future success.  It’s good to keep a record of past failures, and in my opinion, could come in handy during interviews when you’re quizzed “What are your weaknesses?”, “Give us an example where things didn’t go the way you wanted”, etc.  Failing that, it is a good tool for reflection and planning your next growth stage, both personally and professionally.  

To excel and succeed, to reach the next stage or challenge, one has to take risks, try something different - and don’t be afraid of failure. One should embrace failure, because with failure, comes wisdom. A mix of successes and failures provides one with a good balance, well rounded experiences that you can use to your advantage.  The great companies, according to Tina, look for people with a diverse background and talent, willingness to learn, experiment and failure, but wise enough not to repeat past failures.  I, myself, am still searching for a company that embraces this culture - well it is the secret sauce of Silicon Valley after all!!

[April 2011]: So version one of my Failure Resume is below (I found this quite a difficult exercise to complete, this is certainly no 10-minute exercise. In my case, I’m not that experienced, I’ve only got 10 years professional experience, and most of the failures can be rationalised and turned up-side-down to be seen as necessary successes because of my over-arching desire of learning as much as possible, from as many different areas as possible - to be a Jack of all trades in Software):

MUHAMMAD J. KHAN

Professional Failures

Jumping ship too early. My desire to broaden my horizons can be seen as a failure of not giving enough time to stay for a long enough period to establish myself in the role. I find myself changing roles once I feel that I’ve learnt enough or seen enough, eager to seek out the next new challenge. I’ve learnt that you really should allow yourself at least 2 years in a job role before moving onwards. After all practise makes perfect. Just because I excelled in one project doesn’t necessarily mean the next project will be any easier. This failure can be seen as getting bored too quickly, impatient.  It’s a tricky one because you can find yourself falling into the trap of becoming too complacent, watching the years go by and then realising you missed your chance to make that exit you so desperately need right now.

Being made redundant. I left my country of birth after one year of working as a graduate engineer, to work in Ireland. I’d not been overseas before, and neither did I have enough workplace experience (my first company mostly employed people from the local university - we were all friends). It took me a while to settle in with the new company culture, moreover, the company was doing poorly in terms of projects, so for the first 6 months I literally had no real work to do. Although I made efforts to take on work and was proactive, I fear I wasn’t loud enough, blending quietly in the background.  When work did pick up and I was assigned to a good project where I thought I’d played an important role, I falsely assumed that my job was secure in the wave of redundancies taking place. Not so - I was made redundant (they train you to say “my job was made redundant”) - I see that as a personal failure on my part, not doing enough, not contributing to core projects as a failure.  Thankfully, the subsequent 8 years that followed, I made it a point to strategically move when my spider-senses detected a scarcity of work on the horizon.

Falsely assuming managers understood my expectations, not selling myself enough & working too hard. I recently came off an intense project where personally I felt I did an excellent job, even in my appraisals had consistently exceeded my managers expectations.  I assumed that I’d naturally move into a more senior position seeing that I’d proven myself on more than one occasion, doing work proactively, working long hours early into the morning, etc.  Oddly enough, I wasn’t given the appreciation I assumed was coming my way.  I did not communicate clearly enough to the main project stakeholder of my expectations, once the project was over, I found myself wandering, in limbo - all that effort, all my hard work wasn’t good enough to be noticed. Seeing that I wasn’t getting what I wanted, I decided to leave the team, leaving a hole behind now filled in by three people. A new project was kicked off, but it was too late for me to board that ship!  I realise now that expectations must be communicated clearly, and also, you should try NOT to do too much (if you’re overloaded, shout about it, don’t take it on), there’s more important things to worry about than work.

Refusing to play the corporate game.  Perhaps it’s because of my naivety, but for a few years I’d refused to play to corporate game of falsely smiling and being nice to managers, pretending to give a damn when in fact you don’t, just because you want to climb the corporate ladder.  Yes, you need to get on with all your colleagues, but when it comes to work and execution of tasks, one should be judged on only this - the value one adds to the bottom-line.  Maybe I have a cultural or religious bias, ideals that prevent me from mixing and socialising?  I’ve learnt that if you don’t allow yourself a little flexibility without compromising on your core principles, you need to develop some “street savvy, be street-smart” in the workplace, to make the connections and silently manoeuvre to positions advantageous for you.  Don’t expect others to do it for you, you need to consciously make the effort and plan ahead - take time to determine the corporate culture, identify the teams/people you’d rather not get involved with, have a plan of attack targeting the people you’d like to end up with.

Academic Failures

Not doing my best, not 100% focused. I used to be a straight-A/B student, but at university, I dropped the ball in my third & final years (the most important years!). Staying on my own with friends, having unrivalled freedom to experiment, even the distraction of girls, I ended up not attending lectures, avoiding the lecturers, and just sitting for the exams with only a month’s preparation, gaining a 3rd class pass (66% if you average the 4 years), when I could’ve easily got at least 75% had I been focused.  Whilst I did successfully qualify, I regret not using the vast amount of resources that were available to us (e.g. experimenting with business projects, becoming more involved with the university staff, etc).

Poor relationship management. Whilst some of this can be attributed to the unnatural cultural legacy of Apartheid, and that it needed some mental fortitude for students from backgrounds such as myself to break through the invisible barrier between different people (black/indian/coloured/white) - I failed to build any meaningful relationships with the faculty staff at university, so much so, that when I enrolled for a Masters in Ireland, the professor in charge was not entirely positive in his recommendation.  We were not on talking terms socially, I’d only visit him when a report or presentation was due - we did not have a mentor/student relationship.  At university the class would be naturally be separate into racial groups, we didn’t really interact with each other.  Had we learnt how to make these relationships in university, then working in a multicultural company would’ve been a piece of cake.

Personal Failures

Not knowing enough - choosing the wrong career path.  Perhaps I’m being too hash on myself, but one of my failures goes way back to when I finished high school, the period where you decide what to do for the rest of your life.  I feel had I known better, I could’ve made different choices that would’ve set out a path so different to where I am today, although the present one isn’t so bad.  Nevertheless, I failed to do the research, the serious searching that was required at this important juncture of my life.

Not executing my ideas, not taking enough risks.  I have always had ideas that could’ve led to something great, if only I’d acted upon them.  Either I take too long to experiment with the idea, or lack the confidence to push it through, take it to the next level.  I have executed on a couple ideas, but this was within the context of a safe environment of the workplace.  If I failing to execute, I can’t call myself an entrepreneur. I need to stop making excuses (no time, work, family, no money) and take a chance.

Rewarding myself.  I over analyse and keep putting the needs of others ahead of mine.  I am also too hard on myself, and very self-critical. Personal time and space is important, and is a right of an individual. I don’t allow myself enough of this, and also have failed in keeping my interests up-to-date, for example: making sketches, doing some programming (i.e. learning new languages outside of work), or taking time out to be active.

Relationship management. There were times when I found difficult to separate out my work persona from my family persona - and expect my family to be run just as I’m running a project at work, to an extent that I’ve been told I act like a general in the army, too strict with the wife and kids, with too many expectations. Over time, I’ve improved and trying hard to change :-)

[October 2016 Update]: Five years onwards and I've had a relook at my failure resume, read here!

On Self-Awareness

This year I've been focusing on self-awareness, which in my case was really about taking stock of myself in asking searching questions. It started toward the end of last year, when I first began to make sense of a model that I coined as the RAGE model (Reality, Aspirations, Goals, Expectations). In the course of this year, I had applied RAGE to not only a number of my personal & professional situations, but also used it to help guide a few friends and colleagues. I realised that many people find themselves in situations, when faced with different choices, are often conflicted about these choices, find themselves searching, reaching out to me(!). What surprised me most was how I was able to give neutral counsel, often by re-applying the techniques I've come to use on myself - the crux of it is really around asking searching questions. So I decided to use this blog post to capture my learnings to date, as references to the topics I've come across, share them with you...at the very least, I can come back to this post for a refresher when faced with my next dilemma. I myself am still learning...

This post is taking me more than just a few days to write, so I'm releasing it incrementally. There's still some empty sections that will be filled up as I go along. It's broken up as follows, you can read each section independently:

• Real-Life Scenarios
• Know Thyself - RAGE Model
• Dreyfus Model
• Discover Your Generational Archetype
• Cognitive Biases
• Socrates Questions / Socratic Method
• Ego is the Enemy
• Humble Yourself
• Create Your Own Prime Directive
• Antifragile & Black Swan Events
• Happiness Criteria
• Circle of Control / Influence
• Conversation Transformation
• My Own Situational History
• Failure Resume
• Concluding Remarks

What are you willing to walk away from?

Table of Contents

• Introduction
• The Concept of Walking Away
• Insights from 25 Years of Professional Experience
• Personal Experiences with Walking Away
  • 2020: From CTO to Senior Manager
  • 2019: From Consulting to CTO
  • 2013: Senior Manager to Freelance Consulting
  • 2011: Returning to South Africa
• Understanding the Stage of Life You're In
• The Stockdale Paradox and Confronting Reality
• Knowing Your Value System: When to Walk Away on Principle
• Assessing Your Plan: When to Stay the Course and When to Pivot Again
• Conclusion: The Courage to Walk Away and the Wisdom to Reassess
• Further Reading and Resources

Introduction

These are loaded questions of course - there's so much that goes on in one's life, there are many dimensions to life - so we need to be more specific. When I talk about walking away, it's about the career and professional dimension - one's work, job, company, vocation, etc. I found that asking myself this question surfaced at almost all stages of my career. Whether I was walking away from my first choice of degree to study, or walking away from three job offers for my first engineering job, or leaving a role before being promoted, or as recent as 2020 when I decided to walk away from C-level back to mid/senior-management, "What am I willing to walk away from?" has been my resonant, background question driving my self-reflections. Earlier this month, call it a moment of inspiration, these words came to my mind:

“Sometimes it is okay to experiment with new experiences long enough to realise the experience is not worth the reward you originally imagined, and that’s perfectly okay”

The Concept of Walking Away

Being willing to walk away from your work / company / profession / business is not easy. Being prepared is even harder and more important. Asking the question at regular intervals can be somewhat uncomfortable and stressful, so timing is also important. I previously shared my experiences in software engineering, where I advised being careful to jump ship too early, and also included that as learning in my failure resume. IMHO, one needs to create some stability in the role for at least 18-24 months before moving on - I still stand by this. But that's not to say, frequent inspection of your current role and desired trajectory should be put off until 18-24 months. It's good to inspect, but don't do anything rash or hasty. Know what you stand for, have a plan or at least have a clear picture of your desired future state, being intentional about it. Answering the question becomes quite obvious when your realize the current track you're on is actually deviating from the track you intended - and if your intentions are still valid, then being willing to walk away is a very relevant question and important to resolve.

Insights from 25 Years of Professional Experience

Each career move has been a strategic decision to broaden my skill set, take on new challenges, and align my work with my evolving professional goals. This diverse experience has made me a more well-rounded professional, able to bring unique perspectives to each new role.

The career "ladder" is often more complex than it appears. Embracing a non-linear or "squiggly" career path can lead to a rich diversity of experiences. It's important to focus on personal growth and skill development rather than just titles or compensation.

Taking a step in a different direction isn't necessarily a setback. It can lead to growth in unexpected ways, though it's important to be prepared for the challenges that come with change. Every new path is an opportunity to learn and evolve professionally.

While I've made personal choices about the industries I work in, I respect that every professional must make their own decisions based on their individual values and circumstances.

While my current role may not have the same title or compensation as my previous position, it offers unique learning opportunities and the chance to contribute to groundbreaking projects, which I find incredibly fulfilling. I've found that a balanced approach to work often leads to increased productivity and creativity. My experience with alternative work schedules has taught me the value of focused, efficient work time balanced with adequate rest and personal development.

The pace of the tech industry can be relentless, but it's important to find a sustainable rhythm. It's okay to pause, reassess, and adjust your course as needed. Your career should serve your life goals, not the other way around.

Personal Experiences with Walking Away

2025 Update: Decision to leave AWS - click here

2020: From CTO to Senior Manager

In 2020, I made a significant career transition, moving from a CTO position in an industry where I had spent 20 years building expertise to a senior management role in the cloud computing sector. This decision was driven by a desire for new challenges and growth opportunities. I made the decision to prioritize new experiences and growth opportunities over short-term financial gains, recognizing that long-term career satisfaction was more valuable to me at this stage.

The transition involved relocating to a new city with my family and starting fresh in a larger organization. It was a chance to step back from the complexities of C-level responsibilities and re-engage with the technical aspects of the work that had initially drawn me to the tech industry.

While the C-level role offered valuable experiences, I found myself seeking different challenges and a work environment more aligned with my evolving personal and professional values. I was excited by the opportunity to work on cutting-edge computing technologies, which aligned more closely with my personal interests and career goals at that time.

2019: From Consulting to CTO

In 2019, I transitioned from consulting to a CTO role, seeing it as an opportunity for professional growth and increased responsibility. While consulting offered flexibility and valuable experiences, I was eager to have a more direct impact on a company's strategic decisions and operations.

The move to a CTO position presented new challenges and learning opportunities. It meant exchanging the diverse project exposure of consulting for deep engagement with one organization's goals and challenges. This transition allowed me to apply my broad experience to a focused leadership role, accountable for significant business outcomes.

Although the consulting role offered certain advantages, including a flexible schedule and competitive compensation, I was drawn to the CTO position by the prospect of shaping company strategy and directly influencing outcomes. This move aligned with my career goal of expanding my leadership skills and taking on greater responsibility in driving business success.

2013: Senior Manager to Freelance Consulting

In 2013, after successfully leading a senior management role where I contributed to significant business and engineering advancements, I recognized an opportunity for broader professional growth. My experience in Satellite TV and embedded software applications had provided me with deep domain expertise, but I was eager to expand my knowledge into emerging areas like mobile and cloud streaming technologies.

Sensing that my professional growth potential in my current role was plateauing, I made the strategic decision to transition into freelance consulting. This move allowed me to diversify my experience and gain exposure to a wider range of business challenges and technologies.

The shift to consulting opened up numerous opportunities. Over the next five years, I engaged in various high-impact projects, including launching a video streaming business in Africa and leading major corporate-wide programs. This experience provided invaluable exposure to C-level decision-making processes and broadened my understanding of different aspects of the media and technology industries.

Consulting also offered the flexibility to structure my work schedule in a way that optimized my productivity and allowed for continuous learning. I found that this balance enhanced my ability to deliver high-quality results for clients while also pursuing personal development.

As my consulting career progressed, I began to seek new challenges that would allow me to have a more direct, long-term impact on an organization's strategy and operations. This desire for deeper engagement and accountability in business outcomes ultimately led me to consider transitioning back to a full-time leadership role.

2011: Returning to South Africa

In 2011, after building a successful career in the UK, progressing from junior software engineer to Principal Engineer and exploring roles in technical program, project, and product management, I made a significant life decision. I chose to return to South Africa, prioritizing family considerations and seeking a different quality of life.

This transition presented both challenges and opportunities. It required me to adapt my career path to a new professional landscape. I embraced this change, starting in a Scrum Master role which allowed me to apply my technical knowledge in a different capacity and gain new perspectives on software development processes.

The following years demonstrated the value of adaptability and continuous learning in one's career. Over a seven-year period, I progressed through various roles of increasing responsibility, including Program Director and Business Owner, eventually reaching a CTO position in 2017. Each role provided unique learning experiences and opportunities to broaden my skill set.

In 2020, I made another strategic career move, transitioning to a senior engineering management role. This shift allowed me to re-engage more directly with technical aspects of the industry while maintaining a strategic perspective. As of 2024, my role continues to evolve, encompassing responsibilities similar to my previous CTO position, albeit in a larger organizational context.

This journey has reinforced my belief in the importance of remaining flexible and open to diverse experiences throughout one's career. Each transition has contributed to my professional growth, providing valuable insights and skills that inform my current work and future aspirations.

Understanding the Stage of Life You're In

In reflecting on these life stages, I'm reminded of Daniel Levinson's groundbreaking work, "The Seasons of a Man's Life." Levinson proposed that our adult lives unfold in a series of stable periods and transitions, each lasting about 5-7 years. He identified key transition points – around 30, 40, and 50 – where we often reassess our life structures and sometimes make dramatic changes. This framework resonates deeply with my own experience of walking away at different points in my career. Levinson's concept of the 'dream' – our vision for our life – and how it evolves over time, has been particularly relevant. I've found that my willingness to walk away has often coincided with these transition periods, as my 'dream' shifted and I realized my current path no longer aligned with my evolving vision for my life and career. Understanding these natural cycles of stability and change has helped me be more intentional about when and why I choose to walk away, recognizing that these decisions are part of a larger, lifelong process of growth and self-discovery.

You know, as I've navigated through my career, I've realized that the decisions to walk away - and the consequences of those decisions - have been heavily influenced by the stage of life I was in. It's not just about the job or the industry; it's about where you are in your personal journey, including your family situation.

Let me break it down based on my own experiences:

The Twenties: Exploration and Early Family Formation

In my twenties, I was all about exploration, but I was also laying the foundation for my family life. Fresh out of university, I was eager to try everything professionally. Walking away was easier then - I had fewer responsibilities, no mortgage. I could take risks, like leaving South Africa altogether in 2001 for an overseas experience.

But here's the twist - by the time I was 32, I had three kids. Suddenly, my decisions weren't just about me anymore. The stakes were getting higher, and I had to start thinking about stability alongside growth.

Advice for this stage: Take calculated risks, but start thinking long-term. If you're starting a family early like I did, begin balancing your career exploration with the need for stability.

The Thirties: Establishing, Advancing, and Juggling

As I entered my thirties, things really started to shift. I was climbing the career ladder, moving from junior roles to more senior positions. But I was also deep in the trenches of raising young children. Walking away became a lot more complicated.

I had to consider things like long-term career progression and financial stability, all while making sure I had enough time and energy for my family. This was when I made the move from engineering to project management, seeking that seat at the customer's table, but also looking for roles that could provide better work-life balance.

My priorities started to change dramatically. It wasn't just about chasing the next big career move anymore. I had to think about school districts, family time, and building a stable home environment. Every career decision was filtered through the lens of "How will this affect my family?"

Advice for this stage: Focus on building expertise and leadership skills, but don't sacrifice family time. Look for roles that offer flexibility. Your decisions to walk away should be strategic, aimed at positioning yourself for bigger roles in the future while also considering your family's needs.

The Forties: Balancing, Redefining, and Contemplating the Future

Now in my forties - I'm closing in on 47 - I've found that walking away takes on a whole new meaning. It's less about climbing the ladder and more about finding fulfillment and balance. When I left my CTO position in 2020, it wasn't for a bigger title or more money - quite the opposite. It was about aligning my work with my values and finding new challenges, but also about being the kind of parent and partner I wanted to be.

At this stage, I'm juggling career aspirations with family responsibilities, personal health, and long-term financial planning. The decision to walk away from a high-paying, prestigious role wasn't easy, but it was necessary for my personal growth and well-being, and ultimately, for my family's well-being too.

I've realized that my career decisions don't just affect me - they set an example for my kids. Do I want them to see me endlessly chasing titles and money, or do I want to show them that it's okay to prioritize personal growth and family time?

As I approach my fifties, I'm starting to think about slowing down. The high-tech world of software engineering can be relentless, always pushing for the next big thing, the next innovation. It's exciting, but it's also exhausting. I'm beginning to ask myself: Do I need to keep up this pace? Is it time to find a role that allows me to contribute my experience without the constant pressure to be at the cutting edge?

Advice for this stage: Don't be afraid to redefine success. Walking away might mean taking a step back in title or salary for better work-life balance or a more fulfilling role. Remember, you're not just building a career, you're building a life and a family legacy. And it's okay to start thinking about how you want your career to look as you enter your fifties and beyond.

The Fifties and Beyond: Legacy, Mentorship, and Finding Your Pace

As I stand on the cusp of my fifties, this next stage is becoming less abstract and more of a reality to plan for. I imagine that walking away in your fifties and beyond becomes less about personal advancement and more about leaving a legacy, mentoring the next generation, and finding a sustainable pace that allows you to continue contributing without burning out.

In the high-tech world, turning 50 doesn't mean you're obsolete - far from it. But it might mean changing how you engage with the industry. Perhaps it's time to transition from being the one always implementing the latest tech to being the wise voice that guides strategy and mentors the younger engineers.

Advice for this stage: Consider how you can use your wealth of experience to give back, both professionally and to your family. Walking away might mean transitioning to advisory roles, teaching, or even starting your own venture that allows you to pass on your wisdom. Don't be afraid to set boundaries and find a pace that works for you - you've earned it.

The Importance of Reassessment

Here's the thing I've learned: no matter what stage you're in, it's crucial to regularly reassess where you are and where you want to be. Life stages aren't fixed - they're fluid. Major life events like marriage, having children, or dealing with health issues can shift your priorities overnight.

I've made it a habit to do a "life stage check" every few years. I ask myself:

1. What are my current priorities, both professionally and for my family?
2. Does my work align with these priorities?
3. What would I regret not doing if I stay on this path?
4. Am I still growing, or am I just comfortable?
5. How are my career decisions affecting my family life?

These questions have guided my decisions to walk away, even when it seemed counterintuitive to others. Remember, your career should serve your life and your family, not the other way around.

Understanding your life stage isn't about fitting into a predetermined box. It's about recognizing your current needs, responsibilities, and aspirations, including those of your family. It's about being honest with yourself about what matters most right now.

The Stockdale Paradox and Confronting Reality

Throughout my career transitions, I've found myself often applying a principle known as the Stockdale Paradox, even before I knew it had a name. The "Stockdale Paradox" is a concept popularized by Jim Collins in his book "Good to Great". It's about maintaining an unwavering faith that you will prevail in the end, regardless of the difficulties, AND at the same time, confronting the most brutal facts of your current reality, whatever they might be.

In the context of walking away from established positions or industries, this paradox becomes incredibly relevant. When I left my CTO role to start over in a new industry, I had to balance two seemingly contradictory ideas:

1. The unwavering belief that this move would lead to long-term growth and new opportunities (the faith part).
2. The brutal acknowledgment that I was stepping into a role where I had less influence, less domain expertise, and even less compensation (confronting the current reality).

This paradox teaches us to confront the brutal facts of our current reality. It's not about being pessimistic, but about being realistically optimistic. When considering walking away from a job or industry, it's crucial to:

1. Acknowledge the challenges: Recognize the difficulties you'll face in a new role or industry. For me, this meant accepting that I'd be starting from scratch in terms of industry knowledge and connections.
2. Assess your resources: Take stock of what you bring to the table. Your skills, experience, and adaptability are your assets in navigating change.
3. Plan for the worst while hoping for the best: Have contingency plans. When I moved, I made sure I had enough savings to cushion any unforeseen difficulties. I continue to maintain a healthy runway of savings, in the event of me deciding to take a break and pause for while (sabbatical) before considering my next pivot.
4. Maintain faith in your decision: Despite the challenges, hold onto the belief that your decision to walk away will lead to growth and new opportunities in the long run.

I believe in reusing these business techniques to guide my own personal and professional development. Without taking stock of my current reality and having a vision of future self, how will I know how to act when it the time comes to walk away? I've shared other posts on self-awareness you can reference.

Applying the Stockdale Paradox has helped me make tough decisions with both eyes open. It's allowed me to take calculated risks, knowing that while the path might be difficult, I have the resilience and faith to see it through to a positive outcome.

Knowing Your Value System: When to Walk Away on Principle

Throughout my career, I've learned that one of the most important factors in deciding whether to walk away from a job or opportunity is how well it aligns with your personal value system. It's not just about the money, the title, or even the work itself – it's about being able to look yourself in the mirror each morning and feel good about what you do.

Defining Your Red Lines

We all have our own set of values and ethical boundaries. Throughout my career, I've encountered opportunities that, while potentially lucrative, didn't align with my personal values. These situations taught me the importance of having clear ethical guidelines. It's crucial to understand what types of work or industries align with your principles and which ones don't.

It's worth noting that these boundaries can evolve over time as we grow and our perspectives change. For example, earlier in my career, I was hesitant about working in finance, but I've since become open to opportunities in fintech, particularly in areas like ethical banking technologies. The key is to regularly reassess your values and ensure your work continues to align with them as both you and the industry landscape change.

The Tech Ethics Dilemma

In today's world, ethical lines in the tech industry are becoming increasingly complex. Many large tech companies, while driving innovation, also engage in business with various government and military organizations. This has led some engineers to question the potential societal impact of their work. I, too, continually reflect on the indirect effects of my work, considering the challenges faced by technology platform providers in maintaining neutrality, promoting fair and safe use of tech, and balancing the desire for revenue growth with the increasing responsibilities that come with success and scale.

I've observed colleagues grappling with the realization that their code could potentially be used in ways they hadn't anticipated or might not agree with. It's a challenging position to be in, and it underscores the importance of being aware not just of what you're working on, but also how it might be used down the line. Ultimately, the decision to engage with or step away from certain projects or companies is a personal one, shaped by individual values and convictions.

Practical Steps for Aligning Work with Values

1. Define Your Values: Take time to explicitly write down what matters most to you. Is it environmental sustainability? Social justice? Privacy rights? Having a clear understanding of your values makes it easier to evaluate opportunities.
2. Research Potential Employers: Before accepting a job, dig deep into the company's clients, partnerships, and overall mission. Don't just rely on their marketing materials – look for independent reports and employee reviews.
3. Ask Tough Questions: During interviews, don't be afraid to ask about the company's ethical policies and how they handle potential conflicts. Their response (or lack thereof) can be very telling.
4. Stay Informed: Once you're in a role, stay aware of your company's evolving practices and clientele. What was once aligned with your values might shift over time.
5. Be Prepared to Walk Away: If you find your work increasingly conflicting with your values, be prepared to make the tough decision to leave. Your integrity is worth more than any paycheck.

The Bigger Picture

Remember, when you walk away from something that conflicts with your values, you're not just making a decision for yourself. You're setting an example for others in the industry. You're showing that it's possible to have a successful career without compromising your ethics.

Moreover, as more professionals make these stands, companies are forced to reevaluate their practices. We've seen this happening in recent years, with tech workers organizing and demanding more ethical practices from their employers.

Knowing your value system and being willing to walk away when those values are compromised isn't just about feeling good about yourself – though that's important too. It's about actively shaping the kind of industry and world we want to work in. Trust me, it's not always easy. There have been times when I've had to turn down opportunities that looked amazing on paper because something just didn't sit right with me ethically. But you know what? I've never regretted those decisions.

Here's what I want you to take away from this: Really think about your values. Dig deep. What are your non-negotiables? What kind of impact do you want your work to have on the world? And here's the kicker – what are you willing to walk away from to stay true to those values? If you're unsure of where to start, why not look at a framework I created for myself, called the RAGE model? This framework can help you define your value system across all the dimensions of your life, once you've mapped out your current reality against your desired aspirations, you can then reflect on decisions to make with respect to walking away.

It's a tough question, I know. But it's one that's served me well throughout my career. And I bet if you start asking yourself this regularly, you'll find it guiding you towards work that's not just profitable, but truly fulfilling.

Assessing Your Plan: When to Stay the Course and When to Pivot Again

So, you've made the leap. You've walked away from a comfortable position, embarked on a new journey, and set yourself a timeline - say, 3-4 years to achieve your goals in this new role or industry. But what happens when you're in the thick of it, and things aren't quite going as planned? How do you know whether to push through or whether it's time to walk away again?

This is a question I've wrestled with multiple times in my career, and let me tell you, it's not an easy one to answer.

The Case for Staying the Course

We've all heard the advice: "Stay the course." "Push through the difficulties." "Good things come to those who wait." And there's truth to these adages. When I left my CTO position to start over in a new industry, there were moments of doubt, times when I questioned my decision. The learning curve was steep, the pay was less, and the influence I once had seemed a distant memory.

But I had set myself a goal, a timeline. I knew that meaningful change and growth often require time and persistence. In those moments of doubt, I reminded myself of the reasons I made the change in the first place - the desire for new challenges, the need to align my work with my values, the long-term vision I had for my career.

Staying the course allows for:

1. Deep learning and skill development
2. Building meaningful relationships and networks
3. Seeing projects through to completion
4. Potential for unexpected opportunities to arise

The Argument for Pivoting Again

On the flip side, there's an argument to be made for knowing when to cut your losses. As the saying goes, "Don't cling to a mistake just because you spent a lot of time making it."

I've been in situations where, despite my best efforts and intentions, it became clear that the role or company wasn't the right fit. Maybe the company's values didn't align with mine as I had hoped. Perhaps the industry wasn't evolving in the direction I had anticipated. Or maybe my personal circumstances changed, altering my priorities and what I needed from my career.

In these cases, the cost of staying - in terms of personal wellbeing, career progression, or missed opportunities - might outweigh the benefits of persevering.

Reasons to consider pivoting again:

1. Misalignment with personal values or goals
2. Lack of growth or learning opportunities
3. Negative impact on physical health, mental health or work-life balance
4. Better opportunities arise that align more closely with your long-term vision

Finding the Balance: Reflective Assessment

So how do you decide? In my experience, it comes down to regular, honest reflection. Here are some questions I ask myself:

1. Am I still growing and learning, even if it's not in the ways I initially expected?
2. Does this role/company still align with my values and long-term career goals?
3. Can I envision a path to my desired end-state from here?
4. Am I still excited about the potential outcomes, even if the journey is tough?
5. What would be the cost (personal, professional, financial) of walking away now vs. staying longer?

Remember, it's not about giving up at the first sign of difficulty. It's about honestly assessing whether the current path is still the best route to your ultimate destination.

The Time Factor

One argument that often comes up is the idea that "you can't make up for lost time." And while that's technically true, I've found that you can often make up for lost ground. Skills, networks, and experiences gained in one area can often be leveraged in unexpected ways in the future.

When I stepped down from my CTO role, I initially felt like I was moving backwards. But the perspective I gained, the new skills I developed, and the diverse experiences I continue to accumulate are actually invaluable in ways I couldn't have predicted - despite the ups and downs I continue to face daily in my journey of change.

My Personal Approach

Here's what I've learned to do:

1. Set clear goals and timelines when making a change, but be flexible.
2. Regularly reassess (I do this quarterly) - celebrate progress, acknowledge challenges.
3. Stay connected to my network and keep an eye on the broader industry landscape.
4. Trust my gut - often, we know deep down when something isn't right, even if we can't immediately articulate why.
5. Be willing to admit when something isn't working and pivot again if necessary.

Remember, there's no universal right answer. What matters is making conscious, reflective decisions rather than passively drifting or stubbornly sticking to a plan that's no longer serving you.

Your career is a long journey. Sometimes, the bravest thing you can do is to stay and push through the difficulties. Other times, the wisest choice is to recognize that it's time to walk away again. The key is to stay true to yourself, your values, and your long-term vision.

Closing: The Courage to Walk Away and the Wisdom to Reassess

As I've explored throughout this post, the question "What are you willing to walk away from?" is far more complex than it might initially appear. It's a question that has followed me through every stage of my career, from my early days as a fresh graduate to my current position approaching 50 in the relentless world of tech.

I've shared the importance of understanding your life stage, recognizing that what you're willing to walk away from at 25 might be very different from what you'd consider leaving at 45. I've explored the value of diverse experiences, the deceptive nature of the career "ladder," and the importance of aligning your work with your personal values.

The Stockdale Paradox is a good concept that shows the delicate balance of maintaining unwavering faith in our decisions while confronting the brutal facts of our current reality. This paradox becomes particularly relevant when we find ourselves in the midst of a career transition, questioning whether to stay the course or pivot again.

And that's where the discussion on assessing your plan comes in. There's no one-size-fits-all answer to whether you should push through difficulties or walk away again. It requires regular, honest reflection, a clear understanding of your long-term goals, and the courage to make tough decisions.

Throughout all of this, I've emphasized the importance of knowing your value system. In a world where tech giants are increasingly blurring ethical lines, being clear about what you stand for - and what you won't stand for - is becoming more crucial than ever.

So, as you contemplate your own career journey, remember:

1. Regular reassessment is key. Your goals, values, and circumstances will evolve over time. Make sure your career path is evolving with them.
2. Don't be afraid of the squiggly career. Diverse experiences can be your greatest asset in an ever-changing job market.
3. Align your work with your values. No paycheck is worth compromising your integrity.
4. Prepare for the possibility of walking away. Having a financial runway and a strong network can give you the freedom to make bold moves when necessary.
5. Trust your instincts, but also seek out diverse perspectives. The resources we've shared - books, podcasts, and articles - can provide valuable insights as you navigate your path.
6. Remember, you can't make up for lost time, but you can often make up for lost ground. The skills and experiences you gain, even from "detours," can be invaluable in unexpected ways.

In the end, the willingness to walk away - whether from a job, a career path, or even a long-held belief system about what success or happiness looks like - is about having the courage to prioritize growth, fulfillment, and alignment with your values over comfort, convention or what others might say.

It is also not about impulsively quitting every time things get tough! It's about being intentional with your choices, extremely clear about your values and priorities, and being brave enough to make changes when necessary. It's about creating a career that serves your life, not a life that's enslaved to your career.

So, I'll ask you one last time: What are you willing to walk away from? And perhaps more importantly, what future are you walking towards? The journey might be squiggly, it might be challenging, but with clarity of purpose and the courage to make tough choices, it can also be incredibly rewarding.

Remember, in the story of your career, you're not just the protagonist - you're also the author. Don't be afraid to write some bold plot twists!

Further Reading and Resources

Books

1. The Happiness of Pursuit by Chris Guillebeau - This book explores how pursuing our passions can lead to more fulfilling careers and lives.
2. Designing Your Life by Bill Burnett and Dave Evans - Offers practical exercises to help you design a career and life that aligns with your values and goals.
3. Good to Great by Jim Collins - Where I first learned about the Stockdale Paradox. It's primarily about business, but the principles apply well to personal career decisions.
4. Range: Why Generalists Triumph in a Specialized World by David Epstein - This book challenged my thinking about the value of diverse experiences in our careers.
5. The Squiggly Career by Helen Tupper and Sarah Ellis - This book is all about embracing the non-linear career path, which resonates strongly with my own experiences. It offers practical advice on how to thrive in the modern workplace where traditional career ladders are becoming obsolete.
6. Ikigai: The Japanese Secret to a Long and Happy Life by Héctor García and Francesc Miralles - This book explores the Japanese concept of finding purpose in life, which can be incredibly valuable when making career decisions. It's helped me think about how to align my work with what truly fulfills me.
7. Start with Why: How Great Leaders Inspire Everyone to Take Action by Simon Sinek - Sinek's exploration of purpose-driven leadership has implications far beyond just business. It's a great resource for anyone looking to understand their own motivations and how to make decisions that align with their core purpose.
8. What I Wish I Knew When I Was 20 by Tina Seelig - This book offers valuable life lessons and unconventional advice that can be particularly helpful when making big career decisions or considering walking away from a comfortable position.
9. How Will You Measure Your Life? by Clayton M. Christensen - Christensen provides a framework for finding meaning and happiness in life that goes beyond career success. It's a great read for anyone grappling with big life decisions.
10. Solve for Happy: Engineer Your Path to Joy by Mo Gawdat - Gawdat's approach to happiness, informed by his background as a tech executive, offers a unique perspective on finding fulfillment both in and outside of your career.
11. The Barakah Effect by Mohammed Faris - This book illuminates the vital role of good intentions, the importance of seeking Allah’s help, the power of gratitude, and the virtue of patience. Learn how to live a life centred on serving your Lord, cultivate a gardener’s mindset that combines ambition with contentment, navigate challenging family dynamics with grace, and leave a legacy that outlives your earthly existence. Learn how to be more with less… with The Barakah Effect.

Podcasts

1. "How I Built This" with Guy Raz - Interviews with entrepreneurs often reveal moments of walking away and pivoting in their careers.
2. "WorkLife with Adam Grant" - Often touches on topics related to career decisions and work-life alignment.
3. "Pivot" with Jenny Blake - Specifically focused on career changes and transitions.
4. "The Tim Ferriss Show" - While not exclusively about careers, Tim often interviews successful people about their decision-making processes and life philosophies.
5. "Career Tools" - Offers practical advice on various aspects of career management and development.
6. "Squiggly Careers" - A podcast by the authors of "The Squiggly Career" book, discussing how to thrive in the modern workplace.
7. "Finding Mastery" with Dr. Michael Gervais - Explores the psychology of high performance, which can be insightful for making big career decisions.
8. "The Mindset Mentor" with Rob Dial - Offers motivation and strategies for personal growth, including career development.
9. "Happen to Your Career" - Focuses on helping people find and do work they love, often featuring stories of career changes.
10. "Jocko Podcast" with Jocko Willink - While primarily about leadership, Jocko often discusses discipline and decision-making in ways that are relevant to career choices.
11. "Slo Mo: A Podcast with Mo Gawdat" - Mo Gawdat, former Chief Business Officer at Google X and author of "Solve for Happy," discusses happiness, decision-making, and finding purpose in life and work. His unique perspective as a tech executive turned happiness researcher offers valuable insights for anyone contemplating major life or career changes.

Review: What I wish I knew when I was Twenty by Tina Seelig

Tina Seelig runs the Entrepreneurial Thought Leadership at Stanford, a very impressive programme with guest speakers from all the top companies sharing their experiences of past failures and successes - invaluable insights freely available for download.  If you're seeking out knowledge to be a successful entrepreneur, then I highly recommend you visit Stanford E-Corner website.

Back to the book: Very well written, easy language in direct conversation-style, brings out the message in clear and simple terms. Although this book could be seen as yet-another-self-improvement-book-on-leadership-innovation, in that it provides stories and insight's into other people's experiences & resultant lessons learnt; it is unique in the following aspects:

• References to real classroom exercises being taught at Stanford. Don't be fooled into thinking this is a student text, the example challenges can be given to any company team and will be equally, if not, more challenging than the classroom experience
• Tina touches upon subjects that are generally considered taboo
• There is an element of realism, practical advice that is good food for thought
• Topics are light-enough to leave the reader time to analyse his/her own personal situation (For example: the bit on "Failure Resume" (FR) really got me thinking, so much so that I exposed version 1 of my draft FR here)

Another topic that got me going was from the chapter "Turn Lemonade into Helicopters", Pages 129-130, which I'll quote below [I am still working on my own similes :-)]:

...In my course on creativity I focus a great deal on the value of recombining ideas in unusual ways. The more you practice this skill, the more natural it becomes. For example, using similes or metaphors, to describe concepts that on the surface seem completely unrelated offers tools for revealing fresh solutions to familiar problems.....Teams are asked to come up with as many answers as possible to the following statement:

Ideas are like ______________________________________

because __________________________________________

therefore __________________________________________

• Ideas are like babies because everyone think theirs is cute, therefore be objective when judging your own ideas ideas 
• Ideas are like shoes because you need to break them in, therefore take time to evaluate new ideas
• Ideas are like mirrors because they reflect the local environment, therefore consider changing contexts to get more diverse collections of ideas
• Ideas are like bubbles because they easily burst, therefore be gentle with them
• Ideas are like the measles because they are contagious, therefore hang out with other people with ideas if you want to get them yourself
• Ideas are like spider webs because they are stronger than they appear, therefore don't underestimate them

All-in-all, this is a useful addition to my book collection. It will no doubt be used time and again as a reference.  Don't be fooled by the title, the lessons taught are relevant to anyone throughout their personal/professional life, 20s, 30s, 40s and beyond...

How I Used Two AI Models to Re-Architect a Production Backend in 9.5 Hours

One planned. One executed entirely autonmously as lead AI agent co-ordinator (managing 3-4 parallel agents). Human watched each other and steered. Here's what happened.

---

The Problem Nobody Warns You About Your Fast-Built MVP

You build an AI feature on a shoestring budget leveraging free Azure services, like an SWA app (that I later migrated to a standalone service - future blog post). It works in dev. It works in staging. Then it hits production, and the platform underneath it starts fighting you.

Our AI chat feature — a financial intelligence analyst chatbot built on Claude — was deployed inside Azure Functions behind a Static Web App proxy. On paper, it worked. In practice:

• 45-second hard timeout from the SWA proxy killed any complex analysis
• HTTP 500 errors appeared randomly under load
• No reconnection — if your browser tab hiccupped, your 30-second AI response was gone
• Request-bound execution — the AI's "life" was the HTTP request. When the request died, so did the AI

The feature was unreliable enough that users had learned to retry. That's a product failure.

I decided to rip it out of Azure Functions entirely and move it to a standalone always-on service. Not a rewrite — a port. Same AI logic, new runtime.

The question was: could I plan this migration in a way that an autonomous AI agent could execute it end-to-end?

---

The Cast

This project involved three actors:

Me — the architect and orchestrator. I wrote the runbook, designed the execution plan, reviewed every decision, and verified the final result. A taste of the future role of an SDM (Software Development Manager)

Claude (Opus) — my planning partner and shadow observer. Claude reviewed the runbook across three iterations, helped me design the parallel execution strategy, drafted the system prompt for the executor, and then spent the entire execution day "snooping" on progress — reading the codebase in real-time and reporting back.

Codex — the autonomous executor and master agent coordinator. Codex received the runbook and a carefully crafted system prompt, then worked for ~9.5 hours straight spawning multiple agents on demand, producing 21 PRs, ~19,000 lines of code, provisioning Azure infrastructure, and deploying to production. Codex effectively did the job of a senior engineer working with 3-4 engineers

┌─────────────┐     reviews/plans       ┌─────────────┐
│             │◄───────────────────────►│             │
│   Human     │     system prompt       │   Claude    │
│  Architect  │────────────────────────►│  (Opus)     │
│             │     snoop reports       │  Reviewer   │
│             │◄────────────────────────│    Planner  │
└──────┬──────┘                         └─────┬───────┘
       │                                      │
       │  runbook + prompt                    │ reads codebase
       │                                      │ (read-only)
       ▼                                      ▼
┌─────────────────────────────────────────────────────┐
│                                                     │
│                    Codex                            │
│               Autonomous Executor                   │
│                                                     │
│   Thread A ─── Thread B ─── Thread C ─── Thread D   │
│  (critical)    (infra)      (proxy)    (diagnostics)│
│                                                     │
│              21 PRs │ ~19K lines │ 9.5 hours        │
└─────────────────────────────────────────────────────┘

---

Act 1: The Runbook

Why a Runbook, Not a Prompt

Most people using AI coding agents write prompts. I wrote a runbook — a 2,100-line execution plan that reads more like an engineering specification than a chat message.

Why? Because autonomous agents fail in predictable ways:

1. Scope drift — they start "improving" things you didn't ask for
2. Missing context — they make reasonable-sounding decisions based on wrong assumptions
3. No gates — they charge ahead past failure points without stopping
4. Sequential thinking — they do everything in order even when work can be parallelized

The runbook addressed all four:

• Explicit scope fences: "Port, do not rewrite. This is a runtime extraction, not a feature redesign."
• Complete architecture context: every Azure resource name, every file path, every dependency version, every API contract
• Stop gates at every phase: "Stop and escalate if any of the following is true..."
• Parallel thread definitions: five named threads with explicit entry gates and exit artifacts

The runbook went through three review rounds with Claude before I was satisfied. Each round caught real issues:

Round 1 identified that the parallel thread definitions were missing concrete entry/exit gates — they said what work to do, but not what must be true before starting or what artifact proves completion.

Round 2 caught that the integration-branch merge strategy was underspecified. Without explicit merge ordering, parallel agents would create conflicting merges.

Round 3 hardened the execution contract: what the agent can do without permission, what requires stopping and reporting, and how to handle the managed-identity-vs-connection-string decision for Cosmos DB.

The Migration Architecture

The target was clean:

Before:
  Browser → SWA Proxy → Azure Functions (aiChat.js monolith)
                         ↳ 3,356 lines, one file
                         ↳ 45-second timeout
                         ↳ request-bound execution

After:
  Browser → App Service (Express proxy)
              ├── /api/ai/* → Standalone AI Service (Fastify)
              │                ↳ Durable runs (survive disconnects)
              │                ↳ True SSE streaming
              │                ↳ Reconnect by runId
              │                ↳ No hard timeouts
              └── /api/*    → Azure Functions (unchanged)

The AI logic itself — prompts, tools, model selection, orchestration — would be extracted into a shared package (../ai-core) and reused by the new runtime. Port, don't rewrite.

---

The Architecture: Before and After

To appreciate what changed, you need to see what we were working with — and what we built to replace it.

Before: The Monolith

The entire AI feature lived in one Azure Functions HTTP handler. One file. 3,356 lines. Everything from authentication to prompt construction to tool execution to streaming — all coupled to the Azure Functions request lifecycle.

┌─────────────────────────────────────────────────────────────────────┐
│                        BROWSER                                      │
│   AIChatPanel.jsx ──POST /api/ai/chat──►                            │
│                                         │                           │
│   ◄──── raw SSE text stream ────────────┘                           │
│   (no reconnect, no runId, no resume)                               │
└─────────────────────────┬───────────────────────────────────────────┘
                          │
                          ▼
┌─────────────────────────────────────────────────────────────────────┐
│                  SWA / App Service Proxy                            │
│                  ┌──────────────────┐                               │
│                  │  45-second hard  │                               │
│                  │  timeout on ALL  │                               │
│                  │  proxied requests│                               │
│                  └──────────────────┘                               │
│                  /api/* ──► Azure Functions                         │
└─────────────────────────┬───────────────────────────────────────────┘
                          │
                          ▼
┌─────────────────────────────────────────────────────────────────────┐
│              Azure Functions  (app-name-backend-api)                │
│                                                                     │
│  ┌────────────────────────────────────────────────────────────────┐ │
│  │                  aiChat.js  (3,356 lines)                      │ │
│  │                                                                │ │
│  │  ┌──────────┐  ┌───────────────┐  ┌─────────────────────────┐  │ │
│  │  │  Auth    │  │  System       │  │  Anthropic SDK          │  │ │
│  │  │ Bootstrap│  │  Prompt       │  │  (buffered call)        │  │ │
│  │  │ Validate │  │  Builder      │  │                         │  │ │
│  │  └────┬─────┘  └───────┬───────┘  │  ┌───────────────────┐  │  │ │
│  │       │                │          │  │  Tool Dispatch    │  │  │ │
│  │       ▼                ▼          │  │  Loop             │  │  │ │
│  │  ┌──────────┐  ┌───────────────┐  │  │                   │  │  │ │
│  │  │  Feature │  │  Question     │  │  │  Claude response  │  │  │ │
│  │  │  Toggle  │  │  Classifier   │  │  │  ──► tool_use?    │  │  │ │
│  │  │  Check   │  │  Telemetry    │  │  │  ──► execute      │  │  │ │
│  │  └──────────┘  └───────────────┘  │  │  ──► feed back    │  │  │ │
│  │                                   │  │  ──► loop         │  │  │ │
│  │  ┌──────────┐  ┌───────────────┐  │  └───────────────────┘  │  │ │
│  │  │  Model   │  │  Tool Profile │  │                         │  │ │
│  │  │ Selection│  │  Resolution   │  │  request dies = AI dies │  │ │
│  │  │ Policy   │  │   Routing     │  │  no run state persisted │  │ │
│  │  └──────────┘  └───────────────┘  └─────────────────────────┘  │ │
│  │                                                                │ │
│  │  ┌─────────────────────────────────────────────────────────┐   │ │
│  │  │              Tool Executor (22 tools)                   │   │ │
│  │  │  query_products │ query_customers │ query_actuals       │   │ │
│  │  │  run_financial_model │ run_services_model               │   │ │
│  │  │  compare_fiscal_years │ get_margin_analysis             │   │ │
│  │  │  get_customer_concentration │ generate_chart            │   │ │
│  │  │  run_what_if_simulation │ ...12 more                    │   │ │
│  │  └─────────────────────────┬───────────────────────────────┘   │ │
│  └────────────────────────────┼───────────────────────────────────┘ │
│                               │                                     │
│                               ▼                                     │
│                    ┌─────────────────────┐                          │
│                    │     Cosmos DB       │                          │
│                    │  (products, budgets,│                          │
│                    │   actuals, configs, │                          │
│                    │   users, groups)    │                          │
│                    └─────────────────────┘                          │
└─────────────────────────────────────────────────────────────────────┘

Problems:
  ✗ Request-bound: browser disconnect kills the AI mid-thought
  ✗ No run persistence: nothing survives a dropped connection
  ✗ Buffered streaming: text arrives in chunks after provider finishes
  ✗ 45s proxy timeout: complex multi-tool analyses get killed
  ✗ No reconnect: lose your tab, lose your answer
  ✗ Monolith coupling: auth, prompt, tools, streaming all in one file

After: The Decomposed Architecture

The new architecture separates concerns across three layers: a proxy that routes traffic, a standalone AI runtime that manages durable runs, and an extracted core package that holds all the AI logic.

┌─────────────────────────────────────────────────────────────────────┐
│                        BROWSER                                      │
│                                                                     │
│   AIChatPanel.jsx                                                   │
│   │                                                                 │
│   ├── POST /api/ai/chat ──────────────► start run, get runId        │
│   │   ◄── SSE stream (attach immediately)                           │
│   │                                                                 │
│   ├── GET /api/ai/runs/:runId/stream ─► reconnect to live run       │
│   │   ◄── SSE replay (snapshot) + live deltas                       │
│   │                                                                 │
│   ├── GET /api/ai/runs/:runId ────────► poll status; final result   │
│   │                                                                 │
│   ├── POST /api/ai/runs/:runId/cancel ► user-initiated cancel       │
│   │                                                                 │
│   └── POST /api/ai/chat/feedback ─────► thumbs up/down on run       │
│                                                                     │
│   Client persists runId in React state                              │
│   Disconnect ≠ cancel (run continues server-side)                   │
└─────────────────────────┬───────────────────────────────────────────┘
                          │
                          ▼
┌─────────────────────────────────────────────────────────────────────┐
│           Web App Service  (Express proxy — server.js)              │
│                                                                     │
│   ┌─────────────────┐        ┌──────────────────────┐               │
│   │  /api/ai/*      │        │  /api/*              │               │
│   │  ──► AI Service │        │  ──► Azure Functions │               │
│   │  (dedicated)    │        │  (unchanged)         │               │
│   └────────┬────────┘        └──────────┬───────────┘               │
│            │                            │                           │
│   x-internal-proxy-secret      x-internal-proxy-secret              │
│   x-ms-client-principal        x-ms-client-principal                │
└────────────┬────────────────────────────┬───────────────────────────┘
             │                            │
             ▼                            ▼
┌────────────────────────────┐  ┌─────────────────────────┐
│  AI App Service (Fastify)  │  │  Azure Functions        │
│                            │  │  (non-AI APIs)          │
│                            │  │  /api/me                │
│  Dedicated B1 plan         │  │  /api/budgets           │
│  Always-on process         │  │  /api/products          │
│  No hard timeouts          │  │  /api/definitions       │
│                            │  │  etc.                   │
│  ┌──────────────────────┐  │  └─────────────────────────┘
│  │    Route Layer       │  │
│  │  /api/ai/chat        │  │
│  │  /api/ai/runs/:id    │  │
│  │  /api/ai/runs/:id/   │  │
│  │    stream            │  │
│  │  /api/ai/runs/:id/   │  │
│  │    cancel            │  │
│  │  /api/ai/chat/       │  │
│  │    feedback          │  │
│  │  /healthz  /readyz   │  │
│  └──────────┬───────────┘  │
│             │              │
│             ▼              │
│  ┌──────────────────────┐  │
│  │   Runtime Manager    │  │
│  │                      │  │
│  │  Admission Control:  │  │
│  │   2 active/user      │  │
│  │   2 queued/user      │  │
│  │   4 active global    │  │
│  │   8 queued global    │  │
│  │                      │  │
│  │  Run State Machine:  │  │
│  │   queued ──► running │  │
│  │     │         │  │   │  │
│  │     │         ▼  ▼   │  │
│  │     │    completed   │  │
│  │     │    failed      │  │
│  │     ▼    cancelled   │  │
│  │  cancelled_by_admin  │  │
│  │                      │  │
│  │  Watchdog: 10 min    │  │
│  │  Shutdown: 15s grace │  │
│  └──────────┬───────────┘  │
│             │              │
│             ▼              │
│  ┌──────────────────────┐  │
│  │   SSE Event Stream   │  │
│  │                      │  │
│  │  Contract (Phase 5): │  │
│  │   ready ──► status   │  │
│  │   ──► text(delta)    │  │
│  │   ──► tool_start     │  │
│  │   ──► tool_end       │  │
│  │   ──► text(delta)    │  │
│  │   ──► chart          │  │
│  │   ──► follow_ups     │  │
│  │   ──► done           │  │
│  │                      │  │
│  │  Reconnect mode:     │  │
│  │   text(snapshot)     │  │
│  │   + live deltas      │  │
│  └──────────┬───────────┘  │
│             │              │
│             ▼              │
│  ┌──────────────────────┐  │
│  │  Legacy AI Bridge    │  │     ┌──────────────────────────┐
│  │  (adapter pattern)   │──┼────►│        /ai-core          │
│  │                      │  │     │  (workspace package)     │
│  │  Wraps existing AI   │  │     │                          │
│  │  chat logic with     │  │     │  contracts/              │
│  │  new runtime hooks   │  │     │   sseEvents.js           │
│  └──────────────────────┘  │     │   interfaces.js          │
│                            │     │   uiContext.js           │
│                            │     │   chatFeedback.js        │
│                            │     │                          │
│                            │     │  prompts/                │
│                            │     │   systemPrompt.js        │
│                            │     │                          │
│                            │     │  tools/                  │
│                            │     │   definitions.js (22)    │
│                            │     │   anthropic.js           │
│                            │     │                          │
│                            │     │  orchestration/          │
│                            │     │   questionTelemetry.js   │
│                            │     │   modelSelection.js      │
│                            │     │   toolProfiles.js        │
│                            │     │                          │
│                            │     │  domain/                 │
│                            │     │   financialModel.js      │
│                            │     │   servicesModel.js       │
│                            │     └──────────────────────────┘
│                            │
│  ┌──────────────────────┐  │
│  │  Data Access Layer   │  │
│  │  (injected adapters) │  │
│  │                      │  │
│  │  getGroupDoc()       │  │
│  │  hasAiPermission()   │  │
│  │  getUserProfile()    │  │
│  │  canViewBu()         │  │
│  │  probeReadiness()    │  │
│  └──────────┬───────────┘  │
│             │              │
└─────────────┼──────────────┘
              │
      ┌───────┴────────┐
      │                │
      ▼                ▼
┌───────────┐  ┌──────────────────────────────────┐
│ Cosmos DB │  │  Azure Storage                   │
│           │  │                                  │
│ products  │  │                                  │
│ customers │  │  Table:  AiRuns (run metadata)   │
│ actuals   │  │  Queue:  ai-run-dispatch         │
│ configs   │  │  Blob:   ai-run-events           │
│ users     │  │  Blob:   ai-run-snapshots        │
│ groups    │  │  Blob:   ai-run-transcripts      │
│ rbac      │  │                                  │
└───────────┘  └──────────────────────────────────┘

Improvements:
  ✓ Durable runs: AI continues even if browser disconnects
  ✓ Reconnect by runId: pick up where you left off
  ✓ True SSE streaming: tokens arrive as they're generated
  ✓ No proxy timeout: dedicated service, no 45s wall
  ✓ Admission control: queuing, per-user limits, watchdog
  ✓ Graceful shutdown: in-flight runs drain before restart
  ✓ Separated concerns: proxy / runtime / core / storage
  ✓ Dependency injection: testable without Azure

The Run Lifecycle

This is the part that makes the new architecture fundamentally different. A "run" is now a first-class entity that exists independently of any HTTP connection.

  Client                    AI Service                     Storage
    │                          │                              │
    │  POST /api/ai/chat       │                              │
    │─────────────────────────►│                              │
    │                          │  create run (queued)         │
    │                          │─────────────────────────────►│
    │                          │                              │
    │  ◄─ SSE: ready{runId} ── │                              │
    │  ◄─ SSE: status{...}  ── │  transition: running         │
    │                          │─────────────────────────────►│
    │                          │                              │
    │                          │  ┌─────────────────────┐     │
    │  ◄─ SSE: text{delta} ─── │  │  Claude streaming   │     │
    │  ◄─ SSE: text{delta} ─── │  │  response arrives   │     │
    │  ◄─ SSE: text{delta} ─── │  │  token by token     │     │
    │                          │  └─────────────────────┘     │
    │                          │                              │
    │  ◄─ SSE: tool_start ──── │  Claude requests tool_use    │
    │                          │  execute tool (Cosmos query) │
    │  ◄─ SSE: tool_end ────── │  feed result back to Claude  │
    │                          │                              │
    │  ◄─ SSE: text{delta} ─── │  Claude continues response   │
    │  ◄─ SSE: text{delta} ─── │                              │
    │                          │                              │
    │       ╔═══════════════╗  │                              │
    │       ║ DISCONNECT!   ║  │                              │
    │       ║ tab closed    ║  │  run keeps going...          │
    │       ╚═══════════════╝  │  events persisted to blob    │
    │                          │─────────────────────────────►│
    │                          │                              │
    │  (reconnect)             │                              │
    │  GET /runs/:id/stream    │                              │
    │─────────────────────────►│  replay from storage         │
    │                          │◄─────────────────────────────│
    │  ◄─ SSE: text{snapshot}─ │  full text so far            │
    │  ◄─ SSE: text{delta} ─── │  live deltas resume          │
    │                          │                              │
    │  ◄─ SSE: follow_ups ──── │                              │
    │  ◄─ SSE: done ────────── │  transition: completed       │
    │                          │─────────────────────────────►│
    │                          │  archive transcript          │
    │                          │─────────────────────────────►│
    │                          │                              │

The key insight: the run outlives the connection. When you disconnect, the AI doesn't stop thinking. When you reconnect, you get a snapshot of everything that happened while you were away, followed by live deltas. This is what makes AI chat feel like a real product instead of a fragile demo.

---

Act 2: Designing the Parallel Execution Plan

This is where Claude earned its keep as a planning partner.

A naive sequential execution would have looked like: baseline → scaffold → extract → build runtime → add streaming → update proxy → provision infra → deploy → diagnostics → cutover. That's a 12+ hour critical path with no parallelism.

Instead, we designed five concurrent threads:

Thread	Scope	Can Start When
A (Critical Path)	Baseline → Scaffold → Extract → Runtime → Streaming	Immediately
B (Infrastructure)	Azure provisioning, deploy workflow	Immediately (parallel with A)
C (Integration)	Proxy routing, frontend run-client	After A delivers runtime + SSE contract
D (Diagnostics)	Cost ledger, health monitoring	After B provisions resources
E (Cutover)	Dark deploy, brownout test, go-live	After A+B+C converge

The key insight: infrastructure work (Thread B) has zero code dependencies on the AI logic (Thread A). You can provision Azure resources, set up managed identity, create storage primitives, and write deploy workflows while the extraction work is still happening. This saved hours.

Time ────────────────────────────────────────────────────────────────────────►

Thread A: [Phase 0][Phase 1][Phase 2a][Phase 2b][Phase 3][ Phase 5 ]
Thread B: [Phase 7-infra  ][Phase 7-deploy]
Thread C:                                       [P6-back][P6-front]
Thread D:                                                [Phase 8 ]
Thread E:                                                          [Phase 9]
                                                                      │
                                                              All threads
                                                               converge

The Integration Branch Model

With multiple threads writing code simultaneously, merge strategy matters. We used:

1. Integration branch (codex/ai-migration-main) as the merge target
2. Named feature branches per thread (codex/ai-migration-a-phase3, codex/ai-migration-b-deploy, etc.)
3. Explicit merge order: Phase 1 first → Thread B infra → Thread A checkpoints in order → Thread C after SSE freeze → Thread E last
4. Master updated only after verified integration checkpoints

This kept master stable throughout the entire migration.

---

Act 3: The System Prompt

The runbook was the what. The system prompt was the how.

Here's the actual prompt I gave Codex (sensitive values redacted):

## Mission
Execute the AI service migration runbook at
docs/TODO_MigrateAIFeatureToStandaloneAIServince.md.
That document is the single source of truth for scope,
architecture, sequencing, gates, and success criteria.

## Execution Model - Parallel Threads
Spawn parallel agent threads with explicit sequencing:

- Thread A (critical path): Phase 0 → 1 → 2a → 2b → 3 → 5
- Thread B (infra/deploy): Phase 7 infra starts immediately.
  Phase 7 deploy-workflow waits for Phase 1.
- Thread C: Phase 6 backend after Phase 3.
  Phase 6 frontend after Phase 5 SSE contract freeze.
- Thread D: Phase 8 after Phase 7 + runtime telemetry stable.
- Thread E: Phase 9 after A+B+C converge.

Start Thread A and Thread B in parallel immediately.

## First Actions
1. Run required preflight commands (Section 6)
2. Begin Thread A Phase 0: baseline + characterization tests
3. Begin Thread B Phase 7 infra: Azure provisioning
4. Use integration branch codex/ai-migration-main

## Operating Rules
- Follow the runbook literally
- Update Migration Status table at every checkpoint
- Stop on stop gates
- Merge order matters
- One PR per bounded checkpoint
- No rewrites — this is a port
- Never commit or log secrets

## Do NOT Need Permission For
- Creating branches, PRs, running CLI commands
- Provisioning Azure resources, setting GitHub variables/secrets
- Making the connection-string fallback decision if managed
  identity fails

## MUST Stop and Report If
- Any stop gate failure
- Any az or gh command fails due to auth/RBAC
- Any ambiguity where the runbook doesn't specify a clear path
- Any temptation to redesign rather than port

Design Principles Behind the Prompt

Three deliberate choices:

1. Point at the runbook, don't repeat it. The prompt says "that document is the single source of truth." Duplicating architecture context in the prompt creates drift — two sources that can disagree.

2. Explicit permission boundaries. The "do NOT need permission" section prevents the agent from stalling on confirmations for actions the runbook already authorizes. The "MUST stop" section catches the failure modes where human judgment is actually needed.

3. Parallel execution is called out explicitly. Without this, Codex defaults to sequential execution. You have to tell it to spawn concurrent work.

---

Act 4: The Miss — Worktree Isolation

Here's where I have to be honest about a gap.

The runbook specified parallel threads. The prompt told Codex to spawn them. But neither document specified how to physically isolate the parallel work. We described the logical dependency graph but not the physical isolation model.

When I checked on Codex's early progress, I found it was working sequentially in a single directory, mixing Phase 0 and Phase 7 changes in the same checkout. The parallel execution model existed on paper but not in practice.

This matters because git doesn't handle concurrent modifications to the same working directory gracefully. You need separate worktrees — independent checkouts of the repository where each thread can work without stepping on the others.

I had to inject a mid-session correction:

"Each parallel agent thread should work in its own git worktree with a named branch convention (e.g., codex/ai-migration-a-phase3). The orchestrating agent uses the primary checkout for coordination only."

Codex received this, stashed its in-progress work, split changes into proper branches, created worktrees, and adopted the model. It recovered well, but the gap shouldn't have existed.

Lesson learned: the DAG (dependency graph) is necessary but not sufficient. You need to specify the physical isolation model — worktrees, branch naming, merge-back protocol — not just the logical thread dependencies.

I've since added this to my standard checklist for any multi-agent execution plan.

---

Act 5: The Shadow — 12 Snoop Reports

While Codex worked, I periodically asked Claude to observe the codebase and report on progress. Claude had read-only access — it could inspect files, read git history, check PR statuses, and review code quality, but couldn't modify anything.

This created a powerful feedback loop: one AI executes, another AI reviews, and a human makes decisions based on both perspectives.

Here's the timeline of observations:

Report	Time	Finding
#1-#5	Early	Phase 0 + Phase 7 infra done. Worktree issue identified.
#6	Mid	Phase 2b merged. 1,863 lines extracted into ai-core with parity tests.
#7	Mid	Phase 3 worktree created but no commits yet (Codex working locally).
#8	Mid	Phase 3 merged. 916-line runtime manager with full run state machine.
#9	Late	Phase 5 + Phase 6 backend both merged. True streaming + proxy split.
#10	Late	Phase 6 frontend merged. Durable run client with reconnect.
#11	Final	All phases complete. 20 PRs merged. Integration promoted to master.
#12	Post	Hotfix for missing AI_CHAT_BOOTSTRAP_SECRET on AI service.

What the Shadow Caught

The snoop reports weren't just progress tracking. They caught real issues:

1. Lost artifact: The Phase 2a SDK migration audit was written to a gitignored directory. When the worktree was removed, the artifact disappeared. Flagged in report #6.

2. Session continuity gap: Codex sessions have quota limits. If a session expires mid-work, the next session needs to know where to pick up. We drafted a HANDOVER.md protocol — a structured file that captures current phase, what's done, what's blocked, branch state, and next action.

3. Code quality validation: Report #8 confirmed the Phase 3 runtime manager had proper admission control, watchdog timers, graceful shutdown, and queue drain — not just placeholder stubs.

4. Contract compliance: Report #10 verified the frontend was consuming the Phase 5 SSE contract correctly — text(mode=delta) for live tokens, text(mode=snapshot) for reconnect reconstruction.

---

Act 6: The Numbers

Timeline (March 31, 2026)

Time	Event
08:52	First runbook commit
09:31	Codex starts — PR #4 (Phase 7 infra)
09:41	PR #5 (Phase 0 baseline)
10:16	PR #6 (Phase 1 workspace scaffold)
12:41	PR #12 (Phase 2a contracts)
13:27	PR #13 (Phase 2b extraction — 1,863 lines)
14:29	PR #14 (Phase 3 runtime — 916-line manager)
15:08	PR #15 (Phase 6 backend proxy split)
15:28	PR #16 (Phase 5 provider streaming + SSE freeze)
15:59	PR #17 (Phase 6 frontend run-client)
17:30	PR #18 (Phase 8 diagnostics)
19:07	PR #19 (Phase 9 cutover)
19:10	PR #20 (Master promotion)
Total	~9.5 hours, 21 PRs, ~19,000 net lines

Code Produced

Component	Lines	Purpose
ai-service/	~4,500	Fastify runtime, routes, run manager, storage
packages/ai-core/	~3,800	Extracted AI logic, contracts, domain models
Infrastructure scripts	~1,700	Provisioning, verification, packaging
Deploy workflow	~250	CI/CD pipeline
Frontend changes	~1,100	Run-aware chat panel, API client
Tests	~1,500	Characterization, parity, integration, contract
Remaining	~6,000+	Internalized API libs, domain model moves, etc.

What Got Deployed

• Standalone Fastify AI service on app-name
• Dedicated App Service plan, storage account, managed identity
• Express proxy splitting /api/ai to AI service, /api to Functions
• GitHub Actions deploy workflow with pre/post-deploy health probes
• Run-aware frontend with reconnect, cancel, and status polling

---

Act 7: The Hotfix

Twenty PRs merged. Three deploy workflows green. Dark deploy verified. Brownout rollback proven.

Then I opened the chat panel and got "Access Denied."

The AI_CHAT_BOOTSTRAP_SECRET — a shared secret used to verify frontend bootstrap tokens — was configured on the Azure Functions app but had never been provisioned to the new AI service. The dark deploy health probes (which test /healthz and /readyz) didn't catch it because they don't send authenticated chat requests.

Codex fixed it in one commit: added the secret to the provisioning script, the verification checklist, and the readiness probe. PR #21 merged and deployed.

Lesson: Dark deploys verify infrastructure health, not end-to-end user flows. You need both.

---

What I Learned About Multi-Model Development

The Reviewer/Executor Pattern

The most powerful pattern I discovered is using two AI models in complementary roles:

• Model A (Claude Opus): Planner, reviewer, and observer. It reviews the plan before execution, monitors progress during execution, and validates quality after execution. It never touches the code directly.

• Model B (Codex): Autonomous executor. It receives a plan and system prompt, then works independently — creating branches, writing code, provisioning infrastructure, creating PRs, and deploying.

This separation of concerns mirrors how engineering teams work: architects review, developers execute, and neither role is diminished by the other.

How to Write Plans for Autonomous Agents

1. Be exhaustive about context. Every file path, resource name, dependency version, and API contract. Agents can't ask clarifying questions mid-execution the way humans can.

2. Define stop gates, not just tasks. "Do X" is a task. "Do X, but stop if Y is true" is a gate. Gates prevent agents from charging past failure points.

3. Specify physical isolation, not just logical dependencies. Parallel threads need worktrees, branch naming conventions, and merge protocols — not just a DAG.

4. Include permission boundaries. "You can do X without asking" prevents stalling. "You must stop for Y" prevents runaway execution.

5. Plan for session turnover. Long-running agents hit quota limits. Build in handover protocols — structured files that capture state for the next session.

How to Monitor Autonomous Agents

The "snoop" pattern — having a second model periodically observe the codebase — turned out to be valuable for three reasons:

1. Progress tracking without interruption. The observer reads git history and file state without disturbing the executor.

2. Quality validation in real-time. The observer can assess whether extracted code maintains behavioral parity, whether contracts are properly frozen, and whether test coverage is meaningful.

3. Issue detection before merge. The observer caught the lost SDK audit artifact and the session continuity gap before they became blocking problems.

The Human's Role Changes, But Doesn't Disappear

I didn't write 19,000 lines of code. But I:

• Designed the migration architecture
• Wrote a 2,100-line runbook that eliminated ambiguity
• Reviewed the plan across three iterations with an AI partner
• Identified the worktree isolation gap and injected a correction mid-flight
• Monitored execution through 12 observation cycles
• Made the "big bang, fix forward" cutover decision
• Performed the first manual end-to-end test that found the bootstrap secret issue

The human role shifts from writing code to writing specifications precise enough that code writes itself. That's a different skill, but it's still a skill — and it's the skill that determined whether this project took 9 hours or 9 days.

---

Try This Yourself

If you want to experiment with multi-model development:

1. Start with a runbook, not a prompt. Write down everything the agent needs to know before it starts. If you find yourself thinking "it'll figure that out," write it down instead.

2. Use one model to review the plan before another executes it. The reviewer will catch ambiguities, missing gates, and underspecified contracts.

3. Set up a shadow observer. Ask a model to periodically read your codebase and report on the executor's progress. You'll catch issues faster than waiting for CI failures.

4. Plan for failure. Include stop gates, rollback strategies, and session handover protocols. Autonomous agents don't get tired, but they do get stuck — and they need structured ways to communicate that.

5. Test the real user path. Health probes are necessary but not sufficient. Someone needs to open the browser and click the button.

The future of software development isn't AI replacing developers. It's developers who can orchestrate AI systems effectively building things that would have been impractical before.

The runbook and system prompt used in this project are shared below for anyone who wants to adapt this approach.

Appendix: The Full Runbook

Below is the complete 2,100-line migration runbook I wrote to drive this project — the single document that governed every phase, gate, and decision. All Azure resource names and company identifiers have been redacted. The structure, sequencing, execution model, and stop gates are authentic.

This is the actual artifact that Codex followed autonomously for 9.5 hours. If you want to adapt this approach for your own projects, this is your template.

Scroll within the frame to read the full runbook. The original is approximately 2,100 lines covering 22 sections.

---

Mo Khan is just an old-timer engineer-turned-manager, who left coding a long time ago, and is having so much fun learning again and building with AI tools with a special interest in AI-augmented development workflows, cloud architecture, and autonomous agent orchestration.