My experiment with ChatGPT as a recommendations engine using Netflix data

As some readers might have learnt about me by now, is that I enjoy data analytics, and more recently been curious about the capabilities of ChatGPT. So I decided to take my Netflix data and give it to ChatGPT code interpreter to see what it comes up with.

If you didn't know, as a Netflix user, you can get access to all your data by requesting it from under account management on the web site:

After a couple of days, you will receive an email from Netflix with a zipped file "netflix-report". Here's what my archive looks like:

The section that's important is CONTENT_INTERACTION. Recommendation engines are all about understanding a user's viewing behaviour: their content preferences, search history, viewing history, events that happen during playback, viewing activity - all inputs into what is called "viewer engagement". 

For 20 years I worked as a software professional building video technology systems. In 2004, I joined a research team building video recommendation engines. At the time, we focused on rules-based engines and used Amazon as the general poster-child, north-star - even then, Amazon's retail recommeder system was making headwaves. In 2004, it was still the early days of the internet. Set Top Boxes, the devices that enabled you access to satellite TV, were not always online. So the scope of our work was constrained to the embedded world of this device: limited compute, memory and storage power. We knew there needed to be a multi-pronged approach to video recommendations back then, with the bulk processing intelligence being done in the "back end" - headend - and recommendations eventually broadcast to the set top box over the air. Anyway, building software that learns viewing behaviours of users was an art back then - and even almost two decades later - remains still an art - although with the proliferation of many software-as-a-services providers, we now have so many options available "Recommendation engines-as-a-service". In my previous role as CTO for Africa's largest video entertainment company, over the years built our own stack, leverage open source recommendation engines. With new capabilities like Amazon Sagemaker and other services, I got our team to start thinking differently and transform to out-of-the-box recommender systems. Don't reinvent the wheel. Build flywheels instead. I left that team in 2020, 4 years later, I'm told the team's completed their transformation and using almost all their recommendation components off-the-shelf instead of building their own.

So what happens to content recommenders now that we have tools like ChatGPT, code interpreters and powerful data analysis? Could video recommendation engines be further disrupted - i.e. democratizing content recommendations to the new world of personal intelligent agents?

Experiment: ChatGPT as a content recommendation expert - Epic Fail

First, I asked ChatGPT to analyse the Netflix data set, then provide an analysis as a video recommendations expert. I've shared my Netflix data. Act as an expert video recommendations engine, recommender system, using all the state-of-the-art concepts for understanding viewer behaviour. Analyse the datasets and produce a detailed analysis and insights report into my viewing habits. What are my preferences? If you were a video content expert, how would you categorise me? What content will you recommend to me? Etc.

Feedback - Results, much better luck with DeepNote

A bit disappointing - ChatGPT struggles to analyse consistently, without errors. Too many errors, load failures and errors analyzing. It starts strong and then bombs out. I find it to be rather inconsistent. Had to wait a while and create a few sessions to get going. A bit of a time waster. It's a shame that ChatGPT has to restart its chain of analysis everytime there's a failure. "Regenerate" kicks off the process all over again. 3 sessions later, then I get bombed out as I've exhausted my time window - come back later. I pay for ChatGPT and still have to accept limited time-based usage. Crazy times. Switched over to Claude to see if it gave better results. Nope, Claude has severe limitations in file size uploads. Waste of time. Google Bard, equally useless. 

I then switched to Deepnote (an alternative to Noteable which is now dead), my experience was much better there. It turns out the Netflix data is sparse enough, missing vital data attributes like content genre classifications (like action, comedy, drama, romance, history, etc.) which makes it difficult to draw conclusions on viewing behaviour. This means we need to find a content database like TMDB to map my Netflix data to (i.e. complete the dataset by appending vital content attributes like genre, actor, director, studio, etc.), then we can build up a complete profile view of my usage, the content my household is inclined to and the for the last phase of the experiment, get the AI to recommend content from another catalog to surface recommendations for me. To get a glimpse of the potential power of an AI-coPilot used in DeepNote, I've shared my project as public here.

I'm not investing more time going down this rabbit hole today! However, it is certainly food for thought on potential new GPT-agents one can build now that OpenAI have released its GPT store. I suspect it's only a matter of time that someone builds a content recommender-gpt on ChatGPT's store, that can provide more diverse recommendations outside the closed walls of a streaming provider. I suspect we will have a resurgence in Content-Aggregator Wars in 2024!

Personametry 2023 - Wheel of life streams, hours logged

It's that time of the year when I sit down and review my time tracking data for the year. This time round, playing around with ChatGPT (whilst still maintaining my AWS Quicksight dashboards). I was quite looking forward to continuing to use Noteable's AI/Data tools, sadly the service shut down :-(

I've shared my dataset openly and also you can look at the ChatGPT transcript of the conversation and prompts here.

ChatGPT's overall summary:

• Rest and Sleep (Individual): A substantial 2,856.11 hours, akin to the restorative power of a long, rejuvenating sleep every night.

• Professional Work: Impressively, 2,051.20 hours were dedicated to your professional life, showing a solid commitment to career growth and achievements.

• Family Time (Family-Man): A heartwarming 1,398.79 hours spent with family, which is like embarking on a year-long road trip with your loved ones, filled with bonding and cherished memories.

• Spirituality (Individual): A profound 1,106.44 hours in spiritual pursuits, reflecting a journey akin to a spiritual retreat, deepening your inner peace and understanding.

• Personal Time (Individual): 467.55 hours of 'Me Time', ensuring a healthy balance of personal space and activities, like taking a short, refreshing break each day.

• Marital/Wife (Husband): 451.91 hours dedicated to marital life, resonating with the quality time spent in building a strong, supportive partnership.

• Knowledge and Learning (Individual): 202.88 hours of enriching your mind, like attending a series of insightful lectures throughout the year.

• Health and Fitness (Individual): 145.07 hours, a commitment to staying active and healthy, similar to a consistent routine of short, daily workouts.

• Social (Friend): 117.06 hours, akin to enjoying meaningful social gatherings, reinforcing bonds and making memories.

Looking back, my time tracking is reaching steady state. Not sure I'm going to keep reporting these stats going forward, but I'll probably keep fine-tuning. For 2023 Scorecard, I'm glad to have focused and met largely my spiritual goals, work/professional goals was maintained year-on-year, despite the spike in work hours in the last quarter (I got "promoted", with extra responsibilities which means more work hours). Health and Fitness hours took a dive compared to 2022, but it seems that my Strava/Garmin activity tracking shows better performance in 2023 compared to 2022 (so more time logged doesn't necessarily equate to being for fit, or achieving more). Social time decreased because we lost some friends to emigration, and generally our year was focused on the kids as my son approached his final high school matric exams, we prioritised staying at home.

If you've read all my previous posts on Personametry, you would've learnt that as part of this experiment, even way back when I started in 2016, I wanted a personal-assistant AI to help me with my RAGE / Personametry framework. I wanted to engage in meaningful conversations with an AI-assistant, coach me, act as my council, gatekeeper, motivator, etc. I believe there is still a need for this and the current state-of-the-art still has some catching up to do. Nevertheless, I am quite enjoying my interactions with ChatGPT. It was able to understand my personametry data, derive some insights, create the views I would generally take a couple hours doing myself. It's just a matter of time until we get truly personalized agents, like a personametry-gpt agent. Here's what ChatGPT offered I should look into for 2024:

Here's my Personametry slides as usual:

46 today

I asked ChatGPT to write a poem, I turned 46 today.  

In honor of your journey and the resilience you've shown, here's a poem that weaves in elements of your life story:

From Natal's fields to corporate climbs,

In the land where the sugar cane twines,

A boy with a dream, in apartheid's times,

Grew with a flame that brightly shines.

Hitching a ride, 600km through the night,

To an interview held in morning's light,

Persistence his creed, ambition his might,

Against all odds, he would take flight.

Engineering called, though medicine's voice was clear,

Yet no regret, for tech was near.

In computers found, a passion so dear,

Self-taught, self-made, amidst career.

Mo, at forty-six, your story's just begun,

The lion's roar can't be undone.

Your flame rekindled, the past is spun,

Into a future brightly won.

On this day, let's not be remiss,

Celebrate the life that is Mo's bliss,

Forty-six years, it's hit or miss,

But Mo's tale, is one of grit and wist.

So happy birthday, with a cheer so loud,

For Mo stands out, in any crowd.

At forty-six, be humbly proud,

For your life’s story speaks volumes, unbowed 

————

At forty-six, the heart still ticks,

A journey through life’s blend and mix.

Not young nor old, but squarely mid,

With years of wisdom, no longer a kid.

Each wrinkle earned, a tale to tell,

Of laughs and tears, and times we fell.

At forty-six, life’s sweet refrain,

Plays on in joy, despite the pain.

So raise a toast to all those days,

The quiet growth in subtle ways.

At forty-six, still much to do,

Another year of being true.

For in the mirror, the truth’s not stark,

It’s just a number; light a spark.

Embrace the day, let’s not be brisk,

Life’s rich at the age of forty-six.

———

At forty-six, not just a number, but a force,

Rekindling fires from a powerful source.

A story of grit, hard work, and might,

From Natal's fields to tech's soaring heights.

A father's legacy, a machinist's son,

To a leader of tech, second to none.

In every challenge, a lion's heart,

Against life's storms, a steadfast part.

So rise, be bold, reclaim your flame,

In this year's chapter, make your name.

Forge ahead with courage, let passion dictate,

At forty-six, it's your story to create.

Wrap up '23 with my Personal Scorecard RAGE / Wheel of Life review

In 2016 I started to be more intentional with designing my lifestyle across my personal and professional streams. Without studying deeply the literature or taking personal development training courses, I used my  own work experience as a professional engineer to guide me in designing the framework, which I've called "RAGE" that stands for Reality Aspirations Goals Expectations. At the time, I treated myself as a product I'm building, using similes from agile software development, using the concept of Personas (typically products are designed to meet the needs of different user types, i.e. "personas"). Bringing these concepts into personal lifestyle design, I described my aspirations, goals and expectations in the form of agile user stories. 

This helped me in framing my goals to realise tangible outcomes. Almost eight years later, I've maintained discipline in applying the RAGE framework, year-on-year in my pursuit of the various aspirations defined in each persona, i.e. as a Husband, Father,  Professional, Friend, Individual, etc. Coming from an engineering background, I dived deep into prioritising these streams of life, first starting with 20+ personas, having stabilized on a handful of core personas from 2020 onwards.

Interestingly enough, this post, led me to learning about something called "The Wheel of Life". It so happened, according to ChatGPT, the exact date of the creation of the "Wheel of Life" model by Paul J. Meyer is not well documented in public sources -- but it is known that Paul J. Meyer founded the Success Motivation Institute, Inc. in 1960. Honestly, I'm not sure how I missed Meyer's model all this time, but I'm also comforted by the fact that I've created my own model from scratch through my own attempts in crafting a better version of myself through my RAGE model. Anyway, ChatGPT has helped summarize the differences between Meyer's model and Mo Khan's RAGE model in this post. I do like the radial diagram, which is something I hadn't included in my personametry insights as yet.

I'm writing this post in December 2023, the last post for the year - as I break away from the world and embark on a journey making the minor pilgrimage to Mecca (Makkah) for the first time in my life. I've waited a long time to save up for this journey, I made intention at the start of 2023 to do the Umrah, and by the grace of Allah (God), it is about to come true, finishing off 2023 with my most important '23 goal realised - Subhanallah (Glory be to Allah)! And aiming for Hajj (the major pilgrimage) in 4 years time insha Allah (God willing).

Work-wise, my last week was spent completing end of year performance reviews. At Amazon, we treat performance reviews with such seriousness, due care and attention to detail -- that I've never seen done before in other companies I worked with (a post for another day).  For some time now, I have taken mechanisms from my work life and adopted to my personal life. Just as in my RAGE model, breaking down the different domains of life-work streams, I've used business performance scorecards in my tracking progress through the year of my personal performance. If we maintain work logs, performance cards, "baseball cards" for our jobs, is not more important to track our all our streams of life?

Why should we only focus on professional performance reviews and not spare a moment to deeply reflect our performance as a human on this earth? How are we doing personally? How are we performing across the various dimensions of our life? If we live by a value system, how do we know how we're measuring up to that value system? When the year has come and gone, how do you measure progress? Is it only career that matters? Surely not? If not, then why don't we spend as much time, if not more time, doing our own self-assessment and scorecards?? Are we afraid of the reality check? How serious are we in terms of changing ourselves for the better? How are we improving? Do we need to improve? When someone asks you "How's life? How you're doing?" - what's your frame of reference for answering that question? What's the first thing that comes to mind? Is it our work, career? If we're going through a bad patch at work, does this mean our whole life is negative? Often this is exactly what happens. People anchor on the immediate negatives, closest, most recent experiences because without a holistic value system to reference, how should one respond to "How are you doing, really?"

So how am I doing, really - what's my 2023 Performance Scorecard like?

I am sharing my own personal tracking here, not as a means to show off - but as a means to help people, providing an example of how I've managed to use a system that is working for me. This system, called RAGE, may not work for you - but it should give you an idea of the kind of work you need to do, the kind of thinking you need to apply, and the kind of discipline your should consider -- if you are indeed determined to make a concerted effort at changing your current situation, and creating a new trajectory you can aim for to improve the quality, outcome and other aspirations you desire. 

Inspired by Vaugan from Scary Management Playbook's LinkedIn post, here is my Wheel of Life, unpacked in detail, according to my RAGE model. In January '24, I will follow up with the data analytics on time spent in my personas. Given my RAG status, it looks like my wheel of life pretty balanced and aligned with my life-work-stream choices.

My RAGE model compared to The Wheel of Life

 

[I asked ChatGPT to report on the differences between Meyer's Wheel of Life and Mo Khan's RAGE]

Comparative Report: Meyer's Wheel of Life vs. Mo Khan's RAGE Model

Introduction

This report provides a comparative analysis of two influential personal development tools: Meyer's Wheel of Life and Mo Khan's RAGE Model. Both models are designed to enhance self-awareness and personal growth, yet they differ significantly in their approach and methodology.

Meyer's Wheel of Life

• Overview: The Wheel of Life is a holistic self-assessment tool designed by Paul J. Meyer, a pioneer in the field of motivational thinking and self-improvement.
• Components: It typically includes segments such as Career, Personal Growth, Health, Family & Friends, Finances, Spirituality, Recreation, and Physical Environment.
• Function: Users rate their satisfaction in each area, visually representing life balance and identifying areas for improvement.
• Application: Widely used in coaching and self-help, it guides personal development and goal setting.
• Learn More: Success Motivation Institute

Mo Khan's RAGE Model

• Overview: The RAGE Model, an acronym for Reality, Aspirations, Goals, Expectations, is a framework for personal development conceptualized by Mo Khan.
• Components: It involves introspection across various life 'personas', assessing the current reality, setting aspirations, establishing goals, and managing expectations for each persona.
• Function: The model uses agile management techniques and a detailed scoring system for prioritization and tracking progress.
• Application: Khan’s approach is detailed and systematic, suitable for those who prefer an analytical approach to personal development.
• Learn More: Mo Khan’s Blog

Similarities

1. Holistic Approach: Both models provide a comprehensive view of personal life, emphasizing multiple areas or aspects.
2. Self-Reflection: They encourage self-assessment as a means of identifying areas of focus.
3. Goal-Oriented: Each model promotes setting and pursuing personal goals.
4. Personal Development: They are tools for enhancing self-awareness and guiding growth.

Differences

1. Complexity: The Wheel of Life is simpler and more visual, while the RAGE Model is more complex and analytical.
2. Approach: Meyer’s model is qualitative, focusing on self-rated satisfaction, whereas Khan’s model is quantitative, employing a scoring system.
3. Focus Areas: The Wheel of Life has predefined life segments; the RAGE Model allows for personalized 'personas'.
4. Tracking: The RAGE Model includes rigorous tracking mechanisms, unlike the Wheel of Life.
5. Background: Meyer's model stems from motivational training, while Khan’s is influenced by agile product management.

Conclusion

Meyer's Wheel of Life and Mo Khan's RAGE Model, while sharing the goal of fostering personal growth, differ in structure, complexity, and application. The Wheel of Life is a straightforward tool for periodic self-assessment, ideal for a broad audience. In contrast, the RAGE Model offers a detailed framework for systematically setting, prioritizing, and tracking personal development goals, appealing to those who prefer a more structured approach. Both models provide valuable insights into personal development, and the choice between them depends on individual preferences and the desired level of detail and structure in personal growth planning.