PayTV Operators should STOP re-inventing the wheel and should NOT see Google as Evil!

I have been working within the PayTV Software & Systems space for almost fifteen years now, and from the very beginning with Set-Top-Box software, I really wasn't impressed by the software technology. Apart from the hardware being fairly interesting, basically a device for decoding video/audio/data streams based on MPEG / DVB / ATSC / etc protocols and standards, when it came to software, there wasn't really a "wow" factor in it. Of course, we can't forget the really crucial element, the heart of the system, the crown jewels, the revenue-generator, the very interesting & complicated black magic technology called Conditional Access (CA) which is really really cool, the rest of the building blocks was really around the Set-Top-Box user experience / application, which really wasn't all that new: 

• Essentially the device needed an operating system, a way to draw stuff on-screen, a user navigation interface, and some data source driving the application, traditionally called the Electronic Program Guide (EPG). The STB software thus re-applied knowledge well-known in the computer industry for decades (Model-View-Controller MVC design pattern emerged in the late 1970s), simple operating system and driver / hardware abstraction layers, C-code...

As all hardware devices tend to follow Moore's law at some stage, Set-Top-Boxes (STBs) have evolved to quite powerful machines allowing for migration to newer, modern software implementations (although nowhere near the computing power & rich experience offered in the latest smartphones & tablets - post for another day), the software too, has become more accessible than ever, with more STBs using Free Open Source Software (FOSS), particularly the dominance of the Linux Kernel as the popular operating system of note, displacing historical dominance of VxWorks, STOs, NucleusOS, uCos, etc, etc.

However, there are some components in the STB software stack that remain fundamentally closed: The STB Middleware, EPG, Conditional Access (CA) clients. Okay, ignoring the CA client, which has always been fundamental and will never go away for years to come -- the classic Middleware/EPG components really don't need to be that closed anymore. 

There is also the backend / headend information service data generators that are traditionally closed, although vendors purport "open protocols", the PayTV Operator generally obfuscates the openness by forcing business-specific modifications in the protocols, apparently unique to each PayTV Operator/Broadcaster...

Traditionally & historically though, these software components were provided by third-party vendors that PayTV Operators just accepted as the norm. Highly closed, difficult to integrate with open systems, these vendors capitalised on providing a closed system, to the extent of locking in the PayTV Operator to the entire stack, some vendors reaped the benefits of providing the end-to-end system, one-stop-shop for everything. Later, PayTV operators decided to take more control, diversify the ecosystem by enforcing the use of multiple components, not being locked-in by just one vendor, promoting open standards for integration, and more recently taking more ownership of some of the development and integration...

Yet, the models within which most PayTV operators continue to work - is still pretty much a closed one. There is an aversion to sharing, opening up technology to other parties with a view to extending partnerships as well as creating new strategic relationships. There is an huge element of mistrust, not-invented-here attitude, we-can-implement-this-in-house, etc, etc!! I am really dumbfounded with that approach...

Agile Africa 2013 Conference - Day Two

Yesterday was Day Two, concluding the very first Agile Conference in Africa, an event that was organised in a short space of time (2 months) by a team of just five people - hats off to the event organisers and sponsors. Indeed it was quite a significant milestone, a stake firmly grounded that Agile in Africa is growing. 

The second day was filled to the brim by an impressive lineup of speakers, a bit over ambitious in my humble opinion, since sometimes the audience needs more time with the keynote speakers (1 hour isn't enough), and on starting late and not keeping to time, something that could be improved for the next conference. Having said that, the day was awesome, personally, I took away a different kind of appreciation for the companies in Africa, bold enough to push Agile adoption, the openness and willingness to experiment and being OK with failure...

The talks that really resonated well with me included Martin Fowler's keynote, Ana Forrsman's review of JSE's agile experiences, Derrydean Dadzie's first time mistakes clash of Corporate/Start-up culture & Aslam Khan's sharing of personal challenges of teams operating Beyond Apartheid & Democracy (so relevant today, but people walk on eggshells around this. I myself have observed, even today, the subtle office racial nuances at play even in 2013...). I also appreciated Ivar Jacobson's first-time presentation introducing the SEMAT kernel, although I really had background knowledge of this concept from earlier this year - I think most people left that talk wondering more about the wider implications of SEMAT kernel and effects of "formalising" a way of working or thinking when it comes to producing software...

I have briefly summarized my take-away and notes here:

• Software Design in the 21st Century - Martin Fowler
• Agile and SEMAT - Perfect Partners - Ivar Jacobson
• To be Agile or Not to be Agile - JSE Corporate Perspective - Ana Forssman
• Doing Agile in a brutally waterfall enterprise environment - Derrydean Dadzie
• Enterprise Agile Adoption with Parental Guidance - Riaan du Toit
• Beyond Apartheid & Democracy - Aslam Khan
• Beyond hype and dogma: How South African teams increased their value - Martin Cronje
• Measuring Throughput - Danie Roux
• Software Development Revolution in Africa - Ernest Mnkandla
• Questions and Answer Forum

Agile Africa 2013 Conference - Day One

I attended the inaugural Agile Africa Conference in Johannesburg yesterday, hosted by the JCSE & ThoughtWorks, sponsored by an interesting array of partners. A two-day conference hosting guest speakers from all over, quite an impressive list of speakers.

I attended more out of curiosity to gauge the talent in Africa/South Africa, as I've been quite skeptical about the breadth, depth and appreciation of software engineering practices in Africa, due to my limited exposure to the local industry after having spent ten years in Europe working with Agile from way back in 2004/5, coming back to work SA felt like I'd been set back by the way software was done 15-20 years ago. Not only that, it seems corporate structures of large companies in SA are still set in this mindset that is so out of date with current modern day practices...

In my two years of being in SA, I've touched base with a few consultants and quite honestly I was not overly impressed. I felt that people were just jumping on the Agile bandwagon, without fully appreciating what adopting Agile really meant, the impact on company organisation, culture and productivity. My limited experience & interaction with these Agile Trainers also led me to believe it was just the next gravy train, people went on Agile courses, got certified as Scrum Masters / Coaches and then went out to offer expert training without being in the trenches themselves, seeing the transformation of teams through this process of Agile Adaptation / Evolution, basically training us without any substance/depth.

So I was looking forward to getting an insight into the Agile community in South Africa - and boy, was I pleasantly surprised! The turnout for the conference was impressive, the Alex Theatre in Braamfontein was almost full to capacity - there is indeed lots of interest in Johannesburg alone. The line of local speakers impressive (respect!). The presentations were world-class, engaging, interactive, inspiring, motivating - but most of all, they echoed the values that I've shared on Agile all along, renewing my own energy and passion, since the last two years have been quite a struggle to get a company to move from one level of Agile maturity to the next level, in the context of a major product delivery, in what's become quite a difficult and rather ambiguous Agile journey.

What I've also taken away is that the speakers haven't really conveyed truly enlightening insights to me, that I myself have been in Agile for 8+ years, and I've built up my own body of knowledge (through my own work experience, training & readings) that I think is possibly worth sharing, especially with companies working with Large-Scale Software Systems in Digital TV projects. The white papers I've written have been my first attempt at distilling this knowledge, but I realise now (well, this has been on my mind for a long time now) that I need to do more, since quite frankly, the medium isn't really working for me. My white papers are long and tedious to read, often my readers suffer from Communication Saturation. I've made strides at shortening my posts, using visualisations, etc - but now I think I should really put myself out there, ask for feedback on my writings from these esteemed speakers, and also take the leap into presenting my own case studies at similar conferences! Time will tell...

I've summarized my personal key take-aways from the first day here:

• Successful Agile Transitions: Patterns to ensure success (or failure) - Mitch Lacey
• Agile for Creative Industries - Trevor Kimenye
• Agile Testing Mindset - Sam Laing & Karen Greaves
• Teams and Sticky Notes versus Organisational Agility! - Arrie van der Dussen
• The Process Increments Method: Case studies from Egypt - Amr Noaman Abdel-Hamid

The Projects Manager as a Shepherd

It is now eight years and counting that I've been involved in managing Software & Systems Projects as Projects / Programme / Portfolio Manager, enough time I suppose, to have built up at least some experiences that might count as pearls of wisdom, that I could impart on fellow young Project Manager newbies entering the role; or to young engineers doubting the usefulness & effectiveness of the various Projects Manager roles.
I started off as an EIT (Engineer-in-Training), following the usual technical ladder reaching the point of Senior/Principal Engineer. I could have stayed on the technical path (I enjoyed writing code, wrote some really good code too [it's a great feeling when a consultant comes to you after your code has been in circulation for three years saying "your code is one of the most beautiful pieces of code I had to extend, it was really well written, a pleasure to maintain"]) but I always intended to learn everything about the Software business and so couldn't see myself inclined to writing code, debugging, doing maintenance, support or integration forever. I also did not want to become a traditional Line Manager that gets to look after lots of people (although I could) doing appraisals, managing performance and dealing with Administration & HR issues, not to mention Office Politicking. So I delved into Technical Project Management, just around the time that Agile/XP practices were gaining strength and coming of age.

I felt that Project Management touched on a variety of aspects of the business of Software, Management, Business, Execution & Delivery - so I wanted to experience it. The role includes managing stakeholders up and down the business, engaging with engineers directly, doing demos and exhibitions, dealing with customers and third parties, setting up contracts, commercials and strategic planning, interacting with sales/marketing and also contained (to my pleasant surprise) a lot of coaching, mentoring and up-skilling project team members -- so pretty much touching on all areas of the business, a little more exciting than being just a Software Development Manager IMHO...

Quite recently, a young graduate newbie once called me a Pseudo-Technical Manager, inferring to someone who has dabbled in Technical/Engineering and just wasn't cut-out for the Techie nitty-gritty stuff. A young Project Manager would have been outraged at this insult of this arrogant young-fool, child of the Y generation - challenging the new PM in the first week of the job! Alas, I just smiled at him for his passion in being an engineer and for his naivete in not understanding the bigger picture. I could have pointed this graduate software engineer to all the code I've written and products delivered that was light years ahead of the current company's portfolio - but instead I let it lay (He will grow in time, and besides, PMs have a thick skin - we don't get offended or take things personally)...

Projects Managers, whether they come from a Technical (Software Background) or not (Construction, Mining, Psychology, Finance), regardless of the new buzzwords that come and go, are here to stay because the role offers value. Plain-and-simple. Scrum Master, Product Owner, Product Manager or not, there is still a need for a role to provide the Map, a Path to get to the Destination, clearing the Path, removing Obstacles and ultimately Executing & Delivering the Project through its People. Although, having said that, I have been biased, perhaps a little prejudiced against Project Managers assigned to a Software/Systems Project that have really never experienced the art of Software/Systems Engineering before, first hand, in the trenches. That is, I used to say "If you ain't written code before and delivered real software products, then you're ain't managing my project". In the same vein, I was also not convinced that you can be a CTO or CEO of a Technical Organisation, having never been technical yourself (a post for another day...). But through my own personal growth I have come to work and appreciate a Projects Managers from variety of backgrounds, even those of whom have never written a line of code, deserving of my appreciation...

Coming back to point, a Projects Manager is a much needed (but sometimes undervalued) role in an organisation. One of my first PM mentors stressed upon me that a PM is really a Service Provider: PMs provide a service to the project team/engineers. That concept did indeed take some time to warm up with me, as will most PM newbies experience. The first inclination of a young PM is to be misled or disillusioned with power, that "with great power comes great responsibility", "I am Project Manager now, I call the shots, I drive the plan, crack the whip!", "I set my meetings around my calendar, people must just commit because I commit", etc, etc. On the other hand, another early mentor gave me this advice "If you ain't pissing people off, you're not doing your job as a PM. A PM asks difficult questions, follows-through, pisses people off. I measure your progress by the number of complaints coming back from people - you're being a hard-ass PM!"

Young Project Managers must start off slowly, beware of jumping into the role with the wrong attitude as if you're in charge, as a PM, you have to earn your stripes, work with and through people to gain respect, build relationships and ultimately become influential and a driver, through action and taking the lead. A PM must build up a level of Emotional Intelligence (something which I at first was wary of), understand the nuances of Cultural Dynamics and Neuroscience, working with and through people...Building relationships and effective communication skills are probably the two most important soft skills you will need as a PM...

So this Project-Manager-as-a-Service spiel stuck with me for a long time, and continues to do so today. Recently, I was playing with this metaphor that a Project Manager is really a Shepherd, as in a shepherd of a herd of livestock/cattle. I googled for a bit for similar metaphors but couldn't find anything that related to the message I had in my head. So I decided to write about it here as a first attempt, hopefully it will grow and mature over time depending on public feedback.

• The Project Manager as a Shepherd metaphor
• Books that have helped me in my own PM Journey

Agile Architecture - Roadworks Analogy

I've had some interesting, somewhat frustrating conversations recently, around the roles, responsibilities and expectations of a Systems Architect. Refer to a previous post on how I broke down the various architecture roles for Digital TV Systems Projects.

What triggered the recent discussions is around understanding how architecture fits in with the Agile process. Before people who think themselves Agile Aficionados (Agileficionados) singing praise that architecture can be done just-in-time, please remember the context which I refer to: Embedded Set-Top-Box or Backend Systems Engineering.

We are not in the business of writing web applications, smart phone apps, tablets, or any desktop application, or even a web site -- where these environments are usually proven, solid and stable that fosters rapid application development cycles.

Digital TV projects, especially Set-Top-Box projects are a little more complicated than iterating through web page developments, switching versions easily through redirecting URLs, eliciting real world customer feedback for alpha/beta sites or apps, etc. STB projects generally involved multiple vendors or software suppliers, each with its own development processes. Testing is often done in a limited lab environment, whilst real world customer feedback demands an end-to-end engineering system to be deployed in a live broadcasting environment (which is currently in active use, generating revenues for the PayTV operator, so engineers better be damn sure they're not going to disrupt business operations, etc).

So some STB project teams are indeed embracing Agile (or at least trying to be Agile in the context of their component-worlds). I have written about the difficulty of doing Agile in these projects that I'm not going to repeat here. How then does one deal with the concept of architecture in Agile? Do we stick to Just-in-time, just enough or do we embrace a little structure and discipline, in keeping with Mile-wide, Inch-thick view of the future, but not waste time with too futuristic features??

For Digital TV projects, I prefer the mile-wide, inch thick approach, where we're looking at least a year ahead into the future, to develop new features and enhancing the roadmap. This is especially true for long lead features, like Recommendations, Targeted Advertising, Home Networking, or even designing the next evolution of your Middleware or EPG. And I believe that Architecture, can still be done incrementally, in an Agile-manner, although with a little discipline called foresight and pragmatic proactivity. To support this picture, I created the Paving-the-Road analogy: