Common Challenges with Shared Component Development in Agile

In a previous post, I shared a method of Software Development & Integration that component teams adopted for a very large scale project, where the development & integration team spanned in excess of 250 people, geographically dispersed across the world, where the system software stack was a Set-Top-Box Middleware product, consisting of eighty (80+) software components, each component owned by individual component teams, a component team size being anywhere between five and twenty people (all software engineers). These component teams implemented Agile/Scrum at their own individual team level, and had to also deliver their components into project integration streams, with multiple project delivery timelines overlapping simultaneously using a common iteration heartbeat of six week cycles.

This was grand-scale, iterative development & continuous integration that did come with an enormous infrastructure overhead as well as quite a structured implementation of Agile Product Management. Please refer to my earlier post that introduced that particular case study.

In that case study, I maintained that the principles of development & engineering strategies equally apply to small, singular component teams as well as large-scale, distributed teams. In the end, it is just managing a component development & delivery stream.

The purpose of this post is to drill into some of the challenges that impact Set-Top-Box Software Development projects, regardless of the component impacted - especially when component teams have to maintain a common product code-base, yet at the same time manage multiple project deliveries to customers, often with overlapping timelines. I will touch on some of the scenarios development teams face, and highlight the implications of co-ordination & control aspects, in relation to how going Agile/Scrum possibly makes it simpler or more complicated...

This post is structured as follows:

• Typical Project/Product Management Flow
• Implications on Maintaining Common Architecture / Code Base for Component Teams
• Pros/Cons of various Options
• Concluding Remarks

Lightweight Open (Open Source) Streaming Software Stack

Continuing with putting myself out there and sharing some of the ideas I had in the past for possible start-ups, here is one that I journal'ed in October 2006, back when really no one was talking about user-generated content or become your own private-broadcaster.  Youtube only released live channels somewhere around 2010/2011, so the emphasis for me here is that my ideas were not that wildly off... So yeah, we have most of this already, but in 2006 when I had the idea, no one really was pushing publicly for this, and it flowed naturally as a feature extension to you-tube, and now other social networking sites as well!

-----------------------------

Posted 17 October 2006 

I imagine in the not-too-distant-future that any device with a display (reasonable size), integrated camera (this is the future), network connection (ethernet, wifi, cell radio, usb) and built-in speaker/microphone - will want to participate in the live media streaming phenomenon. Be it mobile phones, handheld games, integrated digital cameras(camera with networking, combo PDAs, etc), PSPs, GameCubes, XBox'es, STBs, etc, etc.

The numbers speak for themselves: Mobile phones have surpassed the 1 billion mark in 2004/2005. Handheld games have topped the 100 million mark; and the number of digital cameras is growing all the time. Not forgetting integrated PDAs that are GPS receivers, mobile phones and media players all-in-one.

All these devices provide for some form of content consumption. Coupled with a network connection, the next stage would be to share this content or even stream it...(Forget power requirements for now - assume the power/battery life is already sorted)

Basic Idea: Provide an open lightweight Streaming Software Stack (not a thick middleware) that can be easily ported to a variety of platforms; and integrated seamlessly with other technologies. The software will enable any device with a network connection with or without an integrated display to stream to anyone that's interested (variety of models will probably exist here). It would be better if there was an integrated camera that you could use to stream your live video (could act as video phone).

Could charge a royalty for this stack, or sell it once off - easily re-use STB middleware IPR to provide this stack - If I had my own start-up; I would definitely want to include this software stack in my portforlio!!

If this were possible (bandwidth, network & streaming issues aside) the scenario would be:

- A service provider on the web provides the facility to members of streaming content live (near realtime) or recorded

- Members will have user accounts, etc

- Members themselves will be able to setup individual user accounts that grant web surfers access to their content

For example: My portable (A/V capture) device will connect to the intermediary streaming server. The streaming server will receive and simulataneous stream the content to interested users (i.e. connected).

This will really open up the world of content sharing - literally, anyone can be a broadcaster. People can act as reporters, reporting all sorts of interesting/weird things - people can share live experiences with family/friends located at different ends of the globe, etc, etc....

PayTV Operators should STOP re-inventing the wheel and should NOT see Google as Evil!

I have been working within the PayTV Software & Systems space for almost fifteen years now, and from the very beginning with Set-Top-Box software, I really wasn't impressed by the software technology. Apart from the hardware being fairly interesting, basically a device for decoding video/audio/data streams based on MPEG / DVB / ATSC / etc protocols and standards, when it came to software, there wasn't really a "wow" factor in it. Of course, we can't forget the really crucial element, the heart of the system, the crown jewels, the revenue-generator, the very interesting & complicated black magic technology called Conditional Access (CA) which is really really cool, the rest of the building blocks was really around the Set-Top-Box user experience / application, which really wasn't all that new: 

• Essentially the device needed an operating system, a way to draw stuff on-screen, a user navigation interface, and some data source driving the application, traditionally called the Electronic Program Guide (EPG). The STB software thus re-applied knowledge well-known in the computer industry for decades (Model-View-Controller MVC design pattern emerged in the late 1970s), simple operating system and driver / hardware abstraction layers, C-code...

As all hardware devices tend to follow Moore's law at some stage, Set-Top-Boxes (STBs) have evolved to quite powerful machines allowing for migration to newer, modern software implementations (although nowhere near the computing power & rich experience offered in the latest smartphones & tablets - post for another day), the software too, has become more accessible than ever, with more STBs using Free Open Source Software (FOSS), particularly the dominance of the Linux Kernel as the popular operating system of note, displacing historical dominance of VxWorks, STOs, NucleusOS, uCos, etc, etc.

However, there are some components in the STB software stack that remain fundamentally closed: The STB Middleware, EPG, Conditional Access (CA) clients. Okay, ignoring the CA client, which has always been fundamental and will never go away for years to come -- the classic Middleware/EPG components really don't need to be that closed anymore. 

There is also the backend / headend information service data generators that are traditionally closed, although vendors purport "open protocols", the PayTV Operator generally obfuscates the openness by forcing business-specific modifications in the protocols, apparently unique to each PayTV Operator/Broadcaster...

Traditionally & historically though, these software components were provided by third-party vendors that PayTV Operators just accepted as the norm. Highly closed, difficult to integrate with open systems, these vendors capitalised on providing a closed system, to the extent of locking in the PayTV Operator to the entire stack, some vendors reaped the benefits of providing the end-to-end system, one-stop-shop for everything. Later, PayTV operators decided to take more control, diversify the ecosystem by enforcing the use of multiple components, not being locked-in by just one vendor, promoting open standards for integration, and more recently taking more ownership of some of the development and integration...

Yet, the models within which most PayTV operators continue to work - is still pretty much a closed one. There is an aversion to sharing, opening up technology to other parties with a view to extending partnerships as well as creating new strategic relationships. There is an huge element of mistrust, not-invented-here attitude, we-can-implement-this-in-house, etc, etc!! I am really dumbfounded with that approach...