Nissan Motors Production Line and Software Systems Development

This week I was invited by @Farid from Crossbolt to tour the Nissan Production Line in Pretoria, to see first-hand the Lean / Kanban / Kaizen production model in action: Cars come off the line every four minutes (cycle time), Zero defects straight-through rate.

I jumped at the opportunity without hesitation. Although I've ran software projects as a factory-line model before, I had never actually been in a real car production factory before. Fareed had decided to take his IT Operations team on this tour, after doing the theoretical knowledge-share on Lean.

He felt it would make more sense for his team to witness first hand the theory in action, where the metaphors can be easily visualised and applied to an IT Ops / Systems Integration space. Not a bad idea...

This post is just a brief dump of my personal take-aways from the visit. I will try to expand on them in later posts once I've let the ideas play around my head for a while to morph into shape. Despite what some people might say: Software is a creative art, it can't be rushed and boxed into a production line, when it comes to pushing out consumer production software, I beg to differ: there are indeed many parallels from the manufacturing sector (which is seriously way more mature than software development) that can be drawn upon and applied to software teams - incidentally this is the roots of Scrum / Lean anyway - just taking the same concepts and applying it to software teams...

There's plenty of info on Kaizen / Kanban / Lean / Poka Yoke / Scrum - I won't go into detail here. For context though, the line we visited was a multi-model line. This means that the line produces more than one type of vehicle in a continuous flow. Any team on the line could be working on a different model at the same time. Some car production lines specialise in just one model, but this line builds at least four different models. Because the flow is continuous, a car comes off the production line every four minutes, like clockwork. The team working on the line therefore, are cross-functional and cross-skilled. 

Take-aways:

Career Ladders: Avoiding chaos and anarchy in Software / Systems Projects

This post is part rant & part structured discussion around the topic of career development in the domain of Software & Systems Engineering. I have been in the business of core software engineering for fifteen years (15), and was fortunate to experience working with highly rated professional companies (from Silicon Valley, internationally renowned) whose sole purpose was producing software systems from scratch, including as providing software design services to other software companies - these companies maintained true to engineering as a discipline.

A graduate engineer would enter the company with a fairly good idea of the where he/she is starting, and the various options & opportunities that lay ahead in terms of career growth, and, depending on the company - this graduate could spend

the next 10-15 years with just one company alone and never get bored, traversing many job disciplines as he/she so desired. 

This is the baggage (rightly or wrongly) that I come with, and hence my surprise to learn that some companies in South Africa are still making a serious mistake of not having a simple template that maps roughly the career ladders for the team, covering Development/Test/Integration Engineers, Architects and Managers. I firmly believe this is a recipe for chaos & anarchy that must be avoided as far as possible, and the solution to this problem is to map out a simple Career Jobs Ladder for your technical department.

Don't get me wrong: I am all for meritocracy, flexibility and not bureaucracy - but it really irks me to see things happen in the workplace that just don't make sense, especially when promotions happen when there is no real truth, track record or backed-up peer recognition that warrants a role / title change. 

I have seen the following happen as an example of chaos:

• An integration engineer with no prior design or architecture experience is promoted to Architect
• An software engineer with no prior architecture accolades is promoted to Architect
• A software engineer with no prior team leading, people management or facilitation experience is promoted to Scrum Master
• A team lead with no prior project management or scrum mastering track record becomes an Agile Manager
• An integration engineer with no product management experience becomes a Product Manager
• A new recruit with no prior experience in the technology domain joins as a Senior Solutions Specialist
• A new architect who has never architected any product before enters as a Solutions Architect
• An enterprise architect who has never delivered to market any real enterprise-class systems product that has a user base of more than fifty is made Enterprise Delivery Architect
• A component architect who's only worked on a single software module / component becomes Enterprise-wide Solutions Architect
• An integration engineer who's only experience in embedded devices joins an enterprise systems integration team as a Senior Integration Specialist
• An automation specialist or tech lead who is misunderstood as the Head of Automation

To an observer, the above scenarios are candidates for chaos (What do all these roles mean? What's the job spec? Is there a clear map that shows the progression of one role to another?, etc. etc.), although these cited migrations would not be that far fetched if there was a career ladder to hand, that facilitates the growth path - that can be used to take the individual on the journey to reaching his/her desired goal....

And this is where anarchy comes in (again, a little rant on my part):

Agile Africa Conference - Content made available

Earlier this year I wrote about my experiences from the first Agile Conference hosted in Africa:

• Day One
• Day Two

The kind folks at JCSE have now shared the documents from the conference here, and allowed me to link independently to my own copies, should their site ever disappear. But for now, you can find the slides for each here:

• Martin Cronje – How Teams increased their value
• Sam & Karen – Agile Testing Mindset
• Riaan du Doit – Agile Adoption with Parental Guidance
• Aslam Khan – Apartheid and democracy
• Derrydean Dadzie – Case of 2 worlds
• Ana Forssman – To be Agile or not to be Agile

Clips have also been made available on youtube, click here for the full playlist.

Enjoy!

Common Challenges with Shared Component Development in Agile

In a previous post, I shared a method of Software Development & Integration that component teams adopted for a very large scale project, where the development & integration team spanned in excess of 250 people, geographically dispersed across the world, where the system software stack was a Set-Top-Box Middleware product, consisting of eighty (80+) software components, each component owned by individual component teams, a component team size being anywhere between five and twenty people (all software engineers). These component teams implemented Agile/Scrum at their own individual team level, and had to also deliver their components into project integration streams, with multiple project delivery timelines overlapping simultaneously using a common iteration heartbeat of six week cycles.

This was grand-scale, iterative development & continuous integration that did come with an enormous infrastructure overhead as well as quite a structured implementation of Agile Product Management. Please refer to my earlier post that introduced that particular case study.

In that case study, I maintained that the principles of development & engineering strategies equally apply to small, singular component teams as well as large-scale, distributed teams. In the end, it is just managing a component development & delivery stream.

The purpose of this post is to drill into some of the challenges that impact Set-Top-Box Software Development projects, regardless of the component impacted - especially when component teams have to maintain a common product code-base, yet at the same time manage multiple project deliveries to customers, often with overlapping timelines. I will touch on some of the scenarios development teams face, and highlight the implications of co-ordination & control aspects, in relation to how going Agile/Scrum possibly makes it simpler or more complicated...

This post is structured as follows:

• Typical Project/Product Management Flow
• Implications on Maintaining Common Architecture / Code Base for Component Teams
• Pros/Cons of various Options
• Concluding Remarks

Lightweight Open (Open Source) Streaming Software Stack

Continuing with putting myself out there and sharing some of the ideas I had in the past for possible start-ups, here is one that I journal'ed in October 2006, back when really no one was talking about user-generated content or become your own private-broadcaster.  Youtube only released live channels somewhere around 2010/2011, so the emphasis for me here is that my ideas were not that wildly off... So yeah, we have most of this already, but in 2006 when I had the idea, no one really was pushing publicly for this, and it flowed naturally as a feature extension to you-tube, and now other social networking sites as well!

-----------------------------

Posted 17 October 2006 

I imagine in the not-too-distant-future that any device with a display (reasonable size), integrated camera (this is the future), network connection (ethernet, wifi, cell radio, usb) and built-in speaker/microphone - will want to participate in the live media streaming phenomenon. Be it mobile phones, handheld games, integrated digital cameras(camera with networking, combo PDAs, etc), PSPs, GameCubes, XBox'es, STBs, etc, etc.

The numbers speak for themselves: Mobile phones have surpassed the 1 billion mark in 2004/2005. Handheld games have topped the 100 million mark; and the number of digital cameras is growing all the time. Not forgetting integrated PDAs that are GPS receivers, mobile phones and media players all-in-one.

All these devices provide for some form of content consumption. Coupled with a network connection, the next stage would be to share this content or even stream it...(Forget power requirements for now - assume the power/battery life is already sorted)

Basic Idea: Provide an open lightweight Streaming Software Stack (not a thick middleware) that can be easily ported to a variety of platforms; and integrated seamlessly with other technologies. The software will enable any device with a network connection with or without an integrated display to stream to anyone that's interested (variety of models will probably exist here). It would be better if there was an integrated camera that you could use to stream your live video (could act as video phone).

Could charge a royalty for this stack, or sell it once off - easily re-use STB middleware IPR to provide this stack - If I had my own start-up; I would definitely want to include this software stack in my portforlio!!

If this were possible (bandwidth, network & streaming issues aside) the scenario would be:

- A service provider on the web provides the facility to members of streaming content live (near realtime) or recorded

- Members will have user accounts, etc

- Members themselves will be able to setup individual user accounts that grant web surfers access to their content

For example: My portable (A/V capture) device will connect to the intermediary streaming server. The streaming server will receive and simulataneous stream the content to interested users (i.e. connected).

This will really open up the world of content sharing - literally, anyone can be a broadcaster. People can act as reporters, reporting all sorts of interesting/weird things - people can share live experiences with family/friends located at different ends of the globe, etc, etc....

PayTV Operators should STOP re-inventing the wheel and should NOT see Google as Evil!

I have been working within the PayTV Software & Systems space for almost fifteen years now, and from the very beginning with Set-Top-Box software, I really wasn't impressed by the software technology. Apart from the hardware being fairly interesting, basically a device for decoding video/audio/data streams based on MPEG / DVB / ATSC / etc protocols and standards, when it came to software, there wasn't really a "wow" factor in it. Of course, we can't forget the really crucial element, the heart of the system, the crown jewels, the revenue-generator, the very interesting & complicated black magic technology called Conditional Access (CA) which is really really cool, the rest of the building blocks was really around the Set-Top-Box user experience / application, which really wasn't all that new: 

• Essentially the device needed an operating system, a way to draw stuff on-screen, a user navigation interface, and some data source driving the application, traditionally called the Electronic Program Guide (EPG). The STB software thus re-applied knowledge well-known in the computer industry for decades (Model-View-Controller MVC design pattern emerged in the late 1970s), simple operating system and driver / hardware abstraction layers, C-code...

As all hardware devices tend to follow Moore's law at some stage, Set-Top-Boxes (STBs) have evolved to quite powerful machines allowing for migration to newer, modern software implementations (although nowhere near the computing power & rich experience offered in the latest smartphones & tablets - post for another day), the software too, has become more accessible than ever, with more STBs using Free Open Source Software (FOSS), particularly the dominance of the Linux Kernel as the popular operating system of note, displacing historical dominance of VxWorks, STOs, NucleusOS, uCos, etc, etc.

However, there are some components in the STB software stack that remain fundamentally closed: The STB Middleware, EPG, Conditional Access (CA) clients. Okay, ignoring the CA client, which has always been fundamental and will never go away for years to come -- the classic Middleware/EPG components really don't need to be that closed anymore. 

There is also the backend / headend information service data generators that are traditionally closed, although vendors purport "open protocols", the PayTV Operator generally obfuscates the openness by forcing business-specific modifications in the protocols, apparently unique to each PayTV Operator/Broadcaster...

Traditionally & historically though, these software components were provided by third-party vendors that PayTV Operators just accepted as the norm. Highly closed, difficult to integrate with open systems, these vendors capitalised on providing a closed system, to the extent of locking in the PayTV Operator to the entire stack, some vendors reaped the benefits of providing the end-to-end system, one-stop-shop for everything. Later, PayTV operators decided to take more control, diversify the ecosystem by enforcing the use of multiple components, not being locked-in by just one vendor, promoting open standards for integration, and more recently taking more ownership of some of the development and integration...

Yet, the models within which most PayTV operators continue to work - is still pretty much a closed one. There is an aversion to sharing, opening up technology to other parties with a view to extending partnerships as well as creating new strategic relationships. There is an huge element of mistrust, not-invented-here attitude, we-can-implement-this-in-house, etc, etc!! I am really dumbfounded with that approach...