Appreciating the Role of System Integration

Last year I started my series of posts around the different aspects of DTV system projects. Starting out by describing the a typical end-to-end system architecture and the importance of clearly identifying the roles and responsibilities of the architect team (see post: Architect Roles), followed by a write-up on considerations for setting up a programme organisation structure (see post: Managing a DTV Programme) and more recently, a brief overview of auditing the STB System Integration process (see post: SI/Dev Process Audits).

I still have a few more topics in the pipeline, however the aim of this post is to highlight the worlds of System Integration, as this can often be overlooked and open to interpretation, introduces dangerous assumptions and automatically puts the programme at risk.  With a large-scale DTV deployment, the programme must at the outset, clearly define the boundaries of system testing, and establish clear roles for the system integration effort.

This post is especially relevant to TV Operators deciding to become more involved with technical ownership of their value chain, as previously highlighted, more and more companies are taking the plunge into doing their own Software Development, and System Integration. This post is another humble attempt at raising awareness that such operators should proceed with caution, ensuring the full extent and gravity of the undertaking is understood, because System Integration is a significant investment in people (highly experienced technical experts required), equipment (investment in costly infrastructure) and time (starting from scratch, building this competency from the ground up is at least a 5 year maturity period).

Recall the end-to-end architecture originally present a few posts back:

As highlighted in previous posts, the world of Digital TV consists of Broadcasters / PayTV operators employing one of more component vendors or Technology Service Providers offering specialist services of Software development for Headend/STB components, Conditional Access Providers, Billing, Scheduling  & Play-out Systems and System Integration services.  This is made possible by basing the various technologies on open standards (MPEG/DVB/IET/IEEE) and thus allows for creating an end-to-end system consisting of a variety of components from different, independent vendors.  However, there are some TV operators who opt to use one supplier, often referred to as the Primary System Integrator, who are tasked with not only ensuring the end-to-end system is architected correctly, but quite often, the Primary System Integrator provides the bulk of the system components end-to-end (i.e. dual role of component developer supplier and integrator).

System Integration and Architecture are linked, they go hand-in-hand. They are not separate entities, although in theory the act of architecting is different to the act of integrating, but integration drives the architecture.  Looking at the above figure showing the different architectural streams, the following becomes clear:

• Solutions Architect is replaced with Solutions Integrator
• Headend Architect is replaced with Headend Integrator
• STB Architect is replaced with STB Integrator

This now exposes us to the world and levels of System Integration.  A Programme needs to define its System Integration strategy at the outset, success of the programme depends on a solid System Integration strategy.

If you're a broadcaster taking ownership of SI, you need to decide in what capacity you will be involved in:

• STB SI
• This is generally taking responsibility for integrating the various components of the STB stack (typically Platform Drivers, CA Component, Middleware, UI & Interactive Engines) providing a stable build (STB image).
• Interfacing with a broadcast/backend is part of this responsibility but STB SI assumes the necessary backend/headend infrastructure is in place
• Responsible for managing and driving through component deliveries in terms of quality expectations and to some extent project time-lines
• It is always useful to have access to component source code to aid in investigating and characterising bugs, but this isn't always possible as it has commercial implications with IPR, etc.

• Headend SI
• Within the Headend system itself, there are a number of component systems that must come together in implementing a specific macro-component features, like Conditional Access, Scheduling, On-Demand Catalogue/Playout, IPTV streaming.
• Generally though, headend SI is about integrating these headend components either individually or collectively, proving the interfaces are working as designed (i.e. verifying the input/output is as specified by the protocols, essentially obeying the Interface Control Definitions ICDs)
• Test tools might be used as test harnesses, or the STB itself can be used depending on the situation
• There is a fine line between Headend SI and End-to-End SI or Solution SI, which can often lead to confusion and false assumptions

• End-to-End SI or Solution Integration
• This is about testing the solution in a representative environment, as close to real life operations as possible
• It brings together proven systems at the component level, i.e. STB, Headend and other dependent infrastructure systems like Transport (Encoders, multiplexors, IP networking), Billing & Subscriber Management
• There is often a dependency or interface with systems that are currently live (live components are considered crown jewels, usually managed by a separate delivery/operations team).
• Failover, Recovery and Upgrade scenarios are performed as part of this exercise
• Requires investment in one or two end-to-end labs (Some broadcasters impose this on their major vendors to ensure appropriate end-to-end testing is setup locally at the vendor premises; and some vendors have end-to-end labs set-up on the operator's site)
• Often confusion around how this activity defers from System Delivery/Operations - think of this as pre-delivery, the last mile of the programme's development phase.
• Becomes really important to define an end-to-end SI strategy where the overall DTV value change is a conglomerate of components provided by different business units offering different services, e.g. a Broadcaster might have separate departments, sometimes independent units each offering a slice of the system (Broadcast, Interactive, VOD, Guide, Operations, R&D) - all these business units must come into alignment at the outset of planning the programme...

In a nutshell, the following diagram tries to illustrate the worlds of SI, the intersections shaded show the typical integration points required. The worlds are interconnected, and can't be expected to be tested and qualified in isolation of other systems (during development phase, unit/component testing is expected, but as soon as we have stable elements to make up the E2E system, then its time to kick-off your End-to-End SI activity). Typically, E2E testing must start well in advance of launch/go-live, typically 6-8 months of E2E testing is not unheard of...

Worlds of SI

The world of Delivery/Operations

The absolute last mile in completing a DTV Programme, is the actual launch - go live!  This is usually under the control of an Operational entity who's sole purpose is to ensure the smooth operational deployment of the systems on live - 99.9999% of reliability is expected, the transition from End-to-End SI to Live Deployment must have the highest probability of success.  In parallel to the E2E SI activity, there is usually a Delivery/Operations team that are mirroring the E2E SI effort, making preparations for launch.  This delivery team must be included in the Programme Schedule from the start, so they can make the necessary equipment procurements, network configurations and deployment schedules in advance to ensure the smooth transition.  Practically though, the delivery/operations team might re-use the same people from E2E SI, but it is important to ensure that everyone understands the milestones and phases of this activity.   Forward planning is crucial to avoid ambiguity and confusion down the line, which goes back to my point earlier that the boundaries and scope for each Testing/Integration activity are clearly understood.

As always, I'm interested in your comments and feedback. 

SI/Development Project/Process Audits...

Earlier this year I wrote about setting up the right Roles & Responsibilities in managing a typical DTV project, covering the Architecture & Project Team requirements.  In this post, I'd like to share my experiences with the Project Audits.

The scope of Project Audits is in itself a vast topic; mostly around the area of auditing the Project Management aspects of the project, along with ensuring effective Quality Management processes are in place, proper Project Governance is enforced, etc. The IEEE SWEBOK dedicates a chapter on Sofware Quality, of which Project Audits is considered as part of this knowledge-area, summarising the process as:

“The purpose of a software audit is to provide an independent evaluation of the conformance of software products and processes to applicable regulations, standards, guidelines, plans, and procedures” [IEEE1028-97]. The audit is a formally organized activity, with participants having specific roles, such as lead auditor, another auditor, a recorder, or an initiator, and includes a representative of the audited organization. The audit will identify instances of nonconformance and produce a report requiring the team to take corrective action.

While there may be many formal names for reviews and audits such as those identified in the standard (IEEE1028- 97), the important point is that they can occur on almost any product at any stage of the development or maintenance process.

What to audit in DTV Projects?
Again, with a bias towards the STB Software Development delivery project, audits are introduced to review two main areas:

• STB Development / Test Processes across the stack (Drivers, Middleware, Applications)
• STB System Integration Processes

If you're a product vendor supplying any of the STB components, you might want to review and assess your own development/test processes to gauge how well you're doing against industry best practices. It's also good for your portfolio when pitching for new business to potentional operators.

If you're a TV operator and dependent on product vendors supplying components to your value chain, you need to ensure that your vendors are indeed delivering quality components & systems. Typically (and this can vary with the type of contract in place), the Operator (customer) provides the Product/System requirements to Vendors; Vendors implement and test their components according to the spec; produce test results (self-assessments), making it difficult for the Operator to dig too much into the detail, to find out exactly how the results came about, or gauge the quality of the underlying development processes, typically until it's too late in the project life-cycle.  This can of course be remedied by building into the contracts the requirement for auditing the vendor's various processes, at any stage of the project life-cycle.

Some operators outsource the whole testing and verification effort to third-party test houses. These test houses, depending on the strength of the commercial contracts in place, could very well take complete ownership of the test & quality management, and enforce audits on encumbent component providers.

As noted in my previous posts, more and more Operators/Broadcasters are taking more ownership over the product development, testing and integration. Take for instance, the task of STB System Integration.  Whilst there are well known companies like S3 & Red Embedded, that specialise in STB SI, operators like Sky, DirecTV, Foxtel, Multichoice have dipped their feet into the waters; attempting to own STB SI activity internally, in-house. Incidentally, there are more players on the STB SI front, some STB manufacturers like Pace, UEC; as well as service houses in India like Wipro & Tata Elxsi. NDS Professional Services is also a well known player in the System Integration field.

In my rather short stint of 11 years with STB projects, the world of DTV is quite small; the popular companies of choice for SI, from my experience are:  S3, Red Embedded & NDS.

In my current company, being a DTV Operator/Broadcaster, we are running our own development & STB Integration projects. Because we're relatively new in this game, having built up a good team of people, making every effort to source the right people with the right skill-set (which is really challenging in South Africa), we'd been through a few rounds of Development & SI, using third-party vendors supplying Middleware & Drivers, we decided now was a good a time as any to perform a self-diagnostic, a review and audit of our current processes, specifically focussing on STB System Integration and Supplier Deliverables coming in. Although we have enough knowledge in the team to perform a self-assessment ourselves, and despite having learnt and corrected a lot of mistakes from past experience; we opted for a professional consultancy to do the audit, and chose S3.

Why we chose S3?
S3 despite being a relatively small company in Ireland, has been involved with TV technology for many years; and is well respected in the DTV community; for their specialist design services & system integration knowledge. They've leveraged the wealth of past project experiences in producing useful products like S3 StormTest & Decision Line, a STB automated testing system that has only in increased in popularity since its inception (which we're currently using by the way). The S3 Engage Portal is also another useful product to manage and co-ordinate a complicated DTV SI project.  S3 are also known to be frank & objective, offering feedback and analysis that is open, without hiding the nasty details. S3 were chosen as SI for some big players in Europe; and they're also being used in some other projects of ours.  Based on strength of previous relationships; despite the fact that I previously worked at S3! The world of DTV is small, the other companies mentioned above are also competent and capable.

What areas do you focus on for an SI audit?
Of course I can't share with you specifics of the S3 Audit process, but if you're interested in learning more please contact S3, or speak to this guy.
In a nutshell, S3 shares a model of SI that is well known to the industry:

• Incremental / Iterative approach, built on a strong foundation of domain expertise, using experienced System Integrators.  The foundation cornerstones that deliver an effective SI process, rely on the following:
• Strong Technical Expertise
• Strong Project Management
• Usage of Advanced Tools & Infrastructure to support the process
• Strong Quality Assurance Practices

S3 have been kind enough to allow me to publish a slide on their overview of SI process:

S3's Integration Model: SI is a continuous cycle once all Components are in a suitable state of completeness

Based on the above cornerstones, the audit focuses on 48 topics of concern, that largely fall into the following categories - again, the below is generally best practices as evoked by IEEE SWEBOK Chaper 11 on Software Quality which proves S3 appreciate Industry best practices:

• Project Management
• Co-ordination & Communications
• Requirements Management
• Defect Management
• Testing
• Quality Assurance
• Supplier Management
• Release Delivery
• Resourcing
• Software Architecture
• Integration & Configuration Managmeent
• Software Development

My Own List of Audit Criteria
Since this knowledge comes from my own past experience, I've no problem sharing information with with you. If you want more details, feel free to contact me, or leave a comment. The list below is something I put together in advance of the audit we've just undertaken.

If I were doing a STB audit, this is what I'd be interested in:

• Foundations of the STB Component (Middleware/UI) architecture design
• Design philosophy – apart from being MHP/Jave/MHEG/etc compliant, what is the design ethos behind the architecture, i.e. does it scale?
• Design patterns – what abstractions are used to ensure flexibility with changing requirements without major impact to design?
• Performance requirements – is the product being architected with performance considerations in mind?
• Security considerations – is security checks in place, to prevent buffer overruns, etc?
• Stability considerations – how do we ensure stability checks are enforced in design / implementation?
• Open Source components usage – if open source is being used, is there a process for choosing components, etc?
• Inter-component Interface design and control – assuming the middleware is a collection of components, how are component interfaces managed?
• Documentation – Requirements, Design – is there a documentation standard in place? Middleware design, component design, etc?
• Process for introducing change to architecture? New components to implement new features? – How are new components introduced to the system?
• API ICD Controls – how do we manage API changes??
• Is there an SDK?
• Quality of API Documentation? Is it usable, self-explanatory, auto generated??
• New APIs must be communicated to application developers
• Changes in APIs must be controlled
• APIs must be backwards compatible
• API Deprecation rules

• Testing the STB (Middleware/UI/Drivers) stack – what is the testing strategy employed?
• Depending on the architecture, assuming it’s component based architecture – do we have a continuous build and integration system in place essentially for regression testing:
• Component based testing – unit tests on the component, using mocks for other components, assuming interfaces are controlled
• Component group testing – where you collect related components together and test as unit, e.g. PVR engine can be tested in isolation
• Middleware stack testing – test the middleware as a whole, i.e. collection of components forming the MW
• Regression Suite – does one exist, how is it tested, and how often is it tested?
• Spread of STB hardware platforms used for testing
• Use of a Simulator
• How often do we check for regression – daily, weekly, just before release?
• How is new feature development controlled – i.e. new features to be delivered without causing regression in previous working functionality
• Is there a concept of a middleware integration team?
• Middleware development is separate to middleware integration, separate teams
• Development teams do component development, and component unit testing
• Middleware integration team takes development components and integrate – test new features against last stable middleware as well as test new functionality
• Middleware integration team consists of technical people who can code – best practise is to use a middeware test harness that exercises the middleware, not just the external Application APIs, but speak directly to component APIs themselves
• Is there a separate team owning the typical STB Hardware Abstraction Layer / Driver Interface?
• Who defines the HAL spec / interface?
• Change control procedures in place to support HAL spec?
• Testing of the HAL?
• Is there a test harness that validates new platform ports?
• Is there a separate Platform Integration team?

• Supporting multiple customers and build profiles
• How does the architecture support the needs of different customer profiles: Satellite / DTT / Cable / IP ?
• For the same customer with different project requirements, how is the codebase managed?
• Build, Release, Configuration Management and Bbranching philosophy?

• Multi-site development
• How is multi-site development managed?
• What configuration management system is used? Does it support multi-site development well?
• Are off-shore teams used as support arms, or do actual development?
• So off-shore team have any ownership over components?
• How is communication managed with off-shore teams??
• Planning releases
• How are new releases planned?
• How is work assigned to development teams?
• What is the usual process for planning / starting new development?
• How much time is spent designing new features?
• Is testing included as part of the initial design review?
• Are new features delivered back to main product code tree in isolation or in big chunks??
• How much time is allocated to testing on reference platform?
• How much time is allocated to testing on customer production platforms?
• How is the backlog prioritized?
• How long is the development cycle?
• What is the exit criteria for development?
• How long is the test cycle?
• What is the exit criteria for testing?
• Is Middleware integration part of planning?
• Is full-stack integration part of planning?
• How are defects planned?
• Does the plan cater for stabilization period for new features?
• Who decides on the branching strategy nearing up to release?
• Are new tests identified as part of the planning process?

• Defect Management
• Is there a defect management process in place?
• How are defects managed and factored into the development process?
• How often is root cause analysis performed?
• How is quality being managed and tracked internally?
• What defect tracking tool is being used?
• What metrics can be generated to gauge the quality of all components in the stack??
• Average submission rate
• Closure rate
• Time it takes to close the average Showstopper
• Most suspect components
• Most complex components

• Development practises – industry standard coding practices should be in place
• Coding standard
• Architecture rules – e.g. system calls to avoid, avoid using strcpy, etc?
• Is CUnit test framework or any other suitable framework used for native component testing?
• Is JUnit being used?
• Do component unit tests map to component requirements?
• Do middleware tests map to system requirements or use cases?
• Are any of the following code quality metrics being looked at:
• Zero compiler warnings
• Zero MISRA warnings for native C components
• Which Java best practises being used?
• Code Coverage:
• Line coverage tools
• Branch Coverage
• Code complexity
• Function points
• Other static analysis tools like Klocwork
• Development organisational structure
• Size of development teams
• Structure of development teams – is it based on architecture components, i.e. notion of component teams, with component owners?
• If using multisite development, what is the line management structure – is the team a matrix, with virtual management teams?
• Do you have separate groups for:
• Development: Separate UI development, Separate Middleware team?
• Integration – is there a concept of middleware integration team?
• Platform – is there a concept of a team specialising in low level HAL/Kernel/Drivers/Platform
• What are the defined roles and responsibilities:
• Product Architect  - the person owning the product architecture, maintains the integrity of the philosophy, keeping everyone in check?
• Middleware Component Architect – do you have people owning specific components in the middleware, ensuring the interfaces are respected. For e.g. PVR engine usually is a separate architect?
• Lead Developer / Component Owner – someone responsible for delivering component to spec, owns the internal design and implementation of the component?
• Engineers – what is the mix of people and skillsets across the organisation

Meet Calvin, the security guard with Hope...

Calvin, on the day he passed his Drivers Licence

Meet Calvin, the Fidelity security guard onsite at my company.


A few months ago I would not have thought I'd help someone out by providing a renewed sense of Hope, Trust and Faith in fellow citizens of South Africa...but somehow I did. I am certainly not feeling overly proud, wishing to boast this to the world, in fact, we're taught to keep the good that we do a secret, so that it promotes humility and gratitude to God that it was only by God's will that some good has come out of your actions. There's a saying that if you do a good deed, it should be such that your left hand doesn't learn what your right hand has done, so keep it secret...

But in this day and age, it's necessary to share the little good that has been done; and with the power of social media, perhaps the same act could be repeated by others, creating a stimulus for good and create further joy in this world. So I feel compelled to publish a humble success that I hope can lead to other successes. For now I've helped someone by creating hope, and instilling self-confidence, that no matter what one's situation is, given the right motivation and support, if you're willing to try to change your life for the better, you will succeed, even at the risk of failure, as one of my mantras that I repeat often goes "If at first your don't succeed, try, try and try again"

Calvin is the security guard at my company who manages entry/exit of vehicles into the car park. We got to know each other in the first couple of months as I was using a hire car at the time, and used the visitor car park quite frequently.  We only got to striking a good conversation when I braved the walk to the local Civic Centre offices in Randburg to enquire about my SA Drivers licence.  I say "braved the walk" because nobody in Joburg really walks anywhere, and I returning from the UK after 10 years, have grown accustomed to walking anywhere.  Against the concerns of some colleagues, I decided to walk the 800 metres to the Office, to get a sense of the security myself....suffice to say, it's safe and I've repeated the walk several times hence.

On my back from the Licence office, I spotted Calvin and waited for him, a) so I could chat and b) to have a security guard as company :-) On this brief walk back to the office, I learnt that Calvin spoke very well, had strong opinions about corruption and the state of the country, he was quite curious to find out about London. He was in awe about the underground trains and more so, when I told him about the level of transparency and accountability the politicians have with their constituents, that there is a strict control over corruption and personal expenditures, so much so that politicians public resign...he wished the same could be done in SA.

I would then stop at his post now and again for a chat, and one day in August I found him really stressing about his situation. He tells me how he's been a security guard for the last 6 years, how he wanted to study engineering and was forced to leave school at an early age when his father passed away...and now he's got his Truck Drivers Learners certificate that would soon expire (Dec 2011) and he's got no money to do what's required to get his Drivers Licence before it expires. I knew in my heart immediately that I should help this guy out, and told him not to worry, these things have a way of working out, and left him with that thought - went for my meetings, and returned in the afternoon to continue the conversation.

I learnt that Calvin was serious to change his life for the better. As a security guard, he'd become friends with many of the truck drivers the company uses, and so he was aiming to get his Code 14 drivers licence with the aim of working for a trucking company. Truck drivers certainly earn much more than a security guard, so he'd started the process of, had passed his Learners test over a year ago, but just didn't have the funds to go for driving lessons and do the exam.  He also wants to study engine mechanics -- I recommended he focus on one goal at a time, first get the licence, change jobs and then think about the next step...

Without consulting my wife, I told Calvin that I'll help him get his licence.  Obviously, to a seasoned South African, all alarm bells would be going off at that moment, around trust, being taken for a ride, taken advantage off, etc... Dismissing all of the negativity, I proposed to Calvin that I will contribute the majority of the costs towards the licencing, provided he contributes some of his hard earned money to the cause.

I didn't want to do everything (although I had the means to fully sponsor it), but I felt that if Calvin contributes some of his own money, he'll be compelled and motivated to work hard, and succeed... I also grew up in tough times, I value commitment and dedication, Nothing comes for Mahala...

With this contract in place, we worked together on a plan of action and aimed that by end of November, Calvin should have been on enough driving lessons to be comfortable with taking the final exam.  Calvin surpassed expectations, and with five lessons managed to pass his Code 14/C1 Drivers exam!  We were both beaming with joy that day - you can see it in the photo!

Proof!

So what costed me just under R2000-00, which I could've easily used up on toys and take-aways, I helped enabled Calvin to step out of what was a depressing situation, and instil a renewed sense of hope.

It's not over yet...
Getting his licence is the first step. My aim is for Calvin to land a job by the start of the new year.  He must leave his job of Security Guard of the last 6 years, and take the plunge into a world full of possibilities.  I will continue to look out for Calvin, my company has a campaign called "Be More" and I'm going to send this blog post to the senior management and HR to see if the company can reciprocate or think about setting up similar initiatives...
I am also going to contact Talk Radio 702 to see if this is an example for Lead SA...

Doing more for South Africa...
When I left SA to work overseas, one of my ambitions was to return home and add a valuable contribution socially and professionally. I've made a humble start with the social aspect, but I do have bigger ambitions for the helping the professional outlook.

I'll post about this topic later, but I have a bee in my bonnet with the lack of skills/competencies in the IT/Software development field...I was really surprised that much of the workforce in my current company is outsourced to contractors from India, that there isn't any talent in our country. I blame the schools and tertiary institutions for not doing enough - so much so - that I strongly believe that it's a waste of time and money to go to a SA university: If you're interested in IT or programming, teach yourself, become self-taught, work on Open Source Projects, that's your ticket to landing a decent job and earning a salary...I've been mulling over setting up an Open Source Software academy where people can learn from high school, real world software engineering, contributing to real world projects without needing to attend University...

Agile Project Management with Scrum by Ken Schwaber

Agile Project Management with Scrum (Microsoft Professional) is a well written, easy flowing book that is clearly borne from someone whose had first-hand, real-world experiences of running and managing a variety of Software Projects over a number of years, Ken Schwaber, who is considered the father of Scrum who humbly takes no responsibility for being assigned that title, and so points to much earlier endeavours of many people, pointing to Babatunde's Process Dynamics, Modeling & Control for his first oral presentation of the theory behind Scrum; and sites Degrace's Wicked Problems, Righteous Solutions as the first people to call Scrum Scrum.

Regardless of who is attributed to formalising the Scrum processes, it is a software development methodology that is here to stay, and anyone working in software projects not familiar with Scrum should indeed rethink their strategy. Still, with so many books out there claiming to distil the secrets of Agile/Scrum, finding the right book is indeed challenging -- as with so many books, people focus on the theory without expounding on the practice, the real-world experiences and encounters that as a manager, and especially as someone transitioning from classic project management principles (not necessarily connected to the Watefall process of software development).

This book is definitely different, one that I recommend and fully endorse 100% - so if you're reading this and don't have a copy, get one now!

Why do I like this book so much?

Well, as a developer I'd worked on many projects of different styles, more recently explored with Agile/XP concepts. I've been on training courses, presentations & and participated in practical workshops such as the agile lego game. I've contributed as a developer, lead certain development efforts with a few engineers, then later went on to managing development projects using Waterfall as well as Agile; and more recently came off a massive project with 300+ people, where at the macro-level applied Agile/Scrum principles across the programme, but I didn't have much involvement in the day-to-day planning with individual component teams, where we'd assigned team leaders to act as Scrum Masters. We maintained one huge Product Backlog, not so different to that as Ken describes in Chapter 9 Scaling Projects Using Scrum.  We were developing a product that would deliver to multiple customers, each customer adding unique features contributing to the final feature set, the product itself would service tens of millions of homes -- so we had a strong Product Management group maintaining the heartbeat of the multiple projects using Sprint Time-boxing. We had macro daily planning and status updates, call it Scrum of Scrums.  So I was pretty much focussed on the high level programme and product management, than the low-level activities that a Scrum Master would normally be dealing with, I'd instruct as necessary....

As I was moving to a new job that placed me in a Scrum Master role, I needed to brush up on my Scrum Master knowledge and needed to prepare or translate my heavy Project Management experiences to Scrum principles. This book was certainly written with that purpose in mind. I was looking for real world use cases and reflections of actual projects, something that Ken describes well.

Ken touches on the key concepts of Scrum by using example projects as references, with frank and honest feedback. He includes much of the areas that would trip someone up coming in cold to Agile, and also does not dismiss the value and usefulness of applying the rigour of classic project management dogma (reporting, tracking, metrics, predictions) as old-school, out-with-the-old, in-with-the-new.  In fact, Ken advocates entirely the opposite of that. It certainly takes a while to master Scrum as a Scrum Master, and therefore takes much longer for a company with seasoned managers who know no better, to transition and accept Scrum from day one.  So as important stakeholders in the project and business, these people have a right to be given information in a format that is understandable, and this too, can still be achieved using Scrum.

What really resonated with me was...

Ken has re-affirmed much of my own understanding in that Agile/Scrum is no excuse to cut corners. Scrum is not a short-cut from applying rigour and due diligence.  Way too often, people who are either new to Scrum or have been previously burned by management, fall in love with the concepts of Scrum, especially the tenets of autonomy, freedom and do-what-it-takes attitude to get the job done, immediately put up barriers when someone starts talking about processes...

• "Oh that's waterfall approach, we are young developers, we don't do it like that anymore. We are light on process, and don't need documentation. We don't need to have a process for teaching new engineers, it's a collective, the engineer will be absorbed into the team and learn on the job. We don't have release notes any more, it's automated by Git. We don't need a version numbering system, Git labels are fine. We don't do need pre-planning, we do just enough because it's going to change anyway...."

Principles I connected with...

• The importance of taking time out to produce a Product Backlog early in the project. The argument of not knowing what you're creating because it's unknown is not good enough. Even if you're aiming to brainstorm to produce an initial Proof-of-Concept (POC), that POC is driven by meeting the high level needs of the Product Owner, so it is not an impossible activity to put thoughts onto paper, call it your wishlist, to-do list, whatever - there must be a Product Backlog to start with...how else do you determine the goals, and measure your progress accordingly??  So if you think you're doing Scrum and don't have a Product Backlog, then please go back and reconsider what you're doing...

• Ensuring the Roles/Responsibilities are clearly identified - especially the ownership of the Product Backlog - Is your project big enough to have its own Product Owner, or is this also something the Scrum Master can absorb?? The Product Owner's role is pivotal to setting feature priorities only, but not responsible for driving through scheduling or people management. That is the Scrum Master's job...So spend time to clearly outline the roles and responsibilities...

• Spend enough time Planning - the planning process described by Ken is a useful starting point. Spending a full day on planning and getting the team to uncover the tasks before the Sprint begins is definitely valuable.

• Due diligence must be followed by Scrum Master - measuring progress and reporting on progress, generating metrics and predicting the future are essential aspects to Scrum Management.  Don't be fooled by Scrum being lighter than classic project management. Scrum teams still have a responsibility to meet business objectives, how you go about doing it is the Scrum teams own business, but when asked with business-type questions, then the Scrum team better have enough data to provide professional responses

• Don't misuse Waterfall defence mechanisms - who says you don't do requirements/design/testing - Scrum says you ensure enough is done during the sprint such that at the end of the Sprint you produce a release that is shippable - so a Sprint must support requirements/design/testing/validation/release - same steps as waterfall, but it's constrained to happen within the sprint itself. Of course, one doesn't have to have heavy processes, but it is about applying core engineering principles.

• Self-organising teams are difficult to create, and takes time to master.  As a Scrum Master you have to continuously monitor the team's performance and interactions.  The hard truth is that in the real world, using distributed development or even multicultural teams with mixed permanent/contractor staff, a truly self-organising team is a rare find.

• Common sense - a large part of Scrum is essentially maintaining a common sense and pragmatic approach to things.  One doesn't have to be a certified Scrum Master to manage Scrum projects, but care should be taken in obeying & applying the rules of Scrum, which Ken concludes in Appendix A:
• The ScrumMaster is responsible for ensuring that everyone related to a project, whether chickens or pigs, follows the rules of Scrum.  These rules hold the Scrum process together so that everyone knows how to play. If the rules aren't enforced, people waste time figuring out what to  do.  If the rules are disputed, time is lost while everyone watis for a resolution. These rules have worked in literally thousands of successful projects. If someone wants to change the rules, use the Sprint retrospective meeting as a forum for discussion. Rule changes should originate from the Team, not management.  Rule changes should be entertained if and only if the Scrum Master is convinced that the Team and everyone involved understands how Scrum works in enough depth that they will be skillful and mindful in changing the rules. No rules can be changed until the Scrum Master has determined that this state has been reached.

Watch the man himself here @ GoogleTechTalks:

Managing a Digital TV Programme: Consideration for Project/Organisational Structures...

Following on from my recent post on the role and placement of the "architect" in a DTV systems project; in the same vein I'd like to expound on the role of the project team in the same end-to-end system delivery, touching on overall organisational structure, considering the concepts of:

• End-to-End Product Owner
• Feature Product Owner
• Component Product Owner
• End-to-End Programme Manager
• Headend Programme Manager
• STB Programme Manager
• Headend Component Project Manager
• STB Component Project Manager
• Headend Scrum Master / Tech Lead
• STB Scrum Master / Tech Lead

The collection of some or all these roles will make up your DTV Project team. These roles might each be performed by a person each, or we might have someone performing multiple roles.  Defining these structures up-front at the start of the project is considered Project Management 101, but you'll be surprised there are real world projects operating today where time, effort and preparation hasn't been invested in defining these organisations structures, and are operating in a quasi-chaotic state. I've been on projects where people run like headless chickens, not knowing who's responsible for what, where the boundaries lay in terms of roles and responsibilities, where due diligence isn't practiced through the overall programme: architecture, development, integration, testing & delivery...

The question of whether these roles are managed by a specific organisation unit like your classic Project Office or whether these roles come together naturally through different business units and operate as a virtual project team -- is outside the scope of this discussion.  My aim with this post is to highlight from my experiences what seems to work for DTV projects involving a launch of a new platform like VOD, or replacing old Middleware with new; again with more of a bias towards STB software, although the Headend will not be so dissimilar.  In essence, this can be drilled down to generic component product development at the micro-level, and at the macro-level these are rolled up in the end-to-end system as previously highlighted here for architecture, and the figure below that shows largely how a project team can be structured around the system.

It doesn't really matter about the internal business organisational structure, what matters is that the project has a defined structure and that these roles exist; the roles and responsibilities are well defined and clear, ownership and accountability is understood by all parties, that everyone is connected and on the same page. Of course you'd think this is Product/Project Management 101, that this is a no-brainer -- not so. 

In my experience, companies don't always have this done right, yet you might argue that products are still successfully launched, projects executed and closed, etc. Indeed, but IMHO such projects generally rely on a few individuals that go above and beyond the call of duty, the proactive, driven by a sense of urgency to get things done; filling in the gaps in process & roles -- that companies have grown to depend on.  Whilst this is great and generally seen as heroic efforts, the fire-fighting and critical moments can be saved if a little time is spent up-front by defining & clarifying the project structure.

Again, in the context of DTV projects - the structure depends on the viewpoint.  In general, if you're a broadcaster / PayTV operator (BSkyB, DirecTV, Multichoice, etc), you will have to implement a detailed project team as there is a much wider impact on the business (development, lab, testing, validation, going live, operations, support, customer service) than compared to one of your vendors (NDS, OpenTV, Irdeto, etc). If you're a Professional Services outfit acting as Systems Integrator and ultimately responsible for the delivery of the project, then you might have to mirror the same project structure of your customer.

Let us look at a picture to see where these roles can be placed. Notice this map is exactly the same layout as the architects post:

Figure 1: Overview of DTV System highlighting Project Roles

If you're an Operator focused on content delivery only, then...

You generally hire a System Integrator to manage the design and end-to-end delivery of the system on your behalf. These operators make a conscious decision not to be technical experts and thus rely on preferred partners to manage new development and operations.  The operator would still have someone who's responsible for product ownership, which could be someone from Marketing/Sales. There would also be a Programme or Project manager assigned to manage the project.

This is typical of first-time operators entering the market from cold, there are some professional services companies like NDS, who can manage not only the end-to-end systems development or customisation (generally this is off-the-shelf to enable shortest time to market, followed by product updates post launch) but also able to commission a broadcast centre and operations control from scratch...

So essentially this is an operator that outsource most of the Delivery Project Management - the organisational structure will pretty much be the same as Figure 1.

If you're a mature Operator with an established product and adding new features, but without technical ownership, then...

This scenario again relies on the Operator leaning on preferred partners for providing the technical solution, possibly spanning multiple vendors involved in development of components, system integration, delivery and deployment as well as operational support.  There would be a Product Owner / Manager that defines the high level requirements for the product's feature set. There could also be a Project Office team that takes ownership for the overall co-ordination effort from start to finish, or this could be outsourced to a professional services company to manage the whole project.

Some Operators have separate business entities owning features of the overall value chain, so the primary product spans multiple product owners. It becomes more challenging to manage this distribution of product owners especially when trying to manage an consistent brand and user experience, and coherent presence in the market. But companies are doing this today, it works to an extent, the key being effective communication is pivotal to successful product launches.

Generally though, in this scenario, the Operator has established strong relationships with its vendors, and projects may follow a typical template plan that would be driven by a Project Office team.

You might find the Operator would enlist a Project Board to manage the project's priorities, resources and budgets - that a Product Owner/Manager and Programme Manager reports and takes direction from.

Again, the project organisational breakdown will follow Figure 1.

If you're a mature Operator and responsible for developing and delivering new features, with technical ownership, then...

This can be very complicated and challenging if this is the first time the Operator is responsible for technical ownership. As touched on in my previous post, more and more Operators are taking more ownership in managing DTV projects, around the area of STB & Headend Systems Integration, EPG Development, Interactive Applications development, and some Operators are even developing their own Middleware.

In this scenario, it then becomes fundamental that the organisation starts on the correct footing, ensuring the foundations are cemented correctly, processes and practices firmly put in place as early as possible -- not doing so will definitely result in chaos, which I personally have come to label a "Project Fiasco".

So the roles I'd expect to see an organisational structure clearly defining the Architecture roles, Development, Integration and Testing; as well as a well defined Project Team.

I would expect to see a variant of Figures 1, 2 & 3 for the organisational structure.  There will be interaction with vendors. The relationships, roles & responsibilities must be explicitly communicated....

If you're a Product Vendor that delivers components to Operators, then...

Product Vendors are generally companies that provide software components for STB or Headend components that enable the Operator to function.  As a Product Vendor, the challenge is to internal manage your development organisation to maintain a consistent product line offering features that can be easily customized, tailored or profiled in or out to meet the requirements of your customer, the operator. Typically vendors manage multiple customers. Some customers more important than others - the customer usually generating the most of amount of revenue is assumed the lead customer, if not, the customer that in fact drives forward your product development and innovation.

Vendors typically have a Customer Delivery Manager assigned as an anchor support point, interface to the customer. This Delivery Manager ensures the customer's requirements are accepted by the product development team, and ensures the resulting plans are in accordance with the customer's expectations.  The vendor may choose to use a Product Steering Group that ultimately makes the decisions on customer priorities. Depending on the extent of the products being supplied by the vendor, I would expect to see an organisational structure not so dissimilar to Figure 3.

Brief Definitions

• Product Owner / Product Manager
• Operator - responsible for defining the user experience, high level requirements of the features for the product. For example, Operators looking at VOD will have a Product Owner owning the feature set, along with priorities. The Product Owner takes direction from the Project Board or even the business in understanding the market priorities and urgency of the business w.r.t. launch targets. The Product Owner / Manager will use the services of the architects and project team to get the product specified, designed, implemented and ultimately delivered.  The extent of ownership is around product specification only...The Product Owner/Manager gets actively involved in prioritizing defects leading up to the launch phase of the project, setting priorities, accepting waivers or concessions, etc.

• Vendor - as a vendor, the Product Owner / Manager is different to the role taken on by an Operator.  As a vendor, the Product Owner is concerned with satisfying as many customers as possible, without much deviation or customer-specific features; and will work hard to maintain a consistent offering. The Product Owner generally consists of a Product Team that centrally manage the product backlog. Depending on the size of the projects and demands from customers, this could be one person, or a team of people. If it's a Headend component with specific technical requirements (generally Headend component is a translator - data in, data out component) that implements a protocol, then this could be one person.  However, if this product is a STB Middleware that must sustain multiple customers, and potentially support tens of millions of boxes, then you'd need a separate team managing the product. This Product Management team will own the roadmap for the project, control the work assigned to customer projects, and ensures a coherent release schedule.  The team will also ensure the product specification and requirements are kept-up-to-date supporting the generic product as well as the customer's profile of the product. This Product team will consist of project managers driving through the product roadmap.

• In some cases the term Project Owner / Key Accounts Owner / Customer Delivery Manager is used which basically is the person who's neck is on the chopping block - i.e. the person accountable for the project delivery, who has seniority to make judgement calls to steer the project along in his/her preferred direction, etc - basically the ultimate person in charge, the "X says so" person...

• Programme Manager - this can be generalised both from Operator and Vendor side - and is typically the person responsible for the high level planning of the various work streams that make up the ultimate delivery.  There can be Programme Managers for each system, primarily Headend & STB, as well as managing the overall End-to-End delivery plan.  The Programme Manager works side-by-side with Product Owner in understanding business priorities, but the Programme Manager has autonomy in defining the plan and driving through the execution of that plan.  Programme Managers lean on their respective project managers to take responsibility for the detailed planning that delivers on the high level milestones.

• Project Manager - is responsible for planning, managing and driving through the deliverable of the assigned component/project. Reporting and taking instruction from the Programme Manager, the Project Manager handles the day-to-day issues/planning/tracking, escalating to next level (Programme Manager) as required.

• Scrum Master - can be a Project Manager or Team Leader employing the philosophy of Scrum/Agile to get the component/project team to deliver according to the milestone plan put forward by the Programme Manager and Project Manager. 

What a Product Owner / Manager is not responsible for...

When not acting in the capacity of the Project Owner, i.e. the main person accountable for the success of the delivery, then the Product Owner / Manager should only be focused on managing the Product requirements and Feature set, setting expectations and aligning priorities.  A Product Owner has no influence on the resource structure of the team, the scheduling of the development....that's what the Project Team, including the Programme Manager is there for - the Project Team provides a service to the organisation...Depending on the size/complexity of the deliverable, in DTV projects, they are large enough to warrant a clear separation of boundaries between Project Team & Product Team....Having people stomp on each others toes is not conducive to productivity...

We are an Agile Company, we don't like heavy Project Organisation...

Sure, depends on what kind of company you are. If you're in Application Development writing EPG UIs or Interactive applications, then of course, you can be light on the ground.

If you're in the Middleware business and big enough to be mentioned in the same sentence with NDS, OpenTV, etc -- and you don't have these structures in place, then you're at a competitive disadvantage. I have worked on small and large projects alike, the lack of a clearly defined and sensible Project/Organisation structure does not work...you might reach there eventually through luck, a lot of hard work and unnecessary fire-fighting...

For Operators taking the plunge at taking more technical ownership in your projects, my advice to you is "Do your homework". Spend time up-front investigating, planning and settling on a processes that has been known to work, i.e. proven - by getting real consultants in to advise...By real consultants, I mean not the people who know the theory but have never managed a real life project from Start-to-Finish, not someone who's never worked in the DTV industry before...Invest in the right people up-front, employing or head-hunting candidates who have first-hand experience and proven track records, and pay heed to the advice henceforth...

Typical Project Org Structure for a typical Operator

Below is a simple model of how an Operator might define an Organisational Structure of a DTV project...

Figure 2: A possible Org/Project Structure from the Operator's viewpoint

Typical Org Structure for a Vendor supplying components...

Below is a complicated model of a Vendor who's main business is STB development. Some vendors specialising in components in specific components will own a slice of this structure...

Figure 3: Org/Project Structure from a STB Software Vendor's Viewpoint

PDFs of Diagrams

Figure 1

Figure 2

Figure 3

Your Feedback

Please note my posts are always based on my first-hand experiences from past projects, and is a reflection of reality instead of theory. I am always interested in feedback, please feel free to comment or rate this post!