What makes OTT Streaming platforms different to DTH/DTT STB Broadcast?

In the early days of this blog, I shared in deep detail the technical and project landscape of PayTV systems for the traditional broadcast headend and set-top-box (STB) software platforms and projects as subject matter expert in this area.

In this post, I will provide a high level overview of the core differences and challenges between broadcast DTH/SetTopBox systems versus OTT platforms. OTT has become a hot topic, with the nascent rise of OTT (Over-the-Top) internet streaming platforms that many a traditional PayTV operator is now transitioning to, in light of the growing competition for eyeballs & cord-cutters driven by the likes of primarily SVOD (Subscription Video On Demand) players like Netflix, Amazon & the like. The challenge though, is that these PayTV operators tend to trip up by bringing PayTV/DTH thinking into the OTT-world, which is why I felt the need to write about this, because I have, and still am currently experiencing these challenges. Suffice to say, it becomes very tricky to disagree can call out a high level broadcast TV executive's assertions (expecting like-for-like DTH & OTT experience) as not completely accurate when it comes to OTT engineering...

The main aim of this post is thus targeted at senior executives who come from a broadcast world, and may not fully grasp the nuances of an online streaming world, specifically pointing out the spectrum of control of the end-to-end platform that PayTV operators generally enjoyed much comfort in that world, in contrast to the new world of online, where it becomes impractical to expect the level of control previously enjoyed - basically you can't control all the components of the internet, unless you invest in owning a large part of the network, which some operators have attempted & haven't really succeeded!! Simply put, the internet was not designed for Live TV streaming unlike traditional broadcast TV - the internet is somewhat chaotic & messy to say the least. In addition to this level of infrastructure control & lack thereof, an OTT platform is largely software systems designed for internet-scale (i.e. massively distributed server-side backend components) which requires a different skillset and experiences traditionally enjoyed in the closed ecosystem of broadcast & set top box device software development teams.

Another point of this post responds to the standpoint from traditional PayTV executives who may have spent a large part of their work in building broadcast systems with an engineering mindset anchored to a past era that is no longer relevant in the OTT world. Expressing opinions such as "engineering systems either work or they don't, it's either a 1 or 0, no in between states..with an expectation of 100% always-on availability..." in my view is somewhat misinformed & borders on belittling the engineering challenges faced in the online world. Whist every responsible engineer understands such demands and expectations of any system, a key point to remember is that broadcast TV has benefited from many years of refinement, standardisation, systems & processes specialised for the domain of TV - and when it comes to the internet - well, it was never really designed for video consumption, the speed of innovation has outpaced the development of standards & so is a much more messy world compared to traditional broadcast TV (satellite/terrestrial).

Disclaimer: This post is not about downplaying the broadcast engineering world, in fact, a lot of OTT operations highly depend on the upstream capabilities from PayTV operator's existing broadcast engineering infrastructure. Broadcast engineering systems are a marvel of engineering in its own respects, systems are highly complicated & superior engineering. OTT systems, and the point of my post, is to show the extension of the value chain, highlighting the new software & systems engineering challenges that present itself in the OTT world, that will be new concepts for existing broadcast engineering teams to learn & appreciate.

Key points for TL;DR folks (especially the PayTV Exec CXOs)

• It's not a simple thing to compare OTT platforms wearing hat of a traditional DTH broadcast guy, it  is not simply a like-for-like comparison.
• Don't be naive and pass value-judgements just because you were an engineer in the broadcast TV world (Headend or Set Top Box) does not necessarily make you qualified to understand OTT engineering domain. You can't bring broadcast/STB-thinking-mindset to the world of OTT.
• PayTV operators have historically enjoyed a high level of end-to-end control, not so in OTT world. The internet is outside the PayTV operator's control.
• There are far more components in the value chain of OTT than broadcast (backend, frontend and device software stack is more complicated in OTT than STB software).
• High level of device & platform fragmentation in OTT that a traditional PayTV operator must consider.
• The internet was never designed for TV consumption & has innovated at a much faster pace than traditional broadcast TV, the internet is a messy place, broadcast TV is somewhat cleaner and more determinstic.
• OTT engineering teams have different challenges compared to traditional broadcast & set top box engineering.
• PayTV operators need to plan carefully the convergence (DTH & OTT) of engineering teams and be prepared for a "clash of worlds".
• Set Top Box software systems enjoy far more security control & protection than its OTT sibling.
• Peak events & load challenges are unique to OTT platforms compared to broadcast DTH STB (scaling challenges don't really exist in DTH/STB).
• Redundancy, Fast Rollbacks, Failover & Disaster Recovery are easier to achieve in OTT than in broadcast DTH/STB systems.

How TV is PayTV changing?

Today, the world is different and changing all the time. With the rise of the browser, "streaming over the top" has advanced at a much faster pace than the traditional broadcast world. There are so many topics I can talk about, which might become future posts. Take for example how software patterns for distributed systems had to evolve for internet web scale. A mistake often made in the early development of online-equivalent of traditional TV streaming, is one where the software stack for the streaming world emulated that of a STB, expecting real-time processing, synchronous calls to APIs, etc. Depending where you started, you might have a lot of legacy code to clean-up & undo design patterns that worked previously, but can't scale sufficiently to deal with exponential growth of online users. Some PayTV operators may have anticipated this online streaming by building their own online streaming platform, maybe 5-7 years back, resulting in some legacy already. Today, with the rise of cloud providers, and the constant comparison with the god of streaming (Netflix), PayTV executives expect the same level of performance & reliability from their in-house engineering team. Deciding whether your legacy online platform has reached the end of its shelf life, whether to continue to the restructure journey (e.g. migration to cloud), or hit refresh, starting from scratch build, or use off-the-shelf platforms, is a topic that has kept me awake many a sleepless night. A topic for another day. Another topic that makes traditional PayTV executives a little uneasy, is that the engineering, development, delivery & operational management for OTT platforms are much more dynamic & agile in nature, the fact that apps can be patched and upgraded quickly, that deployments to production happens almost daily, challenging traditional governance and change control procedures that demand varying levels of sign-off & approval, that developers can access production environment, etc. are topics IMHO that every senior executive from the old-world of PayTV, needs to come to terms with - i.e. don't expect your digital OTT engineers to behave the same way as your broadcast engineers! Anyway, these topics are best left for future posts...

Right, now that's off my chest, let's look at the big topics that speak to differences in traditional broadcast satellite TV platforms compared to OTT platforms:

Satellite PayTV Operators enjoy End-to-End control of Broadcast Platform

Apart from the satellite transponders in space, the majority of engineering services is usually under direct control of the PayTV operator. This includes reception of content streams / signals, encoding, encryption & broadcasting packetised transport streams, using satellite uplink & downlink stations. The headend consists of robust hardware & software components, that implement well matured and estabilished specifications, governed by international bodies. Encryption of services is handled through conditional access technology that has been around for 25+ years already. These systems are expected to just work 24/7/365 & highly reliable and fault tolerant. PayTV operators may manage these headend services themselves, or as a managed service using a preferred partner or systems integrator. These systems are highly deterministic & can be tested thoroughly with defined input/output acceptance tests, using reference client side decoders to prove end-to-end, the transmission & reception of these TV signals are working to spec.

On the client side, PayTV operators provide a device called a Set Top Box. This is a consumer device that enables signal reception, decryption & viewing of the TV signal. This device consists of an EPG application, running on an operating system (middleware), that's hosted on the device platform by way of device drivers. This STB software ecosystem is under total control of the PayTV operator. All customers will have the same version of software installed, the operator manages software updates and can force the devices to upgrade/download new software updates. Because the operator has upstream and downstream control of the environment, even if there's multiple vendors providing components and services, generally debugging and troubleshooting end-to-end is possible.

An operator might have different variants of hardware for its STBs, but effectively these devices share a common middleware & application stack, drivers will be specific to the hardware. Even with these variables, the operator has full knowledge and control of these components. Nothing can change on the headend components, encryption services & client-side STB without the knowledge of the operator, nothing! It's essentially a closed system.

In terms of satellite capacity, this bandwidth is usually provisioned and planned in advance to match the PayTV Operators content/channels growth projections. The signal is received by all consumer STBs. It is pretty much a fully deterministic system end-to-end. The STB is a closed, fully controlled environment, although I have seen some STB software that becomes unstable over time, that operators build in a "reboot every morning" to offer some stability of user experience (think memory leaks)!

About the only things outside of operator control are mother nature, customer home installations & of course reliable power/electricity (anomalies in space which is rare, heavy rain causing loss of signal, and poor installations). Assume the rest of the value chain (like fibre links, satellite uplinks, etc. are being managed well enough). Depending on how much legacy old STBs the operator may have in the field, this poses a challenge in terms of signalling/bandwidth to keep these services running, and end consumer is usually recommended to upgrade their home set-up to make use of the newer installation & STBs.

In short, whilst there is some variability in what's within operator control, these amount to just a handful & thus quality of service to end consumer largely sits with the PayTV operator.

In contrast, OTT Platform Operators enjoy less End-to-End distribution control

In the broadcast world, all the end consumer does is: Install a satellite dish, make sure the signal strength is strong, connect the STB decoder, activate the subscription - and voila - guaranteed quality TV signals received. There aren't too many components in the chain, it's simple. Customer signs up with the PayTV operator and expects quality TV beamed to his TV, always on, period (unless it's Africa where electricity is an issue).

In the OTT world, what replaces the satellite broadcast, is the internet that sits between the end user and the operator - and this is not a simple world - the operator relinquishes end-to-end control.

First, consider the OTT customer. He has to get a decent internet connection, by signing up with an ISP (Internet Service Provider). The speed/bandwidth of his connection needs to be at least 10 Mbps connection, with an expectation of consistency & reliability that his connection is always on and available. After-all he is paying for the service from his telco or ISP. But this is not always guaranteed. Unless the customer reads the fine print, ISPs & Telcos have a challenge of maintaining their service levels as more and more users join their network, consume bandwidth, especially during peak times of the day (evenings, weekends, etc.). When it comes to fixed line internet / fibre, this isn't usually a problem, however, some internet providers have something called fair use policy & limits people from "hogging the line", and may revert to traffic shaping or rate limiting. So consumers need to be aware of the nature of their package & underlying terms & conditions of the service. Then there is the data consumption to consider, some people subscribe to data limits, e.g. 50GB or 100GB caps. This is okay for normal internet consumption, but as soon as you start consuming video, especially high quality live streaming content, the data can be consumed pretty quickly.

On the ISP side, depending on the size of their subscriber base, they might be challenged to provide decent levels of service, speed and bandwidth to their consumers. ISP networks also need to have a big enough pipe (i.e. connection to the internet) to serve their customers. 

Wireless/Mobile network operators providing 4G/5G/LTE internet access face similar challenges. Their capacity is constrained by their physical infrastructure (spectrum availability, base stations, transmitters, capacity of cell networks, congestion traffic management, etc.).

Some PayTV/OTT operators may also act as the ISP, however the same challenges face them as for normal telco / ISP providers. Most common though, OTT operators have to rely on partnerships and goodwill of ISPs for allowing OTT video traffic to flow through their networks. 

To help provide a decent video experience to their customers, OTT operators employ the help of CDN (Content Delivery Network) providers, by aiming to ensure video content is available as close to the customer as possible, by deploying video caches within as many telco/ISP networks as possible. The OTT operator has the choice of building out this video CDN in-house, or rely on partnerships. When partnerships are used, the question of who has responsibility for the end customer experience comes into play. Who's responsibility is it to ensure the OTT customer gets the best streaming experience possible?

Some technologists tend to argue the point that customer experience is the responsibility of the operator and therefore should own the network & CDN as much as possible. This then gives operator end-to-end control.

My opinion - it just depends on the OTT Operator's appetite for risk, amount of funding & of course level of engineering skills. I prefer not to re-invent the wheel, and instead partner with the best network & CDN providers in the world. There are CDN providers with decades experience, knowledge and pioneers in the field of video encoding, transcoding, compression & distrubution CDNs - why build your own? How important is ownership of the technology platform to an OTT operator? Is the operator in the content business or technology business? Is application user experience, understanding the customer, analytics and recommendations more important than underlying networking infrastructure? What level of investment is the operator willing to make?

Video streaming CDN providers are widely available - it's no longer a pioneering, niche field. Whilst a VOD CDN might be something an operator can practically own and build in-house, even to the extent of using open source, freely available caching software, there's still investment in capex and operating costs of management of the infrastructure. Live streaming video is somewhat more complicated than VOD content.

Satellite broadcast enjoys reach by way of beaming from space, covering a large footprint of geography. This distribution mechanism is very convenient, and guarantees bitrates at constant levels. The customer STB is likely to receive good quality signals by virtue of this direct-link (wireless) connection.

OTT distribution shares somewhat similar challenges to a mobile / terrestrial network. Just as these radio networks require cell towers to located within geographical regions, OTT networks need to provide similar distribution of network points, but not as many as these radio base station sites.

Depending on when the OTT provider began its journey into OTT video, this could vary between providing CDN caching infrastructure within telco and ISP operator points-of-presence stations, or choose to be reside instead at large internet exchange points, these are meeting points where the world's internet service providers come together and allow networks to peer with one another. Peering gives operators the benefit of jumping on these big highways free-of-charge (typically) so content can be exchanged between these networks. A more recent option is to offload the CDN to the cloud providers like Microsoft Azure, AWS or Google Cloud. I prefer partnering via managed services than build-in-yourself, as long as the operating costs are manageable and makes sense.

So bottom line: Satellite enjoys far more control end-to-end compared to OTT distribution. OTT requires more partnerships and a higher tolerance for risk, something which traditional PayTV execs might find uncomfortable.

The internet is full of variability, can't be avoided but can be effectively managed through diligent monitoring & partner service level agreements.

OTT Applications fragmentation compared to STB Application Development

In the broadcast DTH world, the client-side software ecosystem, call it the STB application stack, is usually under the control of the PayTV operator. This software system has benefited from decades of experience in building & harmonising the system capabilities & features. There was lots of focus from standards bodies such as ISO/DVB/MHP, etc. providing reference designs and protocols for set top box software requirements. This opened up a niche for companies in the "middleware software" business, essentially providing the operating system for the device. These middleware providers took great care in abstracting details of device hardware & chipsets, exposing a platform SDK for application development. PayTV operators tend to have multiple hardware variants often using different hardware manufacturers to spread costs & risk, with the comfort of a middlware SDK providing consistency of application user experience. Generally, this application is referred to as the EPG "Electronic Program Guide" - the main user interface of the STB. These applications started their journey as apps built on native C/C++, then transitioned to Java, and later Flash/Actionscript, HTML and now more recently moving to web app technologies built on frameworks such as React/node.js. (I've experienced all these transformations).

The good thing about the STB world is consistency and ownership of the software stack. Building the EPG once and running on various hardware platforms is made possible by Middleware SDK platform, with minimal customisation. Through standardisation of interfaces and robust test frameworks, once the platform SDK is qualified for hardware variants, the EPG software can be ported or "just run" on any device compliant with the middleware SDK.

The operator is in full control of the experience and controls the platform SDK versions, and is usually responsible for integrating the full stack. Such operators have full control over software upgrades & having a big influence on platform software roadmaps, etc,

Not so in the OTT world...Device/Platform SDKs are more than 1...
In the OTT world, the new EPG application has to contend with multiple devices not under operator control. On top of this, i.e. not having control of device hardware, the platform SDK for application development is also not under operator control. These platforms are provided by other enterprises, that provide utility for generalised application development:

• Web browser support. Although there are web standards defining complicance of HTML specifications, browsers come with their own tweaks and features. An OTT app therefore needs to comply with a minimum acceptable implementation that would make the app work for multiple web browsers (Internet Explorer, Chrome, Firefox, Safari, etc.). 
• Depending on when the OTT operator began their application development, the webapps may have been built using different codebases, tailored for specific platforms and as the web matured, these apps are now converging on common codebases.
• Underlying platform operating systems: Linux, Windows, MacOS - are not under control of OTT operator
• User devices have all sorts of applications & drivers installed, as well as web plugins, etc. Unlike the STB world which is completely locked down and in control by the PayTV Operator, the new client environments are very open. This makes debugging and customer support all the more challenging, compared to STB EPG applications.
• Mobile/Tablet Platforms - dominated by iOS and Android - not under control of PayTV Operator
• OTT Apps are built on these unique platforms that offer their own flavours of application user experience design, rules and frameoworks, that app developers must comply with. So unlike a standardised EPG application on PayTV operator's hardware, where PayTV operator standardises on the UX making it consistent across all hardware decoders, nuances in the mobile device world, does not make this possible. So OTT app designers need to understand the frameworks and constraints of these ecosystems when designing the applications. Software engineers also need to be expert at more than one platform environment, which means
• Instead of pool of application developers writing one EPG application in the STB world, OTT development teams often are specialised by platforms, hence needing separate teams and skills to develop for iOS, Android, other.
• OTT applications have to keep pace with software updates from iOS & Android - again not under control of PayTV/OTT Operator.
• This leads to many versions of applications in the field that need to support not only the recent platform OS versions of iOS & Android, but also provide the best customer experience possible to customers who have old devices not compatible with the latest OS software updates (this is especially true for Android devices, due to its rather open model, unlike the closed iOS ecosystem, Android enjoys much freedoms, often with OEMs building & forking their own customisations for Android).
• No longer is the PayTV operator in control of the end consumer device
• Smart TV platforms - following the success of the mobile application ecosystems, Smart TV  (STV) manufacturers created their own platform SDKs and app stores as well. There is very little standardisation. OTT Operators want to get their apps on many devices as possible, with Smart TVs being increasingly used by customers, OTT Apps are desirable for these TV platforms. STVs also provided bespoke platform SDKs initially (Samsung, Hisense, LG, etc.) which meant that OTT operators develop yet another variant for these applications.
• Converging on a common TV platform SDK makes sense, but this is at the mercy of STV manufacturers. At the very least, a minmally compliant web browser stack does pave the way for some standardisation, and more recently with the increasing popularity of AndroidTV as a good platform.
• Smart TVs also have many variants of hardware models and software versions, which makes OTT application management a little tricky, backwards compatibility challenges, and overall software code management when it comes to device-specific bugs to fix.
• This complexity rarely exists at STB level where PayTV operators have full control of their hardware and software versions, and generally the middleware OS provider and device manufacturer aim to abstract much of the complexity away from the STB application developer.
• Software updates are not aligned in the OTT world
• Set Top Box software updates again benefit from being under full control of the PayTV operator. Customers really have no choice in delaying software updates, the operator is in full control.
• In the OTT space, with a variety of devices & platforms to support, we're are the mercy of application store review processes: some are quick & can be done within a week, whilst others like Smart TV updates, takes longer. This thus comes with challenges on the software release update process to manage and control the lifetime of app versions in the field for different platforms.
• Customers have full control of their personal device settings, leaving no guarantee that updates will be actioned in a timely manner. This creates unknowns for the software support teams which also calls for discipline in managing software version variants as well as placing some challenges on customer support people as customers don't necessarily have the recent updates installed.
• OTT devices have variations of chipsets & video player components for video decoding & playback
• A vital component to the customer experience is video playback. Video needs to start up in good time, and the streaming experience must be free from buffering & other issues such as lip sync, stuttering & so on. 
• Since the devices and platform OSes are not standardised, OTT devices have different video component capabilities, although respecting the MPEG standards. An OTT provider has to decide whether to go native on the player component, build their own, or use COTS player components from vendors who manage all the complexity. This then requires specialist skills in the OTT software teams around player & video components, perhaps not too different to a set top box video engineer.
• In the STB world however, these challenges don't really exist at the EPG application layer, because the specifics and technicalities are abstracted by the middleware and hardware/chipset vendors who drive the implementation. The choice of chipset and video codec support is still owned by the PayTV operator, therefore, from a software point of view, there is really no variation. One STB software build is needed, application developers are not exposed to that level of complexity.
• In the OTT world, depending on the consumption device, the stream will have different formats for the device (HLS, MPEG-Dash, Smooth Streaming, etc.) where as in the DTH broadcast world, it's really just one output format that all STBs can consume.
• Conditional Access (CA) Systems versus Digital Rights Management (DRM)
• In the STB world, CA is the primary means of providing security of content, built on strong encryption developed over the years to make it extremely difficult to exploit, due to the tight coupling of chipset security, hardware protection and tight association with physical smartcards (recently CA providers moving away from smartcards to reduce cost, however there's still a tight association with CA vendor & chipset providers to conform to security implementation of key ladders, etc.) & frequent key cycling has resulted in PayTV operators having a good handle on platform security. The STB software contains a CA software component that is largely consistent across all device variants under PayTV control, with some STBs capable of running multiple CA systems (simulcrypt).
• In the OTT world, because of the many platforms available to consumers, OTT platforms have to support multiple encryption requirements in the form of DRM, which is largely software, confirming to open standards for these devices to comply with for key management associated with hardware. However, the PayTV operator really has no real say into custom security measures. So a form of multi-DRM has to be supported (Widevine for Google, Fairplay for Apple, Playready for Microsoft), thus depending on the device & implementation of the DRM, a customer's experience might not be consistent (for example, time to play video is no longer truly deterministic as compared to a STB experience). CA providers naturally have evolved to providing DRM services, however there are some OTT software teams who may have opted to build their own DRM services. This could be something strange for a PayTV executive to accept though.
• Build your own OTT-device to own content aggregation
• Some PayTV operators consider building their own OTT devices or Hybrid-STBs to help gain control of the platform, aiming to address the concern of providing their OTT apps on third-party "open" platforms where they're not in control of the home page, content experience - "Why should we be at the mercy of another provider for content aggregation? Our content and apps need to be promoted first"
• This brings another set of challenges because although we now move back into the world of control enjoyed by STB software, it means for responsibility for integration, tight coupling to the hardware/software stack, blurring the lines between having an open, portable OTT app that enjoys benefits of a platform SDK provided by a third-party, to owning most of this environment. It almost comes back full circle to traditional STB projects, but can be a nice bridge and gap to close, especially if the PayTV operator had separate technology teams working in isolation (i.e. a separate OTT team and separate DTH STB development team).
• The ease of deployment depends on the software platform chosen for the device, today the strong contenders are AndroidTV versus RDK, Android TV stack provides out-of-the-box re-usability with some custom development, RDK on the other hand, necessitates a full blown software integration for a typical STB stack. OTT application developers prefer open platforms, and thus promote the support of web browser engines, for application reuse, "Build once, run everywhere" mindset.
• When PayTV operators decide to go down this "own-device" route, it will likely slow things down with additional challenges on the OTT apps team of having another device platform to support - the challenge though is a clash between worlds, from an engineering management perspective, needs to be carefully considered.
• OTT platforms not fully in control of the product roadmap & platform updates
• In the STB world, all updates are managed by the PayTV operator, who enjoys the luxury of choosing when to apply updates to the systems
• In OTT, because operators don't enjoy platform control, software updates need to be tracked and managed, sometimes, this is not always possible. Some vendors remove browser support, or might make updates to their DRM/security support, and if caught by surprise, could leave OTT services broken, upsetting many customers.
• OTT architects & software component owners therefore need to keep very close to changes happening on dependent services, most of this is usually abstracted and handled by middleware vendors in the STB world.
• OTT engineering teams therefore need to be more flexible & deliver emergency changes to handle these break-fix changes

Security Threats more exposed in OTT compared to DTH Broadcast

In the STB world, through many years of refinement, it has become increasingly complex to crack and pirate, although not always impossible, but the reality is that it is quite difficult to break. This is because historically of the hardware-level protection built into custom hardware, paired with deep integration of security layers within the hardware chipset, as well as the pairing of the decoder with smart card technology. This is done through tight partnerships with Conditional Access (CA) vendors, who take responsibility for securing the platform. Along with frequent key cycling (crypto periods), and by way of standard protocols, and very long key ladders, and up-to 256-bit encryption keys, these platforms enjoy a robust level of security. An attacker has to be very skillful, and also have access to large amounts of funding investing in tools & deep-probe machines at hardware level to break the codes. Even with the movement to software-defined CA, virtual smartcard technology, it is still quite a solid ecosystem to crack. It is impossible to "credential share" by swapping smartcards with decoders because of the physical pairing of hardware devices, the only way is to take the physical decoder and the smartcard, to share with friends and family, assuming these people have the necessary cable installation installed.

In OTT applications, as this is largely a software based ecosystem, the potential for exploits are greater than the rather closed system of broadcast/STB:

• Credential sharing - this is a simple case of sharing one's username and password with family and friends. Most OTT apps provide for simultaneous concurrent usage of at least two devices streaming at the same time. This is a handy way of sharing an acoount with a friend or family member, saving on subscriptions. This leads to revenue leakage on the OTT Operator side, which is a challenge that has yet to be solved, so is a pretty hot subject in the OTT streaming world.
• Compromising accounts - brute force. Apart from conscious and open sharing of credentials as above, OTT platforms are more susceptible to brute force attacks compromising accounts. This places an expectation on users to have strong passwords for these apps, but is a behaviour that needs to be enforced, OTT apps to most users are not that important in terms of security as say, compared to one's online banking profile. Nevertheless, OTT application providers have to balance security against the demands of seamless customer user experience, the app needs to flow simply to provide a great usage experience, so overly emphasising security measures might not provide the best experience. 
• OTT software still needs to provide adequate measures on monitoring, detection & prevention of brute force attacks on credentials. This implies the OTT engineering team need security experts, which traditionally is a skill and resource provided by CA/security vendors.
• Piracy of OTT streams - due to the rather open nature of online systems, it is far easier to learn and re-engineer webapps than in the closed STB system. By virtue of freely available tools and debugging environments, an OTT app's code can be exposed in websites (javascript, etc.), intercepted & modified to trick backend systems. This exposes such systems to pirates, making it much easier, lowering barrier to entry in terms of costs / investment needed to run exploits & attacks. Most exploits in the online world (like phishing) are also prevalent in the OTT apps world.
• Denial of Service attacks - OTT apps are more susceptible to DDos attacks compared to broadcast world, in fact, it's probably nonexistent in broadcast, unless the internal headend / transmission network is infiltrated, or a DTH STB is connected to the internet. For the points of this post, focus on nonconnected STBs means this problem is nonexistent. OTT on-the-other-hand need robust software, network protection, firewalls and additional security prevention methods implemented to provide a high-level of service. Expecting your system to either work or not, a 1 or 0, is rather a naive mindset IMHO.
• OTT Region-blocking is complicated compared to DTH/STB. In DTH broadcast, in a multi-country PayTV network, STBs can be easily locked down by territory, by way of signalling protocols for DVB as well as CA-security mechanisms. Also, the satellite footprint makes it impossible to access PayTV signals if the region does not fall under satellite footprint. 
• Not so in the OTT world. The internet has no boundaries. Country blocking is easily bypassed by using VPNs (Virtual Private Networks), and other mechanisms to fool the systems. VPN blocking might be a good deterrent, however keeping up-to-date with VPNs that pop-up almost on a daily basis, is something an average OTT engineering team is not equipped to handle, and hence must partner or use off-the-shelf services that manage this. 
• It is far more easier to circumvent region-blocking in OTT world than the DTH/STB world. PayTV executives may need a crash course on how the internet works, and main ways of exploiting the network, as well as the idea of "control" may be almost an impossibility in the internet, where no boundaries exist. This might be a bitter pill to swallow. 
• Device Concurrency Limits do not exist in DTH/STB world but is a mandatory requirement for OTT world. This implies complexity on engineering system to manage in real-time, the number of concurrent streaming users and devices at any moment in time, consuming the service, and if device limits are reached, prevent any further streaming access until devices become freed up for the user. This logic is complex to implement to say the least, such complexity does not exist in traditional broadcast software Set Top Box management. An improper implementation of concurrency opens up security risks - but more importantly, on the nature of OTT scaling, the more users on the platform, the more checks needs to be done, the more loaded the backend services become, which could either slow down the system, or bring it to a complete halt altogether. Naturally, an uninformed PayTV executive, might not understand that scalability is never infinite, despite how nice the coolaid tastes from the various cloud providers...anyway, I digress, the important message here is that OTT software systems offer different challenges that a PayTV executive, even with a broadcast engineering background but not an internet one, will need to appreciate.

DTH Broadcast not impacted by peak events, scaling on demand for OTT

In most ways, a broadcast DTH system with a reasonable satellite footprint, coupled with a solid Set Top Box platform, is a PayTV operator's first choice. It's a simple, straightforward world, where you're in control. Scale is a matter of satellite footprint and penetration of STBs. All users receive the same signal, no matter if it's 1, 2, 10m customers. There isn't any additional infrastructure or systems needed to support increased in customer growth - apart from maybe - the cycling of keys on a more frequent basis, getting more transponder bandwidth, to manage CA entitlement messages - but nothing compared to what's needed for OTT systems.

An OTT system largely consists of web application services in the backend that are constrained by typical topics: network, CPU, memory, database, and of course software implementation (poorly designed and coded applications not for internet-scale is many an OTT-engineering teams worst nightmare). Anyhow, the simple requirements for scaling an OTT platform are:

• System must be able to support 1, 2...10m users concurrently streaming without any degradation of customer experience
• System must be able to handle peak events - where generally, customers wait for last minute to enter the platform (login/authorise/validate eligibilith/etc). Handling a million logins in one minute is not an unusual expectation from an OTT platform that provides live streaming channels (compared on an SVOD service which does not experience such peaks).

Depending on where the OTT engineering team had started, most would've been typically before the advent of cloud services or before cloud-native stacks became mainstream, which meant traditional IT service/app management, running virtual machines, which relied on physical infrastructure being provisioned etc. The lead times were generally long anyway, with application developers erring on the side of overprovisioning - which means having hardware/services installed but running idle most of the time, and would only see the light of day during spikes in demand.

Nowadays, the expectation is that systems are self-expanding, scaling on-demand: when demands are high, the platform automatically scales up/out to support the load, and when demand decreases, the systems scale down accordingly. This is the ultimate place to be, however, there is still the constraint that there is really no infinite capacity. Even with cloud service providers, this is all about tenancy and shared occupancy, no one really owns guaranteed capacity unless it's made explicit and paid upfront (which kind of defeats the purpose of having "resources on-tap"). So even with a scalable platform running off-the-cloud, there is no guarantee that limits in capacity won't be reached...again, when a PayTV executive demands 100% reliability during peak events, one has to wonder...

Not only is the OTT software backend systems impacted by huge demand during peak events, but so too is the streaming CDN which relies on internetworking of ISPs, Telcos and the strength of internet backbones in country. Some networks are unable to cope with a huge spike in demand, or may have not done implemented additional relief mechanisms like undoing rate limiting or throttling, expanding more capacity in advance, changing priorities or QoS...all of these are outside the control of an OTT operator, however, an OTT-engineering team still has to have the relevant relationships with networking providers in place, to secure the best customer experience possible...a lot of these things are behind-the-scenes and not made explicit, magic that happens in the background that no one really notices, until there's an issue (like buffering or slow start-up video times, etc.).

In terms of engineering skills-set differences, PayTV executives need to bear in mind that design patterns, implementation practices and even ways-of-working are much different in OTT than DTH STB software development, embedded world versus highly parallel, distributed network scale, client/server/stateless development).

Disaster Recovery / Redundancy / Rollback better in OTT than DTH Broadcast

When it comes to disaster recovery or business continuity requirements, the DTH broadcast world is much more complex and expensive to implement, as compared to an OTT service. Depending on the type of OTT service, it could get equally complicated, for example, if the OTT service includes live streaming channels, which usually has a dependency on the broadcast engineering team providing access to the channel signals. In an SVOD world however, disaster recovery is a matter of duplicating the stack and supporting services in multiple data centres or cloud regions or availability zones (an SVOD service not designed for cloud probably needs to be thrown away).
Where the OTT service is a replica of the PayTV service on DTH, but effectively offering just a "Dishless" service to customers, then there is very tight coupling with the existing PayTV broadcast engineering operations, so much so, that the OTT systems are "married" to the broadcast systems, so DR for live streaming on OTT becomes tightly coupled to the availability of the broadcast environment.
Depending on the way the PayTV organisation has been structured to manage the different product lines and customer segments, the OTT arm of the business could be entirely independent from the main PayTV business, which may mean that OTT engineering teams may have licence to think and implement things differently. This might be cool in the initial stages, but as OTT matures and the PayTV operator decides to consolidate costs and drive through engineering efficiencies, engineering teams need to work closely together in imagining and shaping the future of the group business, and not content with each other.
OTT engineering teams can benefit tremendously from existing broadcast engineering techniques, implementations and sunk-cost infrastructure. When it comes to sourcing content, transmission and contibution, even to the point of CDN delivery and leveraging economies of scale, OTT engineering teams should not become arrogant to think they can do things differently...
DR in OTT benefits from the cloud by way of having the capability of using "just-in-time" when needed, thus reducing upfront costs of infrastructure capex (having DR in place just to tick box a risk register, for a situation or event that might not even occur)...

The network is outside the control of the TV Operator

When it comes to streaming, a large part of the customer experience is somewhat outside the control of the operator, because the network that sits between the customer and the operator is usually a vast interconnected world of components, hardware & networking policies in between - it's the internet of course. Take for example the rise of mobile networks, as this is fast over taking the world of fixed line internet, where people's most like device of choice to consume video will be mobile 3G/4G/5G/wireless. Telco operators thus have to content with providing many services to its own customers (voice, text, web browsing, social media, etc.) in addition to video streaming services. How does a TV operator maintain some control of the quality of service or experience - through intensifying its partner relationships with telco providers if possible. Still, this is not always possible. Telco networks have the capability to throttle, shape, deprioritise and even lower video quality/bitrates at the edge, bypassing the OTT operator completely, leaving the OTT customer screaming and complaining about quality of service about the OTT operator, when the customer's own internet provider is the source of the problem. Without strict network agreements in place taking care of video QoE/QoS priorities, backed by commercial incentives for the Telco networks, OTT provider will be at the mercy of such players...the reverse is true as well. 

Some Telco operators are just not big enough, they don't have the capacity and throughput to carry video traffic, but the OTT customer does not see this as a Telco problem, it just manifests as a poor, crappy experience provided by the OTT service provider! 

Why do I write about this stuff anyway?

<#humbleBrag If you're looking for an DTV technology expert, happy to have a chat for CTO roles!>
I've been working in engineering Digital TV software systems all my professional life of twenty years. Starting in the early 2000s, where there were just a handful of technology platform providers available (compared to today's world in online streaming where almost every Joe provides a streaming platform off-the-shelf), I have seen almost every transition in this space. I started with writing set-top-box applications built on middleware SDKs (then OpenTV & NDS MediaHighway platforms), to writing middleware & operating systems code for these embedded devices, called Set Top Boxes (STBs). I branched out into broadcast headend software systems that provided the necessary signalling that is core to powering STB applications, like the classic EPG (Electronic Program Guide). I studied and mastered the various IEEE MPEG & ISO/DVB standards in detail, as the middleware software had to implement the protocols (transport stream specifications) from scratch so the STB could interpret the data. I once implemented from scratch, STB client-side for decoding the full specification for signalling terrestrial networks (UK-D Book), which included a driver hardware abstraction layer, graphics library, application engine management and signalling.

I later branched out further to join the IPTV wave, building real-time streaming engines that converted these satellite broadcast streams into IP packets (entailed transforming MPEG packets, signalling, wrapping over IP packetisation - converting an MPTS to SPTS, adding transcoding & encryption then playout of IP), doing live streaming encryption on-the-fly in real-time as well as media file-based offline-encryption packaged VOD (Video On Demand) assets for consumption over the network. Back then we built all these systems from the ground up on standard hardware, running on standard servers (it was truly pioneering stuff). The network streaming was primarily over UDP multicast. We had to build everything, from the control plane managing the clusters with seamless failover, full management console for administrating the IP headend, etc. We built our own transcoding engines, taking in the MPEG source and converting to other formats - all before the time that internet streaming started to become mainstream. This meant going down to the detail of coding the MPEG signalling standards down to the bits & bytes of the video containers, timestamping & metadata signalling. We built these distributed computing platforms from scratch, there weren't any widely available frameworks for abstracting the network & infrastructure as we have today. CDNs were just being built around the same time keeping pace with the ever changing landscape of web technologies, back then IPTV systems took the form of traditional "headends" that sat in telco/broadcast data centres, Live TV was mostly UDP multicast, with VOD files being stored close to the edges.  The consuming devices at the time were still STBs, and we were just starting on the web browsers as the client. I can truly say I've experienced just about the full journey of these systems, apart from writing any security-side code for conditional access & digital rights management (and also the distribution satellite systems was not my thing, I focused on building software).

I then transitioned into engineering management roles that was all about software delivery of these value chains, and became quite an expert at it. In the last few years, my focus has been online, which is really the future of TV. There was actually a period in between for Mobile-TV broadcaast, DVB-H which tried to precede OTT, some PayTV operators gambled this would take off, in hindsight, that time and investment could have been better spent learning & developing their OTT platform instead.  In my opinion the classic broadcast TV systems using unconnected STBs is dead, time to move on. This technology is now commodity, PayTV operators shouldn't invest too much in this world, they should rather sweat their assets and drive all operating costs down to be lean & efficient, stop manufacturing bespoke custom STBs & applications IMHO. The days of closed proprietary STB middleware software is also dead, time to openly embrace more modern stacks, that come very close to seamless integration with the online apps world.
</#humbleBrag>