Building a new, all IP broadcast facility? – Read this.

The last three years of my career have been predominantly spent leading the design and implementation of a live news production facility for a leading financial data services provider in London.

It is one of the very first (and I think I’m right in saying) currently the largest, IP based production facility in the world and includes many other significant innovations and advances in broadcast facility design.

So, as this ambitious programme completes, where is the industry at with its transition from SDI to IP (LiveIP) routing?

Before looking forward, let’s take a moment to consider where we’ve come from.

When we started this build in early 2015, no architectural patterns existed for LiveIP and no one understood the network requirements because no one had built anything like this before.

Standards were in flux. There were no 2059, 2110 or NMOS standards. 100Gb/PSM4 and single mode 25Gb were roadmap. There was a lot of competition in the control and compressed signal spaces. There was also a lot of educated guesswork presented as fact. We’re all Google experts when it comes to design but that’s the thin end of the wedge when it comes to the leadership of high cadence, high innovation programmes.

There were however some 2022-6 capable endpoints available from industry pioneers and some interops had been conducted. Early trials and vendor roadmaps indicated that LiveIP was viable within programme timelines, just.

To bring the facility to fruition, we’d be responsible for leading the architectural design, driving product roadmaps/design and continual product and integration testing. It wasn’t easy. To get this working, we’ve broken just about everything, multiple times. This has been expected, planned for and mitigation strategies employed. We failed early and fast, a lot. We learnt a huge amount.

To provide a glimpse of this process, it took four attempts and a series of full scale network tests to get the PTP design right so as not to overload the network switch CPU’s, GM’s or endpoints. We finally figured that bit out in May 2016. We’ve flooded the network and broken endpoints in many interesting and alarming ways and have taken network and endpoint firmware updates throughout 2017.

Through it all, vendor engagement has been superb. I think it’s fair to say that the programme has played an important role in driving forward knowledge exchange between switch manufacturers and vendors and some great innovations are resulting.

In November 2017, the facility went live one week ahead of schedule.

To look at everything working now, you might wonder what all the fuss has been about. Every second, several terabits of data quietly pass through the network. We can shut down and upgrade half the network with zero impact to services, as hitless switching operates seamlessly across hundreds of endpoints and thousands of flows.  The architecture is open and endpoint integration is simple. Operationally, you wouldn’t know that the heart of the facility was IP.

So, looking forward, should you build on IP?

Yes. This is sufficiently mature and relevant technology.

What pointers should you keep in mind?

Well, it depends on the shape of your project.

In general terms, great progress has been made but there’s still much to do and you need to know what you’re doing. We’re not yet at the plug and play stage. There’s a fair chance that you’ll be targeting roadmap product features.

2110 standardisation is going great, but implementations are bedding in and some are more mature than others. Look around and you’ll see that many vendors are still finishing off 2022 implementations.

The control and monitoring side of things is also still settling. Pay a lot of attention to this area.

With the push towards leaf / spine, make sure you understand what the implications are if you go down this route.

Keep in mind that open standards do not equate to open architecture. Be careful that the architecture and control paradigm of your principle endpoint vendor does not preclude the simple integration of all other vendor products. Understand your control and telemetry handling and how it will work across different vendor products. Think carefully before outsourcing important design decisions, and responsibilities. An outsourced risk isn’t necessarily a mitigated risk, it might just be veiled for a while.

Learn all about PTP, multicast and network configuration / logical design.

LiveIP represents a major industry transition and we are in the midst of it. Things are going well but you should plan with this in mind and employ strategies to manage and accelerate the process.

Get educated, lead the technical discussions, own your architecture and network design and when you’re done, share your experience.

On a final note, even in an IP facility, there’s a place for SDI.

Part 2 of this article can be found here: Building a New All IP Broadcast Facility Read This – Part 2