Systems and Networks · July 30, 2026

Redundancy in Mission-Critical Systems

At an airport, the paging system is not allowed to fail. Not for a gate change, and not for an evacuation. So we do not build one system. We build two.

Backup amplifiers, backup processors, backup network connections. When a unit fails, the system moves to its backup automatically and the monitoring team gets a ping. We work the problem in the background while the airport keeps running, and the passenger waiting at the gate never knows anything failed. Our Oculus project has backups for almost everything. Backup video servers keep the system running even if one fails.

The Oculus, Houston Airport System
The Oculus, Houston Airport System

All that backup equipment means failures happen more often, not less. More amplifiers, more processors, more network paths, more things that can break. The goal was always preventing outages rather than preventing failures.

Which is why you cannot make everything redundant. Follow the logic out far enough and you get to a backup display fed by backup microphones on its own dedicated power source, and that has no end point. So the work is judgment. We decide which failures are unacceptable, which ones are likely, and which ones justify the cost of a backup, and the budget goes there. Redundancy costs money, space, and equipment, and it has to be aimed at what actually cannot fail.

For show systems we usually run Dante on primary and secondary networks, two separate sets of cables and switches carrying the same audio. Sometimes we go further and put an analog failover underneath.

Years ago I designed the DSP for a college basketball arena that way, with primary audio over the network and analog as the fallback. I came back about a month later to make some tweaks and found that half the system had dropped off the network when cables were cut somewhere in the building. It had failed over to analog and stayed up. Nobody at the arena knew there had ever been a failure.

The best redundancy is the kind nobody ever notices.

Read the original post and the discussion on LinkedIn