Understanding what our backups can actually survive
Pat Stahl | October 30, 2026
If the power goes out, what goes dark, and when? It is a straightforward question, but the answer can get complicated quickly. The control room might stay lit while a network device loses power. The transmitter might keep running after its program feed disappears. The backup application might be available, but nobody can sign in to use it.
There is a well-worn story that NBC Pages would share during tours of 30 Rock when I first worked there. According to the story, the backup power systems worked so well during the Northeast blackout on August 14, 2003, that operators inside only learned about it when it became a national news story. Later, I heard another part of the story: the fuel pump connecting the large storage tank to the generator’s day tank was not connected to protected power. That is the type of detail that interests me. A generator can be ready to run while something it depends on is not.
Contemporary reporting in TVWeek describes the fuel, power and operational challenges stations faced during that blackout. [1]
After more than 20 years in media technology, I have become careful about the word redundant. It can describe a tested system with independent paths and a clear recovery process. It can also mean that someone bought a second encoder to keep on a shelf. Either can be useful, but I want to know which failure the backup can handle and what it takes to put it to work.
Know what the backup protects
As broadcasters replace some satellite feeds and dedicated circuits with managed IP services, the dependencies change. The same is true when an application moves outside the building. Lower costs and new capabilities can make those changes worthwhile. We still need to understand what happens when the replacement becomes unavailable.
Most stations cannot justify a second version of their entire operation. Even an elaborate backup can share a point of failure with the primary system. What we should be able to explain is the service we can preserve when an essential connection, power source or supplier fails.
Follow that explanation all the way to the audience. Can we still gather and produce local news? Will captions remain available? Can the transmitter receive a useful program? A channel can remain on air while the newsroom loses the ability to produce its next newscast.
It also helps to ask what happens after the first failure. A successful switchover can leave the station running with less protection for hours or days. Someone needs to recognize that condition and decide how to operate until the damaged system is restored.
A generator needs more than a successful start
During the January 2025 Eaton Fire, engineers on Mount Wilson above Los Angeles had to sustain broadcasting through extended utility interruptions. iHeartMedia’s Doug Irwin told Radio World that its stations ran on generators for about two weeks, apart from a 36-hour restoration of utility power. A supplier delivered fuel daily. NBC engineer Dennis Doty reported consumption of about 2,000 gallons a day at NBC’s facility. [2] Starting the generators was the beginning of an operation that required continuing attention to fuel and site access.
Preparation also extended beyond the equipment room. CAL FIRE found that a 113-acre prescribed burn completed in January 2020 helped firefighters contain a spot fire and protect KPFK’s main transmitter five years later. [3] iHeart had auxiliary transmission sites, too. One was less than a mile from Mount Wilson’s main antenna group and remained exposed to the same fire threat; another was about 20 miles west in the Hollywood Hills. Neither was needed during the incident. [2]
KUSC’s nearby Mount Harvard site illustrates another limit. Its tower, antenna and transmitters survived, but heat damaged the cable between transmitter and antenna. The station reported reduced-power operation and an emergency Hollywood Hills installation with less coverage. Replacing more than 200 feet of heavy cable required riggers, freight and usable mountain roads. [4] Keeping equipment alive and restoring normal reception were different jobs.
Figure 1 UPS coverage must bridge the transfer interval; sustained generator operation then depends on fuel, cooling and maintenance. These are operating stages, not guaranteed runtimes.
At our own facilities, start by checking what has unprotected power, generator power or an uninterruptible power supply (UPS) backed by a generator. Will the UPS carry the required load through transfer? Include cooling, fuel pumps and the ordinary tools people need to work. Test under load through an agreed procedure that protects the live service. Measure fuel burn, establish when the next delivery is needed and name the person who will arrange it. Check access and payment arrangements before the usual supplier becomes unavailable.
Then follow the generator’s running hours. Know when maintenance is due, who can perform it and whether the unit must stop. If the broadcast must continue during service, check that the remaining equipment can carry the required load. An N+1 design provides one additional unit beyond the number needed for the load; shared fuel, controls and cooling still need examination. Have the required supplies available and act before the first limit is reached.
A second connection changes the risk
On August 30, 2022, construction damaged Breezeline’s primary fiber near Fredericksburg, Virginia. Traffic moved to a redundant path without interrupting customer service. Nearly three hours later, a truck accident in Sudlersville, Maryland, damaged the backup circuit. Customers in Delaware, Maryland and Virginia then lost service. [5]
The first failure was handled successfully. The network then operated with less protection until a separate accident removed the remaining path. A shared conduit, river crossing or building riser creates another risk: one event can remove both connections. For a station, knowing which condition it faces helps determine what to do while repairs are underway.
Gray CTO Bob Kroeger described the same concern in TVNewsCheck’s October 29, 2026, reporting. Two providers entering different parts of a station can still share infrastructure farther down the road. He also described emergency use of bonded cellular and Starlink after studio-to-transmitter link (STL) failures. [6] Those alternatives deserve a place in the plan once their capacity and operating limits have been tested.
Figure 2 Two provider routes converge at one entry and riser. The map and WASS example are illustrative; they do not describe WASS’s actual infrastructure.
In January 2026, fiber damage near Cody, Wyoming, interrupted streaming distribution for Montana Television Network stations KPAX, KXLF, KRTV and KTVQ. Affected streaming services carried national CBS programming instead of local news. Local programming remained available over the air, on cable and through DirecTV and Dish. [7]
The substitute feed kept television available to those viewers, but the station’s local service was missing. That is a useful distinction when deciding what to monitor. A successful connection to the app tells us less than confirming that viewers can watch the intended local newscast.
Check each circuit, then consider losing one provider or all of them. Confirm physical routes, shared power and whether the remaining path can carry the required traffic. Follow the consequences through field contribution, cloud tools and the STL. An independent microwave STL still needs functioning equipment, power and a usable program source. Recheck diversity after carrier routing or delivery arrangements change.
A provider outage can reach the newsroom
Outside services deserve the same examination. Microsoft 365, AI tools, bonded cellular and news wires are all part of putting content on air. The service provider’s own backup arrangements matter, but so do the connections and credentials staff use to reach it.
During the AWS outage on December 7, 2021, Gray News reported that the content management system used to update its stations’ websites was affected. NPR also reported problems accessing its news apps. These were disruptions to digital publishing and audience access, rather than documented over-the-air failures. [8, 9] The dependency reached a particular part of the service; understanding that scope is the starting point for a useful fallback.
Figure 3 Both news services depend on company SSO for new access. It does not describe an outage at either service.
Suppose two news services use the company’s single sign-on (SSO), provided through Microsoft Entra ID, as in Figure 3. If that sign-in service fails, staff may lose new access to both. Existing sessions might continue, so test both conditions. Check an approved independent sign-in method, where supported, and what material is already available locally. A mobile phone can help with feeds, scripts and video, provided it can reach the required service and does not depend on the failed sign-in.
On July 19, 2024, a defective CrowdStrike security-content update crashed affected Windows systems. At my station, we scrambled to stop further updates and restore affected computers. Timing and luck prevented a major incident. Sky News reported that it could not broadcast live television for several hours. [10, 11] Geographically separate systems could still share a dependency that causes failure.
CrowdStrike’s post-incident review explains an important detail: the faulty update carried rapidly delivered threat-detection content, separate from normal sensor software releases. The controls customers used to hold back sensor-version releases did not govern rapid content in the same way. [12] LiveU CTO Dan Pisarski later made the related point in TV Tech: physical separation offers limited protection when the alternate computers share the same software vulnerability. [13] A cloud backup needs to avoid that dependency, too.
Scripps offers a practical example of what helped. Executives told TV Tech that Macs served as a backup because they were unaffected by the Windows failure. An established incident-management process brought engineering, IT and business operations together. Some local stations initially could not produce local news even though their channels remained on air. Scripps said nearly 90 percent could air local news by 8 a.m. Eastern, and all could before noon. [14] The alternate equipment and a coordinated response helped restore a capability that a signal alone could not replace.
Recovery had dependencies of its own. CrowdStrike reverted the defective content after 78 minutes, but that did not automatically repair every crashed computer. Microsoft documented manual recovery procedures that could require Safe Mode, recovery tools and BitLocker encryption-recovery keys. [12, 15] Could our team obtain those instructions and keys if its normal computers were unavailable? Could it support remote staff while restoring the systems needed for the next newscast?
CrowdStrike subsequently reported additional testing, staged deployment and greater customer control over rapid-content updates. [16] The practical task is to examine the controls now available for each service, including security tools.
Make the assumptions visible
Get engineering, editorial and the business team in a room. Start with a diagram of how a local newscast reaches the audience, including the tools people use to coordinate recovery. Then work through a failure mode analysis: review each essential service and component, and establish what happens to the operation when it becomes unavailable.
Consider losing power, one internet connection, all internet connections and an essential cloud service. For each, ask what stops and how the team will know. What can we still produce with the systems that remain available, and how long can we sustain that service? Name the person responsible for recovery, the outside contacts they need and the access required to begin.
Test those assumptions under controlled conditions that protect the live service. Include starting the fallback, working with its limitations and returning to normal operations. A plan is more useful when the people who will carry it out have practiced it. Revisit the answers as workflows, equipment and providers change.
There will be risks the business chooses to accept. Put their consequences in terms colleagues can evaluate: a shorter newscast, reduced coverage or a longer recovery time. That gives leaders a basis for deciding where another investment would help and where an agreed limitation is reasonable. When someone tells me a system is redundant, I want to understand what we can still deliver when it is needed.
Sources
[1] TVWeek. Blackout Reveals Weak Spots in News Ops’ Preparedness. August 25, 2003. Read source
[2] Radio World. How Mount Wilson Survived Another Scare. June 10, 2025. Read source
[3] CAL FIRE. Eaton Fire Fuel Treatment Effectiveness Report. Fire Year 2025, pp. 1–2. Read source
[4] Classical California / KUSC. How the Eaton Fire Is Affecting KUSC Reception. February 6, 2025; interview with Ron Thompson. Read source
[5] Light Reading. Breezeline outage caused by two separate cuts to fiber facilities. August 31, 2022. Read source
[6] TVNewsCheck / Glen Dickson. Broadcasters Prioritize Streamlining Linear, Digital Tech & Workflows. October 29, 2026. Read source
[7] KPAX / MTN News. Crews restore KPAX to streaming after fiber line damaged. January 7, 2026 update. Read source
[8] Gray News via WSAZ. Major Amazon Web Services outage crashes several sites, apps. December 7, 2021. Read source
[9] NPR via GPB. Amazon announces progress after an outage disrupted sites. December 7–8, 2021. Read source
[10] CrowdStrike. To Our Customers and Partners. July 19, 2024. Read source
[11] Sky News. CrowdStrike IT outage devastating for businesses. July 19, 2024. Read source
[12] CrowdStrike. Falcon Content Update Preliminary Post Incident Report. July 24, 2024; updated July 25. Read source
[13] TV Tech / Jenny Priestley and Tom Butts. One Year Later Has the Media Industry Learned from the CrowdStrike Outage. July 21, 2025; includes Dan Pisarski’s assessment. Read source
[14] TV Tech / Phil Kurz. Scripps’ Incident Management Planning Minimizes CrowdStrike Disruptions. July 26, 2024; interviews with Scripps executives. Read source
[15] Microsoft Support. KB5042421 CrowdStrike issue impacting Windows endpoints. Recovery guidance for the July 2024 incident. Read source
[16] Adam Meyers / CrowdStrike. Written testimony before the House Homeland Security Subcommittee on Cybersecurity and Infrastructure Protection. September 24, 2024. Read source
Diagrams are illustrative. Incident details and company assessments are attributed to the linked sources. The 30 Rock fuel-pump story and station experience during CrowdStrike are the author’s recollections; the TVWeek article provides contemporary blackout context rather than verification of the pump account.
