AT&T outage reveals weaknesses in emergency communications
Problems with telephone and Internet service from Quincy to Portola disrupted normal operations of everything from gas stations to
emergency service communications Sunday, Aug. 16.
“On a small scale we were shown how unorganized and how unprepared we are at certain times,” Supervisor Robert Meacher commented.
Plumas County Sheriff’s Department Deputy Mike Grant estimated the phone problems began around 8:45 a.m. and were mostly resolved by 2:30 p.m.
There were many problems caused by the loss of service as some gas pumps refused to take credit cards, at least one ATM machine wouldn’t give out money, some supermarkets limited the amount of cash back, “contactless” credit cards that activate by being near a sensor didn’t work in some places and food stamp cards were rejected in some instances.
All of these problems were attributed to services that require a machine to get verification from an outside source.
If someone walks into a supermarket, pays with a debit card and asks for cash back, some services require that information be sent over the phone lines to make sure that the person actually has enough money in his account to make that withdrawal.
Different companies, chains and businesses have a myriad of policies and protocols for how to address a situation when that information can’t be verified.
In this incident one store limited the amount of cash back to $40, while other establishments simply didn’t want to take the risk of giving out money they weren’t sure the customer had.
Businesses that used satellite phones were exempt from these problems, which seemed to be centered primarily on landlines, cell phones and Internet lines run by AT&T.
One AT&T representative unofficially referred to the incident as a “catastrophic repeater outage.” Repeaters are essentially devices that receive some form of signal and repeat it to another receiver.
They are used for many reasons, for example, to get radio messages past obstructions like hills by putting a repeater on top of the hill so a signal goes up to the hill and then down to the destination on the other side.
Gina Pernetti of AT&T Corporate Communications for the West Region gave the official response in an e-mail.
She simply commented that the problem was caused by “a power-related issue.” Pernetti explained cell phones were affected because “all cell-phone calls, at some point, are routed through landlines.”
When asked what areas were affected, she responded, “Roughly, southern Plumas County—Portola, Loyalton, Blairsden, Sierraville, Quincy numbers.”
Although all of the effects of the service outage listed above were certainly inconveniencing and, given the right circumstances, could have led to major problems or dangers for families living day to day, the most frightening part of the incident was its drastic impact on the communications of Plumas first responders and other emergency service providers.
Deputy Grant, the resident PCSO communications expert, explained the situation.
He began by saying the county’s new “reverse 911” call system can’t function without landline phone service.
Grant added that a third of the PCSO’s radio capability was compromised by the outage.
The deputy pointed out dispatch phones also were out: dispatchers could not receive 911 calls for about 45 minutes.
During that time dispatchers were using a non-AT&T cell phone to call hospitals and fire stations to get people with non-AT&T cell phones to stand by there, creating an ad hoc communication system.
Dispatchers also called Sierra County, which takes over dispatching duties for Plumas when issues like these arise.
Grant said he couldn’t tell if Sierra County lost phone service simultaneously or shortly after Plumas, but in any event the dispatch center in that county quickly notified Nevada County that it would have to cover both Plumas and Sierra counties.
He also said that residents in the Chester and Greenville area had working phones, but got a busy signal when calling 911 for part of the outage because there was essentially nowhere for their call to go with dispatch phones out of service in Quincy.
This actually put them at a disadvantage compared to people in Quincy and Portola who had non-A&T cell phones as they could call 911 and their calls would be routed to another dispatch center.
People in Chester and Greenville possibly had a larger problem in terms of reaching emergency services and were probably less likely to know that there was even an issue with communications as their phones were working fine.
Meanwhile, a sergeant and dispatchers activated a satellite phone that allowed them to speak to other dispatch centers and receive calls from them as well.
At 11:30 a.m., a second backup communication system was assembling in Portola, as Plumas Amateur Radio Club and Amateur Radio Emergency Service members Dennis Dickinson and his father, Roy Dickinson, were setting up ham radio stations in the Eastern Plumas District Hospital parking lot and making connections with amateur radio operators in Quincy and Reno, Nev.
The duo notified local ARES and Radio Amateur Coordinated Emergency Services members in both areas of the service outage using radio repeaters on Mount Hough and Mount Rose.
They said that if the outage had gone on, other operators would have moved their equipment to strategic locations to assist emergency services in communicating.
Summing up the incident, Grant said he would like to get secondary communications fully prepared so the 45-minute set-up time would be shorter in the future.
He added that this was only the second time a communication outage of this magnitude had hit Plumas County in recent memory.
Grant said adding redundancy to the emergency services communication system occurred as a result of the last incident, which he thought would be enough to avoid it happening again.
He said it was hard to predict the next way something could go wrong until there was an actual incident like this one.
He also said many of the problems, such as Chester residents getting a busy signal, were hard to envision fixes for when the corporation that owns the infrastructure that malfunctioned was unwilling to share more information about what went wrong.