* 2003 blackout plunged 50 million into darkness
* Hundreds of millions invested in avoiding a repeat
* Cybersecurity the new focus for largest U.S. grid
By Scott DiSavino
NEW YORK, Aug 12 It was just another hot August
afternoon for New York electricity grid chief Rick Gonzales, as
he sat in the control room helping to make sure the state's 8
million homes had enough power to keep the lights on and the air
While the temperature pushed toward 90 degrees Fahrenheit,
the grid serving New York City and the rest of the state of 20
million people wasn't under any strain, Gonzales recalled, as
the clock ticked past 4 p.m. on the East Coast.
But 500 miles away, a few benign tree branches were about to
render Gonzales' territory - and that of 30 million unsuspecting
others - powerless.
"In less than five seconds we went from normal operations to
most of the state blacked out," Gonzales said recently.
"Alarms were ringing and our computers started flashing red.
We didn't know if we were getting bad information or if this was
a true blackout."
They soon found out. The Northeast blackout of Aug. 14,
2003, was the biggest in North American history, and laid bare
the vulnerabilities of an aging electric grid in the most
populous region of the United States.
More than 50 million people from Ohio to Ontario would lose
power, at an estimated cost of almost $10 billion. Coming just
two years after Sept. 11, the blackout reignited fears of what a
targeted attack could achieve if, as an official report
eventually concluded, a few untrimmed tree branches in Ohio
could do that much damage.
Ten years on, electricity experts say a similar event is far
less likely to happen in the U.S. Northeast. Grids and power
companies have overhauled their operations, and regulators now
have the authority to punish any company that doesn't adhere to
rules covering everything from maintenance schedules to how
close tree branches can get to power lines.
Of course as Hurricane Sandy demonstrated last year, "We'll
never be able to eliminate (blackouts) entirely," said Jay Apt,
a former NASA astronaut and physicist and director of the
Carnegie Mellon Electricity Industry Center. "The frequency of
large and small blackouts has not really changed since we became
good at keeping statistics on them 30 years ago."
And while grid operators learn lessons from every major
blackout, they face diverse threats, including attacks on
Eight months after the blackout, a joint U.S.-Canada task
force issued a report on its causes along with recommendations
for the prevention of future cascading power outages.
The power industry and energy regulators have since spent
hundreds of millions of dollars on new monitoring equipment and
improved computer systems, in addition to adopting new standards
and improving worker training and equipment maintenance.
To make sure power companies follow the rules, the 2005
Energy Policy Act gave the Federal Energy Regulatory Commission
(FERC) the ability to impose fines of up to $1 million per
violation per day for failure to comply with the standards.
In 2009, Florida Power and Light, the biggest electricity
company in the state, paid a $25 million settlement over FERC
allegations that one of its workers had caused a blackout the
previous year by not following proper procedures.
"That change from voluntary to mandatory reliability
standards is something we strongly supported since we were
impacted by the lack of clear rules," said Gonzales, who is now
the chief operating officer of the New York Independent System
Operator (NYISO), which runs the state's power grid.
The task force determined that Ohio power company
FirstEnergy Corp triggered the 2003 blackout because it
didn't understand how vulnerable its system was.
A failure in FirstEnergy's computer system was compounded by
human error after a technician called in to fix it forgot to
turn back on a key application for monitoring the grid.
At the same time, the Midwest grid operator was experiencing
its own computer problems. The two together were essentially
flying blind, and did not detect the shutdown of several power
lines that had been hit by wayward trees.
With no reaction from FirstEnergy or the grid operator,
other lines quickly became overloaded. What followed was a power
surge that grew large enough to jump state lines, first knocking
out power in Michigan, rolling through Ontario and ultimately
knocking New York offline. The whole event took little over an
hour from when the first branch hit the power line in Ohio.
A spokesman for FirstEnergy, one of four investor-owned
power companies in Ohio, said it has made efforts to strengthen
the Ohio power grid. "FirstEnergy has added new facilities and
equipment, enhanced operator training, and implemented new
standards that are mandatory and enforceable in order to help
enhance the reliability of the transmission grid," said Mark
Since 2003 Ohio has had numerous widespread blackouts caused
by snow and thunder storms, but the number of outages caused by
faulty equipment or human error has been falling, according to a
company that tracks blackouts and their causes.
Eaton, a power services company, said in its annual report
that Ohio had only 28 blackouts last year due to equipment
failures or human error. While the state still ranks seventh in
the United States for power outages, it was second only to
California in 2009, when it had 42.
SHARPER SKILLS THROUGH SIMULATIONS
Gonzales said that in 2003 there was no early warning that
could have made a difference.
"We had no idea something was wrong with the power system in
Ohio. It was a normal, warm summer day."
To help speed the recovery, the grid operator relied on just
"one guy in a truck" navigating his way through congested
traffic to reconnect the state to the wider grid at a substation
just north of the New Jersey border, where they still had power.
New York utilities restored power to about two-thirds of
homes and businesses by 8 a.m. Eastern time Friday, about 16
hours after the blackout started. Service was completely back
after about 30 hours.
Gonzales says the same lack of information from neighboring
grids would be improbable now. New technology allows the grid
operator to take far more sophisticated measurements of its own
system's power flows as well as neighboring systems', and faster
Apt, at Carnegie Mellon, said training of grid controllers
now includes frequent test runs to sharpen their skills in the
event of a major disruption to the grid.
"We're doing a lot more simulation now," he said. "It's like
what they say about airline pilots - it's years of boredom
interspersed with moments of stark terror."
While the grid is now seen as less susceptible to human
error, malicious threats have risen.
Cyber intrusions and attempted attacks on grid computer
systems have increased dramatically over the last decade,
according to the U.S. Department of Homeland Security, raising
fears of both widespread power cuts and threats to nuclear
Gerry Cauley, president of the North American Electric
Reliability Corp (NERC), the agency enforcing grid standards,
said the power industry has made significant progress to address
cybersecurity threats since he returned as president in 2010.
"I walked in the door and I felt a black hole of worry, risk
and problems in terms of security. We were not in good shape at
NERC or in the industry in terms of understanding what we should
be doing and where we needed to focus," Cauley told an industry
meeting earlier this year.
"We have been able to turn that around. It has become a way
of life for us," Cauley said.
The power industry has put mandatory cybersecurity standards
in place, including the requirement to share information quickly
between grid operators, companies and regulators in the event of
an attack. The U.S. government is also looking at implementing
its own rules.
The other big challenge is the changing way the United
States generates power.
Since 2003 enough coal plants to power 15 million homes have
shut down or converted to natural gas as their primary fuel
source. That has left some grid operators questioning if they
will ever face natural gas supply squeezes on the coldest days
when they compete with energy companies providing gas to heat
Renewables, like wind and solar power, also present some
difficulty because they're less predictable. Grids and power
companies are now experimenting with ways of cutting demand at
times of peak consumption.
PJM Interconnection, which oversees a grid serving 61
million people in 13 parts of states from New Jersey to
Illinois, uses so-called demand response programs, which
compensate large companies and consumers for cutting back the
power they use when the grid is under strain.
That can reduce electricity use for a short period of time
by as much as 10,000 megawatts - the equivalent of enough
electricity to power New York City on an average day.
PJM is also experimenting with using giant batteries, and
programs that may enable owners of electric vehicles to sell
power back even as they are charging their cars.
Chantal-Aimee Hendrzak, PJM executive director of system
operations support, said the grid is looking at integrating a
host of unconventional electric technologies.
"It does not have to be something new," she said.
"Batteries, water heaters, flywheels, electric vehicles can all
respond in less than a second to add (or remove) power. The more
devices that can be used to balance the grid, the fewer big
power plants have to be held in reserve."
(Additional reporting by David Sheppard and Joe Silha in New
York and Eileen O'Grady in Houston; editing by Jonathan Leff and