[Note that this article is a transcript of the video embedded above.]

November 1965 saw one of the most widespread power outages in North American history. On the freezing cold evening of the 9th, the grid was operating at maximum capacity as people tried to stay warm when a misconfigured relay tripped a breaker on a key transmission line. The loss of that single line cascaded into a loss of service for over 30 million people in the northeast US plus parts of Ontario in Canada. Restoring electricity to that many people is no simple task. In this case, the startup began with a little 12 megawatt gas generator in Southampton, New York. That’s about the capacity of four wind turbines, but it was enough to get power plants in Long Island back online which were able to power up all of New York City, eventually returning service to all those 30 million people

The grid is a little bit of a house of cards. It’s not necessarily flimsy, but if the whole thing gets knocked down, you have to rebuild it one card at a time and from the ground up. Restoring power after a major blackout is one of the most high stakes operations you can imagine. The consequences of messing it up are enormous, but there’s no way to practice a real-life scenario. It seems as simple as flipping a switch, but restoring power is more complicated than you might think. And I built a model power grid here in the studio to show you how this works. This is my last video in a deep dive series on widespread outages to the power grid, so go back and check out those other videos if you want to learn more. I’m Grady and this is Practical Engineering. In today’s episode we’re talking about black starts of the grid.

An ideal grid keeps running indefinitely. Maybe it sustains localized damage from lightning strikes, vehicle accidents, hurricanes, floods, and wayward squirrels, but the protective devices trigger circuit breakers to isolate those faults and keep them from disrupting the rest of the system. But, we know that no grid is perfect, and occasionally the damage lines up just right or the protective devices behave in unexpected ways that cascade into a widespread outage. I sometimes use the word blackout kind of freely to refer to any amount of electrical service disruption, but it’s really meant to describe an event like this: a widespread outage across most or all of an interconnected area. Lots of engineering, dedicated service from linesworkers, plenty of lessons learned from past mishaps, and a little bit of good fortune have all meant that we don’t see too many true blackouts these days, but they still happen, and they’re still a grid operator’s worst nightmare. We explored the extreme consequences that come from a large-scale blackout in a previous video. With those consequences in mind, the task of bringing a power grid back online from nothing (called a black start) is frightfully consequential with significant repercussions if things go wrong.

The main reason why black starts are so complicated is that it takes power to make power. Most large-scale generating plants - from coal-powered, to gas-powered, to nuclear - need a fair amount of electricity just to operate. That sounds counterintuitive, and of course configurations and equipment vary from plant to plant, but power generating stations are enormous industrial facilities. They have blowers and scrubbers, precipitators and reactors, compressors, computers, lights, coffee makers, control panels and pumps (so many pumps): lubrication pumps, fuel pumps, feedwater pumps, cooling water pumps, and much much more. Most of this equipment is both necessary for the plant to run and requires electricity. Even the generators themselves need electricity to operate.

I don’t own a grid scale, three-phase generator (yet), but I do have an alternator for a pickup truck, and they are remarkably similar devices. You probably already know that moving a conductor through a magnetic field generates a current. This physical phenomenon, called induction, is the basis for almost all electricity generation on the grid. Some source of motion we call the prime mover, often a steam-powered turbine, spins a shaft called a rotor inside a set of coils. But you won’t see a magnet on the rotor of a grid-scale generator, just like (if you look closely inside the case) you won’t see a magnet inside my alternator. You just see another winding of copper wire. Turns out that this physical phenomenon works both ways. If you put a current through a coil of wire, you get a magnetic field. If that coil is on a rotor, you can spin it like so.

This is my model power plant. I got this idea from a video by Bellingham Technical College, but their model was a little more sophisticated than mine. Let me give you a tour. On the right we have the prime mover. Don’t worry about the fact that it’s an electric motor. My model power plant consumes more energy than it creates, but I didn’t want to build a mini steam turbine just for this demonstration. The thing that’s important is that the prime mover drives a 3-phase generator, in my case through this belt. And the generator you already saw is a car alternator that I “modified” to create an Alternating Current instead of a Direct Current like what’s used in a vehicle. The alternator is connected to some resistors that simulate loads on the grid. And I have an oscilloscope hooked up to one of the phases so we can see the AC waveform. Yeah, all this is so we can just see that sine wave on the oscilloscope. It could have been a couple of tiny 3-phase motors; It could even have just been a signal generator. But, you guys love these models so I thought you deserved something slightly grander in scale. There’s a few other things here too, including a second model power plant, but we’ll get to those in a minute.

The alternator I used in my model has two brushes of graphite that ride along the rotor so that we can supply current to the coil inside to create an electromagnet. This is called excitation, and it has a major benefit over using permanent magnets in a generator: it’s adjustable. Let’s power up the prime mover to see how it works. If there’s no excitation current, there’s no magnetic field, which means there’s no power. We’re just spinning two inert coils of wire right next to each other. But watch what happens when I apply some current to the brushes. Now the rotor is excited, and I have to say, I’m pretty excited too, because I can see that we’re generating power. As I increase the excitation current, we can see that the voltage across the resistor is higher, so we’re generating more power. Of course, this additional power doesn’t come for free. It also puts more and more mechanical load on the prime mover. You can see when I spin the alternator with no excitation current, it turns freely. But when I increase the current, it becomes more difficult to spin. Modern power plants adjust the excitation current in a generator to regulate the voltage of electricity leaving the facility, something that would be much harder to do in a device that used permanent magnets that don’t need electricity to create a magnetic field.

The power for the excitation system can come from the generator, but, like the other equipment I mentioned, it can’t start working until the plant is running. In fact, power plants often use around 5 to 10 percent of all the electricity they generate. That’s why a black start of a large power plant is often called bootstrapping, because the facility has to pick itself up by the bootstraps. It needs a significant amount of power both to start and maintain its own creation of power, and that poses an obvious challenge. You might be familiar with the standby generators used at hospitals, cell phone towers, city water pumps, and many other critical facilities where a power outage could have severe consequences. Lots of people even have small ones for their homes. These generators use diesel or natural gas for fuel and large banks of batteries to get started. Imagine the standby generator capacity that would be needed at a major power plant. Five percent of the nearest plant to my house, even at a quarter of its nameplate capacity, is 18 megawatts. That’s more than 100 of these.

It’s just not feasible to maintain that amount of standby generation capacity at every power plant. Instead, we designate black start sources that can either spin up without support using batteries and standby devices or that can remain energized without a connection to the rest of the grid. Obviously, these blackstart power plants are more expensive to build and maintain, so we only have so many of them spread across each grid. Their combined capacity can only supply a small fraction of electricity demands, but we don’t need them for that during a blackout. We just need them to create enough power so that larger base load plants can spin up. Hydropower plants are often used as blackstart sources because they only need a little bit of electricity to open the gates and excite the generators to produce electricity. Some wind turbines and solar plants could be used as blackstart sources, but most aren’t set up for it because they don’t produce power 24-7.

But, producing enough power to get the bigger plants started is only the first of many hurdles to restoring service during a blackout. The next step is to get the power to the plants. Luckily, we have some major extension cords stretched across the landscape. We normally call them transmission lines, but during a blackout, they are cranking paths. That’s because you can’t just energize transmission lines with blackstart sources. First those lines have to be isolated so that you don’t inadvertently try to power up cities along the way. All the substations along a predetermined cranking path disconnect their transformers to isolate the transmission lines and create a direct route. Once the blackstart source starts up and energizes the cranking path, a baseload power plant can draw electricity directly from the line, allowing it to spin up.

One trick to speed up recovery is to blackstart individual islands within the larger grid. That provides more flexibility and robustness in the process. But it creates a new challenge: synchronization. Let’s go back to the model to see how this works. I have both generating stations running now, each powering their own separate grid. This switch will connect the two together. But you can’t just flip it willy nilly. Take a look at my oscilloscope and it’s easy to see that these two grids aren’t synchronized. They’re running at slightly different frequencies. If I just flip the switch when the voltage isn’t equal between the two grids, there’s a surge in current as the two generators mechanically synchronize. We’re only playing with a few volts here, so it’s a little hard to see on camera. If I flip the switch when the two generators are out of sync, they jerk as the magnetic fields equalize their current. If the difference is big enough, the two generators actually fight against each other, essentially trying to drive each other like motors. It’s kind of fun with this little model, but something like this in a real power plant would cause tremendous damage to equipment. So during a black start, each island, and in fact each individual power plant that comes online, has to be perfectly synchronized (and this is true outside of black start conditions as well).

I can adjust the speed of my motors to get them spinning at nearly the exact same speed, then flip the switch when the waveforms match up just right. That prevents the surges of power between the two systems at the moment they’re connected. You can see that the traces on the oscilloscope are identical now, showing that our two island grids are interconnected. One way to check this is to simply connect a light between the same phase on the two grids. If the light comes on, you know there’s a difference in voltage between them and they aren’t synchronized. If the light goes off and stays off, there’s no voltage difference, meaning you’re good to throw the breaker. Older plants were equipped with a synchroscope that would show both whether the plant was spinning at the same speed as the grid (or faster or slower) and whether the phase angle was a match. I bought an old one for this video, but it needs much higher voltages than I’m willing to play with in the studio, so let’s just animate over the top of it. Operators would manually bring their generators up to speed, making slight adjustments to match the frequency of the rest of the grid. But matching the speed isn’t enough, you also have to match the phase, so this was a careful dance. As soon as the synchroscope needle both stopped moving and was pointing directly up, the operator could close the breaker.

During a black start, utilities can start restoring power to their customers, slowly matching generation capacity with demand as more and more power plants come online. Generally, the most critical loads will be prioritized during the recovery like natural gas infrastructure, communications, hospitals, and military installations. But even connecting customers adds complexity to restoration.

Some of our most power-hungry appliances only get more hungry the longer they’ve been offline. For example an outage during the summer means all the buildings are heating up with no access to air conditioning. When the power does come back on, it’s not just a few air conditioners ready to run. It’s all of them at once. Add that to refrigerators, furnaces, freezers, and hot water heaters, and you can imagine the enormous initial demand on the grid after an extended outage. And don’t forget that many of these appliances use inductive motors that have huge inrush currents. For example, here’s an ammeter on the motor of my table saw while I start it up. It draws a whopping 28 amps as it gets up to speed before settling down to 4 amps at no load. Imagine the demand from thousands of motors like this starting all at the exact same instant. The technical term for this is cold load pickup, and it can be as high as eight to ten times normal electrical demands before the diversity of loads starts to average out again, usually after about 30 minutes. So, grid operators have to be very deliberate about how many customers they restore service to at a time. If you ever see your neighbor a few blocks away getting power before you, keep in mind this delicate balancing act that operators have to perform in order to get the grid through the cold load pickup for each new group of customers that go online.

The ability to black start a power grid quickly after a total collapse is so important because electricity is vital to our health and safety. After the 2003 blackout in the US, new reliability standards were issued, including one that requires grid operators to have detailed system restoration plans. That includes maintaining blackstart sources, even though it’s often incredibly expensive. Some standby equipment mostly just does just that: stands by. But it still has to be carefully maintained and regularly tested in the rare case that it gets called into service. Also, the grid is incredibly vulnerable during a blackstart, and if something goes wrong, breakers can trip and you might have to start all over again. Utilities have strict security measures to try and ensure that no one could intentionally disable or frustrate the black start process. Finally, they do detailed analysis to make sure they can bring their grid up from scratch, including testing and even running drills to practice the procedures; All this cost and effort and careful engineering just to ensure that we can get the grid back up and running to power homes and businesses after a major blackout.