[Note that this article is a transcript of the video embedded above.]

Early in the morning of December 14, 2005, pumps were nearly finished filling the upper reservoir at the Taum Sauk power station, marking the end of the daily cycle. Water rose to the top of the rockfill embankment, reaching the concrete parapet wall that ran along the top of the dam. But the water didn’t stop. One of the two pumps shut off, but the other kept running, and soon, the water was lapping over the wall. Within minutes, those splashes turned into a steady stream cascading over the parapet, pouring against the embankment on the other side. The rockfill eroded slowly at first, but the hole grew deeper and wider. The pump finally shut off, but it was too late—the footing of the parapet wall had already been undermined. The wall tipped over, and a massive surge of water was unleashed down the mountainside headed directly toward a state park.

This award-winning pumped storage facility, considered a model of modern engineering, immediately became the center of intense scrutiny. And what the investigations found would change a lot about the field of dam safety. I’m Grady, and this is Practical Engineering.

When it was built in the 1960s, the Taum Sauk pumped storage plant was unlike really any other power plant in the world, at least in terms of size. South of St. Louis in the Ozark Mountains, it was designed to meet a very specific need. I’ve talked about pumped storage on the channel before, and Taum Sauk was one of the largest facilities of its time. Built by Union Electric, which eventually merged with Ameren, the whole plant is basically a battery. It’s actually a net consumer of electricity, which is normally not a good thing for a power plant. But managing the power grid isn’t only about how much electricity you can produce, but also when you can produce it. Large coal plants in the Missouri area could make lots of power, but they couldn’t ramp that production up and down to accommodate fluctuating demands throughout the day. So, Union Electric proposed a clever solution, one that’s pretty common today, but was innovative for its time. Two reservoirs were constructed: one low on the east fork of the Black River and another near the top of Proffitt Mountain, Missouri’s sixth-highest peak. Between them, a hydroelectric plant with two reversible turbines.

When electrical demands are low, rather than reducing the output of thermal power plants, that energy can go toward pumping water from the lower to the upper reservoir, usually overnight. Then when demand spikes during the day, all that stored potential energy created from cheap electricity can be harvested and put back on the grid by reversing the system to generate hydropower. Of course, you don’t get all the power out that you put in. Some of that water evaporates or leaks out, and there are losses of energy in the pumping and generation. But, with an overall efficiency of around 70%, it was more than enough to justify the enormous cost of building and operating two reservoirs and a power plant that doesn’t produce any of its own electricity.

The most striking part of the whole facility is the upper reservoir. It’s just such an unusual sight: a circular dam, sometimes called a ring dike or ring levee, perched on top of a mountain. This is not usually an efficient way to build a dam. We typically construct them across valleys so that the natural topography can form the sides and back of the reservoir. With a so-called “off-channel reservoir” you have to build the dam all the way around, increasing the costs and the engineering complexity. But there are no valleys at the tops of mountains, and that height is an essential part of a pumped storage facility. The power available from falling water is really simple to calculate: multiply gravitational acceleration, the density of the fluid, the volumetric flow rate, and the difference in height, called head. We can really only change two of these. So for a specific power output needed for a specific duration, you can trade height for flow. The greater the difference in height between the two reservoirs in a pumped storage facility, the less water you need to move, which reduces the size of all the infrastructure, and thus saves costs. The mountains in southeast Missouri provided a perfect location for the project, creating about 750 feet or 230 meters of height between the upper and lower reservoirs.

Actually the whole facility is named after the highest mountain in Missouri, Taum Sauk, which was the original site for the upper reservoir until there was too much pushback about building a project there, so they moved it to a slightly lower peak nearby. And they encountered some challenging conditions during construction, forcing the engineers to realign the dam to avoid an area of weak geology, giving it that unique kidney bean shape. The original dam was built as a rockfill embankment - basically just dumping a long pile of rocks around the perimeter of the reservoir. Rockfill usually works well as an embankment if you have a good source of material nearby. It’s really strong, doesn’t require a lot of compaction, and it doesn’t settle much over time like soil fills do. One thing rockfill doesn’t do well is hold back water. Too many spaces between the rocks. So concrete panels were installed all along the inside of the reservoir to make the embankment water-tight. A tunnel connected a morning glory inlet through the mountain to the generating plant. The inlet was set into a basin 20 feet or 6 meters below the bottom of the reservoir to suppress the potential for a vortex to form as it was drained each day. The whole project was designed to be operated remotely with no on-site technicians required, another innovation for the time, but one of the many decisions that would prove disastrous.

For most of its life, the Taum Sauk station operated on average around 100 days per year, usually during the hot summer months when electricity demands were more variable between night and day. Deregulation of electric power markets in the 1990s opened up the possibility of selling power to other utilities. Those 100 days per year went up to 300, meaning the upper reservoir cycled up and down, often twice per day, nearly every day of the year. And that was starting to cause some problems. The upper reservoir had dealt with leaks essentially since it started operating in the 1960s. Several projects were implemented throughout its life to deal with the issue, but the increased cycles of filling and draining were only making things worse. At one point, small ponds were built beside the reservoir to capture some of the leakage and pump it back inside. In the fall of 2004, Ameren decided to bring out the big guns and spent more than two million dollars to install a geomembrane liner to cover the entire reservoir. That essentially fixed the problem, but it caused a few new ones too.

About a year later, in September 2005, the Institute of Electrical and Electronics Engineers, or “I-triple-E” declared the plant an “Engineering Milestone” for its innovations in the world of electrical infrastructure. On the day before the ceremony, some of the participants took a tour of the upper reservoir and witnessed water pouring over the parapet wall on one side of the dam. The operators quickly switched from pumping mode to generation to get the water back down. They chalked up the issue to high winds from a remnant tropical storm that caused the overtopping, but just to be safe, they hired a dive inspection team to check on the level sensors. And what they saw was concerning.

When that geomembrane liner was installed in the reservoir, there was a valid concern that any penetrations might cause leaks in the future. But the reservoir needed level sensors installed for the control system to be operated remotely. So, instead of mounting those sensors directly to the concrete through the liner along their length, the engineers tried something different. Two cables were run between anchors at the top and bottom of the embankment slope. The conduits for the sensors were attached to those cables, minimizing the number of penetrations needed. Unfortunately, the mounting system was underdesigned. Those conduits were buoyant, and also subject to strong currents as the reservoir filled and emptied each day. Sometime after the spring of 2004, they had become dislodged and deflected, so the sensors inside were providing readings that were lower than the actual water level.

Based on those findings, the operators decided to reprogram the control system to subtract two feet from the upper set point on the pumps. The original design called for two feet or 600 millimeters of freeboard between the top of the wall and the maximum water surface. They figured that doubling that distance would be enough to avoid issues until permanent repairs could be made during the annual maintenance period when the reservoir was drained. Unfortunately, they would never get the chance.

Less than three months after that first time someone observed the reservoir overflowing, on December 14, it happened again, this time in the early dawn when no one was around to notice. Once the parapet wall collapsed, the water quickly eroded down through the dam, emptying roughly 6 billion liters or 200 million cubic feet of water down the steep mountainside straight toward Johnson’s Shut-Ins State Park, stripping away trees and rocks as it surged. By pure luck, the failure happened in the winter when the park was practically empty, but the park superintendent, his wife, and three kids (one of whom was only seven months old) were swept away when the water demolished their house. Incredibly, the entire family survived the event, but not without suffering from injuries and hypothermia. The wave of water flowed into the lower reservoir, where it would have gone anyway later that day, so there were no major downstream impacts.

The event was investigated by the Federal Energy Regulatory Commission, and the conclusions were surprising. Like most events of this kind, a series of small oversights, which on their own wouldn’t have resulted in a disaster, combined to cause hundreds of millions of dollars in damage, plus the effects on the family I mentioned. First was the embankment. That rockfill it was supposed to be built from wasn’t quite as rocky as the engineers who designed it intended. There was a lot more soil mixed into the fill, resulting in more settlement of the embankment over time. Unsound areas of soil in the embankment’s foundation were also not properly cleaned out, making the settlement even worse. From construction to failure, some parts of the parapet wall were a full two feet or 600 millimeters lower than where they started. That settlement wasn’t taken into consideration when the level sensors were replaced after the lining project in 2004. And with the sensors unattached and free to move around, there was no way for the logic controllers to know the actual elevation of the water in the reservoir.

Failsafe probes were installed on the parapet wall to provide a backup that would automatically shut off the pumps if the level got too high, but they were installed in a location that was actually higher than the top of settled parts of the embankment wall. If the water hit those sensors, it was already overtopping parts of the wall. And they were incorrectly programmed in a way that required both sensors to be activated before the pumps shut off. That first site visit when water was running over the wall didn’t trigger those failsafe sensors, but no one thought to check them. And rather than ground truth the important elevations like the top of wall and all sensor levels, they just decided to add a couple feet of margin and postpone a permanent fix. It would have been so easy to have someone on-site during those last few minutes of filling the reservoir each day to verify the levels against the electronic measurements, or even a closed circuit camera, especially after the enormous red flag of seeing it happen a few months earlier, but no one knew it was overtopping. And the owner hadn’t notified the regulator the first time it happened, so there was no oversight for how Ameren responded.

Probably the most significant error of all happened well before the facility was ever built. The design had no spillway. As an off-channel reservoir, there were only two ways water could get in: rain falling on top or water being pumped in. With enough freeboard for a rainstorm and the redundancies built into the control system, the designers never envisioned a need for a way to let water safely run over the top. Unfortunately, when you rely on complicated systems for safety, the likelihood for things to go wrong goes way up. These types of events are sometimes called “normal accidents,” a term coined by Charles Perrow. The idea is that, when systems are complicated, and especially when the safety measures themselves add to a project’s complexity, failures are much more likely, even expected. In other words, failure becomes normal. Compared to an industrial control system, a spillway is dead simple. Once the water gets to the crest, it just goes out. They’re not failproof - I’ve talked about several spillway failures in previous videos - but there are a lot fewer ways that things can go wrong.

FERC fined the owner 15 million dollars for the failure, the largest penalty they’ve ever issued. Five million of that went into a fund to improve the area around the project, although some recent reporting has alleged that those funds have been mismanaged. The State of Missouri also sued, and agreed to a 177 million dollar settlement, much of which went toward restorations at the state park, which held a reopening ceremony in 2010.

At the same time as Johnson’s Shut-Ins State Park was undergoing renovation, crews were working on rebuilding the upper reservoir at Taum Sauk. To avoid a relicensing process, the dam was built on the same alignment and to the same size as the original project. Rather than trying to repair the flawed rockfill embankment, Ameren and their consultants went with an innovative design. The new dam was built using roller-compacted concrete, a dry concrete mix that’s handled using earth-moving equipment and compacted into place using rollers. The new design would address both the settlement and leakage issues the original structure struggled with while still taking advantage of the material from the original embankment. That rock fill was crushed and processed into aggregate for the concrete, reducing the amount of material hauled into the remote site. Maybe most importantly, they included a spillway this time. The structure is the largest roller-compacted concrete dam in the United States. The plant reopened in 2010, was rededicated as an IEEE milestone, and the project won the US Society on Dams’ award of Excellence in the Constructed Project.

The failure at Taum Sauk was a wake-up call for the professional community. The regulator, FERC, implemented some big changes to its oversight of dam safety in the wake of the collapse. They put together a task force that issued a technical guidance document specifically addressing the challenges of pumped storage facilities that was circulated to the owners. They also updated rules for owners, requiring them to have an internal dam safety program and a Chief Dam Safety Engineer who is responsible for overseeing it, a role Ameren didn’t have at the time. The event triggered states as far as Hawaii to bolster their dam safety programs. And most importantly, the failure demonstrated the need for overflow spillways, even for off-channel reservoirs with redundant control systems meant to avoid overfilling.

If you’re paying attention to issues related to the electrical grid, you know the importance of storage. Particularly as intermittent sources of power become a large part of our portfolio, ways to balance out those mismatches in supply and demand are only becoming more important. Pumped storage has traditionally been the only large-scale way to do this economically, but obviously, it comes with a tradeoff. Dams are among the riskiest structures that humans build. They don’t fail very often, but when they do, those failures usually come with serious consequences to people, property, and the environment. And because they don’t fail that often, those lessons come slowly and tragically. But, with battery storage becoming cheaper and much more widespread, it will be interesting to see how the economics of pumped storage change. By 2030, some are predicting the US will have more than 400 gigawatt hours of battery storage on the grid; that’s more than 100 Taum Sauks. We’re right at the beginning of some major changes in how energy is stored. Those batteries have a lot of technical differences in how they interact with the grid, and they come with their own environmental challenges and safety considerations, but the risk profile is a lot different than building a major reservoir at the top of a mountain. As energy infrastructure keeps evolving, those differences in risks are probably going to shape the future of how we store power, and at what cost.