(this page is currently being edited, please pardon the machine translation until this is complete)
(complete)
I’m David Torcivia.
I'm Daniel forkner.
[0:03] And this is Ashes Ashes, a podcast about systemic issues, cracks in civilization, collapse of the environment, and if we're unlucky, the end of the world.
[0:14] But if we learn from all this maybe we can stop that; the world might be broken but it doesn't have to be.
In December of 2017 and underground electrical fire at a substation serving the Hartsfield-Jackson airport in Atlanta caused a power outage that lasted 11 hours.
More than 1,000 flights were cancelled stranding passengers and putting stress on Airport Logistics around the world.
That's right today we're going to be talking about the electrical grid.
[0:46] Now Daniel what is the electrical grid.
[0:48] Well David it's very simple, at least in concept, and it comes in three parts. You have generators of power that includes a diversity of sources including renewables, and these generators are connected to high voltage transmission lines that carry that power long distances to distribution center, and those distribution centers then use transformers to step that voltage down so it can be distributed to homes and businesses.
And those are the three parts to the grid: power generation; transmission; and distribution.
[1:19] Well that sounds pretty simple by itself, but when you actually start digging into the grid it's unbelievably complicated, and to understand how it ended up like that we need to understand how the grid first started.
[1:30] It was really cobbled-together right? It wasn't something that was designed in the way we typically think of designing a complex system, but kind of emerged piece by piece.
[1:40] Exactly that's the big thing to take away from this.
The grid as it stands today evolved. This wasn't something that we set out and built like the interstate system or something, it was a very organic evolution first of small localized individual power generation, to the very massive power generators on an industrial scale that we have today.
[1:59] It really all got started in 1882 with a small power station in New York called The Pearl Street Station - this was designed and built by Thomas Edison - to provide power for the financial district of New York, and it's sort of funny at the time you know we measure power generation today in megawatts or gigawatts, but this was just measured in lamps. It could power 400 lamps which was considered state-of-the-art at the time.
[2:22] Wait you mean lamps like my desk lamp that I have on right now?
[2:25] Yeah like light bulbs. Like this could turn on 400 light bulbs which was of course the main use of electrical power at the time, and it was only later as we started electrifying more and more of our society that we started branching out from just electric lighting.
Now that generation facility didn't last that long, it burnt down just a few years later, which is a great demonstration of why it's maybe not the best idea to have power generation in the middle of a city, but at the time that was how things worked.
You would build the power generator, often in times in a small building, a small generator, right next to the places that it was going to be distributing that electricity.
And there were no standards, there was no design, everything was just sort of haphazard. In fact we hadn't even agreed on what type of power we were generating. Edison was pushing the DC power, there were other people pushing AC power. There was no agreement on voltages or anything, and as time went on and we started to understand electricity and electrical generation better, we started concentrating these into larger and larger generators to make use of economies of scale.
General Electric started building these huge generators, and then we put them far off from the city and ran distribution lines over to where the power was needed. This was really the start of the evolution of the grid that we recognize today.
It really got going leading up to World War I to support the factories that needed to produce all these products for the war, and then the boom of dam construction during the Great Recession - things like the Tennessee Valley Authority; the construction of Hoover Dam - is what kicked off the massive grid that we know today, and by World War II our grid system was basically modernized.
[3:54] And this grid that we have come to depend on - and our dependence on it is really growing in ways that we really take for granted...
I mean our transportation networks depend on this, like we saw in the Atlanta airport, we require it for all our basic necessities at home: cooking, air conditioning –
Even Water?
Even water like our water treatment centers that ensure that our water is drinkable, and partly because of the way this grid evolved and was cobbled together, and just the way that it works that generation has to constantly meet our consumption levels, it's extremely sensitive and fragile that's constantly at risk of failure, and that risk is going up.
The risk is rising because number one we have an aging system. When you look at a power line for example, you might not think there’s much to it, but these are extremely expensive, and a lot of our power lines were designed with a 50 year life expectancy. They were built in the fifties and sixties, they weren't meant to handle the extreme heat and weather patterns that we experience today, and the Department of Energy (DOE) estimates that it would cost 900 billion dollars to update these aging lines.
Part of that is to meet the rising threat that climate change is putting on our power grid. Our transmission and distribution lines are already at full capacity during hot temperatures in the summer when demand for electricity peaks, and rising heat causes these power lines to physically expand and sag, which increases the risk of power outages at the same time we need them the most.
Enormous blackouts have occurred from a single power line that sags and shorts out from foliage contact.
[5:27] Foliage maintenance is actually one of the largest areas that these utilities are investing in protecting the grid, and it may seem silly to think of it this way. When we think of grid protections we might think of investments in new construction, or physical security, or maybe cyber security stuff which is a threat we'll talk about later, but actually the greatest threat to the grid statistically speaking are trees.
There are huge crews, obviously you've seen them before, going around cutting trees back away from these lines. That’s a major area of investment for these utilities. They have new tools, they use a chemical defoliant; I've even seen drones that have chainsaws and flamethrowers on them in order to clear trees and other plants like Kudzu or Ivy away from these power lines in areas that are harder to reach for maintenance crews.
But these areas are also some of the first that are cut when profits start getting tight. So when economic downturns happen, or when power generation gets more expensive due to reasons like increasing prices of natural gas, or coal, or other important components of generation. Then these crews get cut and then you start seeing -
[6:30] The crews get cut but the trees don’t get cut.
[6:36] There you go that’s another unintentional pun. But yeah exactly so the crews get cut, the trees don't, and we start seeing more and more outages from this problem.
[6:45] Not to mention the increased weather extremes we no experience as a result of climate change. In 2015, 25% of all power outages in the Western United States were caused by weather.
[6:56] And part of the reason that things as simple as increased temperatures or changes of weather can throw off the grid is because it is just so complicated. The grid is not this static, unchanging thing, but it’s a constantly evolving, constantly growing, shrinking, adjusting, almost living being. We add generation, we add distribution, we change the type of generation… all this stuff is moving around constantly and it's not just one single grid across the country either, there's three major grids: one on the West Coast; one in the East; and then Texas has their own for a variety of reasons, the most prominent of which is trying to avoid Federal Regulations by keeping all the power generated within state lines.
But as the grid grows and evolved and has to deal with new problems, new challenges, and some of those are even just in the way that we generate electricity itself.
[7:46] Obviously there's a big push for renewables these days.
[7:50] As there should be.
[7:51] Because of our desire to get away from fossil fuels and cut back on our carbon emissions.
[7:56] No let's rephrase that Daniel. Not desire because that doesn't sell it enough, but this is an actual need for civilization to continue surviving. We have to do this. Let's not understate that concept.
[8:07] Well we certainly have to cut back our greenhouse gas emissions I agree with you David, but in this push for just adding renewable energy to our grid we've neglected the fact that the grid itself represents a limit to our ability to add these renewable.
The big problem with renewables is you cannot control the amount of energy produced like you can with a steady fossil fuel plant, and unfortunately climate change is making this volatility worse. Solar and wind have always been highly variable - and that trend is still rising - but now we’re seeing fluctuations in things like hydroelectric generation, as the climate affects precipitation, snow melt, and other factors. This is only negative. Complexity and uncertainty in the grid is only bad for us. Over production does not help us, and underproduction is bad for obvious reasons.
[8:58] The reason that that's the case is that thing that we mentioned earlier about supply and demand. Right now the grid can’t store energy in any significant amount; it's always about immediately balancing the energy produced with the amount of energy that is needed at any one moment, so power generation constantly fluxes more in certain times, down in other times, to keep this push and pull between suppliers and consumers even and balanced, because imbalance on either side of this can throw the entire grid out of whack and bring the whole thing crashing down.
[9:28] For illustration 13% of homes in Hawaii have solar panels, and on sunny days they produce more power than the entire state can use; it’s wasted. And when it comes to wind farms ...
Actually David, I want to bring up a quick aside - one quirky consequence of our push for renewables. Utility companies used to be more vertically integrated. Like Edison’s pearl street plant, if you owned the power plant, you'd also build and own the transmission and distribution networks, selling power directly to the consumer. In the 90s we started breaking those structures up to increase competition; so now one company owns the power plant, one owns the transmission lines, one bills customers, you get the idea.
Then we started subsidizing renewables like wind farms. In many cases, the subsidies don’t kick in unless a company is generating power. So what happens is, I own a wind farm power company, and I go out and find the best land with the best wind to build my turbines. When i finish, I hook them up to your transmission lines David.
And I start generating power. But then, because I built my farm in the middle of a desert maybe that doesn’t serve a lot of people, I start producing more power than you’re willing to buy. So I have to go to another transmission company and the same thing happens, and a regulatory body calls me up and says “shut your power off.” And I say “Why should I?” So they end up paying me to turn off my turbines.
[11:07] Wow the efficiency of the markets never ceases to amaze me, but this seems like a crazy made-up sort of story, there's no way that someone would build a wind farm way far away somewhere without any customers in order to distribute that electricity to right?
[11:21] The world's largest wind farm is in China's Gobi Desert. It’s in the Northern part of the country, and it sits idle most of the time. There are 7,000 turbines there but most of it is turned off, and it's because it's extremely expensive to build those long-distance transmission lines which will carry that power away from the wind farm and into the highly populated cities that really need it.
[11:45] These examples really illustrate why bringing the grid up to date with renewable energy isn't just about installing more solar panels or building more turbines, but it's about really overhauling the grid itself; about building more lines that are capable of carrying these things from where there is ample solar power or wind power - which is often somewhere out of the middle of nowhere - to places where people actually need this power.
But unfortunately so much of our investment, so much of the attention that we pay in this renewable construction is just that: only on renewables, and we ignore the grid itself which is just as important in carrying this power to where it needs to be used,
[12:19] and another way that comes into play is with energy storage.
[12:22] Grid storage is really interesting to me. Large scale grid storage is really interesting. I always assumed - if I thought about it at all, which is really not much - that we would have large batteries for our electrical grid just like the Li-ion ones we have for our phones - and there are some early commercial batteries out there on the grid, and on paper batteries could be pretty efficient solution to grid storage, but it doesn’t look like these will be possible to scale due to a lack of resources and the costs associated.
So what we actually use are not batteries, but water. We push water up a tunnel from one reservoir to another using excess electricity, and then that water comes rushing back down when we need it, turning turbines along the way.
But water isn’t the only method, it’s just the one we use the most, but it’s this idea of potential energy storage. There are proposals for example to build train cars in the desert with 300 ton payloads we can push up an 8-mile incline, and then let it roll back down when necessary to generate some power.
There are some other interesting ideas along these lines, but you can see these will be location dependent, and it’s not clear if efficient large-scale storage across the country can be met with these methods.
[13:37] But this is not something that we're going to have a choice about as we start bringing more renewable energy online.
[13:42] At least not in the way that we've been doing things.
[13:44] Exactly Daniel. We're going to have to invest in these grid storage technologies because the sun doesn't shine at night, because the winds aren't steady and even, because if we want to get to a place where we have a hundred percent renewable power production we're going to have to be able to store energy in order to smooth out the peaks and valleys of this unpredictable power generation scheme.
In the meantime though we are able to balance this out with traditional generation that is able to be ramped up and down much more quickly, and especially with natural gas which very quickly allows us to increase or decrease our generation, so much so that there are some products - called peekers – that are designed solely to balance out the uneven power generation from these renewable sources.
Okay we’re just talking about the way grid, and we’re just talking about some problems that we’re going to face in the future moving forward modernizing this grid, to bring it to a more responsible and environmental generation scheme,
But there are a lot of real threats to the grid right now, and one of those is the aging transformers.
[14:43] Yeah David we mentioned that this grid is a complex system, and very vulnerable right to risks and disruptions. And the most terrifying part of our grid is the vulnerability associated with these transformers.
Transformers are found at pretty much every single stage within the grid, both at the generation side, and the distribution side, and the role of a transformer is simply to change the voltage level of the power so it can be used more efficiently. Whether that’s raising it so that it can travel most efficiently along transmission lines, or whether it's bringing that voltage level back down once it gets to a distribution network so that it can go in your home where your outlets are very, very low form of that voltage.
So these transformers are pretty much the Lynchpins that hold our grid together. They’re the critical pieces.
And a lot of them are old. The average age of these large scale transformers in our country is 40 years old; some of them are over 70 years old, and although they are designed to last a long time, the risk of failure increases with age, so there’s no timely warning for failure, they just get too old and one day stop working.
The reason these transformers are so terrifying, is because when they do fail, and you have to replace them, it is very expensive, and it takes a long time.
[16:02] Right and it's these large transformers, the ones that that do the big step-up, step-down that are the concerns really.
So there are small transformers all over the grid; you can walk outside your house, look at the telephone pole, you'll see like a little round trash can looking thing attached to it. Those are small transformers.
[16:17] I walked around my neighborhood the other day, and I was actually counting how many homes are hooked up for each of these little small transformers, and in my neighborhood it’s about 4 houses.
[16:25] Right, and they’re everywhere, they’re ubiquitous, you see them in every part of your life. Once you start looking for them you’ll notice just how many there are.
And these aren’t really at risk; they’re small, they’re standardized; they’re easy to put out. But what these very large transformers are is where the real risk in the system is.
[16:41] And now these things cost millions of dollars; they weigh up to eight hundred thousand pounds; they’re enormous, they're basically the size of shipping containers maybe larger, and they're entirely steel and copper. Just huge, giant, massive things. You’ve probably driven passed them before; they're surrounded by chain-link fences with all these wires and stuff going into it, and all this metal and copper and steel in them make them very expensive. Both those commodities - the copper and the special electrical steel they used to construct these - are not very common and can be very expensive on these markets. There's a high demand for them and a lot of variability in the price. So just buying the metal alone to build these things is over half the price of the entire transformer.
[17:20] And they takes so long to acquire because of their massive size and the manufacturing that’s involved. So in some cased it can take up to twenty months before the ordering and final installation of one of these transformers. Absolute best-case scenario you're looking at 5 months.
They take a long time to manufacture because these things are all custom-built. Typically, only one is built at a time for each design, so there are no spares and no interchangeable parts. And because of their massive size, these things are really (really) hard to transport. This really can't be overstated.
In the United States we have to import about 80% of all large transformers, and transporting them is a huge undertaking. The most common way is by rail, but only specialized rail cars can handle the weight, and we have 30 of these specialized rail cars - total - in the entire united states. Not to mention as we discussed in our infrastructure episode, our commercial railways have problems and many lines are damaged or being removed.
[18:22] When those railroads don't work out because they aren't going where you need the transformer, or because they're broken or backed up or whatever problem it is, you have to move these on the road. And these things are giant right? So you need these specialized semi-trailers, they call them caterpillars, they're covered with 200 wheels, just ridiculous, humongous, almost transformer-like things and they require so many permits to drive through any sort of municipality. Workers have to physically move power lines to get this out of the way, and the whole process of moving this transformer from the factory to its final destination is an enormous headache in every possible way.
[18:56] You just said that we use transformer-like trailers to move transformers.
[19:00] Yeah well you know Michael Bay Transformer, and critical infrastructure Transformer, it all goes hand-in-hand really.
[19:06] I mean when these things explode it's pretty magnificent so.
[19:10] That's true y'all should look up YouTube videos on it because this is straight up Michael Bay explosions.
[19:15] Michael Bay get on it.
But all this to say, if you have all the time in the world, no problem, but if you are in an emergency situation, where power is down, and you need to replace these to get electricity flowing again, you don't have 20 months. And besides, that long lead time is assuming you have a perfectly flowing logistics network, with cranes and ships, trains and communications; and all the infrastructure you can expect to have delays in the event of blackouts.
[19:45] We're rambling a lot right now about transformers.
We really want you to understand how important these transformers are in the security of the grid as a whole.
These transformers live at places called substations. Those are those chain-link fenced, walled-off areas that we talked about earlier that you drive past on the road. Some of these are very critical. They connect power generators to the rest of the grid, and when a substation fails it can have cascading effects across the entire grid, breaking the whole thing down.
There have been very serious studies done by the government and by utility companies themselves that's say as little as 9 substations knocked out could bring down the entire US grid for 18 months.
Let that sink in for a second.
[20:25] To put it in perspective David. There are somewhere around 55,000 substastions in the United States, so 9 is a very small number. And to be fair not every substation is equal rights? There's estimation that 350 out of those 55,000 substations are the most critical, but it's kind of hard to estimate at the same time because like we said the grid is constantly fluctuating so a substation that's risky one day might not be risky the other day depending on how we're using electricity in that moment and the specific combination of substations that you would need to take out to cause a massive blackout really depends on the situation.
[21:05] And this is where the story starts getting fun because so far we've talked about the risks of this, and our threats have been things like trees and weather, but maybe you don't realize that the grid is actually almost constantly under attack.
[21:17] There was a report from the government a few years ago that had at least a hundred attacks on the grid every year. That’s more than an attack every 4 days and those are just the attacks that we know about, and some of these are very serious.
[21:29] Despite the seriousness, most substations are so incredibly lacking and security it's kind of funny.
For example, one substation in Arizona - the Liberty substation just outside pheonix - is an important substation that connects many northern and southern states on the western grid. And in 2013, a series of physical attacks were carried out against this station.
First, someone cut the fiber-optic cables to Liberty, disabling communications for a couple hours. They never figured out who or why. But then two weeks later, multiple alarms started going off in a nearby control center signaling that something was wrong at the substation. These alarms went off for two days before someone was sent to check on it, and when they arrived, they found the fence had been cut open, the control building broken into, and a bunch of computers had been messed with.
[22:21] You mean somebody actually broke in and was messing with computers, hopefully they at least had some sort of security camera where they could check and figure out who did this right?
[22:29] There were some cameras, and a security team checked them. But most of them just pointed at the sky.
So they installed some new cameras, and a security trailer, but then two months later, another break-in occurred at the same station, and when they checked the new cameras, they discovered that none of them worked because they hadn’t been programmed right.
[22:50] This really inspires confidence in the security of the grid and these utility operators, and these kinds of attacks aren't just one-offs. My favorite story is one that remains a mystery today.
This is the attack on Metcalf substation in California.
[23:05] This was such a terrifying attack that the government and utilities got together afterwards to introduce a number of new and dramatic security measures to prevent this from happening again. But as Daniel mentioned who knows how effective this actually is in practice.
Let's talk about the story for a minute.
I love this store because it's so dramatic, and there's a lot of mystery about why this happened and who was behind it still to this day.
Okay so this is 2013 in Coyote, California a little outside San Jose. There’s a company called Metcalf that has a critical substation there. And so the night that this happened around 1 in the morning there was a disruption in fiber optic cables. Somebody broke into a vault outside of Highway 101 just close to this substation that we’re going to be talking about, broke in, lifted this huge metal plate off the ground - they said it took a couple of people - and they came in and they cut a bunch of fiber optic cables from AT&T. It took AT&T a little while to figure out - which won't surprise anybody who has AT&T as a provider.
10 minutes later there was another set of cables cut that disrupted information for Level 3, one of the providers of the backbone of the internet, and this was in another vault that was cut close to the substation.
30 minutes later a camera recording the substation noticed a streak of light in the distance.
[24:23] And now investigators later would decide that they think this streak of light was a waved flashlight that kicked off all the following:
Immediately after the streak happened - this is at 1:31 a.m. - there were dozens of sparks and the flash of rifles in the distance recorded by this camera, and you could see the sparks as these bullets hit the chain-link fence and shattered and exploded and all this action in the camera kicked off a warning, an alarm, on their camera which detected the motion.
So this is 1:37 a.m. a few minutes after the firing started.
At 1:41, 10 minutes after the signal was given, the sheriff's department received a 911 call; it was actually the plant engineer of the people generating the power - he happen to hear gunshots - and he called in 911 said “I hear a bunch of gunfire in the distance you should probably check it out,” and so Santa Clara Sheriff recognized this and they drove out to the substation. They got there just 10 minutes later but by the time they got there everything was quiet. That Sheriff arrived just one minute before another flashlight signaled the end of the attack and the end of the gunfire. What are they shooting at you might ask? Well it's these very large transformers.
Transformers are actually physically simple things, it's just bunches of wires wrapped around it but there's one important part in that is the fact that it gets very hot. The was they cool these trasnformers are with huge drums of oil, and these snipers had targeted these oil tanks, put 100s of rounds of bullets in them, and let the oil leak out.
The police arrived, nobody noticed, it was dark can’t blame them. And the oil slowly Drained out.
[25:46] Over 52,000 gallons of oil leaked out of these transformers, and after a little bit they overheating and exploded. A worker arrived a few hours later to survey the damage but it had already been done. This kicked off all sorts of alarm bells in the utility company, they called up the FBI, the FBI sent people out there, and they started investigating this because it was such a well-planned well carried off attack, nobody was sure exactly who did it but they were positive that this was a highly trained military team.
Fingers were pointed at China and Russia saying it had to have been operatives from one of these nations. They found bullets from where the attackers had fired but they’d been cleaned of fingerprints. They found stones marking where the attackers should fire from meaning that they’d come out and scouted this site before and knew exactly where to show up to inflict maximum damage.
The firing of oil drums shows that they understood exactly what these transformers were and the fact that they needed to target these things in order to inflict maximum damage. But to this day nobody is sure why they did this. The power company was able to quickly route power around the substation. Silicon Valley continued having power though the company did ask them to reduce their power consumption for the day until they could get things back figured out. But why would they have this attack?
The big fear is that this is because it's a practice run, saying “if we needed to attack these transformer substations, how would we do it and would it work?”
And so this event shows that the idea that you can attack this, knock this out very easily, with no repercussions, isn't just possible but simple.
[27:13] It's very possible that that was a dry run practice to see how a physical attack carried out with such precision and sophistication could affect our grid. But where we’ve seen a lot more practice attacks, and where we see the greatest risk going forward is actually in cyber.
But before we go into cyber attacks David I think we should just point out - some people might be wondering “okay if these transformers are such a big risk, surely the government and surely utility companies are putting practices into place to protect them or to replace them in the event of an emergency right?.
[27:45] Yeah that would make sense.
There are some very minimal programs and plans in place to have spare transformers in reserve. If you ask the utility industry they'll give you a pretty optimistic perspective about their spare capabilities, but in reality it's not so good. Utilities that do store spares do so in locations that can serve a limited number of substations, and from these places it will still take multiple days - 10 or more - to deliver the spares onsite. But more importantly, there are not a lot of these types of reserves, and utilities have to choose between: really expensive custom spares that can only be used to replace specific transformers,
[28:27] Because again many of these transformers are custom designed for just that one install location; there's no one-size that fits all in terms of these very large transformers.
[28:36] And because there are so many substations - and again the risk is not a single substation but a combination of them so - even if you could have identical spares for a particular substation, that's not going to help you if all the substations around you go out right?
So there is a push for more versatile transformers, ones that can be suitable for short-term deployment following emergencies. But these also have problems. Because they're more versatile usually they're less efficient; the load they bear is not ideal. And because you can't cycle these into regular use they end up just sitting in a warehouse somewhere and growing old and Storage. And that's the trade-off that these companies face, and a lot of them just aren't incentivized.
Like you said David even though foliage - literally trees and plants - sometimes represent the biggest threat to our grid, utility companies don't even want to pay to send people out with chainsaws to cut those back so you can understand their reluctance to spend millions and millions of dollars on spare transformers that they're not even sure they're going to have to use.
[29:35] So that really emphasizes how critical these transformers are and how delicate a part they are of this grid system, and as we see more and more attacks - both physical and cyber - this is something we need to be concerned about.
But let’s turn back to cyber attacks for a second. Our grid is getting increasingly more connected to the internet.
[29:53] And this is a good thing when it comes to balancing loads, controlling how electricity flows, and as a whole makes our grid more reliable. At the same time it also makes us more vulnerable, and that's the concern with this cyber security.
Now to understand cyber security attacks we really need to look at two sections of it. So there's first something called stage 1 attacks, and then stage 2 attacks.
A stage 1 attack might be something similar to that physical test that we saw in Metcalf where somebody comes in, they break into a closed system, and I say this in terms of virtual break-in. This might be a virus, this might be a phishing link somebody clicked on, whatever it is they get access to these systems.
Now access to these systems by itself usually isn't very useful. These utilities, these power generation companies, these distribution systems are incredibly complex. They use very customized ICS - industrial control systems – that are custom to each individual specific install. So there’s no one way you can break in and then press some buttons and you know you’re going to be shutting down transformers, re-routing power or shutting down the grid because each system is so unique just like transformers.
So these stage 1 attacks are focused on breaking into the system and then waiting. Spending months learning everything you can about how this system functions, about what this button does, about how to control the flow of electricity and other information in order to do maximum damage. And these are the vast majority of attacks that we see right now. There are concerns that Russia, China, other countries are breaking into our systems and sometimes we’ve pointed our fingers and said “yes we know that you broke in and we can see right here;” this happened in a case recently with some Russian hackers breaking into these Industrial Systems. But whether it's for the state or criminal syndicates is harder to determine.
But this is happening all the time constantly. The controls of our grid, the way that electricity is routed, is known. This is a known thing at this point, and so then the question becomes “well what happens when we hit stage 2?”
[31:47] That's exactly what happened in Ukraine in 2015.
It's thought to be the first successful Cyber attack on an electrical grid and it was so alarming to the security community that it's been studied a lot by security experts, military experts, the US government has spent a lot of time talking about this in connection to our own grid.
[32:08] This is such a great example of what happens when everything goes wrong.
Okay so these hackers, whoever they were and we still don't know to this day, they broke into this system using these phishing links; they had tricked some engineers to click on - they opened word documents and the document prompted them “click okay to enable macros” which is something Word does and it's supposed to be part of the product, so they clicked okay and they got viruses on their systems. These viruses laid dormant for months as the hackers looked around, figured out everything they could, because this Ukrainian facility was actually very well secured. They had great firewalls, lots of logs, and it took them a very long time to figure out how to infiltrate this, to get into the ICS, and then be able to actually inflict some sort of damage.
When it finally came that they had enough information and were ready to attack it was almost comical how it went down. The plant operator was sitting there at his control desk and then all of sudden his mouse started moving by itself and clicking off all these things, and moving around and activating stuff, disabling things.
He just panicked. He look at this and he's like “oh my God what is going on?” he's trying to shake his mouse, the mouse obviously doesn't work because they've shut down control for that, and he just sat there dumbfounded not knowing what was happened until he realized “oh my God we've been hacked,” and immediately started running around trying to damage control, but at this point the damage had been done.
The power went out all across the city, 30 substations, over 230,000 people were all plunged into darkness.
[33:30] Now fortunately for the Ukrainians they still had a system that had lots of manual reboots available on the physical controls so they were able to go back and get power back up within 9 hours, and this is important because a lot of their digital controllers have their firmware re-written, and even months later they still didn't have full control over their system and were having to physically replace parts of the grid slowly to bring it back under their control.
In this way they're older technology actually saved them, but here in the US we moved away from these manual reboots.
[33:57] Yeah.
We have been moving away from that; we've been getting rid of a lot of our manual override switches and controls within our electric grid, and with that go the lineman and other experts that know how to fix grid problem manually.
What’s so interesting to me is that traditionally it's been very difficult to manipulate computer systems to impact the Grid. At most you could maybe shut a computer down but you wouldn't be able to interact with a power network directly, but because we've increased our reliance on smart tech, it has increase the number of doors a hacker can open to gain access to our network, even if that is just hacking an engineer’s computer so that you can see what they see right?
And once a hacker gets into our networks, because of the level of sophistication of these monitoring and control systems - these smart tech devices - these hackers can covertly stay in the grid, cohabitate with the engineers and study the infrastructure in depth. In some cases even taking screenshots of actual control panels and the layout of different buttons, and then they can use that engineering knowledge really to plan and execute commands, disrupt power flow causing blackouts, and potentially damaging equipment like those transformers that are so important.
And many security experts and utility experts that discuss these cyber attacks have a very narrow idea of the motives behind a potential cyber attack.
[35:19] Because right now the level of resources and sophistication needed to carry out one of these attacks is high, these experts assume that the only motivation for such an attack would come from a state-backed group in the context of some political conflict like war.
But don't you think David that the range of potential motives go way beyond that?
Yeah absolutely. I’m saying this as a purely hypothetical thought experiment, but if I was some sort of radical eco-group, one of the most effective methods that I would try and attempt either is one: get access to these power system so you can shut that down, mess up the grid that way because that would save a lot of greenhouse gases right? Or two: mount a physical attack on these transformers like we saw in Metcalf and if I hit enough you know then you get to a point where it takes a long time to replace these transformers, you might have 18 months until the whole grid can get back online because it's hard to build transformers without electricity and I would make a huge dent in the emissions of the United States, and a positive effect for the environment as a whole. But that just hypothetical speaking.
[36:25] That's pretty creative David but it's hard to imagine an eco-terrorist having the type of financial backing or incentives to carry out one of these attacks so let me give you one better:
Let’s say I’m a small hedge-fund manager with a billion dollars. I could bet against some commodity price – like bananas – and then hire a group of hackers to shut down the power grid associated with some sensitive logistics infrastructure in Ecuador for example, during an important export month…
[36:54] Why would you do that when you could just install a dictator in Guatemala?
That’s probably a little too obvious now that we’ve seen that play out.
[37:03] But you could do this, you could shut the grid off in Ecuador during an important month, you could cause a lot of bananas to spoil affecting their price, and then make huge profits on the financial bet you make. And in this complex world who could possibly connect that back to me?
Maybe you could even do that in a politically conflicted area so that the media just assumes it’s related to the conflict going on there.
[37:25] Right I mean our conversation has been about the grid primarily in the US but these facts are true almost anywhere. It doesn’t matter where you go these transformers are the same; they're just as vulnerable here in the US as they are in Europe, as they are in South America, as they are in Africa.
These problems exist everywhere, and all it takes is somebody that has the motivation, and the resources – the US is concerned about groups like Anonymous, or some of the online Jihadi groups having the knowledge to pull this off - or like you mentioned the economic resources in order to get the physical strength, physical gun-power you need, in order to pull an act like Metcalf off at a wider scale.
[38:00] Security Experts are even saying because the number of cyber attacks are increasing, and because of their level of sophistication, they’re telling utility companies “hey, just expect this to happen; expect your grid to fail in the event of a Cyber attack, and just focus on how to recover as fast as possible.”
Why don’t we look at a couple more examples of risks that we face to our electrical grid.
[38:22] Well I think this will be a fun place to play “what if” when it comes to war.
[38:26] So you mentioned one of these motivations that the government is so concerned about is that these are done by nation states, and the only motivation they can understand is war.
Well if it ever came to that, what would happen? And I think this image that we have in our head from media and from our experience of the past many decades of the threat of nuclear annihilation is that when war breaks out, when the war to end all wars World War 3, that we will have alarms on our phone going off telling us missiles are inbound like we just saw in Hawaii, and that that will be how the world looks as we wait for it to end.
But I posit that instead when war breaks out, the only thing we'll know that happens is that the lights go off.
Any attack from one major nation to another at this point is going to begin with Cyber attacks. The missiles aren’t going to be flying immediately; there's not going to be boots on the ground. What will first happened is armies of hackers will be attacking each other. Our grids will go down, utilities will be destroyed, and we won't have any idea what's going on, or even really that we're at war at all.
[39:26] I'll just wake up, I'll try to turn on my TV, and it won't turn on. How will I even know what's going on?
[39:31] We might have a couple of days left of cellphones but the major systems that hold up our communication networks - the internet, broadcast - they have generators but those only last for a little while, and when those pumps run out, when you can't fill them back up anymore, then we’ll just have no idea and we'll all truly be on our own.
[39:47] I don't own a TV by the way. Don’t buy a TV.
[39:50] I own three TVs.
Nice.
And what does it look like when this system that we have become dependent on, that we take for granted, shuts down?
In 2011, a combination of equipment failures and some human error combined to shut down a transmission line somewhere outside of Arizona, and it led to this cascading effect where 11 minutes later the grid went completely dark in San Diego.
The effect was immediate, and it was quite profound.
It took them 12 hours to restore power, and in that time, 7 million people were without power, including about a million in Mexico. Traffic lights stopped working and there were major gridlocks on roads. People had to stay outside in some places because it was so hot they were at risk of overheating if they stayed indoors. Flights were cancelled. One of the hospitals in the area lost power because their generator only lasted 2 hours which just goes to show how prepared we are for these type of things. There were major food losses at grocery stores and restaurants, and at one point 1.9 million gallons of sewage were dumped into a nearby Lagoon because this pumping station failed. The beaches had to be shut down, and drinking water was contaminated so a lot of people had to resort to boiling water to avoid infection.
[41:09] Yeah the power going out sucks. I mean it's silly to think about right? We know how dependent we are on electricity, but there are lots of little systems that we don't think about all the time: pumping fuel into your gas so you say “well if the power goes out I can always drive somewhere” but if you don't have gas you can't refuel it because pumps require electricity.
A lot of Municipal Water Systems require electricity to even pump the water, so if you're not in one of the few cities that have gravity-fed water you won't even have water flowing out of your sink after a few hours or days.
Everything is dependent on some level on this electricity and even a few hours of delays - much less many days - can have cascading effects not just on the electrical grid itself but also in the ways that we live our lives, and people die.
[41:51] It sounds like if there is a Cyber attack on a grid that kicks off some kind of war, we're going to be too busy just trying to survive to even notice right?
What are some other attacks that could happen that could affect our grid?
[42:02] So what we're talking about a lot in this show are things that are called Low Probability High Impact Events.
These are things that aren't particularly likely to happen - at least in the grand scheme of things - but if they did, mess everything up sometimes permanently.
[42:15] One of the biggest fears for this is something called an electromagnetic pulse, and there are a couple ways this could come about, but the most popular and man-made one would be with a high-altitude - extremely high altitude - 250 miles above earth nuclear detonation.
[42:31] Such a detonation would send this electromagnetic pulse down to earth. It would overload and destroy transformers, and potentially cause this cascading effect where enough substations are disrupted that it just causes the grid to overload, and all of a sudden, we have massive blackouts potentially across the entire country.
[42:49] There are a lot of disagreements on just how likely this is, and just how much damage this would actually do. In fact these disagreement exists even within the government itself, with some groups saying that this is maybe the single greatest military threat that we face as a nation, and others saying “you know what? The science just isn't there; we're not sure we should worry about it at all.”
[43:07] But depending on the fact that nuclear war maybe just isn't that likely – at least we hope it's not - there's also a natural phenomenon that has the same effect as these EMP detonations.
That’s something called a coronal mass ejection or CME.
[43:22] So the magnetic forces at the sun are powerful.
It makes the surface highly volatile, and it causes powerful solar flares. These are occurring all the time, and sometimes accompanying these solar flares are these massive outburst of particles and magnetic energy known as coronal mass ejections.
These ejections fly away from the sun in giant arcs - I mean these are really really big - and if their trajectory sends them towards earth, we are in trouble.
[43:50] And these events aren’t unprecedented.
Like Daniel mentioned they happen all the time and in fact have struck Earth.
Now not all of them are extremely powerful or x-classes as astronomers call them, of the highest class with the most energy that have the highest potential to disrupt our life but just under 160 years ago, one of these extremely powerful X-Class flares did hit Earth.
This is called the Solar Storm of 1859 or more popularly the Carrington Event. Now this storm is the largest that has ever hit Earth, at least in recorded history. This event was actually witnessed live by two astronomers. One of them named Carrington, who gave his name to the event, and another one called Hodgson.
[44:27] Both independently recorded this massive solar flare breaking out, and 17 and 1/2 hours later after the solar flare was witnessed coming out of the Sun, this Coronal mass ejection, these particles thrown out from the Sun, struck Earth. This caused one of the largest Aurora events ever recorded. People as far south as Columbia were able to see the Northern Lights.
The energy put into the system was so powerful that the infant Telegraph system all over North America and Europe failed. Electrical lines were literally catching on fire. Telegraph operators were getting shocked as sparks flew out of their Telegraphs.
This is the kind of power the sun can put out and put into the system, and remember at the time this fledgling electrical grid - which was mostly used for telegraphing - was small, relatively self-contained, and didn't have a huge effect on life.
Imagine if this happened today.
[45:14] Immediately, we could lose 15% of all our satellites, while many others have their life expectancy shortened by 2 to 5 years. So that’s GPS, communications networks, you name it.
More importantly however, when these things slam into earth, it causes huge disruptions to the magnetosphere, the furthest boundary of our magnetic field. This impact compressed the magnetic field on one side, stretches it on the other, and when it snaps back, terawatts of power and energy come surging into the atmosphere, and into the ground following the path of least resistance.
And what better path than, I don’t know David, one made up of a bunch of metal wires?
[46:00] You mean the giant antenna system that we built that we know as the grid today?
Anyone who stood in a field during an electrical storm and ran for the lowest point should know that you don't want to stand next to giant metal structures, and that's exactly what this energy goes straight into.
[46:14] Remember we don't have grid storage; power that enters the grid has to be consumed or it destroys equipment. So this surge would obviously be much greater than our demand, meaning a lot of lines catch fire and a ton of transformers explode.
[46:28] An event like this could be global, or it could be localized like happened in the March 1989 geomagnetic storm in Quebec, where a much, much weaker flare than this Carrington class event hit Quebec, overloaded their power systems and threw the whole province into darkness for many many hours.
These events like this are not uncommon, they happen all the time. In fact last year we had radio blackouts and disruptions from a much smaller flare. In fact NASA estimates that a Carrington class event, one of these super power geomagnetic storms, has a 12% chance of striking the Earth every decade. Basically a 1 in 8 chance of civilization being destroyed every 10 years.
And we've had some very close calls before. In 2012 a CME that made Carrington look small in comparison narrowly missed Earth by a matter of hours, and if it had hit us the world today would look very different and maybe more like the Stone Age.
[47:19] I think it's pretty clear we're not prepared for something that has such a high probability of happening.
[47:24] Yeah I mean 12% - I was shocked when I read that - and there's a number of studies that has confirmed this number. Maybe it's just because making the grid resilient to an event like this is too expensive; it's impossible; something we can’t afford, and the fact that we do have a little bit of notice that this is going to happen right?
So we see the CME occur – we have daily probability measurements of exactly how likely to get a large flare come out - and then we know that is heading towards Earth, and we do calculations to see if we're going to be stuck or not, and we can maybe a 90 minute heads up, that's enough time to start rerouting power to shut down some grids in order to try and build in the excess capacity that we can handle this terawatts burst.
But if it happens when we’re paying attention, or caught off guard, or if this flare is just too big then that's game over.
[48:05] I'm just thinking like a 90-minute heads-up, out of that many utility companies, you call up I'm a utility company that serves all of New York City and you tell me I need to shut my power down because there's an X percentage chance a solar flares going to disrupt me?
[48:18] Maybe I choose to take that risk because I don't want to lose my reputation right?
[48:23] Yea, and what it's a dud?
Well on that depressing note maybe we should talk about what we can do to help with all these threats.
[48:30] On a larger scale as a society we should be aware of these risks and we should be moving away from such a crippling dependence on something that is so frail and so vulnerable. Right? The fact that the power can go out for just 12 hours and all of a sudden we have sewage that's infiltrating our water treatment, we have airports shutting down, we have traffic backing up, we have food spoiling and people going hungry… this is not acceptable right? And maybe the reason we've got to this position is that we live in an economy that does its best to maximize revenue and minimize costs. Optimized for efficiency in other words, and that has made us retreat from very sober ideas like redundancy in systems. Despite what our economic textbooks say we have to look at what's actually done in practice, and the reality is we don't like to prepare for risk. We have this mentality that “hey if we have money sitting in a bank account” right from the perspective of a Financial Manager, “it's a waste! We need to get that money out of the bank and in to the economy where it can be circulated, where we can loan it out, and it can grow and be productive.”
And we’ve let that idea trickle into just about every system that we depend on. We're operating on this “just-in-time how can we maximize every single bit of resource that we have and not let anything go to waste” and the result is that we have no ability to recover from and prepare for shocks to our system.
[49:56] And so designing ways to prevent these shocks in the first place, and then to be able to quickly recover are what we need to be doing.
So one on the utility scale we say “why aren’t we investing money into systems that are more resilient to this?” Why don't we have transformers spares? Why can't we standardize transformers in a way that we can quickly deploy them and have multiple options out there? Why aren’t we updating our gridlines when we need to? Why aren’t we building a more resilient system; in investing in this?
We're so dependent on electricity, on the grid is a whole, that it seems silly that these utilities companies and the government as a whole aren’t pouring as much money as possible into making the system modernized and resilient, both from natural threats – climate; foliage; things like CMEs; as well as man-made attacks, physical and otherwise.
On an even larger scale maybe we should start talking about how we can better decentralize these systems, so we’re not so dependent on these massive grids, on these enormous transformers, on the way that power is distributed currently.
By moving the grid back to each of us like we saw when the grid was first being developed, maybe we can begin to start offloading this threat. A solar panel and a battery pack in your house? Windmills and batteries in schools? These kinds of things can help make ourselves less dependent on the grid as a whole and make us more resilient to these attacks, both man-made and otherwise.
[51:11] And before you say “well hold up; I've seen some neighborhoods that have lots of solar panels on the roofs, it seems like we are going in that direction,” there are problems with some of these large-scale integrations of renewables that seem to be decentralized but in fact are not.
[51:26] Right so a lot of these installs are about generating power, selling it to the power company, and then the power company powers your house. And it's because it's cheap, it's easy, and the companies that install these things - companies like SolarCity - make money off this process.
They loan you out these things and say “well you'll get some money off this,” because it's complicated and expensive to wire house to run directly off solar panels, it’s easier to do it on one install, put the power back into the grid, and then you're good to go.
[51:50] Which means that if the grid fails and there is a blackout, you can drive home to your house that has solar panels on the roof, and you don't have power because that solar panel doesn't go to your house! It goes to the grid and you rely on the grid for that power.
[52:04] Right so moving to a process where we take that power, put it in our home, and we can be on a separate grid of our own control, of our own design, and then feed that excess power back into the main grid when we need to or we want to, well that might be something that we need to start moving towards as a whole in these new solar and wind installs.
[52:21] And as individuals, preparing for when the grid does go down because it will.
As we have been building these complex systems and relying more and more on automated computer systems, we're also seeing more and more significant power outages. Between the 50s and the 80s, there were less than five significant power outages per year, and now we're looking at over 300 as of 2011. So as individuals we need to think are we prepared for an event where the lights go out?
Do we have water stored away? Do we have food in storage that we can use when we can't go to grocery stores because that shipping network that brings in our food from really far away doesn't work right now?
[52:59] Do we have plans in place to communicate with loved ones to figure out where to meet them if the phones stop working, if the internet goes off?
All these things would seem sort of weird and maybe make you sound like a prepper, well you know it's not crazy. These things are going to happen; they likely do happen to you each year. But the next time, maybe soon, when the lights go off, they might not turn back on - at least for a while. You have to ask yourself: if that happens; when that happens, am I prepared?
As a larger question, is my community prepared? Are my neighbors good to go? Are my friends? Are my family?
These kinds of scenarios aren't just about being an individual, hunkered down in your personal bunker ready to shoot your neighbors when they come and try and take your food. No that's a fantasy that plays out. What you need to be ready for is how do we come together as a community to help each other and make sure that we all get through this together?
[53:45] Which is ultimately in our best interests as individuals because we're not going to be able to survive without the help of others, and food stored in the house, and guns and ammo and water that you keep in the basement might last you for a little bit but it's not going to be enough in the long-term.
We need communities. We need people taking care of each other.
As part of this mentality maybe we can start questioning the media and our politicians, and are bankers when they say “our economy is doing great we've grown by X percent this year,” we should ask them instead “okay well that's nice but given the fact that failures within our electrical grid has the potential to wipe out all the gains we’ve made in the economy – decades of growth just gone out the window – how are we doing to prepare ourselves for that?
Because metrics like preparedness, resiliency, redundancy in our systems, those might be the more important metrics to monitor going forward.
[54:39] Hang on Daniel I got to loosen my tinfoil hat a little here it's constricting my brain flow I think at least that's how I feel –
What brain flow?
That’s the bigger question.
But we are serious about all this. This all sounds silly again, and we know this is maybe the second time in a few weeks that we’ve gone off into what sounds like conspiracy sort of things, but this is not conspiracy this is about practicality and about making sure that you and your community are safe and protected, and you can trust all that if and when these things happen.
Don’t take for granted our civilization; don’t take for granted society; don’t take for granted the things that you depend upon but ask yourself “if this stopped working, would I be okay?” and that’s a good way to live your life.
[55:22] And that brings us to the end of another episode of Ashes Ashes.
If you want to find out more information about these solar flares; about attacks on our electrical system and the general threats that our aging infrastructure suffers, you can visit our website, ashesashes.org we have a full transcript of this episode and many links, sources, and more information.
A lot of time and research goes into making these episodes possible. We will never use ads to support this show. So if you enjoy it, and would like us to keep going, you can support us by giving us a review and recommending us to a friend.
Also we have an email address. It’s contact AT ashes ashes dot org, and we’d love to hear from you. Let us know what you think - positive or negative - and if you have any stories about power outages or ideas on preparing for something like this, let us know and maybe we can share that in an upcoming episode or on our website.
You can also find more information not just on our website but also on your favorite social media network. We're on all of them at ashes ashes cast.
We’ve got a super exciting episode next week that we really hope you tune into.
I know I'm excited about it, David’s excited about it.
Yep I'm trying to get healthy right now with this in mind.
It's going to be a good one.
Until then, this is Ashes Ashes.
Bye.
Bye bye.