The good data hall

I'm Julia Diaz. I'm a mechanical engineer and I have spent most of my career doing energy efficiency optimization work.

Started in buildings, did a lot of work in casinos, which are actually really similar to data centers. Surprisingly, they are 24/7 facilities. You lose a lot of money when they're down. And I did energy efficiency work at Meta on their data center team for about seven years.

I am Mo Elsayed. I joined Page about three years ago, right after my second PhD in engineering, and my background is in architectural engineering. Then I cross the gap to computational design and then another gap to automation engineering, which is basically autonomous integrated system.

So my PhD was on autonomous drones. So in order to work on autonomous drones, you need to leverage a lot of information, a lot of data and process them. And this is not viable through the old school classic computing. So that got me into machine learning and artificial intelligence. And that was around ten years ago.

It can be a lot of different things. Kind of trite answer is it's a house for your computer.

It can be big or small, big on the order of like a football field, small on the order of the IT closet in your office. Typically what you see are computers arranged in something called a rack, which is exactly what it sounds like. You've got your racks lined up in rows.

The traditional architectures that most people do now is something called hot aisle containment, where you have the computers facing each other so that the hot air from the back comes out into one contained space. And then there's a whole host of different ways to eject that hot air and add new cold air. Inside the data hall, you will see fans to move that air around, or you'll see water piping like chilled water piping, depending on how you're doing your cooling.

And then everything else is going to live outside of the data hall. So sometimes your cooling system is in a dedicated mechanical room, like what you would see in like a high rise hotel or something. You have a mechanical room. Typically they call them the penthouse. It's a very disappointing penthouse if you're expecting something bougie.

Or it'll live just outside, depending on your equipment. Sometimes it just lives outside. Like if you're doing an evaporative cooling, for example, you just have a big tank of water outside and that's the vast majority of your system. Chillers, a lot of times those will live outside. Cooling towers, those live outside. And this is what people see when they drive by a data center. You see all that equipment on the roof or you see all that equipment on the ground. A lot of that is your cooling system.

Artificial intelligence, by definition, is when we train the computer to do something and infer information in data sets, and it gives us a result or an answer that is not confined within the input. This is the moment when you say, okay, this is machine learning. The computer didn’t learn something.

To simplify it, the ChatGPT, the most well known large language model, it gives you a result based on learning from all the information online, right? So it has all this body of literature so it can give you an answer. It's now it has the logic to connect the dots and to generate answers in a myriad of topics.

So what we see on our side, the biggest change has been something called densification. 

If you want to think about energy per square foot in a building, a lot of the latest hardware that people use to run these models is about ten times more energy intense per square foot than what we saw even 3 or 4 years ago.

And so what you end up doing in the real paradigm shift that this is driving is it becomes really difficult to use air to cool something that's that dense.

So if you think about, like taking a step back, how long it takes to boil a pot of water versus how long it takes to run a space heater and heat up your room, air contains a lot less energy than water does. And so, with the less dense the old hardware, you could use air. You could move air through the rack. It would take all the heat away and everyone was happy.

And now with this 10x densification, what we see is that a lot of times we run into the limit of physics with air and how much energy that air can take away from the computer.

And so people are having to switch to using liquids like water to cool that hardware. And that drives a total redesign in how you make your building, because what's inside it has fundamentally changed.

Bringing that back to the technology that we're using, it has a huge impact. And this is the whole new idea that we're discussing that everyone is it's the million dollar question. Actually, $1 billion. Is this situation.

It is since the computing power has changed or the way that you're computing has changed. So the classic data center used to have a CPU, a central processing unit. The technology is jumping into GPU, a graphics processing unit, and those two different components have very different behavior.

At the end of the day, you're cooling a chip, that hasn't changed, but the way or the concept of cooling this chip has opened up other doors and other capabilities that would have otherwise not been viable only with CPUs.

So there is this question what is the most efficient way to cool these new data centers? And that opens up the door to mix your cards a little bit.

And the concept here is that you have this limited space, right, that you need to cool. And then how much can I reduce from the amount of air circulating through this confined space, or reduce the amount of water if it's liquid, to reduce the amount of energy that I need to cool the air or cool the liquid.

And then there is the hybrid. Okay. How about in certain months. So you have to think more about the locations where we are and how the weather patterns are. Do we have dry winters but humid summers? Do we have humidity throughout the year? Do we have dry days and humid nights like what's happening?

And depending on that you do all these permutations like, okay, let's try this system. No, let's try a hybrid of that system. No, let's try 30% of that system, but 70% of the other system.

And this gives us this amount of savings that we're aiming for. So it has become this kind of like equation of solving.

It used to be simpler to like tell the client like, okay, this is the best way to do this. But now we have all these sorts of methods and hybrid methods that we can propose. So these new technologies are changing the rules of the game.

Something that I want to pull out of that is the concept of efficiency as reduction, because I think that's really important.

It can be easy to say, oh, we’ll be the most efficient, we’ll be the most green by throwing some new technology at it or some whiz bang algorithm. And there's definitely space for that.

But I think at its heart, to me that's really what efficiency is. It's doing more with less.

So you were kind of talking about the two how it becomes this optimization problem. We always fight that tension mechanically in are you going to use more water or are you going to use more electricity.

And so waterless technologies are the 100% electric side of things. And typically they work the same way your refrigerator does, just at a much larger scale.

And then you have a couple of options. You can use that same technology to make cold air, which is what we've seen some folks do in the past.

More commonly, especially now with the liquid cooling, we see folks using that technology to make cold water. Typically, folks, it comes with something called an air cooled chiller, which is big. It's, it's like a big refrigerator, basically that makes you cold water.

And so depending on the outdoor air temperature and its conditions, sometimes they can be incredibly efficient. And sometimes they can use even up to half the energy of a data center, depending on what you're doing.

Something that we're very excited about with liquid cooling is that we see the working temperatures go much higher. So traditional data hall can use water up to maybe 68°F, which means for most of the year you're going to need to use electricity to make water that cold.

What we're seeing now is that some of these new chips can tolerate temperatures up to 40 degrees C, which is what, like 108, 110 Fahrenheit. It's very hot.

And so what you can do then is you can mostly not use electricity for the whole cooling process with refrigerant, and instead you can just kind of use a fan to cool off the water for like 99% of the year.

And that's super efficient. It's very efficient. That's something that people will call them or their fluid coolers or dry coolers. You see them a lot in other industrial processes and like refineries and food processing and that sort of thing.

And we're actually seeing now in a couple of crypto mining facilities, they will take the fluid that's directly touching the chip and just run it outside into that fluid cooler and bring it right back in.

And you can do this in a really hot place. We see people doing this in like West Texas because you just need to get that water below, like 105°F.

That you can do at most places in the US most of the year. And so that's something that has the potential to be a huge paradigm shift for the industry and just really use a lot less electricity.

But we don't see that a ton of places. Normally it's the refrigeration cycle.

Two big players that I'm familiar with, both Google and Meta, have these really ambitious carbon reduction goals.

And we see carbon in data centers in a couple different places. The main place we see something called embodied carbon, the carbon it takes to make the data center, is in the concrete.

Data centers use vast amounts of concrete, and it's incredibly carbon intensive to make concrete.

And so the main place that we see people, that I see people anyway, doing embodied carbon reductions is in the concrete itself, is looking at different mixes, looking at ways to make it lower carbon, and people actually use it for carbon capture sometimes. They'll pump CO2 into the concrete to have it in there as it cures.

There's a whole bunch of stuff happening there.

And then in terms of the operation, mostly we see people focusing on their electrical sourcing. There's a couple of different ways you can do that.

A lot of folks will say they're 100% renewable powered. There's a lot of different things that can mean.

And Google in particular is really trying to be 100% renewable powered 24/7, which is a totally different ballgame than just saying, hey, we've you know, we have enough solar panels that over the course of a year, they generate the power we're using.

Google is saying for every point in time I want all of the electricity going into my data center to be generated by a non-carbon producing source, which is totally different.

And so those are the big trends that we see.

I think it depends on your perspective, Tully. So, let's put it this way.

You are sitting on your couch, having your phone in your hand, and you are unaware where is the energy that is being consumed or the carbon dioxide that is being emitted on behalf of your very, very small phone that you have in your palm of your hand, and you're doing a simple process on or like asking a question or like putting in a prompt to ChatGPT.

So to reel that back in, why do we have to care and why the sector of data centers, data halls under extreme scrutiny from people and from the media and from everyone thinking about that?

It's because it is both intensive in two ways.

The first way, which Julia mentioned, was the size, the sheer size of these data halls. As we rely more on these systems and these AIs.

And the other side of it is that we also need to cool these systems. So this is the ongoing process. The energy that we consume to call these systems requires a lot of resources.

And then that means more carbon dioxide. So this is the other part of the equation here.

So two sources of carbon dioxide. The building itself and then the processing.

And then as we rely more and more on these AIs and cloud computing, we rely more on these data centers. And which means that the service providers they are being these big emitters of carbon dioxide.

And it's not that they want to, it's just the demand is so high. You have to build more data centers and more data halls. And that means way more carbon dioxide emitted.

So this is why we need to think about ways and methods to reduce that.

So the first way is materials which is like carbon that is embodied into concrete. So recently we had a data centers that were built out of mass timber. And that reduces that kind of carbon.

But then the second portion of it, which we're talking about today, which is like the methods of cooling and that also reduces that carbon footprint significantly.

Yeah. So designing for both expansion and retrofit is a huge part of this.

So I want to talk about retrofits first. So if you think about going back to that cell phone example, if you think about the cell phone you have now and the cell phone you had five years ago, they're very different. Right.

And so the same thing happens to the hardware inside of a data center. And so over time, what we see is that we need these buildings to be adaptable, because if we go back to that embodied carbon of the building, tearing it down and building a new one every five years is incredibly energy consum- you know that just that's so wasteful versus being able to take what you have and retrofit it.

I mean, the most energy efficient building is the building you already have, right? Because it takes so much to build a building in the first place.

And so we see most folks when they build a data center, most owners, design in the ability to retrofit, you know, and that can mean a lot of different things.

Typically it's very tactical. It's like adding extra ports or adding extra space or, you know, you can do a lot of things on the electrical side to make it flexible so that if you need to add more equipment later, you don't need to, like go buy a new substation.

So that is something that folks do a lot. And that's a big piece of reducing people's carbon footprint because once it's built, getting the most out of it is huge.

Kind of tied to that concept of retrofit ability is expansion. You know, when you build a data center, typically you're not just building one on a little island, you're typically building a whole campus of them.

And so you think about how they're all going to work together and expand. Pretty much every data center I've worked on, they have extra land or they have extra power, they have extra water.

And so from day one, the conversation is, okay, we're going to build one. And then when we build the next one, what will it look like? What do we need? How do we ensure that we don't need to like re-dig a pipe or rerun electrical wire.

So it's something that folks typically just do from the beginning.

Yeah. And I think this is directly related or tied to the fact that the more we automate, the more we introduce computing to industries that never had that, never thought about it, the more we will need.

So this is why when you start a data hall, you're not thinking, this is the only one that I'm building. No no no no, you have to have like a fresh, nice, large piece of land and then you start having it as a modular design, right?

So it's like, okay, we're going to repeat that 100 times. And this is even the short term expectancy.

Can you imagine that a process that used to be so bespoke, such as AI, few people worldwide used to know or actually care about that, and now a company like Apple or a company like Google or a company like Microsoft, they are switching our entire system, things that never needed that, into AI.

So with the new Apple, the new iPhones, Google services, Meta is doing that. It gives more to the customer, to the clients, and it gives more capability.

Like who wants the normal emoji when they can have a customized emoji to encapsulate the moment. So that really comes into play. And like okay, this is this is going to sell well. Right. So we need to incorporate it in every single one out there, every single phone.

So that automatically overnight we didn't multiply that. We actually I don't know how like what's the what's the mathematical formula to this is like X to 1 million, 10 billion, 1.1 billion the demand that you had yesterday.

So the expansion is embedded. It's baked in the DNA of data halls. They are born with the future in them.

Oh yeah. No one has any idea what's coming.

Yeah, the uncertainty here is the technology that we're dealing with is called disruptive technology.

This is the kind of technology that we do not know or can't anticipate how much we're progressing.

And the anecdote here is that with the introduction of open large language models and AI to the wider adoption, it's even now leaping faster than it used to be.

So before the AI, we kind of expected that the efficiency of the processor chip is going to double every couple of years. Right? So from the Intel eighth generation to the 9th, 10th, like we could actually predict, we could draw a line on a curve and say, okay, this is where we're going to be in like ten years.

But now, because there is this new thing that is AI and machine learning, the disruptions has been even a faster increment. So it's like we don't need to wait a year.

So in the matter of, I think maybe six months, we had GPT 2.5, 3, 3.5, 4, 4.0, 4.1 like it is just every few weeks we have something new and we have new capabilities, and the industry is discussing new chipsets or new capabilities.

So it's even catalyzing the process of the new technologies. So we have absolutely no idea what's going to be there, not even ten years. We're talking five, four.

Oh, 18 months.

Yeah, yeah. The new Smart USA initiative by the, the NIST, it has the timeline of 36 months to come up with a backbone for digital twins. 36 month.

This is how cramped we are on a timeline. And we need to get something new out there.

Like you can extrapolate from that how disruptive this technology is going to be.

What we see in the built environment is data centers have always had this tension, and we talked about it a little bit between the lifespan of the building and the lifespan of the hardware. Right?

I mean, you build a building, you want it to last, you know, 20, 30 years. That's a reasonable timeline for a building like this.

If you're putting new hardware in there every 18 months and you don't know 18 months from now, what it's going to be is hugely problematic for people because they want to make sure that these investments, you know, like it takes 18 months to build a building.

And that's pretty fast for a data center. If you break ground tomorrow and then 18 months later, you've built something that's useless, that's a huge waste.

And so we see people doing all sorts of things to try and account for this, this disruption. And I don't think anyone's figured out a magic bullet answer, but it's definitely something that is top of mind for people every day.

Something we've seen that's been really interesting is some of the 20 year old data center designs, because they were built before we were really, really good at air cooling, have actually proven to be more flexible, and we've seen them kind of come back into vogue over time with this new hardware, because 20 years ago, people had to design for flexibility the same way that we're having to today.

So that's been really interesting to kind of go back in the archive and see like, oh, well, this worked. Maybe we do this again.

Even though kind of fell out of favor, it wasn't the most efficient thing for the past like five, ten years.

That's the same mic drop that, it has the same effect as we are saying that today in 2024, we are recommissioning decommissioned nuclear plants in Pennsylvania.

Exactly. Yeah, it's been interesting just to kind of go back and see the designs that have really stood the test of time and say, oh, maybe these are a good fit for the next round of disruption.

This is a new chapter for everyone, actually, from policy makers to clients to engineers and architects. Everyone's asking, how can we cater for this high demand?

It's for two reasons. The first one is that power grids are not linear in their behavior, meaning there is a point in time where you have extreme high demand, and then there are points of time where you have lower demand.

So the same is happening with the data halls that we cannot predict when you are going to have this extreme spike in energy consumption and the other part or the auxiliary part to that, to understand why we thought that this is linear, like this used to be the case with the CPU technology, with the last generation of data halls, the new generation of data is based on GPUs.

And the difference here is that CPU they run even on idle, they are running on their same energy demand. You can basically average it to a flat line, so you can expect how much energy it's using.

Now with the GPU, that is a different story. The GPUs, they are like a race car that basically sometimes goes into like the fifth gear, sixth gear, and they're like running fast, blazing hot, fast.

And then when you're not needing them, when you're not requiring extreme energy from them, they go back into idle. And idle does not use as much electricity.

With this extreme change and swings in the energy, our normal city grid does not have that kind of flexibility when it comes to catering for demand.

So what you will see is that you will see blackouts. You will see parts of the grid will shut down because we cannot cater for that. We do not have enough power.

So where are the sources of energy and how can I offset that with something that is flexible and green? We have hydro and then we have solar, which is pretty good.

But it's also not that flexible given that you're depending on the sun. And then the third which is wind power. Yes. You need like a huge wind field to be able to generate enough electricity for that.

So with this we're thinking again about something that generates so much energy, such as nuclear power. And anecdotally it is green. Nuclear power does not have much carbon footprint per se.

It has other challenges, such as getting rid of the radioactive material and the risks of operation were even seeing now new technologies similar to the one that you have in a submarine, basically, you have your own small nuclear generator that powers the data.

So whether it's a small nuclear generator or a big nuclear generation plant, all of these are just we're thinking about ways and methods that we can offset this generation of energy can offset this amount of energy without having such a huge impact.

Yeah. And to add to that, so I am in California. We've had an unstable electric grid for like years and years and years.

And what we see is that as we add more renewable sources to the grid, it just becomes more and more unstable.

And you need to make these really fundamental changes to how the electric grid is designed and how it runs in a way that is much bigger than data centers.

I mean, it's electric cars, it's decarbonization, it's old infrastructure, to be honest, that we should be replacing anyway.

And it becomes this much larger conversation that I think data centers have been kind of the canary in the coal mine for, because they are in easily identifiable new, large sorts of power consumption.

I mean, something like half of all new cars I think, sold in California last year were electric. There's huge there's a game changer in terms of the electric grid, but because it's individual people, it's much harder to be like, oh, well, you're electric cars causing grid instability.

It's much easier to say, oh, well, it's the data center down the road that is having problems.

And I think that also provides a really great opportunity. Like there's so much money in data centers, there's so much money in these new technologies.

And a lot of the big hyperscalers have, to their credit, really push the envelope forward already when it comes to things like getting more renewables on the grid, providing a guaranteed market for new technologies, that's a huge thing, right?

If you have some experimental technology, you know that the AWS data center down the road is going to be a happy customer of whatever you can generate, and that really helps these new technologies bloom and grow in a stable environment, dependent of what's happening politically, which is hugely valuable right now.

And so I think it's both a big challenge and also a really cool opportunity that we have to look at things like, you know, grid harmonization, kind of going back to what we were talking about earlier, how the grid is designed to kind of do this steady up and down with these new AI loads, I mean, I've seen some megawatt jumps in power.

That's huge. That's like turning on, I don't know, a thousand houses at once, from 0 to 100% in a minute.

And so if there's a time in a place to really talk about how do we change the way that we interact with the electric grid, these data centers are a huge opportunity to talk about how do we make the grid smarter, how do we interact with the grid?

It already happens at a small scale, depending on where you are. For example, my house, I get charged every 15 minutes for power based on how many other people want power. My parents in Western Maryland, a person comes to their house once every couple of months and writes down how much power they use.

There's a huge gap in terms of the data we have in our ability to interact with the grid depending on where we are in the US specifically.

And I think this is a huge opportunity to have the money and have the drive to really make those upgrades everywhere and get us into that dialog with the grid in a way that will change everything, not just data centers.

It'll change, you know, it'll let people plug their cars in and have their cars feed back to the grid when the grid needs power.

And, you know, there's a lot of places you can't do that right now because you just don't even know what's happening.

And so I think this is a really golden opportunity that we have to think about what we want the next version of the power grid to be and how it can move and change with us as a society.

That's a great question. And I think it's very topical because was it that Azure outage recently that took down, like all of the US's airlines?

A small code that went wrong. So can you imagine the damage just because part of like a small outage in an area, it just it falls. It's a it's what they call the cascading impact.

You have one piece of of chess that falls and the entire board is impacted worldwide.

Yeah. There I mean they're they've become part of our critical infrastructure, I think in a way that no one really took a step back and realized until it was too late.

We can't go back now. We're not going back to mainframes and people's offices. And that was a brittle system, too, in a different way.

And so large scale data center outages or instabilities that cause even corner cases of data center outages, depending on what's running in those data centers, can be hugely catastrophic.

Yes, especially that you have some services that cannot be taken down, like medical facilities.

The more compelling part here is that we're seeing AI seep into all of these subnetworks.

So Health Network is utilizing AI, and rather than having every process on their machines, on their local machines. No, they're now part of a cloud.

So imagine going to the doctor and they're trying to pull out your file and they have no idea what's your medical past.

So there are some, or like an airliner that is boarding people onto onto our flight. So these are critical or the banking from like Chase Bank was down for like ten hours last year and it was catastrophic.

No one knew what they're like. The money, where is it going? How can I don't? And they're talking about like a million transactions every few minutes on their credit cards.

So it's quite a lot. And any downtime is going to be catastrophic. So that's why it's like of utter importance to have these running.

And under no circumstance would these data centers be down even for a minute.