1、NA-04-4-3 Liquid Cooling-Friend or Foe Donald L. Beaty, P.E. Member ASHRAE ABSTRACT Liquid cooling is NOT a new technology and has proven to be an efective means of cooling, especially high-density loads. With the rapid growth of heatflux densities at the chip, board, and rack level, why does there
2、appear to be such reluc- tance to use liquid cooling for electronic equipment? The focus of this paper is not to promote or contest the increased use of liquid cooling but rather to examine the advantages and disadvantages of liquid cooling. This includes reviewing thesituation from NOTONLYa thermal
3、 engineering viewpoint but from the end user as well. Some often believe that to even propose liquid cooling to un end user can be a fatal mistake andpotentially the end of the relationship. This paper is focused on examining the controversy including perceptions and risks associated with liquid coo
4、ling. INTRODUCTION Load densities are continuing to rise, resulting in increased high-density cooling needs. From a practical ther- mal engineering standpoint, a natural consideration should be liquid cooling. However, the reputation and ultimately the acceptance of liquids in a datacom center is li
5、mited. This paper provides an overview of the issues and insight into whether to consider liquid cooling for high-densiy loads. Although it is beneficial to provide this information to IT manufacturers, cooling equipment manufacturers, facility design engineers, and construction engineers, a critica
6、l focus of this paper is on the stakeholders (e.g., end users, project sponsor). Many datacom stakeholders are reluctant if not passionate about NOT using liquid cooling (the term datacom means data centers and telecom facilities). For the purposes of this paper, liquid cooling is defined as cooling
7、 meda connected directly to the equipment it serves being LIQUID. This equipment may be the actual individual rack-mounted components (servers, etc.) or it may be a liquid- based cooling system that has been integrated within the enclosed rack itself. The scope of this paper is to discuss the distri
8、bution of the liquid cooling media to the equipment (rack) and not the distribution of cooling within the enclosed rack or the component equipment. The term rack has different definitions in the telecom industry versus data centers; for the purposes of this paper, the broadest definition for rack wi
9、ll be used, which is “an open frame or an enclosed cabinet that houses electronic equipment.” Water leaks and other failures in datacom rooms are seldom publicized or reported since they bring unwanted attention to a failure and can damage the image of people, firms, or facilities. There is no pract
10、ical way to establish the root cause for the limited use of liquid cooling in datacom facilities, but certainly one of the main reasons has nothing to do with cooling but rather the presence of any water or liquid near electronic equipment. Therefore, a major focus is to examine the risk of liquid i
11、n a datacom room. Although IT manufacturers thermal engineers prefer liquid cooling for certain applications, they are not using liquid cooling systems due to lack of customer acceptance. In this competitive market, it is not easy for a manufacturer to design and produce the same equipment in both a
12、ir cooled and liquid cooled versions. Since all customers will accept air cooling while many are reluctant to accept liquid, the current de facto standard is air cooled and, consequently, the majority of facilities are not designed to deploy a liquid cooling solution at any scale in a cost-effective
13、 manner. Donald L. Beaty is president of DLB Associates Consulting Engineers, P.C., Ocean, N.J. 02004 ASHRAE. 643 This is quite a transition since in 1987 an article in a major magazine identified the mainframe watercooled product as 92% of the market. The vast majority of the most powerful mainfram
14、es and supercomputers in the 1980s were liquid cooled. To the best of this authors knowledge, only one current supercomputer manufacturer actively ships their equipment as liquid cooled. There are third party manufactur- ers, predominantly in the rack manufacturing industry, that have developed encl
15、osures that require piped liquid connec- tions to them in order to recirculate chilled air to cool the envi- ronment only within the confines of the enclosure. Other critical industries, such as the biomedical, pharma- ceutical, and food processing industries, regularly use chilled water for equipme
16、nt cooling. The system includes no-leak quick connect fittings, simplified configuration, and simpli- fied troubleshooting. Therefore, although the computer indus- try has been relatively inactive in liquid cooling, other applications continued to promote its refinement and contin- ued improvement.
17、This paper is organized into the following sections: Managing Liquid by Design. This section provides some insight into the strategies that are used when designing a facility or portion of a facility to manage the risk of liquids. This section is focused on liquid in gen- eral and not specifically t
18、he media related to liquid cool- ing. It demonstrates that managing liquid such as roof drainage piping, sprinkler piping, and air-conditioning piping is already commonplace in datacom rooms. The point here is to not look at liquid cooling in isolation but rather to view it in a similar way to other
19、 liquid systems such as makeup water to humidifiers, condenser water piping, storm piping, sprinkler piping, etc. Some Information about Liquid Cooling. This section is not intended to be comprehensive but just provide a cursory overview for liquid cooling systems including some brief history. Altho
20、ugh the scope of this paper does not include examining how liquid cooling is accomplished within the electronic equipment, this sec- tion does include an introduction to the material in the bibliography that does address cooling within the pack- aging of electronic equipment. Managing Liquid Cooling
21、 Systems by Design. Unlike the first section, which focused on the general topic of liquid, this section focuses specifically on liquid cool- ing. There are many ways of organizing and designing a liquid cooling system; as with all choices, there are tradeoffs in cost, flexibility, reliability, etc.
22、 Conclusion. Liquid cooling is not the solution for all situations and may never be an acceptable solution for certain stakeholders based on their personal preferences. However, with the load densities increasing and approaching the thermal limitations of air cooling, liq- uid cooling can be a solut
23、ion whose cost I benefit I risk metrics proves to be a viable or even leading alternative. MANAGING LIQUID BY DESIGN Liquid escaping and landing on electronic equipment is an unacceptable condition. Certainly it is very desirable to completely avoid the presence of liquid in a datacom room (especial
24、ly directly above electronic equipment) but some- times it is simply not practical to do so. Some of the potential sources of liquid include: Coolant or water piping to cooling equipment (e.g., con- denser water, glycol to dry coolers, chilled water, refrig- erant) Makeup water for humidification sy
25、stems Condensate piping from cooling equipment Plumbing piping (such as potable water, hot water, and sanitary sewer that may feed non-datacom uses on the same floor or the floor above) Storm piping (piping from roof drains to the disposal system) Roof leaks Fire suppression systems The first step i
26、n managing the risk of liquid is to look for practical ways to route the piping and other sources of liquid so that they are not in the vicinity of critical electronic equip- ment. The second step is to select quality materials and construction techniques and equipment that reduce the prob- ability
27、of failure. Equally vital is to aggressively commission the installation to achieve maximum quality of installation and performance, thereby further minimizing risk. The third step is to incorporate containment or diversion. One form of containment is to use gutters, pans, or other forms of directio
28、nal diversion. The other approach is to use double- wall piping similar to that used for fuel oil piping (if the inner pipe leaks it is contained by the outer pipe, plus the presence of liquid in the annular space between the inner and outer pipe is monitored and alarmed). The fourth step is to add
29、monitoring and alarming such as monitoring pressure, flow, and presence of moisture. The fifth step is to include means of isolating sections of piping to effectively deal with leaks, changes, maintenance, etc. These means are typically control intensive through the use of automated valves and alter
30、nate flow routes and may even incorporate automated drainage in the event of a leak. Figures 1 to 9 are photographs of various datacom facil- ities throughout the country (including those dedicated to the telecom industry). These figures demonstrate that the pres- ence of liquid piping in datacom ro
31、oms is common. Some conclusions are: The preferred approach is to route piping as far as prac- tical from above electronic equipment, typically around the perimeter of the room. 644 ASHRAE Transactions: Symposia Figure I Storm piping example. Figure 2 Sprinkler piping example. Figure 3 Piping direct
32、ly over racks. Figure 5 Liquid diverted away from rucks. Figure 4 Piping in close proximity to rucks. Figure 6 Containment pun example. .Containment pan under horizontal air handler ASHRAE Transactions: Symposia 645 Figure 7 insterstitial roof example, Figure 8 CRAC unit piping overview. Sometimes,
33、circumstances involving rapid deployment or maximizing the datacom room rentable area results in piping located above electronic equipment. Gutters, pans, and diverters are used and reduce the risk. 8 It is uncommon for individuals or organizations to publi- cize their failures so little is availabl
34、e describing liquid systems leaks or failures in datacom rooms-but they have occurred. They include roof leaks, storm drainage piping fail- ures, floods from other floors, pipe failures, human error, etc. These types of problems are impossible to completely elimi- nate, but it is the responsibility
35、ofthe design team and the oper- ations staff to agree on the allocation in terms of the amount of effort and associated cost to minimize the risk. Since piping is not new to a datacom room nor is manag- ing the risk, the use of liquid cooling should not be eliminated from consideration simply becaus
36、e it is a liquid. Liquid cool- ing should be scrutinized the same as any other liquid system or source of liquid in a datacom room. In fact, as the loads continue to rise, so does the challenge of cooling with air due to the limits of heat sink surface area and contact time. Liquids have superior th
37、ermal characteris- tics to air, making liquid cooling a viable choice as the concen- tration of load continues to rise. Under high-density loads, the effort or measures to successfully cool with air cooling could become extreme to where the risk of failure is far greater than that of liquid cool- in
38、g. Therefore, it comes back to the fundamental challenge of balancing the advantages and disadvantages of any system considered, including its cost, ease of maintenance, flexibility, reliability, etc. SOME INFORMATION ABOUT LIQUID COOLING Definition of Air Cooling and Liquid Cooling The most common
39、source of cooling to electronic equip- ment from the datacom room / facility is chilled air. Chilled air is delivered to the air inlets of the electronic equipment Figure 9 CRAC unit piping connections. through an underfloor, overhead, or local air delivery system. The effortless availability of air
40、 lends itself to an easier appli- cation as the cooling media for electronic equipment. Enhancements in designs to control airflow paths and the contact time with the heat exchanging device attached to the chips themselves have all helped to increase the efficiency of an air-cooled system. For liqui
41、d cooling systems, instead of utilizing air inlets in the face of the equipment packaging, pipes or hoses are physically connected to electronic equipment or the enclosed rack. In some cases, electronic equipment will have a combi- nation of air and liquid cooling which means there will be air inlet
42、s and liquid connections. Fundamentally, the two major types of liquid cooling media available for datacom facilities are water and refriger- ant. Additional technologies, such as direct immersion or liquid spray impingement, utilize dielectric (nonconducting) fluids that would be in direct contact
43、with the actual chips. For simplicity, most of the liquid cooling discussion in this paper is focused on the liquid being water, since that was the most common source of cooling for major computer equip- 646 ASHRAE Transactions: Symposia ment in the past. In fact, as mentioned in the Introduction, a
44、 1987 magazine article identified the mainframe water-cooled product to be 92% of the market with technologies such as water-cooled cold plates and aidwater hybrid cooling methods also being used. There does not appear to be much in the way of formal definitions of air cooling and liquid cooling; mo
45、st definitions focus on a component of a cooling system such as an evapo- rator, condenser, or compressor. For the purposes ofthis paper, the definitions are: Air cooling - chilled air is produced and is the direct source of cooling at the point of use. For example, the heat within the packaging of
46、equipment in a rack is removed by chilled air. Liquid cooling - chilled water is produced and is the direct source of cooling at the point of use. For example, the heat within the packaging of equipment in a rack is removed by chilled water. However, it is the dissipation of that transported heat th
47、rough the use of forced convection that is now rapidly approaching the thermal limitations of air as a cooling medium. The air-cooled measures that are on the verge of being exhausted have resulted in larger and noisier internal fan assemblies and larger (albeit more sophisticated) heat sink designs
48、 due to the necessity to increase the surface area of the heat sink to aid the cooling process. Liquids have more desirable thermal conductivity and specific heat ratings, resulting in lower flow rates required to extract larger amounts of heat away from heat sources within the equipment packaging.
49、Instead of blowing air across a heat sink, it is much more efficient to have liquid filling the cham- bers of a cold plate at the location of the heat transported from the CPU. The liquid cooling issues being addressed by this paper predominantly deal with the supply and return of that liquid to the packaging. Some General Commentary about the History of Liquid Cooling A critical issue with air cooling is the performance of the cold aisle including the static and dynamic interface between the cold aisle and the equipment air inlets. Liquid cooling is not dependent on the cold aisle. Furthe
