ASHRAE OR-16-C036-2016 Data Center IT Energy Recovery Satisfying ASHRAE W2-W5 Liquid-Cooling Classes.pdf

1、Tahir Cader is a power and cooling strategist with HP servers. Roy Dragseth is a section leader and is responsible for the HPC services at UiT The Arctic University of Norway in Tromso, Norway. John Peterson is a technical program manager for HP in Takoma Park, MD. Data Center IT Energy Recovery: Sa

2、tisfying ASHRAE W2-W5 Liquid-Cooling Classes Roy Dragseth, MSc Tahir Cader, PhD John Peterson, PE Member ASHRAEABSTRACT HEADING Recapture of the waste heat from data centers remains a difficulty undertaking. For liquid-cooled data centers, the temperature of the water has been too low to make it eco

3、nomically feasible to put full-blown heat recovery systems in place. The University of Tromso (UiT), in collaboration with an IT manufacturer, has undertaken a detailed study to investigate the technical and economic feasibility of capturing the waste heat from its current data center. A single rack

4、 of servers has been converted to liquid-cooling (CPUs only). The rack was tested using inlet water temperatures that satisfy ASHRAE W2 W5 liquid-cooling classes. Under all inlet water temperatures investigated, the key IT device temperatures remained well within specification, while producing rack

5、exhaust water temperatures as high as 147.2 degrees F (64 degrees C). Using the current UiT cost of energy, and assuming that 50% of the waste heat from a 1 MW data center is re-used, there is the potential to save over 375,000 Euros annually. As UiT is building out a new 2 MW data center, the poten

6、tial for greater heat recapture can greatly expand the re-use of the data center waste heat to the broader campus. INTRODUCTION ASHRAE is aiming for buildings to achieve net-zero energy use, including energy-intensive mission critical facilities, by the year 2030. Often data center heat is considere

7、d a bane, and energy recovery from the exhaust water source is not an option because the heat rejection and distribution is limited by exhaust water temperature, along with other physical and monetary constraints. However, opportunities are becoming more available for new and existing facilities as

8、data center densities and water cooling temperatures move steadily higher. In 2011 ASHRAEs Technical Committee (TC) 9.9 Mission Critical Facilities, Technology Spaces, and Electronic Equipment released a whitepaper entitled Thermal Guidelines for Liquid Cooled Data Processing Environments to propose

9、 five (5) classes of water-cooled IT (Information Technology) equipment, based on entering cooling water temperatures. The 2011 paper depicts re-use of data center heat, but for the higher temperatures the heat re-use was considered largely theoretical. This paper shows the actual results of liquid-

10、cooling IT equipment with supply water temperatures from the W2 to W5 ranges. Testing was performed in a 500 kW data center to understand the percentage of heat rejected to the liquid-cooling system at various IT performance loads. Using the results from the quantification of percentage heat rejecte

11、d to the water, a simple payback analysis was conducted in order to estimate the potential monetary savings from the re-use of the waste heat captured from the exhaust water stream from the IT equipment. Air Handler Campus Building Heating Plant Data Center RadiatorSpecialty Equipment Air Cooling Wa

12、ter cooled RacksHeat Rejection ASHRAE LIQUID-COOLING CLASSES ASHRAEs thermal guidelines for liquid-cooled data processing equipment consists of increasing temperature ranges as shown in Table 1 ASHRAE 2011. Note that the facility water is for the water supply to the IT equipment. At the higher tempe

13、ratures, the exhaust water temperatures begin to be more viable as a stand-alone heat source for building heating, pre-heating hot water, etc. For the results discussed in this paper, the supply water temperature begins at 68 degrees F (20.0 degrees C) and increases in increments of 18 degrees F (10

14、.0 degrees C) up to 122 degrees F (50.0 degrees C). These measurements span the liquid-cooled classes of W2 to W5. For the heating systems, the temperature of the water returning from the data center equipment is of more interest - for this work, the achieved range was 104 degrees F (40.0 degrees C)

15、 to 149 degrees F (65.0 degrees C). Since higher temperature liquid-cooling can be used to more efficiently allow for the re-use of data center heat, a minimum lower exhaust water temperature should be established for a design before it can be considered a viable heat source. UNIVERSITY OF TROMSO CA

16、MPUS HEATING SYSTEM Figure 1 shows a simplified arrangement of the campus-wide heating system at the University of Tromso. The figure shows the connections between the IT equipment, the cooling plant, heating plant, and the hot water heating system supporting a campus building. Also shown is a suppl

17、emental heating plant that will be used to boost the heat supplied by the data center; at present, the campus needs approach 10 MW, while the data center is slated for 2 MW at the outset. Indirect heat transfer via liquid-to-liquid heat exchangers (L2L HX) protects each system from the other, and al

18、so prevents equipment damage as a result of water quality issues, flows, temperature swings, etc. In addition, the L2L HXs allow for flow and pressure de-coupling of any two water loops, thereby allowing for variable performance of each loop. TEST SET-UP TO QUANTIFY WASTE HEAT RECOVERY The Universit

19、y of Tromso (UiT), located in Tromso (Norway), is aggressively pursuing liquid-cooled IT and the re-use of the waste heat from its data center. UiT currently has a small number of liquid-cooled racks that it is using to better understand how to optimize its design for an upcoming 2 MW data center. I

20、n May-June 2015, several tests, at different input Table 1: ASHRAE Liquid Cooled Guidelines Liquid Cooling Classes IT Equipment Supply WaterTemperature Range (F/C)W1 35.6-62.6 / 2-17 W2 35.6-80.6 / 2-27 W3 35.6-89.6 / 2-32 W4 35.6-113.0 / 2-45 W5 133.0 / 45 Figure 1: Basic schematic diagram of the d

21、ata center supporting heating system Figure 2: Liquid cooled server; liquid cooling is provided for the CPUs, (other components) Heat Exchanger RackWater distribution system Pump water temperatures and workloads, were conducted to measure the quantity of waste heat that can be captured from servers

22、with liquid-cooled CPUs. It is important to note that the server manufacturer has agreed to engage in this R HPL is a suite of benchmarks test used to stress IT systems in order to gauge their performance. Following are the main variables used during the testing: IT workloads: idle, 50%, 80% and 100

23、%;IT inlet air temperature: 68 degrees F (20 degrees C);Water inlet temperature:o 68 degrees F (20 degrees C)o 86 degrees F (30 degrees C)o 104 degrees F (40 degrees C)o 122 degrees F (50 degrees C);Water flow rate: fixed at 2.3 GPM (520 L/hr);Fan speed: 10% and 20% of max speed.The set-up was adequ

24、ately instrumented to capture all key power consumption values, air and water temperatures, water flow rate(s), etc. All IT data was captured using a utility supplied by the IT equipment manufacturer with data for the key devices reported in the results section. Figure 3: Liquid-cooled racks in the

25、data centerFigure 4: Schematic diagram of water distributionFigure 5: Plate frame heat exchanger to move heat from the IT system to the campus heating system Recapturing Waste Heat The city of Tromso is located at the 70 northern latitude and the Atlantic Ocean Gulf stream gives it a coastal climate

26、 with mild winters and somewhat rainy summers. The average yearly temperature is 39.4 degrees F (4 degrees C) with a general temperature range of 14 degrees F (-10 degrees C) to 77 degrees F (25 degrees C). UiT has detailedweather statistics for more than 20 years (http:/weather.cs.uit.no) and analy

27、sis of the collected data shows an average outside air temperature below 53.6 degrees F (12 degrees C) for 90% of the year, and above 77 degrees F (25 degrees C) for less than 9 hours a year. Furthermore, the campus has a need to heat its buildings when the temperature dipsbelow 53.6 degrees F (12 d

28、egrees C); this provides a unique opportunity to reduce the cost of the data center operations by capturing the waste heat from the IT equipment. The current heating infrastructure on campus requires a supply water temperature above 122 degrees F (50 degrees C), with a return water temperature of ab

29、out 104 degrees F (40 degrees C) indicating the need to run the data center with supply water temperatures in the W4 class or above. A new 2 MW data center is under construction and will be situated on campus, so all of the waste heat from the data center can be used for at least 10 months out of a

30、given year. The large amount of energy that can potentially be captured and delivered will exceed the needs of the datacenter building, so the plan is to distribute the excess energy across campus to other locations through the universitys own heat distribution system. The anticipated shift towards

31、water-cooled systems has resulted in data center heat recycling being included into plans for new buildings on campus. For instance, floor heating will have lower input temperature requirements (W3) than traditional radiator-based heating systems (W4-W5). UiTs stated goal is to achieve at least 80%

32、recycling and re-use of the waste heat created by the new data center. A key objective for this study is to demonstrate the ability to accomplish this 80% goal without introducing heat pumps or other costly heat conversion infrastructure. This makes a direct economic benefit from data center heat re

33、cycling much more viable and makes it possible to expand the HPC capacity without any significant impact on the total energy budget of the university. In addition, the Norwegian summer temperatures do not require any significant use of air conditioning equipment in private and public buildings, resu

34、lting in a significantly lower power price in the summer months when there is a reduced need for heating the buildings. RESULTSWhen designing a data center to maximize the re-use of the waste heat from the IT equipment, the percentage of heat rejected to the cooling water and the IT rack exhaust wat

35、er temperature are two key variables that should be maximized. The focus of this work has been placed on these two variables. Figures 6 and 7 present two charts that show how the “% of Heat (Rejected) to Water” varies as a function of cooling water temperature and % of maximum Figure 6: Percentage o

36、f heat rejected to water versus percentage of maximum IT power at 10% fan speed Figure 7: Percentage of heat rejected to water versus percentage of maximum IT power at 20% fan speed Idle Idle Table 2: Percentage of heat rejected directly to water for 10%and 20% fan speedspower. The % of heat (reject

37、ed) to water is defined as the ratio of the “heat captured by water” to the “total rack power”. The “heat captured by water” is quantified via an energy balance over the water entering and leaving the rack, while the “total rack power” is measured at the % CPU utilization to the rack power at 100% C

38、PU utilization. Each % of Max Power point, going from lowest to highest, corresponds to the following % CPU utilization values: idle, 50%, 80% and 100% (also discussed in the Tests Conducted section). Figure 6 shows data for the chassis fans running at 10% of maximum speed, while Figure 7 shows data

39、 at 20% of maximum fan speed. All tests were conducted at an inlet air temperature of 68 degrees F (20 degrees C). Both Figures 6 and 7 show a strong dependence on both inlet water temperature and % of max power. The greatest percentage heat rejected to water (66%) is achieved when the temperature o

40、f the air entering the servers matches the temperature of the water entering the server cooling loops, which in this case is 68 degrees F (20 degrees C). This occurs because there is no temperature gradient between the air and water when both are at the same temperature. In reality, the air heats up

41、 as it travels through the server, as does the water, but when the air and water temperatures match at their inlets, the temperature gradients that are established across the server are minimized this highlights a very complicated relationship between air temperature, water temperature, IT load, air

42、 flow rate, and water flow rate. As the inlet water temperature increases, and the inlet air temperature remains fixed at 68 degrees F (20 degrees C), the percentage of heat rejected to water declines steadily. The relationship between the % heat rejected to water and the % of max power is also quit

43、e complicated. The curves in Figures 6 and 7 also show that the percent heat rejected to water changes with percent of max power. The expectation was that the percent heat rejected to water would remain the same if variables such as fan speed, inlet water temperature and inlet air temperature remain

44、ed the same (which they do), yet the curves show a certain dependency. By comparing the curves for the 10% fan speed case to those for the 20% fan speed case, we see a distinct impact of fan speed and consequently air flow rate. For example, at 86 degrees F (30 degrees C) inlet water, 63% of the hea

45、t is rejected to water at 10% fans speed, while that number drops to 60% at 20% fan speed. Again, heat losses from the water-side to the air-side are contributing to the reduction in percent heat rejected to water. For convenience and ease of comparison, the data from Figures 6 and 7 have been tabul

46、ated in Table 2. Figure 8 presents the IT rack exhaust water temperature as a function of percent of max power. This data is for the fans running at 20% of max speed only. Since the air and water flow rates are fixed, the rack exhaust water temperature is directly proportional to the CPU heat reject

47、ed to water, which in turn is proportional to the CPU power draw. As as result, as the percent of maximum power increases, the rack exhaust water temperature increases. In addition, the rack exhaust water temperature increases with an increase in inlet water temperature as was expected. For all test

48、s conducted, all key data from the motherboard and chassis sensors was collected and analyzed. Tables 3 and 4 present the results for the CPUs (1 and 2), CPU voltage regulators, DIMMs and hard drives. For the CPUs, the on-board management system does not report case temperatures lower than 104 degre

49、es F (40 degrees C), so a 104 degree F (40 degree C) reading in the table means that the CPU is at 104 degrees F (40 degrees C) or less. Water temperatureCPU utilizationHeat to water (10% of max fan speed)Heat to water (20% of max fan speed)(F/C) (%) (%) (%)Idle 0.51 0.5050 0.65 0.6380 0.67 0.61100 0.66 0.64Idle no data 0.2450 0.58 0.5580 0.62 0.59100 0.63 0.60Idle no data no data50 0.53 0.4680 0.57 0.52100 0.58 0.55Idle no data no data50 0.41 0.3580 0.50 0.46100 0.53 0.4968 / 2086 / 30104 / 40120 / 50Figure 8: IT rack exhaust water temperature as a percentage of IT chas