1、NA-04-4-2 Thermal Profile of a High-Density Data Center-Methodology to Thermally - Characterize a Data Center Roger R. Schmidt, Ph.D., P.E. Member ASHRAE ABSTRACT The heat dissipated by large sewers and switching equip- ment is reaching levels that make it ve y dijFcult to cool these systems in data
2、 centers or telecommunications rooms. Some of the highestpoweredsystems are dissipatingupwardof2000 W/ fi(21,500 W/m2) based on the equipment footprint. When systems dissipate this amount ofheat and then are clustered together within a data centel; signjcant cooling challenges can result. This paper
3、 describes the thermal projle of a 74j x 84ft(22.6 m x25.6m) data center and the measurement tech- niques employed to fully capture the detailed thermal environ- ment. In aportion of the data center(48j x 56ftr14.6 m x 17. I m) that encompasses the servers the heat flux is 170 W/ fi(1830 W/m2). Most
4、 racks within this area dissipated 6.8 kW while a couple dissipated upward of26 kK Detailed measure- ments were taken in this data center of electronic equipment power usage; perforated floor tile airflow; cable cutout airflow; computer room air conditioning (CRAC) airflow, temperatures andpower usa
5、ge; electronic equipment inlet air temperatures. In addition to these measurements, thephysical features ofthe data center were recorded such that a detailed CFD model could be employed to compare the results. The empirical as well as theflow modeling data are presented and compared. INTRODUCTION Th
6、e heat dissipated by electronic equipment is increasing at a very rapid rate. This increase is summarized in Figure 1, generated by a consortium of 17 industry equipment manu- facturers and published by the Uptime Institute showing that over the last five years the heat flux of rack level servers an
7、d storage doubled. In 2003 the maximum rack heat flux for serv- ers displayed is 15000 W/m2, which translates into 11 kW dissipated from a 19 in. (482.6 mm) rack. Racks with this heat load distributed in a data center present a real challenge for facility engineers in providing adequate cooling to t
8、hese high- density racks. And the problem is exacerbated when clusters of such racks are placed in a data center where zones of high- density servers are beyond the cooling capabilities of the data center. Many data centers employ raised floors to distribute the chilled air to these racks. One of th
9、e challenges to this arrangement is to provide the proper distribution of chilled air to the racks provided by computer room air-conditioning (CRAC) units situated on the raised floor. Kang et al. (2001) and Schmidt et al. (2001) provide some insight in the flow distribution from perforated tiles, a
10、nd Schmidt et al. (2001) show results of the perforated tile distribution on the inlet temperatures of racks located on the raised floor. Figure 1 Equipment heat densities. Roger Schmidt is a distinguished engineer at IBM Corporation, Poughkeepsie, N.Y. 02004 ASHRAE. 635 Figure 3 Cold aisle showing
11、racks Figure 2 Data center layout. Detailed thermal profiles of data centers for the most part do not exist in the literature. Pate1 et al. (2001) developed a three-dimensional model of a laboratory data center and experimentally verified the numerical results to ensure the specified inlet air tempe
12、ratures to the computer systems met the temperature limits. Schmidt (1 997) thermally profiled a non-raised-floor data center in a small office and then compared the results to a CFD model of the space. The results compared favorably. Data from other electronic equipment rooms are available, but it
13、is very difficult to glean important information or correlate or compare data. The motivation for this paper is twofold. First, this paper provides some basic information on the thermal/flow data collected from a high-density data center. Second, it provides amethodology that others can follow in co
14、llecting thermal and airflow data from data centers so that data can be assimilated to make comparisons. This database can then provide the basis for future data center air cooling design and aid in the under- standing of deploying racks of higher heat loads in the future. LAYOUT OF DATA CENTER The
15、data center profiled is the National Center for Envi- ronmental Prediction (NCEP) located in Bethesda, Maryland. All the equipment is located on a raised floor in an enclosed area of 74 ft x 84 ft (22.6 m x 25.6 m). A plan view ofthe data center, indicating the location of the electronic equipment,
16、power distribution units (PDU), CRAC units, and perforated floor tiles, is shown in Figure 2. Most of the servers (5 1 racks) are IBM Model 7040(p690). The other systems are a mix of Fgure 4 Blockages underneath rased$ooz switching, communications, and storage equipment. The key classes of equipment
17、 are highlighted in Figure 2. The ceiling height, as measured from the raised floor to the ceiling, is 10 ft (3.05 m) with a raised floor height of 17 in. (431.8 mm). Computer room air-conditioning units (seven operational CRAC units) as well as power distribution units (six opera- tional PDU units)
18、 are located around the perimeter of the room. Potential expansion is anticipated and additional PDU and CRAC units are also shown in Figure 2. The servers are located in a cold aisleihot aisle arrangement with aisle widths of approximately 4 ft (1.2 m) (two floor tiles wide). The cold aisles were p
19、opulated with 25% open tiles with the dampers removed on all the tiles. A cold aisle, showing the rows of racks, is seen in Figure 3. In addition, underfloor blockages occurred beneath the raised floor. These were either insulated chilled water pipes, as shown in Figure 4, or cabling located beneath
20、 the server equipment. When the data center was first populated with equipment, high rack inlet air temperatures were measured at a number of rack locations. The problem was that the perimeter between the raised floor and subfloor was not blocked off, and the chilled air from the CRAC units was exit
21、ing to other portions 636 ASHRAE Transactions: Symposia of the building (this data center was centrally located among other raised floor data and office space). In addition, the total heat dissipation by the electronic equipment in the room exceeded the sensible cooling capacity of the CRAC units. B
22、ased on these problems, an additional CRAC unit was installed and the entire perimeter of the region between the raised floor and subfloor was enclosed. (Although the “before“ results will not be presented in this paper, the result- ing flow increased by about 50% and the rack inlet tempera- tures d
23、ecreased on average about 5C with these two noted changes). MEASUREMENT TOOLS The airflow through the perforated floor tiles, cable cutouts, and CRAC units was measured with an Alnor velom- eter. The unit was calibrated on a wind tunnel and all measure- ments were adjusted based on the calibration (
24、velometer was measuring approximately 4% low for the range of airflows measured). In addition to this correction, the reading of the velometer also needs to be corrected for the reduction in airflow caused by the flow impedance of the unit. The unit was modeled using a computational fluid dynamics s
25、oftware pack- age, where the resulting correction for the unit (for the 500 ch scale) is given by Corrected Flow Rate (cfm) = 1.1 1 x Measured Flow Rate (cfm) - 16.6 (for flows greater, than 200 cm) . The results presented in the remainder of the paper includes the above two noted corrections. The t
26、emperatures were measured with a handheld Omega “23 meter using a type-T thermocouple. Since temperature differences, and not absolute temperatures, were of most importance, the meter was not calibrated, although the error in the thermocouple and instrument is estimated to be 11 .O“C. Temperature di
27、fference errors were estimated to be hl .O“C, resulting primarily from cycling of the CRAC units. Voltage and current measurements of the CRAC units were made with a handheld Fluke voltmeter (model 87) and a current clamp-on Fluke meter (model 33). Manufacturers of this equipment reported the error
28、in these devices as %0.7% and therefore, measuring a couple of these systems was planned. Also, there were two fully configured IBM model p655 racks that dissipated a very high heat load. Given that there were only two systems, it was planned to also measure them. However, since the communi- cations
29、 could not be established with these racks, the same configured racks in another lab were measured. The results for the p690s were 7.2 and 6.6 kW while two p655s were 25.8 and 26.5 kW. These power measurements were made with apower tool connected directly to the racks. The breakdown of the data-proc
30、essing rack input powers is shown in Table 1. For those rack input powers that were not measured, estimates were obtained from the power profile of each. The rack input powers are displayed as a bar graph in Figures 5 to 9 with each rack power bar somewhat lined up with the racks shown in the pictur
31、e of the layout at the top of each figure. Airflow Measurements The airflow from the perforated floor tiles was measured with an Alnor velometer. This flow tool fits exactly over one perforated tile so it provides an excellent tool for rapidly profiling the flow throughout the data center. Measured
32、flows from each tile or cable cutout was very stable, varying by less than 10 cfm (0.28 m3/min). The measured flow rates from each perforated tile and cable cutout are shown in Figures 5-9. As in the display of the rack powers, the airflows from the perfo- rated tiles and cable cutouts are aligned w
33、ith the physical layout of the data center shown in the picture at the top of each figure. Measuring the cable cutout airflows could not be achieved directly, since it would have been impossible to locate the flow tool directly over the cable cutout that is within the footprint of the rack and at th
34、e rear of the rack. However, an alternative method was proposed and verified to obtain an estimate of the airflow through a cable cutout (or other open- ings throughout the data center such as within the PDU foot- print). The technique will now be described. First, a cable cutout in which it is desi
35、red to know the airflow is completely blocked with foam materials. Second, a tile with a cutout ofthe shape of the cable cutout was provided and placed in the open- ing nearest the desired cable opening. To match the blockage contributed by the cables, a piece of tape was used to block a portion of
36、the cutout. The flow through the modified tile was then measured with the flow tool. Then the blockage was removed from the desired cable cutout and the airflow through the modified tile repeated. Comparison of these flows, with and without blockage of the cable cutout, showed no discern- ible diffe
37、rence in the flow rates measured. Therefore, all cable cutouts were measured with this modified tile without block- ing the actual cable cutout (this technique saved a significant amount oftime). When some of the cutouts were of different size, the modified tile was adjusted to approximate the actua
38、l opening. The airflow measurements from the cable cutouts are also shown in Figures 5-9. Similar to the perforated tile results, the airflows for the cable cutouts are somewhat aligned with the physical layout shown at the top of each figure. The overall flow of the data center then could be estima
39、ted based on all the measurements of the flow from the perforated floor tiles and cable cutouts. The sum of all these measure- ments was 70,896 cfin (2,008 m3/min) (after adjusting for the calibration of the flowmeter and flowmeter resistance). Again, the perimeter of the data center below the raise
40、d floor was completely enclosed such that no air escaped the room. One additional area offlow not accounted for is the leakage of air that occurs between the perforated tiles. Tate Access (2002) states that a typical air leakage of 0.69 cfm/ft2 (0.21 m3/min per m2) occurs at a static pressure of 0.0
41、5 in. (1.27 mm) of water. Since flow modeling (see “Comparison to Tileflow” section) of the flow beneath the floor showed underfloor static pressures of approximately 0.03 in. (0.76 mm) of water, the air leakage is estimated to be 0.50 cfin/ft2 (0.15 m3/min per m2). With the data center having an ar
42、ea of 6200 ft2 (576 m2) the total air leakage is estimated to be 3200 cfm (90.6 m3/min). This then results in a total estimated data center airflow rate of 74,100 cfm (2098 m3/min). No leakage occurred at the walls around the perimeter of the data center since the side walls rested on top of the rai
43、sed floor and not on the subfloor. The estimated error in the total data center flow rate is estimated to 638 ASHRAE Transactions: Symposia 10 9 8 7 6 5 4 3 2 1 h I U I I m I I II II8 z h I II I I b I I I II III Figure 6 Environmental characteristics of racks in row 9. pH(MtMnu F- Ra 17 ant IILINOCI
44、.IUVII*IUIW Rd*!z.=mT Figure 8 Environmental characteristics of racks in row 18. AB c O E F GH I J K L MN O P QR s T U VWXY ZAABBCCCDDEEFFGGHH II uiotu Rack Pmsr- Rar16 I UIIIL u RIB mm GI I IKLYUOIORTUVIXI2MIEE RD. tauon Figure 7 Environmental characteristics of racks in row 15. 26 25 24 23 ABCDEFG
45、HIJKLMNOPQRSTUVWXYZBffffi 0 H I J K L UN O P Q R S T U Y W X Y ZMLI0CCDDEeWCG RwLW C.M. Opning FI- -Rara Figure 9 Environmental characteristics of rack in row 23. ASHRAE Transactions: Symposia 639 be 4.5% (10% for cable cutouts and 4% for perforated floor tiles). The final flow measurement focused o
46、n the flow through each CRAC unit. These flows are difficult to measure since there is no easy way to obtain the airflow exhausting the CRAC unit beneath the floor since it is very nonuniform and highly turbulent. Nor is it easy to obtain the total flow entering the CRAC unit at its inlet. The estim
47、ated flow from each CRAC units is 10,585 cfm (299.8 m3/min). However, each CRAC unit will have some differences in flow due to back- pressures under the floor, filter cleanliness differences, varia- tions in the unit, etc. So, to obtain some estimate of the variation in airflow between the units, th
48、e Alnor velometer was employed. Basically a grid of flow measurements were taken across the face of the CRAC unit and then used to tabu- late the average velocity. From the average velocity and area, the total airflow into the CRAC unit was computed. First, the velometer was placed above the CRAC un
49、it at the inlet to measure a portion of the flow entering the unit. Obviously, the flow is not the same since the opening into the top of the tool (14 in. x 14 in.) (355.6 mm x 355.6 mm) is not the same as the bottom (23 x 23 in.) (584.2 mm x 584.2 mm). Therefore, the flow will be less than actual. With measurements at six loca- tions, which are then averaged and multiplied by the area of the inlet of the CRAC unit, the measured flow was about two- thirds of the actual flow through the CRAC unit (CFD model- ing of the tool at the inlet proved this to be correct). Thes
