1、OR-05-1 2-2 High Density Cooling of Data Centers and Telecom Facilities-Part I Donald L. Beaty, PE Member ASHRAE Neil Chauhan Daniel Dyer, PE Associate Member ASHRAE ABSTRACT The rapid growth trend of electronic equipment heat density is causing concern regarding how to cool these high, if not extre
2、me, loads. Successfully cooling these loads is a complex problem that is made even more dificult by the lack of practical field experience with high density loads. High density loads amplifi the impact on the cooling system of many variables (infrastructure configurations, spatial allocations, etc.)
3、 thatpreviously may not have been the focus of the design engineer: The need for a solution is imme- diate, but (due in part to the dot-com crash) there has been a lack of resources to proactively address the problem. There- fore, the industry has fallen behind and requires accelerated recovery. The
4、 compounded eflect of these issues creates a need for a more holistic approach than those previously utilized with a view to a coordinatedprocess that results in planned solutions meeting the approval of all aflectedparties. Thefirst step toward this holistic approach has been taken by ASHRAE TC9.9
5、with thepublication ofThenna1 Guidelines for Data Processing Environments. Following on from that publication, this two-part paper examines some of the topics thatshould be consideredas apart of the holistic approach and provides some background on the high density cooling topic in general. INTRODUC
6、TION Overview of Pari One The topics in this field are not easily related to each other since the typical contributors for the topics are often from different sectors of the industry. Specifically, the following areas will be discussed across the two-part paper: The collaborative process: compensati
7、ng for a lack of historical data (Part 1) Load calculation at the predesign phase (Part 1) The impact of space planning with respect to cooling difficulty (Part 2) High-level presentation of some cooling system choices (Pari 2) Retrofitting high density loads into existing facilities (Part 2) This f
8、irst part of the two-part paper will cover two sections that represent the extremities in the types of chal- lenges the industry faces in trying to address the issues of high density loads. The first section will provide an overview of how the lack of awareness of the impact of deploying more powerf
9、ul computer equipment in a datacom facility can result in a prob- lem that is overly constrained and compounded with an unre- alistic budget and suggests an alternative approach that can be utilized to prevent escalation. The second section provides methods and techniques to understand and predict l
10、oads at the pre-design phase. The section also covers variables in defining the load, such as the configuration of the support spaces for the building and the area considered when calculating using the more common metrics. Prior to discussing the sections themselves, some back- ground is required on
11、 the context of the discussion in the form of definitions, issues, and influences that are being considered. Donald L. Beaty is president and Neil Chauhan and Daniel Dyer are engineers at DLB Associates, Consulting Engineers, Ocean, New Jersey. 02005 ASHRAE. 92 1 Some Definitions ing since it could
12、imply that high density loads do not exist. As this two-part paper will show, there is a lot of vague- ness with regard to the thresholds where a load is considered high density. Some sectors of the industry will focus on kW per rack and other sectors will want to focus on the average watts per squa
13、re foot (watts per square meter) in a given computer room. For the purposes of this paper, high density cooling means loads that have reached a threshold of 5 to 10 kW per rack or an average density of 100 to 150 watts per square foot (1000 to 1500 watts per square meter) across a computer room. Alt
14、hough the two threshold ranges mentioned are not interchangeably related (i.e., 10 kW per rack represents a much higher load than 150 watts per square foot 1500 watts per square meter), the values do represent the current industry perceptions ofthe range of the high density thresholds. A more tangib
15、le method of understanding the thresholds is to consider that a percentage of higher density racks within a computer room will drive up the average watts per square foot (meter). The term rack has different definitions in the telecom industry vs. data centers, but for this paper, the broadest defi-
16、nition will be used, which is “an open frame or enclosed cabi- net that houses electronic equipment.” This paper is not aimed specifically at any particular type of high density equipment deployment. Certainly there are many scenarios that could exist including: New construction versus retrofitting
17、an existing build- ing. Retrofit of an existing datacom facility versus retrofit of a building with a different (non-datacom) usage. Small-scale deployment versus a large-scale deployment of high density loads (e.g., addition of one to three high density racks versus addition of multiple rows of hig
18、h density racks). The scenarios listed above all have their own unique chal- lenges and constraints, and in some scenarios, a feasibility study for a high density equipment deployment may result in either extensive infrastructure changes being required or the deployment in a particular scenario may
19、be deemed unfeasi- ble. The fact is that high density loads do exist (currently, specific loads of 15 to 25 kW per rack or more have been measured). Although high density installations are few, the quantity is increasing. The existence of any high density loads establishes a need to address how to c
20、ool those loads. To summarize, the claim is not that all loads or most of the loads will be high density, but rather some of the loads will be high density in some of the facilities. Based on the product heat density trend charts, history supports a trend of significant increasing loads at the compo
21、nent, board, and equipment level. As a result, the current existence of some high density loads combined with the projected fture loads warrants the development of strategies to effectively handle high density loads. Background on Issues and Influences this a challenging problem to solve: 1. Followi
22、ng are some of the issues or influences that make Within the datacom environment (the term datacom refers both to data centers and telecom facilities), the IT equip- ment loads continue to rise. However, practical field expe- rience and actual measured data for those loads are not readily available
23、for the following reasons: High density loads (5 to 10 kW per rack or greater) have seldom been experienced in datacom facilities so there is a shortage of empirical data. The speed to construct versus the speed of IT equip- ment load growth creates a lag in completed instal- lations. Consequently,
24、the field data lags behind the IT equipment loads that are typically announced to be shipped within the next 12 months. Due to nondisclosure agreements (NDA), concern for security, and to keep a competitive edge, often there is a reluctance to share data, especially fail- ures, on the latest technol
25、ogy installed. 2. The functions of IT, facility design and construction, and facility operation and maintenance are often driven by, and report to, independent departments or people, with each one placing a different emphasis on any given attribute such as: Do High Density Loads Exist? cost There ar
26、e plenty of studies and information showing over- stated loads (i.e., measured load that is a mere fraction of the design or connected load). There are people who claim they have never seen loads above 25 watts per square foot (250 watts per square meter) and certainly not above 50 watts per square
27、foot (500 watts per square meter). These same people would then argue that, even if they double or triple that load (which is unlikely), it does not support the high density levels being projected or claimed. Determining whether high density loads exist based on personal experience or national avera
28、ges can be very mislead- * Reliability Energy efficiency Speed to marketldeployment Maintenance The different emphases result in budgets that are indepen- dent of one another. This, in turn, creates or promotes independence and working in isolation rather than work- ing on a multi-functional, collab
29、orative basis, addressing the project and decisions in a holistic way, including effective balancing of criteria. 922 ASH RAE Transactions: Symposia 3. 4. High density cooling is thought of as a challenge in terms of thermodynamics and physics. As a result, it both attracts and drives us toward focu
30、sing solely on technical cooling issues. Although these issues are critical, there are other related technical issues and also some nontechnical consid- erations that are just as critical but have a greater risk of receiving little attention: Technical Cooling Issues-These specifically focus on usin
31、g analytical heat transfer and fluid dynamics study to address the specific heat load challenge presented and determine the cooling sys- tem design. Other Technical Issues-These fall into two main categories, directly related and indirectly related: Directly related technical issues include con- sid
32、ering the inn uences and constraints that complicate or adversely affect the performance of the cooling system. In other words, instead of prematurely trying to implement a cooling system in the physical environment that has been presented, consideration needs to be given to whether the environment
33、itself can be modi- fied to better serve the cooling system. For example, this could include increasing the height of the raised floor caviy in the high den- sity load area. Indirectly related technical issues consider the variables affecting the heat loads, including the high density load location
34、and the various IT equipment options (e.g., using fiber versus cop- per connectivity can impact freedom on dis- tances and congestion). Through optimizing these variables, it may be possible to reduce the cooling loads before even considering how to successful cool them. Nontechnical Considerations-
35、These include intangibles such as process, procedure, and the fn- damental essence of problem solving. Unique and constantly changing problems such as high density cooling require a heavy focus on process to enable effective collaboration and holistic assessment/per- formance. This is counterintuiti
36、ve to the way the industry and the project participants are currently operating. The legacy methodology used for projecting or calculating the amount of heat dissipation that needs to be cooled is poorly defined and not optimized for addressing high density heat dissipation loads. A common approach
37、is to use watts per square foot (watts per square meter). Due to the lack of consis- tent definition of the attributes used for that calcula- tion (discussed in detail later), this number could easily vary by as much as 25% to 50%. Since the loads are rapidly changing, the increases in the watts per
38、 square foot (meter) values are made using adjustment factors to the previous industry standard watts per square foot (meter) values. How- ever, the potential variation in the definition makes those original numbers an ineffective baseline. THE COLLABORATIVE PROCESS: COMPENSATING FOR LACK OF HISTORI
39、CAL DATA In a paper presented at the International Mechanical Engi- neering Exposition and Congress in Washington, DC, in November 2003, entitled “The Challenges of Electronic Cool- ing: Past, Present and Future,” IBMs Richard Chu (2003) talked about the need for a collaborative effort to handle the
40、 higher density loads: All of these considerations clearly represent an increased level of challenges for thermal engineers. It also means that thermal engineers must be an integral part of the design process from the very beginning and work very closely with electrical and packaging engineers to ac
41、hieve a truly holistic design. Although the reference above refers specifically to the collaboration of the facets of IT equipment design, this paper proposes that the envelope of the collaborative approach be further expanded to provide, not just the holistic design of a specific product, but a hol
42、istic facility-based solution to the implementation of the product. Since there is a void in the practical field experience of high density loads, fundamental problem-solving approaches need to be undertaken in order to address the problem at hand. Fundamental problem-solving techniques involve proc
43、ess, communication, collaboration, and a holistic focus. In order to provide a more tangible insight into the abstract nature of problem solving, consider the following brief descriptions of two different project approaches and the impact of each to the same initial problem. In the two approaches, w
44、e will consider the introduction of a higher density load through the deployment of blade server equip- ment. The blade server equipment is the result of technology compaction, which allows for a greater processing density over the same equipment volume. The greater processing density also results i
45、n a greater power density and a greater heat density. Approach One (Non-Collaborative) Step 1. Staff in the IT department determines that they need to procure and deploy blade servers, which repre- sent a technology they have never used before. They interact with the IT equipment manufacturers and s
46、elect a manufacturer and product. Step 2. The IT staff obtains preliminary pricing from the ASHRAE Transactions: Symposia 923 manufacturer and submits for funding. Little or no con- sideration is given at this time for additional deployment costs to augment the support or infrastructure services (i.
47、e., cooling). Management approves the pricing for the IT equipment after going through the costbenefit met- rics as a part of their approval process. Step 3. The IT equipment is procured and the facilities department is notified that new equipment is coming and the datacom room must be modified to a
48、ccommo- date the equipment. Step 4. The facilities department discovers the equip- ment loads are far beyond what they have ever cooled before. Due to their current experience with projected loads not being realized, their first reaction is skepticism and the published loads are declared as being gr
49、ossly overstated. Step 5. The facilities department asks their counterparts at other firms and discover people are thinking these incredible loads could be real. Step 6. The facilities department hires a mechanical consulting engineer and assigns them the task of “figure out how to cool this.” No budget for this scope was assigned previously and management is blindsided by an additional cost that was not considered in their previ- ous metrics. Consequently, compounding the difficulty of accomplishing the actual cooling challenge is the fact that there are only minimal financial resource
