1、2008 ASHRAE 3ABSTRACTMany data centers today have inappropriate temperatureand relative humidity environments due to being operated inef-ficiently and ineffectively. This paper will highlight the worstcase practices found in many data centers, their impact on theoperation and environment of the faci
2、lity, and how many ofthese situations can be quickly and inexpensively corrected.A few of the areas of greatest concerns include themismatch of IT expectations and facilities (primarily cooling)capability, which leads to greater exposures of catastrophicfailures in infrastructure equipment. Mismatch
3、ed electricaland cooling infrastructure lead to a waste of investment, aswell as, exposure of catastrophic failures of infrastructureequipment. Inappropriate equipment layout and no masterplan leads to inefficient utilization of floor space and coolingcapacity. Failure to measure and monitor key par
4、ameters leadsto uncontrolled application of cooling resources. Bypassairflow is a large contributor to inefficient use of coolingcapacity.INTRODUCTIONThere is a great deal of talk about the “Best Practices” toemploy in data centers. There have been numerous articles,white papers 1, and presentations
5、 2 made on the subject.This paper is going to take a slightly different approach, thatof identifying some of the worst practices that go on in datacenters. From this approach numerous poor to bad practicesthat are not usually addressed will be highlighted. These badpractices lead to exposure of the
6、computer and infrastructureequipment to unscheduled outages and therefore loss ofsystem availability. In addition many of these practices createa wasteful environment, where up to twice the energy isconsumed operating the data center than is actuallyneeded. 3 The two driving forces in many data cent
7、ers today areavailability, which has always been the key, plus hardware andinfrastructure efficiency, which is the new paradigm in wellrun installations. The later being driven by the recent effort toconserve electrical power in both computer equipment andwithin the facility. This paper will highlig
8、ht many situations,including management decisions, poor design and implemen-tation, and poor maintenance practices that make achievingthe goals of availability and efficiency difficult, if not impos-sible to achieve. System Availability and Infrastructure CapabilityIn todays demanding environment sy
9、stem availability isassumed to be “24 X Forever” in many data centers. In somecases this is based on sound business requirements. In many itis based on unsupported Information Technologys (IT)demands. Another driving force for such availability comes frommultiple users / tenants in the computer room
10、. Each has thecapability of accepting a scheduled maintenance outage some-time during the year. Unfortunately they can not coordinate thetimes when they can be down, so the demand is essentially“24 X Forever”.To support such high availability both the power and cool-ing systems must be both fault to
11、lerant (be able to sustain afailure in any component within the generation and deliverysystem), and be concurrently maintainable (be able to haveany component tested, repaired, or replaced) without having toshut down any component of the computer system(s), asSome Worst Case Practices in Data Center
12、sRobert F. Sullivan, PhDRobert F. Sullivan is a senior consultant with the Uptime Institute, Morgan Hill, CA.NY-08-0012008, American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc. (www.ashrae.org). Published in ASHRAE Transactions, Volume 114, Part 1. For personal use only. A
13、dditional reproduction, distribution, or transmission in either print or digital form is not permitted without ASHRAEs prior written permission.4 ASHRAE Transactionsshown in Figure 2. A tier structure system categorizes the abil-ity of data centers to meet these availability requirements. 4The probl
14、em occurs when the power and cooling infra-structure can not support the “24 X Forever” availability. Inmany data centers there might be redundant capability incertain components of both the power and cooling system.However, in many the distribution path is single threaded, asshown in Figure 1. In p
15、ower this might be non-redundant crit-ical switch gear and/or a single distribution of power betweenthe UPS and the PDUs. In the cooling system the non-redun-dant component is quite often the distribution piping. If a valveneeds to be replaced, a leak repaired, or additional coolingequipment added,
16、the cooling system has to be shut down. In an environment where scheduled outages are notacceptable, due to the real or perceived system availabilitydemands, and the infrastructure can not support the availabil-ity, the only “tolerated” outage is an unscheduled one. You fixit when it breaks and hope
17、 that does not happen to often. There is yet another aspect to the infrastructure not meet-ing the IT demands for availability. This is where the powerand cooling systems are not matched in their ability to supportsuch an availability demand. Usually the power system is morerobust than the cooling s
18、ystem. We have been concerned withuninterruptible power for 30 years or more. Therefore, thereare many solutions available to supply uninterruptible powerthat is also concurrently maintainable to the computer hard-ware. However, the concern about cooling is a more recentphenomenon. Therefore, there
19、are many data centers that havecooling systems with “aged” technology and designs. There isusually redundant cooling capacity on the raised floor andquite often in the refrigeration and heat dissipation compo-nents. The weak link is the pumps, piping, and control valves.They are single threaded and
20、can not be changed or servicedwith the cooling system operating.The inefficiency in this latest scenario is the waste of capi-tal expenditure. A great deal of money is spent on uninterrupt-ible power and the cooling system can not support suchavailability. It used to be that if power to the data cen
21、ter waslost and the UPS system kept the computer equipment oper-ating the heat loads were low enough to be able to open thewindows and doors, turn on the fans and continue to run. Withthe high density of equipment and power dissipation that existsin many data centers today that is not a viable optio
22、n.Another exposure is the inability to provide continuouscooling, which is defined as the capability to continue to coolthe computer equipment in the case of a loss of power. This isthe mechanical equivalent of the battery system on the UPSsystem. 5With todays high heat load environments the time it
23、 takesa computer room, or section thereof, to reach critical temper-atures is limited. Data collected has shown that at a heatdensity of 40 Watts/ft2it takes 10 minutes for the room temper-ature to rise over 25F and exceed the manufacturers maxi-mum operating temperature 3. At a heat density of 100W
24、atts/ft2that time is three to five minutes. Thermal modelingby a number of sources indicates at 300 Watts/ft2the time hasdecreased to less than one minute.The general response to this lack of a continuous coolingcapability is the use of emergency backup power in the formof engine generators. “My gen
25、erators can start in 10 secondsand be on line in less than half a minute.” is a general statementby many facility people. However, data continues to indicatethere are numerous incidents annually where when utilitypower was lost the engines did not start or the generators didnot pick up the load. Thi
26、s later scenario can be due to switchgear failure, the failure of multiple generators to sync witheach other, starter battery failures, or even the diesel enginesrunning out of fuel because the pumps supplying the day tankswere not on backup power. Inefficient Use of Computer EquipmentToo many serve
27、rs, storage arrays, and even communica-tion equipment in computer rooms are either not being used orthe utilization is very low. An informal survey of users indi-cates that 6% to 20% of the computer equipment in datacenters is no longer being used. It is sitting there idle burningpower and cooling c
28、apacity “24 X 7” and doing no usefulwork. It is estimated the same percentages of equipmentsupports a legacy application and can not be replaced becausethe application is proprietary to the business and/or is an oldcommercial piece of software that is no longer supported bythe operating systems on t
29、he new server technology. AgainFigure 1 Single power path tier I and II infrastructure. Figure 2 Dual power path tier IV infrastructure.ASHRAE Transactions 5this equipment is burning power and cooling capacity“24 X 7” and actually computing once a week, month, quarter,or even once annually.Finally t
30、here is the mindset that, “That is my server,supporting my application, and I dont want anything elserunning on it.” If there is a problem with the other applica-tion(s) it will impact my ability to do my job and the hardwaremanufacturer will point at the software as the cause of theproblems. This m
31、indset is a major deterrent to employingvirtualization in computer hardware. This results in the utilization of the latest technology serv-ers being estimated to be from below 10% to no higher than20%. Again burning power and cooling capacity “24 X 7”Lack of a Master Plan for the Computer RoomMany c
32、omputer rooms have a disarray of equipmentlayout, both infrastructure and computer cabinets. First of allthere is the “circle the wagons” cooling unit arrangementcreating a uniformly distributed cooling environment in a non-uniformly distributed heat load environment, as shown inFigure 3. Even worse
33、 is the random placement of coolingunits, where they face in every direction throughout the room.Since the underfloor airflow from most cooling units is direc-tional, 75% to 80% of the air moves away from the front of theunit, each unit controls its own plume under the raised floorand the air from a
34、djacent units will not penetrate this plume,it becomes difficult to identify which cooling unit is coolingwhich piece of equipment. 7 What makes the problem worseis the discovered fact that the return air pattern in such a roommay not match the supply air flow pattern. This leads to unitsnot sensing
35、 the heat load they are cooling which creates hotand cold spots throughout the room.The random placement of computer cabinets, with rowsrunning at right angles to each other and open areas betweengroups of cabinets, leads to the recirculation of hot exhaust air,within the room, to the intake of adja
36、cent cabinets and/or rowsas the hot air attempts returns to the cooling units.The legacy layout of cabinets, where all the cabinets facethe same direction and have a minimal aisle width, usuallyonly 32” to 36”, creates a situation where every aisle is both acold aisle and a hot aisle. The hot exhaus
37、t air moves across theaisle and preheats the cold air coming from under the raisedfloor, as shown in Figure 4. It is difficult to impossible to prop-erly cool the servers at the top of subsequent rows under thesecircumstances. Power dissipation is limited to less than 5 kWin cabinets with the legacy
38、 configuration, no matter how muchcold air is coming through the perforated tile or grate in frontof the cabinet. This has been verified by the author whenattempting to correct hot spots in computer rooms with legacyequipment layouts.In the underfloor layout of the computer room airflowblockages bec
39、ome a major problem. A few of these includechilled or condenser water pipes placed directly in front ofcooling units, cable bundles stacked to fill the underfloorcreating airflow dams, and cable trays running down the coldaisle, blocking airflow through the perforated tiles.It is frustrating to see
40、computer rooms that have beenengineered, constructed, and populated in the last five yearsthat employ “circle the wagon” cooling and legacy layout ofcomputer cabinets. This is 1980s technology being employedin the 21stcentury.Some installations that employ the Cold Aisle / Hot Aislecabinet arrangeme
41、nt do so improperly. Due to the decision thatall cable cutouts will occur at the edge of tiles and not in thecenter, the backs of cabinets are placed on tiles seams ratherthan the front edge. Quite often this creates an Cold Aisle thatis only 36” wide or even if is four feet wide there is only onero
42、w of tiles accessible for the placement of perforated tiles andthe management of cooling capacity. Another reason the Cold Aisles are squeezed narrowerthan recommended is to get another row of cabinets across thewidth of the room into the layout. Both situations cut in halfthe capability of deliveri
43、ng the recommended cooling capacityto the installed equipment cabinets. There is only one perfo-rated tile serving the two cabinets on each side of the ColdAisle, instead of one dedicated to each cabinet. Not Measuring and Monitoring in the Data CenterIf you do not measure, you can not monitor. If y
44、ou do notmonitor you have no knowledge of or control over what ishappening on the raised floor and within the infrastructuresystems. Under these circumstances chaos rules and the partywith the most political clout wins the arguments and controlswhat happens in the data center. This leads to a mismat
45、ch ofavailability and capability, mismatched infrastructuresystems, misplaced hardware, inefficient cooling systems, etc.Inefficiencies that result from not measuring and moni-toring can result in large capital and operating expenses. Poorcomputer equipment installation practices result in thecreati
46、on of hot spots in the computer room. The ultimate “fix”for hot spots is to install additional cooling capacity. Interest-ingly, in many instances when this is done the number andseverity of hot spots increases! 7 Add more cooling and theenvironment gets worse? This is caused by the fixed heat load.
47、With the addition of cooling capacity each cooling unit isFigure 3 Poor master planning cooling unit layout.6 ASHRAE Transactionsrequired to do less work. This raises the underfloor supplytemperature resulting in the increase in hot spots in the room. Other consequences for not measuring and monitor
48、ingare the unexpected hardware installation. The servers aredelivered on Wednesday, Operations is informed they will beinstalled on Saturday and be on line Sunday. Facilitys is toldon Thursday where they will be installed and how many powerwhips will be required. And no one considers how these newse
49、rvers will be cooled. Incapacity in the Data CenterIncapacity is defined as the infrastructure that is operatingbut not contributing to the power or cooling of the computerequipment. The majority of issues that lead to incapacity in adata center are associated with the cooling system. There canbe overcapacity in the power system, but this usually is asso-ciated with installing too much UPS or backup engine gener-ator capacity too early in the life cycle of a data center. The majority of mechanical incapacity in the data centeris associated with the cooling system.
