1、Liquid Cooling Guidelines for Datacom Equipment CentersThis publication was prepared in cooperation with TC 9.9, Mission Critical Facilities,Technology Spaces, and Electronic Equipment.Any updates/errata to this publication will be posted on the ASHRAE Web site at www.ashrae.org/publicationupdates.A
2、merican Society of Heating, Refrigeratingand Air-Conditioning Engineers, Inc.Liquid Cooling Guidelines for Datacom Equipment CentersISBN-10: 1-933742-05-4ISBN-13: 978-1-933742-05-22006 American Society of Heating, Refrigeratingand Air-Conditioning Engineers, Inc.1791 Tullie Circle, NEAtlanta, GA 303
3、29www.ashrae.orgAll rights reserved.Printed in the United States of AmericaPrinted on 30% post-consumer waste using soy-based inks.Cover design by Tracy Becker.ASHRAE has compiled this publication with care, but ASHRAE has not investigated, and ASHRAEexpressly disclaims any duty to investigate, any
4、product, service, process, procedure, design, or the likethat may be described herein. The appearance of any technical data or editorial material in this publi-cation does not constitute endorsement, warranty, or guaranty by ASHRAE of any product, service,process, procedure, design, or the like. ASH
5、RAE does not warrant that the information in the publi-cation is free of errors, and ASHRAE does not necessarily agree with any statement or opinion in thispublication. The entire risk of the use of any information in this publication is assumed by the user.No part of this book may be reproduced wit
6、hout permission in writing from ASHRAE, except by areviewer who may quote brief passages or reproduce illustrations in a review with appropriate credit;nor may any part of this book be reproduced, stored in a retrieval system, or transmitted in any wayor by any meanselectronic, photocopying, recordi
7、ng, or otherwithout permission in writing fromASHRAE._Library of Congress Cataloging-in-Publication DataLiquid cooling guidelines for datacom equipment centers.p. cm.Includes bibliographical references and index.ISBN 978-1-933742-05-2 (pbk.)1. Data processing service centers-Cooling. 2. Hydronics. 3
8、. Electronic apparatus and appliances-Cooling. 4. Electronic digital computers-Cooling. I. American Society of Heating, Refrigerating and Air-Conditioning Engineers. TH7688.C64L57 2006697.9316-dc222006026115ASHRAE STAFFSPECIAL PUBLICATIONSMildred GeshwilerEditorChristina HelmsAssociate EditorCindy S
9、heffield MichaelsAssistant EditorMichshell PhillipsAdministrative AssistantPUBLISHING SERVICESDavid SoltisManagerTracy BeckerGraphic Applications SpecialistJayne JacksonPublication Traffic AdministratorPUBLISHERW. Stephen ComstockContentsAcknowledgments. . . . . . . . . . . . . . . . . . . . . . . .
10、 . . . . . . . . . . . . . . . . . .viiChapter 1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Definitions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.2 Liquid Cooling Systems . . . . . . . . . . . . . . . . . . .
11、 . . . . . . . . . . . 4Chapter 2 Facility Cooling Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.2 Equipment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
12、. . 122.3 Heat Rejection Equipment . . . . . . . . . . . . . . . . . . . . . . . . . . . 152.4 Pumps. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162.5 Energy Recovery Equipment . . . . . . . . . . . . . . . . . . . . . . . . . 17Chapter 3 Facility Piping Desi
13、gn . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213.1 General. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213.2 Spatial Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223.3 Basic Piping Architecture . . . . . . . . . . . .
14、 . . . . . . . . . . . . . . . . 223.4 Piping Arrangements for the Cooling Plant. . . . . . . . . . . . . . . 323.5 Water Treatment Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353.6 Earthquake Protection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35Chapter 4 Liq
15、uid Cooling Implementation forDatacom Equipment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 374.1 Overview of Liquid-Cooled Racks and Cabinets. . . . . . . . . . . 374.2 Overview of Air- and Liquid-Cooled Datacom Equipment . . . 424.3 Overview of Coolant Distribution Unit (CDU) . . . .
16、. . . . . . . . . 45vi ContentsChapter 5 Liquid Cooling Infrastructure Requirementsfor Chilled-Water Systems . . . . . . . . . . . . . . . . . . . . . . . .535.1 Building Facility Chilled-Water Systems (CHWS) . . . . . . . . . .535.2 Non-Chilled-Water Facility Systems. . . . . . . . . . . . . . . .
17、. . . . .69Chapter 6 Liquid Cooling Infrastructure Requirementsfor Technology Cooling Systems. . . . . . . . . . . . . . . . . . .736.1 Water-Based Technology Cooling System. . . . . . . . . . . . . . . .736.2 Non-Water-Based Technology Cooling System . . . . . . . . . . . .77References and Bibliogr
18、aphy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .79Glossary of Terms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .81Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .87Index . . . . . . . .
19、 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .89viiAcknowledgmentsRepresentatives from the following companies participated in producingthis publication:APCAavidCray Inc.Dell ComputersDLB Associates Consulting EngineersEYPMCFHewlett PackardIBMIntel Corporati
20、onLawrence Berkeley National LabsLiebert CorporationLytronMallory scalability; ease of installation, commissioning,and operation; ease of maintenance and troubleshooting; and availability and reli-ability. Any one or combination of the five considerations just mentioned can signif-icantly change how
21、 the facility cooling systems are designed, what equipment isused, and the overall system architecture. 2.1.1 FlexibilityData center cooling systems should be designed with features that will mini-mize or eliminate system outages associated with new equipment installation.These features should be ad
22、ded to both the central plant cooling systems and build-ing chilled-water piping architecture. Some of these features include valved andcapped piping connections for future equipment, such as water-cooled racks,central station air handlers, CRACs, and central plant equipment. The central plantshould
23、 be configured to add additional chillers, pumps, and cooling towers as theload increases. A properly managed load and growth plan or strategy should bedeveloped and employed to incorporate future computer and cooling systems.Overall flexibility is often limited by the pipe sizes used in the central
24、 plantand distribution system. After a data center is online, changing pipe size toincrease capacity is typically prohibitive from both outage risk and implementa-tion cost perspectives.8 Facility Cooling Systems2.1.2 ScalabilityThe building cooling systems should be designed to accommodate future l
25、oadgrowth of the computer equipment. Unless adequately planned growth and expan-sion is included in the data center, it will be obsolete in a very short time. Computertechnology changes every two to five years, and the cooling system will need to beexpanded to accommodate this load growth. Therefore
26、, building piping architecture(CWS and CHWS) should be designed to support a future building cooling loaddensity. Although first cost often dictates pump selection and pipe sizing, pumpingenergy, flexibility, and chilled-water storage will need to be considered to determinea total cost of ownership.
27、 The central plant should have enough space for future chillers, pumps, and cool-ing towers. The central plant chilled- and condenser-water system pipe headersshould be sized to operate efficiently from day one through the ramp up and for thefuture projected load.Sizing piping for the full buildout
28、or future growth will save energy and allowsmaller active components during the early life of the building. If the budget doesnot allow for any additional building costs for future growth, the owner needs tomake sure that real estate is available next to the existing central plant. 2.1.3 Ease of Ins
29、tallation, Commissioning, and OperationCooling service equipment should be designed such that it can be installed ineasy, visible, and readily accessible locations. Commissioning is an effective strategy to verify that the cooling systems areoperating as intended in the original building design and
30、should be considered forevery project. In Design Considerations for Datacom Equipment Centers (ASHRAE2005a), Chapter 12 provides helpful information regarding cooling services forequipment cooling systems. This chapter details the five steps of formal commission-ing activities, starting with the fac
31、ilitys intent and performance requirements (asdetermined by the project team), and following with the Owners Program document,the Basis-of-Design document, and the project Commissioning Plan. These activitiesinclude factory acceptance tests, field component verification, system constructionverificat
32、ion, site acceptance testing, and integrated systems testing. Commissioning at full load (full flow) to prove hydraulic capacity should be arequirement. Loop isolation segments should be commissioned to prove circulationcapacity around each segment with looped supply and return to provide concurrent
33、maintainability without loss of cooling service.Data centers should be designed with a central control or command center tooversee building operations. The control center should house all of the building oper-ations systems, such as security monitoring, energy management and controlsystems (EMCS), s
34、ystem control and data acquisition (SCADA) systems, buildingautomation systems (BAS), and fire alarms. This control can be co-located with thecomputer system control room. It should be staffed for 24-hour operation. All emer-Liquid Cooling Guidelines for Datacom Equipment Centers9gency procedures, p
35、rotocols, and a personnel list should be included in the controlcenter and updated as changes occur. Power and cooling loads on facility equipment,such as UPS, chillers, and electrical feeders, should be monitored to determine loadgrowth and available capacity. Configuration management groups or boa
36、rds,consisting of IT and facilities departments, should be established to control andmanage data center infrastructure. 2.1.4 Ease of Maintenance and TroubleshootingThe ease of maintenance and the ability to troubleshoot problems quickly andaccurately are essential elements of a high-availability da
37、tacom facility. The firstelement of this planning should be to maintain adequate working clearance aroundcooling equipment. Manufacturers recommendations for working clearancesshould be used as a minimum for serviceability areas. Designers need to provideaccess to allow maintenance and operation of
38、valves, control devices and sensors,and large equipment. Lifts, hoists, and cranes may be mounted within the centralplant to help facilitate removal of heavy equipment and components. These devicesshould be incorporated into the existing structural members of the plant room. Forexample, a hoist and
39、rail system could be placed above the chillers in the plant roomto facilitate rapid removal of end plates and/or compressors. Also, space should beprovided to facilitate demolition and removal of entire cooling system components,such as chillers. If space is at a premium, tube pull areas for each ch
40、iller can be sharedsince usually one chiller is being worked on at a time. Chilled and condenser waterpipes should be routed to avoid conflict with removal of cooling system equipment.Mechanical equipment, such as pumps and chillers, should be arranged in such amanner as to facilitate complete repla
41、cement. Isolation valves must also be locatedto allow for replacement without interrupting service, which makes layout andassembly of the entire piping system important.Building owners should consider a computerized maintenance managementsystem (CMMS) to help manage equipment maintenance. These syst
42、ems can recordmaintenance history and automatically dispatch work orders for future maintenance.Manufacturers specific maintenance requirements and frequencies can be input ordownloaded into the CMMS. It is much easier and desirable to coordinate equipmentoutages and maintenance than to deal with an
43、 unscheduled equipment failure due tolack of adequate maintenance. Energy management and control systems (EMCS) or building automationsystems (BAS) sensors and device outputs can be trended over time and used forsystem diagnostics and troubleshooting. EMCS data can also be used to monitor andcharact
44、erize system performance over time. For example, chiller amp readings and/or chilled-water temperature differentials and flows can be used to calculate andmonitor chiller efficiency and load growth. Control systems should have a fail-safecondition that allows mechanical flow and ON operation. Typica
45、l general buildingmanagement systems shut down on the loss of control. Data center systems shouldturn on to keep systems online.10 Facility Cooling SystemsLoad growth over time can be compared to chilled-water capacity, and thisinformation can be used to project time frames for plant expansion or in
46、crease inindividual component capacity, i.e., replace an 800 ton chiller with a 1200 ton chiller.In addition, flowmeters and pressure sensors can be installed in the building coolingdistribution or secondary loop and used to monitor chilled-water flows and capac-ities in the piping architecture. Thi
47、s information can be used to determine the bestlocation in which to install the newest water-cooled computer equipment or used tocalibrate a network model of the chilled-water systems. Finally, analog thermome-ters, pressure gauges, and flow-measuring instrumentation (orifice plates, balancingvalves
48、, etc.) should be installed in chilled-water piping and used to gain additionalinformation on system performance. Sensors need placement in both the primaryloop and the auxiliary loop to allow control if either part of the loop is being serviced.Manual operation may also be used if the alternate loo
49、p is temporary. 2.1.5 Availability and ReliabilityA key to having a reliable system and maximizing availability is an adequateamount of redundant equipment to perform routine maintenance. If N represents thenumber of pieces to satisfy the normal cooling capacity, then often reliability stan-dards are considered in terms of redundant pieces compared to the baseline of N.Some examples would be: N + 1full capacity plus one additional pieceN + 2full capacity plus two additional pieces2Ntwice the quantity of pieces required for full
