1、,Architecture of the Earth System Modeling Framework,GMAO Seasonal Forecast,NCAR/LANL CCSM,NCEP Forecast,GFDL FMS Suite,MITgcm,NASA GMAO Analysis,Climate,Data Assimilation,Weather,Cecelia DeLuca GEM Snowmass, CO,Outline,Background and Motivation Applications Architecture Implementation Status Future
2、 Plans Conclusions,Motivation for ESMF,In climate research and NWP. increased emphasis on detailed representation of individual physical processes; requires many teams of specialists to contribute components to an overall modeling system In computing technology. increase in hardware and software com
3、plexity in high-performance computing, shift toward the use of scalable computing architectures In software development of frameworks, such as the GFDL Flexible Modeling System (FMS) and Goddard Earth Modeling System (GEMS) that encourage software reuse and interoperabilityThe ESMF is a focused comm
4、unity effort to tame the complexity of models and the computing environment. It leverages, unifies and extends existing software frameworks, creating new opportunities for scientific contribution and collaboration.,ESMF Project Description,GOALS: To increase software reuse, interoperability, ease of
5、 use and performance portability in climate, weather, and data assimilation applications PRODUCTS: Core framework: Software for coupling geophysical components and utilities for building components Applications: Deployment of the ESMF in 15 of the nations leading climate and weather models, assembly
6、 of 8 new science-motivated applications METRICS:RESOURCES and TIMELINE: $9.8M over 3 years, starting February 2002,Outline,Background and Motivation Applications Architecture Implementation Status Future Plans Conclusions,Modeling Applications,Data Assimilation Applications,ESMF Interoperability De
7、monstrations,Interoperability Experiments Completed,NCAR Community Atmospheric Model (CAM) coupled to NCEP Spectral Statistical Interpolation (SSI) System, both set up as ESMF components Experiment utilizes same observational stream used operationally at NCEP,3 NCAR Community Atmospheric Model (CAM)
8、 coupled to MITgcm ocean Atmosphere, ocean, and coupler are set up as ESMF components Uses ESMF regridding tools,1 GFDL B-grid atmosphere coupled to MITgcm ocean Atmosphere, ocean, and coupler are set up as ESMF components Uses ESMF regridding tools,2,Temperature SSI importTemperature SSI exportTemp
9、erature difference,Outline,Background and Motivation Applications Architecture Implementation Status Future Plans Conclusions,Characteristics of Weather and Climate Simulation,Mix of global transforms and local communications Load balancing for diurnal cycle, event (e.g. storm) tracking Applications
10、 typically require 10s of GFLOPS, 100s of PEs but can go to 10s of TFLOPS, 1000s of PEs Required Unix/Linux platforms span laptop to Earth Simulator Multi-component applications: component hierarchies, ensembles, and exchanges Data and grid transformations between components Applications may be MPMD
11、/SPMD, concurrent/sequential, combinations Parallelization via MPI, OpenMP, shmem, combinations Large applications (typically 100,000+ lines of source code),Platforms,assim,sea ice,ocean,land,atm,physics,dycore,assim_atm,atmland,Seasonal Forecast,coupler,ESMF Architecture,ESMF provides an environmen
12、t for assembling geophysical components into applications, with support for ensembles and hierarchies. ESMF provides a toolkit that components use to increase interoperability improve performance portability abstract common services,Hierarchies and Ensembles,assim,sea ice,ocean,land,atm,physics,dyco
13、re,assim_atm,atmland,Seasonal Forecast,coupler,ESMF encourages applications to be assembled hierarchically and intuitively Coupling interfaces are standard at each layer Components can be used in different contexts,assim_atm,Ensemble Forecast,assim_atm,assim_atm,ESMF supports ensembles with multiple
14、 instances of components running sequentially (and soon, concurrently),ESMF Class Structure,DELayout Communications,State Data imported or exported,Bundle Collection of fields,GridComp Land, ocean, atm, model,F90,Superstructure,Infrastructure,Field Physical field, e.g. pressure,Grid LogRect, Unstruc
15、t, etc.,Data,Communications,C+,Regrid Computes interp weights,CplComp Xfers between GridComps,Utilities Machine, TimeMgr, LogErr, I/O, Config, Base etc.,Array Hybrid F90/C+ arrays,Route Stores comm paths,DistGrid Grid decomposition,PhysGrid Math description,ESMF Data Classes,Model data is contained
16、in a hierarchy of multi-use classes. The user can reference a Fortran array to an Array or Field, or retrieve a Fortran array out of an Array or Field. Array holds a cross-language Fortran / C+ array Field holds an Array, an associated Grid, and metadata Bundle collection of Fields on the same Grid
17、State contains States, Bundles, Fields, and/or Arrays Component associated with an Import and Export State,ESMF DataMap Classes,These classes give the user a systematic way of expressing interleaving and memory layout, also hierarchically (partially implemented) ArrayDataMap relation of array to dec
18、omposition and grid, row / column major order, complex type interleave FieldDataMap interleave of vector components BundleDataMap interleave of Fields in a Bundle,ESMF Standard Methods,ESMF uses consistent names and behavior throughout the framework, for example Create / Destroy create a new object,
19、 e.g. FieldCreate Set / Get set or get a value, e.g. ArrayGetDataPtr Add / Get / Remove add to, retrieve from, or remove from a list, e.g. StateAddField Print to print debugging info, e.g. BundlePrint And so on,Superstructure,ESMF is a standard component architecture, similar to CCA but designed for
20、 the Earth modeling domain and for ease of use with Fortran codes Components and States are superstructure classes All couplers are the same derived type (ESMF_CplComp) and have a standard set of methods with prescribed interfaces All component models (atm, ocean, etc.) are the same derived type (ES
21、MF_GridComp) and have a standard set of methods with prescribed interfaces Data is transferred between components using States. ESMF components can interoperate with CCA components demonstrated at SC03,Becoming an ESMF GridComp,ESMF GridComps have 2 parts: one part user code, one part ESMF code The
22、ESMF part is a GridComp derived type with standard methods including Initialize, Run, Finalize User code must also be divided into Initialize, Run, and Finalize methods these can be multi-phase (e.g. Run phase 1, Run phase 2) User code interfaces must follow a standard form that means copying or ref
23、erencing data to ESMF State structures Users write a public SetServices method that contains ESMF SetEntryPoint calls - these associate a user method (“POPinit”) with a framework method (the Initialize call for a GridComp named “POP”) Now youre an ESMF GridComp,Infrastructure,Data classes are Bundle
24、s, Fields, and Arrays Tools for expressing interleaved data stuctures Tools for resource allocation, decomposition, load balancing Toolkits for communications, time management, logging, IO,Virtual Machine (VM),VM handles resource allocation Elements are Persistent Execution Threads or PETs PETs refl
25、ect the physical computer, and are one-to-one with Posix threads or MPI processes Parent Components assign PETs to child Components PETs will soon have option for computational and latency / bandwidth weights The VM communications layer does simpleMPI-like communications between PETs (alternative co
26、mmunication mechanisms are layered underneath),DELayout,Handles decomposition Elements are Decomposition Elements, or DEs (decomposition thats 2 pieces in x by 4 pieces in y is a 2 by 4 DELayout) DELayout maps DEs to PETs, can have more than one DE per PET (for cache blocking, user-managed OpenMP th
27、reading) A DELayout can have a simple connectivity or more complex connectivity, with weights between DEs - users specify dimensions where greater connection speed is needed DEs will also have computation weights Array, Field, and Bundle methods perform inter-DE communications,ESMF Communications,Co
28、mmunication methods include Regrid, Redist, Halo, Gather, Scatter, etc. Communications methods are implemented at multiple levels, e.g. FieldHalo, ArrayHalo Communications hide underlying ability to switch between shared and distributed memory parallelism,Load Balancing,Three levels of graphs: Virtu
29、al Machine machine-level PETs will have computational and connectivity weights DELayout DE chunks have connectivity weights, will have computational weights Grid grid cells will have computational and connectivity weightsIntended to support standard load balancing packages (e.g. Parmetis) and user-d
30、eveloped load balancing schemes,Outline,Background and Motivation Applications Architecture Implementation Status Future Plans Conclusions,Open Development,Open source Currently 800 unit tests, 15 system tests are bundled with the ESMF distribution, can be run in non-exhaustive or exhaustive modes R
31、esults of nightly tests on many platforms are accessible on a Test and Validation webpage Test coverage, lines of code, requirements status are available on a Metrics webpage Exhaustive Reference Manual, including design and implementation notes, is available on a Downloads and Documentation webpage
32、Development is designed to allow users clear visibility into the workings and status of the system, to allow users to perform their own diagnostics, and to encourage community ownership,Port Status,SGI IBM Compaq Linux (Intel, PGI, NAG, Absoft, Lahey),Outline,Background and Motivation Applications A
33、rchitecture Implementation Status Future Plans Conclusions,ESMF Key Accomplishments,Public delivery of prototype ESMF v1.0 in May 2003 Monthly ESMF internal releases with steadily increasing functionality Completion of first 3 coupling demonstrations using ESMF in March 2004 NCAR CAM with NCEP SSI N
34、CAR CAM with MITgcm ocean GFDL B-grid atmosphere with MITgcm ocean All codes above running as ESMF components and coupled using the framework, codes available from Applications link on website Other codes running as ESMF components: MOM4, GEOS-5 Less than 2% lines of source code change Delivered ESM
35、F v2.0 in June 2004 3rd Community Meeting to be held on 15 July 2004 at NCAR,ESMF Priorities,Status Components, States, Bundles, Fields mature On-line parallel regridding (bilinear, 1st order conservative) completed Other parallel methods, e.g. halo, redist, low-level comms implemented Comm methods
36、overloaded for r4 and r8 Communications layer with uniform interface to shared / distributed memory, hooks for load balancing Near-term priorities Concurrent components currently ESMF only runs in sequential mode More optimized grids (tripolar, spectral, cubed sphere) and more regridding methods (bi
37、cubic, 2nd order conservative) from SCRIP Comms optimization and load balancing capability IO (based on WRF IO) Development schedule on-line, see Development link, SourceForge site tasks,Outline,Background and Motivation Applications Architecture Implementation Status Future Plans Conclusions,Next S
38、teps,Integration with data archives and metadata standardization efforts, anticipate collaboration with Earth System Grid (ESG) and European infrastructure project PRISM Integration with scientific model intercomparison projects (MIPs), anticipate collaboration with the Program for Climate Model Dia
39、gnosis and Intercomparison (PCMDI), other community efforts Integration with visualization and diagnostic tools for end-to-end modeling support, anticipate collaboration with the Earth Science Portal (ESP) ESMF “vision” for the future articulated in multi-agency white paper on the Publications and T
40、alks webpage,ESMF Multi-Agency Follow-on,3 ESMF FTEs at NCAR slated to have ongoing funding through core NCAR funds NASA commitment to follow-on support, level TBD DoD and NSF proposals outstanding Working with other agencies to secure additional funds,Outline,Background and Motivation Applications
41、Architecture Implementation Status Future Plans Conclusions,ESMF Overall,Clear, simple hierarchy of data classes Multi-use objects mean that the same object can carry information about decomposition, communications, IO, coupling Tools for multithreading, cache blocking, and load balancing are being
42、integrated into the architecture Objects have consistent naming and behavior across the framework,The Benefits,Standard interfaces to modeling components promote increased interoperability between centers, faster movement of modeling components from research to operations The ability to construct mo
43、dels hierarchically enables developers to add new modeling components more systematically and easily, facilitates development of complex coupled systems Multi-use objects mean that the same data structure can carry information about decomposition, communications, IO, coupling this makes code smaller
44、 and simpler, and therefore less bug-prone and easier to maintain Shared utilities encourage efficient code development, higher quality tools, more robust codes,More Information,ESMF website: http:/www.esmf.ucar.edu,Acknowledgements The ESMF is sponsored by the NASA Goddard Earth Science Technology Office.,
