An IntegratedInstrumentation Architecturefor NGI Applications.ppt

1、An Integrated Instrumentation Architecture for NGI Applications,Ian Foster, Darcy Quesnel, Steven Tuecke Argonne National Laboratory The University of Chicago,DOE NGI Instrumentation Project,“A Uniform Instrumentation, Event, and Adaptation Framework for Network-Aware Middleware and Advanced Network

2、 Applications” With UIUC (Dan Reed, Ruth Aydt) “Produce uniform notification and adaptation mechanisms, with the goal of catalyzing the development of both network-aware middleware and sophisticated network-aware applications”,Motivation,Environment incorporates multiple sensors Sources of events re

3、lating to behavior of resources, middleware, and applications Significant advantages to having uniform mechanisms for publishing/discovering sensors and for accessing sensor data E.g., find all sensors for path A-B Including historical data Enables end-to-end, top-to-bottom, past-to-present analysis

4、,Examples of Sensors,Network devices E.g., routers End system devices E.g., computers, storage systems Grid services E.g., Globus HBM, Network Weather Service Libraries E.g., CAVERNsoft, MPI Applications,For Example .,H,R,R,R,H,S,(SNMP),S,(SNMP),S,(SNMP),S,(netstat),S,(netstat),H/W,App,Libs,Sys,GRAM

5、,HBM,NWS,.,S,S,S,S,MPICH,globus-io,CAVERNsoft,DPSS,S,S,S,S,S,S,Three Project Components,1. Mechanisms for creating, publishing, discovering, and accessing sensors 2. Synthesis and analysis techniques for identifying qualitative behavior and trends in sensor data 3. Adaptation techniques that exploit

6、 sensor data to adjust middleware and application configurations to improve performance Argonne focus: (1) and (3); UIUC: (2), (3),Current Approach,Use a directory service (LDAP) to register and publish event sources Publish: source, type, contact online, archive Discover: “find all event sources of

7、 type X” Use NetLogger format for data Develop sensor manager to handle publish, subscribe, archiving Use SQL database as archive Initial sensor set based on Globus libraries, applications, NetLogger-accessible devices,Initial Instrumentation Architecture,LDAP,Sensor Manager,SQL,Netarchive,MySQL,Pub

8、lish (“netstat, host A, time T,contact X”),Subscribe,Discover (“what event sources for route A to B?”),Application,Events in NetLogger format,Sensor,Sensor,Archive,File,Sensor Manager,We are building a program which: Archives sensor event streams Redirects sensor event streams to clients using a pub

9、lish/subscribe interface Generates sensor event streams from archive, based on query language Publishes interfaces and index to LDAP Relation to other work Superset of Netlogd (simple archiver) Might exploit Netarchiver (MySQL indexing),Archiving Events,How to archive sensor event streams? SQL: Save

10、 each event as a record in an SQL database Advantage: Rich query support Netarchive: Save each event into file. Use SQL database to build index of file contents Advantage: Performance and scale? We will explore the use of SQL databases Premise: Most sensors will not produce high volume event streams

11、; hence optimize for simplicity and rich query support,Bandwidth/Latency ANL-NASA Ames,NCSA Origin Nodes,ANL CPU Load,Bandwidth/Latency ANL-Indiana,Applying Info Infrastructure to Instrumentation,Publishing & Discovering Sensors,Globus LDAP-based Metacomputing Directory Service (MDS) provides scalab

12、le, global infrastructure for publishing and discovering sensor managers Sensors stream events to a sensor manager Sensor manager publishes availability of streams into LDAP Clients discover sensor managers from LDAP, and can subscribe to either current or archived sensor event streams directly from

13、 sensor managers,Initial Applications,Replica creation in “Data Grid” applications Online and historical instrumentation for large data transfers (app, lib, network) Involves DPSS, globus-io Also application-level selection of replicas, based on sensor information MPI-based video streaming (Karonis,

14、 Papka),Security,Grid Security Infrastructure (GSI) will be used throughout, hence possible to say e.g. “Manager M accepts only streams from sensors of user U” “Manager N only publishes streams to clients of users A, B, C” As a first step, we have augmented the Netlogger C client with GSI,Instrument

15、ation Architecture Showing Actuators,LDAP,Publish,Discover,Subscribe,Events,Publish,Discover,Subscribe,Events,Future Directions,XML Netlogger is an ASCII based format If you using ASCII, why not use XML? XML database could be used for archive Events Performance related events should be just one part

16、 of a larger, integrated event system Typing Netlogger is weakly typed Various advantages to strongly typed events,Future Directions (2): Publish/Subscribe for Sensors,In first version: Netlogger based sensors stream events to manager Manager publishes sensor availability to LDAP Clients subscribe t

17、o sensor manager for events In later version: Sensor can publish existence to LDAP Client can subscribe directly to sensor for events,Network Weather Service (R. Wolski et al., U.Tenn),Scalable, fault tolerant system for Real-time performance measurements Predictions of future state When installed o

18、n N hosts, delivers: Network performance (=N2 via netperf) Host cpu-load measurements (N) We (USC/ISI crew) are working to integrate this into MDS; hopefully will eventually be consistent with approach described here (to be discussed),Structure of NWS data in MDS (old),c=US,o=ISI,o=Globus,nn= the Internet,N2 Network performance entries for N hosts,N sets of cpu info for N hosts,.,