The ADEPTDigital Library Architecture.ppt

1、The ADEPT Digital Library Architecture,JAMES FREW & GREG JANE Alexandria Digital Library Project University of California, Santa Barbara,ADEPT Introduction,Alexandria Digital Earth ProtoType (ADEPT) is: Distributed digital library for geo-referenced information Services supporting DL federation and

2、interoperation Large geospatial collectionsGoal: an Internet “library” layer Organization Persistence Accessibility Scalability Lots of collections Big collections small collections Heterogeneous contents,Outline,Core library architecture Metadata interoperability Other features Query translation Co

3、llection discovery,ADEPT Core Library Architecture,Architectural Elements,Item structured descriptions (“reports”) contents (optional)Library set of collections client (public) servicesCollection set of items library (internal) services= a distributed catalog system,The Big Picture,library (middlewa

4、re server),client,item,item,item,item,library (middleware server),proxy,collection,collection,collection,ADEPT,Role of the Middleware,collection,collection,logical view,collection discovery service,client,thesaurus/ vocabulary,Middleware Interfaces,ClientsConfiguration collection-id Collection(colle

5、ction-id) report Query(query) query-id Results(query-id) holding-id Metadata(collection-id, holding-id, view) reportLibrariesCollection report Query(query, accumulator) query-thread Metadata(holding-id, view) report Collections,Metadata Reports,Collection Metadata that applies to entire collection B

6、ucket Items bucket metadata Scan Brief (“1-line”) subset of bucket report Full All the items metadata, in whatever format Browse URL(s) reduced-resolution graphics Access URL(s) content (if available),A Complete System: The Boxology,HTTP transport,SDLIP proxy, other clients,web browser,generic DB dr

7、iver,query translator,M I D D L E W A R E,C L I E N T,S E R V E R,web intermediary/ XMLHTML converter,RDBMS,JDBC,configuration files, Python scripts,RMI transport,proxy driver,HTTP,HTTP,XML,XML,group driver,thesauri,OR,paradigm library,Z39.50 driver,Metadata Interoperability,ADEPTs Interoperability

8、Problem,Distributed, heterogeneous collections locally, autonomously created and managedMinimize impact on collection providers allow use of native metadataUniform client services common high-level interface across collections discover and exploit collection-specific interfacesAssumptions items have

9、 metadata items have sufficient, “good” metadata i.e., this is a metadata interoperability problem,Bucket Concept,Abstract metadata category Strongly typed Well-defined search semantics query terms query operators Explicitly mapped from source metadata (FGDC, 1.3, “Time period of content”, “2001-09-

10、08”)Bucket-level search uniform across all collections e.g.: search all collections for items whose Originator bucket contains the phrase “geological survey”,Bucket Properties,name Coverage date semantic definition The time period to which the item is relevant. data type (strictly observed) calendar

11、 date or range of calendar dates syntactic representation (strictly observed) ISO 8601,What is a bucket? (2/2),Source metadata is mapped to buckets buckets hold not just simple values “2001-09-08” but rather, explicit representations of mappings (FGDC, 1.3, “Time period of content”, “2001-09-08”) ma

12、y have multiple values per bucket Bucket definition includes search semantics query terms ISO 8601 date range query operators contains, overlaps, is-contained-in Some semantics are fuzzy to accommodate multiple implementations,Collection-level aggregation,Collection-level metadata describes buckets

13、supported by the collection item-level metadata mappings statistical overviews item counts spatiotemporal coverage histograms Example (de-XML-ized) in collection foo, the Originator bucket is supported and the following item fields are mapped to it: (FGDC, 1.1/8.1, “Citation/Originator”) 973 items (

14、USGS DOQ, PRODUCER, “Producer”) 973 items (DC, Creator, “Creator”) 1249 items unknown 6 items,Searching collections,Bucket-level uniform across all collections example search all collections for items whose Originator bucket contains the phrase “geological survey” Field-level collection-specific but

15、 discovery and invocation mechanisms are uniform functionally equivalent to searching the entire bucket plus additional constraint example search collection foo for items whose FGDC 1.1/8.1 field within the Originator bucket contains the phrase,Bucket types (1/7),6 bucket types: spatial, temporal, h

16、ierarchical, textual, qualified textual, numeric Type captures the portion of the bucket definition that has functional implications data type & syntactic representation query terms query operators Complete bucket definition name semantic definition bucket type,Bucket types (2/7),Spatial data type:

17、any of several types of geometric regions defined in WGS84 latitude/longitude coordinates syntax: defined by ADEPT query terms: WGS84 box or polygon operators: contains, overlaps, is-contained-in example query:,Bucket types (3/7),Temporal data type: calendar date or range of calendar dates syntax: I

18、SO 8601 query term: range of calendar dates operators: contains, overlaps, is-contained-in example query: ,Bucket types (4/7),Hierarchical data type: term drawn from a controlled vocabulary (thesaurus, etc.) one-to-one relationship between hierarchical buckets and vocabularies query term: vocabulary

19、 term operator: is-a example query: ,Bucket types (5/7),Textual data type: text query term: text operators: contains-all-words, contains-any-words, contains-phrase example query: ,Bucket types (6/7),Qualified textual data type: text with optional associated namespace query term: same query operator:

20、 matches example query: ,Bucket types (7/7),Numeric data type: real number query term: real number query operators: standard relational operators example query: ,Bucket types vs. buckets,Bucket types are defined architecturally Buckets in use are defined by collections and items need standard bucket

21、s, defined conventionally, to support cross-collection uniformity ADL core buckets simple; universal; easily useful Bucket descriptions in the following slides: bucket type semantic definition effective treatment of multiple values in searching comparison to Dublin Core,ADL core buckets (1/6),Subjec

22、t-related text Title Assigned term Originator Geographic location Coverage date Object type Feature type Format Identifier,ADL core buckets (2/6),Subject-related text type: textual description: text indicative of items subject not necessarily from controlled vocabularies superset of Title and Assign

23、ed term multiple values: concatenated compare: DC.SubjectTitle type: textual description: items title subset of Subject-related text multiple values: concatenated compare: DC.Title,ADL core buckets (3/6),Assigned term type: textual description: subject-related terms from controlled vocabularies subs

24、et of Subject-related text multiple values: concatenated compare: qualified DC.SubjectOriginator type: textual description: names of entities related to items origin multiple values: concatenated compare: DC.Creator + DC.Publisher,ADL core buckets (4/6),Geographic location type: spatial description:

25、 subset of Earths surface related to item multiple values: union compare: DC.Coverage.SpatialCoverage date type: temporal description: calendar dates related to item multiple values: union compare: DC.Coverage.Temporal,ADL core buckets (5/6),Object type type: hierarchical vocabulary: ADL Object Type

26、 Thesaurus (image, map, thesis, sound recording, etc.) multiple values: unioned compare: DC.TypeFeature type type: hierarchical vocabulary: ADL Feature Type Thesaurus (river, mountain, park, city, etc.) multiple values: unioned compare: none,ADL core buckets (6/6),Format type: hierarchical vocabular

27、y: ADL Object Format Thesaurus (loosely based on MIME) multiple values: union compare: DC.FormatIdentifier type: qualified textual description: unique identifiers for item multiple values: treated separately compare: DC.Identifier,Bucket Summary,Strongly typed, abstract metadata category, with defin

28、ed search semantics, to which source metadata is mapped.Supports discovery/search across distributed, heterogeneous collections that use metadata structures of their choosing.Supports “drill-down” searching of item-level metadata elements.,Challenges,Metadata is like life: refuses to follow the rule

29、s unknown semantics inconsistent typing/syntax unknown or unidentifiable sources poor/inconsistent quality proliferation of overlapping vocabularies . Reality check: Dublin Core won adapt buckets to qualified Dublin Core incorporate fallback mechanism or polymorphism e.g, treat fields as thesauri/co

30、ntrolled vocabularies or as text,Query Translation,ADEPT query language,Domain collection of items item = (unique ID, field, ) field = (name, value) bucket = (name, union or concatenation of fields) Queries atomic constraint: (attribute name, operator, target) return items that have at least 1 value

31、 for the attribute, for which at least one value matches the target arbitrary Boolean combinations AND, OR, AND NOT,The problem,Algorithmically translate ADEPT queries to SQL accommodate all possible SQL implementations configurable by mere mortals generate “reasonable” SQL make up for DB deficienci

32、es stupid things like order of tables & conditions matter incorporate optimizer hints and directives,Approach,Python-based translator 1500 lines Extensible “paradigms” describe atomic translation techniques 15 paradigms Each paradigm 100 lines (50 Python code, 20 assertions, 30 documentation) Intrin

33、sic etc. Configuration file describes: buckets, fields, paradigms, paradigm configuration Boolean override rules misc: external identifier table, optimizer clauses,Collection Discovery,The problem,Distributed queries: necessary evil necessary to achieve scalability performance autonomy introduce sca

34、lability, performance, and reliability problemsAmelioration strategies increase server performance/reliability replication, DIENST connectivity regions turn into offline problem Web search engines, OAI harvesting model identify relevant collections to query (ADEPT) analogous to Web search engine,App

35、roach,Build on collection-level metadata spatial & temporal density histograms item counts per collection categorization schemesUpload periodically to central server Use Euler histograms to support range queries,Challenges,Relevance not necessarily Boolean worldwide, petabyte, 1cm resolution database vs. world map drawn on napkin weight by resolution or minimum feature size but sometimes you want the napkinThe JOIN problem statistics computed independentlyText overviews STARTS?,Thats All, Folks,