An Introduction toData Warehousing Concept and Technology.ppt

1、An Introduction to Data Warehousing Concept and Technology,Mort Anvari,March 2, 2002,Data Warehousing Concept Data Access Technology Enterprise Real-Time Knowledge Architecture for Data Warehousing Data Collection and Delivery,Topics,Benson & Parkers “Square Wheel”,Business Environment,Technology En

2、vironment,Business Planning,Business Operations,Benson & Parkers “Square Wheel”,Business Environment,Technology Environment,Business Planning,Business Operations,Technology Planning,Technology Operations,Benson & Parkers “Square Wheel”,Business Environment,Technology Environment,Business Planning,Bu

3、siness Operations,Technology Planning,Technology Operations,Alignment,Impact,Organization,Opportunity,Benson & Parkers “Square Wheel”,Business Environment,Technology Environment,Business Planning,Business Operations,Technology Planning,Technology Operations,Alignment,Impact,Organization,Opportunity,

4、Information Technology has to do more than just align itself with the business, it has to help the business have the maximum impact in the marketplace.,Data Access and Delivery System,Technology Evolution,New classes of computers New classes of communications New classes of technology (image, sound,

5、 video, multimedia) New classes of software Much more complex technical environment Cooperative Processing/Client-Server Distributed Data Bases LANs, WANs, etc. Obsolescence Problem Multiple Legacy Systems,IT Impact on Business,HP,IBM,DEC,Compaq,Enterprise Network Computing and Client/Server Technol

6、ogy are changing the way organizations look at all of their information systems,Data Jail,Obsolescence IT Wastes,The Existing Enterprise,Support Existing Products Support Existing Customers Support Existing Organization Support Existing Workforce Support Existing Technology,Controlling the (Global)

7、Real-time Organization,RTO = 24 x 7 x E,(Where E means every major market),Information and the Enterprise,Organizational needs for data Organizational needs for information Organizational needs for knowledge,Information and the Enterprise,An Insurance IS Executive estimated that his organization cou

8、ld only access something like 1% of all the data on their data base A Bell Labs report has indicated that the amount of data doubles every 5 years, . and they can only use about 5% of it! Data Warehousing is Data Delivery System,Needs for Data,Data = Values (Measurements) Data to operate Data to con

9、trol Data to plan,Needs for Information,Information = Content + Structure (Relationships) Structure of the Real-world Relating data to the business Cross functional processes Relating data to the real world External DB External Data Feeds (D&B, Reuters, etc.) Text, Image, Voice, Video, etc. Statisti

10、cal Studies,Needs for Knowledge,Knowledge = Goals + Actions + Learning Learning more about our business Learning more about our market Learning more about the business environment Knowledge is the area in which Data Warehousing and Data Mining are potentially critical technologies,Data, Information

11、and Knowledge,Data Centers Information Centers Knowledge Centers,Data Bases Information Bases Knowledge Bases,Old Data Never Dies,Note that none of the early computing styles have ever gone away!,Batch On-line Minis PCs Networking Enterprise Computing (Peer to Peer, Network to Network),60s,70s,80s,9

12、0s,Operational vs. Informational Systems,Information Access Today,Operational vs. Informational Systems,Information Access Today,Operational Systems,Mafg.,Ord. Entry,Operational vs. Informational Systems,Information Access Today,Operational Systems,Informational Systems,Operational vs. Informational

13、 Systems,Information Access Today,Operational Systems,Informational Systems,Estimating& Analysis,Marketing Systems,Product Planning,Operational vs. Informational Systems,Information Access Today,Operational Systems,Informational Systems,Information Delivery System,Operational vs. Informational Syste

14、ms,Information Access Today,Operational Systems,Informational Systems,Information Delivery System,Data Warehousing is fundamentally an issue of Enterprise Data Architecture,Operational vs. Informational Systems,Operational Systems,Informational Systems,Information Delivery System,Operational vs. Inf

15、ormational Systems,Operational Systems,Informational Systems,Information Delivery System,Data Warehouse,Operational vs. Informational Systems,Operational Systems,Information Delivery System,Data Warehouse,Informational Systems,Data Marts,Operational vs. Informational Systems,Operational Systems,Info

16、rmation Delivery System,Informational Systems,Data Warehouse,External Data,Data Garages,Operational vs. Informational Systems,End User Evolution,Data Base Management Systems users Ad Hoc Reports users Todays Customer Demands Automated Real-Time Response. End User Systems Decision Support Systems Exe

17、cutive Information Systems Information Centers,Ways to Organize Data,Tables Flexible, Simple Hierarchies Speed, Natural ReportingNetworks Multiple Directions, Complex Structure Lists Updating Complex Structure Matrices / Array Manipulate Multiple Dimensions Inverted Files Unplanned queries, text ret

18、rieval Objects Complex structures, hide structure Multidimensional Data Bases (Data Warehousing),End User Computing Evolution,Data Warehousing,Data Warehouse can be thought of as an automated version of the Information Center that was widely popular in the mid-1980s or even ultimately as the automat

19、ion of Information Resource Management. And while technologies such as client-server have begun to put enormous computing and graphics power in the hands of individuals, however, these technologies have not, in general, provided the link to the operational data that end users need to make critical b

20、usiness decisions.,Data Warehouse Requirements,Support for Universal Access to Multi-platform Data Bases Support for Multiple User Types Separation of Operational and Informational Concerns Support for Networked Data Support for Directories, Repositories and Information Models, Support for Advanced

21、End User Interfaces,Access to Heterogeneous Data,HP,IBM,DEC,Compaq,Multiple User Types (Knowledge workers),Top Executives Managers Analysts Planners Product Developers Consultants Lawyers etc.,Separation of Operational and Informational Concerns,Operational Systems Response Time Reliability Security

22、 Recoverability Informational Systems Flexibility, Performance, Ease of Navigation Large numbers of different views Manage Huge Amounts of Data (VLDBs) Need to drill down/drill thru into data Need to draw on data from many sources,Support for Networked Data,All the data that is required to support i

23、nformational needs is often not on the same operational data base. The need for Labor Negotiations, for example, may come from a variety of operational data bases, such as Manufacturing, Personnel, and Accounting.Distributed Systems,Support for Advanced End User Interfaces,Dimensions of Data Warehou

24、sing,Performance,Flexibility,Scalability,Ease of Use,Quality,Connection to the Operational Data,Distributed Data,Security,Enterprise Knowledge Architecture for Data Warehousing,Operational vs. Informational Systems,Operational Systems,Informational Systems,Information Delivery System,Operational vs.

25、 Informational Systems,Enterprise Network Computer Architecture,Data Mart,Freeing the “Data in Jail”,The Information Access Layer,The Legacy Data Layer,The External Data Layer,The Data Access Layer,The Data Access Layer,Data Access Filter,The Data Access Layer,SQL Queries,The Data Access Layer,SQL Q

26、ueriesSQL Answers,Application Messaging,The Meta-Data Repository Layer,The Process Management Layer,The Core Data Warehouse,Data Staging and Quality,Data Mart (Post-process/Indexing),Post- Proc.& Indexing,Goals of Warehouse,1. Performance (Canned queries, MD Analysis, Ad hoc, Impact on Operational S

27、ystem) 2. Flexibility (MD Flex, Ad hoc, Change data structure) 3. Scalability (No. of Users, Volume of Data) 4. Ease of Use (Location, Formulation, Navigation, Manipulation) 5. Data Quality (Consistent, Correct, Timely, Integrated) 6. Connection to the Detail Business Transactions,Virtual Warehouse,

28、Virtual Warehouse,Virtual Warehouse,A Virtual Data Warehouse approach is often chosen when there are infrequent demands for data and management wants to determine if/how users will use operational data.,Virtual Warehouse,One of the weaknesses of a Virtual Data Warehouse approach is that user queries

29、 are made against operational DBs.One way to minimize this problem is to build a “Query Monitor” to check the performance characteristics of a query before executing it.,Distributed Data Warehouse,Distributed Data Warehouse,A Distributed Data Warehouse is similar in most respects to a Central Data W

30、arehouse, except that the data is distributed to separate mini-Data Warehouses (Data Marts ) on local or specialized servers,Information Access Tools,Desktop DBs Spreadsheets 4GL/Desktop Query Tools Decision Support Systems (DSS) Multi-dimensional DBs (MDDs) OLAP (On-line Analytical Processing Execu

31、tive Information Systems (EIS) Data Visualization Tools Data Mining Tools Business Modeling and Simulation Tools,Data Warehousing Tools and Technology,Desktop Data Bases:Structured for Database Manipulation Provides facility for selecting, and loading of Desktop DBs from Informational DBs Provides a

32、bility to Create Highly “Personalized” Informational SystemsExamples Access Paradox dBase/FoxPro/Clipper,Enterprise Network Computer Architecture,Spreadsheets:Structured to get any subset of Information Ability to Interface with standard Spreadsheet tools (ExamplesExcel1-2-3Quatro Pro,Enterprise Net

33、work Computer Architecture,Ad Hoc Query Systems:Tailored for Flexible Reporting Ability to do Sophisticated Analysis Functions Aimed a a variety of users from casual to the power userExamples Focus for Windows (IBI) SAS Business Objects GQL (Anadyne) Esperant (Software AG) Forrest & Trees (Platinum)

34、 Visualizer (IBM) Impromptu (Cognos) Beacon (Prodea),Enterprise Network Computer Architecture,Multi-dimensional Databases (MDDB) OLAP (On-line analytical processing):Highly Structured Data Tailored for Financial Modeling Tailored for “Power Users” Ability to do Sophisticated Financial “What-if” Anal

35、ysis Ability to “drill-down” from high-level to Detail DataExamplesAcumate (Kenan Tech.)Beacon (Prodea)CrossTarget (Dimensional Insight) eSSbase (Arbor)Oracle Express (Oracle),Enterprise Network Computer Architecture,Executive Information Systems (EIS):Highly Structured Data Tailored for Non-technic

36、al Users Ability to “slice and dice” data Ability to “drill-down”ExamplesCommander OLAP ServerPilot (Lightship)VBPowerbuilder,Enterprise Network Computer Architecture,Data Visualization:Automatic Categorization Visualization of Multi-dimensional dataAutomatic Analysis and/or IndexingExamplesWinViz (

37、IBI)dbExpress (Computer Concepts)Data Explorer (IBM)ARC Info/ARC ViewStrategic Mapping,Enterprise Network Computer Architecture,Data Mining:High Speed Analysis of Detail Data Constructs Business Patterns Provides Statistical SupportExamplesIBM beta-testInformation HarvesterIDISd.b.ExpressDataMind,En

38、terprise Network Computer Architecture,Business Modeling and Simulation:Business Feedback Model Direct Manipulation Business Gaming Management/Operations TrainingExamplesSimRefinerySimTelephoneiThinkMicroworlds,3. Meta-data Repository Layer,Data Dictionary/ RepositoryMeta-data ModelingMeta-data Upda

39、tingMeta-dataExampleso Platinumo Rochadeo MSPo Data Atlas (IBM)o MS/TI,3. Process (Systems) Management,Process ManagementSchedulingExecutionSubscriptionExampleso Data Harvestero Data Hubo Detect and Alert (Comshare),3. Post-processing/Indexing Layer,Post-processing/ IndexingExamples Sybase IQ Accelerator OMNIdex Oracle 7.3 eSSbase IRI Express,