Measuring system performance.ppt

1、Measuring system performance,The library,A system view,Environment,Transformational process,Inputs,Outputs,energy money materials personnel information,products services,U s e r s,System performance measures,recall,precision,relevance,Robert Taylors four levels of question formation,The actual but u

2、nexpressed need for information (the visceral need),Q1,The conscious, within-brain description of the need (the conscious need),Q2,The formal statement of the need (the formalized need),Q3,The question as presented to the infor- mation system (the compromised need),Q4,Taylor, Robert S. 1968. Questio

3、n-negotiation and information seeking in libraries. College & Research Libraries 29(3): 178-194 (May 1968).,System-defined relevance,find health AND feet,The health of the lumber 90% industry in terms of cubic feetof lumber produced,“My feet are killing me.“,Information retrieval process,Question fo

4、rmulation,Relevancy determination,System: Which documentsare relevant to the query?,User: Are these documentsrelevant to my needs?,Defining relevance,System-defined relevance,User-defined relevance,vs.,Objective Often topical. Does it match the query?,Subjective. Situational. Is it useful?,User-defi

5、ned relevance,The effect of lysergic acid diethylamide ingestion on toenail fungus in cloned mice,“My feet are killing me.“,Soothing remedies for aching feet,Controlling the body by controlling the mind- meditative techniques for dealing with pain,Determining topical relevance,Analyze work as to wha

6、t it is about Assign to the document one or more terms from a finite list of topics Users can then search on those topic indicators,Recall,Recall =,No. of relevant documents retrievedTotal no. of relevant documents in the file,Precision,Precision =,No. of relevant documents retrievedTotal no. of doc

7、uments retrieved from the file,Precision vs. Recall,An inverse relationship,As the level of recall rises the level of precision generally declines and vice versa.The Cranfield experiments (1957 & 1962) Cyril Cleverdon, p.i.,Precision vs. Recall,Subject: sexual dimorphism,Word stemming:,sex sexes sex

8、ual sexy sexier sexiest,Field-specific searches:,DE,TI/sexual()dimorphism,Recall,Precision,Recall,Precision,User-defined relevance,“Relevance appears to be a subjective quality, unique between the individual and a given document supporting the assumption that relevance can only be judged by the info

9、rmation user.“ Miranda Pao,Years later,The effect of lysergic acid diethylamide ingestion on toenail fungus in cloned mice,“My feet are still killing me.“,Soothing remedies for aching feet,Controlling the body by controlling the mind- meditative techniques for dealing with pain,Factors affecting rel

10、evance (1),Purpose of the information Situation of the user Level at which the information source is written Journal of the Amer. Med. Assn. Healthy times,Factors affecting relevance (2),Subject knowledge of the user Is the data new to the user? Does the information relate to the users prior knowled

11、ge? Values - ethical, social, philosophical, political, religious, legal,User-defined relevance,Subjectivity and fluidity make it difficult to use as measuring tool for system performance,Incorporating user-defined relevance into information retrieval systems (1),User performs search System retrieve

12、s results,. . .,Incorporating user-defined relevance into information retrieval systems (2),System asks user if he/she would like to retrieve similar documents Search for other documents with similar word frequencies Search for other documents with same subject descriptors,Search for other documents

13、 with same subject descriptors,Main Author: Title:Subject(s):,Gribbin, John R. In search of Schrodingers cat : quantum physics and reality / by John Gribbin.Schrodinger, Erwin, 1887-1961. Quantum theory History. Reality.,A,A,A,Assisting users in determining relevancy,Indexing terms,Title,Citation da

14、ta,Abstract,Source: Barry, Carol L. 1998. Document representations and clues to document relevance. Journal of the American Society for Information Science 49(14):1293-1303.,Document representation research,Titles,Full text,Title: Getting good grades in graduate school,Title: How to impress your adv

15、isor in graduate school,Title: Writing a dissertation,Title: The well-written graduate paper,Getting good grades in graduate school The best way to get good grades is to study hard,How to impress your advisor in graduate school Never show up late for a meeting with your advisor,Writing a dissertatio

16、n The first thing to do is to pick a topic that truly interests you,The well-written graduate paper Before finalizing your topic do a preliminary search on,How relevant are these?,How relevant are these?,Document representation research,Titles,Citation data,Indexing terms,Abstracts,Full text,Full te

17、xt,Full text,Full text,How relevant are these?,How relevant are these?,Utility studies - Indications that user found relevant materials,Citation & abstract databases User requests citations be formatted for printing User requests citations be sent by e-mail User downloads citations Full-text databas

18、es Pull up the full text Print the article Download the article to their Blackberry,Utility studies - Indications that user found relevant materials,Search,Short list,If user stops may not have found a relevant article,chocolate,Utility studies - Indications that user found relevant materials,Search

19、,Short list,Modifies search,View full citation data for article,View full text of article,Download or print article,Assume that user found article relevant,Characteristics of searches that produce relevant materials,Subject searching Utilization of Boolean operators Search modification Increased tim

20、e in display activities User of greater number of databases,Cooper, Michael Dr. and Hui-Min Chen. 2001. Predicting the relevance of a library catalog search. Journal of the American Society for Information Science and Technology 52 (10):813-827.,Importance of abstract (1),Indication as to depth/scop

21、e of the article Delineates methodology-indication of reliability and validity Gives indication as to content novelty,Authors studied leg-hair count variations of Drosophila in Kawainui Marsh,Random sampling in 40 sectors during March, June, September & December,Greater variation in June,Importance

22、of abstract (2),Basis for research may indicate recency Delineation of results indicates “tangibility“ (important, useful data),American housing market was selected because it is always robust.,Authors concluded that American teenagers listen to rock music.,Types of abstracts,Indicative Informative

23、Critical (evaluative),(Not common in library databases),Indicative abstract,Indicates what the document is about but doesnt report findings,Title: A review of the current literature on relevance.,Abstract: The author reviews the current literature on relevance.,Informative abstract,Acts as a substit

24、ute for the document,Title: The effects of library school on the mental health of library students,Abstract: The authors performed longitudinal studies on 32 graduate students in 8 library and information science programs and found a significant increase in aberrant psychological traits over time.,(

25、fictitious title and abstracts),Abstract creation,Author-produced Vendor-added Automated abstracting,Automated abstracting,Word counts Remove stop words Weight remaining words according to frequency Search for sentences with highest density of most frequently-occurring words,1. Word count,Title: Sea

26、sonal variations in the feral cat population of Fargo,the 81 is 68 a 56 to 42 cats 61 number 45 season 27 winter 11,summer 11 spring 11 fall 11 monthly 10 temperature 61 variation 12 food 10 availability 10,average 9 concept 7 per 8 over 9 immediate 5 implement 3 mortality 8 survival 9,2. Eliminate

27、stop words,Title: Seasonal variations in the feral cat population of Fargo,the 81 is 68 a 56 to 42 cats 61 number 45 season 27 winter 11,summer 11 spring 11 fall 11 monthly 10 temperature 61 variation 12 food 10 availability 10,average 9 concept 7 per 8 over 9 immediate 5 implement 3 mortality 8 sur

28、vival 9,3. Rank by frequency,Title: Seasonal variations in the feral cat population of Fargo,cats 61 temperature 61 number 45 seasonal 27 variation 12 winter 11,summer 11 spring 11 fall 11 monthly 10 food 10 availability 10,average 9 survival 9 mortality 8 concept 7 immediate 5 implement 3,4. Search

29、 for sentences with highest density of high frequency words,Title: Seasonal variations in the feral cat population of Fargo,We found a significant seasonal variation in the number of cats.The highest number of cats are found in the summer, the lowest number of cats in the winter.,Automated abstract,

30、. The Childrens Internet Protection Act (CIPA) sets conditions on public libraries receipt of federal financial assistance for Internet access. . It would not have been possible for the broadcasting station to limit the use of federal funds to all non-editorializing activities. . The instant Court d

31、istinguished Velazquez, restricting its holding to situations in which the grantee is “pitted . . . against the Government. . “ Justice Stevens asserted that the filtering condition was unconstitutional because it distorted the normal usage of library Internet terminals as sources of a wide array of

32、 information. . A condition mandating Internet filters distorts this mission by “denying patrons access to constitutionally protected speech that libraries would otherwise provide. .,Relevance and information overload,In this age of information overload, tools to aid the user in determining relevance are increasingly critical.,