Topic 7 - Part IMeasurement and Evaluation第七讲： 第一部.ppt

1、Topic 7 : Part I Measurement and Evaluation 第七讲： 第一部份 量度及評估,Paul Gandel Syracuse University Singapore Management University,Methods to Measure and Display Outcomes 量度方法及展示结果,2,Basics of Measurement and Evaluation 量度及评估的基本因素,Defining measurement and evaluation Frameworks for measurement Outcomes-base

2、d evaluation Balance scorecard approach,3,为量度及评估下定义量度架构基于结果的评估平衡计分卡法,Why Measure? 为何要量度？,Reference point Where service was in the past Current situation Future trends Informs Decision makers Managers Staff Sets Priorities,4,参考点 过往的服务 目前的景况 未来的趋势 报告 决策者 经理 员工 设定优次序,Problem 问题,Not too little data NOT

3、THE RIGHT DATA!,5,数据量不宜太少 不是正确的数据！,Measurement vs. Evaluation 量度與评估,Measurement is collecting the raw numbers Evaluation is the determination of the value of the service Evaluation requires a perspective Creating indicators allows for incorporating some context,6,量度是收集原始数目评估是决定服务的价值评估需要一套正确的方法 设定指标时

4、容许加入某些内容,Measurement Framework 量度架构,7,From Joseph Mathews Measuring for Results,Measurement Framework量度架构,Resources 资源,Utilization 效用,Capability 能力,Impact & Effect 影响及效果,Input Measures 投入量度,Output Measures 產出量度,Process Measures 过程量度,Outcomes 结果,Demand,Value 价值,Quality 质量,Feedback,8,Input Measures 投入

5、量度,Measure the capacity/potential Easy to collect and report Examples: Income/Expenditures Staffing Collection (total, growth, recency) Infrastructure (cost, speed, space, number of workstations, etc.) User base statistics,9,量度能力或潜力容易收集及报告 范例: 收入/ 支出 人力资源 馆藏 (总数，增长，最近购买的) 基础设施(成本，速度，空间，工作间数目等) 基于用户的

6、统计,Process/Efficiency Measures 过程 / 效率量度,Are we doing things correctly? Efficiencyhow economical is an activity? Cost per transaction for reference service, document delivery, etc. Productivityfocus on time to complete task or activity. Time to catalog an item, time to process a transaction, etc. Sy

7、stem Reliabilityfocus on percent of uptime or availability of the system,10,我们是否做得正确? 效率 活动是否够经济？每项咨询参考及文献送递等服务的成本生产力 着眼于完成任务或活动的时间。将一项馆藏分类的时间，处理一项服务的时间。 系统可靠性 着眼于正常运作时间的百分比或系统可用性,Output Measures 产出量度,Looks at the use of the Library or a particular service Services. Counts and use per capita statist

8、ics. Annual circulation, attendance, etc Qualitywhat was provided and how was it provided. Lead to measures of satisfaction and value.,11,考虑图书馆或某项专门服务的用量 服务。以人均统计方法计算。书籍流通量，出席率等。 质量 有何服务及如何提供該服务。带出满意度及价值的量度。,Satisfaction Measures 满意度量度,Blanket question = useless information Ask about specific compon

9、ents of the experience Ask about how the user is doing not just how the service is doing Ask about expectations (LIBQUAL) Ask about importancewas this service important to the user?,12,一般的问题= 没用的信息询问某经验的特别部分询问用户的感受如何，而不单只是询问服务如何 了解期望(LIBQUAL)询问重要性 该项服务对用户是否重要?,Priority and Performance 优先次序及表現,What p

10、riority should the library give to the following? How important? How well does the library perform the following services or functions? Usually use a 7 point Likert scale,13,图书馆如何为以下项目定优先次序？ 有多重要？图书馆在以下服务或功能表现如何？通常用 Likert 的 7 级分表,14,Success Metrics: Purpose 成功规范：目的,To establish clear accountability

11、 and clarity of purpose Both measured group and the group for whom measures are collected are clearly stated To create focus and alignment The measures collected are in line with the mission, goals, vision, and values of the library,Partially from Matthews, J. (2004) Measuring for Results: The Dimen

12、sions of Public Library Effectiveness,设立清晰的责任和明确的目的 明确定义被量度的小组及征用资料的小组重点设定及调整 收集的数据应与图书馆的使命、目标、愿景及价值配合,15,Success Metrics: Purpose 成功规范：目的,To enable continuous refinement Measures are reviewed and changed to keep up with changes in environment To provide rigorous indicators Measures are well-defined

13、 using unambiguous language and with as little bias as possible,Partially from Matthews, J. (2004) Measuring for Results: The Dimensions of Public Library Effectiveness,容许持续改善 检讨及改变数据以配合环境转变提供严密的指标 量度指标应用不含糊的语言作严格的定义以尽量减少偏见,16,Balancing Success Metrics 平衡成功规范,Absolute versus relative measures Compar

14、isons add context (but are not always appropriate) Objective versus subjective How much does background of evaluator affect the interpretation of the results? Outcomes over process Focus on outcomes, not on the process Performance versus diagnostic Measure achievements and discover problematic areas

15、 Peer institution benchmarks versus best practice,Partially from Matthews, J. (2004) Measuring for Results,絶对指标与相对指标 Comparisons add context (但并不总是适合)客观与主观 评估人的背景对结果的演绎影响有多深结果与过程 侧重于结果而非过程 表现及诊断法 量度成就及发现有问题的地方同水平学院的标准与优良惯例,17,Types of Performance Indicators 表现指标,Direct versus indirect indicators Mea

16、suring a specific activity vs. measuring a surrogate for an activity (looking at citations as surrogate for measuring value of materials) Lagging versus leading indicators Lagging = actual performance, while leading = prediction of something else Social versus economic indicators Measures indicative

17、 of the impact of the library on a community as compared to the economic importance (and cost) of the library,From Matthews, J. (2004) Measuring for Results: The Dimensions of Public Library Effectiveness,直接与间接指标 量度一项特定的活动与量度一项相关的活动 (考虑以引用次数作为量度实际价值的相关指标)落后的与领先的指标 落后 =实际表现,而领先 =其他预测 社会指标与经济指标 此量度方法表

18、示出与图书馆的经济重要性(及成本)相比图书馆对社会的影响,18,Balanced Scorecard Approach 平衡計分卡,User perspective How do users perceive the library? Is that perspective changing? Internal perspective In which services should the library aim to excel? Information resources perspective What should be in the library collections? Inn

19、ovation perspective How can the library continue to develop new services? Financial perspective How can the library demonstrate the best value for the money invested by the institution in the library?,Based upon: Matthews, J. (2004) Measuring for Results: The Dimensions of Public Library Effectivene

20、ss,用户观感 用户对图书馆有何观感? 他们的观感是否在转变?内部认知 图书馆应着意突出哪几项服务？资源概念 图书馆藏应包含甚么？ 创新观念 图书馆如何继续新服务？财务视角 图书馆如何展示学院对图书馆的投资物有所值？,19,Mission 使命,Process Perspective 流程构面,Financial Perspective 财务构面,Customer Perspective 顾客构面,Balanced Scorecard 平衡计分卡 Library Perspective 以图书馆为例,Information Resources Perspective 资源构面,Organiza

21、tional Dashboards 组织仪表盘,User-friendly tool to display performance Effective and meaningful way to track progress View results: efficiency, effectiveness, and impact Should integrate into the strategic plan a limited number of critical indicators of overall library performance Departments may have th

22、eir own dashboards,20,Source: Butler, L. The Nonprofit Dashboard: A Tool for Tracking Progress (BoardSource, 2007),容易使用的表现展示工具有效及有意义的进度追踪方法 看得见的结果：效率、效力及影响 将图书馆整体表现的某些关键指标纳入策略规划 各部门可有自己的仪表盘,What to measure? Outcomes Mission as spine Strategic initiatives Drivers of success Risk factors Service/resou

23、rce matrixHow to track and display? Combine statistics with visual indicators Graphs and,21,Source: Butler, L. The Nonprofit Dashboard: A Tool for Tracking Progress (BoardSource, 2007),Organizational Dashboards / 组织仪表盘,量度甚么？ 结果 使命是支持因素 策略上的主动权 成功动力 冒险因素 服务/资源矩阵如何追踪及展示将统计数字与看得见的指标结合图形及图表,Relating the

24、 Mission to the Dashboard 将使命与仪表盘联系起来,22,Adapted from: Butler, L. The Nonprofit Dashboard: A Tool for Tracking Progress (BoardSource, 2007),23,Final Step: The Reality Check! 最後的步驟：實際檢驗,Developing measures Class Exercise 设定量度方法 小组研习,Start with a topic area of the library service Think about different

25、 parts of that library services and operations Objectives, Inputs, Output, Efficiency, Quality, Immediate Outcomes, Long-Term Outcomes Consider what could be measured about each part of the process, considering the balances just discussed,24,From Matthews, J. (2004) Measuring for Results: The Dimensions of Public Library Effectiveness,以图书馆服务为一专题开始 考虑图书馆服务的不同部分及运作 目标，投入，产出，效率，质量，实时结果，长远结果 考虑过程的每一部分需量度甚么，考虑之前讨论过的平衡,