1、Monitoring and Evaluation: Indicators,M&E Indicators: Module Objectives,At the end of the session, participants will be able to: Critique indicators Identify criteria for selection of sound indicators Understand how indicators are linked to the frameworks covered in the Frameworks Module Select indi
2、cators and complete an Indicator Reference Sheet,An Indicator is,a variable that measures one aspect of a program/project or health outcomeAn appropriate set of indicators includes at least one indicator for each significant aspect of the program or project (i.e. at least one per box in an M&E frame
3、work),Characteristics of Good Indicators,Valid: accurate measure of a behavior, practice or task Reliable: consistently measurable in the same way by different observers Precise: operationally defined in clear terms Measurable: quantifiable using available tools and methods Timely: provides a measur
4、ement at time intervals relevant and appropriate in terms of program goals and activities Programmatically important: linked to a public health impact or to achieving the objectives that are needed for impact,Indicator measures what it is supposed to measure Direct measures Proxy measures Straightfo
5、rward interpretation: change in value signals a change in focal concept or behavior,Characteristics of Good Indicators: Valid,Accurate measure of a behavior, practice or task,Validity: Class Activity,Is ideal family size a valid indicator of fertility demand? Is the maternal mortality ratio a valid
6、indicator of the impact of a family planning program on womens health? Is “% Willing to buy vegetables from HIV+ shopkeeper” a valid measure of AIDS stigma? Is “More than one non-marital partner in past year” a valid measure of “risky sex”?,Consistently measurable in the same way by different observ
7、ersTypes of measurement error Sampling Error: over-representation of urban populations because access is easier Non-Sampling Error: survey estimates of abortion incidence, due to response bias Subjective Measurement: indicators that ask for personal judgment such as “quality,” “environment” and “pro
8、gress”,Characteristics of Good Indicators: Reliable,Operationally defined in clear termsActivity: Develop definitions for New user Knowledge of AIDS Quality of care Trained provider,Characteristics of Good Indicators: Precise,Quantifiable using available tools and methodsExamples of measures Days of
9、 drug stock outs Total Fertility Rate Indicators related to Poverty Reduction Strategy,Characteristics of Good Indicators: Measurable,Provides a measurement over periods of time of interest with data available for all appropriate intervalsTimeliness Considerations Reporting schedules Recall periods
10、Survey schedules Length of time over which change can be detected,Characteristics of Good Indicators: Timely,Linked to a public health impact or to achieving the objectives needed for impact Example: Condom distribution program Indicator: # of sex partners in past 4 weeks Example: Program to increas
11、e access to oral rehydration salt for childhood diarrhea through community based distributors Indicator: # of ORS packets distributed in past month,Characteristics of Good Indicators: Programmatically Important,Factors to Consider When Selecting Indicators,Logic/link to framework Programmatic needs/
12、information for decision making Resources External requirements (government, donor, headquarters) Data availability Standardized indicators,Operationalizing Indicators,To operationalize an indicator is to identify how a given concept or behavior will be measured:Challenges Subjective judgment Local
13、conditions Unclear yardsticks,Common Indicator Metrics,Counts Number of providers trained Number of condoms distributed Calculations: percentages, rates, ratios % of facilities with trained provider Maternal mortality ratio, Total fertility rate Index, composite measures Quality index comprising the
14、 sum of scores on six quality outcome indicators DALY (Disability Adjusted Life Years) Thresholds Presence, absence Pre-determined level or standard,Anatomy of an Indicator Metric,Indicator 1: # of ANC care providers trained “providers” include any clinician providing direct clinical services to cli
15、ents seeking ANC at public health facilities “trained” refers to attending every day of a five-day training course offered by the program Indicator 2: % of facilities with a provider trained in ANC Numerator: # of public facilities with a provider who attended all 5 days of ANC training offered by t
16、he program Denominator: total # of public facilities providing ANC services,Always Specify the Details!,Threshold indicator for post-abortion family planning Facility provides post-abortion FP if 90% PAC clients received FP counseling 90% PAC clients who want no more children were informed about ste
17、rilization services Facility has all of 6 relevant IEC materials,Composite Indicators,How to measure, “Infection Prevention at facilities providing obstetric care” ? Could construct an index, have various measurement options: “Facility Readiness,” from facility audit Has regulation sharps container
18、Up to date guidelines on hazardous and medical waste disposal Leak-proof lidded container for medical waste, etc. Knowledge, from Provider interview Practice, from Provider-client observation,Issues with Composite Indicators,How to “scale” the index Weighted v. un-weighted components Value of each c
19、omponent: yes/no, present/absent, count of sub-components (e.g., drugs in stock) Zero/one total indicator value? % of maximum possible score? At least 8 out of 10 positive (=1) scores? How to interpret and report?,Example of Composite Indicator: Proper Disposal of Medical Waste Percent of Facilities
20、,Sources of Indicators: Using Pre-Defined Indicators,From past years of the program From related or similar programs From lists of global or recommended indicators Millennium Development Goals HIV/AIDS: UNAIDS/WHO Reproductive Health: Compendium of Indicators for Evaluating Reproductive Health Progr
21、ams Poverty Reduction Strategy Goals (World Bank),Indicator Pyramid,District or Facility Identify progress, problems, and challenges,National/Sub-national Assess effectiveness of response Reflect goals/objectives of national/sub-national response,Global Compare countries Overview world-wide situatio
22、n,Indicator Matrix,Indicator Reference Sheet,Compile detailed documentation for each indicator: Basic information Description Plans for data collection Plans for data analysis, reporting and review Data quality issues Performance data table (baseline and targets),Indicator Strengths & Limitations,Al
23、l indicators have limitations, even those commonly used: Low birthweight: cannot use in rural areas where few births are in facilities Sexual behavior (e.g. condom use, number of partners): self reporting bias Maternal Mortality Ratio: maternal death is infrequent event Couple Years Protection: info
24、rmation on program activity but not on number of people benefiting,Setting Indicator Targets: Useful Information Sources,Past trends Client expectations Donor expectations Expert opinion Research findings What has been accomplished elsewhere,Common Pitfalls in Indicator Selection,Indicators not link
25、ed to program activities Poorly defined indicators Indicators that do not currently exist and cannot realistically be collected Process indicators to measure outcomes & impacts Indicators that are not very sensitive to change Too many indicators,Pitfalls with Selecting Indicators,Indicator not linke
26、d to program activities,IR: Expanded access to AIDS prevention & treatment services Activities: train providers in current clinical protocols Inappropriate Indicator: % of facilities with adequate conditions to provide care Better indicators: # of clinicians trained, # of facilities with a trained p
27、rovider,The program is not aiming to affect facility conditions, only provider skills.,Pitfalls with Selecting Indicators,Indicator poorly defined,Activity: radio campaign to dispel myths about AIDS transmission and prevention Inappropriate Indicator: % of population with AIDS knowledge Better indic
28、ators: % of population with no incorrect beliefs about AIDS (defined as: % correctly rejecting the two most common local misconceptions about AIDS and who know that a healthy-looking person can transmit AIDS),AIDS knowledge does not indicate what is desirable: knowledge that AIDS exists? How it is t
29、ransmitted?,Pitfalls with Selecting Indicators,Data needed for indicator not available,Inappropriate Indicator: % of days per quarter that service delivery points have stock-out of drugs Data issue: Information on stock-outs may not be collected daily Better indicators: % of service delivery points
30、that had a stock out of drugs at some time during the last quarter,If relying on routine data, indicator definition must depend on how data are collected.,Pitfalls with Selecting Indicators,Indicator does not accurately represent desired outcome,IR: Expanded access to antiretroviral treatment (ARVs)
31、 for pregnant women to prevent mother to child transmission of HIV Inappropriate Indicators: % of women on ARVs who are pregnant; % of people on ARVs who are pregnant women Better indicator: % of HIV positive pregnant women who are on ARVs,What does it mean if inappropriate indicators increase? Decr
32、ease? Do they reflect the desired program effect?,Indicator systems - How much is enough?,Rule of thumb At least one or two indicators per key activity or result (ideally, from different data sources) At least one indicator for every core activity (e.g., training, BCC, CSM) No more than 8-10 indicat
33、ors per area of significant program focus Use a mix of data collection strategies/source,Not everything that can be counted counts, and not everything that counts can be counted. Albert Einstein,References,Bertrand, Jane T., Magnani, Robert J, and Rutenberg, Naomi, 1996. Evaluating Family Planning P
34、rograms, with Adaptations for Reproductive Health, Chapel Hill, N.C.: The EVALUATION Project. Bertrand, Jane T. and Escudero Gabriela, 2002. Compendium of Indicators for Evaluating Reproductive Health Programs, vols. 1 and 2, Chapel Hill, N.C.: MEASURE Evaluation. Tsui, Amy. 1998. Frameworks (ppt).
35、Presented at the Summer Institute, University of North Carolina, Chapel Hill. Tsui, Amy. 1999. Frameworks (ppt). Presented at the Summer Institute, University of North Carolina, Chapel Hill. UNICEF. 1998. State of the Worlds Children. USAID/Tanzania Country Strategic Plan, 2005-2014. WHO, 1999. The
36、Evolution of Diarrhoeal and Acute Respiratory Disease Control at WHO: Achievement 1980-1995 Research, Development and Implementation (WHO/CHS/CAH/99.12).,Group Project,Form groups For your project: Identify indicators and define metrics If frameworks are not finished, continue working on frameworks For two indicators, complete indicator reference sheet,
