1、Survey Instrument Development in OB/HRM Research,Prof. Jiing-Lih Larry Farh HKUSTIACMR Guangzhou workshopJuly 2007,2,Too many constructs Constructs are poorly defined Measures do not match constructs Unreliable/invalid measures Level of measurement does not match the level of the theory Fatal flaws
2、for empirical papers!,Construct and Measurement Related Problems in Manuscripts,3,“The construction of the measuring devices is perhaps the most important segment of any study. Many well-conceived research studies have never seen the light of day because of flawed measures.” Schoenfeldt, 1984,“The p
3、oint is not that adequate measurement is nice. It is necessary, crucial, etc. Without it we have nothing.” Korman, 1974, p. 194,“Validation is an unending process.Most psychological measures need to be constantly evaluated and reevaluated to see if they are behaving as they should.” Nunnally & Berns
4、tein, 1994, p. 84,4,Empirical Research Model,Independent and dependent variables are identified by X and Y, respectively. The symbol prime, is used to designate that a variable is specified at the conceptual level. Arrows represent the direction of influence or cause. aconceptual relationship; d-emp
5、irical relationship; b1, b2construct validity; c-internal validity,From Schwab (1999),5,Validity in Research,Construct validity is present when there is a high correspondence between the scores obtained on a measure and the mental definition of a construct it is designed to represent. Internal valid
6、ity is present when variation in scores on a measure of an independent variable is responsible for variations in scores on a measure of a dependent variable. External validity is present when generalizations of findings obtained in a research study, other than statistical generalization, are made ap
7、propriately.,6,Construct Validation,Involves procedures researchers use to develop measures and to make inferences about a measures construct validity It is a continual process No one method alone will give confidence in the construct validity of your measure,7,Construct Validation Steps,From Schwab
8、 (1999),Content validity Factor analysis Reliability Criterion-related/ Convergent/ Discriminant/ Nomological validity,8,Why is it important? How to do it? What are some of the best practices?,Survey Instrument Development,9,Instrumentation in Perspective,Selection and application of a technique tha
9、t operationalizes the construct of interest e.g., physics = colliders e.g., MDs = MRI e.g., OB = Job descriptive index Instruments are devices with their own advantages and disadvantages, some more precise than others, and sophistication doesnt guarantee validity,10,Survey Instruments,3 most common
10、types of instrumentation in social sciences Observation Interview Survey instrumentation Survey instrumentation Most widely used across disciplines Most abused technique-people designing instruments who have little training in the area,11,Why do we do surveys?,To describe the populations: What is go
11、ing on? Theoretical reasons: Why is it going on? Develop and test theory Theory should always guide survey development and data collection,12,What construct does this scale measure? (1),Have a job which leaves you sufficient time for your personal or family life. (.86) Have training opportunities (t
12、o improve your skills or learn new skills). (-.82) Have good physical working conditions (good ventilation and lighting, adequate work space, etc.). (-.69) Fully use your skills and abilities on the job. (-.63) Have considerable freedom to adapt your own approach to the job. (.49) Have challenging w
13、ork to do-work from which you can get a personal sense of accomplishment. (.46) Work with people who cooperate well with one another.(.20) Have a good working relationship with your manager.(.20),Adapted from Heine et al. (2002),13,What construct does this scale measure? (2),I would rather say “no”
14、directly, than risk being misunderstood. (12) Speaking up during a class is not a problem for me. (14) Having a lively imagination is important to me. (12) I am comfortable with being singled out for praise or rewards. (13) I am the same person at home that I am at school. (13) Being able to take ca
15、re of myself is a primary concern for me. (12) I act the same way no mater who I am with. (13) I prefer to be direct and forthright when dealing with people I have just met. (14) I enjoy being unique and different from others in many respects. (13) My personal identity, independent of others, is ver
16、y important to me. (14) I value being in good health above everything. (8),Adapted from Heine et al. (2002),14,Example: Computer satisfaction,15,Construct Definition,Personal computer satisfaction is an emotional response resulting from an evaluation of the speed, durability, and initial price, but
17、not the appearance of a personal computer. This evaluation is expected to depend on variation in the actual characteristics of the computer (e.g., speed) and on the expectations a participant has about those characteristics. When characteristics meet or exceed expectations, the evaluation is expecte
18、d to be positive (satisfaction). When characteristics do not come up to expectations, the evaluation is expected to be negative (dissatisfaction).,From Schwab (1999),16,Hypothetical Computer Satisfaction Questionnaire,Decide how satisfied or dissatisfied you are with each characteristic of your pers
19、onal computer using the scale below. Circle the number that best describes your feelings for each statement.,My satisfaction with:,17,Construct Validity Challenges,From Schwab (1999),18,Scale Development Process,Step1: Item Generation Step 2: Questionnaire Administration Step 3: Initial Item Reducti
20、on Step 4: Confirmatory Factor Analysis Step 5: Convergent/Discriminant Validity Step 6: Replication,From Hinkin (1998),19,Step 1: Item Generation -Deductive Approach,It requires: (a) an understanding of the phenomenon to be investigated; (b) thorough review of the literature to develop the theoreti
21、cal definition of the construct under examination,From Hinkin (1998),20,Step 1: Item Generation-Deductive Approach,Advantages: through adequate construct definitions, items should capture the domain of interest, thus to assure content validity in the final scale Disadvantages: requires the researche
22、rs to possess working knowledge of the phenomena; may not be appropriate for exploratory studies,From Hinkin (1998),21,Step 1: Item Generation - Inductive Approach,Appropriate when the conceptual basis may not result in easily identifiable dimensions for which items can then be generated Frequently
23、researchers develop scales inductively by asking a sample of respondents to provide descriptions of their feelings about their organizations or to describe some aspects of behavior Responses classified into a number of categories by content analysis based on key words or themes or using a sorting pr
24、ocess,22,Step 1: Item Generation - Inductive Approach,Advantages: effective in exploratory research Disadvantages: Without a definition of construct under examination, it is difficult to develop items that will be conceptually consistent. Requires expertise on content analyses Rely on factor analysi
25、s which does not guarantee items which load on the same factors share the same theoretical construct,23,Characteristics of Good Items,As simple and short as possible Language should be familiar to target audience Keep items consistent in terms of perspectives (e.g., assess behaviors vs. affective re
26、sponse) Item should address one single issue (no double-barreled items) Leading questions should be avoided Negatively worded questions should be carefully constructed and placed in the survey,24,What about these items?,I would never drink and drive for fear of that I might be stopped by the police
27、(yes or no) I am always furious (yes or no) I often lose my temper (never to always) 滿招損，謙受益,25,Content Validity Assessment,Basically a judgment call But can be supplemented statistically Proportion of substantive agreement (Anderson & Gerbing, 1991) (see next slide) Item re-translation (Schriesheim
28、 et al. 1990) Content adequacy (Schriesheim et al. 1993),26,Content Validation Ratio,CVR =,2 n,e,- 1,N,n,e,N,is the number of Subject Matter Experts (SMEs) rating the selection tool or skills being assessed is essential to the job, i.e., having good coverage of the KSAs required for the job.,is the
29、total number of experts,CVR = 1 when all judges believe the tool/item is essential; CVR = -1 when none of the judge believes the tool/skill is essential; CVR = 0 means only half of the judges believe that the tool/item is essential.,27,How many items per construct?,4 - 6 items for most constructs. F
30、or initial item generation, twice as many items should be generated,28,Item Scaling,Scale used should generate sufficient variance among respondents for subsequent statistical analyses Likert-type scales are the most frequently used in survey questionnaire. Likert developed the scale to be composed
31、of five equal appearing intervals with a neutral midpoint Coefficient alpha reliability with Likert scales has been shown to increase up to the use of five points, but then it levels off,29,Step 2: Questionnaire Administration,Sample size: Recommendations for item-to-response ratios range from 1:4 t
32、o 1:10 for each set of scales to be factor analyzed e.g., if 30 items were retained to develop three measures, a sample size of 150 observations should be sufficient in exploratory factor analyses. For confirmatory factor analysis, a minimum sample size of 200 has been recommended.,30,Step 3: Initia
33、l Item Reduction,Interitem correlations of the variables to be conducted first. Corrected item-total correlations smaller than 0.4 can be eliminated Exploratory factor analysis. An appropriate loadings greater than 0.40 and /or a loading twice as strong on an appropriate factor than on any other fac
34、tor. Eigenvalues of greater than 1 and a scree test of the percentage of variance explained should also be examined Be aware of construct deficiency problems in deleting items,31,Step 3: Internal Consistency Assessment,Reliability is the accuracy or precision of a measuring instrument and is a neces
35、sary condition for validity Use Cronbachs alpha to measure internal consistency. 0.70 should be served as minimum for newly developed measures.,32,Coefficient alpha,The average of all possible split halve reliabilities.,33,An example of coefficient ,Variance of total = 7.0 ; Total of variance = 3.67
36、,34,How High Cronbach Alpha Needs to be?,In exploratory research where hypothesized measures are developed for new constructs, the Alphas need to exceed .70 In basic research where you use well-established instruments for constructs, the Alphas need to exceed .80. In applied research where you need
37、to make decisions based on the measurement outcomes, the Alphas need to exceed .90.,35,Step 4: Confirmatory Factor Analysis (CFA),Items that load clearly in an exploratory factor analysis may demonstrate a lack of fit in a multiple-indicator measurement model due to lack of external consistency It i
38、s recommended that a Confirmatory Factor Analysis be conducted using the item variance-covariance matrix computed from data collected from an independent sample. Then assess the goodness of fit index, t-value, and chi square,36,Step 5: Convergent/Discriminant Validity,Convergent validitywhen there i
39、s a high correspondence between scores from two or more different measures of the same construct. Discriminat validity-when scores from measures of different constructs do not converge. Multitrait-Multimethod Matrix (MTMM) Nomological networks-relationships between a construct under measurement cons
40、ideration and other constructs. Criterion-related validity,37,Convergent Validity,Construct,Measure B,Measure A,From Schwab (1999),38,Step 6: Replication,Find an independent sample to collect more data using the measure. The replication should include confirmatory factor analysis, assessment of inte
41、rnal consistency, and convergent, discriminant, and criterion-related validity assessment,39,Elements of a MTMM matrix,40,A sample MTMM matrix,Heterotrait-monomethod,Monotrait-monomethod,Monotrait-heteromethod,Heterotrait-heteromethod,Adapted from http:/ SE: self esteem; SD: self disclosure; LC: Loc
42、us of control,(Paper & Pencil self test),41,Interpreting MTMM,Reliability (monotrait-monomethod) should be the highest Monotrait-heteromethod (convergent validity) must be 0 and high Monotrait-heteromethod (convergent validity) heterotrait-monomethod (discriminant validity) heterotrait-heteromethod
43、(i.e., convergent validity should be higher than discriminant validity),42,Inductive Example: Taking Charge (Morrison & Phelps, 1999, AMJ),Open-end survey to 148 MBA to list 152 individuals efforts, and collected 445 statements. Reduce the list to 180 by eliminating redundant and ambiguous ones, and
44、 sort the statements into 19 groups based on similarity. Write a general statement to reflect each group, compare the content of the statements with the construct, and result in 10 prototypical activities to reflect the construct. Pretest it to 20 MBA students, to check for clarity and suggestion fo
45、r wording improvements Pretest the measure with a sample of 152 working MBAs, to assess internal consistency of the items and check whether the 10 specific behaviors were extra-role activities. 77% checked six or more.,43,Open-ended Survey: Taking Charge (Morrison & Phelps, 1999, AMJ),To think of in
46、dividuals with whom they had worked who have actively tried to bring about improvement within their organization. These change efforts could be aimed at any aspect of the org, including the persons job, how work was performed within their dept, and orgal policies or procedures. To focus on efforts t
47、hat went beyond the persons formal role or, efforts that were not required or formally expected. To list specific behaviors that reflected or exemplified the persons change effort.,44,Sample Items: Taking Charge (Morrison & Phelps, 1999, AMJ),Try to institute new methods that are more effective Try
48、to introduce new structure, technologies, or approach to improve efficiency Try to change how his/her job is executed in order to more effective Try to bring about improved procedures for the work unit or department,45,Theoretical Model: Taking Charge (Morrison & Phelps, 1999, AMJ),Top management openness,Group norms,Self-efficacy,Felt responsibility,Expert power,Taking charge,46,Deductive Example: Org. Justice (Colquitt, 2001, JAP),The Dimensionality of Organizational Justice,Organizational Justice,Distributive Justice,Procedural Justice,Interactive Justice,
