1、The Analysis of Population-Based Survey Experiments,Diana C. Mutz University of Pennsylvania,Analysis of Experiments,Simple, straightforward No fancy statistical techniques required Very few questions required Comparison of means (analysis of variance) Many problems result from using observational a
2、nalysis techniques on experimental data People make it more complicated than it needs to be!,The Basics,Well measured Dependent Variable(s) Manipulation check (to ensure that the Independent Variable was successfully manipulated by the experimental treatment),Why Not More?,Causality requires meeting
3、 only 3 conditions:1. Association (The easy part!) 2. Precedence in Time of Independent Variable(We manipulate the Independent Variable) 3. Non-spuriousness of relationship(Random assignment eliminates this problem),The Basics,Well measured Dependent Variable Manipulation checks (to ensure that the
4、Independent Variable was successfully manipulated by the experimental treatment)OPTIONAL: Potential Moderators/Contingent conditions Covariates,An Example,Does Social Trust Influence Willingness to Engage in Online Economic Transactions?,Randomization checks/Balance tests Statistical models for anal
5、ysis Weighting data to population parameters Use and misuse of covariates,Four Issues in Analyses,Two common errors:,Randomization checks/balance tests: They cant tell us what we want to know, and they can lead to inferior model choices Statistical models for analyzing population-based survey experi
6、ments often altogether ignore the fact that they are, indeed, experiments.,The Parameters,We assume. Researcher has control over assignment to conditions Respondents do not undergo attrition differentially as a result of assignment to a specific experimental condition Researcher can ensure that thos
7、e assigned to a given treatment are, in fact, exposed to treatment.,The Parameters,If any one of those 3 requirements is not met, then balance tests can make sense If the randomization mechanism requires pretesting, then balance tests make sense Otherwise, not.,Part I. Balance tests?,Rationales for
8、balance testsCredibility of findings Efficiency of analyses,Origins of this Practice?,Lack of faith in or thorough understanding of probability theory Confusion between frequentist and Bayesian paradigms Mistakenly applying methods for observational analyses to experimental results Field experimenta
9、l literature in which exposure to treatment cannot always be controlled,Credibility of Findings,What does it mean for a randomization to “succeed”? A well-executed random assignment to experimental conditions does not promise to make experimental groups equal on all possible characteristics, or even
10、 a specified subset of them.,Credibility of Findings,“Because the null hypothesis here is that the samples were randomly drawn from the same population, it is true by definition, and needs no data.” (Abelson) Randomization checks are “philosophically unsound, of no practical value, and potentially m
11、isleading.” (Senn) “Any other purpose than to test the randomization mechanism for conducting such a test is fallacious.” (Imai et al.),Credibility of Findings,“p.05” already includes the probability that randomization might have produced an unlikely result Thus experimental findings are credible wi
12、thout any balance tests at all.,Efficiency,Can balance tests profitably inform the analyses of results? What should one do if a balance test fails?,Three proposed “remedies” for failed balance tests,Inclusion of covariates Post-stratification Re-randomization,Inclusion of covariates,Is a failed bala
13、nce test useful for purposes of choosing covariates? Covariates should be chosen in advance, not based on the data. Covariates are chosen for anticipated relationship with the DV; balance tests evaluate the relationship with the IV. So is a balance test informative for model selection?,Inclusion of
14、covariates,NO! If inclusion of a variable as a covariate in the model will increase the efficiency of an analysis, then it would have done so, and to a slightly greater extent, had it not failed the balance test. Thus balance tests are uninformative when it comes to the selection of covariates.,Incl
15、usion of covariates,“Failed” randomization with respect to a covariate should not lead a researcher to include that covariate in the model. If the researcher plans to include a covariate for the sake of efficiency, it should be included in the model regardless of the outcome of a balance test.,Two-s
16、tage analysis: Balance test, then hypothesis test,Changes the appropriate p-value Always excludes X: p1 Always includes X: p2 Not the same p-value that should result after the 2-stage process But most researchers simply report p1 or p2,Why are we doing randomization checks?,If they have no implicati
17、ons for the credibility of our findingsIf they cannot improve the efficiency of our analyses,Why not?,They cant tell us what we want to know They can lead to inferior model choices They can lead to unjustified changes in the interpretation of findings,Part II. Statistical models,Balance tests do not
18、 provide rationales for including additional variables Three examples of model and analysis choices made for the wrong reasons,EXAMPLE 1: “In order to ensure that the experimental conditions were randomly distributedthus establishing the internal validity of our experimentwe performed difference of
19、means tests on the demographic composition of the subjects assigned to each of the three experimental conditions.”,Irrelevant factors often dictate model selection and analysis,“Having established the random assignment of experimental conditions, regression analysis of our data is not required; we n
20、eed only perform an analysis of variance (ANOVA) to test our hypotheses as the control variables that would be employed in a regression were randomly distributed between the three experimental conditions.”,Regressions run amok with survey-experimental findings!,Five dummies for 6 conditions,EXAMPLE
21、2:,Whats the point?,Regression versus analysis of variance is a red herring. So are balance tests. Especially in an experimental analysis, everything needs a reason for being there. True experiments should not have “control” variables! (A few covariates are OK.) The presence of unnecessary variables
22、 in a statistical model should be viewed with suspicion; they can hurt and bias results.,Four Issues in Analyses,Randomization checks/Balance tests Statistical models for analysis Weighting data to population parameters Use and misuse of covariates,Part III. Using Weights,Should population-based exp
23、eriments use population weights supplied by survey houses? Some studies do, some dont; no particular rationale typically given No one correct answer but need to consider: Possibility of heterogeneous effects Power needs Emphasis on generalizability,Possible Weighting Options,No use of weights Weight
24、ing sample as a whole to underlying population parameters Weighting formulated so that individual experimental conditions reflect population parametersEither (1) or (2) benefits through increasing generalizability to full population; (2) is better at reducing noise due to uneven randomization But al
25、l weighting sacrifices power .,Calculating the Loss,If all the full sample weights are squared for a sample of size n, and then summed across all subjects, this sum (call it M1) provides a sense of just how much power is lost through weighting:,If M1 =3000 and n=2000, then the equation will come out
26、 to .33.,=1 1,Weighting in this example lowers power as if we had reduced the sample size by one-third. Instead of a sample of 2000, we effectively have the power of a sample size of 1340.,Calculating the Loss,=1 1,Within-Condition Weights,Calculate via same formula for within-subject Compare loss o
27、f power in within versus whole sample weighting,=1 1,Weighting Recommendations,Request both whole sample and within-condition weights Decision can be made on basis of importance of power relative to generalizability Ultimately depends on expectations about heterogeneity of effects.,Randomization che
28、cks/Balance tests Statistical models for analysis Weighting data to population parameters Use and misuse of covariates,Four Issues in Analyses,Part IV. (In)appropriate Uses of Covariates,Because population-based survey experiments involve survey data, often analyzed as if they were observational stu
29、dies Mistaken use of unnecessary “control” variables,Because population-based survey experiments involve survey data, often analyzed as if they were observational studies Mistaken use of unnecessary “control” variables Not a cure for an unlucky randomization (which isnt necessary in any case) But wh
30、ats the harm? Biased results,Part IV. (In)appropriate Uses of Covariates,Treatment effects and their interactions with other variables,EXAMPLE 3:,But then what are these? “Control variables”,Treatment effects and their interactions with other variables,Appropriate Uses of Covariates,To improve effic
31、iency when selected in advance from pretest measures based on advance knowledge of predictors of dependent variable Better yet, use blocking if equality across conditions on that particular variable is THAT important.,An Embarrassment of Riches,Too many available variables leads to sub-optimal data
32、analysis practices. Researchers need to rely more on the elegance and simplicity of their experimental designs. Equations chock full of “control” variables demonstrate a fundamental misunderstanding of how experiments work. Failed randomization checks should never be used as a rationale for inclusion of a particular covariate.,
