Quasi-Experimentation A Guide to Design and Analysis

Featuring engaging examples from diverse disciplines, this book explains how to use modern approaches to quasi-experimentation to derive credible estimates of treatment effects under the demanding constraints of field settings. Foremost expert Charles S. Reichardt provides an in-depth examination of the design and statistical analysis of pretest–posttest, nonequivalent groups, regression discontinuity, and interrupted time-series designs. He details their relative strengths and weaknesses and offers practical advice about their use. Comparing quasi-experiments to randomized experiments, Reichardt discusses when and why the former might be a better choice than the latter in the face of the contingencies that are likely to arise in practice. Modern methods for elaborating a research design to remove bias from estimates of treatment effects are described, as are tactics for dealing with missing data and noncompliance with treatment assignment. Throughout, mathematical equations are translated into words to enhance accessibility. Adding to its discussion of prototypical quasi-experiments, the book also provides a complete typology of quasi-experimental design options to help the reader craft the best research design to fit the circumstances of a given study.
 

Charles S. Reichardt, PhD, is Professor of Psychology at the University of Denver. He is an elected fellow of the American Psychological Society and an elected member of the Society of Multivariate Experimental Psychology. Dr. Reichardt is a recipient of the Robert Perloff President’s Prize from the Evaluation Research Society and the Tanaka Award for Best Article in Multivariate Behavioral Research. His research focuses on quasi-experimentation.
 

1. Introduction
Overview
1.1 Introduction
1.2 The Definition of Quasi-Experiment
1.3 Why Study Quasi-Experiments
1.4 Overview of the Volume
1.5 Conclusions
1.6 Suggested Reading
2. Cause and Effect
Overview
2.1 Introduction
2.2 Practical Comparisons and Confounds
2.3 The Counterfactual Definition
2.4 The Stable-Unit-Treatment-Value Assumption (SUTVA)
2.5 The Causal Question Being Addressed
2.6 Conventions
2.7 Conclusions
2.8 Suggested Reading
3. Threats to Validity
Overview
3.1 Introduction
3.2 The Size of an Effect
3.3 Construct Validity
3.4 Internal Validity
3.5 Statistical Conclusion Validity
3.6 External Validity
3.7 Trade-offs among Types of Validity
3.8 A Focus on Internal and Statistical Conclusion Validity
3.9 Conclusions
3.10 Suggested Reading
4. Randomized Experiments
Overview
4.1 Introduction
4.2 Between-Groups Randomized Experiments
4.3 Examples of Randomized Experiments Conducted in the Field
4.4 Selection Differences
4.5 Analysis of Data from the Posttest-Only Randomized Experiment
4.6 Analysis of Data from the Pretest–Posttest Randomized Experiment
4.7 Noncompliance with Treatment Assignment
4.8 Missing Data and Attrition
4.9 Cluster-Randomized Experiments
4.10 Other Threats to Validity in Randomized Experiments
4.11 Strengths and Weaknesses
4.12 Conclusions
4.13 Suggested Reading
5. One-Group Posttest-Only Designs
Overview
5.1 Introduction
5.2 Examples of One-Group Posttest-Only Designs
5.3 Strengths and Weaknesses
5.4 Conclusions
5.5 Suggested Reading
6. Pretest–Posttest Designs
Overview
6.1 Introduction
6.2 Examples of Pretest–Posttest Designs
6.3 Threats to Internal Validity
6.4 Design Variations
6.5 Strengths and Weaknesses
6.6 Conclusions
6.7 Suggested Reading
7. Nonequivalent Group Designs
Overview
7.1 Introduction
7.2 Two Basic Nonequivalent Group Designs
7.3 Change-Score Analysis
7.4 Analysis of Covariance
7.5 Matching and Blocking
7.6 Propensity Scores
7.7 Instrumental Variables
7.8 Selection Models
7.9 Sensitivity Analyses and Tests of Ignorability
7.10 Other Threats to Internal Validity besides Selection Differences
7.11 Alternative Nonequivalent Group Designs
7.12 Empirical Evaluations and Best Practices
7.13 Strengths and Weaknesses
7.14 Conclusions
7.15 Suggested Reading
8. Regression Discontinuity Designs
Overview
8.1 Introduction
8.2 The Quantitative Assignment Variable
8.3 Statistical Analysis
8.4 Fuzzy Regression Discontinuity
8.5 Threats to Internal Validity
8.6 Supplemented Designs
8.7 Cluster Regression Discontinuity Designs
8.8 Strengths and Weaknesses
8.9 Conclusions
8.10 Suggested Reading
9. Interrupted Time-Series Designs
Overview
9.1 Introduction
9.2 The Temporal Pattern of the Treatment Effect
9.3 Two Versions of the Design
9.4 The Statistical Analysis of Data When N = 1
9.5 The Statistical Analysis of Data When N Is Large
9.6 Threats to Internal Validity
9.7 Design Supplements I: Multiple Interventions
9.8 Design Supplements II: Basic Comparative ITS Designs
9.9 Design Supplements III: Comparative ITS Designs with Multiple Treatments
9.10 Single-Case Designs
9.11 Strengths and Weaknesses
9.12 Conclusions
9.13 Suggested Reading
10. A Typology of Comparisons
Overview
10.1 Introduction
10.2 The Principle of Parallelism
10.3 Comparisons across Participants
10.4 Comparisons across Times
10.5 Comparisons across Settings
10.6 Comparisons across Outcome Measures
10.7 Within- and Between-Subject Designs
10.8 A Typology of Comparisons
10.9 Random Assignment to Treatment Conditions
10.10 Assignment to Treatment Conditions Based on an Explicit Quantitative Ordering
10.11 Nonequivalent Assignment to Treatment Conditions
10.12 Credibility and Ease of Implementation
10.13 The Most Commonly Used Comparisons
10.14 Conclusions
10.15 Suggested Reading
11. Methods of Design Elaboration
Overview
11.1 Introduction
11.2 Three Methods of Design Elaboration
11.3 The Four Size-of-Effect Factors as Sources for the Two Estimates in Design Elaboration
11.4 Conclusions  
11.5 Suggested Reading
12. Unfocused Design Elaboration and Pattern Matching
Overview
12.1 Introduction
12.2 Four Examples of Unfocused Design Elaboration
12.3 Pattern Matching
12.4 Conclusions
12.5 Suggested Reading
13. Principles of Design and Analysis for Estimating Effects
Overview
13.1 Introduction
13.2 Design Trumps Statistics
13.3 Customized Designs
13.4 Threats to Validity
13.5 The Principle of Parallelism
13.6 The Typology of Simple Comparisons
13.7 Pattern Matching and Design Elaborations
13.8 Size of Effects
13.9 Bracketing Estimates of Effects
13.10 Critical Multiplism
13.11 Mediation
13.12 Moderation
13.13 Implementation
13.14 Qualitative Research Methods
13.15 Honest and Open Reporting of Results
13.16 Conclusions
13.17 Suggested Reading
Appendix: The Problems of Overdetermination and Preemption
A.1 The Problem of Overdetermination
A.2 The Problem of Preemption
References
Glossary
Author Index
Subject Index
About the Author