Methods Matter: Improving Causal Inference in Educational and Social Science Research

Citation

Murnane, R. J., & Willett, J. B. (2011). Methods Matter: Improving Causal Inference in Educational and Social Science Research. Oxford University Press.

Chapter Summary

The book has fourteen chapters. Its authors, both at the Harvard Graduate School of Education, say it grew out of an advanced doctoral seminar, and that their approach throughout is to teach each method inside a substantive context, using published empirical studies from education and other fields to introduce, explain and illustrate it. They also say they chose conceptual, graphical and data-based explanations over intensive mathematics, so that scholars without deep technical training can use the methods.

Chapters 1–3: The Challenge for Educational Research; The Importance of Theory; Designing Research to Address Causal Questions

The opening chapters set the problem. The authors' stated motivation is that much of the quantitative research meant to inform education policy cannot support credible statements about cause and effect. Chapter 2 argues for the role of theory, and Chapter 3 turns to designing research so that a causal question can actually be answered.

Chapters 4–7: Investigator-Designed Randomized Experiments; Challenges in Designing, Implementing, and Learning from Randomized Experiments; Statistical Power and Sample Size; Experimental Research When Participants Are Clustered Within Intact Groups

These four chapters cover experiments that the investigator designs. They move from the logic of randomization to the difficulties of carrying out and interpreting such experiments, then to statistical power and sample size, and finally to the common situation in schools where whole classrooms or schools, not individual pupils, are assigned to treatments. Group-level randomization is one of the topics the authors single out as newly accessible to non-specialists.

Chapters 8–11: Using Natural Experiments to Provide "Arguably Exogenous" Treatment Variability; Estimating Causal Effects Using a Regression-Discontinuity Approach; Introducing Instrumental-Variables Estimation; Using IVE to Recover the Treatment Effect in a Quasi-Experiment

The middle of the book handles quasi-experimental designs, for cases in which randomization was not possible but some feature of the world produced treatment differences that are plausibly unrelated to the outcome. Chapter 8 introduces natural experiments, Chapter 9 regression discontinuity, and Chapters 10–11 instrumental-variables estimation, first as an idea and then applied to recover a treatment effect in a quasi-experiment.

Chapters 12–14: Dealing with Bias in Treatment Effects Estimated from Nonexperimental Data; Methodological Lessons from the Long Quest; Substantive Lessons and New Questions

Chapter 12 deals with bias when only nonexperimental data are available. Propensity scores are among the methods the authors cover for this kind of data. The final two chapters step back to draw methodological lessons from the extended effort to answer causal questions, and substantive lessons together with the new questions that remain.

Key Concepts

Chapters 1–3: Framing causal questions

• Causal inference: judging whether an intervention produced an outcome, as opposed to merely being associated with it.
• Theory: Chapter 2 is devoted to the importance of theory, that is, an account of why an intervention should work, in framing a research question.

Chapters 4–7: Randomized experiments

• Randomized experiment: assigning participants to conditions by chance so that groups are comparable before treatment.
• Statistical power and sample size: planning how many units are needed to detect an effect.
• Clustered participants: designs in which intact groups, such as classes or schools, are the units that are randomized.

Chapters 8–11: Quasi-experimental designs

• Natural experiment: a situation in which treatment variability arises outside the researcher's control but is arguably exogenous.
• Regression discontinuity: estimating a causal effect from a sharp cutoff that determines who receives a treatment.
• Instrumental-variables estimation (IVE): using a source of variation that affects treatment but not the outcome directly to recover a treatment effect.

Chapters 12–14: Observational data and lessons

• Bias in nonexperimental estimates: the central problem when treatment was not assigned at random.
• Propensity scores: one of the methods the authors cover for adjusting comparisons made with nonexperimental data.

Critical Analysis

Strengths

• Methods taught through real studies. The authors describe their method as embedding each technique in substantive research papers from education and other fields, and as discussing in depth the controversies around numerous studies in the United States and abroad. Readers see designs succeed and fail on actual evidence.
• Conceptual rather than heavily mathematical. Willett states that the book relies on conceptual, graphical and data-based explanations so that scholars without deep technical backgrounds can use the methods; this makes it more approachable than a technical econometrics text.
• A coherent ladder of designs. The table of contents moves from randomized experiments, through natural experiments, regression discontinuity and instrumental variables, to nonexperimental adjustment, so each design is introduced as the answer to a limit of the previous one.
• Origin in teaching. The book grew from an advanced doctoral seminar taught over more than a decade, which suits it to seminar use.

Limitations

• Scope is quantitative and causal. The book addresses one family of questions, whether an intervention caused an outcome. Descriptive, qualitative and theory-building research are outside its subject.
• Assumes statistics background. Even without heavy proofs, the topics (power, clustering, instrumental variables, propensity scores) presuppose a prior course in statistics including regression.
• Dated by its 2011 publication. Methods and software have developed since, so recent debates about these designs will need other sources.

Where It Fits in the Course

Within the educational research methods course, Methods Matter is the specialist treatment of causal inference. Creswell and Guetterman's Educational Research is a step-by-step introduction that covers quantitative, qualitative and mixed-methods designs within one framework, and Cohen, Manion and Morrison's Research Methods in Education is a comprehensive reference and course text across the field. Both give the broad map that Murnane and Willett assume. Bell, Waters and Johnson's Doing Your Research Project and Gary Thomas's How to Do Your Research Project are step-by-step guides for first-time researchers carrying out small projects, a different audience altogether.

Tisdell, Merriam and Stuckey-Peyrot's Qualitative Research is the natural counterweight: where Methods Matter asks whether something worked, qualitative inquiry asks how participants understand and experience it. Reading the two together shows what each design can and cannot claim.

How to Use This Book

The book suits doctoral and advanced master's students, and researchers who read or commission evaluations of educational programmes and policies. A prior course in research methods and in statistics through multiple regression is the sensible preparation.

A practical plan:

• Core: Chapters 1–3 for the framing, then Chapters 4–5 on randomized experiments, which supply the standard against which every other design is judged.
• Design-specific: read Chapter 6 and Chapter 7 when planning a study with a power calculation or with classrooms or schools as the unit; read Chapters 8–11 in order, since instrumental variables are introduced before they are applied.
• Closing: Chapter 12 for nonexperimental data and Chapters 13–14 as a synthesis.

Read each chapter beside the published studies it uses, and ask of every design what assumption must hold for the causal claim to stand. Practising that question on a study from your own field is the best use of the book.