Testing Treatments: Better Research for Better Healthcare

Citation

Evans, I., Thornton, H., Chalmers, I., & Glasziou, P. (2011). Testing Treatments: Better Research for Better Healthcare (2nd ed.). Pinter & Martin. Free official edition: https://www.testingtreatments.org/wp-content/uploads/2012/09/TT_2ndEd_English_17oct2011.pdf

Chapter Summary

Chapter 1: New – but is it better?

The opening chapter argues that every treatment, conventional or complementary, needs a fair (unbiased) test, because new treatments turn out worse about as often as they turn out better. It starts with James Lind's 1747 comparison of six scurvy remedies aboard HMS Salisbury, then tells the stories of thalidomide, the painkiller Vioxx, the diabetes drug Avandia and the Björk-Shiley heart valve to show that licensing does not guarantee safety and that harms can take years to surface. A section on Herceptin shows how pressure from campaigners, industry and the press can inflate benefits; one patient describes how a widely quoted "50% benefit" amounted to a gain of 4–5% for her.

Chapter 2: Hoped-for effects that don't materialize

This chapter moves from new treatments to established ones that were adopted on the strength of theory or authority. Its examples are Benjamin Spock's advice to put babies to sleep on their fronts (later linked to many avoidable cot deaths), drugs that suppressed heart-rhythm abnormalities after heart attacks but increased deaths, diethylstilboestrol in pregnancy, hormone replacement therapy, and evening primrose oil for eczema. The key points are that neither theory nor professional opinion is a reliable guide, and that an "established" treatment can still do more harm than good.

Chapter 3: More is not necessarily better

Beginning with the idea of the right dose (low-dose diuretics and aspirin give the benefit; higher doses add only side-effects), the chapter traces a century of increasingly drastic breast cancer treatment: the Halsted radical mastectomy, "extended" radical operations, and high-dose chemotherapy with bone marrow transplantation. A boxed explanation introduces random allocation, and the chapter shows how randomized comparisons eventually established that less mutilating treatment was as effective. It ends by urging doctors and patients to "dare to think about doing less".

Chapter 4: Earlier is not necessarily better

New in this edition, the screening chapter defines screening as a test plus an effective management strategy and insists that offering it to healthy people is itself an intervention. It contrasts clearly useful programmes (newborn screening for phenylketonuria) with cautionary ones: mass neuroblastoma screening of infants, abdominal aortic aneurysm screening, mammography, PSA testing for prostate cancer, chest X-ray screening for lung cancer, commercial genetic tests and whole-body CT scans. Overdiagnosis, false positives and the overselling of benefits are its recurring themes.

Chapter 5: Dealing with uncertainty about the effects of treatments

Here the authors separate the rare treatments whose effects are so dramatic that no formal trial is needed (defibrillation, insulin, laser treatment of portwine stains, imatinib for chronic myeloid leukaemia) from the much more common situation of moderate effects that can only be detected by careful comparison. Cases such as caffeine for breathing problems in premature babies and antibiotics in preterm labour show how unresolved uncertainty persists. The chapter argues that admitting uncertainty is professional, that treatment can be offered within a fair test, and that patients can cope with honest discussion of what is not known.

Chapter 6: Fair tests of treatments

The first of three "technical" chapters explains, without jargon, why comparisons are needed: illnesses often get better on their own, and placebo effects and expectations colour judgement. It develops the principle of comparing like with like, from Thomas Graham Balfour's alternate allocation in 1854 to modern random allocation with concealment of the schedule, and covers crossover and cluster designs, following up everyone allocated, dealing with departures from allocated treatment, and blinded assessment of outcomes. A final section deals with unexpected side-effects, including the UK's Yellow Card reporting scheme.

Chapter 7: Taking account of the play of chance

Using a simple table (4 versus 6 deaths in groups of 10, compared with 40 versus 60 in groups of 100), the chapter shows why the same risk ratio is more convincing when based on larger numbers. It introduces confidence intervals and explains that "statistically significant" does not mean important, illustrating this with aspirin for healthy men, where the estimated benefit for 1,000 men over ten years is close to the number of major bleeds caused. It closes with the need for large multicentre trials.

Chapter 8: Assessing all the relevant, reliable evidence

Since one study is rarely enough, the chapter builds on Austin Bradford Hill's four questions for research reports (why did you start, what did you do, what did you find, what does it mean) to argue for systematic reviews and meta-analysis. It explains how reviews can themselves be biased (unpublished studies, spin, vested interests) and documents the costs of not reviewing evidence first: avoidable harm to patients, harm to research volunteers, and wasted resources. It ends with a journal policy requiring reports of new research to be placed in the context of a systematic review.

Chapter 9: Regulating tests of treatments: help or hindrance?

Also new in this edition, this chapter contends that research regulation forces an artificial split between research and routine treatment: studies of uncertain treatments face heavy oversight while the same treatments can be given outside research with little scrutiny. It examines research ethics committees, information and consent procedures, and proposes a common-sense approach to consent. The authors conclude that regulation adds burdens yet does little to ensure that studies are needed or to follow up approved research.

Chapter 10: Research – good, bad, and unnecessary

Good research is illustrated by the carotid endarterectomy trials for stroke prevention and the Magpie trial of magnesium sulphate for pre-eclampsia. Bad research includes some 2,000 schizophrenia trials that were too small or short and used hundreds of different rating scales, and early epidural trials that rarely measured women's pain. Unnecessary research includes continued placebo-controlled trials of antenatal steroids and of nimodipine for stroke, and repeated aprotinin trials. The chapter closes on distorted priorities: "me-too" drugs, marketing, and a Bristol study in which patients with knee osteoarthritis wanted research on physiotherapy, surgery and coping strategies rather than further drug trials.

Chapter 11: Getting the right research done is everybody's business

This chapter turns to patients and the public as partners in research. It reviews the history of patient involvement (breast cancer activists, AIDS campaigners in the late 1980s), shows how patients can also undermine fair tests (pressure for unproven drugs, conflicts of interest between patient groups and industry), and describes ways of involving citizens in setting research priorities. Its message is that patients need better general knowledge about research and access to impartial information if they are to contribute well.

Chapter 12: So what makes for better healthcare?

The penultimate chapter shows how evidence from fair tests can be combined with individual values through shared decision-making, illustrated by a sample consultation. It answers nine common questions, for example whether anything is worth trying in a life-threatening condition, why relative risks can mislead compared with absolute numbers, whether trial results apply to an individual, what genetic testing can and cannot add, and how to avoid being labelled with a disease. It ends with the Salzburg statement on shared decision-making.

Chapter 13: Research for the right reasons: blueprint for a better future

The closing chapter describes how research money is wasted at four stages (wrong questions, poor design, unpublished results, biased reports) and sets out an eight-point blueprint, from teaching people how to judge claims about treatments to trial registration, full publication and declared conflicts of interest. It finishes with an action plan for readers, such as asking what will happen if nothing is done and being sceptical of headline "breakthroughs".

Key Concepts

Chapters 1–2: New – but is it better?; Hoped-for effects that don't materialize

• Fair test: a comparison of treatments designed to reduce bias and take account of chance, so that differences in outcome can be attributed to the treatments.
• Licensing is not proof of safety: approval by regulators can rest on limited evidence, and adverse effects may only appear after wide use.
• Untested theory: a plausible rationale (for example suppressing arrhythmias) can predict benefit where fair tests later show harm.

Chapters 3–4: More is not necessarily better; Earlier is not necessarily better

• Dose–response trade-off: beyond a certain dose benefits plateau while adverse effects keep rising.
• Screening: a test plus an effective management strategy, justified only if it does more good than harm in a well-run programme.
• Overdiagnosis: detection of disease that would never have caused symptoms, turning healthy people into patients.

Chapter 5: Dealing with uncertainty about the effects of treatments

• Dramatic versus moderate effects: only rarely are effects so large and immediate that controlled comparison is unnecessary; most worthwhile effects are modest.
• Treatment uncertainty: when no one knows which option is better, offering treatment within a fair test is a responsible way to proceed.

Chapters 6–7: Fair tests of treatments; Taking account of the play of chance

• Like with like: comparison groups should differ only in the treatment received, which is best ensured by concealed random allocation before treatment starts.
• Blinding: keeping patients and assessors unaware of allocation so that expectations do not bias the assessment of outcomes.
• Play of chance: random variation that can produce apparent differences, reduced by studying large numbers and expressed with confidence intervals.
• Statistical versus practical significance: a difference unlikely to be due to chance may still be too small to matter to patients.

Chapter 8: Assessing all the relevant, reliable evidence

• Systematic review: a review that uses explicit methods to find, appraise and synthesize all relevant studies of a question.
• Meta-analysis: statistical combination of results from several studies to reduce the play of chance.
• Reporting bias: selective publication of studies or outcomes, often favouring positive results, which distorts the evidence base.

Chapters 9–10: Regulating tests of treatments; Research – good, bad, and unnecessary

• Double standard on consent: clinicians who test uncertain treatments face stricter rules than those who simply prescribe them.
• Unnecessary research: studies whose question had already been answered, which a prior systematic review would have revealed.
• Outcomes that matter to patients: research should measure what patients care about, such as pain relief, rather than convenient proxies.

Chapters 11–13: Getting the right research done; So what makes for better healthcare?; Research for the right reasons

• Patient and public involvement: partnership with patients in choosing research questions and designing studies.
• Shared decision-making: combining the clinician's knowledge of options and evidence with the patient's values and preferences.
• Absolute versus relative risk: presenting results as numbers out of 100 or 1,000 shows the size of benefit more honestly than relative percentages.
• Research waste: loss of research value through wrong questions, poor design, non-publication and biased reporting.

Critical Analysis

Strengths

• Teaches principles through cases. Each idea is anchored in a documented episode (Lind's scurvy test, thalidomide, Vioxx, Spock's sleeping advice, radical mastectomy, neuroblastoma screening), so readers remember why bias and chance matter rather than memorizing definitions.
• Statistics without mathematics. Chapters 6–8 explain random allocation, allocation concealment, blinding, confidence intervals, statistical significance and meta-analysis using small tables and plain language. The aspirin example and the boxed "eye-catching statistics" passage make the difference between relative and absolute risk concrete.
• Patient-centred perspective. Written jointly by clinicians, a researcher and a patient advocate, the book treats patients as partners. Chapters 11–13 give practical guidance (questions to ask, sources to use, an action plan) that is uncommon in methods texts.
• Clear chapter structure. Chapters 1–11 each end with a short "Key points" list (a single point for Chapters 3 and 7), chapters are referenced, and boxed quotations and patient testimonies support the main text. A list of vignettes and an additional-resources section help readers find material.

Limitations

• Dated examples and services. The text was finalized in 2011; the regulatory story of Avandia, screening recommendations, websites (NHS Choices) and figures on research funding reflect that time. Readers should check current guidance on mammography, PSA testing and trial registration rules.
• Strongly UK-centred. Institutions such as NICE, the MHRA, research ethics committees and the NHS dominate the examples, with occasional cases from the USA, Australia and New Zealand; readers elsewhere must translate the regulatory discussion.
• Advocacy rather than neutral survey. The authors argue for positions (for example that research regulation is excessive and that randomized evidence should be the norm). The arguments are well referenced, but competing views on regulation and on observational evidence receive less space.
• No hands-on appraisal skills. The book explains why fair tests matter but does not teach how to read a trial report section by section, calculate effect measures or use critical-appraisal checklists.

Where It Fits in the Course

In a course built around reading research articles, Testing Treatments is the conceptual starting point. It explains why trials compare like with like, why chance and selective reporting matter, and why systematic reviews come before new studies, all in non-technical language. Trisha Greenhalgh and Paul Dijkstra's How to Read a Paper (a short first guide that assumes no statistics) is the natural next step, since it is described as turning these ideas into practical appraisal of different types of paper. Evidence-Based Medicine: How to Practice and Teach EBM by Straus and colleagues (a textbook that assumes some clinical exposure and basic study design) goes further into applying evidence to individual patients and teaching it.

The research articles listed for the course, such as the multicentre observational analysis of inotrope use in cardiac surgery and the studies of perineural glucocorticoids and of intravenous versus oral acetaminophen, are primary research reports. Reading Chapters 6–8 of Testing Treatments first gives students the vocabulary to ask of each article whether like was compared with like, how chance was handled, and whether the outcomes matter to patients. The book does not replace a critical-appraisal manual; it supplies the reasons behind the checklists.

How to Use This Book

The book is written for a general audience as well as for students and clinicians, and it assumes no statistics or research background. Medical, nursing and allied-health students can read it in a few sittings at the start of a course on evidence or research methods.

A practical plan is to read the Introduction and Chapters 1–4 quickly for the case histories, then study Chapters 6, 7 and 8 carefully, since they contain the core methods ideas (comparison groups, random allocation, blinding, chance, systematic reviews). Chapters 5 and 10 are useful for seminar discussion of uncertainty and research waste; Chapters 9 and 11–13 matter most to readers interested in research ethics, patient involvement and shared decision-making. The "Key points" at the end of Chapters 1–11 make convenient revision notes. When reading a research article for the course, try to locate each of the features discussed in Chapter 6 in the article's methods section.

The book has no exercises. The authors point readers to the James Lind Library and to the Testing Treatments Interactive website for further material, and the Additional Resources section lists books and websites on shared decision-making and evaluating evidence.

Edition and Sources

This guide is based on the second edition of Testing Treatments: Better Research for Better Healthcare (Pinter & Martin, 2011; first edition 2006, British Library), read in the free English PDF dated 17 October 2011 published by the authors at https://www.testingtreatments.org/wp-content/uploads/2012/09/TT_2ndEd_English_17oct2011.pdf. It draws on the forewords, preface and introduction, the full table of contents, and all 13 chapters (every chapter opening, all section headings and boxed vignettes, all end-of-chapter key points, and Chapters 1, 6, 7, 12 and 13 together with substantial parts of Chapters 4, 8 and 10 in full).