Does Research Design Affect Study Outcomes in Criminal Justice? (From Annals of the American Academy of Political and Social Science, Volume 578, November 2001, P 50-70, 2001, David P. Farrington, Brandon C, Welsh, eds. -- See NCJ-195740)

• NCJ Number: 195743
• Author(s): David Weisburd; Cynthia M. Lum; Anthony Petrosino
• Date Published: 2001
• Length: 21 pages
• Sale Source: Sage Publications, Inc
• Publication Format: Book (Hardbound)
• Publication Type: Report (Study/Research)
• Language: English
• Country: United States of America

Annotation

This article discusses a preliminary study of the question of whether research design affects study outcomes and advises that findings of systematic reviews may be biased by the inclusion of nonrandomized studies.

Abstract

This article examines the issues of the relationship between research design and study outcomes in a broad review of research evidence on crime and justice, as commissioned by the National Institute of Justice. It was found that design does have a systematic effect on outcomes in criminal justice studies. A weaker design, identified by internal validity, will be more likely to report in favor or treatment and less likely to report a harmful effect of treatment. It was found that because randomized studies tend to have more internal consistency, their comparison with quasi-experimental research designs does result in systematic and statistically significant differences. There is a discussion of why randomized experiments are considered the gold standard in research design as compared to quasi-experimental research design. It is explained that internal validity is the primary difference, as effects of other causes are not randomized out in quasi-experimental designs, such as the difficulty in finding control groups that are not systematically different in one respect or another. Also, it is noted that the direction of bias in any particular study is likely to depend on factors related to specific character of the research. This article also discusses the influence of research design on study outcomes across a large group of studies representing all types of research design and a variety of criminal justice areas. A five-point scale, as found in the Maryland Report, was critiqued as being designed to provide an overall measure of the conclusions reached by investigators rather than the actual outcomes of the studies examined. In conclusion, it was found that design does matter and its effect is systematic in criminal justice studies, that systematic reviews of what works in criminal justice may be strongly biased when including non-randomized studies. It is recommended that the Campbell Collaboration should take this into account in coming to conclusions about the effects of interventions. References