NCJ Number
214552
Journal
The Annals Volume: 587 Dated: May 2003 Pages: 49-68
Date Published
May 2003
Length
20 pages
Annotation
This article proposes that methodological quality standards for evaluation research be based on statistical conclusion validity, internal validity, construct validity, external validity, and descriptive validity.
Abstract
The author suggests the development of a new methodological quality scale based on internal validity, descriptive validity, statistical conclusion validity, construct validity, and external validity. Statistical conclusion validity is concerned with whether the presumed cause (the intervention) and the presumed effect (the outcome) are related. Internal validity refers to the correctness of the key question about whether the intervention actually caused a change in the outcome; it has generally been regarded as the most important type of validity in an evaluation. Construct validity pertains to the adequacy of the operational definition and measurement of the theoretical constructs that underlie the intervention and the outcome. External validity refers to the generalizability of causal relationships across different persons, places, times, and operational definitions of interventions and outcomes. Descriptive validity refers to the adequacy of the presentation of key features of an evaluation in a research report. The author also reviews methodological quality scales, with attention to the type of scale that can assist in making decisions about what evaluations to include and exclude in systematic reviews. The article concludes with an assessment of the validity of Pawson and Tilley's (1997) challenge to the Campbell evaluation tradition. They argue that past evaluation research has failed because of its focus on what works. Instead, they argue, researchers should investigate context-mechanism-outcome configurations. These configurations should be examined with qualitative, narrative, ethnographic research that focuses on people's choices. They also indicate that the purpose of evaluations is to test theories. The author concludes that Pawson and Tilley's arguments do not have any implications for methodological quality standards. 41 references