MEASURING IMPACT - POWER THEORY IN SOCIAL PROGRAM EVALUATION (FROM IMPROVING EVALUATIONS, 1979, BY LOIS-ELLIN DATTA AND ROBERT PERLOFF - SEE NCJ-64392)

• NCJ Number: 64397
• Author(s): R F BORUCH; H GOMEZ
• Date Published: 1979
• Length: 32 pages
• Sponsoring Agency: US Dept of Health, Education, and Welfare
• Corporate Author: Sage Publications, Inc
• Publication Format: Document
• Publication Type: Program/Project Evaluation
• Language: English
• Country: United States of America

Annotation

THE MEASUREMENT OF TREATMENT AND RESPONSE IN SOCIAL PROGRAMS ARE EXAMINED IN ORDER TO BETTER ESTABLISH THE STATISTICAL POWER OF AN EVALUATION DESIGN, ENHANCE THAT POWER, AND UNDERSTAND THE EVALUATION RESULTS.

Abstract

THE DISCUSSION IS CONFINED TO OUTCOME EVALUATION: ESTIMATING IN THE LEAST EQUIVOCAL AND LEAST BIASED WAY POSSIBLE THE RELATIVE EFFECT OF A PROGRAM ON ITS TARGET GROUP. WEAK EVALUATION DESIGN OFTEN CAUSES THE FINDING OF 'NO SIGNIFICANT DIFFERENCES' IN PROGRAM EVALUATIONS. STATISTICAL POWER THEORY SHOULD THEREFORE BE AUGMENTED TO BETTER REFLECT THE REALITIES OF SOCIAL EXPERIMENTATION. THE PROBLEM OF ENSURING STATISTICAL POWER IN EVALUATION DESIGN RESULTS FROM LOSS OF STATISTICAL POWER WHEN A PROGRAM IS TRANSFERRED FROM THE LABORATORY TO THE FIELD. THIS DROP OCCURS BECAUSE OF THE PARTIAL IRRELEVANCE OF RESPONSE VARIABLES AND THE DEGRADATION OF TREATMENT VARIABLES. INVALIDITY OF RESPONSE AND INFIDELITY OF TREATMENT HAVE A MULTIPLICATIVE, RATHER THAN ADDITIVE, EFFECT ON STATISTICAL POWER. BY AUGMENTING THE USUAL ANALYSIS OF VARIANCE MODEL WITH SIMPLE REGRESSION MODELS WHICH RECOGNIZE VARIATION IN THE FIELD, AND BY COMPUTING THE STATISTICAL POWER FOR BOTH THE SIMPLE AND AUGMENTED MODELS, IT IS EASY TO SHOW THE DROP IN POWER. RESPONSE VARIABLES WHICH ARE RELEVANT TO THE PROGRAM MUST BE USED. SUBSTANTIVE THEORY, PRIOR DATA, PILOT TESTS, AND SIDE THEORIES ARE ALL CRITERIA FOR DETERMINING RELEVEANCE TO TREATMENT OF A RESPONSE VARIABLE. IDENTIFYING AND MEASURING TREATMENT IMPLEMENTATION IN FIELD STUDIES INVOLVES THREE PROBLEMS: POLICY PACKAGED AS PROGRAM, STRUCTURALLY INCOMPLETE PROGRAMS, AND INCOMPLETE PROGRAM RECEPTION OR DELIVERY. PILOT TESTS OR SIDE STUDIES SHOULD BE USED TO PROVIDE INFORMATION ON THE EXTENT OF THESE PROBLEMS. THE PROBLEM MUST ALSO BE CHARACTERIZED ALGEBRAICALLY AS WELL AS LITERALLY. IT IS CONCLUDED THAT (1) ORTHODOX EXPERIMENTAL DESIGN AND QUASI-EXPERIMENTAL APPROACHES NEED TO BE AUGMENTED FOR EVALUATING SOCIAL PROGRAMS; (2) KNOWLEDGE FROM MANY DISCIPLINES SHOULD BE USED; (3) PROGRAM MANAGERS, EVALUATION DESIGNERS, MEASUREMENT SPECIALISTS, AND ACADEMIC THEORISTS SHOULD ALL BE INVOLVED IN EVALUATION DESIGN; AND (4) THE POWER OF DESIGNS SHOULD BE ROUTINELY TESTED AND PUBLISHED WITH EVALUATION RESULTS, BECAUSE SOCIAL PROGRAMS OFTEN PRODUCE SMALL EFFECTS. RESEARCH ON BOTH SUBSTANTIVE THEORY AND STATISTICAL MODELS IS RECOMMENDED. EXAMPLES FROM DIVERSE FIELDS, A REFERENCE LIST, AND AN APPENDIX PRESENTING ALGEBRAICALLY A THEORY OF MEASUREMENT IN FIELD EVALUATIONS, ARE INCLUDED. (CFW)