EVALUATIONS - USE WITH CAUTION

• NCJ Number: 50203
• Journal: EVALUATION Volume: 1 Issue: 2 Dated: (1973) Pages: 30-35
• Author(s): S J MUSHKIN
• Date Published: 1973
• Length: 6 pages
• Corporate Author: Minneapolis Medical Research Foundation, Inc
• Publication Format: Article
• Publication Type: Program/Project Evaluation
• Language: English
• Country: United States of America

Annotation

EVALUATION STUDIES ARE HAVING AN INCREASED IMPACT ON POLICY AT A TIME WHEN THE METHODOLOGY OF EVALUATION IS STILL TOO INADEQUATE TO SERVE AS A GUIDE. A CRITIQUE OF COMMON EVALUATION TECHNIQUES SUGGESTS CAUTION.

Abstract

SINCE THE 1960'S, EVALUATION STUDIES HAVE ASSUMED A MAJOR ROLE IN POLICYMAKING, PARTICULARLY IN FUNDING POLICY FOR SOCIAL PROGRAMS. MANPOWER, TRAINING, DRUG TREATMENT PROGRAMS, JUVENILE DELINQUENCY PREVENTION PROGRAMS, AND OPERATION HEADSTART HAVE ALL HAD MONEY REDUCED OR ELIMINATED ALTOGETHER ON THE BASIS OF NEGATIVE EVALUATIONS. THE PUBLIC CLAMORS FOR ACCOUNTABILITY, ACCEPTS RESEARCH RESULTS WHICH YIELD NEGATIVE OR 'NO CLEAR' FINDINGS, AND THEN CONCLUDES THAT ANTICIPATED RESULTS ARE NOT BEING ACHIEVED. YET FEW PEOPLE ACTUALLY EXAMINE THE RESEARCH WHICH RESULTED IN THESE FINDINGS. DISCUSSIONS OF SPECIFIC RESEARCH PROBLEMS ARE INCLUDED. THE SELECTION OF EVALUATIVE CRITERIA IS ESPECIALLY DIFFICULT WHEN EVALUATING SOCIAL PROGRAMS, AND THE MANY POLITICAL COMPROMISES RESULTING IN SUCH PROGRAMS CLOUD THE ISSUE OF WHAT THE PROGRAM GOALS ACTUALLY ARE. COST-BENEFIT ANALYSIS SUFFERS FROM THE DIFFICULTY INVOLVED IN ATTEMPTING TO PLACE A DOLLAR VALUE ON HUMAN LIFE AND SUFFERING. TRUE EXPERIMENTATION IS DIFFICULT IN A FREE SOCIETY AND ALSO RAISES PROFOUND ETHICAL QUESTIONS. THESE SHORTCOMINGS HAVE LED SOME TO URGE THE ADOPTION OF A NONEXPERIMENTAL METHODOLOGY, SUCH AS PROCESS-ORIENTED STUDIES. LONGITUDINAL STUDIES MEASURE A MYRIAD OF IMPACTS OCCURRING OVER A LONG PERIOD OF TIME, WHICH MAKES ISOLATING PARTICULAR ENVIRONMENTAL IMPACTS DIFFICULT. IN A REGRESSION ANALYSIS IT IS NOT POSSIBLE FOR THE DATA TO DISCLOSE WHAT WOULD HAPPEN IF ONE OF THE INDEPENDENT VARIABLES WERE DIFFERENT. THE STATISTICAL PROBLEMS ASSOCIATED WITH REGRESSION ANALYSIS USED OVER A VERY SHORT PERIOD OF TIME ARE MANY, YET THE RESULTS OF REGRESSION ANALYSIS ARE BEING APPLIED AS IF THEY EXAMINED CAUSAL RELATIONSHIPS, YIELDING NEGATIVE FINDINGS IN TERMS OF MEETING OBJECTIVES. THESE SHORTCOMINGS SUGGEST THAT EVALUATION STUDIES SHOULD BE USED VERY CAUTIOUSLY. A NEW ORGANIZATION WOULD BE IN ORDER TO EVALUATE EVALUATION STUDIES AND ISSUE SEALS OF APPROVAL; OTHERWISE MANY WORTHWHILE SOCIAL PROGRAMS ARE GOING TO BE SENTENCED TO A PREMATURE DEATH. (GLR)