NCJ Number
47187
Date Published
1977
Length
19 pages
Annotation
METHODS OF RELATING IMPLEMENTATION AND OUTCOME DATA IN EVALUATIONS OF CAREER EDUCATION PROGRAMS ARE CONSIDERED, AND ISSUES IN THE INTERPRETATION OF MIXED FINDINGS IN SUCH EVALUATIONS ARE DISCUSSED.
Abstract
FEW CAREER EDUCATION EVALUATION STUDIES INTEGRATE IMPLEMENTATION AND OUTCOME DATA. THERE HAS BEEN LITTLE DISCUSSION OF THE NATURE OF THE RELATIONSHIP BETWEEN IMPLEMENTATION AND OUTCOMES. FIVE PROGRAM TYPES MAY BE DISTINGUISHED IN TERMS OF IMPLEMENTATION, AND THREE ANALYTICAL MODELS -- SELECTION, BLOCKING, AND REGRESSION -- HAVE BEEN APPLIED IN RELATING IMPLEMENTATION TO OUTCOMES. SOME IMPLEMENTATION DATA USUALLY ARE COLLECTED AS PART OF SUMMATIVE EVALUATION. IF THESE DATA WERE TREATED INFERENTIALLY RATHER THAN DESCRIPTIVELY, THEY COULD CONTRIBUTE TO THE PRECISION OF THE EVALUATION. THE ABILITY TO DISTINGUISH IMPACT OF WELL-IMPLEMENTED PROGRAMS FROM THE RESULTS OF POORLY IMPLEMENTED PROGRAMS MAY BE PIVOTAL TO POLICY DECISIONS ABOUT THE CUMULATIVE VALUE OF THE IDEA BEING TESTED. PROBLEMS OF INTERPRETATION ARISE WHEN PROGRAM EVALUATION FINDINGS ARE MIXED. FOR EXAMPLE, IN ONE EVALUATION REPORT FOR A CAREER EDUCATION PROGRAM, THE PROPORTION OF POSITIVE FINDINGS INCREASES FROM APPROXIMATELY 30 PERCENT IN THE TEXT AND TABLES, TO 60 PERCENT IN THE SUMMARY, TO 100 PERCENT IN THE CONCLUSIONS AND RECOMMENDATIONS. SEVERAL APPROACHES TO THE 'HALF FULL -HALF EMPTY' QUANDARY HAVE BEEN SUGGESTED (E.G., ADVERSARIAL EVALUATION, EVALUATION AUDITS, EMPHASIZING THE REASONS FOR THE MIXED RESULTS). NONE OF THE SUGGESTED APPROACHES IS PERFECT. HOWEVER, THE PUBLIC INTEREST MAY BE SERVED BETTER BY ADOPTING AT LEAST ONE OF THE APPROACHES THAN BY PERPETUATING RELIANCE ON THE OBJECTIVITY OF EVALUATORS IN AN AREA WHERE INTERPRETATION OF MIXED RESULTS PLAYS SO PROMINENT A ROLE. A LIST OF REFERENCES IS INCLUDED. (LKM)