NCJ Number
50206
Journal
EVALUATION Volume: 1 Issue: 2 Dated: (1973) Pages: 49-55
Date Published
1973
Length
7 pages
Annotation
A STUDY ON EVALUATION REPORTS THAT FACTORS IMPEDING MOST PROGRAM STUDIES INCLUDES UNCLEAR GOALS, TURNOVER IN EVALUATION STAFF, A LACK OF UNDERSTANDING BETWEEN PROGRAM STAFF AND EVALUATORS, AND POOR TIMING.
Abstract
AN EVALUATION OF THE EVALUATION OF 10 APPLIED RESEARCH PROJECTS, 7 FUNDED BY THE NATIONAL INSTITUTE OF MENTAL HEALTH AND 3 BY OTHER FEDERAL AGENCIES, FOUND THAT MUCH RESEARCH FALLS SHORT OF EXPECTATIONS, NOT BECAUSE THE EVALUATORS ARE INEXPERIENCED BUT BECAUSE ORGANIZATIONAL PROBLEMS PREVENT EVALUATORS FROM APPLYING WHAT THEY KNOW. THIS STUDY FOUND LITTLE EVIDENCE TO SUPPORT PREVIOUS ASSUMPTIONS ABOUT THE HARMFUL LONG-RANGE EFFECTS OF PERSONALITY CONFLICTS BETWEEN EVALUATORS AND PROGRAM STAFF. INSTEAD, MAJOR PROBLEMS AROSE FROM THE DIFFERENCES IN PROFESSIONAL ROLES; THE PROGRAM ADMINISTRATOR MUST BELIEVE IN WHAT IS BEING DONE, BUT THE EVALUATOR'S FUNCTION IS TO DOUBT. CONFLICTS ALSO AROSE CONCERNING THE PURPOSE OF THE EVALUATION. EVALUATION WAS TOLERATED IF IT WAS MANDATED BY THE GRANT OR WAS SEEN AS A WAY TO GET ADDITIONAL FUNDING, BUT IF THERE WAS NO CLEAR CONSENSUS OF PURPOSE, FRICTION AROSE. STAFFING OF THE EVALUATION BECAME A MAJOR PROBLEM FOR MOST OF THE PROJECTS. ONE PROJECT HAD 3 DIFFERENT RESEARCH DIRECTORS IN 3 YEARS. TURNOVER AT LOWER LEVELS WAS ALSO HIGH. IN SOME PROJECTS EVALUATORS WERE PAID MORE THAN CLINICIANS, WHICH WAS ANOTHER SOURCE OF FRICTION. WHEN OUTSIDE CONSULTANTS WERE CALLED IN, THEY USUALLY COULD DEVOTE ONLY PART OF THEIR TIME TO THE PROJECT AND, AS A RESULT, MISSED MANY SIGNIFICANT CHANGES IN THE PROGRAM AND POLICY. THE RELUCTANCE OF STAFF TO FILL IN DATA SHEETS WAS CONSTANT THROUGHOUT. TIMING WAS ALSO A PROBLEM IN THAT EVALUATORS OFTEN WANTED BASELINE DATA AND WOULD THEREFORE HOLD UP TRAINING OR OTHER ASPECTS OF THE PROGRAM TO GATHER THEM. AT THE CONCLUSION OF AN EVALUATION, THERE WAS A PROBLEM WITH STAFF LEAVING TO TAKE NEW JOBS AND THE REPORT BEING COMPLETED ON A PART-TIME BASIS, POSSIBLY AT HOME. MOST EVALUATION REPORTS, HOWEVER, REMAIN UNREAD, WHICH MEANS THAT VALID FINDINGS FROM SMALL COMMUNITY PROJECTS MAY GO UNNOTICED. SELECTIVE DISTRIBUTION OF FINDINGS IS NEEDED. (GLR) EVALUATION REPORTS, HOWEVER, REMAIN UNREAD, WHICH MEANS THAT VALID FINDINGS FROM SMALL COMMUNITY PROJECTS MAY GO UNNOTICED. SELECTIVE DISTRIBUTION OF FINDINGS IS NEEDED. (GLR)