Masked Intervention Effects: Analytic Methods for Addressing Low Dosage of Intervention

• NCJ Number: 215608
• Journal: New Directions for Evaluation Issue: 110 Dated: Summer 2006 Pages: 19-32
• Author(s): John E. Lochman; Caroline Boxmeyer; Nicole Powell; David L. Roth; Michael Windle
• Date Published: 2006
• Length: 14 pages
• Grant Number(s): R49 CCR418569; DA-08453; DA-16135
• Sponsoring Agency: National Institute on Drug Abuse; Ctr's for Disease Control and Prevention (CDC)
• Publication Format: Article
• Publication Type: Program/Project Evaluation
• Language: English
• Country: United States of America

Annotation

This article examines how a particular strategy for analyzing evaluation data, intent-to-treat analyses, may underestimate the true effects of interventions.

Abstract

Intent-to-treat (ITT) analyses for program evaluations with randomized experimental designs provide rigorous tests of intervention effects. Once cases are randomized to conditions, all cases are included in analyses. Although this is a widely accepted analytic strategy, estimates of intervention effect sizes can be biased if any participants fail to comply with the assigned treatment condition. Using the example of the Coping Power Program--an evidence-based program for aggressive children--bias would result if attendance at parent sessions varied among parents. ITT compares average outcomes between randomized groups, but ignores compliance. An ITT analysis provides evidence of whether the intervention and control groups differ, but if significant intervention effects are not detected, it is unclear whether the intervention was ineffective or whether noncompliance with the intervention program undermined its effectiveness. Although progress is being made in developing and using new methods of addressing variables in participant attendance in intervention trials, these methods have rarely been compared and have not been applied to preventive interventions designed to reduce children's problem behaviors. This article compares propensity-score analyses and three types of complier-average-causal-effect (CACE) analyses with ITT analyses and as-treated analyses in evaluating the Coping Power Project. Both the propensity-score approach and the CACE approach addressed participants attendance and compliance. These differences in findings when compliance thresholds were set at different levels suggest that intervention researchers should work with program designers to estimate the minimally important level of participation necessary for program success. 26 references