A Synthesis of the 2021 NIJ Forecasting Challenge Winning Reports

• NCJ Number: 309826
• Author(s): Veronica White; Rachel Rief; Caleb D. Hudgins; Meaghan L. Pimsler; Joel Hunt
• Date Published: January 2025
• Length: 22 pages
• Agencies: NIJ

Abstract

In 2021, the National Institute of Justice (NIJ) hosted the Recidivism Forecasting Challenge (the Challenge), which evaluated team forecasts of the probability of individuals on parole recidivating during a specified time interval. NIJ anticipated that the Challenge would inspire teams to apply innovative data science techniques to forecast the probability individuals recidivate and inherently further our general knowledge of what variables are important in forecasting recidivism. Submissions were evaluated based on two different statistics: a Brier score and a fairness score. The Brier score evaluates the forecasts’ accuracy, in mathematical terms, by averaging the forecasts’ squared errors. In less technical terms, it is the average difference between the forecasted probability and the actual outcome. A Brier score, therefore, falls between 0 and 1, where the lower the error (and therefore the Brier score), the more accurate the forecast. Three individual Brier scores were computed: for males in the dataset, females in the dataset, and the average of these two scores. The second statistic looked at the racial fairness of the forecasts and was called the Fairness score. This metric incorporated the difference in false positive rates (forecasted to recidivate but did not recidivate) between Black and white individuals on parole. The difference in false positive rates was used as a penalty that was applied to the Brier score. This was computed for males in the dataset and females in the dataset; no average was calculated. To add to the overall body of knowledge of risk assessment creation, NIJ required the winning teams to submit a short paper describing (1) what type of model/algorithm(s) was used, (2) what variables mattered in their model (when possible), (3) additional variables created or added to their model that NIJ did not provide, (4) if/how models/variables differed between genders, and (5) if/how they accounted for any potential racial bias in their forecasting model, and other key findings. This paper aims to add to the knowledge of risk assessment creation by synthesizing the 25 winning, nonstudent papers.