From 2007 through the present, the FBI Laboratory has led a series of decision analysis “black-box” studies aimed at evaluating the accuracy, repeatability, and reproducibility of pattern evidence examiner decisions in forensic casework. The findings of this research have been presented at more than a dozen scientific conferences and published in more than 20 scientific papers, with several additional publications in development. The results of this research can inform researchers, casework examiners, and the judicial system regarding the significance and effectiveness of forensic pattern evidence examination.
This research has also sparked significant discussion at conferences, in the trade press, and in scientific literature, with commentary on and responses to various aspects of the study. This presentation will focus on addressing and responding to the most frequently encountered critical observations regarding the experimental design and interpretation of results.
Critical factors in project management and experimental design include securing funding and logistics; definitions of objectives; method independence; design of experiments, such as open set design, randomization in samples, sample collection, digital versus physical samples, selection of close non-matches, and numbers of participants, samples, and decisions; sample difficulty and introduced biases to create bounding scenarios; participant recruitment, representativeness, and human subject research protections; and harmonization and imposition of potentially unfamiliar reporting scales.
Factors affecting the interpretation of results include participant drop-out and non-response; volunteer bias; the Hawthorne effect; calculation of accuracy, repeatability, and reproducibility rates; analysis of covariates among metadata and participant demographics; absence of quality control measures and secondary reviews.
One critical area for consideration is how inconclusive results are treated in research, especially compared to forensic casework where ground truth is often unknown and unknowable. Following the publication of our research results, a variety of approaches have been suggested, each with its own set of advantages and disadvantages.
Detailed Learning Objectives
- Attendees will gain insight as to how this research was designed to assess reliability of pattern evidence examinations.
- Attendees will develop a deeper understanding of the context behind the results and the inherent challenges in their interpretation.
- Attendees will understand the ongoing dialogue about the admissibility of pattern evidence testimony.