Coming soon: How does AI perform in the real world?
Much of what we know about the potential of AI applications to support breast cancer screening radiologists is based on simulation studies conducted in retrospective settings, or reader studies conducted in unrealistic screening settings .
These studies do not allow us to understand the true impact on cancer detection (sensitivity) and normal exam triaging (specificity). Nor do we have a sense of AI-radiologist interaction, and the influence of AI on radiologist behaviour. There is a great need for proactive, prospective monitoring of AI software and devices.
A prospective, observational, post-market study
With Vara’s integrated live performance monitoring, we are working with our partners to evaluate the prospective performance of our AI and the interaction with users. This prospective, observational study will provide evidence on the screening performance of radiologists when using AI in real world settings, shedding light on strategies used in daily clinical practice and their effectiveness and safety outcomes. The impact on radiologists’ satisfaction will also be an important factor in determining the optimal integration of AI in population-based breast cancer screening.
This study aims to produce clinical evidence that is generalizable, while being pragmatic about data availability, privacy and willingness of clinicians to participate in such a study.
 Freeman K, Geppert J, Stinton C, Todkill D, Johnson S, Clarke A et al. Use of artificial intelligence for image analysis in breast cancer screening programmes: systematic review of test accuracy BMJ 2021; 374 :n1872 doi:10.1136/bmj.n1872
 Hickman SE, Woitek R, Le EPV, Im YR, Mouritsen Luxhøj C, Aviles-Rivero AI, Baxter GC, MacKay JW, Gilbert FJ. Machine Learning for Workflow Applications in Screening Mammography: Systematic Review and Meta-Analysis. Radiology. 2021 Oct 19:210391. doi: 10.1148/radiol.2021210391. Epub ahead of print. PMID: 34665034.