Machine Learning Model Drift: Predicting Diagnostic Imaging Follow-Up as a Case Example.
Objective: Address model drift in a machine learning (ML) model for predicting diagnostic imaging follow-up using data augmentation with more recent data versus retraining new predictive models.Methods: This institutional review board-approved retrospective study was conducted January 1, 2016, to De...
| Publicado en: | Journal of the American College of Radiology Vol. 19; no. 10; pp. 1162 - 1170 |
|---|---|
| Autores principales: | , , , , |
| Formato: | research Journal Article |
| Publicado: |
Elsevier B.V.
Oct2022
|
| Acceso en línea: | Ver este registro en EBSCOhost |
| fields | @attributes: recordID: 1 pdfLink: plink: https://search.ebscohost.com/login.aspx?direct=true&db=ccm&AN=159577071&site=ehost-live header: @attributes: shortDbName: ccm uiTerm: 159577071 longDbName: CINAHL Complete uiTag: AN controlInfo: bkinfo: dissinfo: jinfo: jid: 15461440 EAO jtl: Journal of the American College of Radiology issn: 15461440 maglogo: N pubinfo: dt: Oct2022 vid: 19 iid: 10 pid: 467 pub: Elsevier B.V. place: New York, New York artinfo: ui: 159577071 159577071 NLM35981636 159577071 10.1016/j.jacr.2022.05.030 NLM35981636 159577071 ppf: 1162 ppct: 8 formats: tig: atl: Machine Learning Model Drift: Predicting Diagnostic Imaging Follow-Up as a Case Example. aug: au: Lacson, Ronilda Eskian, Mahsa Licaros, Andro Kapoor, Neena Khorasani, Ramin affil: Associate Director of Center for Evidence Based Imaging, Department of Radiology, Brigham and Women's Hospital, Boston, Massachusetts sug: subj: Algorithms Radiography Retrospective Design Prospective Studies Funding Source ab: Objective: Address model drift in a machine learning (ML) model for predicting diagnostic imaging follow-up using data augmentation with more recent data versus retraining new predictive models.Methods: This institutional review board-approved retrospective study was conducted January 1, 2016, to December 31, 2020, at a large academic institution. A previously trained ML model was trained on 1,000 radiology reports from 2016 (old data). An additional 1,385 randomly selected reports from 2019 to 2020 (new data) were annotated for follow-up recommendations and randomly divided into two sets: training (n = 900) and testing (n = 485). Support vector machine and random forest (RF) algorithms were constructed and trained using 900 new data reports plus old data (augmented data, new models) and using only new data (new data, new models). The 2016 baseline model was used as comparator as is and trained with augmented data. Recall was compared with baseline using McNemar's test.Results: Follow-up recommendations were contained in 11.3% of reports (157 or 1,385). The baseline model retrained with new data had precision = 0.83 and recall = 0.54; none significantly different from baseline. A new RF model trained with augmented data had significantly better recall versus the baseline model (0.80 versus 0.66, P = .04) and comparable precision (0.90 versus 0.86).Discussion: ML methods for monitoring follow-up recommendations in radiology reports suffer model drift over time. A newly developed RF model achieved better recall with comparable precision versus simply retraining a previously trained original model with augmented data. Thus, regularly assessing and updating these models is necessary using more recent historical data. pubtype: Academic Journal doctype: research Journal Article ougenre: Article language: English refInfo: holdings: @attributes: islocal: N |
|---|