Generalizability of a Machine Learning Model for Improving Utilization of Parathyroid Hormone-Related Peptide Testing across Multiple Clinical Centers.
BACKGROUND: Measuring parathyroid hormone-related peptide (PTHrP) helps diagnose the humoral hypercalcemia of malignancy, but is often ordered for patients with low pretest probability, resulting in poor test utilization. Manual review of results to identify inappropriate PTHrP orders is a cumbersom...
| Publicado en: | Clinical Chemistry Vol. 69; no. 11; pp. 1260 - 1270 |
|---|---|
| Autores principales: | , , , , , , , , |
| Formato: | Journal Article |
| Publicado: |
Oxford University Press / USA
Nov2023
|
| Acceso en línea: | Ver este registro en EBSCOhost |
| fields | @attributes: recordID: 1 pdfLink: plink: https://search.ebscohost.com/login.aspx?direct=true&db=ccm&AN=173595661&site=ehost-live header: @attributes: shortDbName: ccm uiTerm: 173595661 longDbName: CINAHL Complete uiTag: AN controlInfo: bkinfo: dissinfo: jinfo: jid: 00099147 10CS jtl: Clinical Chemistry issn: 00099147 maglogo: N pubinfo: dt: Nov2023 vid: 69 iid: 11 pid: 622 pub: Oxford University Press / USA artinfo: ui: 173595661 10.1093/clinchem/hvad141 173595661 ppf: 1260 ppct: 10 formats: fmt: @attributes: type: P tig: atl: Generalizability of a Machine Learning Model for Improving Utilization of Parathyroid Hormone-Related Peptide Testing across Multiple Clinical Centers. aug: au: Yang, He S. Weishen Pan Yingheng Wang Zaydman, Mark A. Spies, Nicholas C. Zhen Zhao Guise, Theresa A. Meng, Qing H. Fei Wang affil: Department of Pathology and Laboratory Medicine, Weill Cornell Medicine, New York, NY, United States sug: ab: BACKGROUND: Measuring parathyroid hormone-related peptide (PTHrP) helps diagnose the humoral hypercalcemia of malignancy, but is often ordered for patients with low pretest probability, resulting in poor test utilization. Manual review of results to identify inappropriate PTHrP orders is a cumbersome process. METHODS: Using a dataset of 1330 patients from a single institute, we developed a machine learning (ML) model to predict abnormal PTHrP results. We then evaluated the performance of the model on two external datasets. Different strategies (model transporting, retraining, rebuilding, and fine-tuning) were investigated to improve model generalizability. Maximum mean discrepancy (MMD) was adopted to quantify the shift of data distributions across different datasets. RESULTS: The model achieved an area under the receiver operating characteristic curve (AUROC) of 0.936, and a specificity of 0.842 at 0.900 sensitivity in the development cohort. Directly transporting this model to two external datasets resulted in a deterioration of AUROC to 0.838 and 0.737, with the latter having a larger MMD corresponding to a greater data shift compared to the original dataset. Model rebuilding using site-specific data improved AUROC to 0.891 and 0.837 on the two sites, respectively. When external data is insufficient for retraining, a fine-tuning strategy also improved model utility. CONCLUSIONS: ML offers promise to improve PTHrP test utilization while relieving the burden of manual review. Transporting a ready-made model to external datasets may lead to performance deterioration due to data distribution shift. Model retraining or rebuilding could improve generalizability when there are enough data, and model fine-tuning may be favorable when site-specific data is limited. pubtype: Academic Journal doctype: Journal Article ougenre: Article language: English refInfo: holdings: @attributes: islocal: N |
|---|