babylonpolice.com

Objective: Reproducibility is critical for translating machine learning-based
(ML) solutions in computational pathology (CompPath) into practice. However, an
increasing number of studies report difficulties in reproducing ML results. The
NCI Imaging Data Commons (IDC) is a public repository of >120 cancer image
collections, including >38,000 whole-slide images (WSIs), that is designed to
be used with cloud-based ML services. Here, we explore the potential of the IDC
to facilitate reproducibility of CompPath research.

Materials and Methods: The IDC realizes the FAIR principles: All images are
encoded according to the DICOM standard, persistently identified, discoverable
via rich metadata, and accessible via open tools. Taking advantage of this, we
implemented two experiments in which a representative ML-based method for
classifying lung tumor tissue was trained and/or evaluated on different
datasets from the IDC. To assess reproducibility, the experiments were run
multiple times with independent but identically configured sessions of common
ML services.

Results: The AUC values of different runs of the same experiment were
generally consistent and in the same order of magnitude as a similar,
previously published study. However, there were occasional small variations in
AUC values of up to 0.044, indicating a practical limit to reproducibility.

Discussion and conclusion: By realizing the FAIR principles, the IDC enables
other researchers to reuse exactly the same datasets. Cloud-based ML services
enable others to run CompPath experiments in an identically configured
computing environment without having to own high-performance hardware. The
combination of both makes it possible to approach the reproducibility limit.