REVIEW

published: 05 July 2022

doi: 10.3389/fbinf.2022.912809

Open-Source Biomedical Image

Analysis Models: A Meta-Analysis and
Continuous Survey
Rui Li 1†, Vaibhav Sharma 1†, Subasini Thangamani 1† and Artur Yakimovich 1,2,3,4*
1
Center for Advanced Systems Understanding (CASUS), Helmholtz-Zentrum Dresden-Rossendorf e. V. (HZDR), Görlitz,
Germany, 2Bladder Infection and Immunity Group (BIIG), Department of Renal Medicine, Division of Medicine, University College
London, Royal Free Hospital Campus, London, United Kingdom, 3Artificial Intelligence for Life Sciences CIC, Dorset,
United Kingdom, 4Roche Pharma International Informatics, Roche Diagnostics GmbH, Mannheim, Germany

Open-source research software has proven indispensable in modern biomedical image

analysis. A multitude of open-source platforms drive image analysis pipelines and help
disseminate novel analytical approaches and algorithms. Recent advances in machine
learning allow for unprecedented improvement in these approaches. However, these novel
algorithms come with new requirements in order to remain open source. To understand
how these requirements are met, we have collected 50 biomedical image analysis models
Edited by: and performed a meta-analysis of their respective papers, source code, dataset, and
Jan Eglinger, trained model parameters. We concluded that while there are many positive trends in
Friedrich Miescher Institute for
Biomedical Research (FMI), openness, only a fraction of all publications makes all necessary elements available to the
Switzerland research community.
Reviewed by:
Jean-Karim Hériché, Keywords: machine learing, deep learning, open source, bioimaging, image analysis, medical imaging
European Molecular Biology
Laboratory Heidelberg, Germany
Christopher Schmied, INTRODUCTION
Fondazione Human Technopole, Italy
*Correspondence: The source code of data analysis algorithms made freely available for possible redistribution and
Artur Yakimovich modification (i.e. open source) has been beyond any doubt driving the ongoing revolution in Data
[email protected] Science (DS), Machine Learning (ML), and Artificial Intelligence (AI) (Sonnenburg et al., 2007;
†
These authors have contributed Landset et al., 2015; Abadi et al., 2016; Paszke et al., 2019). Encouraging open collaboration, the open-
equally to this work source model of code redistribution allows researchers to build upon their peers’ work on a global
scale fueling the rapid iterative improvement in the respective fields (Sonnenburg et al., 2007).
Specialty section: Conversely, “closed-source” publications not only hamper the development of the field but also make
This article was submitted to it hard for the researchers to reproduce the results disseminated in the research articles. While de jure
Computational BioImaging, all published work resides in the public domain, reverse engineering of an advanced algorithm
a section of the journal implementation may often take weeks or months, making such works hard to reproduce.
Frontiers in Bioinformatics
Needless to say, open source comes in a great variety of shapes and kinds. Remarkably, just
Received: 04 April 2022 making the source code of your research software available publicly or upon request does not per
Accepted: 13 June 2022
se make it open source. Usage and redistribution of any original creation, be it a research article
Published: 05 July 2022
or source code, lies within the legal boundaries of copyright laws, which differ significantly from
Citation: country to country. Therefore, for example, publicly available code without an explicit
Li R, Sharma V, Thangamani S and
attribution of a respective open-source license cannot be counted or treated as open source.
Yakimovich A (2022) Open-Source
Biomedical Image Analysis Models: A
Due to the sheer diversity, it may be difficult to judge which specific license is right for one’s
Meta-Analysis and Continuous Survey. project. Yet the choice of the license must always be dictated by the project and the intent of its
Front. Bioinform. 2:912809. authors. Consulting the licenses list approved by the Open Source Initiative is generally
doi: 10.3389/fbinf.2022.912809 considered to be a good starting point.

Frontiers in Bioinformatics | www.frontiersin.org 1 July 2022 | Volume 2 | Article 912809

Li et al. Open-Source Biomedical Image Analysis Models

The importance of open source software for computational attempt to minimize bias, we made sure that no individual
biomedical image analysis has become self-evident in the past medical imaging modality or biomedical task represents more
3 decades. Packages like ImageJ/Fiji (Schindelin et al., 2012; than 25% of our dataset. Additionally, we have attempted to
Schneider et al., 2012), CellProfiler (Carpenter et al., 2006), sample models published by both the biomedical community
KNIME (Tiwari and Sekhar, 2007), and Icy (de Chaumont et al., (e.g. Nature group journals), engineering community (IEEE
2011) not only perform the bulk of quantification tasks in the wetlabs group journals and conferences), as well as models published
but also serve as platforms for distribution of modules containing as preprints. For each publication we have noted the
cutting-edge algorithms. The ability to install and use these modules biomedical imaging modality, biomedical task (e.g. cancer),
and algorithms by researchers from various fields via a point-and- the open-source license used, reported model performance
click interface made it possible for the research groups without image with respective metric, whether the model is dealing with the
analysis specialists to obtain a qualitatively new level of biomedical supervised task, whether the model parameters can be
insights from their data. Yet, as we transition into the data-driven and downloaded (as well as the respective link), links to code
representation learning paradigm of biomedical image analysis, the and dataset. Noteworthy, performance reporting is highly
availability of datasets and trained model parameters becomes as dependent on a dataset or benchmark. Therefore, to avoid
important as the open-source code. confusion or bias we have recorded the best-reported
The ability to download training parameters may allow researchers performance for illustrative purposes only. Identical
to skip the initial model training and focus on gradual model performance on a different dataset should not be expected.
improvement through a technique known as transfer learning For the purpose of this review, we have split this meta-dataset
(West et al., 2007; Pan and Yang, 2010). Transfer learning has into three tables according to the ML task of the models. The
proven effective in Computer Vision (Deng et al., 2009) and full dataset is available on GitHub (https://github.com/casus/
Natural Language Processing (Wolf et al., 2020) domains (further bim). To ensure the completeness and correctness of this
reviewed in (Yakimovich et al., 2021)). However, the complexity of meta-dataset we invite the research community to contribute
sharing the trained parameters of a model differs significantly between their additions and corrections to our survey meta-dataset.
ML algorithms. For example, while model parameters of a First display table obtained from our meta-dataset
conventional ML algorithm like linear regression may be contains 14 models aimed at biomedical image
conveniently shared in the text of the article, this is impossible for classification (Table 1). The most prevalent imaging
DL models with millions of parameters. This, in turn, requires modalities for this ML task are computed tomography
rethinking conventional approaches to ML/DL models sharing (CT) and digital pathology—both highly clinically relevant
under an open-source license. modalities. We noted that most publications had an open-
In this review, we collate ML models for biomedical image analysis source license clearly defined in their repositories. The
recently published in the peer-reviewed literature and available as consensus between the choices of metric is rather low,
open-source. We describe open-source licenses used, code availability, making it difficult to compare one model to the other.
data availability, biomedical and ML tasks, as well as the availability of Although most models had both source code and datasets
model parameters. We make the collated collection of the open-source available, only 4 out of 14 models had trained model
model available via a GitHub repository and call on the research parameters available for download.
community to contribute their models to it via pull requests. The second display table contains 25 models (Table 2)
Furthermore, we provide descriptive statistics of our observations aimed at biomedical image segmentation—a task relevant for
and discuss the pros and cons of the status quo in the field of obtaining quantitative insights from the biomedical images
biomedical image analysis as well as perspectives in the general DS (e.g. size of the tumor). Similarly, to the models for
context. Several efforts to create biomedical ML model repositories or biomedical image classification, the vast majority of the
so-called “zoos” (e.g. bioimage. io) and web-based task consolidators segmentation models have a well-defined open-source
(Hollandi et al., 2020; Stringer et al., 2021) have been undertaken. license with only a few exceptions. Again, similarly to the
Here, rather than proposing a competing effort, we propose a classification models, the consensus between performance
continuous survey of the field “as is”. We achieve this through metric choices is rather low, although Dice score reports
collating metadata of published papers and their respective source clearly dominated. Conversely, the percentage of models
code, data, and model parameters (also known as weights and with pre-trained parameters available for download is
checkpoints). slightly higher than in the case of the classification models
(36% vs 29%). However, over half of the models do not
provide pre-trained parameters for the download for both
Continuous Biomedical Image Analysis segmentation and classification tasks.
Model Survey Finally, we have also examined biomedical image analysis
To understand the availability, reproducibility, and models aimed at less popular ML tasks including data
accessibility of published biomedical image analysis models generation, object detection or reconstruction (Table 3).
we have collected a survey meta-dataset of 50 model articles Apart from digital pathology, CT scans this group of
and preprints published within the last 10 years. During our models also contains light and electron microscopy.
collection effort, we have prioritized publications with Remarkably, only 19% of models in this group had
accompanying source code freely available online. In an downloadable model parameters. At the same time, almost

Frontiers in Bioinformatics | www.frontiersin.org 2 July 2022 | Volume 2 | Article 912809

Li et al. Open-Source Biomedical Image Analysis Models

TABLE 1 | Biomedical Image Classification Models. Here, AUC is Area under curve, CT is computed tomography.

Imaging Modality Biomed Task License Reported Performance Parameters Download References

CT Lung tumor Apache-2.0 0.93 Accuracy No LaLonde et al. (2020)

CT Lung tumor MIT 0.76 AUC No Guo et al. (2020)
CT Pulmonary nodule GPL-3.0 0.90 Accuracy No Zhu et al. (2018a)
CT Pulmonary nodule MIT 0.96 AUC No Al-Shabi et al. (2019)
CT Pulmonary nodule MIT 0.95 AUC No Dey et al. (2018)
Dermatoscopy Skin tumor N/a 0.93 Accuracy No Datta et al. (2021)
Dermatoscopy Skin tumor MIT 0.81 AUC Yes Zunair and Ben Hamza, (2020)
Mammography Breast tumor CC BY-NC-ND 4.0 0.93 AUC Yes Shen et al. (2021)
Digital Pathology Breast tumor CC BY-NC-ND 4.0 0.63 F1 No Pati et al. (2022)
Mammography Breast tumor CC BY-NC-SA 4.0 0.84 Accuracy Yes Shen et al. (2019)
Digital Pathology Breast tumor MIT 0.93 Accuracy Yes Rakhlin et al. (2018)
Digital Pathology Lung tumor GPL-3.0 0.53 Kappa No Wei et al. (2019)
Digital Pathology Lung tumor MIT 0.97 AUC No Coudray et al. (2018)
Fluorescence microscopy Host-pathogen interactions N/a 0.92 Accuracy No Fisch et al. (2019)

TABLE 2 | Biomedical Image Segmentation Models. Here, CT is computed tomography, DSC is Dice similarity coefficient, AP is Average Precision, IoU is Intersection over
Union, DOF is Depth of field, AUC is Area under curve, SHG is Second harmonic generation microscopy.

Imaging Modality Biomed Task License Reported Performance Parameters References

Download

3D microscopy Nuclei detection MIT 0.937 AP No Hirsch and Kainmueller,

(2020)
CT Kidney tumor GPL-3.0 0.95 Dice No Müller and Kramer, (2021)
CT Pulmonary nodule BSD-3-Clause N/a No Hancock and Magnan, (2019)
CT Pulmonary nodule CC BY-NC-SA 4.0 0.55 IoU Yes Aresta et al. (2019)
CT Pulmonary nodule MIT 0.83 DSC No Keetha et al. (2020)
CT Pancreas & Brain tumor MIT 0.84 Dice No Oktay et al. (2018)
CT, Dermatoscopy Lung tumor and Skin tumor N/a 0.9965 Jaccard No Kaul et al. (2019)
CT Brain tumor Apache 2.0 0.89 Dice No Isensee et al. (2018)
MRI Brain tumor Apache 2.0 0.79 Dice No Wang et al. (2021)
MRI Brain tumor CC BY-NC- 0.76 Dice No Baek et al. (2019)
ND 4.0
Digital Pathology Breast tumor CC BY-NC- 0.893 F1 Yes Le et al. (2020)
ND 4.0
Digital Pathology Lung tumor CC-BY 0.83 Accuracy No Tomita et al. (2019)
Digital Pathology Multiple pathologies MIT N/a No Khened et al. (2021)
Electron microscopy Multiple pathologies MIT 0.5 VI Yes Lee et al. (2017)
Fluorescence Cellular structures N/a 20 x Enhancement in DOF Yes Wu et al. (2019)
microscopy reconstruction
Fluorescence Nuclei detection BSD-3-Clause 0.94 Accuracy Yes Weigert et al. (2020)
microscopy
Microscopy Cellular reconstruction N/a 0.69 AP No Hirsch et al. (2020)
MRI Brain tumor BSD-3-Clause 0.87 Dice Yes Wang et al. (2018)
MRI Brain tumor MIT 0.85 Dice Yes Havaei et al. (2017)
MRI Brain tumor MIT 0.90 Dice No Isensee et al. (2018)
MRI Brain tumor MIT 0.91 Dice No Myronenko, (2019)
SHG Bone disease GPL-3.0 0.78 Accuracy No Schmarje et al. (2019)
Time-lapse microscopy Nuclei detection N/a 0.92 Accuracy Yes Shailja et al. (2021)
Ultrasound imaging Intraventricular hemorrhage MIT 0.89 Dice No Valanarasu et al. (2020)
MRI Brain tumor N/a 0.81 Dice Yes Larrazabal et al. (2021)

all the models in this group had well attributed open-source Trends Meta-Analysis in Biomedical Image
licenses. This may suggest that parameter sharing is not very Analysis Model
common in highly specialized fields like microscopy. To understand general trends in the collection of our open-source
Interestingly, for this and other groups of ML tasks, we models we have computed respective fractions of each descriptive
have found that parameter sharing was more common in category we have assigned to each work. The assignment was
models submitted as a part of a data challenge. This may be performed through careful analysis of the respective research
simply a result of data challenge participation conditions. article, code repository, dataset repository, and the availability of

Frontiers in Bioinformatics | www.frontiersin.org 3 July 2022 | Volume 2 | Article 912809

Li et al. Open-Source Biomedical Image Analysis Models

TABLE 3 | Other Biomedical Image Models. Here, CT is computed tomography.

Imaging Modality Biomed Task ML Task License Parameters References

Download

Mammography Breast tumor Classification & Detection N/a Yes Ribli et al. (2018)
Fluorescence Cellular structures Data generation Apache-2.0 No Eschweiler et al. (2021)
microscopy reconstruction
CT Pulmonary nodule Detection Apache-2.0 No Zhu et al. (2018b)
CT Pulmonary nodule Detection MIT No Li and Fan, (2020)
Digital Pathology Multiple pathologies Graph embedding AGPL 3.0 No Jaume et al. (2021)
Mammography Breast tumor Image Inpainting & Data CC BY-NC- Yes Wu et al. (2018)
generation ND 4.0
Confocal microscopy Cellular structures Reconstruction Apache-2.0 No Vizcaíno et al. (2021)
reconstruction
Cryo-electron Cellular structures Reconstruction GPL-3.0 No Zhong et al. (2019)
microscopy reconstruction
Cryo-electron Protein structures Reconstruction GPL-3.0 No Ullrich et al. (2019)
microscopy reconstruction
Electron microscopy Cellular structures Reconstruction N/a No Guay et al. (2021)
reconstruction
3D microscopy Image acquisition Reconstruction BSD-3-Clause No Saha et al. (2020)

the trained model parameters (Figure 1). While admittedly 50 Remarkably, while providing the model’s source code, as well
papers constitute a relatively small sample size, we have made the as, in most cases, the model’s dataset, an impressive 68% of the
best reasonable effort to ensure the sampling was unbiased. contributions we have reviewed did not provide trained model
Specifically, the set of models we have reviewed addresses the parameters (Figure 1G). Breaking down by the publishers or
following biomedical tasks (from most to least frequent): repositories, 43% and 31% of papers published by Nature group
pulmonary nodule, brain tumor, breast tumor, cellular and Springer respectively provided model parameters. However,
structures reconstruction, lung tumor, cell nuclei detection, only 25% of IEEE papers and 14% of arXiv preprints provided
multiple pathologies, skin tumor, protein structures parameters. Altogether, the low percentage of shared parameters
reconstruction, kidney tumor, pancreas and brain tumor, lung are suggesting that the efforts to reproduce these papers came
tumor and skin tumor, host-pathogen interactions, bone disease, with the caveat of provisioning a hardware setup capable of
image acquisition, intraventricular hemorrhage (Figure 1A). wielding the computational load required by the respective
From the perspective of imaging modalities, the models we model. In some cases that requiresaccess to the high-capacity
reviewed span the following: computed tomography (CT), digital computing. Furthermore, this way, instead of simply building
pathology, magnetic resonance imaging (MRI), mammography, upon the models trained, the efforts of the authors would have to
fluorescence microscopy, 3D microscopy, cryo-electron be first reproduced. Needless to say, should any of the papers
microscopy, dermatoscopy, electron microscopy, confocal become seminal these high-performance computations would
microscopy, CT and dermatoscopy, light and electron have to be repeated time and time again, possibly taking days
microscopy, second harmonic generation microscopy, time- of GPU computation.
lapse microscopy, ultrasound imaging (Figure 1B). From the Interestingly, of the authors who have chosen to make the
perspective of ML tasks these models covered the following: trained parameters available to the readers around 25% have
segmentation, reconstruction classification, object detection, chosen to deposit the parameters on GitHub, while 19% and 6%
imagine inpainting and data generation, graph embedding, have opted for Google drive and Dropbox services respectively.
classification, and detection (Figure 1C). 86% of the models The rest deposited their parameters on the proprietary and other
we have reviewed were addressing supervised tasks and 14% services (Figure 1H).
unsupervised tasks (Figure 1D).
Within our collection of open-source models, we have noted that
32% of the authors have selected the MIT license, 18% have selected DISCUSSION
Apache-2.0, 12%—GPL-3.0, 10%—BSD-3-Clause license, 8%—CC
BY-NC-SA 4.0 license. Remarkably, another 8% have published their The advent of ML and specifically representation learning is
code without license attribution, arguably making it harder for the opening a new horizon for biomedical image analysis. Yet, the
field to understand the freedom to operate with the code made success of these new advanced ML approaches brings about new
available with the paper (Figure 1E). Within these papers, 84% of the requirements and standards to ensure quality and reproducibility
authors made the dataset used to train the model available and clearly (Hernandez-Boussard et al., 2020; Mongan et al., 2020; Norgeot
indicated within the paper or the code repository (Figure 1F). Overall, et al., 2020; Heil et al., 2021; Laine et al., 2021). Several
this amounted to the vast majority of the works which we have minimalistic quality standards applicable to the clinical setting
selected to have a clear open-source license designation, as well as a have been proposed (Hernandez-Boussard et al., 2020; Mongan
dataset available. et al., 2020; Norgeot et al., 2020), and while coming from slightly

Frontiers in Bioinformatics | www.frontiersin.org 4 July 2022 | Volume 2 | Article 912809

Li et al. Open-Source Biomedical Image Analysis Models

FIGURE 1 | Meta-analysis of trends in open-source biomedical image analysis models (A) Biomedical tasks overview and breakdown in our collection (B) Variety of
imaging modalities (C) Machine learning tasks the models are aimed at (D) Are the ML algorithms used for supervised or unsupervised learning tasks (E) Prevalence of
open source licenses used (F) Availability of datasets (G) Availability of trained model parameters (H) Prevalence of platforms used for trained model parameters sharing.
Here, CT is computed tomography, MRI is magnetic resonance imaging.

different perspectives they demonstrate an overlap on essential 2020) and MINIMAR (Hernandez-Boussard et al., 2020)
topics like the dataset description, comparison to baseline and approaches aim to adhere to a clinical tradition. Authors
hyperparameters sharing. For example, CLAIM (Mongan et al., define a checklist including a structure of an academic

Frontiers in Bioinformatics | www.frontiersin.org 5 July 2022 | Volume 2 | Article 912809

Li et al. Open-Source Biomedical Image Analysis Models

biomedical paper, requiring either a lengthy biomedical problem results. Arguably, one of the most intuitive experiences of model
description (CLAIM) or descriptive statistics of the dataset’s parameters sharing for the end-users is currently offered by the
internal structure (MINIMAR). At the same time, MI-CLAIM HuggingFace platform in the domain of natural language
(Norgeot et al., 2020) aims to adhere to the Data Science tradition, processing. This has largely been possible through the
focusing specifically on data preprocessing and baseline platform’s own ML library allowing for improved
comparison. Remarkably, even though item 24 of the CLAIM compatibility (Wolf et al., 2020).
checklist explicitly mentions the importance of specifying the Interestingly, the vast majority of authors have chosen MIT
source of the starting weights (parameters) if transfer learning is and Apache-2.0 as their open-source licenses. Both Apache-2.0
employed, all three approaches fail to explicitly encourage sharing and MIT are known for being permissive, rather than copyleft
of the trained model parameters. Instead of proposing yet another licenses. Furthermore, both licenses are very clearly
checklist, the current survey aims to understand to extend to formulated and easy to use. It is tempting to speculate that
which the model parameters are shared in the biomedical image their popularity is a result of the simplicity and openness that
analysis field and emphasize the importance of parameters these licenses offer.
sharing to foster reproducibility in the field. However, noteworthy, our survey is limited to the papers we
The past 3 decades have successfully demonstrated the reviewed. To improve the representativeness of our meta-
viability of the open-source model for the research software in analysis, as well as encourage the dissemination of the open-
this field, as well as the role of open-source software in fostering source models in biomedical image analysis we call on our peers
scientific progress. However, the change of modeling paradigm to to contribute to our collection via the GitHub repository.
DL requires new checks and balances to ensure the results are Specifically, we invite the researchers to fork our repository,
reproducible and the efforts are not doubled. Furthermore, major make additions to the content of the list following the
computational efforts inevitably come with an environmental contribution guidelines and merge them in via pull request.
footprint (Strubell et al., 2020). Making parameters of the trained This way we hope to not only obtain an up-to-date state of
models available to the research community not only could the field but also ensure the code, datasets and trained model
minimize this footprint, but also open new prospects for the parameters are easier to find.
researcher wishing to fine-tune the pre-trained models to their
task of choice. Such an approach proved incredibly fruitful in the
field of natural language processing (Zhang et al., 2020). AUTHOR CONTRIBUTIONS
Remarkably, in the current survey, we have found that only
32% of the biomedical models we have reviewed made the train AY conceived the idea. AY, ST, VS, and RL reviewed the
model parameters available for download. On one hand, such a published works and collated the data. AY, ST, VS, and RL
low number of trained models available for download may be wrote the manuscript.
explained by the fact that many journals and conferences do not
require trained models to warrant publication. On another hand,
with parameters of some models requiring hundreds of FUNDING
megabytes of storage, there are not many opportunities to
share these files. Interestingly, while some researchers shared This work was partially funded by the Center for Advanced
their trained model parameters via platforms like GitHub, Google Systems Understanding (CASUS) which is financed by
drive, and Dropbox, the vast majority opted for often proprietary Germany’s Federal Ministry of Education and Research
sites to share these parameters (Figure 1H). In our opinion, this (BMBF) and by the Saxon Ministry for Science, Culture and
indicates the necessity of hubs and platforms for sharing trained Tourism (SMWK) with tax funds on the basis of the budget
biomedical image analysis models. approved by the Saxon State Parliament. This work has been
It is worth noting that most cloud storage services like Google partially funded by OPTIMA. OPTIMA is funded through the
drive or Dropbox are more suited for instant file sharing rather IMI2 Joint Undertaking and is listed under grant agreement No.
than archival deposition of model parameters. These storage 101034347. IMI2 receives support from the European Union’s
solutions don’t offer data immutability or digital object Horizon 2020 research and innovation programme and the
identifiers attached to them, and hence can simply be European Federation of Pharmaceutical Industries and
overwritten or disappear leaving crucial content inaccessible. Associations (EFPIA). IMI supports collaborative research
Authors opting for self-hosting of model parameters also likely projects and builds networks of industrial and academic
underestimate the workload of the long-term serving of archival experts in order to boost pharmaceutical innovation in
data. Instead of the aforementioned approaches to model sharing, Europe. The views communicated within are those of
one should take advantage of efforts like BioImage.io, Tensorflow OPTIMA. Neither the IMI nor the European Union, EFPIA,
Hub (Paper, 2021), PyTorch Hub, DLHub (Chard et al., 2019), or or any Associated Partners are responsible for any use that may be
similar in order to foster consistency and reproducibility of their made of the information contained herein.

Frontiers in Bioinformatics | www.frontiersin.org 6 July 2022 | Volume 2 | Article 912809

Li et al. Open-Source Biomedical Image Analysis Models

REFERENCES Heil, B. J., Hoffman, M. M., Markowetz, F., Lee, S. I., Greene, C. S., and Hicks, S. C.
(2021). Reproducibility Standards for Machine Learning in the Life Sciences.
Nat. Methods 18, 1132–1135. doi:10.1038/s41592-021-01256-7
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., et al. (2016). Hernandez-Boussard, T., Bozkurt, S., Ioannidis, J. P. A., and Shah, N. H. (2020).
“TensorFlow: A System for Large-Scale Machine Learning,” in 12th USENIX MINIMAR (MINimum Information for Medical AI Reporting): Developing
Symposium on Operating Systems Design and Implementation (OSDI 16) Reporting Standards for Artificial Intelligence in Health Care. J. Am. Med. Inf.
(usenix.org), 265 Assoc. 27, 2011–2015. doi:10.1093/jamia/ocaa088
Al-Shabi, M., Lan, B. L., Chan, W. Y., Ng, K. H., and Tan, M. (2019). Lung Nodule Hirsch, P., and Kainmueller, D. (2020). “An Auxiliary Task for Learning Nuclei
Classification Using Deep Local-Global Networks. Int. J. Comput. Assist. Radiol. Segmentation in 3D Microscopy Images,” in Proceedings Of the Third
Surg. 14, 1815–1819. doi:10.1007/s11548-019-01981-7 Conference On Medical Imaging With Deep Learning Proceedings of
Aresta, G., Jacobs, C., Araújo, T., Cunha, A., Ramos, I., van Ginneken, B., et al. Machine Learning Research. Editors T. Arbel, I. Ben Ayed, M. de Bruijne,
(2019). iW-Net: an Automatic and Minimalistic Interactive Lung Nodule M. Descoteaux, H. Lombaert, and C. Pal (Montreal, QC, Canada: PML), 304
Segmentation Deep Network. Sci. Rep. 9, 11591. doi:10.1038/s41598-019- Hirsch, P., Mais, L., and Kainmueller, D. (2020). PatchPerPix for Instance
48004-8 Segmentation. arXiv [cs.CV]. Available at: http://arxiv.org/abs/2001.07626
Baek, S., He, Y., Allen, B. G., Buatti, J. M., Smith, B. J., Tong, L., et al. (2019). Deep (Accessed March 30, 2022).
Segmentation Networks Predict Survival of Non-small Cell Lung Cancer. Sci. Hollandi, R., Szkalisity, A., Toth, T., Tasnadi, E., Molnar, C., Mathe, B., et al. (2020).
Rep. 9, 17286. doi:10.1038/s41598-019-53461-2 nucleAIzer: A Parameter-free Deep Learning Framework for Nucleus
Carpenter, A. E., Jones, T. R., Lamprecht, M. R., Clarke, C., Kang, I. H., Friman, O., Segmentation Using Image Style Transfer. Cell. Syst. 10, 453–e6. doi:10.
et al. (2006). CellProfiler: Image Analysis Software for Identifying and 1016/j.cels.2020.04.003
Quantifying Cell Phenotypes. Genome Biol. 7, R100. doi:10.1186/gb-2006-7- Isensee, F., Kickingereder, P., Wick, W., Bendszus, M., and Maier-Hein, K. H.
10-r100 (2018). “Brain Tumor Segmentation and Radiomics Survival Prediction:
Chard, R., Li, Z., Chard, K., Ward, L., Babuji, Y., Woodard, A., et al. (2019). Contribution to the BRATS 2017 Challenge,” in Brainlesion: Glioma,
“DLHub: Model and Data Serving for Science,” in 2019 IEEE International Multiple Sclerosis, Stroke and Traumatic Brain Injuries (Quebec City,
Parallel and Distributed Processing Symposium (IPDPS) (ieeexplore.ieee.org), Canada: Springer International Publishing), 287–297. doi:10.1007/978-3-
283–292. doi:10.1109/ipdps.2019.00038 319-75238-9_25
Coudray, N., Ocampo, P. S., Sakellaropoulos, T., and Narula, N. (2018). Jaume, G., Pati, P., Anklin, V., Foncubierta, A., and Gabrani, M. (2021).
“Classification and Mutation Prediction from Non–small Cell Lung Cancer “HistoCartography: A Toolkit for Graph Analytics in Digital Pathology,” in
Histopathology Images Using Deep Learning,”Nat. Med. Available at: https:// Proceedings Of the MICCAI Workshop On Computational Pathology
www.nature.com/articles/s41591-018-0177-5?sf197831152=1. Proceedings of Machine Learning Research. Editors M. Atzori, N. Burlutskiy,
Datta, S. K., Shaikh, M. A., Srihari, S. N., and Gao, M. (2021). “Soft Attention F. Ciompi, Z. Li, F. Minhas, and H. Müller (PMLR), 117
Improves Skin Cancer Classification Performance,” in Interpretability of Kaul, C., Manandhar, S., and Pears, N. (2019). “Focusnet: An Attention-Based
Machine Intelligence in Medical Image Computing, and Topological Data Fully Convolutional Network for Medical Image Segmentation,” in 2019 IEEE
Analysis and its Applications for Medical Data (Springer International 16th International Symposium on Biomedical Imaging (ISBI 2019)
Publishing), 13–23. doi:10.1007/978-3-030-87444-5_2 (ieeexplore.ieee.org), 455–458. doi:10.1109/isbi.2019.8759477
de Chaumont, F., Dallongeville, S., and Olivo-Marin, J.-C. (2011). “ICY: A New Keetha, N. V., Samson, A. B. P., and Annavarapu, C. S. R. (2020). U-det: A
Open-Source Community Image Processing Software,” in 2011 IEEE Modified U-Net Architecture with Bidirectional Feature Network for Lung
International Symposium on Biomedical Imaging: From Nano to Macro Nodule Segmentation. arXiv [eess.IV]. Available at: http://arxiv.org/abs/2003.
(ieeexplore.ieee.org), 234–237. doi:10.1109/isbi.2011.5872395 09293 (Accessed March 31, 2022).
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. (2009). “ImageNet: A Khened, M., Kori, A., Rajkumar, H., Krishnamurthi, G., and Srinivasan, B. (2021).
Large-Scale Hierarchical Image Database,” in 2009 IEEE Conference on A Generalized Deep Learning Framework for Whole-Slide Image Segmentation
Computer Vision and Pattern Recognition, 248–255. doi:10.1109/cvpr.2009. and Analysis. Sci. Rep. 11, 11579. doi:10.1038/s41598-021-90444-8
5206848 Laine, R. F., Arganda-Carreras, I., Henriques, R., and Jacquemet, G. (2021).
Dey, R., Lu, Z., and Hong, Y. (2018). “Diagnostic Classification of Lung Nodules Avoiding a Replication Crisis in Deep-Learning-Based Bioimage Analysis.
Using 3D Neural Networks,” in 2018 IEEE 15th International Symposium on Nat. Methods 18, 1136–1144. doi:10.1038/s41592-021-01284-3
Biomedical Imaging (ISBI 2018) (ieeexplore.ieee.org), 774–778. doi:10.1109/ LaLonde, R., Torigian, D., and Bagci, U. (2020). “Encoding Visual Attributes in
isbi.2018.8363687 Capsules for Explainable Medical Diagnoses,” in Medical Image Computing and
Eschweiler, D., Rethwisch, M., Jarchow, M., Koppers, S., and Stegmaier, J. Computer Assisted Intervention – MICCAI 2020 (Springer International
(2021). 3D Fluorescence Microscopy Data Synthesis for Segmentation and Publishing), 294–304. doi:10.1007/978-3-030-59710-8_29
Benchmarking. PLoS One 16, e0260509. doi:10.1371/journal.pone. Landset, S., Khoshgoftaar, T. M., Richter, A. N., and Hasanin, T. (2015). A Survey
0260509 of Open Source Tools for Machine Learning with Big Data in the Hadoop
Fisch, D., Yakimovich, A., Clough, B., Wright, J., Bunyan, M., Howell, M., et al. Ecosystem. J. Big Data 2, 1–36. doi:10.1186/s40537-015-0032-1
(2019). Defining Host-Pathogen Interactions Employing an Artificial Larrazabal, A. J., Martínez, C., Dolz, J., and Ferrante, E. (2021). “Orthogonal
Intelligence Workflow. Elife 8, e40560. doi:10.7554/eLife.40560 Ensemble Networks for Biomedical Image Segmentation,” in Medical Image
Guay, M. D., Emam, Z. A. S., Anderson, A. B., Aronova, M. A., Pokrovskaya, I. D., Computing and Computer Assisted Intervention – MICCAI 2021 (Strasbourg,
Storrie, B., et al. (2021). Dense Cellular Segmentation for EM Using 2D-3D France: Springer International Publishing), 594–603. doi:10.1007/978-3-030-
Neural Network Ensembles. Sci. Rep. 11, 2561–2611. doi:10.1038/s41598-021- 87199-4_56
81590-0 Le, H., Gupta, R., Hou, L., Abousamra, S., Fassler, D., Torre-Healy, L., et al. (2020).
Guo, H., Kruger, U., Wang, G., Kalra, M. K., and Yan, P. (2020). Knowledge-Based Utilizing Automated Breast Cancer Detection to Identify Spatial Distributions
Analysis for Mortality Prediction from CT Images. IEEE J. Biomed. Health Inf. of Tumor-Infiltrating Lymphocytes in Invasive Breast Cancer. Am. J. Pathol.
24, 457–464. doi:10.1109/JBHI.2019.2946066 190, 1491–1504. doi:10.1016/j.ajpath.2020.03.012
Hancock, M. C., and Magnan, J. F. (2019). Level Set Image Segmentation with Lee, K., Zung, J., Li, P., Jain, V., and Sebastian Seung, H. (2017). Superhuman
Velocity Term Learned from Data with Applications to Lung Nodule Accuracy on the SNEMI3D Connectomics Challenge. arXiv [cs.CV]. Available
Segmentation. arXiv [eess.IV]. Available at: http://arxiv.org/abs/1910.03191 at: http://arxiv.org/abs/1706.00120.
(Accessed March 31, 2022). Li, Y., and Fan, Y. (2020). DeepSEED: 3D Squeeze-And-Excitation Encoder-
Havaei, M., Davy, A., Warde-Farley, D., Biard, A., Courville, A., Bengio, Y., et al. Decoder Convolutional Neural Networks for Pulmonary Nodule Detection.
(2017). Brain Tumor Segmentation with Deep Neural Networks. Med. Image Proc. IEEE Int. Symp. Biomed. Imaging 2020, 1866–1869. doi:10.1109/
Anal. 35, 18–31. doi:10.1016/j.media.2016.05.004 ISBI45749.2020.9098317

Frontiers in Bioinformatics | www.frontiersin.org 7 July 2022 | Volume 2 | Article 912809

Li et al. Open-Source Biomedical Image Analysis Models

Mongan, J., Moy, L., and Kahn, C. E., Jr (2020). Checklist for Artificial Intelligence Stringer, C., Wang, T., Michaelos, M., and Pachitariu, M. (2021). Cellpose: a
in Medical Imaging (CLAIM): A Guide for Authors and Reviewers. Radiol. Generalist Algorithm for Cellular Segmentation. Nat. Methods 18, 100–106.
Artif. Intell. 2, e200029. doi:10.1148/ryai.2020200029 doi:10.1038/s41592-020-01018-x
Müller, D., and Kramer, F. (2021). MIScnn: a Framework for Medical Image Strubell, E., Ganesh, A., and McCallum, A. (2020). Energy and Policy
Segmentation with Convolutional Neural Networks and Deep Learning. BMC Considerations for Modern Deep Learning Research. AAAI 34,
Med. Imaging 21, 12. doi:10.1186/s12880-020-00543-7 13693–13696. doi:10.1609/aaai.v34i09.7123
Myronenko, A. (2019). “3D MRI Brain Tumor Segmentation Using Autoencoder Tiwari, A., and Sekhar, A. K. (2007). Workflow Based Framework for Life Science
Regularization,” in Brainlesion: Glioma, Multiple Sclerosis, Stroke and Informatics. Comput. Biol. Chem. 31, 305–319. doi:10.1016/j.compbiolchem.
Traumatic Brain Injuries (Granada, Spain: Springer International 2007.08.009
Publishing), 311–320. doi:10.1007/978-3-030-11726-9_28 Tomita, N., Abdollahi, B., Wei, J., Ren, B., Suriawinata, A., and Hassanpour, S.
Norgeot, B., Quer, G., Beaulieu-Jones, B. K., Torkamani, A., Dias, R., (2019). Attention-Based Deep Neural Networks for Detection of Cancerous and
Gianfrancesco, M., et al. (2020). Minimum Information about Clinical Precancerous Esophagus Tissue on Histopathological Slides. JAMA Netw. Open
Artificial Intelligence Modeling: the MI-CLAIM Checklist. Nat. Med. 26, 2, e1914645. doi:10.1001/jamanetworkopen.2019.14645
1320–1324. doi:10.1038/s41591-020-1041-y Ullrich, K., van den Berg, R., Brubaker, M., Fleet, D., and Welling, M. (2019).
Oktay, O., Schlemper, J., Le Folgoc, L., Lee, M., Heinrich, M., Misawa, K., et al. Differentiable Probabilistic Models of Scientific Imaging with the Fourier Slice
(2018). Attention U-Net: Learning where to Look for the Pancreas. arXiv Theorem. arXiv [cs.LG]. Available at: http://arxiv.org/abs/1906.07582
[cs.CV]. Available at: http://arxiv.org/abs/1804.03999 (Accessed March 31, (Accessed March 31, 2022).
2022). Valanarasu, J. M. J., Sindagi, V. A., Hacihaliloglu, I., and Patel, V. M. (2020). “KiU-
Pan, S., and Yang, Q. (2010). A Survey on Transfer Learning. IEEE Transaction Net: Towards Accurate Segmentation of Biomedical Images Using Over-
Knowl. Discov. Data Eng. 22 (10), 191. doi:10.1109/tkde.2009.191 complete Representations,” in Medical Image Computing and Computer
Paper, D. (2021). “Simple Transfer Learning with TensorFlow Hub,” in State-of- Assisted Intervention – MICCAI 2020 (Lima, Peru: Springer International
the-Art Deep Learning Models in TensorFlow: Modern Machine Learning in the Publishing), 363–373. doi:10.1007/978-3-030-59719-1_36
Google Colab Ecosystem. Editor D. Paper (Berkeley, CA: Apress), 153–169. Vizcaíno, J. P., Saltarin, F., Belyaev, Y., Lyck, R., Lasser, T., Favaro, P., et al.
doi:10.1007/978-1-4842-7341-8_6 (2021). Learning to Reconstruct Confocal Microscopy Stacks from Single
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., et al. (2019). Light Field Images. IEEE Trans. Comput. Imaging 7, 775. doi:10.1109/TCI.
PyTorch: An Imperative Style, High-Performance Deep Learning Library. Adv. 2021.3097611
Neural Inf. Process. Syst. 32. Available at: https://proceedings.neurips.cc/paper/ Wang, G., Li, W., Ourselin, S., and Vercauteren, T. (2018). “Automatic Brain Tumor
2019/hash/bdbca288fee7f92f2bfa9f7012727740-Abstract.html (Accessed Segmentation Using Cascaded Anisotropic Convolutional Neural Networks,” in
March 24, 2022). Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries (Quebec
Pati, P., Jaume, G., Foncubierta-Rodríguez, A., Feroce, F., Anniciello, A. M., City, Canada: Springer International Publishing), 178–190. doi:10.1007/978-3-319-
Scognamiglio, G., et al. (2022). Hierarchical Graph Representations in 75238-9_16
Digital Pathology. Med. Image Anal. 75, 102264. doi:10.1016/j.media.2021. Wang, W., Chen, C., Ding, M., Yu, H., Zha, S., and Li, J. (2021). “TransBTS:
102264 Multimodal Brain Tumor Segmentation Using Transformer,” in Medical Image
Rakhlin, A., Shvets, A., Iglovikov, V., and Kalinin, A. A. (2018). “Deep Computing and Computer Assisted Intervention – MICCAI 2021, Strasbourg,
Convolutional Neural Networks for Breast Cancer Histology Image France, September 27–October 1, 2021 (Springer International Publishing),
Analysis,” in Image Analysis and Recognition (Póvoa de Varzim, Portugal: 109–119. doi:10.1007/978-3-030-87193-2_11
Springer International Publishing), 737–744. doi:10.1007/978-3-319- Wei, J. W., Tafe, L. J., Linnik, Y. A., Vaickus, L. J., Tomita, N., and Hassanpour, S.
93000-8_83 (2019). Pathologist-level Classification of Histologic Patterns on Resected Lung
Ribli, D., Horváth, A., Unger, Z., Pollner, P., and Csabai, I. (2018). Detecting and Adenocarcinoma Slides with Deep Neural Networks. Sci. Rep. 9, 3358. doi:10.
Classifying Lesions in Mammograms with Deep Learning. Sci. Rep. 8, 4165. 1038/s41598-019-40041-7
doi:10.1038/s41598-018-22437-z Weigert, M., Schmidt, U., Haase, R., Sugawara, K., and Myers, G. (2020). “Star-convex
Saha, D., Schmidt, U., Zhang, Q., Barbotin, A., Hu, Q., Ji, N., et al. (2020). Practical Polyhedra for 3d Object Detection and Segmentation in Microscopy,” in
Sensorless Aberration Estimation for 3D Microscopy with Deep Learning. Opt. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer
Express 28, 29044–29053. doi:10.1364/OE.401933 Vision, Snowmass village, CO, United States, March 2–5, 2020
Schindelin, J., Arganda-Carreras, I., Frise, E., Kaynig, V., Longair, M., Pietzsch, T., (openaccess.thecvf.com), 3666–3673. doi:10.1109/wacv45572.2020.9093435
et al. (2012). Fiji: an Open-Source Platform for Biological-Image Analysis. Nat. West, J., Ventura, D., and Warnick, S. (2007). Spring Research Presentation: A
Methods 9, 676–682. doi:10.1038/nmeth.2019 Theoretical Foundation for Inductive Transfer, 1. Provo, UT, United States:
Schmarje, L., Zelenka, C., Geisen, U., Glüer, C.-C., and Koch, R. (2019). “2D and Brigham Young University, College of Physical and Mathematical Sciences.
3D Segmentation of Uncertain Local Collagen Fiber Orientations in SHG Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., et al. (2020).
Microscopy,” in Pattern Recognition (Dortmund, Germany: Springer “Transformers: State-Of-The-Art Natural Language Processing,” in
International Publishing), 374–386. doi:10.1007/978-3-030-33676-9_26 Proceedings of the 2020 Conference on Empirical Methods in Natural
Schneider, C. A., Rasband, W. S., and Eliceiri, K. W. (2012). NIH Image to ImageJ: Language Processing: System Demonstrations, Online, November 16–20,
25 Years of Image Analysis. Nat. Methods 9, 671–675. doi:10.1038/nmeth.2089 2020 (Online: Association for Computational Linguistics). doi:10.18653/v1/
Shailja, S., Jiang, J., and Manjunath, B. S. (2021). “Semi Supervised Segmentation 2020.emnlp-demos.6
and Graph-Based Tracking of 3D Nuclei in Time-Lapse Microscopy,” in 2021 Wu, E., Wu, K., Cox, D., and Lotter, W. (2018). “Conditional Infilling GANs
IEEE 18th International Symposium on Biomedical Imaging (Nice, France: ISBI for Data Augmentation in Mammogram Classification,” in Image
IEEE), 385–389. doi:10.1109/isbi48211.2021.9433831 Analysis for Moving Organ, Breast, and Thoracic Images (Granada,
Shen, L., Margolies, L. R., Rothstein, J. H., Fluder, E., McBride, R., and Sieh, W. Spain: Springer International Publishing), 98–106. doi:10.1007/978-3-
(2019). Deep Learning to Improve Breast Cancer Detection on Screening 030-00946-5_11
Mammography. Sci. Rep. 9, 12495. doi:10.1038/s41598-019-48995-4 Wu, Y., Rivenson, Y., Wang, H., Luo, Y., Ben-David, E., Bentolila, L. A., et al.
Shen, Y., Wu, N., Phang, J., Park, J., Liu, K., Tyagi, S., et al. (2021). An Interpretable (2019). Three-dimensional Virtual Refocusing of Fluorescence Microscopy
Classifier for High-Resolution Breast Cancer Screening Images Utilizing Images Using Deep Learning. Nat. Methods 16, 1323–1331. doi:10.1038/
Weakly Supervised Localization. Med. Image Anal. 68, 101908. doi:10.1016/ s41592-019-0622-5
j.media.2020.101908 Yakimovich, A., Beaugnon, A., Huang, Y., and Ozkirimli, E. (2021). Labels in a
Sonnenburg, S., Braun, M. L., Ong, C. S., Bengio, S., Bottou, L., Holmes, G., et al. Haystack: Approaches beyond Supervised Learning in Biomedical
(2007). The Need for Open Source Software in Machine Learning. Available at: Applications. Patterns 2, 100383. doi:10.1016/j.patter.2021.100383
https://www.jmlr.org/papers/volume8/sonnenburg07a/sonnenburg07a.pdf Zhang, N., Li, L., Deng, S., Yu, H., Cheng, X., Zhang, W., et al. (2020). Can Fine-
(Accessed March 24, 2022). Tuning Pre-trained Models Lead to Perfect Nlp? a Study of the Generalizability

Frontiers in Bioinformatics | www.frontiersin.org 8 July 2022 | Volume 2 | Article 912809

Li et al. Open-Source Biomedical Image Analysis Models

of Relation Extraction. Available at: https://openreview.net/forum?id=3yzIj2- Conflict of Interest: AY was employed by Roche Pharma International
eBbZ (Accessed April 4, 2022). Informatics, Roche Diagnostics GmbH, Mannheim, Germany
Zhong, E. D., Bepler, T., Davis, J. H., and Berger, B. (2019). Reconstructing
Continuous Distributions of 3D Protein Structure from Cryo-EM Images. The remaining authors declare that the research was conducted in the absence of
arXiv [q-bio.QM]. Available at: http://arxiv.org/abs/1909.05215 (Accessed any commercial or financial relationships that could be construed as a potential
March 30, 2022). conflict of interest.
Zhu, W., Liu, C., Fan, W., and Xie, X. (2018a). “DeepLung: Deep 3D Dual Path Nets
for Automated Pulmonary Nodule Detection and Classification,” in 2018 IEEE Publisher’s Note: All claims expressed in this article are solely those of the authors
Winter Conference on Applications of Computer Vision (WACV) and do not necessarily represent those of their affiliated organizations, or those of
(ieeexplore.ieee.org), Granada, Spain, March 12–15, 2018, 673–681. doi:10. the publisher, the editors and the reviewers. Any product that may be evaluated in
1109/wacv.2018.00079 this article, or claim that may be made by its manufacturer, is not guaranteed or
Zhu, W., Vang, Y. S., Huang, Y., and Xie, X. (2018b). “DeepEM: Deep 3D ConvNets endorsed by the publisher.
with EM for Weakly Supervised Pulmonary Nodule Detection,” in Medical
Image Computing and Computer Assisted Intervention – MICCAI 2018 Copyright © 2022 Li, Sharma, Thangamani and Yakimovich. This is an open-access
(Springer International Publishing), Lake Tahoe, NV, United States, March article distributed under the terms of the Creative Commons Attribution License (CC BY).
12–15, 2018, 812–820. doi:10.1007/978-3-030-00934-2_90 The use, distribution or reproduction in other forums is permitted, provided the original
Zunair, H., and Ben Hamza, A. (2020). Melanoma Detection Using Adversarial author(s) and the copyright owner(s) are credited and that the original publication in this
Training and Deep Transfer Learning. Phys. Med. Biol. 65, 135005. doi:10.1088/ journal is cited, in accordance with accepted academic practice. No use, distribution or
1361-6560/ab86d3 reproduction is permitted which does not comply with these terms.

Frontiers in Bioinformatics | www.frontiersin.org 9 July 2022 | Volume 2 | Article 912809