2.1 Cytomine-core

Cytomine-Core relies on recent web and database software development technologies. Its underlying data models (Supplementary Note 2) allow to create and store projects. Each project can be accessed by multiple users through authentication. A project can contain multi-gigapixel image sequences and a user-defined ontology, i.e. a structured list of domain-specific semantic terms. Each image instance can be annotated by users or software using annotation objects of various shapes for regions of interest (e.g. a cell or a tissue subregion) and labeled with one or multiple semantic terms from the ontology (e.g. a specific cell type or tissue structure). In addition, metadata (key-value properties, associated files and rich descriptions) can be associated to any project, image and annotation. Such data can be created remotely either by human experts (through Cytomine-WebUI) or automatically (by our analysis modules or any third-party software implementing basic web communication mechanisms). Because these data are identified by URLs they are de facto shared with any authenticated user. Also, as they are represented in standard formats (namely JSON, a lightweight data-interchange format), they can be automatically parsed and generated by registered external applications.

2.2 Cytomine-IMS

Cytomine-IMS backend server provides web services that encapsulate a collection of distributed, specialized image server instances. It is used to upload 5D image sequences (x,y,z,c,t planes) and to dynamically deliver original image areas and annotation masks over the internet – at any pyramid resolution. It supports various standards and specific microscopy image formats (including most of whole-slide scanner formats) either by directly accessing their native formats, or by seamlessl conversion to a pyramidal format during the upload phase (see Supplementary Note 1 for a list of supported formats).

2.3 Cytomine-WebUI

Cytomine-WebUI is a customizable and responsive rich internet application (Fig. 2), accessible through regular web browsers and mobile devices. It allows to create, organize, visualize and edit all data. It includes a zoomable, tile-based viewer for multi-gigapixel images with the visualization of overlaid (human or computer-generated) annotation layers and their properties. Furthermore, an ontology editor, several modules to derive annotation statistics and visualize annotation galleries, a textual search engine and proofreading tools for expert reviewing of annotation objects are part of this user interface. In addition, we have implemented functionalities to allow various forms of collaborative works. One of them is the tracking of all user activities to e.g. allow multiple users to follow remotely another user’s observation paths and actions. Conversely, a blinded mode can be activated to hide image and user information to allow independent studies and reduce bias when analyzing imaging data. An additional module (Cytomine-IRIS, the interobserver reliability study module) also allows independent ground-truth construction and inter-observer annotation statistics e.g. to identify cell type classification disagreements among experts.

Open in a separate window

Fig. 2.

Overview of Cytomine-WebUI: (a) Zoomable multi-gigapixel image viewer (a la Google Maps) with overlaid annotations colored according to ontology terms (Original image size: 19968×25088 pixels). (b) Annotation drawing tools including various shapes and operations on polygons. (c) Gallery of bronchus annotations in current image. (d) Main menu including project listing, ontology editor, storage to upload images, user activity statistics, textual search engine. (e) Selected annotation panel with thumbnail, suggested terms (based on content-based image retrieval algorithm), textual description. (f) Project-specific, user-defined ontology for semantic annotation. (g) Activation of annotation layers of possibly distributed users and softwares. (h) Annotation properties (key-value pairs). (i) Proofreading tools to accept or edit annotations. (j) Job template panel to launch pre-configured processing routines on regions of interest. (k) Gigapixel image overview with current position. (l) Multidimensional image panel with selectors for channel, slice in a z-stack, and time point. (m) Image layer panel to apply on-the-fly tile image processing

3.1 Tissue area quantification

In these use cases, scientists aims at quantifying the size (area) of tissue regions (e.g. the ratio of tumor islets with respect to whole tissue sections). This type of task implies to delineate the whole tissue section as well as the specific regions of interest within the tissue, either manually or semi-automatically (see Supplementary note 5.2.4.1 for a step-by-step guide using automatic recognition algorithms on toy data).

Following these principles, Cytomine enabled semi-automatic tumor area assessment in hundreds of whole lung Hematoxylin–Eosin (H&E) stained digital slides in mice inflammation and cancer research (Marée et al., 2014) (Fig. 3a). Experts (pneumologists and biomedical researchers) first used Cytomine-WebUI drawing tools to provide manual tumoral islets and non-tumour annotations. Cropped images of these annotations were retrieved using web services and fed into our supervised learning algorithms for semantic segmentation. The task was formulated as pixel classification problem using multiple outputs. User interfaces and communication mechanisms to launch algorithms from Cytomine-WebUI were implemented to allow experts to execute training and prediction algorithms in an autonomous way. As our algorithms were not perfectly recognizing tumors, tools were implemented in Cytomine-WebUI to allow scientists to proofread annotations that were generated automatically. Experts are therefore able to accept or reject annotations and edit their shapes using drawing tools which allow to edit vertices, scale, substract or merge polygons, or fill internal holes. These manual operations are automatically translated internally into spatial queries on polygons, and validated annotations are stored in Cytomine-Core. After expert validation, statistics can be exported in standard formats for further analysis.

A similar workflow was used in (Leroi et al., 2015) for semi-automatic tumor delineation in tens of whole Hematoxylin–Diaminobenzidine (HDAB) stained immunohistochemical digital slides in mice lung cancer research (Fig. 3b). Manual annotations (tumor, stroma and necrosis) were provided by experts to build a binary semantic segmentation model whose predictions were then proofread. In this study, this step was followed by quantitative assessment of antibody staining in relevant tissue area.

Finally in (Suarez-Carmona et al., 2015), Cytomine was used to enable independent assessment by two observers (using the blinded configuration mode) of recruitment of CD11b+GR1+myeloid-derived suppressor cells using mouse ear sponge models from whole immunofluorescent stained frozen sections. Experts used manual freehand annotation tools and multidimensional image visualization interface to analyze and merge fluorescent images (Fig. 3c).

3.2 Scoring and object counting

In these use cases, scientists aim at scoring or counting ‘objects’. This type of task implies to define (manually or automatically) regions of interest and count different types of ‘objects’ (e.g. cells marked with a marker-specific antibody) within these regions.

Cytomine was used to enable independent assessment by two observers (using our blinded configuration mode) of tens of thousands of BRCA1 mRNA expression signals and nucleus counts by in situ hybridization assays in tens of formalin-fixed, paraffin-embedded tissues in human breast cancer research (Boukerroucha et al., 2015) (Fig. 3e). This study involved pathologists (for manual tumor delineation) and biomedical researchers (for manual annotation of spots and nucleus using point annotations). It required the development of web services performing polygon intersection operations to count spots within specific regions of interest. In addition, scripts using these web services were implemented to export quantification statistics in standard formats for further statistical analysis. Similarly, experts in sexual maturation research performed manual classification and counting (using point annotations) of thousands of oocytes in whole H&E slides of Chondrostoma nasus (Fig. 3d).

In Weekers et al. (2015), semi-automatic counting of immuno-reactive cells in regions of interest (cortex, medulla, corticomedullary junction) of tens of kidney sections was performed (Fig. 3i). Experts (nephrologists, pathologists and biomedical researchers) first provided manual freehand annotations (regions of interest, positive and negative cells) to train semantic segmentation models. These were applied for positive cell detection whose statistics were exported for each region of interest.

Other applications include manual double-blind scoring within tissue subregions from immuno-histostained digital slides in melanoma cellular microenvironment research (Fig. 3j), and manual point annotations of hundreds of thousands of nuclei for microproteomics from small regions of interest in H&E formalin-fixed paraffin-embedded tissue samples in human breast cancer research (Longuespée et al., 2015) (Fig. 3k).

3.3 Labeled ground truth creation and object classification

In this family of tasks, scientists aim at sorting ‘objects’ (e.g. to detect rare abnormal cells or phenotypes). This type of task implies to detect objects and then classify them according to predefined categories (see Supplementary Note 5.2.4.2 for a detailed guide on using automatic detection and recognition algorithms on cytology toy data, and Supplementary Note 5.2.7.2 that describes how to create independent ground truth data).

Following these principles, Cytomine enabled manual semantic annotation of eleven categories of Danio rerio larva defects (e.g. edema, dead, curved tail,…) in hundreds of brightfield microscopy images by consensus voting of three biologists (Fig. 3h). These annotations were then used as ground-truth to build a worfklow for automatic phenotype classification using tree-based supervised learning (Jeanray et al., 2015).

In Marée et al. (2016), we analyzed tissue components in human renal biopsies (Masson-Trichrome stain). We proposed an automatic glomeruli detection workflow combining image processing operations using Icy (de Chaumont et al., 2012) and variants of our supervised classification algorithms. Icy was registered in Cytomine-Core using our software parameter templating mechanisms and it was therefore able to import and export image and annotation data using our web services (see examples in Supplementary Note 5.3). To build a large ground truth dataset (almost thirty thousand tissue components), glomeruli candidates automatically detected by Icy were analyzed using our proofreading tools for object classification. These interfaces show galleries of classified objects and allow a user to readily validate or correct (by drag and drop) predictions of ontology terms.

Other large ground truth datasets were collected using manual annotation tools. Several thousands of cells were annotated to build a large ground truth dataset in human thyroid cytology for the (ongoing) development of rare cell detection algorithms (Fig. 3f), inspired by previous work on cervical cancer screening (Delga et al., 2014). We also implemented novel user interfaces (Cytomine-IRIS, see Supplementary Note 5.2.7.2) to enable different users to independently assign ontology terms to objects of interest. In particular, it was used by several pathologists to annotate bone marrow cells in order to study inter-observer agreements and build a large concordant ground truth dataset for the design of cell classification algorithms.

We thank Pierre Ansen, Julien Confetti and Olivier Caubo for various code contributions, and Alain Empain for system administration. Natacha Rocks, Fabienne Perin, Didier Cataldo, Caroline Degand and Isabelle Salmon were early testers of the software who provided useful feedback.