Image provider.

Image providers supply tile-based image data, provide search functionality, and read and store metadata like annotations and models. Out-of-the-box, Orbit comes with a local image provider which reads images from the disk in standalone mode. The BioFormats library from the Open Microscopy Environment (https://www.openmicroscopy.org/) is used to read many proprietary WSI file formats. A unique ID based on the md5 hash of the image data is assigned to each image. Metadata is stored in a SQLite database on the local disk. In addition, the image provider OMERO can be used to connect to the open-source image server OMERO [24]. This image provider makes use of the OMERO-API to read tile-based image data and to store metadata. The authentication mechanisms and access rights of OMERO are used in this context. This allows a collaborative workflow in a distributed environment, e.g. biologists can create annotations, and image analysis experts can run analysis algorithms within these annotations. Other image providers can be easily integrated by implementing an IImageProvider interface so that other imaging infrastructures can be used as a data source.

Region of interest definition.

For WSI, definition of the region of interest (ROI) is very important. This defines e.g. where the tissue on the slide is located, or more specifically, which parts of a tissue should be analysed. For this, Orbit provides two ROI definition modules. First, manual annotations can be drawn using the annotation tool panel. Several tools including a ‘magnetic lasso’ can be used to define a combination of ROIs, exclusion regions and explicit inclusion regions (areas which will be included even if excluded by other rules like exclusion annotations). Second, a so-called exclusion model can be trained. This is a pixel-based machine-learning model trained on a lower resolution of the image to define inclusion and exclusion areas. This powerful feature greatly speeds up tedious manual annotation tasks. Such an automated approach can be combined with the manual annotations, which have priority.

Map-reduce paradigm.

Image processing within Orbit is performed in map-reduce manner. The map step is performed at the tile level; the reduce step aggregates all results from the map step. This allows arbitrary algorithms, e.g. ImageJ [25], Fiji [16], or Cell Profiler [26], to run on small in-memory image tiles, even if the algorithm is not designed for WSI. A good example is cell count: the map step counts cells in each tile (e.g. 512x512 pixel), the reduce step is simply a sum function. The advantage of this process is that the map steps can be executed independently of one another, enabling it to be executed in a multi-threaded manner on a local computer or distributed, e.g. using an Apache Spark environment. Orbit comes with built-in algorithms like an advanced structure-based pixel-classification, object segmentation and object classification. In addition, other algorithms or tools like Cell Profiler, ImageJ Macros, or Deep Learning segmentation using TensorFlow [27] can be used in the map step.

This generic map-reduce framework was developed in order to allow the implementation of map and reduce steps, followed by the execution of these steps using different executors. Orbit comes with a local executor which performs map-reduce tasks multithreaded on the local computer. In addition, an executor for Apache Spark can be used to execute tasks on a cluster, if available. This scale-out functionality allows the use of on-premise clusters with a Spark installation, or the use of cloud-based services, such as Amazon’s EMR. Additional executors can be developed easily by implementing the IMapReduceExecutor interface.

Script editor.

An integrated Groovy script editor enables the execution of Groovy Script directly in Orbit (Fig 2). The advantage of the integrated editor is that the code execution runs within the same Java Virtual Machine (JVM) and thus can access all active objects and has the full Orbit plus dependencies class-path available. The main controller class OrbitImageAnalysis is a singleton and can be used to easily access all open images and models. It also can read and process all image data and even write back objects, e.g. polygons or an overlay map.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Integrated script editor.

The integrated script editor allows the execution of Groovy scripts which can access the Orbit API. Results can be visualized as a mark-up on open image frames.

https://doi.org/10.1371/journal.pcbi.1007313.g002

Scripting allows the automated execution of manually performed steps in the user interface, and in addition enables the execution of complex analysis steps which are not integrated within the user interface. Orbit provides sophisticated helper methods in the OrbitHelper class to handle ROIs, classify pixels, segment objects or classify objects. Image data can be requested to execute arbitrary algorithms on tile data, e.g. to run an ImageJ macro on it, which enables the full power of such libraries to be harnessed at the tile level.

The benefit of the script editor is that the workload of image analysis specialists and end-users can be split: the more programming-oriented image analysis specialists can write parameterized scripts so that the end-users can adapt some parameters (e.g. a threshold), and without requiring deep knowledge of scripting, run the script, validate the results, and adjust the parameters if required. The validation can be done visually if the script writes segmentation objects or a classification map back to the image.

For headless script execution, the GroovyExecutor class takes the URL of a Groovy-Script as argument and executes the Groovy code. This allows Orbit to be run on a server or HPC cluster and to submit a script which is located in e.g. a source control system. This feature is supplementary to the map-reduce principle; in fact, a script itself can start map-reduce processes and thus run analysis tasks at scale.

Pixel classification.

One of the basic principles in Orbit is the classification of pixels. Therefore, the user defines a set of classes, e.g. ‘background’, ‘staining 1’, ‘staining 2’ and assigns a colour to each class. One important parameter to set is the so-called structure size. This parameter defines the size of the surrounding context taken into account for each pixel. For each class, the user draws several shapes using a polygon, rectangle or circle tool. The union of all pixels inside these training-shapes define the training set. Several images can be used to define the training set with the goal of covering the entire variability of the tissue structures. For each pixel inside the training set, a feature extraction is performed, which takes into account not just the pixel itself, but a pre-defined area around each pixel delimited by the structure size. Based on this data the following features are computed: min, max, mean, standard deviation, edge-factor, and the middle pixel intensity per channel. For example, the edge-factor for pixel p characterizes the edginess and is defined as (1) W defines the set of pixels inside the structure-size window around pixel p.

These features are used as input for a linear SVM which is used to train a statistical model for pixel classification. This circumvents the need for the user to specify any thresholds, as the tissue class discrimination is done simply by drawing shapes on representative regions. The structure-size here is very important: smaller values discriminate more fine-grained structures, higher values take more surrounding context into account. Orbit always makes use of all computed features and leaves to the SVM to weight the features appropriately. A maximum of 40,000 pixels is randomly picked from all training regions in order to perform a fast training. The training-set accuracy is computed internally (and shown in the log) on the training-set, without cross-validation. This explains the accuracy at which the model can discriminate the training classes and a warning popup appears if the accuracy is below 0.85, which indicates when the training regions should be placed differently. Usually a very small set of training regions per class is sufficient. However, Orbit allows training to be performed across several images in case there is a variation from image to image (e.g. variations in intensity). Comparable approaches like fastER [28] work in a similar manner and may be incorporated in future versions of Orbit.

Object segmentation.

An object segmentation task groups the pixels that define an object, and requires a definition of foreground and background pixels, which can be determined from a pixel classification model that is run as a preceding step. Object segmentation in the biology domain is mainly used to segment cells, as visualized in Fig 3. Here the main challenge, especially in tissues, is that cells often overlap. Orbit provides a standard watershed algorithm [29] in combination with erosion and dilation steps which solves most of the problems. For challenging problems like dense cell clusters, a Mumford-Shah [30] functional-based segmentation algorithm can be used.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Object classification.

Upper image: After an object segmentation the user can define object classes (1), use the object marker tool (2) and mark segmented objects (3). Orbit computes features for each object and uses a SVM to classify the objects. Lower image: Results of the object classification.

https://doi.org/10.1371/journal.pcbi.1007313.g003

Two-level segmentation performs a second segmentation within segmented objects. For this, a second classification model is built and applied within the detected objects of the primary segmentation model. This is very useful for e.g. mRNA detection which is visible as small spots inside cells.

By default, Orbit simply outputs the number of segmented objects; however, in the segmentation settings the Features option can be activated. This enables the feature computation per segmented object which can work with several other parameters like shape descriptors, texture descriptors, location information, and outputs this information per segmented object.

Object classification.

A machine-learning based object classification can be applied after the object segmentation. The idea is that the user can mark segmented objects and assign them to specific classes—without thinking about how to describe the differences between the object classes. To achieve this, Orbit provides an object marker tool which can be used after a segmentation step. For each class, the user marks several objects which defines the training set. Orbit internally computes the object features per segmented shape which will be used to train a SVM. Based on this statistical model, all segmented objects are labelled. The object classification can be seen as an additional step to the object segmentation. Sometimes this is very useful for discriminating objects, sometimes it is more appropriate to analyse the output of the object features with external data analysis tools.

Region of interest.

The region of interest (ROI) is the area in which other algorithms are applied, e.g. an object segmentation only segments objects within the valid ROI and skips the other parts of a WSI. Usually the ROI defines the tissue and excludes the background, but it can also exclude unwanted parts like holes or wrinkles.

Annotations.

Manual annotations can be drawn in the annotation panel using annotation tools like a pencil or a magnetic lasso. By default, an annotation can have a name and is just an informative annotation. However, annotations can also be edited and assigned a special type. Annotations of the type ROI define the region of interest (e.g. area outside the ROIs is excluded). If multiple ROIs are created, the final ROI for the image is the union of all ROI annotations. Exclusion annotations define excluded areas (e.g. inside a ROI annotation), unless another annotation of type inclusion exists. Inclusion annotations have the highest priority and overrule exclusion annotations. All created annotations are saved in the database and are automatically uploaded whenever an image is accessed.

Exclusion models.

Technically an exclusion model is a classification model on a low resolution (approx. one megapixel) with a high structure-size. For each class, the user can define whether it is an inclusion or exclusion class. The training is done the same way as for a standard classification model, but a special exclusion model training automatically uses a low resolution of the image and later maps it to the full resolution. Manual annotations (e.g. exclusion annotations) and exclusion models can be combined, so some parts of a WSI can be explicitly excluded, even if the exclusion model would include it.

Masks.

The masking functionality in Orbit is a generic approach for defining areas which are included and those which are excluded from analysis. In contrast to the exclusion model and manual annotations, masks do not affect the measured size of a ROI.

A classification mask is a standard pixel classification model with additional flags for each class defining whether the class should be included or excluded. In contrast to an exclusion model, the main model usually is applied on a high image resolution. The classification of the pixels is performed on-the-fly tile-by-tile when the main analysis on the particular tile is performed. One characteristic here is that the mask classification and the main analysis can be applied on different channels. This allows a channel co-localization quantification as is often used in fluorescence imaging.

A segmentation mask is a segmentation model in which all pixels inside segmented objects are declared as active pixels used by further analysis. This allows a pixel classification inside segmented objects, or a nested segmentation.

Tools.

Orbit provides many other tools out-of-the-box to cope with the most common image analysis problems. These include (among others) a tool to run Cell Profiler pipelines within the ROI defined by Orbit and read back the cell segmentations, a nerve fibre detection tool, a tissue microarray (TMA) spot detection tool and a rare event detection tool. The latter is a very flexible tool which performs an object segmentation and presents all detected objects to the user who then can manually assign each object to a class. A report outputs the frequencies of the assigned classes.

In addition to the built-in tools, image analysis specialists can create custom tools which are instantiated via Java reflection and listed as ribbons in the custom tab. The developer has access to all open images, annotations and models in order to perform an analysis task.

Training.

To build a new deep learning model from scratch, the first step is to annotate a set of representative objects over several images using the annotation tools. For the glomeruli example we annotated 92 slide images resulting in around 21,000 annotations. This huge amount of annotations was necessary because the aim was to create a model which is able to cope with two species (mouse and rat) and several different staining protocols: H&DAB, FastRed, PAS, and three variations of H&E. The ultimate goal was to detect glomeruli on any WSI. In addition to the object annotations, we drew a ROI annotation to define the region which we considered for the annotations. This implies that outside the ROI there might be further objects which are not taken into account.

Orbit generates 512x512 pixel-sized tile masks which encode objects and background out of the manually drawn annotations (Fig 7). These masks act as the training set for the actual training of the deep learning model. We provide a Python script for training which has to be executed outside of Orbit, ideally on a machine with a GPU (https://www.orbit.bio/deep-learning-object-segmentation/). The script uses TensorFlow as a deep learning framework and runs out-of-the-box, where only the file path of the tile masks has to be defined. As a base CNN we use a ResNet-101 network structure [39] pre-trained on the MSCOCO dataset [23]. The input tensor is a tile image and the output (target value) is its corresponding tile mask. The encoder-decoder network structure is depicted in Fig 7.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Deep-learning object detection.

Top: Annotated objects on the left side, generated segmentation map which is used as training input on the right side. Bottom: The ResNet-101 based encoder-decoder network takes a tile image as input and outputs the tile mask.

https://doi.org/10.1371/journal.pcbi.1007313.g007

Segmentation.

Once the model is created, or a pre-trained model downloaded, objects on images can be segmented. As a first step the CNN model was used to predict the tile masks. We observed that the model detected objects very well if they were located at the centre of the tile, but often only partially when they were located close to the tile border. Fortunately, the false-positive pixel-rate in the tile mask was close to zero, indicating that the model correctly detected a pixel inside an object—but just missed some. This highlighted the need for a segmentation refinement step: arbitrary translations were applied to the tile, for each variant the tile mask was predicted, and the refined result mask was the union of all variant masks with respect to their inverse translation. For the glomeruli detection, four translations were applied: half the tile size up, down, left and right. Based on the tile mask, the object segmentation was performed using a pre-defined segmentation model. Another refinement was applied to cope with tile-crossing objects: each segmented object was centred, a virtual tile around the centre of the object was extracted and used as input for another segmentation step. The idea was that partially segmented objects were to be detected completely if they were located at the centre of the tile. All segmented objects were stored in an object list. A post-processing step eliminated duplicate objects from that list by only keeping the ones which cover others.

Evaluation.

The complete dataset consisted of 90 whole slide kidney images from rat and mouse samples and several types of stains (https://doi.org/10.5061/dryad.fqz612jpc). Fig 6 shows the huge variability between the stainings. For this evaluation, the data set was divided into 58 training and 32 test images. Each slide image contained around 300 glomeruli objects. For each group (species x staining) the test samples were independent but randomly selected to ensure a good stratification. The training set was virtually enhanced with 90- and 180-degree rotation images in order to achieve rotation invariance. The complete training set including the rotations contained 87213 images and masks, each 512x512 pixels. The training of the network was performed on a high-end gaming GPU and lasted 4 days for 410,000 training iterations with a batch size of 2.

The evaluation of the test set was based on the Dice index [40] which scores the overlap between pixels in the range of [0,1] where 1 means a complete overlap. For each ground truth object, the closest segmented object was taken into account and vice versa. The sum of the two overlap ratios was divided by two. We used the dice object index as defined in [41] to score a complete image by using the Dice index for all objects, weighted by its object size.

Table 1 shows the Dice object index results per group and the total. The scores show a good segmentation performance over a set of 32 whole slide images, each containing around 300 glomeruli. This numeric evaluation has been confirmed by a subjective visual evaluation; an example is shown in Fig 6.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Evaluation results of the glomeruli segmentation for each group and total.

Samples of each staining type are illustrated in Fig 6.

https://doi.org/10.1371/journal.pcbi.1007313.t001

Note that the ground truth was defined by manual annotations which might not be fully accurate or may miss some glomeruli. Indeed, a visual inspection did show that the automated detection found some new glomeruli which had not been annotated. Interestingly, the models performed very well despite a high staining variability, e.g. H&E images greatly differ from FastRed and H&DAB images, but the model was able to detect glomeruli regardless of the staining protocol used.