Abstract

INTRODUCTION

Anatomic pathology stands at the threshold of a major transformation. Starting in the late 20^th century and continuing into the 21^st century, digital slide technology has evolved from early attempts to manually digitize the entire glass slide to sophisticated slide formats and robotic scanning devices. The online storage, Internet-based access and archival preservation of digital slides (also known as “whole-slide images”) overcome numerous areas of inefficiency in the conventional workflow of anatomic pathology. For example, they eliminate the need for physical transport and archival storage of glass slides. They also enable within-hospital-network and Internet-based sharing of digital slides for a variety of purposes ranging from diagnoses and quality control to medical research. In addition, there are numerous advances in computer vision and machine learning that enable automated or semi-automated analysis and comparison of digital slides.

For example, an area ripe for the taking is what pathologists affectionately refer to as “wallpaper matching” − namely, comparison of an unknown diagnostic entity encountered under the microscope to entities described and/or photographed in textbooks. This is a long and arduous process with conventional textbooks, hampered by the fact that color imagery is minimal in printed literature because it increases printing costs. Further, only small regions of entire glass slides can be shown in textbooks because their format is too small. If an entire glass slide were to be photographed at ×40 objective (×400 magnification equivalent), the size of the image would roughly be the size of a football field! Using digital slides in conjunction with deep-zoom visualization software and a content-based image search tool called Diamond,[1,2] we hope to circumvent this problem. This would enable pathologists to leverage Internet-based software to greatly improve the speed and accuracy of analysis and to rapidly identify archival material that is comparable to current material.

An important non-technical challenge when transitioning to digital slides will be the ability and willingness of experienced pathologists who are comfortable with glass slides to embrace new technology. Digital pathology will need to offer some benefits (such as improved accuracy, speed or convenience) to practicing pathologists to encourage them to become more proficient with the new workflows. These changes will take time to percolate through the system, possibly a decade or even longer. Newly-trained pathologists who are comfortable with digital slides from the beginning of their careers will accelerate this process of transformation.

One technical challenge is the difficulty of processing digital slides using standard image processing tools and libraries. These are typically designed for images that can be comfortably uncompressed into memory. Unfortunately, digital slides are enormous relative to image data such as personal photographs. Even after aggressive compression, file sizes of hundreds of megabytes to many gigabytes are common in this domain. At full resolution, images routinely exceed memory sizes and often occupy tens of gigabytes when uncompressed. These images are typically multi-resolution, with only a small amount of image data being relevant at any particular resolution and viewpoint.

A second technical challenge is that each vendor implements its own digital slide formats, libraries and viewers. Vendors typically do not document their formats. Even when there is documentation, important details are omitted or inaccurate. Because a vendor's library or viewer is the only way to view a particular digital slide, doctors and researchers suffer from long-term vendor lock-in − they are unable to take advantage of improvements offered by other vendors. In 2010, a Digital Imaging and Communications in Medicine (DICOM) standard for digital slides was released (supplement 145).[3,4] This standard was based on experience with the DICOM standard for visible light images (supplement 15, released in 1999)[5] and the efforts of DICOM Working Group 26.[6] Although the DICOM standard may eventually alleviate the problem of format proliferation, there is no evidence of this yet. Realistically, it may be many years before this standard is widely supported by all vendors. Even after widespread adoption, each vendor may continue to preferentially advocate its own proprietary format and merely provide slow export/import functionality for conversion to/from the DICOM standard. Further, unless an extensive effort of format conversion is undertaken, access to archived digital slides in pre-DICOM formats will remain an important obstacle.

A third technical challenge is that few vendors provide libraries and viewers for non-Windows platforms. While Windows dominates clinical settings, there is significant use of Linux in medical research and some use of Apple Mac OS X. In addition, the explosive growth of mobile platforms such as Android and iOS tablets and smartphones suggests that these platforms may play important roles in future clinical settings. Some vendors offer multi-platform support through a server-based approach, but this hurts performance by adding a network round-trip delay on every digital slide operation. This makes it suboptimal for high-latency settings such as 3G and 4G cellular networks and unusable in non-networked environments. There are also some open-source projects to read and convert digital slides, such as the LOCI Bio-Formats library[7] and the NYU Virtual Microscope project,[8] but these are limited to a particular programming environment such as Java or to a small number of slide formats.

To address these technical challenges, we have created OpenSlide. This is a C library for reading and manipulating digital slides of diverse vendor formats. This library can be linked to applications written in C or C++, as well as to applications written in other programming languages such as Java and Python that support C bindings. The use of OpenSlide enables clean separation of concerns for an application developer. Rather than dealing with proliferation of vendor-specific formats, the developer need only develop his application using the OpenSlide application programming interface (API). It is OpenSlide's responsibility to deal with the idiosyncrasies of different vendor formats and to translate back and forth at runtime between the OpenSlide API and a specific vendor's data format. In this context, the DICOM standard can be viewed as yet another vendor format. OpenSlide currently handles the following vendor formats:

• Aperio SVS

• Hamamatsu VMS

• Hamamatsu VMU

• Leica SCN

• 3DHISTECH MRXS ("MIRAX")

• Trestle TIFF

• Generic tiled TIFF.

Since OpenSlide is released as open-source software under the LGPL v2.1 license, the digital pathology community can collaboratively extend it to support new vendor formats. Today, OpenSlide is supported on many Linux distributions, on Windows and on Mac OS X. Although it is not available yet on Android and iOS, we do not foresee any fundamental obstacles to supporting OpenSlide on these platforms.

DESIGN AND IMPLEMENTATION

OpenSlide aims for a good user experience on desktop-class hardware. Its design leverages the fact that although digital slide formats vary widely across vendors, they all address a common set of problems. As a result, vendor formats typically share some broad high-level features:

• They pre-compute and store downsampled versions of the full-resolution image for quicker access to lower resolutions

• They use various kinds of lossy compression

• They are, to varying degrees, optimized for random access

• They are effectively unbounded in width and height

• They can store slide metadata and additional small images such as thumbnails that serve as annotations

• They may not store pixel data for all regions of the image.

OpenSlide is designed to support these features while shielding applications from the details of the various vendor formats. Its vendor-neutral API provides read-only access to digital slides stored in a file system. Table 1 shows the API calls that relate to file access. For brevity and clarity, the prefix “openslide_” is omitted from the calls shown in Tables 1 through 5 and in their discussion throughout this section.

Table 1

Calls to access files

Open in a separate window

The OpenSlide API provides efficient random access to multiresolution image data using the concept of pyramid levels. Table 2 shows the relevant calls. A digital slide is represented as an ordered list of pyramid levels; level 0 is the highest-resolution level and each subsequent level is a downsampled version of the previous level. In general, no image scaling is performed by the library; the only levels available through the API are those actually stored in the slide file. The centerpiece is the read_region() function, which extracts a rectangular region of a whole-slide image at a particular pyramid level. Image data is returned in uncompressed, premultiplied ARGB (Alpha, Red, Green and Blue) format. Additional calls exist to determine the downsampling factor for each level (relative to level 0) and to determine the next largest level for an arbitrary downsample factor. All image coordinates are specified with respect to the coordinate system of level 0. Pixel data for missing image regions is rendered as transparency.

Table 2

Calls relating to image pyramids

Open in a separate window

The API provides functions for accessing slide metadata and auxiliary images. All other vendor-specific information, such as tile placement and compression formats, is hidden from the application; access to pixel data is only provided in uncompressed, premultiplied ARGB format. This allows OpenSlide to be extended to support new slide formats without changing the external API or the design of applications using the library. As shown in Table 3, OpenSlide exposes slide metadata as a set of properties, each of which is identified by a string and has a string-typed value. OpenSlide exposes generic properties for common slide metadata. Individual vendor drivers also expose properties containing format-specific metadata. In addition, small additional images embedded in the slide file, such as thumbnail or barcode images, are exposed as associated images. Unlike the main slide image, associated images are single-resolution and can only be read in their entirety. Table 4 shows the calls relating to associated images.

Table 3

Calls relating to properties

Open in a separate window

Table 4

Calls relating to associated images

Open in a separate window

Calls for error handling and version control are shown in Table 5. If an unrecoverable error such as an I/O error occurs while a slide is being read, the slide handle transitions terminally into an error state. Subsequent invocations of OpenSlide functions on the handle (other than close()) will have no effect; functions that are expected to return values will instead return an error value; read_region() will clear its destination buffer instead of filling it. The get_error() function can be used to check for errors and obtain a string describing the error. This style of error handling allows programs written in C to check for errors only when convenient, without the need to check after every call.

Table 5

Utility calls

Open in a separate window

OpenSlide is implemented in C99 and uses GLib[9] extensively for memory allocation, data structure manipulation and text file parsing. OpenSlide relies upon LibTIFF,[10] IJG's JPEG library,[11] OpenJPEG,[12] and LibXML2[13] for image reading and processing of the various formats and upon the Cairo graphics library[14] for positioning and rendering slide tiles.

OpenSlide's internal architecture is structured in a manner similar to the device driver model found in operating systems. Application-facing code is linked to vendor-specific code by way of internal constructors and function pointers. When a slide file is first opened, vendor-specific driver code is responsible for parsing the slide's metadata and locating the compressed image data for each region of the slide. Thereafter, read requests are forwarded to the vendor-specific driver, which reads the appropriate compressed data from the slide file and renders it to an ARGB image. To do this, the driver can handle the request itself or – in simple cases – forward it to generic handlers provided by the OpenSlide core.

DIGITAL SLIDE FORMATS

As previously discussed, many vendors provide limited or no public documentation for their slide formats. In some cases documentation is available, but only under disclosure restrictions incompatible with open-source software. As a result, OpenSlide's support for many formats has been implemented via empirical analysis of the raw slide data and a significant amount of trial and error. This is an inherently imperfect and iterative process that relies critically on feedback from the user community. As an OpenSlide driver gets more widely and extensively used, we discover more subtle cases where our understanding of the file format is in error. This leads to modifications that result in a more accurate version of the driver. Over time, the implementation of the driver converges to a stable and accurate version.

This approach is dependent upon the availability of sample slide files for analysis. Because it is often difficult to create these samples without access to a corresponding scanner, we are often dependent on the OpenSlide user community to contribute digital slides in interesting formats. The remainder of this section summarizes the slide formats currently supported by OpenSlide. More detailed documentation, as well as a variety of freely-redistributable slide files, is available on the OpenSlide website.[15]

EXAMPLE APPLICATIONS

Web-Based Viewer

To enable remote viewing of digital slides across the Internet, we have created a web application using the OpenSeadragon AJAX image viewer[21] as shown in Figure 1. The client component of our application is JavaScript that is dynamically loaded into a standard web browser from the web page being browsed. The server component is a web server that returns tiles of specified coordinates and zoom levels on demand. These tiles can be pre-generated using an OpenSlide-based tool and served from the file system or can be produced on demand by a web application that invokes OpenSlide at runtime to generate the tiles. Our experience is that there is no user-perceptible performance difference between dynamically-tiled and statically-tiled viewing sessions. Using dynamic tiling avoids a preprocessing step in the workflow and eliminates the storage overhead of storing tiles. A demo of our viewer is available at http://openslide.org/demo/.

Open in a separate window

Figure 1

Internet viewer for digital slides

PathFind

As mentioned earlier, enabling knowledge from previously encountered entities to be brought to bear on an unknown, but similar-looking diagnostic entity is a valuable capability in pathology. Of course, “similar-looking” is expressed in terms of visual attributes that have specific significance to the pathology of the tissue sample being examined. PathFind is an OpenSlide-based tool that accepts a wide range of similarity-detection software called searchlets. Each searchlet is a plugin for the OpenDiamond^® platform, which enables interactive, semi-automated, hypothesis-driven searches of complex images. (The OpenDiamond platform was created by the same research group that created OpenSlide.) For example, we have created searchlets for similarity detection of cytologic atypia and pagetoid spread in skin lesions, using a machine-learning technique called Semantic Texton Forests.[22] We have also created searchlets for attributes such as nuclear density using ImageJ,[23] an open-source tool supported by the National Institutes of Health. Figure 2 shows the user interface presented to a pathologist by PathFind. The pathologist can zoom and navigate case data just as he does with a microscope and glass slides today. At any point, he can request a search for archival digital slides that have areas similar to the area currently in view. The results that are returned include entire digital slides as well as specific matching regions within each of those digital slides. Figure 3 shows the end-to-end architecture of the whole system with the roles of OpenSlide and the OpenDiamond platform highlighted. For each result, associated metadata such as anonymized patient information and diagnosis are returned.

Open in a separate window

Figure 2

PathFind screenshot with search results

Open in a separate window

Figure 3

How OpenSlide is used by PathFind

FUTURE WORK

OpenSlide is an evolving tool and we see many areas for improvement and enhancement:

• Supporting new digital slide formats and new variants of existing formats is an ongoing challenge. We would like OpenSlide to ultimately support all major formats, including Hamamatsu NDPI, Olympus VSI, Ventana BIF and DICOM

• Language bindings currently exist for Java and Python. We would like to create open-source bindings to additional programming languages such as C# and image-processing toolkits such as MATLAB. GObject bindings would allow OpenSlide to be transparently used by any programming environment that supports GObject Introspection

• OpenSlide is designed primarily to support high-resolution brightfield images. As a result, the API supports only three 8-bit color channels per image. Multiple focal planes, z-slices, or time points are not supported. In the future, we would like to extend OpenSlide to support arbitrary numbers of fluorescence channels and physical axes

• OpenSlide is designed as infrastructure for third-party applications that wish to support digital slides. While we provide simple viewer programs and tools to exercise the library, including the web-based viewer described earlier, there is no high-quality, open-source viewer application that can easily be installed by end users. Such an application would greatly assist pathologists and researchers wishing to view slide files from Mac OS X or Linux, especially in a multi-vendor environment

• The web-based viewer could be extended to support additional features such as annotations, image rotation and synchronized panning/zooming of multiple images. Some of these features already exist in third-party applications built with OpenSlide. In addition, the creation of an embeddable viewer widget for desktop applications, similar to what OpenSeadragon provides in a web browser, would ease third-party development of such applications.

CONCLUSION

Pathology is one of the last medical specialties to be digitized, largely because of the technical problems posed by the capture, storage, transmission, archiving, viewing and annotation of high-resolution, multi-gigabyte, zoomable images. While future improvements in network bandwidth and storage capacity can alleviate these difficulties, their benefit will be limited by the increasing size and complexity of digital slides due to improvements in the resolution, multi-spectral capabilities and other aspects of digital scanner technology. The growth of mobile computing in health-care delivery will exacerbate these challenges because of the need to access digital slides over bandwidth-limited wireless networks from mobile devices with limited screen size. Today's vendor-specific and fragmented approach to digital pathology software offers few opportunities for the latest computer science innovations to be brought to bear on these difficult technical challenges. OpenSlide enables such innovations to be contributed by the best software talent. The merits of specific innovations and their implementations can be evaluated, critiqued and evolved by the entire digital pathology community. We look to a future where this process leads to a vibrant hardware-software ecosystem for digital pathology that fosters interoperability and sharing while preserving ample opportunity for proprietary innovations in hardware and application software.

ACKNOWLEDGMENTS

Footnotes

REFERENCES