Abstract

1 Introduction

With the rapid and significant advances in large-throughput scanning technologies, digital pathology whole slide imaging has recently become an emerging tool not only promising for disease diagnosis and treatment evaluation, but also for high-throughput quantitative information extraction for a wide scope of diseases, ranging from brain (Han et al., 2011; Kong et al., 2011), breast (Petushi et al., 2006), lung (Xing and Yang 2013), colorectal, neuroblastoma (Kong et al., 2009), lymphoma (Cooper et al., 2009), to prostate cancer (Jafari-Khouzani and Soltanian-Zadeh, 2003; Tabesh et al., 2007). Importantly, this new field presents salient merits that help researchers and clinicians better understand the underlying biological mechanisms of pathological evolutions and progressions of distinct diseases (Jara-Lazaro et al., 2010; Kong et al., 2012). Due to the large pathology microcopy imaging scale and overwhelming number of histopathology structures of interest to research, human-based reviewing approaches are not practically feasible for large data sets. In addition, the traditional labor-intensive reviewing process remains a qualitative discipline prone to significant intra-/inter-reader variations and poor reproducibility for diagnosis (Coons et al., 2007). This results in a significant potential utility of whole slide image analysis and the quantitative characterization of pathological structures. A large suite of imaging analytical methods for quantitative analysis of pathology images has, therefore, been developed to meet these needs ( Qi et al., 2012; Foran et al., 2011; Yang et al., 2008; Teodoro et al., 2013). In addition, commercial and open-source products are also available (Aperio, Definiens, ImageJ, among others). However, these tools are bounded by 2D biological structure analysis and limited for processing objects in 3D space.

As micro-anatomic objects in 2D pathology images are only approximate representations of their true 3D structures, information is substantially lost when 3D structures are projected to the 2D image focal plane. In liver disease diagnosis, for instance, structural changes in liver vessels and their spatial relationships in 3D are important (Roberts et al., 2012). 2D profiles of blood vessels in microscopy images of liver tissue are highly dependent on the location and angle of the cutting plane during tissue slide preparation and are substantially different from their 3D shapes in most cases. By contrast, a sequence of microscopy images of liver tissue slices allows for analysis of vessels in 3D space. Therefore, it presents better potential to improve vessel characterization accuracy and to support disease diagnosis and understanding.

Numerous studies on modeling and analysis of vessel structures have been proposed, ranging from vessel segmentation (Lesage et al., 2009), vessel structure tracking (Friman et al., 2009), to the 3D visualization of vascular structures (Kubisch et al., 2012). However, these investigations are limited to either radiology images (such as Computed Tomography, and Magnetic Resonance Imaging), or 3D contrast-enhanced imaging modalities (including Computed Tomography Angiography and Magnetic Resonance Angiography). Compared with radiology image modalities, digital pathology images have much higher resolutions and contain a wealth of information at the cellular level. Therefore, applying 3D vessel analysis methods for radiology images directly to whole slide images is not feasible. Other vessel studies involve the usage of Optical Coherent Tomography (OCT) and confocal microscopy imaging data (Hu et al., 2010; Arribas et al., 2007). Although OCT data modality is non-destructive and accurate for micro-vessels, it loses contextual information of digital pathology. Alternatively, confocal microscopy can also be used to accurately characterize vessels, but limited to small tissue samples.

In this paper, we present a complete 3D vessel analysis framework that can help researchers and clinicians better visualize 3D vessel structures and quantitatively characterize vessel features with 3D microscopy images of liver tissues. With access to a sequence of liver 2D microscopy images, we register all these images to allow for the following analysis in the same spatial coordinate system. Vessels, along with other key micro-anatomic objects of interest in pathology diagnosis, are segmented on each 2D microscopy image. The large number of vessels within livers poses major challenges to recover vessel shapes in 3D by human reviewers. With our computerized analysis, we use Fourier shape descriptors and spatial similarity of vascular structures as visual cues and formulate similarity functions for one-to-one, one-to-two, and one-to-none correspondence cases across adjacent tissue slides. The best multiple vessel associations are found by maximizing the resulting aggregated similarity function with constrained Integer Programming. The resulting mapped vessel cross sections are interpolated to create more intermediate virtual slides for better vessel characterization. Finally, each associated vessel structure is reconstructed and volumetrically rendered in 3D space. We apply our method to a liver microscopy image dataset with sequential tissue slides and reconstruct primary vessels for 3D visualization. By comparing machine-identified vessels with human annotations, we observe a satisfactory concordance suggested by both region-based and distance-based measures. Meanwhile, visual inspections on vessel structures by our method and human annotations also present negligible difference. These results suggest the efficacy of our 3D vessel analysis method for whole slide images of liver tissue sections. Notably, our 3D vessel analysis framework is generic and can be readily applied and extended to the analytics of other 3D biological entities with 3D image datasets of diverse diseases.

The remainder of the paper is organized as follows. Data and Methods used for this study are described in Section 2. We then present and validate the 3D vessel analysis results in Section 3. In Section 4, we discuss how to apply this software to future integrative analysis linked with clinical and molecular endpoints and close the paper with conclusions.

2 Materials and methods

Our workflow for 3D vessel structure analysis consists of multiple components: image registration, vessel segmentation, vessel cross-section association, intermediate slide interpolation, and volumetric rendering. We present the overall workflow in Figure 1. Detailed discussions on these modules and the dataset involved in this study are given as follows.

Open in a separate window

Figure 1

The overall workflow of 3D vessel analysis with whole slide imaging data of liver tissue sections.

2.3 3D vessel association

After blood vessels are identified by our analysis, we find some are recognized as numerous disconnected small parts. To group these disconnected pieces back to their intact vessel cross-sections, we perform intra-frame vessel object grouping, followed by vessel object filtering. Grouping is achieved by an image dilation operation (Vincent, 1993) that grows small vessel parts belonging to the same vessel object and that forces them to be cohesive to a single vessel object. Among all grouped vessel objects, we proceed with selecting the top-K candidates of primary vessels for further analysis based on their sizes, as inclusion of unduly small vessels for analysis not only significantly increases the computational complexity for object association, but also reduces the vessel association accuracy.

The selected top-K vessel objects in each frame are characterized based on their shapes and spatial relationships. We define the problem of frame-by-frame object association in a constrained Integer Programming framework (Hillier and Lieberman, 2001). For our data, three distinct vessel association cases are considered: one-to-one (growing), one-to-two (bifurcation) and one-to-none (disappearing). For each case, the similarity function S is defined as follows:

• one-to-one: the main vessel still continues to extend to the next frame;

  S(vit,vjt+1)=λ1g(vit,vjt+1)+λ2d(vit,vjt+1)

  (1)

• one-to-two: bifurcation occurs and the vessel grows into two small branches;

  S(vit,vj1t+1,vj2t+1)=λ1g(vit,vj1t+1∪vj2t+1)+λ2d(vit,vj1t+1∪vj2t+1)

  (2)

• one-to-none: the vessel stops growing;

  S(vit,v∅t+1)=d(vit,ΩI)

  (3)

where function g(·) compares the similarity of vessel appearance by comparing the Fourier shape descriptors (Zahn et al., 1972) derived from vessel boundaries; d(·) computes the spatial similarity of two vessel objects with their centroids; vit is the i-th vessel object in frame t; Ω_I is the boundary of image I; λ₁ and λ₂ are two constant weights to control the association smoothness subject to λ₁ + λ₂ = 1.

Identification of vessel association is a process of establishing mappings between vessel objects in adjacent image frames based on given criteria and constraints. Assuming we have n_t and n_t+1 vessel objects in frame t and t + 1 and h possible associations between the two frames, we deem such association identification process identical to a multi-object tracking problem (Jiang et al., 2007). Therefore, the optimal associations among vessel objects can be achieved by solving the constrained Integer Programming (Hillier and Lieberman, 2001), based on the pre-defined similarity function:

ArgmaxxsTxsubject to(HTx)i≤fori=1,⋯,nt+nt+1xj∈{0,1}forj=1,⋯,h

(4)

where s is an h × 1 vector with each entry representing the similarity of one vessel object association; H is an h × (n_t + n_t+1) binary matrix with H_ij set to 1 if and only if the i-th object association involves the j-th element in a set concatenated by n_t vessel objects from frame t and n_t+1 vessel objects from frame t+1; (H^Tx)_i denotes the i-th element of (H^Tx) and the constraint (H^Tx)_i ≤ 1 guarantees that each vessel object in a given frame (i.e. t or t + 1) can be selected at most once in the result; the optimal solution x is an h × 1 binary vector where x_i = 1 indicates the selection of the i-th association in the optimal solution. The algorithmic description of the above constrained Integer Programming with simplex solver is presented in Algorithm 1.

Algorithm 1

Constrained Integer Programming with simplex solver for Vessel Object Association (VOA): (fI∗,x∗)=VOA(x,j,fI) (Hillier and Lieberman, 2001)

Require: (HTx)i ≤ 1 for i = 1,···, nt + nt+1 and xj {0,1} for j = 1,···,h
Ensure: ArgmaxxsTx
1:	Suppose fI and fr are current objective values obtained by solving the Integer Programming(IP) problem and LP-relaxation problem, respectively; fI∗ is the optimal objective value and x* is the optimal integer solution
2:	Initialization: x ← [0, · · · , 0]h×1, j ← 1, and fI ← ∞
3:	Output: (fI∗,x∗)=VOA(x,j,fI)
4:	function VOA(x, j, fI) The definition of function VOA(x, j, fI)
5:	if j > h then
6:	return (fI, x)
7:	end if
8:	Perform LP-relaxation by releasing xj {0,1} to a weaker constraint 0 ≤ j ≤ 1
9:	Solve the LP-relaxation problem: (fr, j) = LP_Relax(x1, · · · , xj–1, j, xj+1, · · · ,xh)
10:	if fr > fI and j is NOT an integer then
11:	xj ← 0; (fI0,x0)=VOA(x,j+1,fI)
12:	xj ← 1; (fI1,x1)=VOA(x,j+1,max(fI0,fI))
13:	if fI0≥fI1 then
14:	return (fI0,x0)
15:	else
16:	return (fI1,x1)
17:	end if
18:	else if fr > fI and j is an Integer but j ≠ xj then
19:	fI ← fr; xj ← j
20:	(fI,x) = VOA(x, j + 1, fI)
21:	else if fr < fI then
22:	(fI, x) = VOA(x, j + 1, fI)
23:	end if
24:	end function

Open in a separate window

Solving the constrained Integer Programming problem above provides the optimal associations for vessel objects in adjacent frame pairs. Therefore, by tracking vessel cross-sections through all adjacent frames with the identified associations, we can recover the profile of a vessel structure along the z-axis with shape descriptors and spatial similarity. In Figure 2, vessel in yellow bifurcates, corresponding to the one-to-two association case. Other vessels (green, red, and magenta) represent one-to-one tracking cases.

Open in a separate window

Figure 2

Vessel associations of four chains of color-coded vessels in frame t₁, t₂,..., t_n.

3 Results

We apply the proposed processing framework to a whole slide image dataset consisting of 12 sequential liver slides and extensively evaluate our approach implemented with C and Matlab. In this section, we present results from different modules in our framework and demonstrate method performance measured by both quantitative and qualitative validation analysis.

3.1 Results of individual modules

The first processing module in the analysis pipeline is image registration that helps register to the first image all subsequent ones in the sequence. As this step provides the foundation to reconstruct 3D objects in a meaningful way with their 2D cross-sections from different coordinate systems, it has a profound impact on the following analysis. Our two-step registration method takes into account both global rigid and local non-rigid deformation with phase correlation and B-Spline interpolation by 64 control points. With our image resolution, 64 control points are sufficient to accurately recover non-rigid deformation at different regions in an image. Additional control points would only marginally improve registration accuracy. We present a typical registration result in Figure 3 where frame (n = 10) is registered to the reference image (n = 0). After registration, the registered image in Figure 3(right) is rotated in counterclockwise direction slightly, especially the upper and lower right corners. Nevertheless, not all parts of the registered image are rotated and shifted uniformly because of the non-rigid registration effect.

Open in a separate window

Figure 3

Sequential liver microscopy image registration. (Left): a microscopy image of a liver slice (n = 10) before registration; (Right): same image after registration (slightly rotated counterclockwise).

In liver microscopy images, histopathologic objects of interest to this study at the segmentation step include vessels, nuclei, fat, lumens, and bile ducts. Each type of object is identified and represented with distinct colors. A representative segmentation result is demonstrated in Figure 4, where brown, red, dark blue, light purple, and light yellow colors are used to represent vessels, bile ducts, nuclei, lumens, and fat in liver tissues, respectively. Since we use vessel shape for association analysis in this study, we obtain the vessel boundaries from their masks and superimpose green vessel boundaries on the original image in Figure 4.

Open in a separate window

Figure 4

Liver microscopy image segmentation. (Left): a registered microscopy image of a liver slice (n = 10); (Right): a close-up view of segmentation result within the red box region overlaid on the left image. Note that brown, red, dark blue, light purple, and light yellow colors are used to represent vessels, bile ducts, nuclei, lumens, and fat in liver tissues; Additionally, vessel boundaries are overlaid in green.

In the proposed method, a constrained Integer Programming framework is employed to identify the optimal vessel object association between adjacent frames. The optimal solution is achieved by iteratively solving a series of Linear Programming (LP) relaxation problems. The parameters for object association used in this study are K = 25, λ₁ = 0.68 in Equation (1), and λ₁ = 0.35 in Equation (2). We carefully choose these parameter values after intensive experiments. The value for λ₁ in Equation (1) is intentionally selected to be greater than that in Equation (2), in the sense that shape descriptors are more reliable and discriminative in the one-to-one than one-to-two case. When vessel bifurcates, shapes of the resulting vessel components are substantially deformed from their source. As a result, we assign more weight on spatial similarity for one-to-two case.

We plot objective similarity value for each adjacent frame pair as a function of iteration number in Figure 5, where the objective similarity function defined in Equation (4) increases as iteration number increases. Each curve in Figure 5 represents similarity function change of one frame pair, as indicated by a unique color. The optimal solution is achieved when the objective function reaches its maximum. In Figure 5, we also show time cost for finding the optimal vessel object association for each frame pair with the same color coding scheme of similarity function curves. Note that all time costs are less than 0.7 seconds, with the majority around 0.32 seconds. This suggests that it is efficient to use the constrained Integer Programming framework to establish vessel object association.

Open in a separate window

Figure 5

Similarity value as a function of iteration number, with an inset presenting time cost of vessel object association for each image pair in the sequence. Similarity function curves and time cost bars are coded with the same color scheme.

Interpolation used in our framework enables us to recover 3D vessel structures that exhibit a smooth transition process between two original frames. Without this step, the resulting reconstructed vessels present abrupt changes and step artifacts due to insufficient inter-frame resolution. We present a typical interpolation result in Figure 6, where the leftmost and the rightmost frames are original slices with two identified vessel objects in each. Six intermediate frames are evenly sampled from 50 interpolated frames.

Open in a separate window

Figure 6

Interpolation of intermediate frames. The leftmost and rightmost frames are original, while the six ones in the middle are interpolated.

Note that the interpolation error is bounded as follows (Hazewinkel, 2001):

error(k)≈f(k)−L(k)=f″(ξi)2(k−k1)(k−k2)≤∣l28f″(ξ∗)2∣ξ∗=k1+k22

(6)

where ξ_i [k₁, k₂], i = 1,2,..., P, and l = k₂ – k₁. As a result, the interpolation error decreases as the interval l decreases and the number of sampled data points increases. This suggests that we can achieve improved accuracy by using more sampled points. In this study, the length of interval [k₁, k₂] is normalized to l = 1 and the number of ξ_i is set to P = 50. With these parameter values, we observe continuously changed vessel cross sections as illustrated by two interpolated vessel chains in Figure 6.

As a last step in our pipeline, we render the matched and interpolated vessel objects in 3D space. After interpolation, each object is labeled with a unique integer. Coordinates of vertices on isosurfaces with same label values are recorded and the associations of these vertices for triangular isosurfaces are found. Then, tetrahedral elements are generated to fill up volumes contained by these isosurfaces and 3D mesh is generated with adaptive resolution. The 3D view of six primary vessel objects is presented in Figure 7(a).

Open in a separate window

Figure 7

3D visualization result of (a) machine-identified and (b) human-annotated vessels.

4 Discussion and conclusion

As commercial whole slide imaging scanners are now capable of producing high-magnification, high-resolution images from whole slide tissues in a prompt manner, it increasingly becomes feasible even for a medium-scale study to routinely generate sequential whole slide images of tissue sections. The resulting high-resolution microscopy image data offers a rich source of tissue phenotypic information on clinical relevant histological features and spatial relationships among histopathology hallmarks in specimen architectures. However, 3D microscopy image data is still an underutilized data modality in biomedical research because of the following reasons:

• It is not technically straightforward to develop effective and efficient methods to extract useful information from large-scale whole slide image dataset;

• There is a lack of highly configurable image analysis software to meet diverse research demands from individual studies.

• Paucity of generic methods appropriate for 3D pathology structure investigations contrasts with 2D analysis tools in relative abundance.

As 2D microscopy images can only present histopathological structures at discrete planes, we aim to provide efficient and generic 3D image analysis tools to facilitate biomedical investigations where 3D tissue examinations at microscopic resolution are important. Our vessel structure analysis framework is one such effort to enrich microscopy image analysis tools applicable to a large spectrum of 3D pathology phenotypic hallmarks in numerous diseases. To achieve a reasonable efficiency and accuracy, our current method is focused on primary vessel structure modeling and analysis. As a result, inaccurate segmentation results of small vessels do not affect following modules. Nevertheless, the efficacy of our method is affected by such factors as unduly large inter-slice resolution and tissue slicing orientation during tissue slide preparation. In our future work, we will improve individual analysis modules to reduce errors propagated in our working pipeline. We will also develop methods to accommodate small vessels. Additionally, we will validate our method with more liver tissue sample datasets and extract informative 3D imaging features to help researchers and clinicians better analyze 3D biological structures.

As a follow-up research, we will characterize vessels and other tube-shaped structures with the following generic features. 1) Length: the length of the 3D vessel can be determined by the medial axis with the hamonic skeletonization technique (Yang et al., 2005); 2) Thickness of walls: the vessel wall thickness can be used to measure blood flow volume and to indicate the degree of vascular integrity closely related to oncogenesis; 3) 3D vessel bifurcations: the number of bifurcations along the artery could be a clinically relevant parameter that determines the complexity of the vessel structures; 4) Angles at bifurcations: the angles at bifurcations of the vessels could affect volume of blood flow; 5) Local cross-sectional area: cross-sectional areas at different locations along a vessel can characterize vessel structures. With these fundamental features measured, we will further describe high-level 3D shape features. For vessels, we will calculate the inter-branch distance, characterize the branching pattern, and generate a histogram of branch lengths and thicknesses for the entire 3D tissue volume. With these 3D structure features and their cross-relationships quantitatively characterized, better clinically relevant patient stratification protocols and molecularly correlated phenotypic information could be expected. Our end goal is to combine our methods with such techniques as immunohistochemistry, Fluorescent In Situ Hybridization, or mRNA gene expression data and potentially provide complementary information to enhance investigations on the anatomical features and micro-architecture of histologic tissues, mechanisms of tumor progression, spatial distributions of signatured molecular biomarkers, expression of master regulators in cancer pathways in relation to 3D vessel structural features.

In summary, we present in this paper a new 3D vessel structure analysis method consisting of registering, segmenting, matching, interpolating, and visualizing blood vessels in liver tissues with large-scale microscopy images. We vision this cohort of 3D analysis tool development effort as a new avenue to facilitate pathology review and biomedical research, and a better way to engage researchers and clinicians in using rich molecular and clinical related information from histopathological imaging data.

Acknowledgement

Contributor Information

Yanhui Liang, Department of Biomedical Informatics, Emory University, Atlanta, GA, USA [email protected].

Fusheng Wang, Department of Biomedical Informatics, Emory University, Atlanta, GA, USA [email protected].

Darren Treanor, Department of Pathology Leeds Teaching Hospitals NHS Trust Leeds Institute of Cancer and Pathology The University of Leeds, Leeds LS9 7TF, United Kingdom ten.shn@ronaertnerrad.

Derek Magee, School of Computing, The University of Leeds, Leeds, LS2 9JT, United Kingdom [email protected].

Nick Roberts, Leeds Institute of Cancer and Pathology The University of Leeds, Leeds LS9 7TF, United Kingdom [email protected].

George Teodoro, Department of Computer Science, University of Brasília, Brasília, DF, Brazil rb.bnu@orodoet..

Yangyang Zhu, Department of Mathematics and Computer Science, Emory University, Atlanta, GA, USA [email protected].

Jun Kong, Department of Biomedical Informatics, Emory University, Atlanta, GA, USA.

References