Vol. 28 no.

22 2012, pages 3009–3011

BIOINFORMATICS APPLICATIONS NOTE doi:10.1093/bioinformatics/bts543

Bioimage informatics Advance Access publication September 8, 2012

ImgLib2—generic image processing in Java

Tobias Pietzsch1,y, Stephan Preibisch1,2,y, Pavel Tomančák1 and Stephan Saalfeld1,*,y
1
Max Planck Institute of Molecular Cell Biology and Genetics, 01307 Dresden, Germany and 2Janelia Farm Research
Campus, Howard Hughes Medical Institute, Ashburn, VA 20147, USA
Associate Editor: Jonathan Wren

Downloaded from https://academic.oup.com/bioinformatics/article-abstract/28/22/3009/240540 by guest on 18 April 2020

ABSTRACT representations that will seamlessly work with existing
Summary: ImgLib2 is an open-source Java library for n-dimensional algorithms, and vice versa. ImgLib2 shares basic concepts with
data representation and manipulation with focus on image processing. the Cþþ frameworks ITK (Yoo et al., 2002) and Vigra (Köthe,
It aims at minimizing code duplication by cleanly separating pixel- 2000) for n-dimensional, generic image processing. It is the first
algebra, data access and data representation in memory. Algorithms framework that introduces generic programming to the Java
can be implemented for classes of pixel types and generic access image processing community (Preibisch et al., 2010). We chose
patterns by which they become independent of the specific dimen- Java for its simplicity and wide acceptance among biological
sionality, pixel type and data representation. ImgLib2 illustrates that an researchers due to the popular image processing toolbox
elegant high-level programming interface can be achieved without ImageJ (Rasband, 2012).
sacrificing performance. It provides efficient implementations of
common data types, storage layouts and algorithms. It is the data
model underlying ImageJ2, the KNIME Image Processing toolbox
and an increasing number of Fiji-Plugins. 2 ARCHITECTURE
Availability: ImgLib2 is licensed under BSD. Documentation and The ImgLib2 core design is based on three main concepts:
source code are available at http://imglib2.net and in a public reposi- Accessibles (i.e. images), Accessors and Types. We define an
tory at https://github.com/imagej/imglib. image as any mapping from a subset of n-dimensional
Supplementary Information: Supplementary data are available at Euclidean coordinate space to a generic pixel value type. Image
Bioinformatics Online. properties are expressed by Accessible interfaces: coordinates can
Contact: [email protected] be either integer or real-valued, the coordinate domain can be
Received on July 6, 2012; revised on August 21, 2012; accepted on
either bounded or infinite, the image may support random access
July 28, 2012
at arbitrary coordinates and/or iteration of all samples. Consider
a conventional pixel image. It comprises samples of a specific
value type in bounded n-dimensional space, arranged on an in-
1 INTRODUCTION teger grid and is both random-accessible (at arbitrary integer
coordinates) and iterable. Importantly, ImgLib2 supports con-
Many algorithmic concepts from computer vision and image
cepts beyond the conventional pixel image, e.g. infinite, proced-
processing are applicable to the analysis of biological image
urally generated images or continuous images interpolated from
data. However, re-using existing code is often difficult because
sparsely sampled data.
it is implemented for a specific data type, limited image size or
Access to sample (pixel) values and coordinates is provided by
fixed number of dimensions, e.g. small 2d grayscale images.
Accessor interfaces. These exist in variants for integer and real
Biological imaging techniques generate images of varying dimen-
coordinates, as well as iterating and random access. For iterating
sionality and a multitude of sample types (e.g. wavelength, fre-
accessors, iteration order is subject to implementation, specia-
quency spectra, diffusion tensors) with varying precision.
lized for each memory layout to minimize access time.
Improvements in imaging speed and resolution result in gigantic
Accessors provide value access via Types. ImgLib2 has a hier-
datasets that require well-designed strategies for data handling
archy of Type interfaces that describe algebraic properties of
(e.g. tiled or compressed storage, streaming access). Writing code
families of concrete types. Examples are Comparable types or
that is re-usable across many combinations of dimensionality,
sample type and storage strategy is challenging and requires an NumericTypes that support basic arithmetic operations
appropriate abstraction layer. (þ,,*,/).
We present ImgLib2, an open-source image processing frame- Access patterns and type properties allow fine-grained specifi-
work that achieves code re-usability through a generic interface cation of algorithmic requirements. An algorithm that is built
architecture that abstracts from dimensionality, sample type and using appropriate interfaces applies to any specific image imple-
storage strategy. It is highly extensible, providing developers with menting those interfaces. Re-usability of algorithms is maximized
great flexibility in adding new sample types and image by specifying them for the minimal set of required properties.
Consider, for example, summing all pixel values in an image.
This can be implemented in two lines of Java code for, e.g. a
*To whom correspondence should be addressed.
y
The authors wish it to be known that, in their opinion, the first, second gray-level image stored as a byte[] array. However, it has to be
and the last author should be regarded as joint First Authors. re-implemented, over and over, for every combination of data

ß The Author(s) 2012. Published by Oxford University Press.

This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits
unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
T.Pietzsch et al.

Downloaded from https://academic.oup.com/bioinformatics/article-abstract/28/22/3009/240540 by guest on 18 April 2020

Fig. 1. Visualizes exemplarily the capabilities of ImgLib2. (a) shows an image, virtually extended by a mirroring strategy, arrows mark the original
image boundaries. Four algorithms were applied to sub-image views: (from left to right) anisotropic diffusion, maximally stable extremal regions, Sobel
filtering, Gaussian convolution. (b) shows an extrapolation of sparse data where 2,000 points were randomly sampled from the larger area indicated in
(a). (c) shows an interpolated and affine transformed view of the smaller tilted area indicated in (a)

type, dimensionality and storage layout. Using ImgLib2, this can implementations using image factories. This allows performance
be written generically as tuning for specific datasets without any modification to the al-
gorithm implementation.
for (T value : image) sum.add(value);
We compared the performance of ImgLib2 generic code and
where we specify that image implements IterablehTi and that T special purpose (fixed dimensionality and value type) implemen-
extends NumericTypehTi. The same code handles all pixel images tations for Java primitive type arrays and ImageJ
with appropriate value type, virtual views into such images, spar- (Supplementary Table S1). For simple per-pixel operations, gen-
sely sampled datasets, procedural images, etc. eric ImgLib2 code achieves 100% of the performance of special
In Java, this level of generality requires pixels to be objects. purpose implementations using native arrays. For a more com-
Storing simple pixel values (e.g. bytes) as individual objects, how- plex operation involving an inner loop over the unknown
ever, comes with significant memory overhead. Conversely, number of dimensions, the ImgLib2 code was on average
creating new objects per pixel access introduces significant run- 1.6 slower than native arrays (1.5 slower than ImageJ). We
time overhead and triggers frequent garbage collection. Both consider this a reasonable abstraction penalty as the ImgLib2
approaches do not scale well with large images. To address this code supports any dimensionality, image and value type. In con-
issue, ImgLib2 uses proxy types to access pixel data that can be trast, native arrays and ImageJ images require specialized imple-
mapped into Java primitive type arrays (byte[], float[], etc.). In mentations for each supported dimensionality and value type.
this way, an accessor can re-use one proxy instance for all pixel For the cases tested in our benchmark, this amounts to an
accesses. In the above example, a proxy of type T is instantiated order of magnitude increase in lines of code. Even so, only
once and then re-used in every iteration, changing only internal ImgLib2 is able to handle all test cases due to dimensionality
state. This virtualization pattern has no performance overhead and image size limits of both ImageJ and primitive type arrays.
compared with direct array access, thanks to the optimizations ImgLib2 permits virtualization of sample access. We use
performed by Java’s just-in-time (JIT) compiler. this for accessors that perform on-the-fly coordinate and value
transformations without copying the underlying data. The
Views framework creates accessibles that provide coordinate-
3 IMPLEMENTATION transforming accessors. Integer coordinate transformations in-
ImgLib2 incorporates common value types (BitType, Unsigned- clude slicing, windowing, axes permutations and 90 rotations.
ByteType, ARGBType, ComplexFloatType, etc.) efficiently im- Consecutive transformations are reduced and simplified, yielding
plemented as proxies that map into Java primitive type arrays. accessors with optimal performance. For real coordinates we
Various implementations for pixel data in a discrete n-dimen- support n-dimensional affine transformations. Interpolating
sional grid (conventional pixel images) are provided: ListImg and rasterizing views convert between discrete and continuous
stores pixels as individual object instances and thus supports ar- coordinate spaces. Finally, some algorithms (e.g. convolution)
bitrary value types, but does not scale to large numbers of pixels. require access to pixels outside of the image which are usually
ArrayImg maps proxy types into a single primitive type array, created by padding or mirroring. This is achieved by extending
providing optimal performance and memory efficiency. views, whose accessors generate outside values on demand. Note,
However, Java arrays are limited to a size of 231 (e.g. a square that views may be cascaded and act both as input and output for
2d image with maximally 46,340 px side length) which is easily pixel processing. Similarly, the Converters framework realizes
exceeded in today’s microscopy recordings. CellImg splits the transparent transformation of values. For instance, a
coordinate domain into an n-dimensional grid of cells, each map- FloatType image can be addressed as ByteType using an arbi-
ping into one primitive type array. This enables significantly trary mapping function.
larger images (262 px) at slightly reduced performance. In ImgLib2 uses Bio-Formats (Linkert et al., 2010) to read and
generic code we can transparently switch between image write a large number of image file formats. Interoperability with

3010
 ImgLib2

ImageJ is provided by non-copying wrappers of ImageJ data microscopy. Consequently, ImgLib2 is already being used by
structures as ImgLib2 accessibles and vice versa. This makes it several high-profile projects of the Java bioimaging community
straightforward to integrate ImgLib2 into existing ImageJ-based (Berthold et al., 2009; Rueden et al., 2010; Schindelin et al.,
processing pipelines. Light-weight wrappers for other data 2012). It is easily integrated into other projects providing an
models are easy to implement and currently exist for Java ideal basis for sharing interoperable, generic algorithms.
AWT BufferedImage, Java primitive type arrays and remotely
stored image stacks (Saalfeld et al., 2009). ImgLib2 comprises
a growing collection of generic algorithms that are fundamental ACKNOWLEDGEMENTS
building blocks for n-dimensional image analysis: the Fast ImgLib2 and ImgLib have been supported by Fiji Hackathons
Fourier Transform can be used for tomography reconstruction, at the MPI-CBG, EMBL and LOCI. We gratefully thank all
pattern detection or (de-)convolution; sub-pixel edge detection

Downloaded from https://academic.oup.com/bioinformatics/article-abstract/28/22/3009/240540 by guest on 18 April 2020

developers and users that contributed, notably C. Rueden, B.
(Devernay, 1995), component trees (Nistér and Stewénius, DeZonia, C. Dietz, M. Horn, L. Kamentsky, A. Cardona, J.
2008) and automatically detected interest-points (e.g. DoG and Schindelin, G. Harris, L. Lindsey, M. Longair, J.-Y. Tinevez,
MSER; Lindeberg, 1998; Matas et al., 2002) are important tools N. Perry, J. Funke and S. Jaensch.
for image segmentation, image registration and tracking; k-d
trees enable fast n-dimensional search. Funding: T.P., P.T. and S.S. were funded by MPI-CBG, S.P. was
Sparsely and irregularly sampled data are supported, stored funded by MPI-CBG and HHMI.
either as a sample list or in a k-d tree. Both implement interfaces
for nearest-neighbor search, allowing extrapolation of sparse Conflict of Interest: none declared.
data into a continuous image. Sparsely sampled data, interpol-
ation, extension, coordinate transformation and several algo-
rithms are illustrated in Figure 1. REFERENCES
Berthold,M.R. et al. (2009) Knime—the konstanz information miner: version 2.0
and beyond. SIGKDD Explor. Newsl., 11, 26–31.
4 DISCUSSION Devernay,F. (1995) A non-maxima suppression method for edge detection with
sub-pixel accuracy. Technical Report RR-2724, INRIA.
ImgLib2 is an open-source image processing framework that Köthe,U. (2000) STL-style generic programming with images. Cþþ Report Mag.,
increases code re-usability by promoting generic implementa- 12, 24–30.
tions. It provides an abstraction layer that focuses on flexible Lindeberg,T. (1998) Feature detection with automatic scale selection. Int. J. Comp.
and efficient image storage and access. The core paradigm is a Vision, 30, 79–116.
Linkert,M. et al. (2010) Metadata matters: access to image data in the real world. J.
clean separation of pixel algebra (how sample values are manipu- Cell Biol., 189, 777–782.
lated), data access (how sample coordinates are traversed) and Matas,J., David,M. and Paul L.,R. (2002) Robust wide baseline stereo from max-
data representation (how the samples are stored, laid out in imally stable extremal regions. In: BMVC. Vol. 1, pp. 384–393.
memory or paged to disc). ImgLib2 relies on virtual access to Nistér,D., Stewénius,H., David,F., Philip,T. and Andrew,Z. (2008) Linear time
maximally stable extremal regions. In: ECCV. pp. 183–196.
both sample values and coordinates, facilitating parallelizability
Preibisch,S., Andreas,J. and Christian,M. (2010) Into imglib—generic image pro-
and extensibility. cessing in java. In: Proceedings of the ImageJ User and Developer Conference,
ImgLib2 aims to connect software projects through an inter- Luxembourg.
face design that is easily adapted to existing data structures. Rasband,W.S. ImageJ, U. S. National Institutes of Health, Bethesda, Maryland,
ImgLib2 is the first image processing library available for Java USA, http://rsbweb.nih.gov/ij/, 1997–2012.
Rueden,C., Andreas,J. and Christian,M. (2010) Imagejdev: next generation imagej.
that combines a flexible high-level programming interface with
In: Proceedings of the ImageJ User and Developer Conference,
optimal performance. It enables developers of bioimage analysis Mondorf-les-Bains, Luxembourg.
software to focus on the design of complex algorithms instead of Saalfeld,S. et al. (2009) CATMAID: collaborative annotation toolkit for massive
data management. Conversely, software engineers can develop amounts of image data. Bioinformatics, 25, 1984–1986.
efficient infrastructure without interfering with algorithm design. Schindelin,J. et al. (2012) Fiji: an open-source platform for biological-image ana-
lysis. Nat. Methods, 9, 676–682.
This becomes particularly interesting in the emerging field of Yoo,T.S. and Westwood,J. (2002) Engineering and algorithm design for an image
bioimage informatics that is coping with the enormous amount processing API: a technical report on ITK—the insight toolkit. In: MMVR,
of n-dimensional image data generated by recent developments in pp. 586–592.

3011