Image Compressor 2014 UserGuide
Image Compressor 2014 UserGuide
User Guide
14.00.0001
May 2014
Copyright
Copyright 2014 Intergraph Corporation. All Rights Reserved.
Including software, file formats, and audiovisual displays; may be used pursuant to applicable software license agreement; contains
confidential and proprietary information of Intergraph and/or third parties which is protected by copyright law, trade secret law, and
international treaty, and may not be provided or otherwise made available without proper authorization from Intergraph Corporation.
Terms of Use
Use of this software product is subject to the End User License Agreement ("EULA") delivered with this software product unless the
licensee has a valid signed license for this software product with Intergraph Corporation. If the licensee has a valid signed license for
this software product with Intergraph Corporation, the valid signed license shall take precedence and govern the use of this software
product. Subject to the terms contained within the applicable license agreement, Intergraph Corporation gives licensee permission to
print a reasonable number of copies of the documentation as defined in the applicable license agreement and delivered with the
software product for licensee's internal, non-commercial use. The documentation may not be printed for resale or redistribution.
Trademarks
Intergraph, the Intergraph logo, and GeoMedia are registered trademarks of Intergraph Corporation. Microsoft and Windows are
registered trademarks of Microsoft Corporation. Bing is a trademark of Microsoft Corporation. Google Maps is a trademark of Google
Incorporated. Pictometry Intelligent Images is a registered trademark of Pictometry International Corporation. ERDAS, ERDAS
IMAGINE, Stereo Analyst, IMAGINE Essentials, IMAGINE Advantage, IMAGINE, Professional, IMAGINE VirtualGIS, Mapcomposer,
Viewfinder, Imagizer, LPS, and ERDAS APOLLO are registered trademarks and exclusive property of Intergraph Corporation.
Other brands and product names are trademarks of their respective owners.
Contents
Whats New .................................................................................................................. 4
Overview ...................................................................................................................... 5
Architecture............................................................................................................. 6
Supported Environments ........................................................................................... 7
Installation Guide ........................................................................................................ 8
Windows ................................................................................................................. 8
Uninstallation .................................................................................................. 11
Linux ..................................................................................................................... 12
Uninstallation .................................................................................................. 15
Licensing.................................................................................................................... 16
Overview ............................................................................................................... 16
Installation & Configuration ................................................................................... 17
Compressor Usage ................................................................................................... 18
Wizard mode......................................................................................................... 18
Command-line mode ............................................................................................ 23
Input ............................................................................................................... 24
Output............................................................................................................. 24
Options ........................................................................................................... 25
Region Options ........................................................................................ 31
Advanced Options ................................................................................... 33
Reporting example ......................................................................................... 35
Usage examples ............................................................................................ 36
XML Project File ................................................................................................... 37
Example mosaic.xml ...................................................................................... 37
Supported input formats ....................................................................................... 39
Null block analysis ................................................................................................ 41
Comparison .................................................................................................... 42
FAQ ................................................................................... Error! Bookmark not defined.
Support....................................................................................................................... 50
Whats New
Mosaicking now supports multiband input, multiband output. Previously only RGB/RGBA
output was supported in 14.0
Mosaicking now supports greater than 8bit output for ECW v3 and JPEG2000 files
Parsing of mosaic projects with thousands of datasets now 10 times faster
Overriding of SRS information now possible in Mosaic XML as a new output option
Added support for MBTile, NITF and Terrashare RBD file input
12-bit JPEG compressed file types now supported
Multi-part polygon regions now supported (e.g. polygons with holes)
Further optimizations and bug fixes
Overview
ERDAS Image Compressor is a geospatial image compression application designed to simplify the
creation of ECW and JPEG2000 data. The application provides,
The products functional scope is intentionally narrow. The ERDAS Image Compressor does not,
The Image Compressor can be seen therefore as a transcoder only of input data. For other image
processing tasks, ERDAS IMAGINE 2014 remains the recommended tool for the full processing and
compression workflow.
Architecture
The ERDAS Image Compressor can be broken into input readers, compressor logic and output
writers. The input readers ensure wide industry format support by leveraging the GDAL and
ERMapper libraries. For transcoding of existing ECW or JPEG2000 datasets, there is a direct rendering
path where the ECWJP2 SDK is used both as the input and output pathway to ensure the fastest
throughput.
The compressor tier has the thread, memory, region and processing logic to ensure the input data is
passed as quickly as possible from the input readers to the output writer. Its also responsible for the
spatial intersection tests when passing in vector regions to denote null block or opacity band areas.
The application is thread-safe1 and has been deployed on processing workstations up to 32-cores
with good CPU scaling using the new parallel ECW encoder. Peak performance has been recorded at
120MB/sec encoding a 10 gigapixel image and far exceeds the throughput of other third-party
applications that implement the ERDAS ECWJP2 SDK due to optimizations implemented in the Image
Compressor architecture.
The provided user interface is a thin wrapper around the command-line Image Compressor
executable. For advanced users it is expected that most will interface with the command line tool to
give more control integrating with existing workflows.
Note: Compression throughput speeds vary and are heavily dependent on hardware and the input
data format type and internal structure.
Supported Environments
= One of the operating systems is required.
= One of the database server engines is required.
= One of database client engines is required.
= One of the internet browsers is required.
= One of the third party products is required.
= One of the Intergraph products is required.
R = Required
O = Optional
U = User must install
A = Automatically installed by Intergraph setup
ERDAS Image Compressor
Operating Systems
Windows 7, 64-bit
Windows 8 & 8.1, 64-bit
Windows Server 2008 & 2008 R2 64-bit
Windows Server 2012 & 2012 R2 64-bit
Red Hat Enterprise Linux 6.x, 64-bit
CentOS 6.x, 64-bit
Note: Other Linux distributions such as Debian, Ubuntu, Mint, Fedora and OpenSuSE are considered
viable platforms and may require additional installation requirements.
Installation Guide
Windows
1. Download and execute the installation package from the product website
3. Default target location is C:\Intergraph\Image Compressor 2014\" but can be changed. 170
MB is required
6. Start menu shortcuts will be created by default under the following group,
7. The installation will finally unpack and deploy the required files
10
Uninstallation
To remove or uninstall, use the standard Windows Add/Remove Programs process. For
convenience there is also a shortcut within the Start Menu folder.
11
Linux
1. Download and extract the installer giving executable rights to ERDAS-ImageCompressor_2014.bin
2. Execute the bin file,
3. The installation wizard will then open, mirroring the steps found in the Windows version
12
13
14
Uninstallation
To remove or uninstall, execute the UninstallImageCompressor.sh script located in
/usr/local/Intergraph/image-compressor/ and Remove all components when prompted.
15
Licensing
Overview
The ERDAS Image Compressor is designed to complement existing Intergraph Geospatial 2014
products. The tool uses host based licensing to validate license availability in accordance with the
following table,
Product Family
Version
ERDAS APOLLO
Essentials
14.0
Advantage
14.0
Professional
14.0
ERDAS IMAGINE
Professional
14.0
Essentials
14.0
Advantage
14.0
Professional
14.0
Professional
14.0
GeoMedia WebMap
In the event node-locked or concurrent licenses are unavailable for any of the above products and
versions the compressor will fail.
Host based licensing ensures that if any of the above feature codes are already acquired on the same
machine the Image Compressor will validate but not check out an additional license. For example,
the Image Compressor and APOLLO Essentials can coexist under one 2014 license when run on the
same machine at the same time.
The Image Compressor user interface and command line tools will display the licensing status to help
identify which license was acquired. Where multiple concurrent licenses are available from a central
licensing server, it is advised to borrow licenses explicitly to prevent situations where Image
Compressor may acquire a license required by other users.
16
Once a license is acquired Image Compressor allows unlimited compression of input files in terms of,
1.
2.
3.
4.
17
Compressor Usage
Wizard mode
1. The initial page guides users through the 4 step compression process. It also provides license
status information and about dialog for versioning and license information.
Note: If a license was not found, the Next button will be disabled.
2. The I icon launches the About dialog below to review license, build information and other
relevant help links.
4. Regardless of the wizard selected, the next step is to select the input images.
a. Add images allows individual file selection using the default image filter of
common input file types (*.ecw, *.jp2, *.tif, *.alg, *.ers, *.img, *.vrt) or All files
b. Add folder can be used in conjunction with the Recurse directories option to add
thousands of files within a directory.
c. If mosaic wizard was selected, the order in which the datasets are displayed in this
list box represents the display order of the output mosaic where datasets displayed
at the top will be rendered on top of the output image. You may use the
contextual right click menu or drag and drop to reorder the display order.
19
a. The output filename and directory can be defined using standard file locations or
with templates.
b. The options dialogs match to the command line equivalent. See Command Line
Options chapter below for more information.
6. Opacity and null block options is the final configuration step.
20
8. The progress dialog will then execute the jobs in sequential order. Executing jobs in parallel
is not supported by the UI but can be achieved through direct command line access.
9. Press the Start button to begin the compression jobs.
a. During compression the Status bar will progress relative to the current input image.
Additional metrics are also reported,
i. Null Blocks: The number of blocks written to an ECW v3 file tagged as null.
This number will always show 0 unless a region is passed in and null blocks
are enabled.
ii. Blocks: The number of regions that have been compressed and written to
disk. One block contains all compressed data for all bands for one region,
i.e. for three bands there will be three precincts per block.
iii. Precincts: The number of regions that have been processed by the wavelet
filter. Each precinct is per band.
On the completion of each input image, two actions are supported
iv. Log: to review the standard log output including any errors or performance
information
v. View: will open the output file in the registered application handler such as
ERDAS IMAGINE or ER Viewer to review the output result
ERDAS Image Compressor User Guide
21
When an error occurs, the UI will report the error and only allow the Log to be
reviewed.
At any time during batch compression, the Stop button will halt the current
processing job. Pressing Start will then restart the batch project from the next
image in the list
22
Command-line mode
The command-line mode offers fine-grain control over the compression process. Where possible
defaults are set to ensure that in many cases only the <input> and <output> file is required however
careful consideration of all options is recommended. When the user interface is used, the output
compression log includes the command line options provided.
Note: The available options are identical for both Windows and Linux versions of Image Compressor.
23
Input
For supported input format types see Supported Input Formats chapter. To compress any of these
formats pass in the file location and the Image Compressor will attempt to load it via one of the
supported Input Readers. If reading fails the compressor will exit.
Note: File location should be enclosed in quotes if path may contain spaces.
Output
Image Compressor supports writing ECW and JPEG2000 files where the type is defined by the output
file extension. To write JPEG2000 the output file should be specified as C:\output\compressed.jp2 or
for ECW C:\output\compressed.ecw. Based on the file type different compression options will
become available according to the following format capabilities.
Capability
ECW v2
ECW v3
JPEG2000
Line compression
Tile compression
8-bit unsigned
16-bit unsigned
16-bit signed
Visually lossless
Numerically lossless
Null block support
Opacity band support
Data statistics, histogram
RPC storage
Custom metadata
Georeferencing
GDT
GeoTIFF Tags
Greyscale, RGB,
Multiband
Greyscale, RGB,
Multiband
Greyscale, RGB,
Multiband
14 terapixels
14 terapixels
756 gigapixels
Partial. Custom metadata can be written to JP2 UUID boxes however clients will have to explicitly
understand the presence and structure of the contents.
3
24
Options
-method line|tile
The Image Compressor implements two types of compression algorithms.
Line represents a scan-line based reader that reads across each scan-line, compresses and
continues to the next scan-line until the end of file. This approach has been used since ECWs
inception.
Line is inherently single-threaded and cannot scale across multiple CPU cores however it benefits
from lower memory requirements to compress. Line is particularly suited for scan-line structured
input data such as Strip TIFF or file formats that do not perform well with multi-threaded readers.
Note: Line must be used for the compression of JPEG2000 files.
Tile is a new parallel algorithm introduced in the ERDAS ECWJP2 SDK v5 that reads input data in
discrete tiles across multiple reader threads specified by the threads option. Each thread processes
independently in a thread-pool across the width of the dataset and is then repeated down the
image.
25
The new algorithm results in scaling across CPU Cores however it requires more memory to
compress the same input as line. There is a further trade-off with Disk I/O as the concurrent
threads increase load and requires more data to be processed than line, increasing the likelihood of
reaching a disk bottleneck. Its also possible that the input Data Readers have not been optimized for
multi-threaded reading creating additional bottlenecks. See FAQ for formats that are not threadsafe.
Fast I/O is a requirement in order to feed data to the multiple worker threads otherwise CPU
Utilization will be low and performance may be slower than the line algorithm. Where Data I/O is
sufficient, the tile algorithm can be more than 400% faster than line depending on hardware and
input format used.
Both algorithms are suited for different situations. In order to determine the ideal method for your
situation, benchmarks should be performed particularly where compression speed is an important
measure.
Note: Output ECW files are binary identical irrespective of the compression method used.
-listinputformats true
Retrieves list of supported input formats across all data readers. See Supported Formats chapter for
a formatted version. This option is mutually exclusive from all others.
-targetrate 15
The target compression rate is expressed as a ratio, where 15 represent 15:1 or in other words a
94% size reduction. The compressor uses this target to maintain image compression quality which
can result in the Actual Compression Ratio being higher or lower than the provided value. This is
expected and is commonly seen on datasets with nodata areas or other areas with spectral
characteristics that are highly compressible without sacrificing quality. A 15:1 target compressed
26
image always produces comparable image quality to any other 15:1 target image even though actual
rate may vary.
For RGB or Multiband input, 15:1 to 20:1 target compression rates produce visually lossless output
results. For greyscale, 10:1 is recommended. See the Analyzing the ECW Lifecycle Whitepaper for
more information.
Note: To compress numerically lossless JPEG2000, specify a targetrate value of 1. Any other value
will revert to the lossy compression type. ECW does not support numerically lossless compression so
a targetrate of less than 5 is not recommended.
-opacityband 4
The compressor will by default attempt to retain any opacity or alpha band present in the input file.
This option can be used to force the detection of the opacity channel if it cannot be determined
automatically from the input reader. For example, where an input 4-band image is tagged as Bands #
0 to 3, where Band #3 is actually a mislabelled opacity band.
-version 2 | 3
The ECW file format has two versions available. ECW v2 is the legacy format with the widest industry
support and ensures interoperability with all existing ECW software. ECW v3 was introduced in 2012
and adds new capabilities such as null blocks, improved metadata storage and support for uint16 cell
types.
Defaults to ECW v2. This option is ignored for JPEG2000 output.
-tempdir C:\temp\
As part of every compression process the Image Compressor must maintain a certain amount of
intermediate data on disk before reassembling to form the final output format. These intermediate
files are all stored in the tempdir location that defaults to the system TEMP environment location.
For performance reasons its strongly recommended to write temporary data to a different physical
disk than the input or output drives to maintain throughput. Users should allocate the same amount
of storage at tempdir as at the final target destination. For example, if a 10 GB ECW output is
estimated then at least 10 GB is required at the tempdir location.
Note: The Image Compressor manages the files written to this location. On completion of all
compression jobs the temp files will be deleted however if the compression process is interrupted
temporary files may be left on disk. If no compression is currently running it is recommended to
delete any orphaned temp files.
ERDAS Image Compressor User Guide
27
-threads 4
The number of threads used when compressing via the tile compression method. Defaults to the
detected CPU Core count but can be adjusted to determine ideal concurrency level. Like memcache
using all threads may not yield best performance particularly if Disk I/O is inadequate. On large
deployments with 32 cores or more, optimal throughput may be found setting threads to 8 or 16.
Testing is always recommended.
Value is ignored when method is set to line as it will always be one. See method above for more
information.
-memcache 0.25
Allocate a percentage of total System Memory to a memory pool for the compression process. The
image compressor will calculate the required memory for the Write Memory Cache based on the
input image dimensions and compression method specified (see table). Any remaining memory in
the pool will then be assigned to the Read Memory Cache.
On a machine with 32 GB System Memory for example, a memcache of 0.7 can be visualized as
follows,
If the same input image was then compressed on a machine with only 16GB System Memory, the
same memcache value of 0.7 would yield,
Which would result in an error since the fixed Write Memory Cache requirement of 15gb is larger
than the whole memory pool budget (16gb * 0.7 = 11.2 gb). To remedy, a memcache value of 0.95
(15.2 GB) is required which would require careful management to ensure the system does not begin
to swap to pagefile. Adding additional memory would be recommended in this situation or switching
to the line compression method to reduce the memory requirement but potentially sacrificing
throughput performance.
To compress a smaller hypothetical image on the 32 GB machine with the same memcache value of
0.7, the write cache is reduced and the read cache size increases dramatically to take the remainder
of the memory pool.
28
The smaller image will now also be able to be run on the 16gb machine as the Write requirements no
longer exceeds the memcache budget,
The size of the Read Memory Cache is secondary and increasing this value does not always yield an
increase in performance. In other words, if a memcache of 0.25 is sufficient to compress your input
image it is not always advised to specify memcache of 0.9. It is dependent on the type of input
format, how well that dataset driver responds to a higher read memory cache and disk I/O
performance. In some cases performance could drop given additional overheads managing a large
cache verse simply reading data from disk.
It is always recommended to leave a memory buffer to ensure long running compression jobs do not
reach 100% Memory usage. Image Compressor can fluctuate within a degree of tolerance of the
memory budget before purging, so assigning 99% memory is never advised. This buffer size also
depends on other applications that may be running on the machine.
The following table highlights the fixed write memory cache requirements for different input
characteristics.
Bands
Gigapixels
Output
Format
Compression
Method
Write Memory
Cache (MB)
50,000 x 50,000 px
2.5
ECW
Tile
234
2.5
ECW
Line
143
10
ECW
Tile
416
10
ECW
Line
234
50
ECW
Tile
1,872
50
ECW
Line
962
50
ECW
Tile
4,906
50
ECW
Tile
416
50
ECW
Line
234
50
JPEG2000
Line
202
50
ECW
Tile
1,023
250
ECW
Tile
1,873
250
ECW
Line
963
250
JPEG2000
Line
801
250
ECW
Tile
4,909
250
JPEG2000
Line
2,048
1000
ECW
Tile
3,695
100,000 x 100,000 px
500,000 x 100,000 px
100,000 x 500,000 px
500,000 x 500,000 px
1,000,000 x 1,000,000 px
29
Bands
Gigapixels
Output
Format
Compression
Method
Write Memory
Cache (MB)
1000
ECW
Line
1,873
1000
ECW
Tile
9,102
Note,
30
the drop in memory when using line over tile compression method
the drop in memory when the width of the input changes, despite identical gigapixel value
the large increase in memory requirements moving from 3 to 8 bands
the small memory drop using our default JPEG2000 profile over ECW in line mode
although not shown, write memory requirements do not change depending on the
targetrate. A target of 10:1 will require the same memory as 50:1.
Region Options
-opacity 0 | 1 | 2
The opacity value defines the behaviour of the input region defined by datasetregionfile, -shapefile
and wktfile.
When writing ECW v3, a value of 2 is recommended to benefit from an opacity band as well as
reduced storage requirements and possible performance improvements using the null block
capability. Be aware that null blocks introduce additional calculations in compression, so where the
null area represents a small percentage of the total dataset area (Ratio to data), or the region
complexity is large (a high number of vertices) a value of 0 may compress faster. Testing is
recommended to quantify this difference for your situation. The output quality will be identical
between 0 and 2. ECW users will be unaware whether null blocks are present and behave like any
other ECW file.
When writing ECW v2 and JPEG2000 output, a value of 0 will ensure the output has an opacity band
to make the background transparent.
If no region and no opacity value are specified, Image Compressor will recreate the input opacity
information when detected however a region will override this automatic translation.
The following diagram visualizes each available option where the red polygon is the input region,
No Region and Opacity 1 have opaque backgrounds with no opacity band in the output. Opacity
1 will have reduced storage requirements due to null blocks however will still have a solid
background colour (defined by nullcolor). Only opacity values of 0 or 2 will create an opacity band
and allow underlying data to be shown in this area.
Default value is 2 for ECW v3 output and 0 for ECW v2 and JPEG2000
31
-datasetregionfile region.txt
A file on disk that represents a closed polygon in image coordinates (pixel space) defining the active
or data area of the image. All area outside of this polygon will be tagged as null and/or transparent
as per the opacity option.
Image Compressor will not clip the output image dimensions to the extents of the input region so it
should not be used to subset larger images.
Dataset coordinates
Output result
563,1642;
1071,1681;
1040,1408;
899,1423;
915,1486;
727,1509;
758,1337;
1181,1345;
1181,1775;
625,1760;
625,1939;
1321,1916;
1353,1165;
664,1181
Note: Each coordinate pair must be specified on a new line with a semi-colon separator.
-shapefile vector.shp, layername, fid
Provide an Esri Shapefile to define the non-Null or active area region defined by a closed polygon
feature. Requirements:
Shapefile projection must match the coordinate system of the input raster
FID identifier must map to a single, simple polygon feature
-wktfile region.txt
Identical to worldregionfile however in OGC WKT format. Coordinates must match the dataset
source projection. Multi-polygon type is currently not supported.
Dataset coordinates
Output result
32
Advanced Options
-bitdepth 8 | 16
Force the output cell type to the provided value. If undefined the Image Compressor will maintain
the input celltype so this option is only required in instances where the input type is incorrectly
detected. Int16 is supported in ECW v3 and JPEG2000 output.
Note: Setting bitdepth to 8 (uint8) when the input range is uint16 will not rescale the data. Image
Compressor will merely truncate the data values.
-signed true | false
Used in conjunction with bitdepth 16 to set signed or unsigned int16 output. Signed INT16 is only
supported in JPEG2000 output.
-colorspace greyscale | rgb | multiband
Both ECW and JPEG2000 formats support three colorspaces which are dependent on the band count.
When undefined, Image Compressor will maintain the colorspace of the input data. This option can
be used in conjunction with bandlist to subset multiband input to create RGB or Greyscale output.
-bandlist 0,1,2
Explicitly select which input bands are to be compressed, indexed from 0. Default option is to
compress all bands in original order.
This option allows bands to be reordered such as BGR to RGB (eg. 2,1,0) or to be targeted selectively
when used with the colorspace option. For example a 7 band Multiband input can be mapped to a
RGB 7,4,2 output by specifying colorspace RGB bandlist 7,4,2.
-logfile C:\log.txt
All information printed to the console will also be logged to file, useful for auditing or performance
purposes.
-genstats true | false
When true the Image Compressor will keep count of histogram bins and data statistics such as
mean, mode, standard deviation for writing to the output file. This is calculated during the
compression process and is only available when writing ECW v3 files (with embedded metadata
support). For performance reasons the calculated statistics is based on the input pixels rather than
the output compressed pixels. This tradeoff will be improved in a future release.
Default value true for ECW v3.
-nullvalue min | max
The colour definition to assign to null blocks when opacity is set to 1 or 2. The min/max represents
the smallest or largest pixel value for the specified bitdepth range. For uint8, min will be 0 (black)
and max, 255 (white). Default value min.
-interactive true | false
Compressor will prompt for user input when the following conditions are met,
ERDAS Image Compressor User Guide
33
When using batch compression mode via the GUI, interactive defaults to false. Command line
defaults to true.
-srs EPSG:2180
Override the input projection definition and use the specified EPSG code when writing the
georeferencing information in the output file. This parameter should only be used where the Image
Compressor is unable to detect the source projection correctly. For example, the following IMG input
lists a projection code of WGS84 / LOCAL indicating an offset was detected but EPSG lookup failed.
Where input files are known to have an EPSG code however the code was not detected correctly, the
srs value (-srs EPSG:31984) can be passed into the compressor to set this value explicitly and a
relevant warning will be shown in the log.
Note: The Image Compressor will not perform any reprojection of the source to target projection
systems, nor does it modify in anyway the origin or cell size values. The option should not be used
when the compressor is correctly listing the input projection code. For reprojection needs, ERDAS
IMAGINE is recommended.
-xml (C:\data\mosaic.xml)
Define all compression and mosaic parameters within the XML project file and pass into the
compressor. This option is exclusive of all others, including <input>, <output> and other options.
Example usage: ImageCompressor.exe xml c:\data\mosaic.xml
34
Reporting example
The Image Compressor records a variety of metrics of the input, output, hardware and throughput
for each compression job. When reporting issues or concerns regarding processing speed, always
include the full report as part of the Intergraph Support ticket.
The information is also written to a logfile when used with the logfile option
35
All relevant information on the input and output results is recorded in this report. Warnings and
Errors will also be reported in this area.
Usage examples
GeoTIFF to ECW v2 using the line encoder
ImageCompressor c:\data\test.tif e:\data\test.ecw method line
ArcInfo Binary Grid (AIG) to lossless JPEG2000 INT16
ImageCompressor c:\dem\w001000.adf e:\data\dem.jp2 targetrate 1 signed true
4 band BGRN UINT16 IMG HFA to RGB ECW v3 with target rate of 20:1 and band stats created
ImageCompressor c:\multi\bgrn.img e:\data\rgb.ecw version 3 bandlist 2,1,0
targetrate 20 -genstats true
3 band ERMapper ALG to ECW v3 with null blocks and opacity
ImageCompressor c:\alg\mosaic.alg e:\data\mosaic.ecw version 3 opacity 2 shapefile
c:\alg\region.shp,region,0
Mosaic XML project to ECW v2 at 20:1 target ratio (defined within the XML file)
ImageCompressor -xml project.xml
Panchromatic IMG to lossy JPEG2000 with opacity defined by WKT region
ImageCompressor c:\jp2\data.img e:\data\output.jp2 wktfile c:\jp2\region.txt opacity 0
Multiband 7 band TIF to RGB ECW with band selection
ImageCompressor multiband.tif rgb.ecw bandlist 7,4,2 colorspace rgb
36
Example mosaic.xml
<?xml version="1.0" ?>
<imagecompressor version="1.0">
<!--compression task: attributes describe the options for the process-->
<compresstask interactive="true|false" logfile="C:/temp/compressor.log" loglevel="Info" method="tile"
tempdir="C:/temp" threads="4" memcache="0.25">
<!--inputs: multiple files determine mosaic mode-->
<inputs>
<file bandlist="0,1,2" path="C:/dir/file_rgb1.ecw" zindex="0"/>
<file bandlist="0,1,2" path="C:/dir/file_rgb2.ecw" zindex="1"/>
<file bandlist="7,4,2" path="C:/dir/file_landsat5.ecw" zindex="2"/>
</inputs>
<!--output options-->
<output path="C:/data/output/compressed.ecw" version="2">
<!--output bandlist-->
<bandlist>
<band description="red" id="0"/>
<band description="green" id="1"/>
<band description="blue" id="2"/>
</bandlist>
<!--region definition from shapefile-->
<region layerid="1" layername="file" name="region1" path="file.shp" type="shape"/>
<!--region definition from WKT-->
<region name="region2" path="file.wkt" type="wkt"/>
<!--metadata: ecwv3 only-->
<metadata>
<item name="classification" value="raster image"/>
<item name="acquistiondate" value="2013-09-12"/>
<item name="acquisitionsensorname" value="Landsat 7"/>
<item name="author" value="John Smith"/>
<item name="copyright" value="Intergraph"/>
<item name="company" value="Intergraph"/>
<item name="email" value="[email protected]"/>
<item name="address" value="2 Abbotsford St, West Leederville WA 6007"/>
<item name="telephone" value="(08) 9388 2900"/>
</metadata>
37
38
GTiff: GeoTIFF
ELAS: ELAS
DIPEx: DIPEx
ACE2: ACE2
39
Note: When mosaicking input data, not all GDAL supported formats listed above are available due to nonthread safe operations. The list above is supported for batch processing or single input only.
40
An important observation in these examples are the first and third use-cases. Both have 4 vertices
however clearly the percentage ratio to data is a lot higher in the third example at 95%. Therefore
null blocks will provide the greatest benefit to the third image both in terms of additional file storage
savings and compression speed. The first example will still benefit and is still a good candidate for
null blocks however will not see as significant gains. Null blocks in this way will always provide
varying levels of optimizations depending on input as highlighted in the following examples.
41
Comparison
Sydney Landsat scene
Input image
Dimensions:
15,221 x 14,661 px
(0.223 gigapixel)
Structure:
Opacity band:
false
Projection:
EPSG:32656
Ratio to data:
30.234%
Hardware
Windows 7 / Server 2008 R2
Platform:
CPU Model:
Intel(R) Core(TM) i7 CPU
Q 740 @ 1.73GHz
CPU Model:
Intel(R) Core(TM) i7 CPU
Q 740 @ 1.73GHz
CPU Cores:
CPU Cores:
Memory:
8
8,128.00 MB
Memory cache
42
Memory:
8
8,128.00 MB
Memory cache
System:
512.00 MB
System:
512.00 MB
Read:
1,911.85 MB
Read:
1,911.85 MB
Write:
120.15 MB
Write:
120.15 MB
Threads:
Threads:
Precincts:
Total Blocks:
73376
18344
Precincts:
Total Blocks:
73376
18344
Data Blocks:
13251
Data Blocks:
18344
Null Blocks:
5093
Null Blocks:
Duration:
Duration:
Target Ratio:
15:1
Target Ratio:
15:1
Actual Ratio:
31.3:1
Actual Ratio:
30.7:1
Throughput:
11.5 MB / sec
Output Data
File Name:
Throughput:
8.0 MB / sec
Output Data
f:\landsat-null.ecw
File Name:
f:\landsat-no-null.ecw
File Type:
ECW v3
File Type:
ECW v3
Data Writer:
Data Writer:
Dimensions:
Structure:
File Size:
15,221 x 14,661 px
4 Band, RGB UINT8
20.39 MB
Dimensions:
Structure:
File Size:
15,221 x 14,661 px
4 Band, RGB UINT8
20.79 MB
43
Input image
Dimensions:
372,535 x 477,806 px
(177.999 gigapixel)
Structure:
Opacity band:
true
Projection:
EPSG:28350
33
Ratio to data:
89.731%
Hardware
Windows 7 / Server 2008 R2
Platform:
CPU Model:
Intel(R) Xeon(R) CPU
E5410 @ 2.33GHz
CPU Model:
Intel(R) Xeon(R) CPU
E5410 @ 2.33GHz
CPU Cores:
CPU Cores:
Memory:
44
8
16,380.00 MB
Memory:
8
16,380.00 MB
Memory cache
Memory cache
System:
512.00 MB
System:
512.00 MB
Read:
2,319.47 MB
Read:
2,319.47 MB
Write:
1,775.53 MB
Write:
1,775.53 MB
Threads:
Threads:
Precincts:
57967752
Precincts:
57967752
Total Blocks:
14491938
Total Blocks:
Data Blocks:
1506776
Data Blocks:
14491938
Null Blocks:
12985162
Null Blocks:
Duration:
Duration:
14491938
Target Ratio:
15:1
Target Ratio:
15:1
Actual Ratio:
209.4:1
Actual Ratio:
169.3:1
Throughput:
210.6 MB / sec
Output Data
File Name:
Throughput:
51.2 MB / sec
Output Data
g:\corridor-null.ecw
File Name:
g:\corridor-no-null.ecw
File Type:
ECW v3
File Type:
ECW v3
Data Writer:
Data Writer:
Dimensions:
372,535 x 477,806 px
Dimensions:
372,535 x 477,806 px
Structure:
File Size:
Structure:
File Size:
45
FAQ
Can I purchase the Image Compressor application separately?
Not at this time. Please contact your local Intergraph representative to express interest in an
individual licensing model.
Should I use Image Compressor or IMAGINE Professional?
You are entitled to use both. Depending on your workflow you may find Image Compressor too
limiting or conversely perfectly suitable.
Are files created by Image Compressor any different to those from IMAGINE?
No. There may be slight variations to the metadata stored in the file headers, but otherwise an ECW
or JPEG2000 compressed image will be identical regardless what application is used. Both
applications use the same underlying ECWJP2 SDK version that performs the compression.
What relationship does ERDAS Image Compressor have with the legacy ER Mapper Image
Compressor application?
None. The ERDAS Image Compressor is a completely redesigned and rebuilt compression application.
Despite the similar product naming, the capabilities are not equivalent and is not a direct upgrade
from the legacy product.
What advantages does Image Compressor provide over using the ECWJP2 SDK within GDAL?
Although GDAL has excellent support for the ECWJP2 SDK, a key feature it lacks is the multi-threaded
tile encoder and region management for null block and opacity generation. There are also other
performance implications that limit throughput if GDAL is used for both reading and writing. Image
Compressor will remain quicker at compressing equivalent input even if the GDAL reader is used.
What mosaic file format gives the best performance?
Image Compressor supports both ER Mapper ALG and GDAL VRT file formats. Both support a wide
range of dynamic capabilities including the ability to mosaic separate files together to present them
as a single virtual file or mosaic. The Image Compressor mosaicking process uses the ER Mapper ALG
engine internally for this capability and was chosen for performance reasons and because its threadsafe. VRT is supported however is known to perform slower than an equivalent ALG especially for
large input sizes. VRT files can also only be compressed using the line encoder.
Why cant I define a multi-polygon for opacity/null definition?
This limitation will be removed from a future release of Image Compressor.
I use complex ER Mapper Algorithms that link to Virtual ERS rasters, kernels and so on, will Image
Compressor compress these files?
Yes. Image Compressor uses the full ER Mapper Library thats included in ER Mapper Professional
2014, however in a 64-bit environment. Image Compressor can be used to compress large datasets
46
where memory requirements exceed 32-bit limits but still benefit from the powerful visualization
tools that ER Mapper provides.
Is Image Compressor faster than Product X?
Image Compressor compresses faster than the legacy ER Mapper 7.2 (64-bit), ER Mapper Mosaic
Balance Compressor (MBC), Safe FME and Global Mapper products all of which are used heavily
throughout the industry for compressing to ECW and JPEG2000. These products use the legacy
ECWJP2 SDK v3.x which was released over 7 years ago. In comparison ERDAS IMAGINE 2014 and
Image Compressor 2014 are built on the latest v5.1 ECWJP2 SDK and benefits from years of further
optimizations, bug fixes, ECW v3 and other improvements. Image Compressor is also maintained by
the same development team as the ECWJP2 SDK ensuring the product is the reference, best-of-breed
implementation of the underlying compression engine.
Why does the compressor prevent some input formats from being compressed in tile mode?
Using the tile compression method forces multi-threaded accesses on the input data. For a variety of
reasons many input formats are not thread-safe. Where we are aware of formats that are not
thread-safe we default to the line compression method and will return an error if tile is used.
The GDAL VRT format is one such example of an input format that compresses without issue with a
single thread however causes significant problems when called from multiple threads. This problem
is a general GDAL Data Reader problem and is not specific to Image Compressor, see GDAL
documentation. As improvements are made to the Data Readers like GDAL, Image Compressor will
re-enable these formats.
Can the compressor be used on a headless Linux machine?
Yes, however a known limitation is that we provide only a GUI based installation (BIN) package. This
will be remedied in a future release. Files can be transferred easily to the headless machine after
installation and the command line compressor will function as expected.
What window managers are supported on Linux?
The Image Compressor GUI is built using the Qt cross-platform library and has been tested across
Gnome, KDE and Unity platforms.
Why are some Linux platforms listed as viable?
Due to time constraints, platforms which are viable are deemed to be usable and in some cases
tested however not to the same degree as fully supported platforms and may require additional
libraries to function.
Why is georeferencing sometimes lost in the output files?
The Image Compressor relies on the Data Readers to interpret georeferencing information and
determine an EPSG code. For a variety of reasons this process is not always successful in which case
the Image Compressor will create output files in a WGS84/LOCAL. This definition can be altered to
the correct EPSG code by updating the file header after the fact; a recompression is not required. Or
alternatively the srs command line parameter can be defined prior to compression to force the
output definition.
If no georeferencing is detected at all and the input data is raw, RAW/RAW projection/datum
pairing will be written to the output.
ERDAS Image Compressor User Guide
47
To confirm, check the Compressor log and look for the Projection value set within the Input Data
metadata area.
What settings should I use to get best throughput?
This question is dependent on too many variables and can only be answered after benchmarks and
comparisons conducted on customer hardware with customer input data. Contact Intergraph
Support for assistance.
Are pyramids or overviews required for compression?
No. The Image Compressor only reads the input data at native resolution. If files such as RRD, OVR or
internally defined overviews are present they will be ignored. Generating these files is a redundant
processing step for compression workflows; they do not speed up the compression of ECW or
JPEG2000 files.
Why is the output actual compression ratio significantly different to the target when using ECW v3
null blocks?
Depending on the region supplied and the amount of area defined as null, it is not uncommon to
specify 20:1 target and obtain an actual compression rate of 50:1 or more. When null blocks are
enabled the amount of data written to the output file can be substantially reduced which is reflected
in the actual compression ratio since this divides the uncompressed input size by the output size. It
makes no allowances between null or data areas.
This behaviour can be quantified by customers using the APOLLO Essentials image quality utility or
by your own visual quality assessment by overlaying a sample image with and without null blocks
enabled. The image quality of the data areas will remain identical to the same image compressed
to 20:1 without null blocks even though its actual target rate may be closer to the target rate.
When should null blocks be enabled?
Null blocks were designed to shrink the output file size and speed up compression speed, but these
benefits are directly related to the characteristics of the input. To help identify when null blocks
should be enabled, the compression output will report the Ratio to data as well as the number of
vertices that make up the selected region. Where the ratio to data is low enabling null blocks will not
yield much change to the output file size and in many cases will increase compression time due to
the additional spatial intersection checking. Similarly if the number of vertices is in the thousands,
performing spatial intersection tests against very complex polygons could adversely impact the
compression time negating the compression speed improvements.
The degree to which these trade-offs affect throughput is heavily dependent on the input data but
also the hardware that the compression is performed on. The only reliable way to answer this
question is to perform two identical compression tests, one with opacity 0 (or with no region
specified) and one with opacity 2. This will allow direct comparison of the benefits that null blocks
provide. See the Null Block Analysis chapter for more information and comparisons.
Why does the compressor need to calculate min/max values?
In order to compress signed or unsigned 16-bit output the compressor needs to know the data range
to ensure high quality output when the actual data range is a smaller sample set, for example 11-bit
stored in a 16-bit range. This step is not required for 8-bit output.
48
Depending on the size and format type this range calculation may take a few minutes before the
actual compression begins. This calculation uses an approximate algorithm by taking an overview
image of approximately 2,500 pixel samples to determine the range. If the range reported appears
wildly inaccurate it is recommend to calculate full statistics prior to compression and the Image
Compressor will read existing statistics when detected.
Image Compressor supports reading statistics stored in *.aux.xml, ECW v3, ERS and ALG files.
When compressing large inputs why is there a delay between reaching 100% complete and the
compressor exiting?
For large inputs 100gigapixels or more, there can be a delay of a few minutes between when the
compressor reaches 100% complete and the output statistics are printed and the compressor exits.
This is expected as the compressor needs to reassemble the temporary files into the final output and
delete these files from disk before the compression is deemed complete. You will observe a high
level of disk activity within this period.
49
Support
The Image Compressor product support is available to all customers with active Software
Maintenance on any of the dependant Intergraph Geospatial 2014 products. See the Intergraph SG&I
Support webpage for more information on how to raise support requests.
When reporting problems, please include the example Image Compressor output which will include
all relevant information. Input data may be required.
50