GIS

Data Preparation and Integration

Digesting the Food

1
03/12/20 Ron Briggs, UTDallas POEC 5319 Introduction to GIS
 Data Preparation and Integration:
the necessary steps
• data media conversion
– scanning
– digitizing
• data format conversion
– raster & vector
• data reduction
• error detection and editing
• Geocoding: assigning geographic coordinates
• rectification and registration (one on top of the other)
– overlaying sheets and referencing to the real world
• edge matching & image adjustment (side by side)
– linking & balancing adjacent sheets
• interpolation
• conflation

2
03/12/20 Ron Briggs, UTDallas POEC 5319 Introduction to GIS
 Data Media Conversion--Scanning:
automated recording of map or aerial
• produces raster data
– vectorize using scanner software or GIS
Great if only a raster
– extensive editing may be needed
representation needed.
• electromechanical
– $100-$50,000 instruments Automated creation of vector
– drum or flatbed data via scanning very
– scan resolution depends on price! problematic:
• down to 20 microns (millionth – docs must be clean
of m) – lines at least .1mm
• Scanners v. sensors – complex line work adds error
– Sensors collect data directly in digital – lines shouldn’t be broken with text.
form (e.g. digital cameras) – text may be interpreted as lines
– Photos have higher resolution than – automatic feature detection (road
sensors, therefore scanning still versus railroad) difficult
important, but less so, in 2000
– Still lots of paper maps around e.g.
property ownership records

3
03/12/20 Ron Briggs, UTDallas POEC 5319 Introduction to GIS
 Data Media Conversion--Digitizing:
manually tracing a map or aerial
• Applied to map or aerial photo
• Use hard copy map/photo on
table/tablette, or scanned image on screen
Problems:
(heads-up digitizing) • paper maps unstable
• pen or cursor detects x, y coords – crease and fold
• coordinates are in inches/cms from lower – stretch with humidity ( up to 3%)
left (0,0)
•
– photos more stable (0.2%)
control points (tic marks) relate digitized
coordinates to real world lat/long • map errors transferred to GIS
coordinates – maps often prepared for display
• coordinates captured in stream or point not accuracy
mode
• accuracy of table (but not user!) usually • human hand very shaky
better than 0.1 mm • often generates undershoots,
• all nodes and polys should be marked and overshoots, & double lines
numbered first
– editing and clean-up essential
• essentially a vector approach

4
03/12/20 Ron Briggs, UTDallas POEC 5319 Introduction to GIS
 Data Format Conversion:
Vector raster

vector
raster
• Vector to Vector
– e.g. whole polygon (e.g SAS map data) to 4 possibilities
point/arc/polygon
– computationally intense
vector to raster: line
– no accuracy loss providing data is ‘clean’ – cells assigned if touched by line
– perfectly transitive – stair step appearance of diagonal
• raster to raster lines (called aliasing)
– may involve resampling (see under data – can be visually improved through
reduction) anti aliasing: brightness of cells
– may involve conversion between different varied based on fraction of cell
vendor’s raster formats (e.g. GRID to BIL) covered by the line
• vector to raster: point • raster to vector
– node x,y assigned to closest raster cell – by far the most difficult
– locational shift almost inevitable; error
depends on raster size.
– two points in one cell indistinguishable Transitive: the ability to reproduce the
– not transitive; cannot retrieve original data original data after conversion.
without error

5
03/12/20 Ron Briggs, UTDallas POEC 5319 Introduction to GIS
 Vector to Raster Conversion

Point Orthogonal Line Diagonal Line

(more
problemmatic)

Vector

Note the use of

Raster anti-aliasing to
improve line’s
visual
appearance

6
03/12/20 Ron Briggs, UTDallas POEC 5319 Introduction to GIS
 Raster to Vector Data Conversion: 3-step process
– skeletonizing (or thinning): to reduce rasters to unit width
• peeling approach successively removes outer edges
• medial axis approach determines set of interior pixels farthest from outer edges
– vector extraction: to identify lines
• 4-connected reconstruction
– joins center points of 4-connected neighbors if present
– particularly bad for diagonal line reproduction
• 8-connected reconstruction Available via the
– joins center points of 8-connected neighbors if present ArcScan
– diagonal lines reproduced but adds extra lines
extension for
• 8-connected reconstruction with redundancy elimination
– if 4-connected neighbor line exists, don’t draw diagonal
ArcGIS, as well
– reduces redundant lines as via several
– topological reconstruction: recreates topological structure specialized
– create nodes at line junctions packages from
– construct arcs
other vendors
– define polygons (manual designation required)

7
03/12/20 Ron Briggs, UTDallas POEC 5319 Introduction to GIS
 Raster to Vector Conversion
Skeletonizing

8
03/12/20 Ron Briggs, UTDallas POEC 5319 Introduction to GIS
 Raster to Vector Conversion:
Vector Extraction
4-connect reconstruction
Vector Raster

4-connect reconstruction:
search the 4 surrounding cells and
join center points if present

9
03/12/20 Ron Briggs, UTDallas POEC 5319 Introduction to GIS
 Raster to Vector Conversion:
Vector Extraction
8-connect reconstruction
Vector Raster

8-connect reconstruction:
search the 8 surrounding cells and join
center points if present.

10
03/12/20 Ron Briggs, UTDallas POEC 5319 Introduction to GIS
 Raster to Vector Conversion:
Vector Extraction
8-connect reconstruction with redundancy elimination

Vector Raster

8-connect with redundancy

elimination:
draw diagonal from 8-cell search only if not already
connected by orthogonal from 4-cell search

11
03/12/20 Ron Briggs, UTDallas POEC 5319 Introduction to GIS
• Why?
– conserve space
• Disk in past Data Reduction
• Comm. bandwidth today
– conserve time
• reduce processing time (batch)
• speed response time (interactive) • Thinning (vector data)
• Resampling (raster data) – often applied to data digitized in
– ‘average’ the 4 values in a 2by2 stream mode
neighborhood
– tolerance elimination: remove
– use this 1 value in a single cell occupying
the location of the 4 original cells nearest-neighbor points which
– use mean for interval data; rules required are ‘too close’ (e.g. output device
for ordinal or nominal data resolution insufficient to
– not transitive! distinguish)
– topological elimination: remove
points unnecessary for topo
3 7 16 bytes
structure
2 4
– model-based elimination: fit
polynomial by least squares and
4 bytes
record fewer points along its path
4
1 byte

12
03/12/20 Ron Briggs, UTDallas POEC 5319 Introduction to GIS
 Errors: common errors
• dangling arc (node missing at one
end)
• dangling node (only one arc: could be
legit. e.g cul-de-sac)
• no-node arc intersection
• overshoot
• undershoot
• pseudo node (but perhaps road
surface changes)
• pseudo arc (connects to itself)
• open polygon
• sliver
• gap

13
03/12/20 Ron Briggs, UTDallas POEC 5319 Introduction to GIS
 Errors: detection and removal
• GIS packages commonly use topological structure checking to
detect errors (available in ArcGIS 8.3 and later)
• node snapping used to correct them : moving a feature so its
coordinates correspond exactly with another’s
• snapping conducted based on tolerances -- snap if within 1
foot, for example
• node migration a problem if same node snapped multiple times
• links can be defined to track snaps & control for node migration
• Care must always be taken to assure that topological “cleaning”
does not itself introduce errors (e.g. snapping lines together
which shouldn’t be)

14
03/12/20 Ron Briggs, UTDallas POEC 5319 Introduction to GIS
 Geocoding
assigning spatial coordinates to point data
Address Matching assigns spatial coordinates (explicit location) to
addresses (implicit location)
Address matching requires street network file with street attribute information
(street name and number range for each block) for all street segments (block
sides)
– precise matching of street names can be problematic
– completeness (esp. for ‘new’ streets) important
– PO boxes, building names, and apartment complex names cause problems.
Implementation in ArcGIS is 3-step process
– In ArcCatalog, process street network file to create a Geocoding Service
– In ArcMap, load the geocoding service using Tools/Geocoding/Services Manager
– In ArcMap, geocode a table of addresses using Tools/Geocoding/Geocode Addresses
Point Location Files containing lat/long or x,y coordinates (e.g derived
via GPS) must be converted to shape files for display
– bring table (in .dbf format) into ArcGIS using add data icon
– Right click table name in T of C and select Display X,Y data
Table must contain 3 variables at minimum: Feature ID, x, y
15
03/12/20 Ron Briggs, UTDallas POEC 5319 Introduction to GIS
 Georeferencing: Rectification and Registration
providing true earth location/overlaying layers

• rectification: rearrangment of Two methods

location of objects to correspond to • homogeneous mathematical
a specific reference system (usually transformation via rotation,
geodetic) translation, scaling, skewing
• registration: – used for map projection and
rearrangment of location of objects similar conversions
of one set so they correspond with • differential stretching via
those of another, without reference rubber sheeting
to a specific reference system – used to correctly position
distorted maps or data files
Despite formal difference, often used
interchangeably
•Most commonly used to relate scanned images or photographs to a vector layer,
but also be used to “fix” incorrect positioning of features in a vector layer
•Implemented in ArcMap via the Georeferencing toolbar for images
via the Spatial Adjustment toolbar for vector layers
16
03/12/20 Ron Briggs, UTDallas POEC 5319 Introduction to GIS
 Transformation:
(homogeneous conversion)
• translation of origin
– from digitizer origin for sheet
to ‘true’ origin of GIS file
• rotation of axis
– e.g to true north
translation differential
• scaling of axis
• homogenous:
scaling
• differential (ovals to circles)
• skewing of axis
Changing map projections may
involve all 4 rotation
skewing
17
03/12/20 Ron Briggs, UTDallas POEC 5319 Introduction to GIS
 Rubber Sheeting
(differential conversion)
• GIS file is differentially ‘stretched’ --the more the better
so that tic points in file overlay
corresponding ground control (tie) ground control --well distributed
points on earth’s surface (or tic
points in a second file) --known lat/long of
• polynomial fitted by least squares ground control tie points
between known ground control needed for rectification
coords and tic point coords in GIS (usually obtained with
Map locations GPS)
• derived parameters then applied to
all coordinates in file
--common identifiable
• after conversion tic points on
points in each file
average closer to ground control
needed for registration
points, but not identical
• can’t do this with a paper map! GIS file

18
03/12/20 Ron Briggs, UTDallas POEC 5319 Introduction to GIS
 Edge Matching:
Process
Joining map sheets to create a seamless GIS
• required for topo. consistency even if features line-up
visually
• snapping used to connect features Corresponding features fail
Issues
• acceptable tolerance before ‘further investigation’ of
to match on two sheets:
mismatch
• ‘how far back’ to go on sheet(s) with adjustment s for
mismatch
Causes of mismatch
• paper map shrinkage/expansion
• errors from digitizing/scanning
– georeferencing errors
– accuracy of equipment
– extrapolation or round-off errors
• overlapping map coverage
Implement in ArcView (very crudely) by:
– Read each sheet as a separate theme
– Select File/Extensions and and load Geoprocessing

–
Wizard
Select View/Geoprocessing Wizard
Edge matching in this example
– Choose Merge Themes Together would likely require ‘further research’

Implemented in ArcMAP via

Tools/Geoprocessing/Merge Layers
And also Spatial Adjustment toolbar
19
03/12/20 Ron Briggs, UTDallas POEC 5319 Introduction to GIS
 Image Adjustments
raster/image data issues
Raster data is made from separate images (photos) or tiles which are mosaiced to produce “seamless
image”
Collars: must be removed for seamless image
– Overlap between adjacent images
– Borders of scanned maps
Image Balancing and Feathering: adjusting radiometry for consistent and/or desired image
color, brightness, contrast
– Checker board appearance
– Abrupt line between adjacent images
– Brightness levels wash out detail in highly reflective areas, but enhance detail in low
reflectance areas
– Inconsistent signature for same features, especially water as function of wind or sun relative
to camera (and is it blue?)
Digital Ortho adjustments:
– Ground control (usually with GPS for visible points) to obtain ‘real world’ location
– Ground control for camera’s angle relative to ground
– Camera calibration data to remove lens distortion
– Digital terrain model (dtm) to remove elevation “distance”
(5 mi. on map to mountain top, but 6 mi walking or on photo if mountain is 5,280 feet high!)

20
03/12/20 Ron Briggs, UTDallas POEC 5319 Introduction to GIS
 Interpolation:
to create regular spacings from irregular data
(e.g creating raster elevation surface from set of point height measurements)

• estimating values for • weighting functions

Estimated values

locations with no data – average closest n (2?) points

based on: • ignores distance
– known values, and – fit line between closest 2
– understanding of spatial – fit surface between closest 3
behavior of phenomena • trend surface approaches
– one high order polynomial
• generally, should assign • oscillation a problem
more importance to closer – finite element approach:
known values than those fit separate polynomials for
each local area
further away – kriging: uses correlations of
values with distance

21
03/12/20 Ron Briggs, UTDallas POEC 5319 Introduction to GIS
 Conflation
• create new master coverage from the best spatial and attribute qualities of two or
more source coverages
– combine multiple coverages into one to simplify support
– updated data obtained (e.g. new TIGER file) but need to preserve enhancements made to
earlier version
– two groups modify a single file, then need to recreate single version which preserves mods
• create new master coverage from quality spatial data in one source and quality
attribute data in another
– somewhat narrower definition
• essentially requires application of a variety of processing tools and can be very
labor intensive:
– Arc/Info has Conflation tool which addresses simple applications
• requires common identification of nodes in the two sources
– other add-ins available
• MapMerge from ESEA, Mountain View CA for ArcGIS
• GIS/T-Conflate for transportation applications

22
03/12/20 Ron Briggs, UTDallas POEC 5319 Introduction to GIS
 NAVSTAR Global Positioning System (gps)
Types of Ground Collection and Corrrection
–use to collect ground control for imagery/orthos Autonomous
–or for point/line data (manholes, roads, etc) – Hand-held unit provides 10m accuracy (with SA off)
– $150-$1,500 per unit
NAVSTAR Satellite Program WAAS (wide area augmentation system)
– <3 meter accuracy in practice (spec. is 7m vert/horiz)
• 24 (NAVigation Satellite Time and Ranging) – Base stations (25 across US) monitor satellites
satellites in 11,00 mile orbit provide 24 hour – 2 master stations (E & W coast) calculate corrections
coverage worldwide – upload to two geosynchronous satellites over equator
• – correction signal broadcast to GPS receivers (no special extra equipment
first launched 1978; full system operational needed unlike DGPS)
December 1993. – Began operation June, 1998
• gps receiver computes locations/elevations via – To be expanded to cover Canada, Mexico, Panama
– European EGNO, Asian MSAS under development
signals from simultaneously visible satellites
Differential (DGPS-predecessor to WAAS)
(minimum 3 for 2-D, 4 for 3-D) – accuracy 1-5m depending on equipment/exact method
• Selective Availability (SA) security system – equipment $1,500-$15,000 per receiver
– 100m accuracy with single receiver, if active – correct for SA and other errors via either
• real time correction signals over FM radio
– 10-15m accuracy if inactive
• post process with data from Internet
• SA turned off May 1st, 2000 Kinematic:
– Multiple ways to counteract SA – high accuracy engineering (within cms);
– Even USCG broadcasted correction signal! – two receivers (base station and rover
– must lock-on to satellites
– Europeans threatened to compete
– equipment $15-30K per station
– Regional denial of signal possible
• Russia’s 21-satellite GLONASS (Global Navigation
Satellite System) also available.
 Factors Affecting GPS Accuracy
• Ionosphere
– worst in evening at low altitudes (but ephemerous best there)
• troposhere
– especially water vapor which slows signal
• multipath
– reflected signals from buildings, cliffs, etc
• ephemerous
– position and number of satellites in sky
– 4 required for 3D (horiz. and vertical), 3 for 2D (no elevation)
– ideallly, 3 every 120° horizon. with 20° elev., 1 directly above
• blockage (of satellite signal)
– by foliage, buildings, cliffs, etc.
– WAAS signal espec. subject to blocking by terrain & buildings ‘cos is from
geostationary equatorial satellite

Overall, accuracy better at night than during day.

24
03/12/20 Ron Briggs, UTDallas POEC 5319 Introduction to GIS