Learning Geospatial Analysis with Python - Second Edition

Table of Contents

Learning Geospatial Analysis with Python Second Edition

Credits

About the Author

About the Reviewers

www.PacktPub.com

Support files, eBooks, discount offers, and more

Why subscribe?

Free access for Packt account holders

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Downloading the example code

Downloading the color images of this book

Errata

Piracy

Questions

1. Learning Geospatial Analysis with Python

Geospatial analysis and our world

Beyond disasters

History of geospatial analysis

Geographic information systems

Remote sensing

Elevation data

Computer-aided drafting

Geospatial analysis and computer programming

Object-oriented programming for geospatial analysis

Importance of geospatial analysis

Geographic information system concepts

Thematic maps

Spatial databases

Spatial indexing

Metadata

Map projections

Rendering

Remote sensing concepts

Images as data

Remote sensing and color

Common vector GIS concepts

Data structures

Buffer

Dissolve

Generalize

Intersection

Merge

Point in polygon

Union

Join

Geospatial rules about polygons

Common raster data concepts

Band math

Change detection

Histogram

Feature extraction

Supervised classification

Unsupervised classification

Creating the simplest possible Python GIS

Getting started with Python

Building SimpleGIS

Step by step

Summary

2. Geospatial Data

An overview of common data formats

Data structures

Common traits

Geolocation

Subject information

Spatial indexing

Indexing algorithms

Quadtree index

R-tree index

Grids

Overviews

Metadata

File structure

Vector data

Shapefiles

CAD files

Tag-based and markup-based formats

GeoJSON

Raster data

TIFF files

JPEG, GIF, BMP, and PNG

Compressed formats

ASCII Grids

World files

Point cloud data

Web services

Summary

3. The Geospatial Technology Landscape

Data access

GDAL

OGR

Computational geometry

The PROJ.4 projection library

CGAL

JTS

GEOS

PostGIS

Other spatially-enabled databases

Oracle spatial and graph

ArcSDE

Microsoft SQL Server

MySQL

SpatiaLite

Routing

Esri Network Analyst and Spatial Analyst

pgRouting

Desktop tools (including visualization)

Quantum GIS

OpenEV

GRASS GIS

uDig

gvSIG

OpenJUMP

Google Earth

NASA World Wind

ArcGIS

Metadata management

GeoNetwork

CatMDEdit

Summary

4. Geospatial Python Toolbox

Installing third-party Python modules

Installing GDAL

Windows

Linux

Mac OS X

Python networking libraries for acquiring data

The Python urllib module

FTP

ZIP and TAR files

Python markup and tag-based parsers

The minidom module

ElementTree

Building XML

Well-known text (WKT)

Python JSON libraries

The json module

The geojson module

OGR

PyShp

dbfpy

Shapely

Fiona

GDAL

NumPy

PIL

PNGCanvas

GeoPandas

PyMySQL

PyFPDF

Spectral Python

Summary

5. Python and Geographic Information Systems

Measuring distance

Pythagorean theorem

Haversine formula

Vincenty's formula

Calculating line direction

Coordinate conversion

Reprojection

Editing shapefiles

Accessing the shapefile

Reading shapefile attributes

Reading shapefile geometry

Changing a shapefile

Adding fields

Merging shapefiles

Merging shapefiles with dbfpy

Splitting shapefiles

Subsetting spatially

Performing selections

Point in polygon formula

Bounding Box Selections

Attribute selections

Creating images for visualization

Dot density calculations

Choropleth maps

Using spreadsheets

Using GPS data

Geocoding

Summary

6. Python and Remote Sensing

Swapping image bands

Creating histograms

Performing a histogram stretch

Clipping images

Classifying images

Extracting features from images

Change detection

Summary

7. Python and Elevation Data

ASCII Grid files

Reading grids

Writing grids

Creating a shaded relief

Creating elevation contours

Working with LIDAR

Creating a grid from LIDAR

Using PIL to visualize LIDAR

Creating a triangulated irregular network

Summary

8. Advanced Geospatial Python Modeling

Creating a Normalized Difference Vegetative Index

Setting up the framework

Loading the data

Rasterizing the shapefile

Clipping the bands

Using the NDVI formula

Classifying the NDVI

Additional functions

Loading the NDVI

Preparing the NDVI

Creating classes

Creating a flood inundation model

The flood fill function

Making a flood

Creating a color hillshade

Least cost path analysis

Setting up the test grid

The simple A* algorithm

Generating the test path

Viewing the test output

The real-world example

Loading the grid

Defining the helper functions

The real-world A* algorithm

Generating a real-world path

Routing along streets

Geolocating photos

Summary

9. Real-Time Data

Tracking vehicles

The NextBus agency list

The NextBus route list

NextBus vehicle locations

Mapping NextBus locations

Storm chasing

Reports from the field

Summary

10. Putting It All Together

A typical GPS report

Working with GPX-Reporter.py

Stepping through the program

The initial setup

Working with utility functions

Parsing the GPX

Getting the bounding box

Downloading map and elevation images

Creating the hillshade

Creating maps

Measuring the elevation

Measuring the distance

Retrieving weather data

Summary

Index

Learning Geospatial Analysis with Python Second Edition

Learning Geospatial Analysis with Python Second Edition

Copyright © 2015 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

First published: October 2013

Second edition: December 2015

Production reference: 1211215

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham B3 2PB, UK.

ISBN 978-1-78355-242-9

www.packtpub.com

Credits

Author

Joel Lawhead

Reviewers

Mark Cederholm

Truc Viet Le

John Maurer

Julia Wood

Commissioning Editor

Kartikey Pandey

Acquisition Editors

Kevin Colaco

Usha Iyer

Kartikey Pandey

Content Development Editor

Anish Sukumaran

Technical Editor

Manthan Raja

Copy Editor

Tasneem Fatehi

Project Coordinator

Izzat Contractor

Proofreader

Safis Editing

Indexer

Mariammal Chettiyar

Graphics

Jason Monteiro

Production Coordinator

Arvindkumar Gupta

Cover Work

Arvindkumar Gupta

About the Author

Joel Lawhead is a project management institute-certified Project Management Professional (PMP), certified GIS Professional (GISP), and the Chief Information Officer (CIO) of NVision Solutions Inc., an award-winning firm that specializes in geospatial technology integration and sensor engineering.

Joel began using Python in 1997 and started combining it with geospatial software development in 2000. He is the author of the first edition of Learning Geospatial Analysis with Python and QGIS Python Programming Cookbook, both by Packt Publishing. His Python cookbook recipes were featured in two editions of Python Cookbook, O'Reilly Media. He is also the developer of the widely-used, open source Python Shapefile Library (PyShp). He maintains the geospatial technical blog http://geospatialpython.com/ and the Twitter feed, @SpatialPython, which discusses the use of the Python programming language in the geospatial industry.

In 2011, Joel reverse-engineered and published the undocumented shapefile spatial indexing format and assisted fellow geospatial Python developer, Marc Pfister, in reversing the algorithm used, allowing developers around the world to create better-integrated and more robust geospatial applications.

Joel serves as the lead architect, project manager, and co-developer for geospatial applications used by U.S. government agencies, including NASA, FEMA, NOAA, the U.S. Navy, and many other commercial and non-profit organizations. In 2002, he received the international Esri Special Achievement in GIS award for his work on the Real-Time Emergency Action Coordination Tool (REACT), for emergency management using geospatial analysis.

About the Reviewers

Mark Cederholm, GISP, has over 20 years of experience in developing GIS applications using various Esri technologies, from ARC/INFO AML to ArcObjects to ArcGIS Runtime and Web SDKs. He lives in Flagstaff, Arizona.

He has been a technical reviewer for the book, Developing Mobile Web ArcGIS Applications, Packt Publishing.

Truc Viet Le is currently a PhD candidate in information systems at the Singapore Management University. His research interests primarily involve novel methods for the modeling and predicting of human mobility patterns and trajectories, learning smart strategies for urban transportation, and traffic flow prediction from fine-grained GPS and sensor network data. He uses R and Python every day for his work, where he finds R superb for data manipulation/visualization and Python an ideal environment for machine learning tasks. He is also interested in applying data science for the social work and international development work. When not behind the computer screen, he is an avid traveler, adventurer, and an aspiring travel writer and photographer. His work portfolio and some of his writings can be found on his personal website at http://vietletruc.com/.

He spent a wonderful year at Carnegie Mellon University in Pittsburgh, Pennsylvania, while pursuing his PhD. Previously, he obtained his bachelor's and master's degrees from Nanyang Technological University in computer engineering and mathematical sciences.

John Maurer is a programmer and data manager at the Pacific Islands Ocean Observing System (PacIOOS) in Honolulu, Hawaii. He creates and configures web interfaces and data services to provide access, visualization, and mapping of oceanographic data from a variety of sources, including satellite remote sensing, forecast models, GIS layers, and in situ observations (buoys, sensors, shark tracking, and so on) throughout the insular Pacific. He obtained a graduate certificate in remote sensing as well as a master's degree in geography from the University of Colorado, Boulder, where he developed software to analyze ground-penetrating radar (GPR) for snow accumulation measurements on the Greenland ice sheet. While in Boulder, he worked with the National Snow and Ice Data Center (NSIDC) for eight years, sparking his initial interest in Earth science and all things geospatial: an unexpected but comfortable detour from his undergraduate degree in music, science, and technology at Stanford University.

Julia Wood is currently a Geospatial Information Sciences (GIS) analyst who spends her professional time completing projects as a contractor in the Washington D.C. area. She graduated magna cum laude from the University of Mary Washington in Fredericksburg, Virginia, in the spring of 2014 with a bachelor's degree in both history and geography as well as a minor in GIS. Though her career is still in its early stages, Julia has aspirations to keep growing her skill set, and working on this review has certainly helped expand her professional experience; she hopes to continue learning and eventually work toward a master's degree while still working full time. In her non-work life, she enjoys reading, crafting, cooking, and exploring the big city, one local restaurant at a time.

Reviewing this book for Packt Publishing was Julia's first professional reviewing experience and she hopes that she can pursue similar endeavors in the future.

I'd like to thank my parents, John and Diana, for always encouraging me to do well in my educational and professional endeavors; my sister, Sarrina, and my brother, Jonathan, for offering support and advice when I needed it; and my boyfriend, Max, and my cat, Coco, for keeping me company while I conducted the reviews for this book. A thank you to Packt for letting me be a part of this experience!

www.PacktPub.com

Support files, eBooks, discount offers, and more

For support files and downloads related to your book, please visit www.PacktPub.com.

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at for more details.

At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks.

https://www2.packtpub.com/books/subscription/packtlib

Do you need instant solutions to your IT questions? PacktLib is Packt's online digital book library. Here, you can search, access, and read Packt's entire library of books.

Why subscribe?

Fully searchable across every book published by Packt

Copy and paste, print, and bookmark content

On demand and accessible via a web browser

Free access for Packt account holders

If you have an account with Packt at www.PacktPub.com, you can use this to access PacktLib today and view 9 entirely free books. Simply use your login credentials for immediate access.

Preface

The book starts with a background on geospatial analysis, offers a flow of techniques and technology used, and splits the field into its component specialty areas, such as Geographic Information Systems (GIS), remote sensing, elevation data, advanced modeling, and real-time data. The focus of the book is to lay a strong foundation in using the powerful Python language and framework in order to approach geospatial analysis effectively. In doing so, we'll focus on using pure Python as well as certain Python tools and APIs, and using generic algorithms. The readers will be able to analyze various forms of geospatial data that comes in and learn real-time data tracking and how to apply this to interesting scenarios.

While many third-party geospatial libraries are used throughout the examples, a special effort will made by us to use pure Python, with no dependencies, whenever possible. This focus on pure Python 3 examples is what will set this book apart from nearly all other information in this field. This book may be the only geospatial book using Python 3 on the market currently. We will also go through some popular libraries that weren't in the previous version of the book.

What this book covers

Chapter 1, Learning Geospatial Analysis with Python, introduces geospatial analysis as a way of answering questions about our world. The differences between GIS and remote sensing are explained. Common geospatial analysis processes are demonstrated using illustrations, basic formulas, pseudo code, and Python.

Chapter 2, Geospatial Data, explains the major categories of data as well as several newer formats that are becoming more and more common. Geospatial data comes in many forms. The most challenging part of geospatial analysis is acquiring the data that you need and preparing it for analysis. Familiarity with these data types is essential to understand geospatial analysis.

Chapter 3, The Geospatial Technology Landscape, tells you about the geospatial technology ecosystem that consists of thousands of software libraries and packages. This vast array of choices is overwhelming for newcomers to geospatial analysis. The secret to learning geospatial analysis quickly is understanding the handful of libraries and packages that really matter. Most other software is derived from these critical packages. Understanding the hierarchy of geospatial software and how it's used allows you to quickly comprehend and evaluate any geospatial tool.

Chapter 4, Geospatial Python Toolbox, introduces software and libraries that form the basis of the book and are used throughout. Python's role in the geospatial industry is elaborated: the GIS scripting language, mash-up glue language, and full-blown programming language. Code examples are used to teach data editing concepts, and many of the basic geospatial concepts in Chapter 1, Learning Geospatial Analysis Using Python, are also demonstrated in this chapter.

Chapter 5, Python and Geographic Information Systems, teaches you about simple yet practical Python GIS geospatial products using processes, which can be applied to a variety of problems.

Chapter 6, Python and Remote Sensing, shows you how to work with remote sensing geospatial data. Remote sensing includes some of the most complex and least-documented geospatial operations. This chapter will build a solid core for you and demystify remote sensing using Python.

Chapter 7, Python and Elevation Data, demonstrates the most common uses of elevation data and how to work with its unique properties. Elevation data deserves a chapter all on its own. Elevation data can be contained in almost any geospatial format but is used quite differently from other types of geospatial data.

Chapter 8, Advanced Geospatial Python Modeling, uses Python to teach you the true power of geospatial technology. Geospatial data editing and processing help us understand the world as it is. The true power of geospatial analysis is modeling. Geospatial models help us predict the future, narrow a vast field of choices down to the best options, and visualize concepts that cannot be directly observed in the natural world.

Chapter 9, Real-Time Data, examines the modern phenomena of geospatial analysis. A wise geospatial analyst once said, "As soon as a map is created it is obsolete." Until recently by the time you collected data about the earth, processed it, and created a geospatial product, the world it represented had already changed. But modern geospatial data shatter this notion. Data sets are available over the Internet which are up to the minute or even the second. This data fundamentally changes the way we perform geospatial analysis.

Chapter 10, Putting It All Together, combines the skills from the previous chapters step by step to build a generic corporate system to manage customer support requests and field support personnel that could be applied to virtually any organization.

What you need for this book

You will require Python (3.4 or higher), a minimum hardware requirement of a 300-MHz processor, 128 MB of RAM, 1.5 GB of available hard disk, and Windows, Linux, or OS X operating systems.

Who this book is for

This book is for anyone who wants to understand digital mapping and analysis and who uses Python or any other scripting language for the automation or crunching of data manually. This book primarily targets Python developers, researchers, and analysts who want to perform geospatial, modeling, and GIS analysis with Python.

Conventions

In this book, you will find a number of text styles that distinguish between different kinds of information. Here are some examples of these styles and an explanation of their meaning.

Code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles are shown as follows: The pycsw Python library implements the CSW standard.

A block of code is set as follows:

1.0 encoding=utf-8?>

http://www.opengis.net/kml/2.2>

    Mockingbird Cafe

    Coffee Shop

      -89.329160,30.310964

Any command-line input or output is written as follows:

t.pen(shown=False) t.done()

Note

Warnings or important notes appear in a box like this.

Tip

Tips and tricks appear like this.

Reader feedback

Feedback from our readers is always welcome. Let us know what you think about this book—what you liked or disliked. Reader feedback is important for us as it helps us develop titles that you will really get the most out of.

To send us general feedback, simply e-mail <[email protected]>, and mention the book's title in the subject of your message.

If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, see our author guide at www.packtpub.com/authors.

Customer support

Now that you are the proud owner of a Packt book, we have a number of things to help you to get the most from your purchase.

Downloading the example code

You can download the example code files from your account at http://www.packtpub.com for all the Packt Publishing books you have purchased. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.

Downloading the color images of this book

We also provide you with a PDF file that has color images of the screenshots/diagrams used in this book. The color images will help you better understand the changes in the output. You can download this file from: https://www.packtpub.com/sites/default/files/downloads/2429OS_ColorImages.pdf.

Errata

Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you find a mistake in one of our books—maybe a mistake in the text or the code—we would be grateful if you could report this to us. By doing so, you can save other readers from frustration and help us improve subsequent versions of this book. If you find any errata, please report them by visiting http://www.packtpub.com/submit-errata, selecting your book, clicking on the Errata Submission Form link, and entering the details of your errata. Once your errata are verified, your submission will be accepted and the errata will be uploaded to our website or added to any list of existing errata under the Errata section of that title.

To view the previously submitted errata, go to https://www.packtpub.com/books/content/support and enter the name of the book in the search field. The required information will appear under the Errata section.

Piracy

Piracy of copyrighted material on the Internet is an ongoing problem across all media. At Packt, we take the protection of our copyright and licenses very seriously. If you come across any illegal copies of our works in any form on the Internet, please provide us with the location address or website name immediately so that we can pursue a remedy.

Please contact us at <[email protected]> with a link to the suspected pirated material.

We appreciate your help in protecting our authors and our ability to bring you valuable content.

Questions

If you have a problem with any aspect of this book, you can contact us at <[email protected]>, and we will do our best to address the problem.

Chapter 1. Learning Geospatial Analysis with Python

This chapter is an overview of geospatial analysis. We will see how geospatial technology is currently impacting our world with a case study of one of the worst disease epidemics that the world has ever seen and how geospatial analysis helped stop the deadly virus in its tracks. Next, we'll step through the history of geospatial analysis, which predates computers and even paper maps! Then, we'll examine why you might want to learn a programming language as a geospatial analyst as opposed to just using geographic information system (GIS) applications. We'll realize the importance of making geospatial analysis as accessible as possible to the broadest number of people. Then, we'll step through basic GIS and remote sensing concepts and terminology that will stay with you through the rest of the book. Finally, we'll use Python for geospatial analysis right in the first chapter by building the simplest possible GIS from scratch!

This book assumes some basic knowledge of Python, IT literacy, and at least an awareness of geospatial analysis. This chapter provides a foundation in geospatial analysis, which is needed to attack any subject in the areas of remote sensing and GIS, including the material in all the other chapters of the book.

Geospatial analysis and our world

On March 25, 2014, the world awoke to news from the United Nations World Health Organization (WHO) announcing the early stages of a deadly virus outbreak in West Africa. The fast-moving Ebola virus would spread rapidly over the coming summer months resulting in cases in six countries on three continents, including the United States and Europe.

Government and humanitarian agencies faced a vicious race against time to contain the outbreak. Patients without treatment died in as little as six days after symptoms appeared. The most critical piece of information was the location of new cases relative to the existing cases. The challenge they faced required reporting these new cases in mostly rural areas with limited infrastructure. Knowing where the virus existed in humans provided the foundation for all of the decisions that response agencies needed for containment. The location of cases defined the extent of the outbreak. It allowed governments to prioritize the distribution of containment resources and medical supplies. It allowed them to trace the disease to the first victim. It ultimately allowed them to see if they were making progress in slowing the disease.

This map is a relative heat map of the affected countries based on the number of cases documented and their location:

Unfortunately, the rural conditions and limited number of response personnel at the beginning of the outbreak resulted in a five-day reporting cycle to the Liberian Ministry of Health who initially tracked the virus. Authorities needed better information to bring the outbreak under control as new cases grew exponentially.

The solution came from a Liberian student using open source software from a non-profit Kenyan technology start-up called Ushahidi. Ushahidi is the Swahili word for testimony or witness. A team of developers in Kenya originally developed the system in 2008 to track reports of violence after the disputed presidential election there. Kpetermeni Siakor set up the system in Liberia in 2011 following a similarly controversial election. When the epidemic hit Liberia, Siakor turned Ushahidi into a disease-monitoring platform.

Siakor created a dispatch team of volunteers who received phone calls from the public reporting possible Ebola cases. The details were entered into the Ushahidi database, which was available on a web-based map almost instantly. The Liberian Ministry of Health and other humanitarian organizations could access the website, track the spread of the disease, and properly distribute supplies at health centers. This effort, amplified by the international response, would ultimately contain the epidemic globally. In 2015, cases are receding as the world monitors West African cases in anticipation of the last patient recovering. The following screenshot shows the latest Liberian public Ushahidi map as of April, 2015:

Relief workers also used the Ushahidi disaster mapping system to respond to the 2010 Haiti earthquake. Maps have always been used in disaster relief; however, the modern evolution of GPS-enabled mobile phones, web technology, and open source geospatial software have created a revolution in humanitarian efforts globally.

Note

The Ushahidi API has a Python library that you can find at https://github.com/ushahidi/ushapy.

Beyond disasters

The application of geospatial modeling to disaster relief is one of the most recent and visible case studies. However, the use of geospatial analysis has been increasing steadily over the last 15 years. In 2004, the U.S. Department of Labor declared the geospatial industry as one of 13 high-growth industries in the United States expected to create millions of jobs in the coming decades.

Geospatial analysis can be found in almost every industry, including real estate, oil and gas, agriculture, defense, politics, health, transportation, and oceanography, to name a few. For a good overview of how geospatial analysis is used in dozens of different industries, visit http://geospatialrevolution.psu.edu.

History of geospatial analysis

Geospatial analysis can be traced as far back as 15,000 years ago to the Lascaux cave in southwestern France. In this cave, Paleolithic artists painted commonly hunted animals and what many experts believe are astronomical star maps for either religious ceremonies or potentially even migration patterns of prey. Though crude, these paintings demonstrate an ancient example of humans creating abstract models of the world around them and correlating spatial-temporal features to find relationships. The following image shows one of the paintings with an overlay illustrating the star maps:

Over the centuries, the art of cartography and science of land surveying has developed, but it wasn't until the 1800s that significant advances in geographic analysis emerged. Deadly cholera outbreaks in Europe between 1830 and 1860 led geographers in Paris and London to use geographic analysis for epidemiological studies.

In 1832, Charles Picquet used different half-toned shades of gray to represent deaths per thousand citizens in the 48 districts of Paris as part of a report on the cholera outbreak. In 1854, Dr.