Python Geospatial Analysis Cookbook

Table of Contents

Python Geospatial Analysis Cookbook

Credits

About the Author

About the Reviewers

www.PacktPub.com

Support files, eBooks, discount offers, and more

Why Subscribe?

Free Access for Packt account holders

Preface

What this book covers

What you need for this book

Who this book is for

Sections

Getting ready

How to do it…

How it works…

There's more…

See also

Conventions

Reader feedback

Customer support

Downloading the example code

Downloading the color images of this book

Errata

Piracy

Questions

1. Setting Up Your Geospatial Python Environment

Introduction

Installing virtualenv and virtualenvwrapper

Getting ready

How to do it...

How it works...

There's more…

Installing pyproj and NumPy

Getting ready

How to do it...

How it works...

Installing shapely, matplotlib, and descartes

Getting ready

How to do it...

How it works...

Installing pyshp, geojson, and pandas

Getting ready

How to do it...

How it works...

Installing SciPy, PySAL, and IPython

Getting ready

How to do it...

How it works...

Installing GDAL and OGR

Getting ready

How to do it...

How it works...

See also

Installing GeoDjango and PostgreSQL with PostGIS

Getting ready

How to do it...

How it works...

There's more...

2. Working with Projections

Introduction

Discovering projection(s) of a Shapefile or GeoJSON dataset

Getting ready

How to do it...

How it works...

There's more...

See also

Listing projection(s) from a WMS server

Getting ready

How to do it...

How it works...

Creating a projection definition for a Shapefile if it does not exist

Getting ready

How to do it...

How it works...

Batch setting the projection definition of a folder full of Shapefiles

How to do it...

How it works...

Reprojecting a Shapefile from one projection to another

How to do it...

How it works...

See also

3. Moving Spatial Data from One Format to Another

Introduction

Converting a Shapefile to a PostGIS table using ogr2ogr

Getting ready

How to do it...

How it works...

See also

Batch importing a folder of Shapefiles into PostGIS using ogr2ogr

How to do it...

How it works...

There's more…

Batch exporting a list of tables from PostGIS to Shapefiles

How to do it...

How it works...

There's more...

Converting an OpenStreetMap (OSM) XML to a Shapefile

Getting ready

How to do it...

How it works...

Converting a Shapefile (vector) to a GeoTiff (raster)

Getting ready

How to do it...

How it works...

See also

Converting a raster (GeoTiff) to a vector (Shapefile) using GDAL

Getting ready

How to do it...

How it works...

Creating a Shapefile from point data stored in Microsoft Excel

Getting ready

How to do it...

How it works...

Converting an ESRI ASCII DEM to an image height map

Getting ready

How to do it...

How it works...

4. Working with PostGIS

Introduction

Executing a PostGIS ST_Buffer analysis query and exporting it to GeoJSON

Getting ready

How to do it...

How it works...

Finding out whether a point is inside a polygon

Getting ready

How to do it...

How it works...

There's more...

Splitting LineStrings at intersections using ST_Node

Getting ready

How to do it...

How it works...

Checking the validity of LineStrings

How to do it...

Executing a spatial join and assigning point attributes to a polygon

Getting ready

How to do it...

How it works...

Conducting a complex spatial analysis query using ST_Distance()

Getting ready

How to do it...

How it works...

5. Vector Analysis

Introduction

Clipping LineStrings to an area of interest

Getting ready

How to do it...

How it works...

Splitting polygons with lines

Getting ready

How to do it...

How it works...

Finding the location of a point on a line using linear referencing

How to do it...

How it works...

See also

Snapping a point to the nearest line

How to do it...

How it works...

Calculating 3D ground distance and total elevation gain

Getting ready

How to do it...

How it works...

There's more...

6. Overlay Analysis

Introduction

Punching holes in polygons with a symmetric difference operation

Getting ready

How to do it...

How it works...

Union polygons without merging

Getting ready

How to do it...

How it works...

Union polygons with merging (dissolving)

Getting ready

How to do it...

How it works...

Performing an identity function (difference + intersection)

How to do it...

How it works...

7. Raster Analysis

Introduction

Loading a DEM USGS ACSII CDED into PostGIS

Getting ready

How to do it...

How it works...

Creating an elevation profile

Getting ready

How to do it...

How it works...

Creating a hillshade raster from your DEM with ogr

Getting ready

How to do it...

How it works...

Generating slope and aspect images from your DEM

Getting ready

How to do it...

How it works...

Merging rasters to generate a color relief map

Getting ready

How to do it...

How it works...

8. Network Routing Analysis

Introduction

Finding the Dijkstra shortest path with pgRouting

Getting ready

How to do it...

How it works...

Finding the Dijkstra shortest path with NetworkX in pure Python

Getting ready

How to do it...

How it works...

Generating evacuation polygons based on an indoor shortest path

Getting ready

How to do it...

How it works...

Creating centerlines from polygons

Getting ready

How to do it...

How it works...

Building an indoor routing system in 3D

Getting ready

How to do it...

How it works...

Calculating indoor route walk time

How to do it...

How it works...

9. Topology Checking and Data Validation

Introduction

Creating a rule – only one point inside a polygon

Getting ready

How to do it...

How it works...

A point must be on the starting and ending nodes of a line only

How to do it...

How it works...

LineStrings must not overlap

How to do it...

How it works...

A LineString must not have dangles

How to do it...

How it works...

A polygon centroid must be within a specific distance of a line

How to do it...

How it works...

10. Visualizing Your Analysis

Introduction

Generating a leaflet web map with Folium

Getting ready

How to do it...

How it works...

Setting up TileStache to serve tiles

Getting ready

How to do it...

Visualizing DEM data with Three.js

Getting ready

How to do it...

How it works...

Draping an orthophoto over a DEM

Getting ready

How to do it...

How it works...

11. Web Analysis with GeoDjango

Introduction

Setting up a GeoDjango web application

Getting ready

How to do it...

Creating an indoor web routing service

Getting ready

How to do it...

How it works...

Visualizing an indoor routing service

Getting ready

How to do it...

How it works...

Creating an indoor route-type service

Getting ready

How to do it...

How it works...

Creating an indoor route from room to room

Getting ready

How to do it...

How it works...

A. Other Geospatial Python Libraries

B. Mapping Icon Libraries

Index

Python Geospatial Analysis Cookbook

Python Geospatial Analysis Cookbook

Copyright © 2015 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

First published: November 2015

Production reference: 1251115

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham B3 2PB, UK.

ISBN 978-1-78355-507-9

www.packtpub.com

Credits

Author

Michael Diener

Reviewers

Jáchym Čepický

Richard Marsden

Commissioning Editor

Ashwin Nair

Acquisition Editor

Rebecca Youé

Content Development Editor

Athira Laji

Technical Editor

Chinmay S. Puranik

Copy Editor

Sonia Michelle Cheema

Project Coordinator

Bijal Patel

Proofreader

Safis Editing

Indexer

Hemangini Bari

Graphics

Disha Haria

Production Coordinator

Nilesh Mohite

Cover Work

Nilesh Mohite

About the Author

Michael Diener graduated from Simon Fraser University, British Columbia, Canada, in 2001 with a bachelor of science degree in geography. He began working in 1995 with Environment Canada as a GIS (Geographic Information Systems) analyst and has continued to work with GIS technologies ever since.

In 2008, he founded a company called GOMOGI that is focused on building web and mobile GIS application with open source tools. In 2011, the focus changed to indoor wayfinding and navigation solutions and building the indrz platform that Michael had envisioned.

From time to time, Michael also holds seminars for organizations wanting to explore or discover the possibilities of how GIS can increase productivity and help better answer spatial questions. He is also the creative head of new product development in his company. His technical skills include working with Python to solve a wide range of spatial problems on a daily basis. Through the years, he has developed many spatial applications with Python, including indrz and golfgis, which are two of the products built by his company, GOMOGI.

He is also lecturer of GIS at the Alpen Adria University, Klagenfurt, where he enjoys teaching students the wonderful powers of GIS and explaining how to solve spatial problems with open source GIS and Python.

I would like to dedicate this book to my family, beginning with my loving wife, Silvia, who encouraged and supported me through the writing of this book. I must also thank my three boys, Noah, Levi, and Jasper, for sacrificing many evenings and weekends of playtime for daddy’s book time. My parents, Birte and Hartwig, also deserve a big thank you for always believing in me—thank you for everything.

About the Reviewers

Jáchym Čepický is an open source GIS software consultant, developer, and user.

Richard Marsden has over 15 years of professional software development experience. After starting in the fields of geophysics and oil exploration, he has spent the last 10 years running the Winwaed Software Technology LLC independent software vendor. Winwaed specializes in geospatial tools and applications, including web applications, and operates the http://www.mapping-tools.com website for tools and add-ins for geospatial products such as Caliper Maptitude and Microsoft MapPoint.

Richard has been a technical reviewer for Python Geospatial Development and Python Geospatial Analysis Essentials both by Erik Westra, and Mastering Python Forensics by Dr. Michael Spreitzenbarth and Dr. Johann Uhrmann all by Packt Publishing.

www.PacktPub.com

Support files, eBooks, discount offers, and more

For support files and downloads related to your book, please visit www.PacktPub.com.

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at for more details.

At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks.

https://www2.packtpub.com/books/subscription/packtlib

Do you need instant solutions to your IT questions? PacktLib is Packt’s online digital book library. Here, you can search, access, and read Packt’s entire library of books.

Why Subscribe?

Fully searchable across every book published by Packt

Copy and paste, print, and bookmark content

On demand and accessible via a web browser

Free Access for Packt account holders

If you have an account with Packt at www.PacktPub.com, you can use this to access PacktLib today and view 9 entirely free books. Simply use your login credentials for immediate access.

Preface

Geospatial analysis is not special; it is just different when compared to other types of analysis such as financial market analysis. We work with geometry objects, such as lines, points, and polygons, and connect these geometries to attributes such as business data. We ask where question, such as Where is the nearest pub?, Where are all my customers located?, and Where is my competition located?. The other location questions include, Will this new building cast a shadow over the park?, What is the shortest way to school?, What is the safest way to school for my kids?, Will this building block my view of the mountains?, and Where is the optimal place to build my next store?. Identify the areas that fire trucks can reach from their station in 5 min, 10 min, or 20 min, and so on.

One thing all these questions have in common is the fact that you need to know where certain objects are located in order to answer them. Without the spatial component, you cannot answer such questions and this is what geospatial analysis is all about.

Geospatial features are laid over each other and patterns or trends are easily identified. This ability to see a pattern or trend is geospatial analysis in its simplest form.

Throughout this book, simple and complex code recipes are provided as small working models that can easily be integrated or expanded into a larger project or model.

Analysis is the fun part of GIS, and involves visualizing relationships, identifying trends, and seeing patterns that are not visible in a spreadsheet.

The Python programming language is clean, clear, and concise, making it great for beginners. It also has advanced powers for professionals to help them quickly code solutions to complex problems. Python makes visualization quick and easy for experts or beginners who work with geospatial data. It's that simple.

What this book covers

Chapter 1, Setting Up Your Geospatial Python Environment, explores setting up your computer to handle all software requirements in one go, such as pyproj, NumPy, and Shapely. All your development software needs to enable spatial analysis or geoprocessing on Windows and Linux are met in this chapter.

Chapter 2, Working with Projections, explains how to deal with spatial data that's projected or unprojected. You can learn and discover how to transform your data into a correct projection to prepare for an analysis.

Chapter 3, Moving Spatial Data from One Format to Another, explains how geospatial data comes in many different formats and also how messaging data from one format to another is a daily chore. In this chapter, you will find out about the most common data management tasks.

Chapter 4, Working with PostGIS, shows you how most of our geospatial data is stored in a spatial database and using, accessing, manipulating this data with Python is what this chapter is about.

Chapter 5, Vector Analysis, introduces a very common geospatial data format, that is, the vector data format. To execute analysis functions on vector data, we will explore patterns used to create new data by snapping, clipping, cutting, and overlaying vector datasets followed by determining the 3D ground distance and total elevation gain.

Chapter 6, Overlay Analysis, explains how to combine spatial data to create new data by using the process of overlaying two sets of data over each other.

Chapter 7, Raster Analysis, shows you how to create an elevation profile and quick ways to merge images to perform raster analysis functions on your data.

Chapter 8, Network Routing Analysis, shows you how finding the nearest anything is a common geospatial analysis feature. This chapter will disclose how to go about solving an indoor network type problem and demonstrate some common use cases for wayfinding inside buildings.

Chapter 9, Topology Checking and Data Validation, covers data quality and connections. In this chapter, you will learn how to verify your data for errors using custom topological functions.

Chapter 10, Visualizing Your Analysis, explains how geospatial data is inherently visual and you will learn about presenting your analysis on a web map and a 3D web.

Chapter 11, Web Analysis with GeoDjango, builds on Chapter 8, Network Routing Analysis, where you will create an indoor routing web application. You will easily be able to route a person from point A to point B within a building with real 3D network data. These key features will be presented by bringing together all the parts of the recipes you have learned so far.

Appendix A, Other Geospatial Python Libraries, explains how Python flourishes with geospatial libraries, and you will also find a listing of many popular libraries that are used for data analysis, regardless of whether they're spatial or not. This may trigger your interest.

Appendix B, Mapping Icon Libraries, quickly goes over the icon libraries out there that play a special role in the python geospatial working environment.

What you need for this book

To work with this book, you should be familiar with the programming language Python and the concepts involved in programming. This means that you should be able to install Python 2.7.x on your machine (Windows, Linux, or OS X) if it's not already installed. The concepts related to GIS (Geographic Information Systems) are definitely helpful but not necessary. A primer to this book could be Learning Geospatial Analysis with Python, Joel Lawhead or Python Geospatial Development, Eric Westra, both by Packt Publishing.

Who this book is for

If you are a student, teacher, programmer, geospatial or IT administrator, GIS analyst, researcher, or scientist looking to learn about spatial analysis, then this book is for you. Anyone trying to answer simple to complex spatial analysis questions will get a working demonstration of the power of Python with the help of real-world data. Some of you may be beginners but most of you will probably have a basic understanding of geospatial analysis and programming.

Sections

In this book, you will find several headings that appear frequently (Getting ready, How to do it..., How it works..., There's more..., and See also).

To give clear instructions on how to complete a recipe, we use these sections as follows:

Getting ready

This section tells you what to expect in the recipe, and describes how to set up any software or any preliminary settings required for the recipe.

How to do it…

This section contains the steps required to follow the recipe.

How it works…

This section usually consists of a detailed explanation of what happened in the previous section.

There's more…

This section consists of additional information about the recipe in order to make the reader more knowledgeable about the recipe.

See also

This section provides helpful links to other useful information for the recipe.

Conventions

In this book, you will find a number of styles of text that distinguish between different kinds of information. Here are some examples of these styles, and an explanation of their meaning.

Code words in text are shown as follows: If workon for some reason does not start your virtual environment, you can start it simply by executing source /home/mdiener/.venvs/pygeoan_cb/bin/activate from the command line.

A block of code is set as follows:

#!/usr/bin/env python

# -*- coding: utf-8 -*-

from osgeo import ogr

shp_driver = ogr.GetDriverByName('ESRI Shapefile')

shp_dataset = shp_driver.Open(r'../geodata/schools.shp')

shp_layer = shp_dataset.GetLayer()

shp_srs = shp_layer.GetSpatialRef()

print shp_srs

Any command-line input or output is written as follows:

$ sudo apt-get install python-setuptools python-pip

New terms and important words are shown in bold. Words that you see on the screen, in menus or dialog boxes for example, appear in the text like this: Select Route To: and enter 2 to see the second floor options.

Note

Warnings or important notes appear in a box like this.

Tip

Tips and tricks appear like this.

Reader feedback

Feedback from our readers is always welcome. Let us know what you think about this book—what you liked or may have disliked. Reader feedback is important for us to develop titles that you really get the most out of.

To send us general feedback, simply send an e-mail to <[email protected]>, and mention the book title through the subject of your message.

If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, see our author guide on www.packtpub.com/authors.

Customer support

Now that you are the proud owner of a Packt book, we have a number of things to help you to get the most from your purchase.

Downloading the example code

You can download the example code files for all Packt books you have purchased from your account at http://www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.

Downloading the color images of this book

We also provide you with a PDF file that has color images of the screenshots/diagrams used in this book. The color images will help you better understand the changes in the output. You can download this file from https://www.packtpub.com/sites/default/files/downloads/5079OS_ColorImage.pdf.

Errata

Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you find a mistake in one of our books—maybe a mistake in the text or the code—we would be grateful if you would report this to us. By doing so, you can save other readers from frustration and help us improve subsequent versions of this book. If you find any errata, please report them by visiting http://www.packtpub.com/support, selecting your book, clicking on the errata submission form link, and entering the details of your errata. Once your errata are verified, your submission will be accepted and the errata will be uploaded to our website, or added to any list of existing errata, under the Errata section of that title.

Piracy

Piracy of copyright material on the Internet is an ongoing problem across all media. At Packt, we take the protection of our copyright and licenses very seriously. If you come across any illegal copies of our works, in any form, on the Internet, please provide us with the location address or website name immediately so that we can pursue a remedy.

Please contact us at <[email protected]> with a link to the suspected pirated material.

We appreciate your help in protecting