Foss4g Python
Foss4g Python
Geospatial
gisadvisor.com, LLC.
What we’ll cover
PART 1: LOADING SOFTWARE AND DATA
PART 2: OVERVIEW OF PYTHON
PART 3: PYTHON PACKAGES
PART 4: GEOSPATIAL ADDITIONS TO PYTHON
gisadvisor.com, LLC.
Course Goals
At the end of this course,
you will:
Understand what Python is.
Learn how to use Python to write
scripts for automating geographic
data.
Learn how to use Python packages
to add additional functionality to
your Python code.
gisadvisor.com, LLC.
Part I: Loading our
software and data
gisadvisor.com, LLC.
Part II: An overview of
Python
gisadvisor.com, LLC.
Using Python for Geospatial
Basic Python
Spatial and database tools
QGIS
Arcpy
Special Python packages for
geospatial
Postgres/PostGIS
Microsoft Excel
Geocoding
gisadvisor.com, LLC.
What is Python
Python is an interpreted, interactive, object-oriented
programming language. It incorporates modules,
exceptions, dynamic typing, very high level dynamic data
types, and classes. Python combines remarkable power
with very clear syntax. It has interfaces to many system
calls and libraries, as well as to various window systems,
and is extensible in C or C++. It is also usable as an
extension language for applications that need a
programmable interface. Finally, Python is portable: it runs
on many Unix variants, on the Mac, and on Windows 2000
A Tour
and later. –of Python
www.python.org
gisadvisor.com, LLC.
The many flavors of Python
Versions
2.6, 2.7, 2.8
3.0, 3.3, 3.6
Interpreters
IDLE
WinPy
Geany
gisadvisor.com, LLC.
The Python Language
Python as a calculator
Variables
Structures
Statements
Expressions
Methods and Functions
gisadvisor.com, LLC.
Starting with IDLE
gisadvisor.com, LLC.
Python as a calculator
Just
enter some
numbers
gisadvisor.com, LLC.
Variables and Data Types
A variable is a name for a value. The computer
stores the value in memory. Depending upon what
the variable type is, certain operations can be done
with the variable.
Some variable types include:
Strings
Numbers
Integer
Floating Point
Lists
name = 'Art Lembo'
age = '54'
children = ['Emily','Arthur','Katie']
print age
print name
print children[1]
gisadvisor.com, LLC.
Variables and methods
String
gisadvisor.com, LLC.
Lists
Python supports arrays in the form of lists.
These include a basic list:
mylist = ["art","bob","sue"]
mylist[0]
or, lists within lists:
mylist = [["art",52],["bob",19],["Sue",44]]
mylist[0]
mylist[0][0]
gisadvisor.com, LLC.
List
a = [66.25, 333, 333, 1, 1234.5]
a[1]
print a.reverse
a.insert(2,1.00)
a.remove(1.0)
a.pop(2)
sortlist = a; sortlist.sort(); sortlist
gisadvisor.com, LLC.
Dictionaries
A dictionary is a collection which is unordered, changeable and
indexed. In Python dictionaries are written with curly brackets, and
they have keys and values.
print dict['Name']
gisadvisor.com, LLC.
states = {
'Oregon': 'OR',
'Florida': 'FL',
'California': 'CA',
'New York': 'NY',
'Michigan': 'MI'
}
cities = {
'CA': 'San Francisco',
'MI': 'Detroit',
'FL': 'Jacksonville'
}
gisadvisor.com, LLC.
Methods and Functions
f = open('c:/training/python/addresses.txt')
for line in f:
print line
Files
Read a file
Write a file f = open('c:/training/python/addresses.txt')
o = open('c:/training/python/addout.txt','w')
for line in f:
o.write(line)
o.close()
gisadvisor.com, LLC.
Quiz
gisadvisor.com, LLC.
Quiz
How many letters are in
supercalifragilisticexpialidocious?
What is the 9th letter in the word?
Use split to create a list of the Beatles:
‘John’,’George’,’Paul’,’Ringo’
Create two variables:
Firstname = ‘John’
Lastname = ‘Lennon’
Append the two names into a new
variable called FullName
gisadvisor.com, LLC.
Python as a module
Add code array = [12, 9, 17, 16]
Save sumtotal = 0
for i in array:
Run the module sumtotal = i +
sumtotal
print sumtotal
print sum(array)
Quiz
Instead of the sum, calculate the sum of the
squares (use the pow function)
Print out the sum of the squares and the
average sum of the squares
gisadvisor.com, LLC.
E
C
T
P
rr
o
ro
e d
o
g
re
r
s a
C
W
m
t
o
rr
rD
ie
i e
t
c
s
n ti
i
in
g
g o
g
n
n
L
o
g
i
c
C
o
r
r
e
c
t
i
o
n
gisadvisor.com, LLC.
Algorithm - a finite list of well-defined
instructions for accomplishing some
task
Algorithms are essential to the way computers process
information
Algorithms must be rigorously defined:
specified in the way it applies in all possible circumstances that could
arise.
conditional steps must be systematically dealt with, case-by-case;
the criteria for each case must be clear (and computable).
array = [2,4,6,3]
Have variable for
largest=0
-for largest
i in array:
if i>largest:
Look up in Help:
largest=i
print largest
- list
- for loop
- if..then..else
gisadvisor.com, LLC.
Statements and Control
if
while
for
gisadvisor.com, LLC.
Quiz - Temperature program
gisadvisor.com, LLC.
Python modules (packages)
A module is a file containing Python
definitions and statements.
import math
math.cos(34)
There are many, many “standard” modules
built in Python by default
There are many, many, many, MANY other
packages you can install
gisadvisor.com, LLC.
Default modules
gisadvisor.com, LLC.
SQLite Example (not spatial,
yet)
import sqlite3
conn = sqlite3.Connection('c:/training/python/tompkins.sqlite')
answer = conn.execute('SELECT * FROM parcels LIMIT 3')
for row in answer:
print row
Quiz
Findthe average ASMT value for
each Propclass in parcels
gisadvisor.com, LLC.
import os
os.chdir("c:/program files/qgis 3.8/bin")
import sqlite3
##pkey = input('enter your parcel key: ')
conn = sqlite3.Connection('c:/training/python/tompkins.sqlite')
conn.enable_load_extension(True)
conn.execute('SELECT load_extension("mod_spatialite")')
gisadvisor.com, LLC.
import os
os.chdir("c:/program files/qgis 3.8/bin")
import sqlite3
pkey = input('enter your parcel key: ')
conn = sqlite3.Connection('c:/training/python/tompkins.sqlite')
conn.enable_load_extension(True)
conn.execute('SELECT load_extension("c:/program files/QGIS
3.8/bin/mod_spatialite")')
answer = conn.execute('''SELECT
min(st_distance(parcels.geometry,floodzones.geom)) as dist
FROM parcels, floodzones
WHERE parcels.parcelkey = ''' + '\'' + pkey + '\'')
for row in answer:
if row[0] == 0.0:
print('parcel intersects the floodzone')
else:
print(row[0])
gisadvisor.com, LLC.
pip – the magic of Python
gisadvisor.com, LLC.
Creating an external function
for linear algebra
Pycall.py Ols.py
gisadvisor.com, LLC.
import psycopg2
PostGIS
conn = psycopg2.connect(dbname =
'tompkins',host='127.0.0.1',user='postgres',password='postgres',
port=5433)
cur = conn.cursor()
thesql = ('''SELECT tcparcel.parcelkey, zone
FROM floodzones, tcparcel
WHERE ST_Intersects(tcparcel.geom,floodzones.geom)''')
cur.execute(thesql)
theresult = cur.fetchall()
for j in theresult:
print ('Park ' + j[0] + 'Intersects parcel ' + j[1])
import psycopg2
conn = psycopg2.connect(dbname =
'tompkins',host='127.0.0.1',user='postgres',password='postg
res', port=5433)
cur = conn.cursor()
thesql = input('enter sql: ')
print (thesql)
cur.execute(thesql)
theresult = cur.fetchall()
for j in theresult:
gisadvisor.com, LLC. print(j)
Microsoft Excel
gisadvisor.com, LLC.
Quick and dirty Excel
pip install pypiwin32
import win32com.client
excel = win32com.client.Dispatch("Excel.Application")
mylist = [10,20,30,40,50,60,70,80,90]
myAvg= excel.WorksheetFunction.Average([mylist])
print(myAvg)
gisadvisor.com, LLC.
Advanced Excel
import psycopg2, win32com.client, numpy as np
conn = psycopg2.connect(dbname = 'tompkins',host='127.0.0.1',user='postgres',password='postgres', port='5433')
excel = win32com.client.Dispatch("Excel.Application")
cur = conn.cursor()
thesql = ('SELECT asmt FROM tcparcel')
cur.execute(thesql)
theresult = cur.fetchall()
# Here we will issue an Excel function to determine the standard deviation from the
# selected values
thestdev = excel.WorksheetFunction.StDev(theresult)
print ('The Standard Deviation is: ' + str(thestdev))
gisadvisor.com, LLC.
GEOCODING: GEOCODER, CENSUSGEOCODER,
GOOGLE
SPATIAL OPERATIONS: POSTGIS
stuff
gisadvisor.com, LLC.
gisadvisor.com, LLC.
Geocoding
from googleapi import googleapikey
requests.packages.urllib3.disable_warnings()
import geocoder
apikey = googleapikey
f = open('c:/training/python/addresses.csv','r')
googleout = open('c:/temp/googleoutaddress.csv','w')
googleout.write("\"address\",\"lon\",\"lat\",\"matchcode\" \n")
for line in f:
gc = geocoder.google(line,key=apikey)
googleout.write(gc.address.replace(',','') + ' , ' + str(gc.lng) + ',' +
str(gc.lat) + ',' + gc.accuracy + '\n')
print (line," has an accuracy of ", gc.accuracy)
googleout.close()
f.close()
gisadvisor.com, LLC.
Geocoding
import censusgeocode
cg = censusgeocode.CensusGeocode()
census_gc = cg.onelineaddress('16 Bush Lane Ithaca NY')
Print( census_gc)
import censusgeocode
cg = censusgeocode.CensusGeocode()
f = open('c:/training/python/addresses.csv', 'r')
out = open('c:/temp/outaddress1.csv','w')
out.write('addr, longitude, latitude, accuracy '+'\n')
for line in f:
census_gc = cg.onelineaddress(line)
if len(census_gc) > 0:
out.write(str(line) + ',' + str(census_gc[0]['coordinates']['x']) + ',' +
str(census_gc[0]['coordinates']['y'])+'\n')
gisadvisor.com, LLC.
import psycopg2, geocoder
from googleapi import googleapikey
##requests.packages.urllib3.disable_warnings()
apikey = googleapikey
f = open('c:/training/python/addresses.csv','r')
for line in f:
gc = geocoder.google(line,key=apikey)
if gc.accuracy == 'ROOFTOP':
thesql = 'INSERT INTO newpoints(address,geom) VALUES (\'' + gc.location + '\',ST_SetSRID(ST_Point(' +
str(gc.lng) + ',' + str(gc.lat) + '),4326))'
cur.execute(thesql)
print(thesql)
conn.commit()
f.close()
gisadvisor.com, LLC.
Excel quiz
[16,22,30,45,58,69,77,88,99]
What is the Pearson correlation coefficient?
gisadvisor.com, LLC.
Arcpy Examples
gisadvisor.com, LLC.
Working with Arcpy in ArcGIS
Fire up ArcMap
Click Python window
Start typing code
Look up help files – they are
excellent!
gisadvisor.com, LLC.
Things you do most of the GIS Useage
time…
Load layers
Select layers by attributes
Select layers by some spatial
operation Load Layers Attribute Qry Spatial Qry Other actions
gisadvisor.com, LLC.
Buffer watershed and clip
parcels
import arcpy
arcpy.env.overwriteOutput = True
arcpy.env.workspace = "c:/training/python/tompkins.gdb"
wstemp = arcpy.MakeFeatureLayer_management("watersheds","wsclip","watershed = 'Cascadilla
Creek'")
##testvar = input("How far do you want the buffer in meters")
testvar = 500
wsbuffer = arcpy.Buffer_analysis(wstemp,"wsbuffer",testvar)
wsclip = arcpy.Clip_analysis("parcels","wsbuffer","parclip")
arcpy.Delete_management(wsbuffer)
arcpy.Delete_management(wsclip)
gisadvisor.com, LLC.
Total value of land in the AE
floodzone by land use
import arcpy
import numpy
arcpy.env.workspace = "c:/training/python/tompkins.gdb"
parceltemp = arcpy.MakeFeatureLayer_management("parcels")
floodtemp = arcpy.MakeFeatureLayer_management("floodzones","zone = 'AE'")
gisadvisor.com, LLC.
QGIS Examples
gisadvisor.com, LLC.
Working with Qpy in QGIS
Fire up QGIS
Click Python window
Start typing code
Look up help files – not so
excellent
gisadvisor.com, LLC.
Some QGIS documentation
https://
docs.qgis.org/testing/en/docs/user_manual/pro
cessing/console.html
gisadvisor.com, LLC.
Buffer watershed and clip
parcels
import qgis
fz = QgsProject.instance().mapLayersByName('floodzones')[0]
fz.selectByExpression('zone = \'AE\'',QgsVectorLayer.SetSelection)
parcels = QgsProject.instance().mapLayersByName('tcparcel')[0]
gisadvisor.com, LLC.
import qgis
selectedfz.featureCount()
selpar = processing.runAndLoadResults("native:extractbylocation",
{'INPUT':'parcels','PREDICATE':[0],'INTERSECT':"Matching features",'OUTPUT':'memory:'})
gisadvisor.com,
['OUTPUT'] LLC.
Total value of land in the AE
floodzone by land use
import numpy
sp = QgsProject.instance().mapLayersByName('Extracted (location)')[0]
selected_features = sp.selectAll()
features = sp.getFeatures()
myidx = features.fields().indexFromName('asmt')
myarray = []
for i in features:
myarray.append(i[19])
print(numpy.sum(myarray))
gisadvisor.com, LLC.
Total value of land in the AE
floodzone by land use
import qgis, numpy
floodzones = iface.addVectorLayer('c:/training/python/shapefiles/Floodzones.shp', "", "ogr")
floodzones.setName('floodz')
parcels = iface.addVectorLayer('c:/training/python/tompkins.gdb|layername=parcels', "parcels", "ogr")
selectedfz = processing.runAndLoadResults("native:extractbyexpression", {'INPUT':'floodz', 'EXPRESSION':'ZONE
= \'AE\'','OUTPUT':'memory:'})
selpar = processing.runAndLoadResults("native:extractbylocation",{'INPUT':'parcels','PREDICATE':
[0],'INTERSECT':"Matching features",'OUTPUT':'memory:'})['OUTPUT']
sp = QgsProject.instance().mapLayersByName('Extracted (location)')[0]
selected_features = sp.selectAll()
features = sp.getFeatures()
myidx = parcels.fields().indexFromName('asmt')
myarray = []
for i in features:
myarray.append(i[myidx])
print(numpy.sum(myarray))
gisadvisor.com, LLC.
gisadvisor.com, LLC.
Municipal Risk
Exploration of vector functions in Arcpy
For this exercise, you will explore the use of vector analysis to
determine at-risk properties in certain flood zones, and categorize the
data. Let’s assume our stakeholders are interested in:
• The total value of all land within the ‘AE’ flood zone.
gisadvisor.com, LLC.
Steps – in ArcGIS
import arcpy
myconn =
arcpy.CreateDatabaseConnection_management("c:/temp/","tompkins","POSTGRESQL","127.0.0
.1,5433","DATABASE_AUTH","postgres","postgres","SAVE_USERNAME","tompkins")
arcpy.SelectLayerByAttribute_management("tompkins.public.floodzones","NEW_SELECTION",
"zone='AE'")
arcpy.SelectLayerByLocation_management("tompkins.public.tcparcel","INTERSECT","tompki
ns.public.floodzones",0,"NEW_SELECTION")
arcpy.Statistics_analysis("tompkins.public.tcparcel","statout",
[["asmt","SUM"]],"propclass")
gisadvisor.com, LLC.
Steps – separate script
import arcpy
arcpy.env.overwriteOutput = True
if arcpy.Exists(r'c:\temp\tompkins.sde'):
arcpy.env.workspace = 'c:/temp/tompkins.sde'
myconn = arcpy.env.workspace
else:
myconn =
arcpy.CreateDatabaseConnection_management("c:/temp/","tompkins","POSTGRESQL","127.0.0.1,5433","DAT
ABASE_AUTH","postgres","postgres","SAVE_USERNAME","tompkins")
sdeconn = str(myconn)+"\\"
floodfeat = arcpy.MakeFeatureLayer_management(sdeconn+"tompkins.public.floodzones")
parcelfeat = arcpy.MakeFeatureLayer_management(sdeconn+"tompkins.public.tcparcel")
aeflood = arcpy.SelectLayerByAttribute_management(floodfeat,"NEW_SELECTION","zone='AE'")
parflood =
arcpy.SelectLayerByLocation_management(parcelfeat,"INTERSECT",aeflood,0,"NEW_SELECTION")
##arcpy.Statistics_analysis(parflood,"c:/temp/statoutpf/statoutpf",[["asmt","SUM"]],"propclass")
stattable = arcpy.Statistics_analysis(parflood,"c:/temp/statoutpf.dbf",
[["asmt","SUM"]],"propclass")
for row in arcpy.SearchCursor(stattable):
print row.propclass + " $" + str(row.SUM_asmt)
gisadvisor.com, LLC.
The total value of Residential properties
within the ‘X’ flood zone.
import arcpy
arcpy.env.workspace = r'C:\training\python\tompkins.gdb'
arcpy.MakeFeatureLayer_management('parcels', 'parcel_layer')
arcpy.MakeFeatureLayer_management('floodzones', 'firm_layer')
arcpy.SelectLayerByLocation_management('parcel_layer', 'intersect',
'firm_layer')
arcpy.SelectLayerByAttribute_management('parcel_layer','SUBSET_SELECTION',
"propclass = 'Residential'")
gisadvisor.com, LLC.
import googlemaps
gmaps = googlemaps.Client(xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx')
startAddress = "1440 East Sandy Acres Drive, Salisbury MD"
endAddresses = ["6 Briaroot Drive, Smithtown, NY","7 Hickory Road, Bayville NY",
"16 Bush Lane, Ithaca, NY"]
for endAddress in endAddresses:
directions = gmaps.directions(startAddress, endAddress)
print "time to " + endAddress + ": " + directions[0]['legs'][0]['duration']
['text']
gisadvisor.com, LLC.
import arcpy, math
fc = arcpy.GetParameterAsText(0)
gisadvisor.com, LLC.