Notebooks/01a_Spatial_SQLite.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction to Databases\n",
    "\n",
    "### Using Spatial Extensions - SQLite\n",
    "\n",
    "Based in [this](https://geoalchemy-2.readthedocs.io/en/latest/orm_tutorial.html#orm-tutorial) and [this](https://geoalchemy-2.readthedocs.io/en/latest/spatialite_tutorial.html) tutorials"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://ubuntu.pkgs.org/18.04/ubuntu-universe-amd64/libsqlite3-mod-spatialite_4.3.0a-5build1_amd64.deb.html  \n",
    "! sudo apt-get install libsqlite3-mod-spatialite  \n",
    "! pip install -U sqlalchemy geoalchemy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Connect to the DB\n",
    "\n",
    "Just like when using PostGIS connecting to a SpatiaLite database requires an Engine. This is how you create one for SpatiaLite:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sqlalchemy\n",
    "from sqlalchemy import create_engine, Column, Integer, String\n",
    "from sqlalchemy.event import listen\n",
    "from sqlalchemy.sql import select, func\n",
    "from sqlalchemy.ext.declarative import declarative_base\n",
    "from sqlalchemy.orm import sessionmaker\n",
    "from sqlalchemy.orm import relationship, backref\n",
    "\n",
    "from geoalchemy2 import Geometry, WKTElement"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_spatialite(dbapi_conn, connection_record):\n",
    "    dbapi_conn.enable_load_extension(True)\n",
    "    dbapi_conn.load_extension('/usr/lib/x86_64-linux-gnu/mod_spatialite.so')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#engine = create_engine('postgresql://gis:gis@localhost/gis', echo=True)\n",
    "engine = create_engine('sqlite:///../SampleDBs/gis.sqlite', echo=True)\n",
    "listen(engine, 'connect', load_spatialite)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The call to create_engine creates an engine bound to the database file gis.db. After that a connect listener is registered on the engine. The listener is responsible for loading the SpatiaLite extension, which is a necessary operation for using SpatiaLite through SQL.\n",
    "\n",
    "At this point you can test that you are able to connect to the database:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-11-17 12:23:01,113 INFO sqlalchemy.engine.base.Engine SELECT CAST('test plain returns' AS VARCHAR(60)) AS anon_1\n",
      "2020-11-17 12:23:01,115 INFO sqlalchemy.engine.base.Engine ()\n",
      "2020-11-17 12:23:01,118 INFO sqlalchemy.engine.base.Engine SELECT CAST('test unicode returns' AS VARCHAR(60)) AS anon_1\n",
      "2020-11-17 12:23:01,120 INFO sqlalchemy.engine.base.Engine ()\n"
     ]
    }
   ],
   "source": [
    "conn = engine.connect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One additional step is required for using SpatiaLite: create the geometry_columns and spatial_ref_sys metadata tables. This is done by calling SpatiaLite’s InitSpatialMetaData function:  \n",
    "Note that this operation may take some time the first time it is executed for a database. When InitSpatialMetaData is executed again it will report an error (that can be ignored)  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-11-17 12:23:02,630 INFO sqlalchemy.engine.base.Engine SELECT InitSpatialMetaData() AS \"InitSpatialMetaData_1\"\n",
      "2020-11-17 12:23:02,632 INFO sqlalchemy.engine.base.Engine ()\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<sqlalchemy.engine.result.ResultProxy at 0x7f810808ed00>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "conn.execute(select([func.InitSpatialMetaData()]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before going further we can close the current connection:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "conn.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Declare a Mapping\n",
    "\n",
    "Now that we have a working connection we can go ahead and create a mapping between a Python class and a database table.\n",
    "When using the ORM, the configurational process starts by describing the database tables we’ll be dealing with, and then by defining our own classes which will be mapped to those tables. In modern SQLAlchemy, these two tasks are usually performed together, using a system known as Declarative, which allows us to create classes that include directives to describe the actual database table they will be mapped to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "Base = declarative_base()\n",
    "\n",
    "class Lake(Base):\n",
    "    __tablename__ = 'lake'\n",
    "    id = Column(Integer, primary_key=True)\n",
    "    name = Column(String)\n",
    "    geom = Column(Geometry(geometry_type='POLYGON', management=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Lake class establishes details about the table being mapped, including the name of the table denoted by __tablename__, and three columns id, name, and geom. The id column will be the primary key of the table. The geom column is a geoalchemy2.types.Geometry column whose geometry_type is POLYGON.\n",
    "\n",
    "Setting management to True indicates that the AddGeometryColumn and DiscardGeometryColumn management functions will be used for the creation and removal of the geometry column. This is required with SpatiaLite.\n",
    "\n",
    "### Create the Table in the Database\n",
    "\n",
    "We can now create the lake table in the gis.sqlite database:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-11-17 12:24:54,915 INFO sqlalchemy.engine.base.Engine \n",
      "CREATE TABLE lake (\n",
      "\tid INTEGER NOT NULL, \n",
      "\tname VARCHAR, \n",
      "\tPRIMARY KEY (id)\n",
      ")\n",
      "\n",
      "\n",
      "2020-11-17 12:24:54,919 INFO sqlalchemy.engine.base.Engine ()\n",
      "2020-11-17 12:24:54,951 INFO sqlalchemy.engine.base.Engine COMMIT\n",
      "2020-11-17 12:24:54,955 INFO sqlalchemy.engine.base.Engine SELECT AddGeometryColumn(?, ?, ?, ?, ?) AS \"AddGeometryColumn_1\"\n",
      "2020-11-17 12:24:54,956 INFO sqlalchemy.engine.base.Engine ('lake', 'geom', -1, 'POLYGON', 2)\n",
      "2020-11-17 12:24:55,114 INFO sqlalchemy.engine.base.Engine COMMIT\n",
      "2020-11-17 12:24:55,116 INFO sqlalchemy.engine.base.Engine SELECT CreateSpatialIndex(?, ?) AS \"CreateSpatialIndex_1\"\n",
      "2020-11-17 12:24:55,117 INFO sqlalchemy.engine.base.Engine ('lake', 'geom')\n",
      "2020-11-17 12:24:55,291 INFO sqlalchemy.engine.base.Engine COMMIT\n"
     ]
    }
   ],
   "source": [
    "Lake.__table__.create(engine)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Table('lake', MetaData(bind=None), Column('id', Integer(), table=<lake>, primary_key=True, nullable=False), Column('name', String(), table=<lake>), Column('geom', Geometry(geometry_type='POLYGON', management=True, from_text='ST_GeomFromEWKT', name='geometry'), table=<lake>), schema=None)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Lake.__table__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we wanted to drop the table we’d use this. There’s nothing specific to SpatiaLite here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Lake.__table__.drop(engine)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a Session\n",
    "\n",
    "When using the SQLAlchemy ORM the ORM interacts with the database through a Session."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "Session = sessionmaker(bind=engine)\n",
    "session = Session()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add New Objects\n",
    "\n",
    "We can now create and insert new Lake objects into the database, the same way we’d do it using GeoAlchemy 2 with PostGIS."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-11-17 12:25:40,502 INFO sqlalchemy.engine.base.Engine BEGIN (implicit)\n",
      "2020-11-17 12:25:40,516 INFO sqlalchemy.engine.base.Engine INSERT INTO lake (name, geom) VALUES (?, GeomFromEWKT(?))\n",
      "2020-11-17 12:25:40,519 INFO sqlalchemy.engine.base.Engine ('Majeur', 'POLYGON((0 0,1 0,1 1,0 1,0 0))')\n",
      "2020-11-17 12:25:40,531 INFO sqlalchemy.engine.base.Engine COMMIT\n"
     ]
    }
   ],
   "source": [
    "lake = Lake(name='Majeur', geom='POLYGON((0 0,1 0,1 1,0 1,0 0))')\n",
    "session.add(lake)\n",
    "session.commit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now query the database for Majeur:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-11-17 12:26:04,741 INFO sqlalchemy.engine.base.Engine BEGIN (implicit)\n",
      "2020-11-17 12:26:04,744 INFO sqlalchemy.engine.base.Engine SELECT lake.id AS lake_id, lake.name AS lake_name, AsEWKB(lake.geom) AS lake_geom \n",
      "FROM lake \n",
      "WHERE lake.name = ?\n",
      " LIMIT ? OFFSET ?\n",
      "2020-11-17 12:26:04,748 INFO sqlalchemy.engine.base.Engine ('Majeur', 1, 0)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'Majeur'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "our_lake = session.query(Lake).filter_by(name='Majeur').first()\n",
    "our_lake.name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<WKBElement at 0x7f80f9ae2970; 0103000020FFFFFFFF010000000500000000000000000000000000000000000000000000000000F03F0000000000000000000000000000F03F000000000000F03F0000000000000000000000000000F03F00000000000000000000000000000000>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "our_lake.geom"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "our_lake.id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let’s add more lakes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-11-17 12:27:05,615 INFO sqlalchemy.engine.base.Engine INSERT INTO lake (name, geom) VALUES (?, GeomFromEWKT(?))\n",
      "2020-11-17 12:27:05,617 INFO sqlalchemy.engine.base.Engine ('Garde', 'POLYGON((1 0,3 0,3 2,1 2,1 0))')\n",
      "2020-11-17 12:27:05,620 INFO sqlalchemy.engine.base.Engine INSERT INTO lake (name, geom) VALUES (?, GeomFromEWKT(?))\n",
      "2020-11-17 12:27:05,625 INFO sqlalchemy.engine.base.Engine ('Orta', 'POLYGON((3 0,6 0,6 3,3 3,3 0))')\n",
      "2020-11-17 12:27:05,627 INFO sqlalchemy.engine.base.Engine COMMIT\n"
     ]
    }
   ],
   "source": [
    "session.add_all([Lake(name='Garde', geom='POLYGON((1 0,3 0,3 2,1 2,1 0))'),\n",
    "                 Lake(name='Orta', geom='POLYGON((3 0,6 0,6 3,3 3,3 0))')\n",
    "                ])\n",
    "session.commit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Query\n",
    "\n",
    "#### Let’s make a simple, non-spatial, query:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-11-17 12:30:59,928 INFO sqlalchemy.engine.base.Engine SELECT lake.id AS lake_id, lake.name AS lake_name, AsEWKB(lake.geom) AS lake_geom \n",
      "FROM lake ORDER BY lake.name\n",
      "2020-11-17 12:30:59,931 INFO sqlalchemy.engine.base.Engine ()\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['Garde', 'Majeur', 'Orta']"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "query = session.query(Lake).order_by(Lake.name)\n",
    "\n",
    "#for lake in query:\n",
    "#    print(lake.name)\n",
    "\n",
    "[l.name for l in query]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Now a spatial query:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-11-17 12:34:33,367 INFO sqlalchemy.engine.base.Engine SELECT lake.id AS lake_id, lake.name AS lake_name, AsEWKB(lake.geom) AS lake_geom \n",
      "FROM lake \n",
      "WHERE ST_Contains(lake.geom, ST_GeomFromText(?, ?))\n",
      "2020-11-17 12:34:33,370 INFO sqlalchemy.engine.base.Engine ('POINT(4 1)', -1)\n",
      "Orta\n"
     ]
    }
   ],
   "source": [
    "query = session.query(Lake).filter(func.ST_Contains(Lake.geom, WKTElement('POINT(4 1)')))\n",
    "\n",
    "for lake in query:\n",
    "    print(lake.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Altenatively: Here the ST_Contains function is applied to the Lake.geom column property. In that case the column property is actually passed to the function, as its first argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-11-17 12:36:30,312 INFO sqlalchemy.engine.base.Engine SELECT lake.id AS lake_id, lake.name AS lake_name, AsEWKB(lake.geom) AS lake_geom \n",
      "FROM lake \n",
      "WHERE ST_Contains(lake.geom, ?)\n",
      "2020-11-17 12:36:30,314 INFO sqlalchemy.engine.base.Engine ('POINT(4 1)',)\n",
      "Majeur\n",
      "Garde\n",
      "Orta\n"
     ]
    }
   ],
   "source": [
    "query = session.query(Lake).filter(Lake.geom.ST_Contains('POINT(4 1)')) \n",
    "\n",
    "for lake in query:\n",
    "    print(lake.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here’s another spatial query, using ST_Intersects this time:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-11-17 12:36:47,827 INFO sqlalchemy.engine.base.Engine SELECT lake.id AS lake_id, lake.name AS lake_name, AsEWKB(lake.geom) AS lake_geom \n",
      "FROM lake \n",
      "WHERE ST_Intersects(lake.geom, ST_GeomFromText(?, ?))\n",
      "2020-11-17 12:36:47,828 INFO sqlalchemy.engine.base.Engine ('LINESTRING(2 1,4 1)', -1)\n",
      "Garde\n",
      "Orta\n"
     ]
    }
   ],
   "source": [
    "query = session.query(Lake).filter(Lake.geom.ST_Intersects(WKTElement('LINESTRING(2 1,4 1)')))\n",
    "for lake in query:\n",
    "    print(lake.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also apply relationship functions to geoalchemy2.elements.WKBElement. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-11-17 12:38:50,191 INFO sqlalchemy.engine.base.Engine SELECT lake.id AS lake_id, lake.name AS lake_name, AsEWKB(lake.geom) AS lake_geom \n",
      "FROM lake \n",
      "WHERE lake.name = ?\n",
      "2020-11-17 12:38:50,197 INFO sqlalchemy.engine.base.Engine ('Garde',)\n",
      "2020-11-17 12:38:50,201 INFO sqlalchemy.engine.base.Engine SELECT ST_Intersects(GeomFromEWKB(?), ST_GeomFromText(?, ?)) AS \"ST_Intersects_1\"\n",
      "2020-11-17 12:38:50,203 INFO sqlalchemy.engine.base.Engine ('0103000020FFFFFFFF0100000005000000000000000000F03F00000000000000000000000000000840000000000000000000000000000008400000000000000040000000000000F03F0000000000000040000000000000F03F0000000000000000', 'LINESTRING(2 1,4 1)', -1)\n",
      "1\n"
     ]
    }
   ],
   "source": [
    "lake = session.query(Lake).filter_by(name='Garde').one()\n",
    "print(session.scalar(lake.geom.ST_Intersects(WKTElement('LINESTRING(2 1,4 1)'))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "session.scalar allows executing a clause and returning a scalar value (an integer value in this case).\n",
    "\n",
    "The value 1 indicates that the lake “Garde” does intersects the LINESTRING(2 1,4 1) geometry  \n",
    "\n",
    "The GeoAlchemy functions all start with ST_. Operators are also called as functions, but the function names don’t include the ST_ prefix.  \n",
    "As an example let’s test whether the bounding boxes of geometries intersect. GeoAlchemy provides the intersects function for that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}