Notebooks/06_Spatial_SQLite.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction to Databases\n",
    "\n",
    "### Using Spatial Extensions - SQLite\n",
    "\n",
    "Based in [this](https://geoalchemy-2.readthedocs.io/en/latest/orm_tutorial.html#orm-tutorial) and [this](https://geoalchemy-2.readthedocs.io/en/latest/spatialite_tutorial.html) tutorials"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://ubuntu.pkgs.org/18.04/ubuntu-universe-amd64/libsqlite3-mod-spatialite_4.3.0a-5build1_amd64.deb.html  \n",
    "https://zoomadmin.com/HowToInstall/UbuntuPackage/spatialite-bin  \n",
    "\n",
    "! sudo apt-get install libsqlite3-mod-spatialite spatialite-bin  \n",
    "! pip install -U sqlalchemy geoalchemy geoalchemy2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Connect to the DB\n",
    "\n",
    "Just like when using PostGIS connecting to a SpatiaLite database requires an Engine. This is how you create one for SpatiaLite:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sqlalchemy\n",
    "from sqlalchemy import create_engine, Column, Integer, String\n",
    "from sqlalchemy.event import listen\n",
    "from sqlalchemy.sql import select, func\n",
    "from sqlalchemy.ext.declarative import declarative_base\n",
    "from sqlalchemy.orm import sessionmaker\n",
    "from sqlalchemy.orm import relationship, backref\n",
    "\n",
    "from geoalchemy2 import Geometry, WKTElement"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_spatialite(dbapi_conn, connection_record):\n",
    "    dbapi_conn.enable_load_extension(True)\n",
    "    dbapi_conn.load_extension('/usr/lib/x86_64-linux-gnu/mod_spatialite.so')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "file in use or not found\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    os.remove('../SampleDBs/gis.sqlite')\n",
    "    print(\"removed file\")\n",
    "except:\n",
    "    print(\"file in use or not found\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "#engine = create_engine('postgresql://gis:gis@localhost/gis', echo=True)\n",
    "engine = create_engine('sqlite:///../SampleDBs/gis.sqlite', echo=True)\n",
    "listen(engine, 'connect', load_spatialite)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The call to create_engine creates an engine bound to the database file gis.db. After that a connect listener is registered on the engine. The listener is responsible for loading the SpatiaLite extension, which is a necessary operation for using SpatiaLite through SQL.\n",
    "\n",
    "At this point you can test that you are able to connect to the database:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "conn = engine.connect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I would prefer the items are paid when collected. I do not want to have the money without the itens being taken away. Sorry.One additional step is required for using SpatiaLite: create the geometry_columns and spatial_ref_sys metadata tables. This is done by calling SpatiaLite’s InitSpatialMetaData function:  \n",
    "Note that this operation may take some time the first time it is executed for a database. When InitSpatialMetaData is executed again it will report an error (that can be ignored)  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-10-28 17:26:29,566 INFO sqlalchemy.engine.Engine SELECT InitSpatialMetaData() AS \"InitSpatialMetaData_1\"\n",
      "2021-10-28 17:26:29,567 INFO sqlalchemy.engine.Engine [generated in 0.00104s] ()\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<sqlalchemy.engine.cursor.LegacyCursorResult at 0x7f78d82e1490>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "conn.execute(select([func.InitSpatialMetaData()]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before going further we can close the current connection:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "conn.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Declare a Mapping\n",
    "\n",
    "Now that we have a working connection we can go ahead and create a mapping between a Python class and a database table.\n",
    "When using the ORM, the configurational process starts by describing the database tables we’ll be dealing with, and then by defining our own classes which will be mapped to those tables. In modern SQLAlchemy, these two tasks are usually performed together, using a system known as Declarative, which allows us to create classes that include directives to describe the actual database table they will be mapped to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "Base = declarative_base()\n",
    "\n",
    "class Lake(Base):\n",
    "    __tablename__ = 'lake'\n",
    "    id = Column(Integer, primary_key=True)\n",
    "    name = Column(String)\n",
    "    geom = Column(Geometry(geometry_type='POLYGON', management=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Lake class establishes details about the table being mapped, including the name of the table denoted by __tablename__, and three columns id, name, and geom. The id column will be the primary key of the table. The geom column is a geoalchemy2.types.Geometry column whose geometry_type is POLYGON.\n",
    "\n",
    "Setting management to True indicates that the AddGeometryColumn and DiscardGeometryColumn management functions will be used for the creation and removal of the geometry column. This is required with SpatiaLite.\n",
    "\n",
    "### Create the Table in the Database\n",
    "\n",
    "We can now create the lake table in the gis.sqlite database:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-10-28 18:47:32,768 INFO sqlalchemy.engine.Engine BEGIN (implicit)\n",
      "2021-10-28 18:47:32,771 INFO sqlalchemy.engine.Engine \n",
      "CREATE TABLE lake (\n",
      "\tid INTEGER NOT NULL, \n",
      "\tname VARCHAR, \n",
      "\tPRIMARY KEY (id)\n",
      ")\n",
      "\n",
      "\n",
      "2021-10-28 18:47:32,773 INFO sqlalchemy.engine.Engine [no key 0.00249s] ()\n",
      "2021-10-28 18:47:33,030 INFO sqlalchemy.engine.Engine SELECT AddGeometryColumn(?, ?, ?, ?, ?, ?) AS \"AddGeometryColumn_1\"\n",
      "2021-10-28 18:47:33,032 INFO sqlalchemy.engine.Engine [no key 0.00170s] ('lake', 'geom', -1, 'POLYGON', 2, 0)\n",
      "2021-10-28 18:47:35,115 INFO sqlalchemy.engine.Engine SELECT CreateSpatialIndex(?, ?) AS \"CreateSpatialIndex_1\"\n",
      "2021-10-28 18:47:35,117 INFO sqlalchemy.engine.Engine [generated in 0.00165s] ('lake', 'geom')\n",
      "2021-10-28 18:47:41,951 INFO sqlalchemy.engine.Engine COMMIT\n"
     ]
    }
   ],
   "source": [
    "Lake.__table__.create(engine)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Table('lake', MetaData(), Column('id', Integer(), table=<lake>, primary_key=True, nullable=False), Column('name', String(), table=<lake>), Column('geom', Geometry(geometry_type='POLYGON', management=True, from_text='ST_GeomFromEWKT', name='geometry'), table=<lake>), schema=None)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Lake.__table__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we wanted to drop the table we’d use this. There’s nothing specific to SpatiaLite here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Lake.__table__.drop(engine)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a Session\n",
    "\n",
    "When using the SQLAlchemy ORM the ORM interacts with the database through a Session."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "Session = sessionmaker(bind=engine)\n",
    "session = Session()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add New Objects\n",
    "\n",
    "We can now create and insert new Lake objects into the database, the same way we’d do it using GeoAlchemy 2 with PostGIS."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-10-28 18:47:43,751 INFO sqlalchemy.engine.Engine BEGIN (implicit)\n",
      "2021-10-28 18:47:43,758 INFO sqlalchemy.engine.Engine INSERT INTO lake (name, geom) VALUES (?, GeomFromEWKT(?))\n",
      "2021-10-28 18:47:43,759 INFO sqlalchemy.engine.Engine [generated in 0.00154s] ('Majeur', 'POLYGON((0 0,1 0,1 1,0 1,0 0))')\n",
      "2021-10-28 18:47:43,764 INFO sqlalchemy.engine.Engine COMMIT\n"
     ]
    }
   ],
   "source": [
    "lake = Lake(name='Majeur', geom='POLYGON((0 0,1 0,1 1,0 1,0 0))')\n",
    "session.add(lake)\n",
    "session.commit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now query the database for Majeur:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-10-28 18:47:44,633 INFO sqlalchemy.engine.Engine BEGIN (implicit)\n",
      "2021-10-28 18:47:44,635 INFO sqlalchemy.engine.Engine SELECT lake.id AS lake_id, lake.name AS lake_name, AsEWKB(lake.geom) AS lake_geom \n",
      "FROM lake \n",
      "WHERE lake.name = ?\n",
      " LIMIT ? OFFSET ?\n",
      "2021-10-28 18:47:44,636 INFO sqlalchemy.engine.Engine [generated in 0.00047s] ('Majeur', 1, 0)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'Majeur'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "our_lake = session.query(Lake).filter_by(name='Majeur').first()\n",
    "our_lake.name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<WKBElement at 0x7f786da1b340; 0103000020FFFFFFFF010000000500000000000000000000000000000000000000000000000000F03F0000000000000000000000000000F03F000000000000F03F0000000000000000000000000000F03F00000000000000000000000000000000>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "our_lake.geom"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "our_lake.geom is a geoalchemy2.elements.WKBElement, which a type provided by GeoAlchemy. geoalchemy2.elements.  \n",
    "WKBElement wraps a WKB value returned by the database."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "our_lake.id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let’s add more lakes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-10-28 18:47:45,550 INFO sqlalchemy.engine.Engine INSERT INTO lake (name, geom) VALUES (?, GeomFromEWKT(?))\n",
      "2021-10-28 18:47:45,552 INFO sqlalchemy.engine.Engine [cached since 1.794s ago] ('Garde', 'POLYGON((1 0,3 0,3 2,1 2,1 0))')\n",
      "2021-10-28 18:47:45,555 INFO sqlalchemy.engine.Engine INSERT INTO lake (name, geom) VALUES (?, GeomFromEWKT(?))\n",
      "2021-10-28 18:47:45,557 INFO sqlalchemy.engine.Engine [cached since 1.8s ago] ('Orta', 'POLYGON((3 0,6 0,6 3,3 3,3 0))')\n",
      "2021-10-28 18:47:45,560 INFO sqlalchemy.engine.Engine COMMIT\n"
     ]
    }
   ],
   "source": [
    "session.add_all([Lake(name='Garde', geom='POLYGON((1 0,3 0,3 2,1 2,1 0))'),\n",
    "                 Lake(name='Orta', geom='POLYGON((3 0,6 0,6 3,3 3,3 0))')\n",
    "                ])\n",
    "session.commit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Query\n",
    "\n",
    "#### Let’s make a simple, non-spatial, query:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-10-28 18:47:46,295 INFO sqlalchemy.engine.Engine BEGIN (implicit)\n",
      "2021-10-28 18:47:46,297 INFO sqlalchemy.engine.Engine SELECT lake.id AS lake_id, lake.name AS lake_name, AsEWKB(lake.geom) AS lake_geom \n",
      "FROM lake ORDER BY lake.name\n",
      "2021-10-28 18:47:46,297 INFO sqlalchemy.engine.Engine [generated in 0.00060s] ()\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['Garde', 'Majeur', 'Orta']"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "query = session.query(Lake).order_by(Lake.name)\n",
    "\n",
    "#for lake in query:\n",
    "#    print(lake.name)\n",
    "\n",
    "[l.name for l in query]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Now a spatial query:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-10-28 18:47:46,601 INFO sqlalchemy.engine.Engine SELECT lake.id AS lake_id, lake.name AS lake_name, AsEWKB(lake.geom) AS lake_geom \n",
      "FROM lake \n",
      "WHERE ST_Contains(lake.geom, ?)\n",
      "2021-10-28 18:47:46,603 INFO sqlalchemy.engine.Engine [no key 0.00184s] ('POINT(4 1)',)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['Majeur', 'Garde', 'Orta']"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "query = session.query(Lake).filter(func.ST_Contains(Lake.geom, 'POINT(4 1)'))\n",
    "\n",
    "[l.name for l in query]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Altenatively: Here the ST_Contains function is applied to the Lake.geom column property. In that case the column property is actually passed to the function, as its first argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-10-28 18:47:47,511 INFO sqlalchemy.engine.Engine SELECT lake.id AS lake_id, lake.name AS lake_name, AsEWKB(lake.geom) AS lake_geom \n",
      "FROM lake \n",
      "WHERE ST_Contains(lake.geom, ?)\n",
      "2021-10-28 18:47:47,512 INFO sqlalchemy.engine.Engine [no key 0.00089s] ('POINT(4 1)',)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['Majeur', 'Garde', 'Orta']"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "query = session.query(Lake).filter(Lake.geom.ST_Contains('POINT(4 1)')) \n",
    "\n",
    "[l.name for l in query]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here’s another spatial query, using ST_Intersects this time:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-10-28 18:47:48,737 INFO sqlalchemy.engine.Engine SELECT lake.id AS lake_id, lake.name AS lake_name, AsEWKB(lake.geom) AS lake_geom \n",
      "FROM lake \n",
      "WHERE ST_Intersects(lake.geom, ST_GeomFromText(?, ?))\n",
      "2021-10-28 18:47:48,738 INFO sqlalchemy.engine.Engine [no key 0.00193s] ('LINESTRING(2 1,4 1)', -1)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['Garde', 'Orta']"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "query = session.query(Lake).filter(Lake.geom.ST_Intersects(WKTElement('LINESTRING(2 1,4 1)')))\n",
    "\n",
    "[l.name for l in query]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also apply relationship functions to geoalchemy2.elements.WKBElement. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-10-28 18:47:49,511 INFO sqlalchemy.engine.Engine SELECT lake.id AS lake_id, lake.name AS lake_name, AsEWKB(lake.geom) AS lake_geom \n",
      "FROM lake \n",
      "WHERE lake.name = ?\n",
      "2021-10-28 18:47:49,513 INFO sqlalchemy.engine.Engine [generated in 0.00188s] ('Garde',)\n",
      "2021-10-28 18:47:49,518 INFO sqlalchemy.engine.Engine SELECT ST_Intersects(GeomFromEWKB(?), ST_GeomFromText(?, ?)) AS \"ST_Intersects_1\"\n",
      "2021-10-28 18:47:49,519 INFO sqlalchemy.engine.Engine [no key 0.00065s] ('0103000020FFFFFFFF0100000005000000000000000000F03F00000000000000000000000000000840000000000000000000000000000008400000000000000040000000000000F03F0000000000000040000000000000F03F0000000000000000', 'LINESTRING(2 1,4 1)', -1)\n",
      "1\n"
     ]
    }
   ],
   "source": [
    "lake = session.query(Lake).filter_by(name='Garde').one()\n",
    "print(session.scalar(lake.geom.ST_Intersects(WKTElement('LINESTRING(2 1,4 1)'))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "session.scalar allows executing a clause and returning a scalar value (an integer value in this case).\n",
    "\n",
    "The value 1 indicates that the lake “Garde” does intersects the LINESTRING(2 1,4 1) geometry  \n",
    "\n",
    "The GeoAlchemy functions all start with ST_. Operators are also called as functions, but the function names don’t include the ST_ prefix.  \n",
    "As an example let’s test whether the bounding boxes of geometries intersect. GeoAlchemy provides the intersects function for that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "file in use or not found\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    os.remove('../SampleDBs/gis.sqlite')\n",
    "    print(\"removed file\")\n",
    "except:\n",
    "    print(\"file in use or not found\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}