Notebooks/01a_Spatial_SQLite.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction to Databases\n",
    "\n",
    "### Using Spatial Extensions - SQLite\n",
    "\n",
    "Based in [this](https://geoalchemy-2.readthedocs.io/en/latest/orm_tutorial.html#orm-tutorial) and [this](https://geoalchemy-2.readthedocs.io/en/latest/spatialite_tutorial.html) tutorials"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://ubuntu.pkgs.org/18.04/ubuntu-universe-amd64/libsqlite3-mod-spatialite_4.3.0a-5build1_amd64.deb.html  \n",
    "https://zoomadmin.com/HowToInstall/UbuntuPackage/spatialite-bin  \n",
    "\n",
    "! sudo apt-get install libsqlite3-mod-spatialite spatialite-bin  \n",
    "! pip install -U sqlalchemy geoalchemy geoalchemy2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Connect to the DB\n",
    "\n",
    "Just like when using PostGIS connecting to a SpatiaLite database requires an Engine. This is how you create one for SpatiaLite:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sqlalchemy\n",
    "from sqlalchemy import create_engine, Column, Integer, String\n",
    "from sqlalchemy.event import listen\n",
    "from sqlalchemy.sql import select, func\n",
    "from sqlalchemy.ext.declarative import declarative_base\n",
    "from sqlalchemy.orm import sessionmaker\n",
    "from sqlalchemy.orm import relationship, backref\n",
    "\n",
    "from geoalchemy2 import Geometry, WKTElement"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_spatialite(dbapi_conn, connection_record):\n",
    "    dbapi_conn.enable_load_extension(True)\n",
    "    dbapi_conn.load_extension('/usr/lib/x86_64-linux-gnu/mod_spatialite.so')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#engine = create_engine('postgresql://gis:gis@localhost/gis', echo=True)\n",
    "engine = create_engine('sqlite:///../SampleDBs/gis.sqlite', echo=True)\n",
    "listen(engine, 'connect', load_spatialite)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The call to create_engine creates an engine bound to the database file gis.db. After that a connect listener is registered on the engine. The listener is responsible for loading the SpatiaLite extension, which is a necessary operation for using SpatiaLite through SQL.\n",
    "\n",
    "At this point you can test that you are able to connect to the database:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "conn = engine.connect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One additional step is required for using SpatiaLite: create the geometry_columns and spatial_ref_sys metadata tables. This is done by calling SpatiaLite’s InitSpatialMetaData function:  \n",
    "Note that this operation may take some time the first time it is executed for a database. When InitSpatialMetaData is executed again it will report an error (that can be ignored)  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-09-30 14:15:35,114 INFO sqlalchemy.engine.Engine SELECT InitSpatialMetaData() AS \"InitSpatialMetaData_1\"\n",
      "2021-09-30 14:15:35,115 INFO sqlalchemy.engine.Engine [generated in 0.00168s] ()\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<sqlalchemy.engine.cursor.LegacyCursorResult at 0x7f118040efa0>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "conn.execute(select([func.InitSpatialMetaData()]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before going further we can close the current connection:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "conn.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Declare a Mapping\n",
    "\n",
    "Now that we have a working connection we can go ahead and create a mapping between a Python class and a database table.\n",
    "When using the ORM, the configurational process starts by describing the database tables we’ll be dealing with, and then by defining our own classes which will be mapped to those tables. In modern SQLAlchemy, these two tasks are usually performed together, using a system known as Declarative, which allows us to create classes that include directives to describe the actual database table they will be mapped to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "Base = declarative_base()\n",
    "\n",
    "class Lake(Base):\n",
    "    __tablename__ = 'lake'\n",
    "    id = Column(Integer, primary_key=True)\n",
    "    name = Column(String)\n",
    "    geom = Column(Geometry(geometry_type='POLYGON', management=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Lake class establishes details about the table being mapped, including the name of the table denoted by __tablename__, and three columns id, name, and geom. The id column will be the primary key of the table. The geom column is a geoalchemy2.types.Geometry column whose geometry_type is POLYGON.\n",
    "\n",
    "Setting management to True indicates that the AddGeometryColumn and DiscardGeometryColumn management functions will be used for the creation and removal of the geometry column. This is required with SpatiaLite.\n",
    "\n",
    "### Create the Table in the Database\n",
    "\n",
    "We can now create the lake table in the gis.sqlite database:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-09-30 15:49:34,640 INFO sqlalchemy.engine.Engine BEGIN (implicit)\n",
      "2021-09-30 15:49:34,642 INFO sqlalchemy.engine.Engine \n",
      "CREATE TABLE lake (\n",
      "\tid INTEGER NOT NULL, \n",
      "\tname VARCHAR, \n",
      "\tPRIMARY KEY (id)\n",
      ")\n",
      "\n",
      "\n",
      "2021-09-30 15:49:34,643 INFO sqlalchemy.engine.Engine [no key 0.00138s] ()\n",
      "2021-09-30 15:49:35,448 INFO sqlalchemy.engine.Engine SELECT AddGeometryColumn(?, ?, ?, ?, ?, ?) AS \"AddGeometryColumn_1\"\n",
      "2021-09-30 15:49:35,449 INFO sqlalchemy.engine.Engine [no key 0.00092s] ('lake', 'geom', -1, 'POLYGON', 2, 0)\n",
      "2021-09-30 15:49:41,161 INFO sqlalchemy.engine.Engine SELECT CreateSpatialIndex(?, ?) AS \"CreateSpatialIndex_1\"\n",
      "2021-09-30 15:49:41,162 INFO sqlalchemy.engine.Engine [generated in 0.00089s] ('lake', 'geom')\n",
      "2021-09-30 15:49:47,476 INFO sqlalchemy.engine.Engine COMMIT\n"
     ]
    }
   ],
   "source": [
    "Lake.__table__.create(engine)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Table('lake', MetaData(), Column('id', Integer(), table=<lake>, primary_key=True, nullable=False), Column('name', String(), table=<lake>), Column('geom', Geometry(geometry_type='POLYGON', management=True, from_text='ST_GeomFromEWKT', name='geometry'), table=<lake>), schema=None)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Lake.__table__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we wanted to drop the table we’d use this. There’s nothing specific to SpatiaLite here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Lake.__table__.drop(engine)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a Session\n",
    "\n",
    "When using the SQLAlchemy ORM the ORM interacts with the database through a Session."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "Session = sessionmaker(bind=engine)\n",
    "session = Session()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add New Objects\n",
    "\n",
    "We can now create and insert new Lake objects into the database, the same way we’d do it using GeoAlchemy 2 with PostGIS."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-09-30 15:49:48,135 INFO sqlalchemy.engine.Engine BEGIN (implicit)\n",
      "2021-09-30 15:49:48,138 INFO sqlalchemy.engine.Engine INSERT INTO lake (name, geom) VALUES (?, GeomFromEWKT(?))\n",
      "2021-09-30 15:49:48,139 INFO sqlalchemy.engine.Engine [generated in 0.00168s] ('Majeur', 'POLYGON((0 0,1 0,1 1,0 1,0 0))')\n",
      "2021-09-30 15:49:48,377 INFO sqlalchemy.engine.Engine COMMIT\n"
     ]
    }
   ],
   "source": [
    "lake = Lake(name='Majeur', geom='POLYGON((0 0,1 0,1 1,0 1,0 0))')\n",
    "session.add(lake)\n",
    "session.commit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now query the database for Majeur:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-09-30 15:49:48,969 INFO sqlalchemy.engine.Engine BEGIN (implicit)\n",
      "2021-09-30 15:49:48,972 INFO sqlalchemy.engine.Engine SELECT lake.id AS lake_id, lake.name AS lake_name, AsEWKB(lake.geom) AS lake_geom \n",
      "FROM lake \n",
      "WHERE lake.name = ?\n",
      " LIMIT ? OFFSET ?\n",
      "2021-09-30 15:49:48,973 INFO sqlalchemy.engine.Engine [generated in 0.00121s] ('Majeur', 1, 0)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'Majeur'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "our_lake = session.query(Lake).filter_by(name='Majeur').first()\n",
    "our_lake.name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<WKBElement at 0x7f11438c4e20; 0103000020FFFFFFFF010000000500000000000000000000000000000000000000000000000000F03F0000000000000000000000000000F03F000000000000F03F0000000000000000000000000000F03F00000000000000000000000000000000>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "our_lake.geom"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "our_lake.id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let’s add more lakes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-09-30 15:49:49,666 INFO sqlalchemy.engine.Engine INSERT INTO lake (name, geom) VALUES (?, GeomFromEWKT(?))\n",
      "2021-09-30 15:49:49,670 INFO sqlalchemy.engine.Engine [cached since 1.533s ago] ('Garde', 'POLYGON((1 0,3 0,3 2,1 2,1 0))')\n",
      "2021-09-30 15:49:49,672 INFO sqlalchemy.engine.Engine INSERT INTO lake (name, geom) VALUES (?, GeomFromEWKT(?))\n",
      "2021-09-30 15:49:49,673 INFO sqlalchemy.engine.Engine [cached since 1.536s ago] ('Orta', 'POLYGON((3 0,6 0,6 3,3 3,3 0))')\n",
      "2021-09-30 15:49:49,674 INFO sqlalchemy.engine.Engine COMMIT\n"
     ]
    }
   ],
   "source": [
    "session.add_all([Lake(name='Garde', geom='POLYGON((1 0,3 0,3 2,1 2,1 0))'),\n",
    "                 Lake(name='Orta', geom='POLYGON((3 0,6 0,6 3,3 3,3 0))')\n",
    "                ])\n",
    "session.commit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Query\n",
    "\n",
    "#### Let’s make a simple, non-spatial, query:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-09-30 15:49:50,856 INFO sqlalchemy.engine.Engine BEGIN (implicit)\n",
      "2021-09-30 15:49:50,858 INFO sqlalchemy.engine.Engine SELECT lake.id AS lake_id, lake.name AS lake_name, AsEWKB(lake.geom) AS lake_geom \n",
      "FROM lake ORDER BY lake.name\n",
      "2021-09-30 15:49:50,859 INFO sqlalchemy.engine.Engine [generated in 0.00102s] ()\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['Garde', 'Majeur', 'Orta']"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "query = session.query(Lake).order_by(Lake.name)\n",
    "\n",
    "#for lake in query:\n",
    "#    print(lake.name)\n",
    "\n",
    "[l.name for l in query]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Now a spatial query:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-09-30 15:49:51,251 INFO sqlalchemy.engine.Engine SELECT lake.id AS lake_id, lake.name AS lake_name, AsEWKB(lake.geom) AS lake_geom \n",
      "FROM lake \n",
      "WHERE ST_Contains(lake.geom, ST_GeomFromText(?, ?))\n",
      "2021-09-30 15:49:51,252 INFO sqlalchemy.engine.Engine [no key 0.00156s] ('POINT(4 1)', -1)\n",
      "Orta\n"
     ]
    }
   ],
   "source": [
    "query = session.query(Lake).filter(func.ST_Contains(Lake.geom, WKTElement('POINT(4 1)')))\n",
    "\n",
    "for lake in query:\n",
    "    print(lake.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Altenatively: Here the ST_Contains function is applied to the Lake.geom column property. In that case the column property is actually passed to the function, as its first argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-09-30 15:49:51,421 INFO sqlalchemy.engine.Engine SELECT lake.id AS lake_id, lake.name AS lake_name, AsEWKB(lake.geom) AS lake_geom \n",
      "FROM lake \n",
      "WHERE ST_Contains(lake.geom, ?)\n",
      "2021-09-30 15:49:51,422 INFO sqlalchemy.engine.Engine [no key 0.00093s] ('POINT(4 1)',)\n",
      "Majeur\n",
      "Garde\n",
      "Orta\n"
     ]
    }
   ],
   "source": [
    "query = session.query(Lake).filter(Lake.geom.ST_Contains('POINT(4 1)')) \n",
    "\n",
    "for lake in query:\n",
    "    print(lake.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here’s another spatial query, using ST_Intersects this time:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-09-30 15:49:52,203 INFO sqlalchemy.engine.Engine SELECT lake.id AS lake_id, lake.name AS lake_name, AsEWKB(lake.geom) AS lake_geom \n",
      "FROM lake \n",
      "WHERE ST_Intersects(lake.geom, ST_GeomFromText(?, ?))\n",
      "2021-09-30 15:49:52,204 INFO sqlalchemy.engine.Engine [no key 0.00098s] ('LINESTRING(2 1,4 1)', -1)\n",
      "Garde\n",
      "Orta\n"
     ]
    }
   ],
   "source": [
    "query = session.query(Lake).filter(Lake.geom.ST_Intersects(WKTElement('LINESTRING(2 1,4 1)')))\n",
    "for lake in query:\n",
    "    print(lake.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also apply relationship functions to geoalchemy2.elements.WKBElement. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-09-30 15:49:52,904 INFO sqlalchemy.engine.Engine SELECT lake.id AS lake_id, lake.name AS lake_name, AsEWKB(lake.geom) AS lake_geom \n",
      "FROM lake \n",
      "WHERE lake.name = ?\n",
      "2021-09-30 15:49:52,905 INFO sqlalchemy.engine.Engine [generated in 0.00106s] ('Garde',)\n",
      "2021-09-30 15:49:52,908 INFO sqlalchemy.engine.Engine SELECT ST_Intersects(GeomFromEWKB(?), ST_GeomFromText(?, ?)) AS \"ST_Intersects_1\"\n",
      "2021-09-30 15:49:52,909 INFO sqlalchemy.engine.Engine [no key 0.00095s] ('0103000020FFFFFFFF0100000005000000000000000000F03F00000000000000000000000000000840000000000000000000000000000008400000000000000040000000000000F03F0000000000000040000000000000F03F0000000000000000', 'LINESTRING(2 1,4 1)', -1)\n",
      "1\n"
     ]
    }
   ],
   "source": [
    "lake = session.query(Lake).filter_by(name='Garde').one()\n",
    "print(session.scalar(lake.geom.ST_Intersects(WKTElement('LINESTRING(2 1,4 1)'))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "session.scalar allows executing a clause and returning a scalar value (an integer value in this case).\n",
    "\n",
    "The value 1 indicates that the lake “Garde” does intersects the LINESTRING(2 1,4 1) geometry  \n",
    "\n",
    "The GeoAlchemy functions all start with ST_. Operators are also called as functions, but the function names don’t include the ST_ prefix.  \n",
    "As an example let’s test whether the bounding boxes of geometries intersect. GeoAlchemy provides the intersects function for that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}