Notebooks/05_SQLAlchemy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction to Databases\n",
    "\n",
    "### Using SQLAlchemy \n",
    "\n",
    "Based in [this](https://medium.com/hacking-datascience/sqlalchemy-python-tutorial-abcc2ec77b57), [this](https://medium.com/dataexplorations/sqlalchemy-orm-a-more-pythonic-way-of-interacting-with-your-database-935b57fd2d4d) and [this](https://auth0.com/blog/sqlalchemy-orm-tutorial-for-python-developers/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "SQLAlchemy provides a nice “Pythonic” way of interacting with databases. So rather than dealing with the differences between specific dialects of traditional SQL such as MySQL or PostgreSQL or Oracle, you can leverage the Pythonic framework of SQLAlchemy to streamline your workflow and more efficiently query your data.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install mysqlclient\n",
    "#!pip install python-dotenv\n",
    "#!pip install sqlalchemy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import getpass\n",
    "import pandas as pd\n",
    "\n",
    "import sqlalchemy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Connecting to a database\n",
    "\n",
    "To start interacting with the database we first we need to establish a connection.  \n",
    "\n",
    "Some examples of connecting to various databases can be found [here](http://docs.sqlalchemy.org/en/latest/core/engines.html#postgresql)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Viewing Table Details\n",
    "\n",
    "SQLAlchemy can be used to automatically load tables from a database using something called reflection. Reflection is the process of reading the database and building the metadata based on that information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "engine = sqlalchemy.create_engine('sqlite:///../SampleDBs/chinook.sqlite')\n",
    "connection = engine.connect()\n",
    "metadata = sqlalchemy.MetaData()\n",
    "\n",
    "albums = sqlalchemy.Table('albums', metadata, autoload=True, autoload_with=engine)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['AlbumId', 'Title', 'ArtistId']\n"
     ]
    }
   ],
   "source": [
    "# Print the column names\n",
    "print(albums.columns.keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Table('albums', MetaData(bind=None), Column('AlbumId', INTEGER(), table=<albums>, primary_key=True, nullable=False), Column('Title', NVARCHAR(length=160), table=<albums>, nullable=False), Column('ArtistId', INTEGER(), ForeignKey('artists.ArtistId'), table=<albums>, nullable=False), schema=None)\n"
     ]
    }
   ],
   "source": [
    "# Print full table metadata\n",
    "print(repr(metadata.tables['albums']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Querying\n",
    "\n",
    "ResultProxy: The object returned by the .execute() method. It can be used in a variety of ways to get the data returned by the query.  \n",
    "\n",
    "ResultSet: The actual data asked for in the query when using a fetch method such as .fetchall() on a ResultProxy.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Equivalent to 'SELECT * FROM albums'\n",
    "query = sqlalchemy.select([albums])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(1, 'For Those About To Rock We Salute You', 1),\n",
       " (2, 'Balls to the Wall', 2),\n",
       " (3, 'Restless and Wild', 2)]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ResultProxy = connection.execute(query)\n",
    "ResultSet = ResultProxy.fetchall()\n",
    "#partial_results = ResultProxy.fetchmany(50)\n",
    "ResultSet[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Convert to dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>AlbumId</th>\n",
       "      <th>Title</th>\n",
       "      <th>ArtistId</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>For Those About To Rock We Salute You</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>Balls to the Wall</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>Restless and Wild</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>Let There Be Rock</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>Big Ones</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   AlbumId                                  Title  ArtistId\n",
       "0        1  For Those About To Rock We Salute You         1\n",
       "1        2                      Balls to the Wall         2\n",
       "2        3                      Restless and Wild         2\n",
       "3        4                      Let There Be Rock         1\n",
       "4        5                               Big Ones         3"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame(ResultSet)\n",
    "df.columns = ResultSet[0].keys()\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Filtering data\n",
    "\n",
    "Lets see some examples of raw SQLite Queries and queries using SQLAlchemy.\n",
    "\n",
    "#### where"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(16, 'Caetano Veloso')]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## SQL :SELECT * FROM artists WHERE Name = \"Caetano Veloso\" :\n",
    "\n",
    "artists = sqlalchemy.Table('artists', metadata, autoload=True, autoload_with=engine)\n",
    "\n",
    "query = sqlalchemy.select([artists]).where(artists.columns.Name == 'Caetano Veloso')\n",
    "ResultProxy = connection.execute(query)\n",
    "ResultSet = ResultProxy.fetchall()\n",
    "ResultSet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### in"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Roberto', 'Almeida'), ('Michelle', 'Brooks')]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## SQL : SELECT FirstName, LastName FROM customers WHERE state IN (\"Rio de Janeiro\", \"New York\")\n",
    "\n",
    "customers = sqlalchemy.Table('customers', metadata, autoload=True, autoload_with=engine)\n",
    "\n",
    "query = sqlalchemy.select([customers.columns.FirstName, \n",
    "                           customers.columns.LastName]).where(customers.columns.City.in_(['Rio de Janeiro', 'New York']))\n",
    "ResultProxy = connection.execute(query)\n",
    "ResultSet = ResultProxy.fetchall()\n",
    "ResultSet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### and, or, not"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(12, 'Roberto', 'Almeida', 'Riotur', 'Praça Pio X, 119', 'Rio de Janeiro', 'RJ', 'Brazil', '20040-020', '+55 (21) 2271-7000', '+55 (21) 2271-7070', 'roberto.almeida@riotur.gov.br', 3)]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## SQL :SELECT * FROM customers WHERE City = 'Rio de Janeiro' AND NOT FirstName = 'Roberta'\n",
    "    \n",
    "customers = sqlalchemy.Table('customers', metadata, autoload=True, autoload_with=engine)\n",
    "\n",
    "query = sqlalchemy.select([customers]).where(sqlalchemy.and_(customers.columns.City == \"Rio de Janeiro\",\n",
    "                                                             customers.columns.FirstName != \"Roberta\")\n",
    "                                            )\n",
    "ResultProxy = connection.execute(query)\n",
    "ResultSet = ResultProxy.fetchall()\n",
    "ResultSet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### order by"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(33, 'Ellie', 'Sullivan', None, '5112 48 Street', 'Yellowknife', 'NT', 'Canada', 'X1A 1N6', '+1 (867) 920-2233', None, 'ellie.sullivan@shaw.ca', 3),\n",
       " (32, 'Aaron', 'Mitchell', None, '696 Osborne Street', 'Winnipeg', 'MB', 'Canada', 'R3L 2B9', '+1 (204) 452-6452', None, 'aaronmitchell@yahoo.ca', 4),\n",
       " (49, 'Stanisław', 'Wójcik', None, 'Ordynacka 10', 'Warsaw', None, 'Poland', '00-358', '+48 22 828 37 39', None, 'stanisław.wójcik@wp.pl', 4)]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## SQL : SELECT * FROM customers ORDER BY City DESC, Country\n",
    "\n",
    "customers = sqlalchemy.Table('customers', metadata, autoload=True, autoload_with=engine)\n",
    "\n",
    "query = sqlalchemy.select([customers]).order_by(sqlalchemy.desc(customers.columns.City), customers.columns.Country)\n",
    "ResultProxy = connection.execute(query)\n",
    "ResultSet = ResultProxy.fetchall()\n",
    "ResultSet[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### functions\n",
    "other functions include avg, count, min, max…  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/sqlalchemy/sql/sqltypes.py:666: SAWarning: Dialect sqlite+pysqlite does *not* support Decimal objects natively, and SQLAlchemy must convert from floating point - rounding errors and other issues may occur. Please consider storing Decimal numbers as strings or integers on this platform for lossless storage.\n",
      "  \"storage.\" % (dialect.name, dialect.driver)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[(Decimal('2328.60'),)]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "##SQL : SELECT SUM(Total) FROM invoices \n",
    "    \n",
    "invoices = sqlalchemy.Table('invoices', metadata, autoload=True, autoload_with=engine)\n",
    "\n",
    "query = sqlalchemy.select([sqlalchemy.func.sum(invoices.columns.Total)])\n",
    "ResultProxy = connection.execute(query)\n",
    "ResultSet = ResultProxy.fetchall()\n",
    "ResultSet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### group by"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(Decimal('1.98'), datetime.datetime(2009, 1, 1, 0, 0)),\n",
       " (Decimal('3.96'), datetime.datetime(2009, 1, 2, 0, 0)),\n",
       " (Decimal('5.94'), datetime.datetime(2009, 1, 3, 0, 0)),\n",
       " (Decimal('8.91'), datetime.datetime(2009, 1, 6, 0, 0)),\n",
       " (Decimal('13.86'), datetime.datetime(2009, 1, 11, 0, 0))]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "##SQL : SELECT SUM(Total) as Total, InvoiceDate FROM invoices GROUP BY InvoiceDate\n",
    "\n",
    "invoices = sqlalchemy.Table('invoices', metadata, autoload=True, autoload_with=engine)\n",
    "\n",
    "query = sqlalchemy.select([sqlalchemy.func.sum(invoices.columns.Total).label('Total'), \n",
    "                           invoices.columns.InvoiceDate]).group_by(invoices.columns.InvoiceDate)\n",
    "\n",
    "ResultProxy = connection.execute(query)\n",
    "ResultSet = ResultProxy.fetchall()\n",
    "ResultSet[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### distinct"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(1,), (2,), (3,), (4,), (5,)]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## SQL : SELECT DISTINCT state FROM censusSQLAlchemy :\n",
    "\n",
    "invoices = sqlalchemy.Table('invoices', metadata, autoload=True, autoload_with=engine)\n",
    "query = sqlalchemy.select([invoices.columns.CustomerId.distinct()])\n",
    "\n",
    "ResultProxy = connection.execute(query)\n",
    "ResultSet = ResultProxy.fetchall()\n",
    "ResultSet[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### joins"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "artists = sqlalchemy.Table('artists', metadata, autoload=True, autoload_with=engine)\n",
    "albums = sqlalchemy.Table('albums', metadata, autoload=True, autoload_with=engine)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Automatic Join"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ArtistId</th>\n",
       "      <th>Name</th>\n",
       "      <th>AlbumId</th>\n",
       "      <th>Title</th>\n",
       "      <th>ArtistId</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>AC/DC</td>\n",
       "      <td>1</td>\n",
       "      <td>For Those About To Rock We Salute You</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>AC/DC</td>\n",
       "      <td>2</td>\n",
       "      <td>Balls to the Wall</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>AC/DC</td>\n",
       "      <td>3</td>\n",
       "      <td>Restless and Wild</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>AC/DC</td>\n",
       "      <td>4</td>\n",
       "      <td>Let There Be Rock</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>AC/DC</td>\n",
       "      <td>5</td>\n",
       "      <td>Big Ones</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   ArtistId   Name  AlbumId                                  Title  ArtistId\n",
       "0         1  AC/DC        1  For Those About To Rock We Salute You         1\n",
       "1         1  AC/DC        2                      Balls to the Wall         2\n",
       "2         1  AC/DC        3                      Restless and Wild         2\n",
       "3         1  AC/DC        4                      Let There Be Rock         1\n",
       "4         1  AC/DC        5                               Big Ones         3"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "query = sqlalchemy.select([artists, albums])\n",
    "#query = sqlalchemy.select([artists.columns.Name, albums.columns.Title])\n",
    "\n",
    "results = connection.execute(query).fetchall()\n",
    "df = pd.DataFrame(results)\n",
    "df.columns = results[0].keys()\n",
    "df.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Manual Join"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ArtistId</th>\n",
       "      <th>Name</th>\n",
       "      <th>AlbumId</th>\n",
       "      <th>Title</th>\n",
       "      <th>ArtistId</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>AC/DC</td>\n",
       "      <td>1</td>\n",
       "      <td>For Those About To Rock We Salute You</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>Accept</td>\n",
       "      <td>2</td>\n",
       "      <td>Balls to the Wall</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>Accept</td>\n",
       "      <td>3</td>\n",
       "      <td>Restless and Wild</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>AC/DC</td>\n",
       "      <td>4</td>\n",
       "      <td>Let There Be Rock</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3</td>\n",
       "      <td>Aerosmith</td>\n",
       "      <td>5</td>\n",
       "      <td>Big Ones</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   ArtistId       Name  AlbumId                                  Title  \\\n",
       "0         1      AC/DC        1  For Those About To Rock We Salute You   \n",
       "1         2     Accept        2                      Balls to the Wall   \n",
       "2         2     Accept        3                      Restless and Wild   \n",
       "3         1      AC/DC        4                      Let There Be Rock   \n",
       "4         3  Aerosmith        5                               Big Ones   \n",
       "\n",
       "   ArtistId  \n",
       "0         1  \n",
       "1         2  \n",
       "2         2  \n",
       "3         1  \n",
       "4         3  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "query = sqlalchemy.select([artists, albums])\n",
    "query = query.select_from(artists.join(albums, artists.columns.ArtistId == albums.columns.ArtistId))\n",
    "\n",
    "results = connection.execute(query).fetchall()\n",
    "df = pd.DataFrame(results)\n",
    "df.columns = results[0].keys()\n",
    "df.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating and Inserting Data into Tables\n",
    "\n",
    "By passing the database which is not present, to the engine then sqlalchemy automatically creates a new database."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "engine = sqlalchemy.create_engine('sqlite:///../SampleDBs/test.sqlite') #Create test.sqlite automatically\n",
    "connection = engine.connect()\n",
    "metadata = sqlalchemy.MetaData()\n",
    "\n",
    "emp = sqlalchemy.Table('emp', metadata,\n",
    "                       sqlalchemy.Column('Id', sqlalchemy.Integer()),\n",
    "                       sqlalchemy.Column('name', sqlalchemy.String(255), nullable=False),\n",
    "                       sqlalchemy.Column('salary', sqlalchemy.Float(), default=100.0),\n",
    "                       sqlalchemy.Column('active', sqlalchemy.Boolean(), default=True)\n",
    "                      )\n",
    "\n",
    "metadata.create_all(engine) #Creates the table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Inserting record one by one\n",
    "\n",
    "query = sqlalchemy.insert(emp).values(Id=1, name='naveen', salary=60000.00, active=True) \n",
    "ResultProxy = connection.execute(query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Inserting many records at ones\n",
    "\n",
    "query = sqlalchemy.insert(emp) \n",
    "values_list = [{'Id':'2', 'name':'ram', 'salary':80000, 'active':False},\n",
    "               {'Id':'3', 'name':'ramesh', 'salary':70000, 'active':True}]\n",
    "\n",
    "ResultProxy = connection.execute(query,values_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>name</th>\n",
       "      <th>salary</th>\n",
       "      <th>active</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>naveen</td>\n",
       "      <td>60000.0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>ram</td>\n",
       "      <td>80000.0</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>ramesh</td>\n",
       "      <td>70000.0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Id    name   salary  active\n",
       "0   1  naveen  60000.0    True\n",
       "1   2     ram  80000.0   False\n",
       "2   3  ramesh  70000.0    True"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results = connection.execute(sqlalchemy.select([emp])).fetchall()\n",
    "df = pd.DataFrame(results)\n",
    "df.columns = results[0].keys()\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Updating data in Databases\n",
    "\n",
    "db.update(table_name).values(attribute = new_value).where(condition)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>name</th>\n",
       "      <th>salary</th>\n",
       "      <th>active</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>naveen</td>\n",
       "      <td>60000.0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>ram</td>\n",
       "      <td>80000.0</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>ramesh</td>\n",
       "      <td>70000.0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Id    name   salary  active\n",
       "0   1  naveen  60000.0    True\n",
       "1   2     ram  80000.0   False\n",
       "2   3  ramesh  70000.0    True"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results = connection.execute(sqlalchemy.select([emp])).fetchall()\n",
    "\n",
    "df = pd.DataFrame(results)\n",
    "df.columns = results[0].keys()\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build a statement to update the salary of Id = 1 to to 100000\n",
    "\n",
    "query = sqlalchemy.update(emp).values(salary = 100000)\n",
    "query = query.where(emp.columns.Id == 1)\n",
    "results = connection.execute(query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>name</th>\n",
       "      <th>salary</th>\n",
       "      <th>active</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>naveen</td>\n",
       "      <td>100000.0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>ram</td>\n",
       "      <td>80000.0</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>ramesh</td>\n",
       "      <td>70000.0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Id    name    salary  active\n",
       "0   1  naveen  100000.0    True\n",
       "1   2     ram   80000.0   False\n",
       "2   3  ramesh   70000.0    True"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results = connection.execute(sqlalchemy.select([emp])).fetchall()\n",
    "df = pd.DataFrame(results)\n",
    "df.columns = results[0].keys()\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Delete tuples from Table\n",
    "\n",
    "db.delete(table_name).where(condition)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>name</th>\n",
       "      <th>salary</th>\n",
       "      <th>active</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>naveen</td>\n",
       "      <td>100000.0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>ram</td>\n",
       "      <td>80000.0</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>ramesh</td>\n",
       "      <td>70000.0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Id    name    salary  active\n",
       "0   1  naveen  100000.0    True\n",
       "1   2     ram   80000.0   False\n",
       "2   3  ramesh   70000.0    True"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results = connection.execute(sqlalchemy.select([emp])).fetchall()\n",
    "df = pd.DataFrame(results)\n",
    "df.columns = results[0].keys()\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build a statement to delete where salary < 80000\n",
    "\n",
    "query = sqlalchemy.delete(emp)\n",
    "query = query.where(emp.columns.salary < 80000)\n",
    "results = connection.execute(query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>name</th>\n",
       "      <th>salary</th>\n",
       "      <th>active</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>naveen</td>\n",
       "      <td>100000.0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>ram</td>\n",
       "      <td>80000.0</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Id    name    salary  active\n",
       "0   1  naveen  100000.0    True\n",
       "1   2     ram   80000.0   False"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results = connection.execute(sqlalchemy.select([emp])).fetchall()\n",
    "df = pd.DataFrame(results)\n",
    "df.columns = results[0].keys()\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Dropping a Table\n",
    "\n",
    "table_name.drop(engine) #drops a single table  \n",
    "metadata.drop_all(engine) #drops all the tables in the database"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Removing the database"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.remove('../SampleDBs/test.sqlite')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A full example of Database creation and management using SQLAlchemy ORM\n",
    "\n",
    "![](../SampleDBs/SQLAlchemyORM.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Let's set up this database:\n",
    "\n",
    "points_of_interest is the main table which has a zero-to-many relationship with the other three tables. For example, a given point of interest may have no associated Architects or it may have several Architects."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Option 1 — Raw SQL"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sqlalchemy import create_engine\n",
    "#db = create_engine('dialect+driver://user:pass@host:port/db')\n",
    "#db = create_engine(f'postgresql://{DB_USER}:{DB_PASS}@{IP}:{DB_PORT}/{DB_NAME}')\n",
    "\n",
    "db = create_engine('sqlite:///../SampleDBs/architect.sqlite')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<sqlalchemy.engine.result.ResultProxy at 0x7f0214beb290>"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# create main table\n",
    "db.execute(\"\"\"\n",
    "CREATE TABLE IF NOT EXISTS points_of_interest (\n",
    "    poi_id BIGSERIAL PRIMARY KEY,\n",
    "    name text,\n",
    "    build_year text, \n",
    "    demolished_year text,\n",
    "    address text, \n",
    "    latitude float, \n",
    "    longitude float,\n",
    "    source text, \n",
    "    external_url text, \n",
    "    details text,\n",
    "    image_url text, \n",
    "    heritage_status text, \n",
    "    current_use text, \n",
    "    poi_type text)\n",
    "    \"\"\")\n",
    "\n",
    "# create architectural styles TABLE\n",
    "db.execute(\"\"\"\n",
    "CREATE TABLE IF NOT EXISTS architectural_styles (\n",
    "    poi_id int,\n",
    "    style text)\n",
    "    \"\"\")\n",
    "\n",
    "# create architects TABLE\n",
    "db.execute(\"\"\"\n",
    "CREATE TABLE IF NOT EXISTS architects (\n",
    "    poi_id int,\n",
    "    architect_name text)\n",
    "    \"\"\")\n",
    "\n",
    "# create categories TABLE\n",
    "db.execute(\"\"\"CREATE TABLE IF NOT EXISTS poi_categories (\n",
    "    poi_id int,\n",
    "    category text)\n",
    "    \"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This code does not establish a formal relationship between the tables (i.e. it is up to me as the developer to maintain the links). So, for example, as I started to load data from a pandas data frame into the database, I had to save the generated primary key from the PointsOfInterest table and use that when submitting entries to the ArchitecturalStyles table"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "new_id = new_row.inserted_primary_key[0]\n",
    "if new_id:\n",
    "    db.execute('''INSERT INTO architectural_styles(poi_id, style) VALUES ( {},'{}')'''.format(new_id, row['Style']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "removed file\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    os.remove(\"../SampleDBs/architect.sqlite\")\n",
    "    print(\"removed file\")\n",
    "except:\n",
    "    print(\"file does not exist\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Option 2 — SQLAlchemy ORM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sqlalchemy import create_engine, Column, Integer, String, Sequence, Float, PrimaryKeyConstraint, ForeignKey\n",
    "from sqlalchemy.ext.declarative import declarative_base\n",
    "from sqlalchemy.orm import sessionmaker, relationship, backref"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using SQL Alchemy ORM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "db = create_engine('sqlite:///../SampleDBs/architect.sqlite')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create Tables as Classes\n",
    "\n",
    "Instead of writing SQL INSERT statements, you will define a class to represent each of your tables.\n",
    "\n",
    "+ Class inherits from the declarative_base() class\n",
    "+ define __tablename__ (the actual table name that is created in the database)\n",
    "+ define Columns and their types  \n",
    "\n",
    "How do I set up an auto-incrementing primary key like BIGSERIAL?\n",
    "\n",
    "There is no BIGSERIAL-equivalent column type in ORM, but you can accomplish the same thing by explicitly defining a sequence:\n",
    "\n",
    "poi_id = Column(Integer, Sequence('poi_id_seq'), primary_key=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "Base = declarative_base()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "class PointsOfInterest(Base):\n",
    "    __tablename__ = \"points_of_interest\"\n",
    "    poi_id = Column(Integer, Sequence('poi_id_seq'), primary_key=True)\n",
    "    name = Column(String)\n",
    "    build_year = Column(String)\n",
    "    build_decade = Column(Float)\n",
    "    build_year_clean = Column(Float)\n",
    "    demolished_year = Column(String)\n",
    "    address = Column(String)\n",
    "    latitude = Column(Float)\n",
    "    longitude = Column(Float)\n",
    "    external_url = Column(String)\n",
    "    image_url = Column(String)\n",
    "    heritage_status = Column(String)\n",
    "    current_use = Column(String)\n",
    "    poi_type = Column(String)\n",
    "    poi_type_simple = Column(String)\n",
    "    source = Column(String)\n",
    "    details = Column(String)\n",
    "    \n",
    "    #Defining One to Many relationships with the relationship function on the Parent Table\n",
    "    styles = relationship('ArchitecturalStyles', backref = 'points_of_interest',lazy=True, cascade=\"all, delete-orphan\")\n",
    "    architects = relationship('Architects', backref = 'points_of_interest', lazy=True,cascade=\"all, delete-orphan\")\n",
    "    categories = relationship('POICategories', backref = 'points_of_interest', lazy=True,cascade=\"all, delete-orphan\")\n",
    "    \n",
    "    def print_me(self):\n",
    "        print(f\"id: {self.poi_id}\")\n",
    "        for col in self.__table__.columns:\n",
    "            print(col.name, \":    \", getattr(self,col.name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the supporting / foreign-key tables\n",
    "\n",
    "SQLAlchemy ORM requires you to define a primary key. We have to define a primary key encompassing both columns . This has the added benefit of preventing you from inserting duplicate architectural styles for the same building (or from inserting null entries)  \n",
    "\n",
    "#### One-to-Many Relationship Pattern / Foreign Key\n",
    "\n",
    "You can set up several types of relationship patterns in SQLAlchemy ORM, but I only needed to use the One-To-Many relationship. I found this part confusing and at first only defined the foreign key on the supplementary table. But you can take more advantage of the power of ORM if you also define the relationship on the main table.\n",
    "\n",
    "The first step is indeed to define your Foreignkey and point to the table.column of interest\n",
    "\n",
    "The next step is to define a bidirectional relationship between the two tables (a zero-to-many from PointsOfInterest to ArchitecturalStyles and a many-to-one from ArchitecturalStyles to PointsOfInterest). This makes it easier, for example, when loading data since ORM takes care of automatically using the newly generated poi_id when populating data into the other tables.\n",
    "\n",
    "To do this, add a relationship to the main PointsofInterest class and use the backref parameter to connect the two (see the last cell) \n",
    "\n",
    "+ styles = The relationship with the AchitecturalStyles will be named “styles” (this attribute will be exposed when working with the PointsOfInterest class)\n",
    "+ backref = connects the two classes\n",
    "+ lazy = determines how the supporting tables get loaded when you query the main table. lazy=Trueis the default option and works for me here\n",
    "+ cascade = if I delete a poi_id from the main table, the linked rows in architectural styles will be deleted too (so I don’t end up with orphaned entries)\n",
    "\n",
    "Since I’m referencing the ArchitecturalStyles class from the PointsOfInterest class and vice versa, I kept running into “not yet defined” errors when running the code. The secret was to define the main class first (PointsOfInterest) and use quotes around the referenced class names when defining the relationship( relationship='ArchitecturalStyles'). This tells SQLAlchemy to create a placeholder for a class that will be defined later"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ArchitecturalStyles(Base):\n",
    "    __tablename__= \"architectural_styles\"\n",
    "    __table_args__ = (PrimaryKeyConstraint('poi_id', 'style'),)\n",
    "    poi_id = Column(Integer,ForeignKey('points_of_interest.poi_id'))\n",
    "    style = Column(String)     #Defining the Foreign Key on the Child Table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Architects(Base):\n",
    "    __tablename__ = \"architects\"\n",
    "    __table_args__ = (PrimaryKeyConstraint('poi_id', 'architect_name'),)\n",
    "    poi_id = Column(Integer,ForeignKey('points_of_interest.poi_id'))\n",
    "    architect_name = Column(String)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "class POICategories(Base):\n",
    "    __tablename__ = \"poi_categories\"\n",
    "    __table_args__ = (PrimaryKeyConstraint('poi_id', 'category'),)\n",
    "    poi_id = Column(Integer,ForeignKey('points_of_interest.poi_id'))\n",
    "    category = Column(String)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the database tables\n",
    "\n",
    "Run this command to actually create the tables in the database\n",
    "    \n",
    "    checkfirst = check if table already exists and skip the creation if it already exists"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "PointsOfInterest.__table__.create(bind=db, checkfirst=True)\n",
    "ArchitecturalStyles.__table__.create(bind=db, checkfirst=True)\n",
    "Architects.__table__.create(bind=db, checkfirst=True)\n",
    "POICategories.__table__.create(bind=db, checkfirst=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using SQL Alchemy ORM\n",
    "\n",
    "### Inserting Rows\n",
    "\n",
    "Now that my classes were defined, I could use them in other modules to help load data. As usual, the first step is to import the necessary classes, including our new class definitions (PointsOfInterest etc) and establish a session:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "Session = sessionmaker(bind=db)\n",
    "session = Session()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "session.add(PointsOfInterest(name = 'FGV',\n",
    "                             build_year = 1970,\n",
    "                             build_decade = 70,\n",
    "                             build_year_clean = 1970,\n",
    "                             # demolished_year = ,\n",
    "                             address = 'Praia de Botafogo, 190 - 426 - Botafogo, Rio de Janeiro - RJ, 22250-900, Brasil',\n",
    "                             latitude = -22.94,\n",
    "                             longitude = -43.18,\n",
    "                             external_url = 'https://portal.fgv.br/',\n",
    "                             # image_url = ,\n",
    "                             # heritage_status = ,\n",
    "                             # current_use = ,\n",
    "                             poi_type = 'Building' ,\n",
    "                             poi_type_simple = 'Building',\n",
    "                             source = 'Google Maps',\n",
    "                             # details = ,\n",
    "                            ))\n",
    "\n",
    "\n",
    "session.commit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "session.add(POICategories(poi_id = 1, category = \"Commerce\"))\n",
    "session.commit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "session.add(Architects(poi_id = 1, architect_name = \"Oscar Niemeyer\"))\n",
    "session.add(Architects(poi_id = 1, architect_name = \"Lucio Costa\"))\n",
    "session.commit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Get Count of rows in table\n",
    "session.query(PointsOfInterest).count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Praia de Botafogo, 190 - 426 - Botafogo, Rio de Janeiro - RJ, 22250-900, Brasil\n"
     ]
    }
   ],
   "source": [
    "#get an object by primary key\n",
    "poi = session.query(PointsOfInterest).get(1)\n",
    "print(poi.address)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<sqlalchemy.orm.query.Query at 0x7f0213d1dcd0>"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#filter on particular column/value\n",
    "poi = session.query(PointsOfInterest).filter(PointsOfInterest.build_year==1970)\n",
    "poi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "delete_architect = session.query(Architects).filter(Architects.poi_id==1).first()\n",
    "session.delete(delete_architect)\n",
    "session.commit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Using Pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>poi_id</th>\n",
       "      <th>name</th>\n",
       "      <th>build_year</th>\n",
       "      <th>build_decade</th>\n",
       "      <th>build_year_clean</th>\n",
       "      <th>demolished_year</th>\n",
       "      <th>address</th>\n",
       "      <th>latitude</th>\n",
       "      <th>longitude</th>\n",
       "      <th>external_url</th>\n",
       "      <th>image_url</th>\n",
       "      <th>heritage_status</th>\n",
       "      <th>current_use</th>\n",
       "      <th>poi_type</th>\n",
       "      <th>poi_type_simple</th>\n",
       "      <th>source</th>\n",
       "      <th>details</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>FGV</td>\n",
       "      <td>1970</td>\n",
       "      <td>70.0</td>\n",
       "      <td>1970.0</td>\n",
       "      <td>None</td>\n",
       "      <td>Praia de Botafogo, 190 - 426 - Botafogo, Rio d...</td>\n",
       "      <td>-22.94</td>\n",
       "      <td>-43.18</td>\n",
       "      <td>https://portal.fgv.br/</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Building</td>\n",
       "      <td>Building</td>\n",
       "      <td>Google Maps</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   poi_id name build_year  build_decade  build_year_clean demolished_year  \\\n",
       "0       1  FGV       1970          70.0            1970.0            None   \n",
       "\n",
       "                                             address  latitude  longitude  \\\n",
       "0  Praia de Botafogo, 190 - 426 - Botafogo, Rio d...    -22.94     -43.18   \n",
       "\n",
       "             external_url image_url heritage_status current_use  poi_type  \\\n",
       "0  https://portal.fgv.br/      None            None        None  Building   \n",
       "\n",
       "  poi_type_simple       source details  \n",
       "0        Building  Google Maps    None  "
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_sql(session.query(PointsOfInterest).statement, session.bind)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>poi_id</th>\n",
       "      <th>name</th>\n",
       "      <th>build_year</th>\n",
       "      <th>build_decade</th>\n",
       "      <th>build_year_clean</th>\n",
       "      <th>demolished_year</th>\n",
       "      <th>address</th>\n",
       "      <th>latitude</th>\n",
       "      <th>longitude</th>\n",
       "      <th>external_url</th>\n",
       "      <th>image_url</th>\n",
       "      <th>heritage_status</th>\n",
       "      <th>current_use</th>\n",
       "      <th>poi_type</th>\n",
       "      <th>poi_type_simple</th>\n",
       "      <th>source</th>\n",
       "      <th>details</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>FGV</td>\n",
       "      <td>1970</td>\n",
       "      <td>70.0</td>\n",
       "      <td>1970.0</td>\n",
       "      <td>None</td>\n",
       "      <td>Praia de Botafogo, 190 - 426 - Botafogo, Rio d...</td>\n",
       "      <td>-22.94</td>\n",
       "      <td>-43.18</td>\n",
       "      <td>https://portal.fgv.br/</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Building</td>\n",
       "      <td>Building</td>\n",
       "      <td>Google Maps</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   poi_id name build_year  build_decade  build_year_clean demolished_year  \\\n",
       "0       1  FGV       1970          70.0            1970.0            None   \n",
       "\n",
       "                                             address  latitude  longitude  \\\n",
       "0  Praia de Botafogo, 190 - 426 - Botafogo, Rio d...    -22.94     -43.18   \n",
       "\n",
       "             external_url image_url heritage_status current_use  poi_type  \\\n",
       "0  https://portal.fgv.br/      None            None        None  Building   \n",
       "\n",
       "  poi_type_simple       source details  \n",
       "0        Building  Google Maps    None  "
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.query(\"build_year >= '1900'\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>poi_id</th>\n",
       "      <th>name</th>\n",
       "      <th>build_year</th>\n",
       "      <th>build_decade</th>\n",
       "      <th>build_year_clean</th>\n",
       "      <th>demolished_year</th>\n",
       "      <th>address</th>\n",
       "      <th>latitude</th>\n",
       "      <th>longitude</th>\n",
       "      <th>external_url</th>\n",
       "      <th>image_url</th>\n",
       "      <th>heritage_status</th>\n",
       "      <th>current_use</th>\n",
       "      <th>poi_type</th>\n",
       "      <th>poi_type_simple</th>\n",
       "      <th>source</th>\n",
       "      <th>details</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>FGV</td>\n",
       "      <td>1970</td>\n",
       "      <td>70.0</td>\n",
       "      <td>1970.0</td>\n",
       "      <td>None</td>\n",
       "      <td>Praia de Botafogo, 190 - 426 - Botafogo, Rio d...</td>\n",
       "      <td>-22.94</td>\n",
       "      <td>-43.18</td>\n",
       "      <td>https://portal.fgv.br/</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Building</td>\n",
       "      <td>Building</td>\n",
       "      <td>Google Maps</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   poi_id name build_year  build_decade  build_year_clean demolished_year  \\\n",
       "0       1  FGV       1970          70.0            1970.0            None   \n",
       "\n",
       "                                             address  latitude  longitude  \\\n",
       "0  Praia de Botafogo, 190 - 426 - Botafogo, Rio d...    -22.94     -43.18   \n",
       "\n",
       "             external_url image_url heritage_status current_use  poi_type  \\\n",
       "0  https://portal.fgv.br/      None            None        None  Building   \n",
       "\n",
       "  poi_type_simple       source details  \n",
       "0        Building  Google Maps    None  "
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.query('address.str.contains(\"Botafogo\")', engine='python')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}