Notebooks/Numpy/04_Numpy_NeuralNetworks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "pdmqqvbyL_sz"
   },
   "source": [
    "<h1 style=\"color:rgb(0,120,170)\">Neural Networks and Deep Learning</h1>\n",
    "\n",
    "## [Implementing Neural Networks with Numpy](https://github.com/SurajDonthi/Neural-Networks-from-Scratch)\n",
    "\n",
    "### Part 4: Neural Networks from Scratch  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Activation Function\n",
    "\n",
    "In the hidden layer, we will use the tanh activation function and in the output layer, I will use the sigmoid function. It is easy to find information on both the sigmoid function and the tanh function graph. I don’t want to bore you with explanations, so I will just implement it.\n",
    "\n",
    "![Sigmoid](./Images/2nn0.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sigmoid(x):\n",
    "    return (1 / (1 + np.exp(-x)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Setting Parameters\n",
    "\n",
    "What are parameters and hyperparameters? Parameters are weights and biases. Hyperparameters effect parameters and are before the learning begins. Setting hyperparameters perfectly correctly at first is not a piece of cake, you’ll need to tinker and tweak your values. The learning rate, number of iterations, and regularization rate, among others, can all be considered as hyperparameters.\n",
    "\n",
    "Wondering how to set the matrices sizes? The answer just below!\n",
    "\n",
    "What does all that mean? For example:\n",
    "(layer 0 so L = 0) number of neurons in input layers = 3\n",
    "(layer 1 so L = 1) number of neurons in hidden layers = 5\n",
    "(layer 2 so L = 2) number of neurons in output layers = 1\n",
    "\n",
    "I hope this all makes sense! Let’s set the parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def setParameters(X, Y, hidden_size):\n",
    "    np.random.seed(3)\n",
    "    \n",
    "    input_size = X.shape[0] # number of neurons in input layer\n",
    "    output_size = Y.shape[0] # number of neurons in output layer.\n",
    "    \n",
    "    W1 = np.random.randn(hidden_size, input_size) * np.sqrt(2/input_size)\n",
    "    b1 = np.zeros((hidden_size, 1))\n",
    "    \n",
    "    W2 = np.random.randn(output_size, hidden_size) * np.sqrt(2/hidden_size)\n",
    "    b2 = np.zeros((output_size, 1))\n",
    "    return {'W1': W1, 'W2': W2, 'b1': b1, 'b2': b2}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We define W1, b1, W2, and b2. It doesn’t hurt if you set your biases to zero at first. However, be very careful when initializing weights. Never set the weights to zero at first. Why exactly? Well, if you do, then in Z = Wx + b, Z will always be zero. If you are building a multi-layer neural network, neurons in every layer will behave like there is one neuron. So how do we initialize weights at first? I use he initialization.\n",
    "\n",
    "![Sigmoid](./Images/2nn1.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.69781336, -0.04322357, -1.03899755, -1.1808012 ,  0.50424918]])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hidden_size = 5\n",
    "output_size = 1\n",
    "np.random.randn(output_size, hidden_size) * np.sqrt(2/hidden_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or just use this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-0.00464412, -0.00661853,  0.00751112, -0.00775691, -0.00506107]])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.random.randn(output_size, hidden_size) * 0.01"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Sigmoid](./Images/2nn2.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def forwardPropagation(X, params):\n",
    "    Z1 = np.dot(params['W1'], X) + params['b1']\n",
    "    A1 = np.tanh(Z1)\n",
    "  \n",
    "    Z2 = np.dot(params['W2'], A1) + params['b2']\n",
    "    y = sigmoid(Z2)  \n",
    "    return y, {'Z1': Z1, 'Z2': Z2, 'A1': A1, 'y': y}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Why we are storing {‘Z1’: Z1, ‘Z2’: Z2, ‘A1’: A1, ‘y’: y}? Because we will use them when back-propagating.\n",
    "\n",
    "Cost function\n",
    "\n",
    "We just looked at forward propagation and obtained a prediction (y). We calculate it using a cost function.\n",
    "\n",
    "![Cost](./Images/2nn3.png)\n",
    "![Cost](./Images/2nn4.png)\n",
    "\n",
    "We update our parameters and find the best parameter that gives us the minimum possible cost. I’m not going to delve into derivatives, but note that on the graph above, if you are on the right sight of the parabola, the derivative (slope) will be positive, so the parameter will decrease and move left approaching the parameter that returns the minimum cost. On the left side, the slope will be negative, so the parameter increases towards the value we want. Let’s look at the cost function we will use:\n",
    "\n",
    "![Cost](./Images/2nn5.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def cost(predict, actual):\n",
    "    m = actual.shape[1]\n",
    "    cost__ = -np.sum(np.multiply(np.log(predict), actual) + np.multiply((1 - actual), np.log(1 - predict)))/m\n",
    "    return np.squeeze(cost__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Backpropagation\n",
    "\n",
    "We’ve found the cost, now let’s go back and find the derivative of our weights and biases. In a future piece, I plan to show you how to derivate them step by step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def backPropagation(X, Y, params, cache):\n",
    "    m = X.shape[1]\n",
    "    \n",
    "    dy = cache['y'] - Y\n",
    "    dW2 = (1 / m) * np.dot(dy, np.transpose(cache['A1']))\n",
    "    db2 = (1 / m) * np.sum(dy, axis=1, keepdims=True)\n",
    "    \n",
    "    dZ1 = np.dot(np.transpose(params['W2']), dy) * (1-np.power(cache['A1'], 2))\n",
    "    dW1 = (1 / m) * np.dot(dZ1, np.transpose(X))\n",
    "    db1 = (1 / m) * np.sum(dZ1, axis=1, keepdims=True)\n",
    "    return {\"dW1\": dW1, \"db1\": db1, \"dW2\": dW2, \"db2\": db2}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What are the params and cache in def backPropagation(X, Y, params, cache)? When we use forward propagation, we store values to use during backpropagation. Params are parameters (weight and biases).\n",
    "Updating Parameters\n",
    "\n",
    "Now that we have our derivatives, we can use the equation below:\n",
    "![derivatives](./Images/2nn6.png)\n",
    "\n",
    "In that equation, alpha (α) is the learning rate hyperparameter. We need to set it to some value before the learning begins. The term to the right of the learning rate is the derivative. We know alpha and derivatives, let’s update our parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def updateParameters(gradients, params, learning_rate = 1.2):\n",
    "    W1 = params['W1'] - learning_rate * gradients['dW1']\n",
    "    b1 = params['b1'] - learning_rate * gradients['db1']\n",
    "    W2 = params['W2'] - learning_rate * gradients['dW2']\n",
    "    b2 = params['b2'] - learning_rate * gradients['db2']\n",
    "    return {'W1': W1, 'W2': W2, 'b1': b1, 'b2': b2}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All About Loops\n",
    "\n",
    "We need to run many interations to find the parameters that return the minimum cost. Let’s loops it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fit(X, Y, learning_rate, hidden_size, number_of_iterations = 5000):\n",
    "    params = setParameters(X, Y, hidden_size)\n",
    "    cost_ = []\n",
    "    for j in range(number_of_iterations):\n",
    "        y, cache = forwardPropagation(X, params)\n",
    "        costit = cost(y, Y)\n",
    "        gradients = backPropagation(X, Y, params, cache)\n",
    "        params = updateParameters(gradients, params, learning_rate)\n",
    "        cost_.append(costit)\n",
    "    return params, cost_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hidden_size means the number of neurons in the hidden layer. It looks like a hyperparameter. Because you set it before learning begins! What return params, cost_ tells us. params are the best parameters we found and cost_ is just cost we estimated in every episode.  \n",
    "\n",
    "Let’s Try Our Code!\n",
    "\n",
    "Use sklearn to create a dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sklearn.datasets\n",
    "X, Y = sklearn.datasets.make_moons(n_samples=500, noise=.2)\n",
    "X, Y = X.T, Y.reshape(1, Y.shape[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "X input, Y actual output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "params, cost_ = fit(X, Y, 0.3, hidden_size, 5000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I set the learning rate to 0.3, the number of neurons in the hidden layer was ser to 5 and the number of iterations to 5000.\n",
    "\n",
    "Feel free to try with different values.\n",
    "\n",
    "Let’s draw a graph showing how the cost function changed with every episode:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x1a64ee228c8>]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsIAAAHSCAYAAADmLK3fAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXxcdb3/8fdnlmxN0jXdW9KWUigIFEKLoIILUlwoKmArXAGXKlhR0esF7/35u6LXn+KCCrhUxYsglIqAFSsVBGSRpekGXWgbWmjTNd2Tplnn+/tjziSTkNJpO+mZ5Pt6Ph7zmDnfczLzSU8J73z7Od9jzjkBAAAAvomEXQAAAAAQBoIwAAAAvEQQBgAAgJcIwgAAAPASQRgAAABeIggDAADAS7GwPnjQoEGuvLw8rI8HAACAJxYtWrTDOVfWeTy0IFxeXq7KysqwPh4AAACeMLM3uhqnNQIAAABeIggDAADASwRhAAAAeIkgDAAAAC8RhAEAAOAlgjAAAAC8RBAGAACAlwjCAAAA8BJBGAAAAF4iCAMAAMBLBGEAAAB4iSAMAAAALxGEAQAA4CWCMAAAALxEEAYAAICXCMIAAADwkldBuKU1ob31zWpuTYRdCgAAAELmVRBe9MZunXbz37Vw/a6wSwEAAEDIvArCKS7sAgAAABA6r4KwmYVdAgAAAHKEV0E4xTElDAAA4D2vgnBqQtjRHAEAAOA9v4Jw2AUAAAAgZ3gVhFNojQAAAIBXQbi9NQIAAAC+8yoI0xwBAACAFM+CcJKjNwIAAMB7XgVhWiMAAACQ4lcQDrsAAAAA5AyvgnAbpoQBAAC851UQTt1imRtqAAAAwK8gHHYBAAAAyBleBeEUFo0AAACAV0G4bdUIgjAAAID3/ArCNEcAAAAg4FUQTmFCGAAAAF4F4fbWCKIwAACA77wKwgAAAECKl0GY+WAAAABkFITNbKqZrTazKjO7sYv9t5rZ0uCxxsz2ZL/Uo8eqEQAAAEiJHeoAM4tKukPSBZKqJS00s3nOuZWpY5xzX0k7/ouSJnVDrUeNVSMAAACQksmM8GRJVc65dc65JklzJE17i+NnSLovG8V1H6aEAQAAfJdJEB4haWPadnUw9iZmdpykMZKeOPrSss+YEAYAAEAgkyDcVXw82JTqdEkPOOdau3wjs5lmVmlmlTU1NZnWmHX0CAMAACCTIFwtaVTa9khJmw9y7HS9RVuEc262c67COVdRVlaWeZVZ0nax3DH/ZAAAAOSaTILwQknjzWyMmeUpGXbndT7IzCZI6i/p+eyWmD1cLAcAAICUQwZh51yLpFmSFkhaJWmuc26Fmd1sZhenHTpD0hzXA27blvsVAgAAoLsdcvk0SXLOzZc0v9PYNztt/3f2yuoe7a0RJGEAAADfeXVnORojAAAAkOJVEE6hNQIAAABeBWFWjQAAAECKV0GY5ggAAACkeBaEk3rAwhYAAADoZl4FYW6xDAAAgBS/gnDYBQAAACBneBWEU+iMAAAAgFdB2ILeCG6oAQAAAL+CcNgFAAAAIGd4FYRTaI0AAACAV0G47YYaBGEAAADv+RWEaY4AAABAwKsgnMKEMAAAALwKwtxQAwAAACleBeEUbrEMAAAAP4Nw2AUAAAAgdF4FYVojAAAAkOJVEG7DlDAAAID3vArC3GIZAAAAKX4F4bALAAAAQM7wKginsGgEAAAAvArCbbdYDrcMAAAA5AC/gjDNEQAAAAh4FYRTaI0AAACAV0G4vTWCJAwAAOA7v4Jw2AUAAAAgZ3gVhFNojQAAAIBfQZhVIwAAABDwKgizagQAAABSvArCbeiNAAAA8J5XQZgbagAAACDFryAcdgEAAADIGV4F4RQ6IwAAAOBVELagN8KRhAEAALznVxAOuwAAAADkDK+CcArzwQAAAPAqCBtTwgAAAAh4FYRTaBEGAACAV0E4dWc5cjAAAAC8CsJcLQcAAIAUv4JwgOXTAAAA4FUQ5mI5AAAApPgVhMMuAAAAADnDqyCcQmcEAAAAvArCbbdYZt0IAAAA7/kVhMMuAAAAADnDqyCcQmsEAAAAvArCqVUjyMEAAADwKwjTHAEAAIBARkHYzKaa2WozqzKzGw9yzOVmttLMVpjZvdktM7tojQAAAEDsUAeYWVTSHZIukFQtaaGZzXPOrUw7ZrykmySd65zbbWaDu6vgo9HeGkESBgAA8F0mM8KTJVU559Y555okzZE0rdMxn5V0h3NutyQ557Znt0wAAAAguzIJwiMkbUzbrg7G0p0g6QQze87MXjCzqdkqsDvQGgEAAIBDtkao6+V3O0fJmKTxks6XNFLSM2Z2inNuT4c3MpspaaYkjR49+rCLPVrGtXIAAAAIZDIjXC1pVNr2SEmbuzjmz865ZufcekmrlQzGHTjnZjvnKpxzFWVlZUda8xFj1QgAAACkZBKEF0oab2ZjzCxP0nRJ8zod87Ckd0uSmQ1SslViXTYLzSZHbwQAAID3DhmEnXMtkmZJWiBplaS5zrkVZnazmV0cHLZA0k4zWynpSUn/7pzb2V1FHylaIwAAAJCSSY+wnHPzJc3vNPbNtNdO0g3BI+cxIQwAAADP7iyXRA4GAACAX0GY3ggAAAAEvArCKbRGAAAAwKsg3N4aQRIGAADwnV9BmM4IAAAABLwKwim0RgAAAMCrIJy6WI4cDAAAAK+CMAAAAJDiZxCmNwIAAMB73gVhM1ojAAAA4GMQDrsAAAAA5ATvgrBEZwQAAAA8DMJmxg01AAAA4GEQDrsAAAAA5ATvgrBEawQAAAA8DMKsGgEAAADJxyBMcwQAAADkYRCWaI0AAACAj0HYxKoRAAAA8C8I0xgBAAAAycMgLImr5QAAAOBfEDamhAEAACAPg7DEhDAAAAA8DMImk2PZCAAAAO/5F4RpjQAAAIA8DMIS6wgDAADAwyBsokcYAAAAPgZheiMAAAAgD4OwRGsEAAAAPAzCydYIkjAAAIDvvAvC3GMZAAAAko9BWLRGAAAAwMMgzIQwAAAAJB+DMKtGAAAAQB4GYUncYhkAAAD+BWEzbqgBAAAAH4Nw2AUAAAAgJ3gXhCVWjQAAAICHQdjMuKEGAAAAPAzCYRcAAACAnOBdEJZojQAAAICHQZhlhAEAACB5GIQllk8DAACAl0HYaI0AAACAf0GY1ggAAABIHgbhJKaEAQAAfOddEDaxagQAAAB8DMK0RgAAAEAeBmGJGWEAAAB4GIRN3GIZAAAAPgZhWiMAAACgDIOwmU01s9VmVmVmN3ax/2ozqzGzpcHjM9kvNXtojQAAAEDsUAeYWVTSHZIukFQtaaGZzXPOrex06P3OuVndUGNWmVg8DQAAAJnNCE+WVOWcW+eca5I0R9K07i2r+xi9EQAAAFBmQXiEpI1p29XBWGcfM7OXzewBMxvV1RuZ2UwzqzSzypqamiMoNzsS9EYAAAB4L5Mg3NUUauck+RdJ5c65UyU9Lumurt7IOTfbOVfhnKsoKys7vEqzJBIRvREAAADIKAhXS0qf4R0paXP6Ac65nc65xmDz15LOzE552RcxY0YYAAAAGQXhhZLGm9kYM8uTNF3SvPQDzGxY2ubFklZlr8TsSgbhsKsAAABA2A65aoRzrsXMZklaICkq6U7n3Aozu1lSpXNunqTrzexiSS2Sdkm6uhtrPipm9AgDAAAggyAsSc65+ZLmdxr7ZtrrmyTdlN3SukfEjHWEAQAA4N+d5SLMCAMAAEBeBmEulgMAAICHQdi4WA4AAADyMAhHTHLMCAMAAHjPwyDMjDAAAAC8DMJcLAcAAAAPgzA9wgAAAJA8DML0CAMAAEDyMgizfBoAAAB8DcKJsKsAAABA2LwLwsbFcgAAAJCHQThiJnIwAAAAvAvCzAgDAABA8jAIc7EcAAAAJA+DsJlEDAYAAIB3QZhbLAMAAEDyMghzQw0AAAB4GYTpEQYAAICHQdi4oQYAAADkYRCOsHwaAAAA5GUQ5oYaAAAA8DEIR5gRBgAAgIdB2LhYDgAAAPIwCNMaAQAAAMnLIExrBAAAALwMwtxZDgAAAB4GYWNGGAAAAPIwCNMjDAAAAMnLIMyMMAAAALwMwiyfBgAAAA+DsHGxHAAAAORhEI6Y5JgRBgAA8J6HQZgZYQAAAHgZhLlYDgAAAB4GYTNTgilhAAAA73kXhFlHGAAAAJKXQZjWCAAAAPgYhCNcLAcAAAAPg7AxIwwAAAB5GITpEQYAAIDkZRBmRhgAAABeBmFTK0EYAADAe14GYefEWsIAAACe8y4Ix6MmScwKAwAAeM67IByNJL/lVmaEAQAAvOZdEI5FkjPCza2JkCsBAABAmPwLwqnWCGaEAQAAvOZfEA5mhFsIwgAAAF7zLgjTIwwAAADJwyCcao2gRxgAAMBvGQVhM5tqZqvNrMrMbnyL4y41M2dmFdkrMbtSrRHMCAMAAPjtkEHYzKKS7pB0kaSJkmaY2cQujiuRdL2kF7NdZDZF6REGAACAMpsRniypyjm3zjnXJGmOpGldHPdtSbdIashifVkXj9IjDAAAgMyC8AhJG9O2q4OxNmY2SdIo59wjWaytW0RZRxgAAADKLAhbF2Nt06lmFpF0q6SvHvKNzGaaWaWZVdbU1GReZRbRIwwAAAApsyBcLWlU2vZISZvTtksknSLpKTN7XdLZkuZ1dcGcc262c67COVdRVlZ25FUfBXqEAQAAIGUWhBdKGm9mY8wsT9J0SfNSO51ze51zg5xz5c65ckkvSLrYOVfZLRUfpVSPcEsrQRgAAMBnhwzCzrkWSbMkLZC0StJc59wKM7vZzC7u7gKzrX1GmB5hAAAAn8UyOcg5N1/S/E5j3zzIsecffVndhx5hAAAASF7eWS5ojSAIAwAAeM2/IJxqjaBHGAAAwGveBeFoW2sEPcIAAAA+8y4Ix6MsnwYAAAAPg3A0wvJpAAAA8DAI58eS33JTC60RAAAAPvM2CDe0tIZcCQAAAMLkXxCORyVJjc3MCAMAAPjMvyAczAg3MiMMAADgNe+CcCxiipjUSI8wAACA17wLwmamgnhUDc3MCAMAAPjMuyAsJdsjmBEGAADwm6dBOMrFcgAAAJ7zMwjHI1wsBwAA4Dk/g3AsogZmhAEAALzmaRCOMiMMAADgOU+DMBfLAQAA+M7LIFwQjxKEAQAAPOdlEE7OCNMaAQAA4DM/g3Cci+UAAAB852UQLsqLaX9jS9hlAAAAIEReBuGSgphqGwjCAAAAPvM0CMdV19ii1oQLuxQAAACExMsgXFoQkyTV0R4BAADgLS+DcEkQhGsbmkOuBAAAAGHxNAjHJYk+YQAAAI95GoRTM8IEYQAAAF95GoSTM8J1jbRGAAAA+MrLINyvMBmEd+0nCAMAAPjKyyBcVpIvSdpR1xhyJQAAAAiLl0G4T35MffKi2r6PIAwAAOArL4OwlJwVrmFGGAAAwFveBuHBJQWqqW0IuwwAAACExNsgPKRvgTbvIQgDAAD4ytsgPGZgkap316upJRF2KQAAAAiBt0F4bFmxEk7asKs+7FIAAAAQAm+D8JhBfSRJ63fsD7kSAAAAhMHbIDy2LBmEV2/dF3IlAAAACIO3QbikIK7jBxdr8YY9YZcCAACAEHgbhCXpzNH9tXjDbiUSLuxSAAAAcIx5HYTPHjdAe+qbtayaWWEAAADfeB2E33PiEMUipkeXbw27FAAAABxjXgfhvoVxvXP8ID20ZBPrCQMAAHjG6yAsSVefO0bbaxv156Wbwi4FAAAAx5D3Qfhd4wfp5OGl+tHf16iusSXscgAAAHCMeB+EzUzfvuQUbatt0P/8dWXY5QAAAOAY8T4IS9IZo/vr2vPG6b6XNurOZ9eHXQ4AAACOgVjYBeSKr75/gqq21+nmR1Yq4Zw+886xYZcEAACAbsSMcCAaMd3+iTP0gbcN1Xf+ukrf+ssKtbSykgQAAEBvxYxwmrxYRD+bPklDS1/Vnc+tV9X2Ot0+4wz1LYqHXRoAAACyLKMZYTObamarzazKzG7sYv/nzewVM1tqZs+a2cTsl3psxKIRffPDE3XLx07VC+t26pKfP6f1O/aHXRYAAACy7JBB2Myiku6QdJGkiZJmdBF073XOvc05d7qkWyT9OOuVHmOXnzVK9372bO090KyP/eJfWrqR2zADAAD0JpnMCE+WVOWcW+eca5I0R9K09AOcc/vSNvtIctkrMTxnlQ/Qn649R8X5Mc2Y/YKeeHVb2CUBAAAgSzIJwiMkbUzbrg7GOjCzL5jZa0rOCF+fnfLCN2ZQH/3p2nN0/OBizfz9Ij2+kjAMAADQG2QShK2LsTfN+Drn7nDOjZP0H5L+q8s3MptpZpVmVllTU3N4lYaorCRff/jsFJ08vFTX/WGxnly9PeySAAAAcJQyCcLVkkalbY+UtPktjp8j6ZKudjjnZjvnKpxzFWVlZZlXmQNKC+L6/aem6IShxfrc3Yv0/Gs7wy4JAAAARyGTILxQ0ngzG2NmeZKmS5qXfoCZjU/b/KCktdkrMXf0LYrr7k9N0egBRZp5d6XWbKsNuyQAAAAcoUMGYedci6RZkhZIWiVprnNuhZndbGYXB4fNMrMVZrZU0g2Sruq2ikPWv0+e/veas1QQj+qa3y3Utn0NYZcEAACAI2DOhbPAQ0VFhausrAzls7Nh+aa9uvxXz2vMoD6a+7m3q08+9yYBAADIRWa2yDlX0XmcWywfoVNG9NXPrzhDq7bs0w1zlyqR6BUrxgEAAHiDIHwUzp8wWP/5wYlasGKbbnuiKuxyAAAAcBj49/yj9Klzy7Vy8z7d+vganTisRBeePDTskgAAAJABZoSPkpnpfz5yik4b1U833L9Uq7eykgQAAEBPQBDOgoJ4VL+68kwV5cf02d9Xam99c9glAQAA4BAIwlkytG+Bfnnlmdqy94C++sdlCms1DgAAAGSGIJxFZx7XXzdddJIeX7VNs59eF3Y5AAAAeAsE4Sy75txyXXTKUN2yYLVeWr8r7HIAAABwEAThLDMzff/SUzWqf6G+eN9i7ahrDLskAAAAdIEg3A1KC+L6+RVnak99s740Z4laudkGAABAziEId5OJw0t187ST9VzVTt32xNqwywEAAEAnBOFudHnFKH1k0gj97B9rVfk6/cIAAAC5hCDcjcxMN087WSP7F+lLc5Zq7wHWFwYAAMgVBOFuVlIQ189mTNK2fQ36xkOvsL4wAABAjiAIHwOnj+qnr1xwgv768hb9cVF12OUAAABABOFj5vPnjdPbxw7Uf89boXU1dWGXAwAA4D2C8DESjZhu/fjpyotFdP2cJWpsaQ27JAAAAK8RhI+hoX0LdMvHTtXyTfv0wwWrwy4HAADAawThY+z9Jw/VlWeP1q+fWa9n1taEXQ4AAIC3CMIh+M8PTNT4wcW6Ye4y7eQWzAAAAKEgCIegMC+qn82YpL0HmvX1B15mSTUAAIAQEIRDctKwUt100Yn6x6vbdfcLb4RdDgAAgHcIwiG6+pxynT+hTN/56yqt3lobdjkAAABeIQiHyMz0w8tOU2lBXNfft0QNzSypBgAAcKwQhEM2qDhfP7zsVK3eVqv/N39V2OUAAAB4gyCcA86fMFiffscY3fX8G3ri1W1hlwMAAOAFgnCO+PrUCTppWKm+9seXtX1fQ9jlAAAA9HoE4RyRH4vqthmnq76pRV/94zIlEiypBgAA0J0Iwjnk+MEl+uaHTtYza3fozufWh10OAABAr0YQzjEzJo/ShScP0fcffVXLN+0NuxwAAIBeiyCcY8xM3/voqRrQJ0/Xz1mi+qaWsEsCAADolQjCOah/nzzdevnpWr9jv27+y8qwywEAAOiVCMI56pzjB+na88ZpzsKNenBxddjlAAAA9DoE4Rx2wwUnaMqYAfrGQ6/o1a37wi4HAACgVyEI57BYNKLbPjFJJQVxXXfPYtU10i8MAACQLQThHDe4pEC3z5ikN3bV6z8eeFnOsb4wAABANhCEe4ApYwfq3y+coL++skX/+6/Xwy4HAACgVyAI9xAz3zlW7ztpiP7nr6u06I3dYZcDAADQ4xGEe4hIxPSjy07T8H6FuvaeRdq2ryHskgAAAHo0gnAP0rcortmfPFN1jS2aefciNTS3hl0SAABAj0UQ7mFOHFqqWz9+upZt3KNvPPgKF88BAAAcIYJwD3ThyUN1wwUn6MElm/TrZ9aFXQ4AAECPRBDuob74nuP1gbcN1ff+9qqeWr097HIAAAB6HIJwD2Vm+uFlp2nC0FJ98d4lWr21NuySAAAAehSCcA9WlBfTb66qUGFeVNf87iVWkgAAADgMBOEebkS/Qt159Vnae6BZ1/xuIbdhBgAAyBBBuBc4ZURf3XHFGVq9rVbX/WGxmlsTYZcEAACQ8wjCvcT5Ewbrux85RU+vqdH/eXg5y6oBAAAcQizsApA9Hz9rtDbtPqCfPVGlQcX5+tqFE8IuCQAAIGcRhHuZr1xwgmrqmnT7k1UqLYxp5rvGhV0SAABATiII9zJmpu9ccopqG5r13fmvqqQgrhmTR4ddFgAAQM7JqEfYzKaa2WozqzKzG7vYf4OZrTSzl83sH2Z2XPZLRaaiEdOPLz9d755Qpm889IrmLdscdkkAAAA555BB2Myiku6QdJGkiZJmmNnEToctkVThnDtV0gOSbsl2oTg8ebGIfn7FmTqrfIBuuH+p/rFqW9glAQAA5JRMZoQnS6pyzq1zzjVJmiNpWvoBzrknnXP1weYLkkZmt0wcicK8qH57VYVOGlaqa+9ZTBgGAABIk0kQHiFpY9p2dTB2MJ+W9LeudpjZTDOrNLPKmpqazKvEESspiOueT0/RhKEl+vw9i/T4SsIwAACAlFkQti7Gulyk1syulFQh6Qdd7XfOzXbOVTjnKsrKyjKvEkelb1Fc93xmiiYOK9W1f1ikv6/YGnZJAAAAocskCFdLGpW2PVLSm66+MrP3SfpPSRc75xqzUx6ypW9hXHd/ZopOHt5X1/1hsR5dThgGAAB+yyQIL5Q03szGmFmepOmS5qUfYGaTJP1KyRC8PftlIhtKC+L6/acn620j+2rWvYv156Wbwi4JAAAgNIcMws65FkmzJC2QtErSXOfcCjO72cwuDg77gaRiSX80s6VmNu8gb4eQlRbE9ftPTdaZx/XXl+9fqt8//3rYJQEAAITCnOuy3bfbVVRUuMrKylA+G1JDc6tm3btEj6/apq+87wRd/97jZdZVOzgAAEDPZmaLnHMVncczuqEGep+CeFS/vPIMfeyMkbr18TX61l9WKpEI55ciAACAMHCLZY/FohH94NJT1a8ort8+u1576pt0y6WnKS/G70cAAKD3Iwh7LhIx/dcHT9KAPnn6wYLV2rqvQb+6skJ9i+JhlwYAANCtmPqDzExfePfx+snHT9fiN/boo794Tht31R/6CwEAAHowgjDaXDJphO7+9GTtqGvSJXc8pyUbdoddEgAAQLchCKODKWMH6sHrzlGf/Jimz35Bf3tlS9glAQAAdAuCMN5kXFmxHrruHE0cXqrr7l2s259Yq7CW2QMAAOguBGF0aWBxvu777Nmadtpw/fDvazTr3iWqb2oJuywAAICsIQjjoAriUd368dN100Unav7yLbr0F8+rejcX0QEAgN6BIIy3ZGb63HnjdOfVZ2nj7npNu/05vbhuZ9hlAQAAHDWCMDLy7gmD9fAXzlXforiu+M2LuueFN8IuCQAA4KgQhJGxcWXFevgL5+qd4wfpvx5erhv/9LIamlvDLgsAAOCIEIRxWEoL4vrNVWdp1ruP15yFG3XZL5/n5hsAAKBHIgjjsEUjpq9dOEG/+WSFXt+5Xx++/Vk9tXp72GUBAAAcFoIwjtj7Jg7RX2a9Q0NLC3TN/y7UTx5fo0SC9YYBAEDPQBDGUSkf1EcPXXeuPjJphH7y+Fp96q6F2lPfFHZZAAAAh0QQxlErzIvqR5edpu9ccoqeq9qhD932rJZv2ht2WQAAAG+JIIysMDNdefZxmvu5t6s14fTRX/xLc17awK2ZAQBAziIII6smje6vR774Dk0uH6AbH3xFN8xdpv2N3JoZAADkHoIwsm5gcb7u+tRk3XDBCfrz0k368O3P6tWt+8IuCwAAoAOCMLpFNGK6/r3jdc9npqi2oUXTbn+OVgkAAJBTCMLoVueMG6T5179TZwWtEl++f6nqaJUAAAA5gCCMbldWkmyV+OoFJ+gvyzbr4tue1aottEoAAIBwEYRxTEQjpi++d7zu/ezZqmts0SV3PKd7X6RVAgAAhIcgjGPq7LEDNf9L79TkMQP0jYde0az7lmhvfXPYZQEAAA8RhHHMDSrO113XTNbXp07QguVbNfWnT+tfVTvCLgsAAHiGIIxQRCKm684/Xg9ed44K41Fd8dsX9d35q9TY0hp2aQAAwBMEYYTq1JH99Mj179AVU0Zr9tPrdMkd/9KabbVhlwUAADxAEEboivJi+s4lb9Nvr6rQ9n0N+tBtz+q3z65XIsGFdAAAoPsQhJEz3nvSED365XfpnccP0rcfWanLf/W8XqupC7ssAADQSxGEkVPKSvL1m6sq9OPLT9Pa7XW66KfP6BdPvaaW1kTYpQEAgF6GIIycY2b66Bkj9dgN79J7Txys7z/6qj7y839xEw4AAJBVBGHkrMElBfrFlWfq51ecoS17D+jDtz2r785fxS2aAQBAVhCEkfM+8LZheuwr5+nSM0dq9tPr9N4fPaV5yzZzVzoAAHBUCMLoEfr3ydP3PnaqHrruHA0uKdD19y3RJ379otay1BoAADhCBGH0KJNG99fDXzhX37nkFK3csk9Tf/qM/s/Dy7WjrjHs0gAAQA9DEEaPE42Yrjz7OD3x1fP0icmjde9LG3TeLU/qtn+sVX0T/cMAACAzFlafZUVFhausrAzls9G7vFZTp1sefVULVmzTkNJ8feV9J+hjZ45UPMrveQAAQDKzRc65is7jJAX0eOPKivWrf6vQHz//dg3vV6gbH3xF7/nRU7p/4QY1s/4wAAA4CGaE0as45/Tk6u36yeNr9XL1Xo3sX6hZ7z6eGWIAADx2sBlhgjB6Jeecnlpdo588vkbLqvdqRL9CXXNuuT5+1iiVFMTDLg8AABxDBGF4KRWIf/HP1/TS+hOOQOIAAA9ASURBVF0qyY9pxpTRuvqccg3vVxh2eQAA4BggCMN7yzbu0a+fWae/Ld8qU/JGHf/29uNUcVx/mVnY5QEAgG5CEAYC1bvr9bvnXtfchRtV29ii8YOLdcWU0frIGSPVt5C2CQAAehuCMNBJfVOLHlm2RX94aYOWbdyjgnhEHzp1uC47c6TOKh+gSIRZYgAAegOCMPAWlm/aq3tf2qA/L9mk/U2tGtGvUNNOH66PTBqh8UNKwi4PAAAcBYIwkIH6phY9tnKbHl6ySU+v3aHWhNPJw0t1yekjNPWUoRo1oCjsEgEAwGEiCAOHaUddox5ZtlkPLd2sZRv3SJImDivVhScP1dRThuqEIcVcZAcAQA9AEAaOwoad9fr7yq16dPlWLdqwW85J5QOLdMHEITp/wmBVlPdXfiwadpkAAKALBGEgS7bXNuixldv06PKtenHdLjW1JlQYj+qccQN13oQynXdCmY4b2CfsMgEAQOCogrCZTZX0U0lRSb9xzn2v0/53SfqJpFMlTXfOPXCo9yQIozfY39iiF9bt1D/X1Oifa2r0xs56SdLoAUU6e+wATRkzUFPGDtDI/vQWAwAQliMOwmYWlbRG0gWSqiUtlDTDObcy7ZhySaWSviZpHkEYvnp9x349vbZGz6zdoZfW79LeA82SpBH9CjVlzABNGTtAFeUDNGZgH5ZnAwDgGDlYEI5l8LWTJVU559YFbzRH0jRJbUHYOfd6sC+RlWqBHqp8UB+VD+qjT769XImE0+pttXpx3U69uH6X/rmmRg8u2SRJKi2I6bRR/XR68DhtVD8NKs4PuXoAAPySSRAeIWlj2na1pCndUw7Qe0QippOGleqkYaW6+twxcs6panudFm/YraUb92jpxr2648kqJYJ/lBnZv1Cnjeynk4aVtH3dsL4FrEwBAEA3ySQId/V/4SO6ws7MZkqaKUmjR48+krcAeiwz0/ghJRo/pEQfPyv597++qUXLN+3T0o27tWzjXr2yaa/++sqWtq/pVxTXiUPbg/GJQ0s0tqxYxfmZ/KcLAADeSib/N62WNCpte6SkzUfyYc652ZJmS8ke4SN5D6A3KcqLafKYAZo8ZkDbWG1Ds1ZvrdWqLfu0ckvyec5LG3WgubXtmGF9CzSurFjjyvpo3OBiHV9WrHGDizW4JJ8ZZAAAMpRJEF4oabyZjZG0SdJ0SZ/o1qoAj5UUxFVRnryoLqU14fTGzv1as61Or9UEj+11+tPiTaprbGk7rjg/puMGFmn0gORj1ID218P7FSovFgnjWwIAICcdMgg751rMbJakBUoun3anc26Fmd0sqdI5N8/MzpL0kKT+kj5sZt9yzp3crZUDHolGTGPLijW2rLjDuHNO22sb9dr2OlUF4XjDrnqt2Varf7y6XU0t7devRkwa1rdQowcUaWT/Qg3rV6jhfQs0tG+Bhvcr1NC+BSotiB/rbw0AgNBwQw2gl0okkiF5w676tsfG4Ll6d7221zaq83/+xfkxDUuF477JcDyktEBlJfkqK8nXoOI8DSrOV0Gcu+gBAHqOo1k+DUAPFImYhgahNr0HOaW5NaHttY3asueANu9t0Na9B7R5T4O27m3Qlr0H9OrWWu2oe3NYlpLLvyWDcX5bSE5tDyrOU7+iPA0oylP/ojyVFMRYMxkAkJMIwoCn4tGIRvQr1Ih+hQc9pqkloZ37G1VT26gddcnn1GNHXZNqahu1YvM+7ahtVG1ar3K6iEn9i/LUryiuAX3aQ3K/PnH1T70uiqtvYVylhXGVFMRUWhhXcR4BGgDQvQjCAA4qLxbRsL6FGtb34GE55UBTq3bUNWrn/ibtrm/S7v1N2l3fHDynxpq1cVe9Xq7eo937m9XUevB78JhJJfkxlRQkA3JpEJBLCmIqTR8rSI71yU8+ivNjKsqLJp/zo8qP0cYBAOgaQRhAVhTmRTUqWKkiE8451Te1tgXkfQ3N2negWbUNLW2v97W9Tj5v3FXftr+2oesZ6M7iUVNRXjIg98mPtr1OD8t98mMqzoupKBgvjEdVEI+qMHhdGI+qMC+i/Fj7WEE8qigz1gDQoxGEAYTCzNpmcUf2P/yvb0041TW2tIXn/U0tqmtsUX1jq/Y3JreTz8nttn3BeE1tY4dj0lfYyFReLJIWlKPKj0U6hOeCttCcDNH5seRzXiyi/Fik7Tk/HlVeNKL8eET5qec3HRdtex2LGOtFA0AWEIQB9EjRiKlvYbK3OBuaWhKqb2rRgeZWHWhq1YHmVjU0t+pAUyL53Jw+lrbd1KqG5sSb9u890KwDza1qbE6+b1NLQo0tCbUkjn6lnoipLRy3helYRHmxqPKipng0oljwnBeNtG2nv45Hk6E6HjXFIh1fx2MR5XV6nfy65DF5nV6nv2cskvy6aNQUi5iiESO4A8hZBGEAUDJY5sXy1K+bP6c14dTUkgiCcasag4Dc2NLaFpY7PrePd3VM5/Hm1oRaWp2aWhOqa2zpsJ163dya/PrmVqeWRPK5u0XTQnHqORYE5/TxeDTS6bhkyG4fC8J2tON2NGqKp7bTjk99fcRM0YiC5+S+SMQUtfbnaCT9dfuxHfYHzx32t42lvTZTJKK012nP6fuD4/lFAQgHQRgAjqFoxJLtE3lRSblxAxPnnJqDgJwempMPl/a643ZTSypIJ9Tc4tScSKg14dTS6pLPCaeW1uQseGq7NQje6dstHb4mkfa17e9/oDm5P/m1ae/Z2vFrWltd277mRKLL5f9ykZneHMqt/ReI9EAdseS+SNt28rWlj1uy/Sj1Ph33mczSg/zB90dMb/rMzp9zOPvNOh+rtl8gDrY/Vael1Zc6NjVmkiIRyWRS+lhwjFlyX/r3mvzdo9OY2o+PdHp/S32mOu6Tkt9jh89TF/VZqrauP+/N9fHL0bFAEAYAz5mZ8mLWK2/BnUi0B+NWlwzKrS65nQie018nn9Vxv3NKJNJfq4uxzu+rLsY67e/i89vfVx3GUnUnEk5O7fU5lzwu+Uj+UpNw7V/beX9rItG2P5FoP7b9a12X+w/2OYeqozULrUA+S4Xq9oDdKagrPWAfTnDvGLzTj+8Y0Nt/aWgbU3vIb3stvam+rl5HzHTXpyaH9KfZNYIwAKDXikRMeazuEar0cN4xNCsI6wffn0h0DOddBX2pY1h3klwQxp3U9stDwjnJKRnU1R7YXVCjS+0LPkNtx6S/b/I4p+QvRB3ft/1zgqG0921/7lhv8r3a6u1Un1K/WKTVkn58en1v+efQqb6O9Xb159Bei970Z6U3/Vm4TrUl/2wSHY9VMjjnGoIwAADoNmamqEnRnIxB8F3v+3cwAAAAIAMEYQAAAHiJIAwAAAAvEYQBAADgJYIwAAAAvEQQBgAAgJcIwgAAAPASQRgAAABeIggDAADASwRhAAAAeIkgDAAAAC8RhAEAAOAlgjAAAAC8RBAGAACAlwjCAAAA8BJBGAAAAF4iCAMAAMBLBGEAAAB4yZxz4XywWY2kN0L5cGmQpB0hfTaODc6xHzjPfuA8+4Hz3PuFeY6Pc86VdR4MLQiHycwqnXMVYdeB7sM59gPn2Q+cZz9wnnu/XDzHtEYAAADASwRhAAAAeMnXIDw77ALQ7TjHfuA8+4Hz7AfOc++Xc+fYyx5hAAAAwNcZYQAAAHjOqyBsZlPNbLWZVZnZjWHXg8NjZnea2XYzW542NsDMHjOztcFz/2DczOxnwbl+2czOSPuaq4Lj15rZVWF8L+iamY0ysyfNbJWZrTCzLwXjnOdexMwKzOwlM1sWnOdvBeNjzOzF4Jzdb2Z5wXh+sF0V7C9Pe6+bgvHVZnZhON8R3oqZRc1siZk9EmxznnsZM3vdzF4xs6VmVhmM9Yyf2845Lx6SopJekzRWUp6kZZImhl0Xj8M6h++SdIak5Wljt0i6MXh9o6TvB68/IOlvkkzS2ZJeDMYHSFoXPPcPXvcP+3vj0XY+h0k6I3hdImmNpImc5971CM5XcfA6LunF4PzNlTQ9GP+lpGuD19dJ+mXwerqk+4PXE4Of5fmSxgQ/46Nhf3883nS+b5B0r6RHgm3Ocy97SHpd0qBOYz3i57ZPM8KTJVU559Y555okzZE0LeSacBicc09L2tVpeJqku4LXd0m6JG389y7pBUn9zGyYpAslPeac2+Wc2y3pMUlTu796ZMI5t8U5tzh4XStplaQR4jz3KsH5qgs248HDSXqPpAeC8c7nOXX+H5D0XjOzYHyOc67RObdeUpWSP+uRI8xspKQPSvpNsG3iPPuiR/zc9ikIj5C0MW27OhhDzzbEObdFSoYoSYOD8YOdb/4e9BDBP4tOUnK2kPPcywT/XL5U0nYl/4f3mqQ9zrmW4JD0c9Z2PoP9eyUNFOe5J/iJpK9LSgTbA8V57o2cpL+b2SIzmxmM9Yif27Hu/oAcYl2MsWRG73Ww883fgx7AzIol/UnSl51z+5KTQl0f2sUY57kHcM61SjrdzPpJekjSSV0dFjxznnsgM/uQpO3OuUVmdn5quItDOc8937nOuc1mNljSY2b26lscm1Pn2acZ4WpJo9K2R0raHFItyJ5twT+pKHjeHowf7Hzz9yDHmVlcyRD8B+fcg8Ew57mXcs7tkfSUkr2C/cwsNUGTfs7azmewv6+SbVKc59x2rqSLzex1JdsR36PkDDHnuZdxzm0Onrcr+YvtZPWQn9s+BeGFksYHV6vmKdmIPy/kmnD05klKXVl6laQ/p41/Mrg69WxJe4N/mlkg6f1m1j+4gvX9wRhyQNAP+FtJq5xzP07bxXnuRcysLJgJlpkVSnqfkv3gT0q6NDis83lOnf9LJT3hklfXzJM0PVhtYIyk8ZJeOjbfBQ7FOXeTc26kc65cyf/nPuGcu0Kc517FzPqYWUnqtZI/b5erh/zc9qY1wjnXYmazlPxDjUq60zm3IuSycBjM7D5J50saZGbVkv6vpO9Jmmtmn5a0QdJlweHzlbwytUpSvaRrJMk5t8vMvq3kL0aSdLNzrvMFeAjPuZL+TdIrQf+oJH1DnOfeZpiku8wsquSEzFzn3CNmtlLSHDP7jqQlSv5SpOD5bjOrUnKGcLokOedWmNlcSSsltUj6QtBygdz2H+I89yZDJD0UtLDFJN3rnHvUzBaqB/zc5s5yAAAA8JJPrREAAABAG4IwAAAAvEQQBgAAgJcIwgAAAPASQRgAAABeIggDAADASwRhAAAAeIkgDAAAAC/9fwfmw9/uPD7oAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 864x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(12,8))\n",
    "plt.plot(cost_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Bingo! We did it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.741966337767029\n",
      "0.07587079591697381\n"
     ]
    }
   ],
   "source": [
    "print(cost_[0])\n",
    "print(cost_[-1])"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "Neural Networks with Numpy for Absolute Beginners - Part 3: Logistic Regression",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}