{ "cells": [ { "cell_type": "markdown", "id": "5ddad2d4-6b8e-417b-b2f1-818ec162c523", "metadata": {}, "source": [ "## The Bootstrap\n", "\"Bootstrapping\" refers to computational techniques for making inferences about a statistic beyond point estimates by treating the samples as though they were the populations of interest. Regarding the origin of the term, [An Introduction to the Bootstrap](https://cindy.informatik.uni-bremen.de/cosy/teaching/CM_2011/Eval3/pe_efron_93.pdf) states:\n", "\n", "> The use of the term bootstrap derives from the phrase *to\n", "pull oneself up by one's bootstrap*, widely thought to be based on\n", "one of the eighteenth century Adventures of Baron Munchausen,\n", "by Rudolph Erich Raspe. (The Baron had fallen to the bottom of\n", "a deep lake. Just when it looked like all was lost, he thought to\n", "pick himself up by his own bootstraps.)\n", "\n", "Let us return to the experiment that we considered at the beginning of the discussion of permutation tests. Again, a new medical treatment is intended to prolong life after a form of surgery. Sixteen mice are randomly assigned to either a treatment group or control group. All mice receive the surgery, but only the treatment group will receive the new treatment. The survival time of each mouse after surgery is recorded below.\n" ] }, { "cell_type": "code", "execution_count": 1, "id": "7581fc1a-795e-41b1-9de5-4167100d91d5", "metadata": {}, "outputs": [], "source": [ "# survival times measured in days\n", "import numpy as np\n", "x = np.array([94, 197, 16, 38, 99, 141, 23]) # treatment group\n", "y = np.array([52, 104, 146, 10, 51, 30, 40, 27, 46]) # control group" ] }, { "cell_type": "markdown", "id": "ca5fe064-d6b2-493b-8609-4a8d3b7be900", "metadata": {}, "source": [ "The permutation test allowed us to study whether or not the treatment had any effect on the survival times. In many studies, we are interested not only in whether there is an effect; we are also interested in the _magnitude_ of the effect. It would be misleading to report only the difference in mean survival times, especially since the permutation test and t-test showed that there was a ~$14\\%$ chance of observing such an extreme difference in means due to chance alone. In addition to reporting our statistic (the difference in means), we should also report some measurement of our uncertainty.\n", "\n", "One way of quantifying our uncertainty is the _standard error_ of our statistic. Suppose we were to perform the same experiment (with new mice) repeatedly. Because the mice are random samples from some greater population and there will be some random error in the effect of the treatment, we would not observe the same value of the statistic every time; rather, the values of the statistic would form a distribution. The standard error is the standard deviation of this distribution.\n", "\n", "How do we calculate the standard error if we do not know the underlying distribution from which the mice survival times are sampled? The typical approach, which we will not discuss in detail, assumes that the underlying distributions are normal; from this assumption and some math, statisticians have derived a formula to estimate the standard error of the statistic. This approach is limited in applicability, however, as it may not produce a good estimate if the original distributions are non-normal; moreover, standard error formulas are only available for a few statistics. Instead, we take a different approach, beginning with the mild assumption that the observed samples are representative of the distributions from which they were taken. We estimate the standard error by repeatedly *resampling from the observed data* (*with replacement*), calculating the statistic of the resample each time, and computing the standard deviation of the resulting distribution. This makes sense: to estimate the standard error, we would happily resample from the distribution itself it if were available to us. It's not, so we do the next best thing, which is resampling from the data we already have." ] }, { "cell_type": "code", "execution_count": 2, "id": "e30a2a3c-dae6-4c29-aa9f-d585a4b39213", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Observed Statistic Value: 30.63492063492064\n", "Standard Error: 27.336478035417056\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEGCAYAAABy53LJAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAd7ElEQVR4nO3de7gdVZ3m8e9LkJsNHTFBMQmdgBGMDGoMl0ZtL0ibBCTaAwqtD4i0mSjYareXII7d+vTMkxFHGlokRkTAC4g2QkZCA2Iro2MgIUK4awgRDkSJ2nIRmxh854+qA5vNPntXnZw6Zyd5P8+zn1O1aq21fzs52b9Uraq1ZJuIiIiqthvrACIiYsuSxBEREbUkcURERC1JHBERUUsSR0RE1LL9WAcwGiZMmOCpU6eOdRgREVuUG2+88Ve2J7aXbxOJY+rUqaxcuXKsw4iI2KJI+nmn8lyqioiIWpI4IiKiliSOiIioJYkjIiJqSeKIiIhakjgiIqKWJI6IiKgliSMiImpJ4oiIiFq2iSfHI5o0deEVw267btERIxhJxOjIGUdERNSSxBEREbUkcURERC1JHBERUUsSR0RE1JLEERERtSRxRERELUkcERFRS6OJQ9JsSXdJWiNpYYfjknRWeXy1pJktx86T9KCkW4fo+0OSLGlCk58hIiKerrHEIWkccDYwB5gBHCdpRlu1OcD08jUfOKfl2PnA7CH6ngIcDtw7slFHREQvTZ5xHASssb3W9kbgYmBeW515wIUuLAfGS9oTwPZ1wG+G6PsM4COAmwk9IiKG0mTimATc17I/UJbVrfM0ko4C7rd9c4968yWtlLRyw4YN1aOOiIiumkwc6lDWfoZQpc5TlaVdgNOAT/R6c9tLbM+yPWvixIm9qkdEREVNJo4BYErL/mTggWHUabUPMA24WdK6sv4qSc/f7GgjIqKSJhPHCmC6pGmSdgCOBZa21VkKHF/eXXUI8JDt9UN1aPsW23vYnmp7KkXimWn7Fw19hoiIaNNY4rC9CTgFuAq4A7jE9m2SFkhaUFZbBqwF1gBfBN472F7SRcCPgX0lDUg6qalYIyKiukYXcrK9jCI5tJYtbtk2cPIQbY+r0P/UzQwxIiJqypPjERFRSxJHRETUksQRERG1JHFEREQtjQ6OR0R3UxdesVnt1y06YoQiiaguiSNiC7Y5iSdJJ4Yrl6oiIqKWJI6IiKgliSMiImpJ4oiIiFqSOCIiopYkjoiIqCWJIyIiakniiIiIWvIAYASb/wR3xLYkZxwREVFLEkdERNSSxBEREbUkcURERC2NDo5Lmg2cCYwDzrW9qO24yuNzgceAd9peVR47DzgSeND2/i1tTgfeBGwE7gZOtP3bJj9HbBkywB0xOho745A0DjgbmAPMAI6TNKOt2hxgevmaD5zTcux8YHaHrq8B9rd9APBT4NSRjTwiIrpp8lLVQcAa22ttbwQuBua11ZkHXOjCcmC8pD0BbF8H/Ka9U9tX295U7i4HJjf2CSIi4hmaTByTgPta9gfKsrp1unkXcGWnA5LmS1opaeWGDRtqdBkREd00mTjUoczDqNO5c+k0YBPwtU7HbS+xPcv2rIkTJ1bpMiIiKmhycHwAmNKyPxl4YBh1nkHSCRQD54fZrpRoIiJiZDR5xrECmC5pmqQdgGOBpW11lgLHq3AI8JDt9d06Le/U+ihwlO3Hmgg8IiKG1ljiKAewTwGuAu4ALrF9m6QFkhaU1ZYBa4E1wBeB9w62l3QR8GNgX0kDkk4qD30O2BW4RtJNkhY39RkiIuKZGn2Ow/YyiuTQWra4ZdvAyUO0PW6I8heOZIwREVFPnhyPiIhakjgiIqKWnolD0v696kRExLajyhnHYkk3SHqvpPFNBxQREf2tZ+Kw/Srg7RTPW6yU9HVJhzceWURE9KVKYxy2fwZ8nOL5idcAZ0m6U9JfNRlcRET0nypjHAdIOoPiWYzXA2+y/eJy+4yG44uIiD5T5TmOz1E8nPcx278fLLT9gKSPNxZZRET0pSqJYy7we9tPAEjaDtjJ9mO2v9JodBER0XeqjHF8F9i5ZX+XsiwiIrZBVc44drL96OCO7Ucl7dJgTLGNytKvEVuGKmccv5M0c3BH0iuA33epHxERW7EqZxwfAL4paXCdjD2BtzUWUURE9LWeicP2Ckn7AftSrNh3p+0/NB5ZRET0parTqh8ITC3rv1wSti9sLKqIiOhbPROHpK8A+wA3AU+UxQaSOCIitkFVzjhmATOytndEREC1u6puBZ7fdCAREbFlqHLGMQG4XdINwOODhbaPaiyqiIjoW1USxz8Ot3NJs4EzgXHAubYXtR1XeXwu8BjwTturymPnAUcCD9rev6XN7sA3KAbr1wFvtf0fw40xIiLqqbIexw8ovqCfVW6vAFb1aidpHHA2MAeYARwnaUZbtTnA9PI1Hzin5dj5wOwOXS8ErrU9Hbi23I+IiFFSZVr1dwPfAr5QFk0CLqvQ90HAGttrbW8ELgbmtdWZB1zownJgvKQ9AWxfB/ymQ7/zgAvK7QuAN1eIJSIiRkiVwfGTgVcCD8OTizrtUaHdJOC+lv2BsqxunXbPs72+jGX9ULFImi9ppaSVGzZsqBBuRERUUSVxPF6eMQAgaXuK5zh6UYey9nZV6gyL7SW2Z9meNXHixJHoMiIiqJY4fiDpY8DO5Vrj3wT+T4V2AxTrlA+aDDwwjDrtfjl4Oav8+WCFWCIiYoRUSRwLgQ3ALcB/A5ZRrD/eywpguqRpknYAjgWWttVZChyvwiHAQ4OXobpYCpxQbp8AXF4hloiIGCFVJjn8I8XSsV+s07HtTZJOAa6iuB33PNu3SVpQHl9MkYTmAmsobsc9cbC9pIuA1wITJA0A/2D7S8Ai4BJJJwH3AsfUiSsiIjZPlbmq7qHDuIPtvXu1tb2MIjm0li1u2TbF4HuntscNUf5r4LBe7x0REc2oOlfVoJ0o/oe/ezPhREREv6vyAOCvW1732/5n4PXNhxYREf2oyqWqmS2721GcgezaWEQREdHXqlyq+t8t25so54dqJJqIiOh7Ve6qet1oBBIREVuGKpeq/q7bcdufHblwIiKi31W9q+pAnnp4703AdTx9jqmIiNhGVF3IaabtRwAk/SPwTdt/02RgERHRn6pMObIXsLFlfyPFIkoREbENqnLG8RXgBknfpniC/C3AhY1GFRERfavKXVX/Q9KVwKvLohNt/6TZsCIiol9VuVQFsAvwsO0zgQFJ0xqMKSIi+liVpWP/AfgocGpZ9Czgq00GFRER/avKGcdbgKOA3wHYfoBMORIRsc2qkjg2ltOfG0DSs5sNKSIi+lmVxHGJpC8A4yW9G/guNRd1ioiIrUfXu6okCfgGsB/wMLAv8Anb14xCbBER0Ye6Jg7blnSZ7VcASRbR09SFV4x1CBHRsCqXqpZLOrDxSCIiYotQJXG8jiJ53C1ptaRbJK2u0rmk2ZLukrRG0sIOxyXprPL46tZFo4ZqK+llkpZLuknSSkkHVYklIiJGxpCXqiTtZfteYM5wOpY0DjgbOBwYAFZIWmr79pZqc4Dp5etg4Bzg4B5tPw180vaVkuaW+68dTowREVFftzOOywBs/xz4rO2ft74q9H0QsMb2WtsbgYuBeW115gEXurCc4s6tPXu0NbBbuf2nwAMVYomIiBHSbXBcLdt7D6PvSTx9zY4BirOKXnUm9Wj7AeAqSZ+hSHyHdnpzSfOB+QB77bXXMMKPiIhOup1xeIjtqtShrL2foep0a/se4IO2pwAfBL7U6c1tL7E9y/asiRMnVgw5IiJ66XbG8VJJD1N8ie9cblPu2/ZuQzcFirOEKS37k3nmZaWh6uzQpe0JwPvL7W8C5/aIIyIiRtCQZxy2x9nezfautrcvtwf3eyUNgBXAdEnTJO0AHMtTy88OWgocX95ddQjwkO31Pdo+ALym3H498LPKnzYiIjZblYWchsX2JkmnAFcB44DzbN8maUF5fDGwDJgLrAEeA07s1rbs+t3AmZK2B/6TchwjIiJGR2OJA8D2Mork0Fq2uGXbwMlV25blPwReMbKRRkREVVUXcoqIiACSOCIioqZuT44/QpfbcCsOkEdExFZmyMRhe1cASZ8CfgF8heJW3LeTFQAjIrZZVS5VvdH2520/Yvth2+cA/7XpwCIioj9VuavqCUlvp5gvysBxwBONRhURjductVPWLTpiBCOJLU2VM46/Bt4K/LJ8HVOWRUTENqjnGYftdTxzVtuIiNhG9TzjkPQiSddKurXcP0DSx5sPLSIi+lGVS1VfBE4F/gBgezXF3FEREbENqpI4drF9Q1vZpiaCiYiI/lclcfxK0j6UDwNKOhpY32hUERHRt6rcjnsysATYT9L9wD0UDwFGRMQ2qEri+LntN0h6NrCd7UeaDioiIvpXlUtV90haAhwCPNpwPBER0eeqJI59ge9SXLK6R9LnJL2q2bAiIqJf9Uwctn9v+xLbfwW8HNgN+EHjkUVERF+qtB6HpNdI+jywCtiJYgqSiIjYBvUcHJd0D3ATcAnwYdu/q9q5pNnAmRTrhp9re1HbcZXH51KsOf5O26t6tZX0PuAUiudJrrD9kaoxRcTmywSJ27auiUPSOODLtj9Vt+Oy7dnA4cAAsELSUtu3t1SbA0wvXwcD5wAHd2sr6XUUc2cdYPtxSXvUjS0iIoava+Kw/UT5RV07cQAHAWtsrwWQdDHFF35r4pgHXGjbwHJJ4yXtCUzt0vY9wCLbj5cxPjiM2KKLzfnfZERs/aqMcfy/8k6qV0uaOfiq0G4ScF/L/kBZVqVOt7YvAl4t6XpJP5B0YKc3lzRf0kpJKzds2FAh3IiIqKLKA4CHlj9bzzoMvL5HO3Uoa1/DfKg63dpuDzyH4rmSA4FLJO1dnrU8VdleQvHEO7NmzRpy7fSIiKinynocrxtm3wPAlJb9ycADFevs0KXtAHBpmShukPRHYAKQ04qIiFFQZT2O50n6kqQry/0Zkk6q0PcKYLqkaZJ2oJiKfWlbnaXA8SocAjxke32PtpdRnu1IehFFkvlVhXgiImIEVBnjOB+4CnhBuf9T4AO9GtneRHHL7FXAHcAltm+TtEDSgrLaMmAtsIZi3Y/3dmtbtjkP2LtcWOpi4IT2y1QREdGcKmMcE2xfIulUKL7UJT1RpXPbyyiSQ2vZ4pZtU0xlUqltWb4ReEeV94+IiJFX5Yzjd5Key1PrcRwCPNRoVBER0beqnHH8HcX4wj6SfgRMBI5uNKqIiOhbVe6qWiXpNRSz5Aq4y/YfGo8sIiL6UpW7qo4Bdi4Hp98MfKPiA4AREbEVqjLG8d9tP1KuwfFG4AKKOaUiImIbVCVxDN5BdQRwju3LKZ6diIiIbVCVxHG/pC9QrMGxTNKOFdtFRMRWqEoCeCvFg3izbf8W2B34cJNBRURE/6qydOxjwDpgTrmA0p62r246sIiI6E9V7qr6BMWA+HMpJhP8sqSPNx1YRET0pyoPAB4HvNz2fwJIWkSx9vg/NRlYRET0pypjHOuAnVr2dwTubiSaiIjoe0OecUj6F4r5qR4HbpN0Tbl/OPDD0QkvIiL6TbdLVSvLnzcC324p/35j0URERN8bMnHYvgBA0k7ACynONu4eHOuIiIht05BjHJK2l/RpiqVaLwC+Ctwn6dOSnjVaAUZERH/pNjh+OsXDftNsv8L2y4F9gPHAZ0YhtoiI6EPdEseRwLttPzJYYPth4D3A3KYDi4iI/tQtcbjTWt62n6BcDTAiIrY93RLH7ZKOby+U9A7gziqdS5ot6S5JayQt7HBcks4qj69uXeejQtsPSbKkCVViiYiIkdHtdtyTgUslvYvillwDBwI7A2/p1bGkccDZFM99DAArJC21fXtLtTnA9PJ1MMU6Hwf3aitpSnns3hqfNSIiRkC323Hvp/gSfz3wEoplY6+0fW3Fvg8C1theCyDpYmAe0Jo45gEXlpfElksaL2lPYGqPtmcAHwEurxhLRESMkCprjn8P+N4w+p4E3NeyP0BxVtGrzqRubSUdBdxv+2ZJQ765pPnAfIC99tprGOFHREQnVSY5HK5O3+rtg+pD1elYLmkX4DTgL3u9ue0lwBKAWbNmbXOD+VMXXjHWIUR0tDm/m+sWHTGCkcRwNbmS3wAwpWV/MvBAxTpDle8DTANulrSuLF8l6fkjGnlERAypycSxApguaZqkHYBjgaVtdZYCx5d3Vx0CPGR7/VBtbd9iew/bU21PpUgwM23/osHPERERLRq7VGV7k6RTKJadHQecZ/s2SQvK44uBZRQPE64BHgNO7Na2qVgjIqK6Jsc4sL2MIjm0li1u2TbFbb+V2naoM3Xzo4yIiDqavFQVERFboSSOiIioJYkjIiJqSeKIiIhakjgiIqKWJI6IiKgliSMiImpp9DmO2DyZbyoi+lHOOCIiopYkjoiIqCWJIyIiakniiIiIWpI4IiKiliSOiIioJYkjIiJqSeKIiIhakjgiIqKWJI6IiKgliSMiImppNHFImi3pLklrJC3scFySziqPr5Y0s1dbSadLurOs/21J45v8DBER8XSNJQ5J44CzgTnADOA4STPaqs0Bppev+cA5FdpeA+xv+wDgp8CpTX2GiIh4pibPOA4C1thea3sjcDEwr63OPOBCF5YD4yXt2a2t7attbyrbLwcmN/gZIiKiTZOJYxJwX8v+QFlWpU6VtgDvAq7s9OaS5ktaKWnlhg0baoYeERFDaXI9DnUoc8U6PdtKOg3YBHyt05vbXgIsAZg1a1b7+0bENmhz1rhZt+iIEYxky9Zk4hgAprTsTwYeqFhnh25tJZ0AHAkcZjtJISJiFDV5qWoFMF3SNEk7AMcCS9vqLAWOL++uOgR4yPb6bm0lzQY+Chxl+7EG44+IiA4aO+OwvUnSKcBVwDjgPNu3SVpQHl8MLAPmAmuAx4ATu7Utu/4csCNwjSSA5bYXNPU5IiLi6Rpdc9z2Mork0Fq2uGXbwMlV25blLxzhMCMiooZGE0dExEjanMHtGDmZciQiImpJ4oiIiFqSOCIiopYkjoiIqCWD4w3KQF5EbI1yxhEREbUkcURERC1JHBERUUsSR0RE1JLEERERtSRxRERELbkdNyKigiwC9ZQkjh7yLEZExNPlUlVERNSSxBEREbUkcURERC1JHBERUUsSR0RE1NJo4pA0W9JdktZIWtjhuCSdVR5fLWlmr7aSdpd0jaSflT+f0+RniIiIp2vsdlxJ44CzgcOBAWCFpKW2b2+pNgeYXr4OBs4BDu7RdiFwre1FZUJZCHy0qc8REbG5xvK2/iaeIWnyjOMgYI3ttbY3AhcD89rqzAMudGE5MF7Snj3azgMuKLcvAN7c4GeIiIg2TT4AOAm4r2V/gOKsoledST3aPs/2egDb6yXt0enNJc0H5pe7j0q6azgfoosJwK9GuM/N1Y8xQX/GlZiqSUzV9WNcE/S/NiumP+tU2GTiUIcyV6xTpW1XtpcAS+q0qUPSStuzmup/OPoxJujPuBJTNYmpun6Mq6mYmrxUNQBMadmfDDxQsU63tr8sL2dR/nxwBGOOiIgemkwcK4DpkqZJ2gE4FljaVmcpcHx5d9UhwEPlZahubZcCJ5TbJwCXN/gZIiKiTWOXqmxvknQKcBUwDjjP9m2SFpTHFwPLgLnAGuAx4MRubcuuFwGXSDoJuBc4pqnP0ENjl8E2Qz/GBP0ZV2KqJjFV149xNRKT7FpDBxERsY3Lk+MREVFLEkdERNSSxDFMkj4kyZImtJSdWk6RcpekN45iLKdLurOctuXbksaPdUzle3edcmaUYpgi6d8l3SHpNknvL8vHfOoaSeMk/UTSd/oopvGSvlX+Pt0h6c/HOi5JHyz/7m6VdJGknUY7JknnSXpQ0q0tZUPGMBr/7oaIaVS+C5I4hkHSFIrpUO5tKZtBcffXS4DZwOfLqVNGwzXA/rYPAH4KnDrWMbVMGzMHmAEcV8Yz2jYBf2/7xcAhwMllHINT10wHri33R9v7gTta9vshpjOBf7O9H/DSMr4xi0vSJOBvgVm296e4WebYMYjpfIp/Q606xjCK/+46xTQq3wVJHMNzBvARnv5Q4jzgYtuP276H4k6xg0YjGNtX295U7i6neO5lTGOi2pQzjbO93vaqcvsRii/CSYzx1DWSJgNHAOe2FI91TLsBfwF8CcD2Rtu/Heu4KO7+3FnS9sAuFM90jWpMtq8DftNWPFQMo/LvrlNMo/VdkMRRk6SjgPtt39x2aKjpU0bbu4Ary+2xjKlf/jyeJGkq8HLgetqmrgE6Tl3ToH+m+M/HH1vKxjqmvYENwJfLS2jnSnr2WMZl+37gMxRn9+spnvW6eixjajFUDP3yu9/Yd0GTU45ssSR9F3h+h0OnAR8D/rJTsw5lI3avc7eYbF9e1jmN4tLM10Yjph7G8r2fQdKfAP8KfMD2w1Kn8EYtliOBB23fKOm1YxbIM20PzATeZ/t6SWcyNpfLnlSOG8wDpgG/Bb4p6R1jGVMFY/673/R3QRJHB7bf0Klc0n+h+AW+ufzimQysknQQ1aZYGfGYWmI7ATgSOMxPPZzTaEw9jOV7P42kZ1Ekja/ZvrQs/qWkPcuJMkd76ppXAkdJmgvsBOwm6atjHBMUf2cDtq8v979FkTjGMq43APfY3gAg6VLg0DGOadBQMYzp7/5ofBfkUlUNtm+xvYftqbanUvxlzLT9C4qpUI6VtKOkaRRrjNwwGnFJmk2xJslRth9rOTRmMVFtypnGqcjwXwLusP3ZlkNjNnWN7VNtTy5/h44Fvmf7HWMZUxnXL4D7JO1bFh0G3D7Gcd0LHCJpl/Lv8jCKcap+mHpoqBi2/u8C23kN8wWsAya07J8G3A3cBcwZxTjWUFy/vKl8LR7rmMr3nktxZ8fdFJfUxuLv6FUUp+SrW/585gLPpbgT5mflz93HKL7XAt8pt8c8JuBlwMryz+sy4DljHRfwSeBO4FbgK8COox0TcBHFGMsfKP7DeFK3GEbj390QMY3Kd0GmHImIiFpyqSoiImpJ4oiIiFqSOCIiopYkjoiIqCWJIyIiakniiC2KpO+3z+wp6QOSPt+jzayG47qonJH0g23lbx7u5I6SXlY+JFi1/nhJ761bT9ILJH1rpOrH1i+JI7Y0F1E8NNfq2LJ8TEh6PnCo7QNsn9F2+M0UswMPx8sonjmpajzQM3G017P9gO2jR7B+bOWSOGJL8y3gSEk7wpMTF74A+KGkcyStVLF2wyc7NZb0aMv20ZLOL7cnSvpXSSvK1ys7tN1J0pcl3VJOAvi68tDVwB6SbpL06pb6hwJHAaeXx/YpX/8m6UZJ/1fSfmXdY1SsN3GzpOvKp+0/BbytbPu2tlheIumG8thqSdOBRcA+Zdnpkv5E0rWSVpUxD85O3F5vqso1HSr221p/nKTPlP2vlvS+yn+TseUa7SdT88prc1/AFcC8cnshcHq5vXv5cxzwfeCAcv/7FOs5ADza0s/RwPnl9teBV5Xbe1FMUdL+vn8PfLnc3o9iOoydgKnArUPEej5wdMv+tcD0cvtgiulGAG4BJpXb48uf7wQ+N0S//wK8vdzeAdi5PQ6Kueh2K7cnUDxVrA71ntyv2G9r/fdQzAO2fevfQV5b9yuTHMaWaPBy1eXlz3eV5W+VNJ/iC3NPiktEqyv2+QZghp6aNXc3Sbu6WMNj0KsovlixfaeknwMvAh6u8gYqZug9lGKG18HiHcufPwLOl3QJcGmH5u1+DJymYl2PS23/TM+c8VfA/5T0FxTTt08CnjcC/bZ6A8W0FpsAbLevWRFboSSO2BJdBnxW0kxgZ9uryonbPgQcaPs/yktQO3Vo2zrHTuvx7YA/t/37Lu+7uXOxbwf81vbLnhGUvUDSwRSLO90k6Rl12up/XdL1Zf2rJP0NsLat2tuBicArbP9B0jo6/5nU7beVGMPp8mNsZIwjtji2H6W4/HQeTw2K7wb8DnhI0vMolqzt5JeSXixpO+AtLeVXA6cM7gzxxX0dxZcxkl5EcUnrrh7hPgLsWsb9MHCPpGPKPiTppeX2Pravt/0J4FcUU2A/2badpL2BtbbPopj59IAO9f+UYt2PP5TjMX/WHtMw+211NbBAxep8SNq9x59HbAWSOGJLdRHFmtgXA7hYkfEnwG0UCeVHQ7RbCHwH+B7FzKKD/haYVQ7w3g4s6ND288A4SbcA3wDeafvxHnFeDHy4HEzfhyLxnCTp5jLWwQHr08sB5lspEtTNwL9TXD57xuA48DbgVkk3UYy3XGj718CPykH20ykW8ZklaWX5vneWf1bt9er22+pcirGe1eVn+usefx6xFcjsuBERUUvOOCIiopYkjoiIqCWJIyIiakniiIiIWpI4IiKiliSOiIioJYkjIiJq+f8XyCrp0n5dgQAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "import matplotlib.pyplot as plt\n", "\n", "rng = np.random.default_rng()\n", "\n", "def statistic(x, y, axis=0):\n", " return np.mean(x, axis=axis) - np.mean(y, axis=axis)\n", "\n", "def bootstrap_distribution(x, y):\n", " nx, ny = len(x), len(y)\n", " N = 1000\n", " bootstrap_distribution = []\n", " for i in range(N):\n", " # random indices to resample from x and y\n", " ix = rng.integers(0, nx, size=nx)\n", " iy = rng.integers(0, ny, size=ny)\n", " xi = x[ix]\n", " yi = y[iy]\n", " stat = statistic(xi, yi)\n", " bootstrap_distribution.append(stat)\n", " return bootstrap_distribution\n", "\n", "boot_dist = bootstrap_distribution(x, y)\n", "\n", "plt.hist(boot_dist, density=True, bins=20)\n", "plt.xlabel(\"Value of test statistic\")\n", "plt.ylabel(\"Observed Frequency\")\n", "\n", "observed_statistic = statistic(x, y)\n", "standard_error = np.std(boot_dist, ddof=1)\n", "print(f\"Observed Statistic Value: {observed_statistic}\")\n", "print(f\"Standard Error: {standard_error}\")" ] }, { "cell_type": "markdown", "id": "b7802b49-0a53-4f5f-95b1-a0710ab98a81", "metadata": {}, "source": [ "This is precisely what `bootstrap` does." ] }, { "cell_type": "code", "execution_count": 3, "id": "530d48d9-8944-43e3-848b-37831e3630f7", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "26.78533925848876" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from scipy import stats\n", "# `n_resamples=1000` indicates that the statistic will be calculated for\n", "# each of 1000 resamples.\n", "# The meaning of `method='percentile'` will be discussed below\n", "res = stats.bootstrap((x, y), statistic, n_resamples=1000, method='percentile')\n", "assert res.standard_error == np.std(res.bootstrap_distribution, ddof=1)\n", "res.standard_error" ] }, { "cell_type": "markdown", "id": "bce2cf1c-4a82-4438-afb7-1832690132f2", "metadata": {}, "source": [ "The two standard errors estimates differ slightly because the bootstrap algorithm is inherently stochastic, but that is OK. The best we can hope for is an approximation, and these two approximations agree with one another quite well.\n", "\n", "An even better way of quantifying the uncertainty, especially when the distribution of the statisic is non-normal, is to produce a *confidence interval* on the statistic. Suppose we perform the experiment repeatedly and produce a \"95% confidence interval\" $(l_i, u_i)$ from the data in each experiment $i$; this means that we should expect the true value of the statistic (the difference in the *population* means) to be between $l_i$ and $u_i$ in 95% of the replications $i$." ] }, { "cell_type": "code", "execution_count": 4, "id": "590cc9d0-0b34-4faf-829b-8302b4230553", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "ConfidenceInterval(low=-18.494047619047613, high=82.84563492063491)" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "res.confidence_interval # 95% confidence interval by default" ] }, { "cell_type": "markdown", "id": "935b014a-ec43-4c5f-a480-c804edea3e6a", "metadata": {}, "source": [ "By choosing `method='percentile'` above, we indicated that bootstrap should estimate this confidence interval as the central 95% of the bootstrap distribution - that is, the boundaries of our interval will be the 2.5 and 97.5 percentiles of the bootstrap distribution." ] }, { "cell_type": "code", "execution_count": 5, "id": "86857327-0c35-4f5b-9529-9b1c65edd133", "metadata": {}, "outputs": [], "source": [ "ci_percentile = stats.scoreatpercentile(res.bootstrap_distribution, [2.5, 97.5])\n", "np.testing.assert_allclose(res.confidence_interval, ci_percentile) # confidence interval is the central 95% of the bootstrap distribution " ] }, { "cell_type": "markdown", "id": "86f64385-b4a6-4015-8222-3adc85572384", "metadata": {}, "source": [ "Again, this means that if we were to perform the mice experiment repeatedly and each time use `bootstrap` to compute such a confidence interval from the data, we would expect the confidence interval to contain the true value of the difference in mean survival times 95% of the time. Note also that our confidence interval contains 0. This is closely related to our conclusion from the hypothesis tests above: our data is not inconsistent with the null hypothesis that the treatment has no effect." ] }, { "cell_type": "markdown", "id": "9e000136-9159-4361-9da8-b2d0f61d30ae", "metadata": {}, "source": [ "### Single-Sample, Scalar-Valued Statistics (and Confidence Intervals)\n", "This definition of a confidence interval can be difficult to interpret correctly, so we illustrate with a simpler example. Suppose there is an election with only two candidates, `0` and `1`, and all voters will vote for either one or the other (never both, and never for neither). We wish to estimate the percentage of voters who will vote for candidate `1` by performing an experiment before the election: we will ask a random sample of 1000 voters who the will vote for on election day. The results are stored in the array `sample`." ] }, { "cell_type": "code", "execution_count": 6, "id": "2a72dd5a-7de6-4d7b-8d6d-e618f7c9f3fa", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "249 for candidate 0, 751 for candidate 1\n" ] } ], "source": [ "# Rather than entering `sample` directly, let's generate one to work with.\n", "# To simulate the results of such an experiment, suppose that the true \n", "# (but unknown) percentage of voters who will vote for candidate `1` is 75%. \n", "# If we sample voters at random from the population before the election and \n", "# ask them who they will vote for, the responses will follow a Bernoulli \n", "# distribution with shape parameter `p=0.75`.\n", "p = 0.75\n", "dist = stats.bernoulli(p=p)\n", "sample = dist.rvs(size=1000)\n", "vote_for_0 = np.sum(sample == 0)\n", "vote_for_1 = np.sum(sample == 1)\n", "print(f\"{vote_for_0} for candidate 0, {vote_for_1} for candidate 1\")" ] }, { "cell_type": "markdown", "id": "c9f9532d-46ee-4d59-ab3b-af452790fab1", "metadata": {}, "source": [ "The statistic we wish to estimate is the percentage of voters who will vote for candidate 1, so we can produce a *point estimate* of the statistic from the sample as:" ] }, { "cell_type": "code", "execution_count": 7, "id": "89b46532-a3b2-448f-b47a-22b92529074c", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0.751" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def statistic(sample, axis=0):\n", " return np.sum(sample, axis=axis) / sample.shape[axis]\n", "statistic(sample)" ] }, { "cell_type": "markdown", "id": "b30681e0-52d0-41ac-8e90-11ecee5bf235", "metadata": {}, "source": [ "`bootstrap` can produce a confidence interval around the point estimate." ] }, { "cell_type": "code", "execution_count": 8, "id": "ee6d8613-22ca-4a59-bf66-c86ed1bdb726", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "ConfidenceInterval(low=0.726, high=0.772)" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# As with `permutation_test`, the first argument of `bootstrap` needs to be a *sequence* of samples\n", "data = (sample,)\n", "# Passing `confidence_level=0.9` produces a 90% confidence interval\n", "res = stats.bootstrap(data, statistic, confidence_level=0.9)\n", "res.confidence_interval" ] }, { "cell_type": "markdown", "id": "593bb317-2f96-40a3-8f7f-1499015ea2b2", "metadata": {}, "source": [ "Suppose we perform the same experiment $100$ times, each time collecting new data from the same population, but computing the confidence interval in the same way." ] }, { "cell_type": "code", "execution_count": 9, "id": "b09ee5a2-1adb-4430-8118-ad16f2ed2549", "metadata": {}, "outputs": [], "source": [ "# lower and upper limits of confidence intervals produced by `bootstrap`\n", "n_replications = 100 # 100 replications of the same experiment\n", "n_observations = 1000 # 1000 observations per sample\n", "\n", "# Draw 100 new samples from the same population of voters, each with 1000 observations\n", "sample = dist.rvs(size=(100, 1000)) \n", "\n", "# bootstrap the 90% confidence interval for all 100 samples (10 at a time)\n", "res = stats.bootstrap((sample,), statistic, confidence_level=0.9, axis=1, batch=10)\n", "li, ui = res.confidence_interval\n", " \n", "# This was equivalent to (but faster than) the following \n", "# li = np.empty((n_replications,))\n", "# ui = np.empty((n_replications,))\n", "# for i in range(n_replications):\n", "# sample = dist.rvs(size=n_observations) # collect a new sample from the same population of voters\n", "# res = stats.bootstrap((sample,), statistic, confidence_level=0.9, vectorized=False)\n", "# li[i], ui[i] = res.confidence_interval" ] }, { "cell_type": "markdown", "id": "55cbf673-1d90-4436-9755-12c79cd4c054", "metadata": {}, "source": [ "We expect that the confidence interval will contains the true value of the statistic ($p=0.75$) approximately 90% of the time." ] }, { "cell_type": "code", "execution_count": 10, "id": "9856f8c6-4f4e-43e6-98e7-476202f27bcb", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "87\n" ] } ], "source": [ "contained = (li < p) & (p < ui)\n", "print(np.sum(contained))" ] }, { "cell_type": "markdown", "id": "c172f0e3-40cf-49ec-841e-0a3a7ab7bf64", "metadata": {}, "source": [ "### Paired-Sample, Vector-Valued Statistics\n", "\n", "[An Introduction to the Bootstrap](https://books.google.com/books?id=MWC1DwAAQBAJ&printsec=frontcover) considers a small data set collected when studying a medical device for continuously delivering an anti-inflammatory hormone to test subjects. The arrays `x` and `y` record the number of hours the device was worn and the amount of hormone remaining in the device, respectively." ] }, { "cell_type": "code", "execution_count": 11, "id": "07480ef9-594e-4eb6-9661-25cced033299", "metadata": {}, "outputs": [], "source": [ "x = np.array([99, 152, 293, 155, 196, 53, 184, 171, 52, 376, 385, 402, 29, 76, 296, 151, 177, 209, 119, 188, 115, 88, 58, 49, 150, 107, 125]) # hours worn\n", "y = np.array([25.8, 20.5, 14.3, 23.2, 20.6, 31.1, 20.9, 20.9, 30.4, 16.3, 11.6, 11.8, 32.5, 32.0, 18.0, 24.1, 26.5, 25.8, 28.8, 22.0, 29.7, 28.9, 32.8, 32.5, 25.4, 31.7, 28.5]) # amount remaining (units unspecified)" ] }, { "cell_type": "markdown", "id": "e1e416f8-7f0e-46f8-9584-388fd67602fe", "metadata": {}, "source": [ "A standard linear regression is performed in SciPy as follows." ] }, { "cell_type": "code", "execution_count": 12, "id": "f64d81c2-d847-425a-bee0-dc53e8efed05", "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAjpklEQVR4nO3de3zU1Z3/8deHEAMCxRDAHxUKRi2gCCFEjOKCiFiveK21RcW2SlWqVX9bxVovXW2Lq6J1VbZIFa3UpRVFyuLWleJddBMvCEVW5FJRChhAQVFJcvaPM3GSMJPMJDPz/c7M+/l4zGPmnHy/8/34BT+cnO+5mHMOERHJPh2CDkBERNpGCVxEJEspgYuIZCklcBGRLKUELiKSpTpm8mI9e/Z0AwYMyOQlRUSyXnV19UfOuV7N6zOawAcMGEBVVVUmLykikvXMbH2senWhiIhkKSVwEZEspQQuIpKlMtoHLiLhtHv3bjZs2MDnn38edCh5rVOnTvTt25fCwsKEjlcCFxE2bNhAt27dGDBgAGYWdDh5yTlHTU0NGzZsYP/990/oHHWhiAiff/45JSUlSt4BMjNKSkqS+i1ICRyoXr+Ne5espnr9tqBDEQmMknfwkv0zyPsulOr125g4aylf1tazV8cOzLmwkhH9i4MOS0SkVXnfAl+6poYva+upd7C7tp6la2qCDklEWnDiiSeyffv2Fo+54YYbeOaZZ9r0/c8++ywnn3xyq8cdffTRrU5MvOuuu/jss8/aFEcisiKBt7eLo6XzK0tL2KtjBwoMCjt2oLK0pL3hikgaOOeor69n0aJF7LPPPi0e+y//8i8ce+yxmQmsBXmfwKvXb2PGtfcw5ZiD+M3PZyadxBu6SO54ehUTZy3d4/wR/YuZc2ElVx03UN0nIklI9bOj6dOnM2TIEIYMGcJdd90FwLp16xg8eDCXXnop5eXlvP/++wwYMICPPvoIgJtvvplBgwYxfvx4vvvd73L77bcDcMEFF/DYY48BfgmPG2+8kfLycg499FDeeecdAF577TWOPPJIhg8fzpFHHsmqVatajG/Xrl2cc845DB06lO985zvs2rXrq59dcsklVFRUcMghh3DjjTcCcPfdd/Phhx8yduxYxo4dG/e49gh9H/jSNTUUfPEFAA8/MpUdT98N76+FvfZK+PzmXSTNk/SI/sVK3CJJSPWzo+rqah588EFeffVVnHMcfvjhjBkzhuLiYlatWsWDDz7Ifffd1+Scqqoq5s2bxxtvvEFtbS3l5eWMGDEi5vf37NmT119/nfvuu4/bb7+dWbNmMWjQIJ5//nk6duzIM888w89+9jPmzZsXN8YZM2aw9957s2zZMpYtW0Z5eflXP/vlL39Jjx49qKurY9y4cSxbtozLL7+c6dOns2TJEnr27Bn3uKFDh7b5voW+BV5ZWsJzB4/i7HP/FYBumz+EoiKYMyfh89VFIpJaqX529OKLL3L66afTpUsXunbtyhlnnMELL7wAQP/+/amsrIx5zqmnnkrnzp3p1q0bp5xyStzvP+OMMwAYMWIE69atA+Djjz/m29/+NkOGDOHKK69kxYoVLcb4/PPPc+655wIwdOjQJon3j3/8I+Xl5QwfPpwVK1bwt7/9LeZ3JHpcokKfwBu6OMb84HSq19bAqaf6H5x7LphBKw8z1EUiknqpbhi1tLl6ly5dkj6nuaKiIgAKCgqora0F4Prrr2fs2LEsX76cP//5zwmNv441zG/t2rXcfvvtLF68mGXLlnHSSSfF/K5Ej0tG6BM4+CQ8ZeyBjBjQA+bPh8Z9VcXFcPPNiZ2fhuStMeSSj1LdMBo9ejTz58/ns88+49NPP+WJJ57gn/7pn1o856ijjvoq8e7cuZP//M//TOqaH3/8Mfvttx8As2fPTijGOZHf/JcvX86yZcsA+OSTT+jSpQvdu3dn06ZNPPXUU1+d061bN3bs2NHqcW0V+j7wmL75TXAOrr4abrsNbrjBv9avh298I2NhaAy55LNUPjsqLy/nggsuYOTIkQBceOGFDB8+/KvujlgOO+wwJkyYwLBhw+jfvz8VFRV079494WteffXVTJo0ienTp3PMMce0evwll1zC97//fYYOHUpZWdlXsQ4bNozhw4dzyCGHUFpayqhRo746Z/LkyZxwwgn06dOHJUuWxD2urSyZX0Paq6KiwqV8Q4ctW6B372j5/PPhoYdSe4047l2ymjueXkW9gwKDq44byJSxB6b8OtXrt7F0TQ2VpSX6B0LSYuXKlQwePDjoMJK2c+dOunbtymeffcbo0aOZOXNmk4eL2SjWn4WZVTvnKpofmxVdKC3q1cu3xmfM8OWHH/Z949XVab90Jh6QtjYMUiSfTZ48mbKyMsrLyznzzDOzPnknKzu7UGK5+GKYNAn69IGPP4aKChg5El55BTqk59+phn7AdLaOExkGKZKv/vCHPwQdQqCyvwXeWOfOflRKw8OM116DgoJoOQ3S+YAUNAxSROLLnRZ4YyeeCHV1MGoULF0KDesa1NRAjx7BxpakTLTyRSQ75WYCB99t8sor8Prr0DA7q6QEysrgjTcCDS1ZmikqIrHkVhdKLOXl/iFngzffBDMW/ir5dVVERMIk9xN4A+fgxRe/Kp583Y8YMaCHkrhISNx9990MHjyYiRMnsmDBAqZNmwbA/Pnzm0w5nz17Nh9++GFS371u3TqGDBnSYn1VVRWXX355O/4LMi9/EjjAqFHc+9d3m1SNGNDDj2ARkUDdd999LFq0iDlz5jBhwgSmTp0KpCaBJ6KiooK777475d+bTvmVwPGjOgb9fBFnT7w1Wvnb3/qx4/X1wQUmkscuvvhi1qxZw4QJE7jzzjuZPXs2P/7xj3n55ZdZsGABP/3pTykrK+PWW2+lqqqKiRMnUlZWxq5du6iurmbMmDGMGDGCb33rW2zcuBHwKxwOGzaMI444gnvvvbfVGBpv5HDTTTfxgx/8gKOPPprS0tImif2RRx5h5MiRlJWV8aMf/Yi6urr03JQE5O5DzDiiozoOovqXF/kWeIOCAr+2ytatwQUoErQrrvDPilKprAwia3zH8u///u/813/911dLrzasTXLkkUcyYcIETj75ZM466ywAnnrqKW6//XYqKirYvXs3l112GU8++SS9evVi7ty5XHfddTzwwAN8//vf59/+7d8YM2YMP/3pT5MO+Z133mHJkiXs2LGDgQMHcskll7B69Wrmzp3LSy+9RGFhIZdeeilz5szh/PPPb8NNab+8S+DQbFSHc7B5M+y7ry9v2+Zb41k45FAk36xatYrly5czfvx4AOrq6ujTpw8ff/wx27dvZ8yYMQCcd955SS8eddJJJ1FUVERRURG9e/dm06ZNLF68mOrqag477DDAb/LQu/FSHhmWdwk85roivXv7RN6xox8/Dn7IITQdwSKSD1poKYeNc45DDjmEV155pUn99u3bk97hvbmGJWghugytc45Jkybx61//ul3fnSp51Qfe6roitbX+1ZiZn9EpIoFovCRr8/LAgQPZsmXLVwl89+7drFixgn322Yfu3bvzYmTk2ZwEN4Bpzbhx43jsscfYvHkzAFu3bmX9+vUp+e62yKsEntAuIgUFvtV93nnRusMP94lcRDLunHPO4bbbbmP48OG89957XHDBBVx88cWUlZVRV1fHY489xjXXXMOwYcMoKyvj5ZdfBuDBBx9kypQpHHHEEXTu3DklsRx88MHccsstHHfccQwdOpTx48d/9dA0CK0uJ2tmnYDngSJ8l8tjzrkbzawHMBcYAKwDznbOtTioOi3LySahoQW+u7aewkTX726euO++Gy67LH1BigQgW5eTzUWpXk72C+AY59wwoAw43swqganAYufcQcDiSDnU2rSLiHPw+99Hy5dfrta4iIRCqwnceTsjxcLIywGnAg07JzwEnJaOAFOtTasHnnvung8zzeD441MbnIhIEhLqAzezAjN7E9gM/Ldz7lVgX+fcRoDIe8yxNGY22cyqzKxqy5YtKQo7IM5B452r//IXn8i//DKQcLQfp6RSJnfnktiS/TNIaBihc64OKDOzfYAnzGzPRQXinzsTmAm+Dzyp6MLo4IN9Im/cjdIw3CiD/wNoP05JpU6dOlFTU0NJSUm7h99J2zjnqKmpoVOnTgmfk9Q4cOfcdjN7Fjge2GRmfZxzG82sD751nj+cg507oVu3aJ1ZxjZW1k49kkp9+/Zlw4YNZP1vyVmuU6dO9O3bN+HjW03gZtYL2B1J3p2BY4FbgQXAJGBa5P3JNkWczbp29Yl8yJBo10r//v49za3xhp16GkbUaKceaY/CwkL233//oMOQJCXSAu8DPGRmBfg+8z865xaa2SvAH83sh8DfgW+nMc5wW77cJ+zGe2+awcKFcNJJabmkduoRkVbHgadS0OPAM+L66+GWW5rW6eGQiLRDe8aBSzJuvjn2kMOrrw4mHhHJWUrg6eKcH2bY4LbbfCJXa1xEUkQJPJ2OO27PhN2hAxx4YDDxiEhOUQJPs+r12xj080UcMWV2tPK993xr/JNPUn4tTewRyR9K4GnWMF57Y9eeHDB1YdMfdu+esnVVWl0qV0RyjhJ4mjWM1y4wKOzYgep1W/ecem8Gy5a16zrxlspVq1wkd+XdjjyZFne8tnNw2mnwZGT+07Bh0fo2iDWxR9PtRXKbEngGNNmDs7H58/17424UM/jNb/yytUleo/k/FPcuWa3p9iI5TF0oYeCcH2bY4Cc/aVPfePOlcpt332i6vUhu0UzMsGmeuMeMgWefbfPXxdzEWUSySryZmOpCCRvn4NVXobLSl597zif13buhY/J/XHG7b0Qk66kLJYwOP3zPh5mFhdrKTUSaUAIPM+dg69amdWawdm0w8YhIqCiBh11x8Z6t8dJStcZFRAk8azgH9fVN68zg0UeDiUdEAqcEnk0aVjMcNSpa973vqTUukqeUwLPRiy/GXnP8rLOCiUdEAqEEns2cg3vuiZbnzVNrXCSPKIFnuylTYrfGlchFcp4SeK5wDl57rWmdGWzfHkg4IpJ+SuC55LDD9myNFxerNS6So5TAc5FzsHNn0zozePnlYOIRkbRQAs9VXbrs2RofNUqtcZEcogSe65yL/ZDz+uuDiUdEUkYJPF8413Sc+C23hKo1rq3fRJKn5WTzyZ/+5N+b7wDUuzds2hRMTKCt30TaSC3wfOQczJ0bLW/e7BN5XV0g4cTbkFlEWqYEnq/OPnvPvvGOHQPpVtHWbyJtoy6UfOccvP8+fOMb0Toz+PvfoV+/jIQQa0NmEWmdErj4RO1c09Z3Q0LP0J6p2vpNJHnqQgmJUIzCcA5qa5vWmUUffuaAUNxnkRRRCzwEQjUKo6DAJ/JeveCjj3zd2Wf79wy1xtMlVPdZJAXUAg+BUI7C2LIl9gSgb387mHhSIJT3WaQdlMBDINSjMJyD666Llh97LFQTgJIR6vss0gbmMvhrcUVFhauqqsrY9bJJ9fpt4R+FEStxZ1m3SlbcZ5FmzKzaOVfRvF594CGRDaMwqtdt5Zc3P8Ljv7s8WmkGn34Ke+8dXGBJyIb7LJKoVrtQzKyfmS0xs5VmtsLMfhKpv8nMPjCzNyOvE9MfrgRp6Zoa3uxVyoBrFjb9QZcuWdutIpLNEukDrwX+v3NuMFAJTDGzgyM/u9M5VxZ5LUpblBIKjfuQB/18EW8sW9f0ADNYujSQ2ETyUatdKM65jcDGyOcdZrYS2C/dgUn4NJ8xObx/8Z4TgI44wr9nWd+4SDZKahSKmQ0AhgOvRqp+bGbLzOwBM4vZsWhmk82sysyqtmzZ0r5oJXAj+hczZeyBTfuR4605PnVqZoMTyTMJJ3Az6wrMA65wzn0CzAAOAMrwLfQ7Yp3nnJvpnKtwzlX06tWr/RFLeDkHBxwQLd96q/rGRdIooQRuZoX45D3HOfc4gHNuk3OuzjlXD9wPjExfmJI1Vq+O3RovLAwmHpEclsgoFAN+B6x0zk1vVN+n0WGnA8tTH55kLefgjka/lNXW+kSuvnGRlGl1Io+ZHQW8ALwN1EeqfwZ8F9994oB1wI8iDzzj0kSePJUDE4BEgtTmiTzOuReBWB2ZGjYoiXEOli+HQw+N1pnBBx/A178eXFwiWU5roUjS2rQk65Ahe7a699tPDzlF2kFT6SUp7V6S1Tn48ksoKorWmfk9OhuWrRWRhKgFLklJyZKse+21Z2v8O99Ra1wkSUrgkpSULskabwLQKae0L0iRPKEuFElKWjYgdg7OOAOeeMKXFy7UkEORBCiBS9LSsiTr44/798bdKA2flchFYlIXioSLc9Fk3sDMP/gUkSaUwCV8Tj99z1Z3UZEecoo0owQu4eWc31y5MTNYuzaYeERCRglc2qVNk3qS0bPnnq3x0lK1xkXQQ0xph3ZP6klGw5DDDo3aHJoAJHlOLXBps5RM6klGw9DCM86I1mkCkOQxJXBps5RO6knGvHmxJwDdcENmri8SEq0uJ5tKWk4291Sv35baST3JevhhmDSpaV19vVrlklPiLSerBC65oXnCPvRQWLYsmFhEUixeAlcXiuQG52Ddumj57bd9Uv/008BCEkk3JXDJHf3779k33rWrulMkZymBS+5xDr74ommdGaxcGUw8ImmiBC65qWHN8fHjo3UHH6zWuOQUJXDJbU8/HXvI4TPPBBOPSAopgUvWaNe0fefgN7+JlsePV2tcsp4SuGSFhmn7dzy9iomzlrYtiV9+eezW+K9/nZogRTJMCVyyQkqn7TsHf/tbtPyzn/lEXl/f/kBFMkgJXLJCyqftDx68Z2u8oACOPbZ93yuSQVqNULJCWvbiBJ/EP/kEunf35cWLfWt82zbYZ5/UXEMkTZTAJWukZS9OgK99zSfyU07xGyoDFEeuo/04JcTUhSLS4M9/3rMf3MxPyxcJISVwkcYa1hy/445o3dChGnIooaQELhLLVVfFHnL46KPBxCMSgxK4SEucg2efjZa/9z21xiU0lMAl45KZUZn2TZMTMWZM7Nb4ZZcFE49IhDZ0kIxKZiPkjG6anKgNG6Bfv6Z1X3zhF88SSRNt6CChkMyMykSPzWgrvW9f3xrv3z9aV1QEBx6Y/muLNKNx4NImbd0Ls2FG5e7a+lZnVCZybGCt9HXrfMu7Uydffu89363ywQfw9a+n//oiKIFLG7QnaSYzozKRY2O10jPWzVJU5Fvjl14KM2b4uv328++aACQZoC4USVp7F5Ya0b+YKWMPTCjRtnZsytdIaYv77ov9kPOFFzIfi+QVtcAlacl0g6Rb2tZIaQvnYM4cOPdcXx49OlovkgatjkIxs37Aw8D/A+qBmc6535hZD2AuMABYB5ztnGvxKZJGoeSOtvaB543mY8Xvugt+8pNAQpHsF28USiIJvA/Qxzn3upl1A6qB04ALgK3OuWlmNhUods5d09J3KYFLXnnrLSgra1pXX6+JQJK0Ng8jdM5tdM69Hvm8A1gJ7AecCjwUOewhfFIXkQbDhu3ZfdKhA5x+ejDxSM5J6iGmmQ0AhgOvAvs65zaCT/JA7zjnTDazKjOr2rJlSzvDFclCzsHWrdHy/Pm+Fb5jR2AhSW5IOIGbWVdgHnCFc+6TRM9zzs10zlU45yp69erVlhhFsl9xsU/kDQ82wa9D3kEDwaTtEvrbY2aF+OQ9xzn3eKR6U6R/vKGffHN6QhTJIc89B3V10bJzvjW+alVwMUnWajWBm5kBvwNWOuemN/rRAmBS5PMk4MnUhyeSgzp08In75pujdYMG6eGmJC2RFvgo4DzgGDN7M/I6EZgGjDezd4HxkbKIJOrnP489AWj+/EDCkezT6kQe59yLQLymwbjUhiOSh5yDv/wFjj/elxtGqWgCkLRCT1BEwuBb34rdGr/22mDikaygBC4SJs7BmjXR8rRpPpHX1gYXk4SWErhI2Oy/v0/kxY2WKCgshPLy4GKSUFICFwmrrVvhs8+i5Tfe8K3xzRqxK54SuEiYde7sW+PnnRet23dfDTkUQAlcJDs8/LBfCKsxM3jttWDikVBQAhfJFma+NT5zZrTu8MPVGs9jSuAi2eaii2IPObz//mDikcAogYtkK+eadqFMnhxtpUteUAIXyWaHHRZ7zfHGDz0lZymBi+QC55oOL3zkEd8abzwMsRXV67dx75LVVK9vcWdECRFtaiySK3r18ol8xAh4/XVf16UL7LMPbGs5KVev38bEWUv5sraevTp2YM6FldrrNAuoBS6Sa6qrYffuaHn7dt8aX7s27ilL19TwZW099Q5219azdE1N+uOUdlMCF8lFHTv61vjUqdG60tK4Qw4rS0vYq2MHCgwKO3agsrQkQ4FKe7S6K30qaVd6kYA0T9xPPRVdvjaiev02lq6pobK0RN0nIRNvV3r1gYvkA+fgySfhtNN8+YQTovURI/oXK3FnGXWhiOSLU0+NPQGo8dZuklWUwEXyjXNNN1G+4QafyJuvtSKhpwQuko+++U2fyBv3jRcUqDWeZZTARfJZfT3s2hUt33ADfOMb8OWXwcUkCVMCF8l3nTr51vjs2b78/vtQVORnc0qoKYGLiDdpkm+Rn366L593nu9i2b490LAkPiVwEYkyg8cfh//932hdcbH6xkNKCVxE9nTQQb5b5ZprfLlhpMr69cHGJU0ogYtIfNOmwZYt0fKAAXD++YGFI00pgYtIy3r29K3x3/7Wl3//e98ar64ONq40yLYldZXARSQxkyf79cWLI9PtKypg5Eioqws2rhRpWFL3jqdXMXHW0qxI4krgIpK4zp1h61ZYtMiX/+d//MqHCxcGG1cKZOOSukrgIpK8E07wLe8jj/TlU06Bbt2S2gEobLJxSV0lcBFpmw4d4KWXorv/7NzpdwCaMSPYuNpoRP9i5lxYyVXHDcyaHYm0HriIpMYPfgAPPhgtb9oEvXsHF08OibceuFrgIpIaDzwAf/97tLzvvvDP/xxcPHlACVxEUqdfPz/k8Fe/8uU77vBDDhvP7JSUUQIXkdS79tqma6gMHOgfdGawyzYfKIGLSHp07+4T9h/+4MsLF/oHny+8EGxcOUQJXETS67vf9euLl5b68ujRfq2V3buDjSsHtJrAzewBM9tsZssb1d1kZh+Y2ZuR14npDVNEslphIbz3Hjz3nC+vXg177QVz5wYbV5ZLpAU+Gzg+Rv2dzrmyyGtRasMSkZw0erRfc/ykk3z5nHP8Q85PPgk2rizVagJ3zj0PbM1ALCKSD8x8f/jKldG67t3h1luDiylLtacP/MdmtizSxRJ3ypKZTTazKjOr2tJ4WUoRyW+DBvmHnFde6ctTp/rkvmFDsHFlkbYm8BnAAUAZsBG4I96BzrmZzrkK51xFr1692ng5EclZ06fDP/4RLffrBxddFFw8WaRNCdw5t8k5V+ecqwfuB0amNiwRySv77utb4/fc48uzZvnW+FtvBRtXyLUpgZtZn0bF04Hl8Y4VEUnYlCnw6ad+USyAsjI46ij/4FP2kMgwwkeBV4CBZrbBzH4I/KuZvW1my4CxwJVpjlNE8sXee/uVDRcs8OWXXoKCAvjLX4KNK4S0GqFInqtev42la2qoLC0J3xKqdXVw+OHR7dtKSvxDzk6dgo0rw7QaoYjsIfTbiBUUQFWV3/kHoKbG7wo0a1awcYWEErhIHsuabcQqKvxDzokTffmii/xDzpqQxpshSuAieSzrthF75BFYuzZa7tkTrrsuuHgCpj5wkTwX6j7wlvziF3DTTdHy6tVwwAGBhZNO8frAlcBFJHtt2wY9ekTLZ54Jf/qT717JIXqIKSK5p7jY940/9JAvz5vn1xxfujTYuDJECVxEst/558Pnn8N++/nyEUfAkCFQWxtsXGmmBC4iuaGoyI8RX7zYl1es8OuQP/54sHGlkRK4iOSWY47xU+/HjfPlM8+Ejh397M4cowQuIrnHDJ55BpZHlmmqq4Nu3eDOO4ONK8WUwEUkdx1yiH/IOWWKL191lU/uGzcGG1eKKIGLSO675x748MNo+etfjyb1LKYELiL5oU8f3xqfPt2X77vPt8ZXrAg2rnZQAheR/HLllbBjh3+wCX644bhxPrlnGSVwEck/XbvC7t1+4g/AX//qJwD99a/BxpUkJXARyV9nnOET+ZAhvjxuHPTtC198EWxcCVICF5H81rEjvP02vPKKL3/wgd8womF6fogpgYuIAFRW+glAZ53lyxdc4B9ybgvZJheNKIGLiDQw86sZrl4drevRwy9dG0JK4CIizR1wgB+Vcu21vnzTTT65N95MIgSUwEVE4vnVr+Cjj6Ll0lL43vdCM+RQCVxEpCUlJT5h33+/Lz/6qB9y2LDRcgKq12/j3iWrU75ptBK4iEgiLrwQdu3y+3ACjBwJ5eV+oawWVK/fxsRZS7nj6VVMnLU0pUlcCVxEJEHVm3Zx7x9f4d2H/uQr3njDD0NcsCDuOUvX1PBlbT31DnbX1rN0TU3K4lECFxFJQOOW9CnvdqF6zUdw1FH+h6eeCnvvDZ9+usd5laUl7NWxAwUGhR07UFlakrKYlMBFRBKwR0t63TZ44QV4801/wK5dfor+Pfc0OW9E/2LmXFjJVccNZM6FlYzoX5yymJTARUQSELclPWyYf8j5wx/68mWX+SGHmzZ9de6I/sVMGXtgSpM3gLkMDoepqKhwVVVVGbueiEgqVa/fxtI1NVSWlsROxhs2QL9+0fIVV6RkFyAzq3bOVTSvVwtcRCRBrbak+/b1rfFp03z5rrt8a/ydd9ISjxK4iEiqXXMNfPxxtDx4MPzjHym/jBK4iEg6fO1rvjU+dy6ceCJ06ZLyS3RM+TeKiEjU2Wf7VxqoBS4ikqWUwEVEspQSuIhIllICFxHJUkrgIiJZqtUEbmYPmNlmM1veqK6Hmf23mb0beU/t/FAREWlVIi3w2cDxzeqmAoudcwcBiyNlERHJoFYTuHPueWBrs+pTgYcinx8CTkttWCIi0pq2TuTZ1zm3EcA5t9HMesc70MwmA5MjxZ1mtqqN10yFnsBHrR4VLMWYGoqx/cIeH+RPjP1jVaZ9JqZzbiYwM93XSYSZVcVa0StMFGNqKMb2C3t8oBjbOgplk5n1AYi8b05dSCIikoi2JvAFwKTI50nAk6kJR0REEpXIMMJHgVeAgWa2wcx+CEwDxpvZu8D4SDkbhKIrpxWKMTUUY/uFPT7I8xgzuiOPiIikjmZiiohkKSVwEZEsldMJ3MzWmdnbZvammVVF6gJdBiDZpQnM7FozW21mq8zsWwHFd5OZfRC5j2+a2YlBxRe5Zj8zW2JmK81shZn9JFIfpvsYL8ZQ3Esz62Rmr5nZW5H4fhGpD9M9jBdjKO5hs1gLzOwNM1sYKWfmPjrncvYFrAN6Nqv7V2Bq5PNU4NYMxzQaKAeWtxYTcDDwFlAE7A+8BxQEEN9NwD/HODbj8UWu2wcoj3zuBvxvJJYw3cd4MYbiXgIGdI18LgReBSpDdg/jxRiKe9js2lcBfwAWRsoZuY853QKPI9BlAFxySxOcCvyHc+4L59xaYDUwMoD44sl4fOBn/zrnXo983gGsBPYjXPcxXozxZDRG5+2MFAsjL0e47mG8GOMJ5O+jmfUFTgJmNYsl7fcx1xO4A542s+rIlH5otgwAEHcZgAyKF9N+wPuNjttAy0kgnX5sZssiXSwNvw4GHp+ZDQCG41tnobyPzWKEkNzLyK/9b+In4v23cy509zBOjBCSexhxF3A1UN+oLiP3MdcT+CjnXDlwAjDFzEYHHVCSLEZdEOM+ZwAHAGXARuCOSH2g8ZlZV2AecIVz7pOWDo1Rl5E4Y8QYmnvpnKtzzpUBfYGRZjakhcMDuYdxYgzNPTSzk4HNzrnqRE+JUdfmGHM6gTvnPoy8bwaewP+qEsZlAOLFtAHo1+i4vsCHGY4N59ymyP9I9cD9RH/lCyw+MyvEJ8Y5zrnHI9Whuo+xYgzjvXTObQeexS8bHap7GCvGkN3DUcAEM1sH/AdwjJk9QobuY84mcDPrYmbdGj4DxwHLCecyAPFiWgCcY2ZFZrY/cBDwWqaDa/iLGHE6/j4GFp+ZGfA7YKVzbnqjH4XmPsaLMSz30sx6mdk+kc+dgWOBdwjXPYwZY1juIYBz7lrnXF/n3ADgHOCvzrlzydR9zMQT2iBeQCn+ae9bwArgukh9CX4Tincj7z0yHNej+F/7duP/Nf5hSzEB1+GfVK8CTggovt8DbwPLIn8B+wQVX+SaR+F/7VwGvBl5nRiy+xgvxlDcS2Ao8EYkjuXADZH6MN3DeDGG4h7GiPdooqNQMnIfNZVeRCRL5WwXiohIrlMCFxHJUkrgIiJZSglcRCRLKYGLiGQpJXARkSylBC4ikqX+D/bCIRuaBXJmAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "The slope estimate is: -0.0574462986976377\n", "The intercept estimate is: 34.16752817399911\n", "The slope standard error is: 0.004464173160311544\n", "The intercept standard error is: 0.8671972620941928\n" ] } ], "source": [ "res_lr = stats.linregress(x, y)\n", "\n", "plt.plot(x, y, '.', label='original data')\n", "plt.plot(x, res_lr.intercept + res_lr.slope*x, 'r', label='fitted line')\n", "plt.legend()\n", "plt.show()\n", "print(f\"The slope estimate is: {res_lr.slope}\")\n", "print(f\"The intercept estimate is: {res_lr.intercept}\")\n", "print(f\"The slope standard error is: {res_lr.stderr}\")\n", "print(f\"The intercept standard error is: {res_lr.intercept_stderr}\")" ] }, { "cell_type": "markdown", "id": "3aa96aa6-be6a-4e9c-a55c-f5e4e14fb6c4", "metadata": {}, "source": [ "`linregress` produces point estimates of the slope and intercept as well as standard errors for each statistic, assuming that the residuals between the best fit line and the data are normally distributed. We can test the normality assumption using `stats.shapiro`." ] }, { "cell_type": "code", "execution_count": 13, "id": "b5eb0671-a179-47e7-809c-8bc39f56843d", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "ShapiroResult(statistic=0.9171469211578369, pvalue=0.03371993452310562)" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "e = y - res_lr.intercept + res_lr.slope*x\n", "stats.shapiro(e)" ] }, { "cell_type": "markdown", "id": "77de4703-5400-4a4b-a928-0af8878c330a", "metadata": {}, "source": [ "Although the $p$-value is not small enough to conclusively reject the null hypothesis at all reasonable confidence levels, it does suggest that we might want to relax the residual normality assumption. `bootstrap` makes no such assumption about the residuals, and it can go beyond the standard errors, producing bias-corrected confidence intervals. The standard errors produced by `bootstrap` match those produced by `linregress` fairly well; however `linregress` may overestimate these quantities for this data." ] }, { "cell_type": "code", "execution_count": 14, "id": "b393954f-53a8-4e69-baf2-6f75f79020f3", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The slope standard error is: 0.004260082018022721\n", "The intercept standard error is: 0.7292901429594613\n", "The confidence interval on the slope is: (-0.06516915578638195, -0.04868503288913481)\n", "The confidence interval on the intercept is: (32.58844595440181, 35.45478220570269)\n" ] } ], "source": [ "def statistic(x, y):\n", " res = stats.linregress(x, y)\n", " return res.slope, res.intercept\n", "\n", "res = stats.bootstrap((x, y), statistic, vectorized=False, paired=True)\n", "\n", "print(f\"The slope standard error is: {res.standard_error[0]}\")\n", "print(f\"The intercept standard error is: {res.standard_error[1]}\")\n", "print(f\"The confidence interval on the slope is: {res.confidence_interval.low[0], res.confidence_interval.high[0]}\")\n", "print(f\"The confidence interval on the intercept is: {res.confidence_interval.low[1], res.confidence_interval.high[1]}\")" ] }, { "cell_type": "markdown", "id": "1dceaaf7-ba8e-44d7-afb7-1cc1e74ed491", "metadata": {}, "source": [ "Again, because the statistic has multiple values, a visualization of the bootstrap distribution may be more informative." ] }, { "cell_type": "code", "execution_count": 15, "id": "662eee2a-0c36-4f9e-bbaa-8896f328aa8c", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[]" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAABP8klEQVR4nO29d3ij13Xn/72oBEiABMHOGU7jSKPqUTVlxbFcIztucouysdyleLPKJi5ry5ZjO3GP2zprrXddlFjFtvSLbEtRXOJVrCguY3nGGhVrpCkcksNhB0ASBEDU+/vj8OC+aOwFJM/ned6HxEV7AWm+7+H3nqK01hAEQRA2H7aNPgFBEARheYiAC4IgbFJEwAVBEDYpIuCCIAibFBFwQRCETYpjPd+sqalJ7969ez3fUhAEYdNz5MiRCa11c/H6ugr47t27cfjw4fV8S0EQhE2PUqq/3LpYKIIgCJsUEXBBEIRNyoICrpSqUUo9qpR6XCn1e6XU386tf1wpdVYpdXTueMXan64gCILALMYDTwJ4kdZ6RinlBPALpdSP5+77stb6C2t3eoIgCEIlFhRwTc1SZuZuOucOaaAiCIKwwSzKA1dK2ZVSRwGMAfiZ1vo3c3fdrJR6Qil1u1IqUOG5NymlDiulDo+Pj6/OWQuCIAiLE3CtdVZrfRDADgBXKqUuBPA1APsAHAQwDOCLFZ77da315Vrry5ubS9IYBUEQhGWypCwUrfUkgIcBXKu1Hp0T9hyAbwC4cvVPz5DNruWrC4IgbD4Wk4XSrJRqmPvdA+AlAJ5RSrVbHnYdgKfW5AwB5HIk4KmUCLkgCAKzmCyUdgDfVkrZQYJ/r9b6QaXUnUqpg6ANzT4Af75WJ6kUCbdSdDubBZxOc1sQBGE7spgslCcAXFJm/YY1OaMyZLNAJgM4HEA6bX4qRUIuCIKwHVnXXijLRSk6UikTdSsF2O20ZrfTIQiCsJ3YFAKeSgHxOIm2y0XRuFLkjdvmXHyxVQRB2G5sCgFPJoHeXiAQAOrryUJxOACtSbizWROBi60iCMJ2YVMIeDgMPP444PMBXV3Ajh2A10ui7XCYLJVs1gi72CqCIGx1NoWAz84CJ08CtbXA+Dhw5gywcyeJeU2NEetcjh7rdNIhtoogCFuZTSHgk5PAM8+QWAcCFJGHQsDgILBvH9DRAbjd5Ifb7eSRp9Mk3lrTutgqgiBsNTaFgI+MAP39JMINDSTezc1AWxuJeUsLsH8/0NpKForNRkc6TRugLhcJufjjgiBsJTaFgJ85Azz5JInyjh1AMEjCPT4ONDUBnZ10u7MT2LWLhDyXM6mGyWShkNtsJPTMkf4IDvWG0LM3iMt2le3JJQiCUHVsCgEPhehnLgcMDNDR0UFCHokAExPA0BAJ+vCwEfKWFvLBWbQTCRJzt5vE3OEAHjsTwZ998xBSmRxcDhvuflePiLggCJuCTSHgz0xE4O8JYXYgiNQQievQEB0dHSTU4TAwOkq3R0fJdtm1i8R8KBnBbwdCuKQjiIM7AwVC/ssTIaQyOeQ0kM7kcKg3JAIuCMKmoOoF/Eh/BNHZ7+L+kW/hEy+9Cb/42ZvyIg4YIW9ro2NyEjh7lqyW4WEATRH8U98hZHI5OB02/O839uCC1gCyWYroL2oNwmW3IZ2l+3v2BjfsswqCICyFqhfwQ70hXDhyHNf0/Q7X9L0bwLtxFX6FQ7iq4HEjI3Tw5ubkJHnnyb0hpN05aFCE/cixEM5vCcBup03O85oDuO2NPXjsLHngl+yU6FsQhM1B1Qt4z94gvnzxH2HKVYuv3/9pAMCv8TwAwCvwr/gxCmcpj4+bzc22NqAmFQQusgEqB6VssIeDOHrUbIYCwLnBAC5sC8BmI+GvqQE8HskfFwShulnSQIeN4LJdAbzC24N7Rt8Od0cYL8DD+ft+hD+GhsKf4a6S501MAE89BZw8FEDNr3rgPXUu2o714NRvAjhyBDh8GDh6lDZIuQgokaCoPBajzdFkcuHzO9IfwW0/P4kj/ZHV+9CCIAiLoOojcAAIZAOYPkTWxiN4ARQ0LsbjeBwHAQB34QbchRvwXnwRX8Z7C547OQlM/jIAlyuA9nZgpp4KgBobgYsvJgFvaaHKzkCABDyXow3O6WkSd5+vfP74kX7JYBEEYeOo+ggcAHp6gD17CteewHOgoLEXp5ACqeuX8D5oKHwOHwDNmTCkUlQM9MQTwO9/Dxw7Bvz7vwMPPEDR+GOPUcQ+M0NinUyaiHxqirJciqcBHeotzWBZCyTKFwShHJsiAm9pAS66CNi9Gzh1ivLAmdPYCzdSaMEofosr0IUz+AA+jw/g87gDN+AduB3Zoo85OkpHQwPQ3g6MjVH64cGDZL0Eg5SC6PNRG1u7nTzxyUnKJ29oIH+8Z28QLocN6czaZbBIlC8IQiU2hYB7PNS4amgIOO88oLubouneXirSAYAxtGIXBuDDNB7Ci3EFDuMtuBNvwZ34Cf4Ir8UPkURNwetOTpq0w9FRk354+eW0HgyStWK3U2TOQh4OU5R+aVcAd7+rZ02rOMtF+SLggiAAm0TAm5pIuL1eiojHxsij3rWLMk6eftrYG1H4cSV+CxeSuA+vxyvxr7gWP8UsPACALvTjDLoKXn96mg5rNWdHB3DVVbTu99N7KQVEo1TB6fFQ06wDTYE1FdT1iPIFQdicKK31wo9aJS6//HJ9+PDhJT9vZgY4dIhaxY6MkIUyNUUbkJOT5FWHw8Czz9LvVmzI4jv4L/gT3Fuw/u59n8M/Jm4sKApi/H5KQWxvpwj8qqtI0Ovr6TanFzqdJOQOh0k9XAukV4sgbG+UUke01peXrG8GAU8k5opykuRRJ5MUhZ89S4IeiZCAJ5MUQZ84QZF6IRq6zJ7tP9S+G38V+1rZ9/X7Sbibm4G9e0nIOzvNOmA6HHo81F/F5Vo7IRcEYXuyqQV8dpZsjUTCeN7hMEXmk5NkfYyMkHhHIiTeoRD55OFw4Wv5e05i6tD+kvf4NXrwPPy67Pv7/dThsK2N/Pcrr6RIPBikdESey8ni7XYbURcEQVgplQR8U3jgjMtFP7NZygQJBs2xezdZK6OjJOJ1dbQeClH0PjFBz50dCKLrvT+CsuXQ/4VX5l/7KhyCBnkjqigFkT3ykRG6kJw8CZxzDm127txJHr3fTxcXbpLl9VI+ud1O9oogCMJqsykEXClK6UunKRrX2gikx0NHYyN51Lt2kb0yPEw//X5KQzx9miL1FICZp3YAANwdYaSGAnnhZioJeTRKx+ioEfLzzqMUx64uumDU1Rkhj8fpvHM5itJFyAVBWE02hYA7nSTYPJQhkzFDGmpqzIAGp5OEPBgkj5qFfGSE7m86N4Kz5xyCtuWgszbE5oSchfpJXIgL8fv8+y4k5Lyh+vTTwIUXAuefTwVH9fWFQu520xpPBRIhFwRhNVhQwJVSNQAeAeCee/w/a60/ppRqBHAPgN0A+gC8SWu9JqWCvFHIE+hTKbrNsy+TSRLx2lozyNjno9L4zk4S8mAQ+GUkBNhzUApQyKFubwhhSxbKRXgKAPB13Igb8c38uhHyHGCJ1mdm6BgZAfr6aG7ngQMk5OecQxed2lo652TSzPRMJMhaYUtIEARhOSy4iamUUgBqtdYzSikngF8A+CsArwMQ1lp/Vil1C4CA1vqD873WcjcxGZ6uk8vR7VSKRNwq5DyVXmv6na2MiQngiaEI7jhzCFmdg9I2tB3rwXRvAH195bJWgL/AbbgNN5esuzGLFNyl627a7Ny7l6yVCy8kIXe76YLidtNFyOuliNxuFyEXBGFhViULRSnlBQn4fwVwB4BrtNbDSql2AA9rrc+d7/krFXAmlyOB5uKdbNYMMOYIPR6nxzgcdN/kJNkeTw5F8NR4CLaJINLDAUQi1H1wdJSyVmZmSt/vRXgID+ElJestGMU4WkrWXS6ycHbtooj84ouBffvIVvF6KRLnYqCGBhJxtoAEQRCKWZGAK6XsAI4A6AZwm9b6g0qpSa11g+UxEa11SZWJUuomADcBQFdX12X9/f3L/xRFcDTOgq618cdZ1GdnSdCVovs4zTAWo5/Hj1M++fi4qfIcGCDBL6YbJ3AC55SsPwdH8QSeU7LucFBp/s6dwAUXUETOQl5ba4S8tpYicquXvxikwEcQtgerFYE3APgBgL8E8IvFCLiV1YrAi2Eht9orbKGk0yTmsRjd5owQ7jDI/b/7+kjIh4bo9sSEEfZi6jGJSZR+1OvwffwQ15Ws22yU5tjRQdH4gQO02dnQYITc5TLWilJ0226v/JmlyZUgbB9WJQ9caz2plHoYwLUARpVS7RYLZWx1TnXpKGXEzmqvcFFNNkuPqa2lDcR4nKJgn4+i9dFREs99+yhr5cwZEvTWVhLy4WESdmYKDVDQsCODDEy4/AO8DgBwKz6JT+PW/HouR423envpovDUU+SRn3suvWdjI50bWz8eD212ApWFXJpcCYKwmCyUZgDpOfH2AHgJgM8BeADAWwF8du7n/Wt5oouFqyJtNhJOpej3+noSSLudRD2TIb+b/WqlSKhraqgHyv79JLinT5OQd3SYAcpMFo58iqE1l/xT+Ag+hY/g+7gOr8f3C87v9Gk6BgeBJ58kIb/gAtr4DAZNuiRnrTQ00POKhVyaXAmCsJgslIsBfBuAHTQA4l6t9d8ppYIA7gXQBWAAwBu11uHKr7R2Fsp8lPPJs1kSyGSSbs/MmCwWrclemZ6maD0Wox7kvb20FomQ0A8Olg54AFBSFAQAY2hGa4U/UPbtowtGdzeJ+f79VNnp9dJfCW43ReR+Pz3eKuTigQvC9mBT90JZDcr55NksRbvsjadSJNiZDN0/PW2yV5JJipz7+2mjMxKh/O+JifKZK+WEHCgtCmL276dIf8cO4LLLyDNvaSHxrqujn14v2T7Awh65IAhbhy3RC2UlWH1ya+TMPnkmQxE355BHo8abTiRIsD0eEtbBQbJSBgeNkIfD9Hv+/eaEOowAApjMr1eq7jxxgo79+8kn7+ykDc+9e6mJFldzxmJGxAGJyAVhO7NtBNxKuciVW8FmsyTmNhuJfixmbIxMhrJSGhooUt6zh8S2r48idY7IrR0Q2zp60Xr9IXzr/o/jpaceza8vJOT79tFmans7pR92d5Ooezz0F0M0ahpoKQU8NRLBDbdLVoogbCe2pYAzLORKGWuFqyMdDhJsu93klsfjZiqQ2019wpubKSofGqLmVizk3BWxpisEZc/hxjd8FDoL3PS9f8Otg/+QPwcWcjsyyMFcWU6domPPHnrtJ54gj3zvXjpYyN1uuqD88nj5rBSJygVh67KtBZzhzBW73ZTr822n0xQFcYGN00m+9MwMCX1rK208dnaScB8/TlkrIyNAaDoInbUBOgeds+Hvch/HR/AVvA734T68IX8OPHi5ARFMoSG/zlkrnZ0U7be3U3n+3r2m38rsLLDPH4TTbkMmm4PTbsOVu4OSKy4IWxwR8CLK+eQ2G4k2e+X8GLebPHJro6rmZjPpnjY8Axg83INJVwgTTwfzI9y+j9dDQeMgHsNjuDT/XlwgdADH8CwO5NfPnqWjuZmyYI4fNxH6uecCLbUBfPqlPTg2EcLV5wRxXnMA3/r1SckVF4QtjAh4BYp98mJ7JZslz5zTEb1eitJDIdpsZCEPhYC+vgBGRgIYaQWGshSlM0dxCRQ0OnAWZ7Ejv/4MzgMAXIsf46e4Nr8+Pk5HUxNZKyzk+/YB3d0BvKQzAL+dvPiL24Jw2W1IZyVXXBC2ItsmjXClVCrX51TERILWuffKxITJJee0wzNnTLn+8DAdxdQggQS8JeufwS34MD5Tst7URFF4ezv93L+fPPnGRspWGYhH8PQEeeBX7g3kL0KCIGwetn0e+Gpi7YJoXeNUxGzWCHkkQhubs7NmY3NwkBpmjYyQoA8PG9/dUH4I82M4iEvxWMl6UxN1P+zooPzxiy+mKUFNTfQXQSBAfyVwto0IuSBsHrZ9HvhqUix82WyhvaI1eea5HHnjwSBVd3q9RmB37CBBP32aIvS+PhJ0I+SqbJn+JThaNgVxYoKOkydpg3N0lMT74EF6z9ZWEnIu1xchF4TNjwj4CmDh474r1iiaZ2E6neSNe71ka0SjlALY0kIFOu3tlDfe1maEfGio8LXKCbn1tlXIp6aAxx6j19m5k0S9oYGqOzs6SNQbGsijr60VIReEzYwI+CpQrsqTo3Ie85bJkI3Cvb9jMWNtxGIkqJEI8OyzFD0PDtKRTlveZwlCHonQcfo0eeK8ufrc55osmaYmupD4fHSh4bF1IuSCsDkQAV9lFrJXADO/s76eBHRqivLKYzGK0iMRap7FFsvgIPnpzFKEPBqlrocnT9ImZzhM73HFFRT1j4/TOTQ1UWUni7jTKUIuCNWOCPgaUVzlabVE/H6TllhTYwYwT0+b3ivNzSS27e3kjff2krUyO2teh4X6UVyBK2A2h8sNYU4kgKefJiHft88Mer70UvLjGxpI0INB+j2TkYhcEKodEfA1hqs8uWcJYKLyujr6vaaGxJI3GaenSeSnpwuj5IkJ6pMyOlo4hPlK/BYAcAduwA24K7/OWSx+TCEK6kebSgHHjlH++L59FO03NFDWys6d9Ht7O71ffT354yLkglCdSBrhBlAuDZE3Qbk1bSJBAj40RCI7PW0icU5DHB4mi6SYd+Eb+AaNIS3gHDxbMtNTKcpa2b2bBPvCC42Qd3YaIfd4SMRFyAVh/ZE88CqEBdxaHMTr8bgpEorFqAgoHKbfh4aoTJ/neA4OksAXcwUexaN4bsn6q/AAHsSrCtZsNvLId++m6P/88ymPPBAwqY+NjeavBclaEYT1QwS8iikeMsHkcpSFMjtLnnQ8bop/kkkTiff3G0G3WitMEBOYQHPJ+sfxMfwtPl6yvmsXiXl9PQn5jh3kyXd0kKC3tFAfGLd79YRcuiYKQmVEwDcJlewVjsq52nNykuyURMLM6uTByQMD5acEFQ9hZh7B8/ECPFKy3tFBfcjr66mVbUcHefIdHWbT0+MxEbljmTsq0jVREOZHKjE3CcXRLF9feTBzLmeiX84dZ5tj1y4SXBbywUESeqbSEOY/xH+WTUHkCwO/T309tbDt7KT34yKhjg7KnmERd5ZeI+blUG/5XuaCIMyPCHiVwkKeyxVmr9hslL3i9ZKN4nBQDnckQmIbDJKP3d1NlsrJkyTk1ilBwNJyybkDYjBIBUHBIGWwdHTQwd0RuVyfJxq53Yv7rD17g3A5bEhnpGuiICwFEfAqx5qGyLCQW+d52my0ybhjB4lsIGDGvvX1GTGfmip8/aUIeShkBJwFnRtodXXRGk8pCgTMxaemZv7PeNmuAO5+V4944IKwRETAq4SFNvGs5fpMLkfC7XKReGezFJH7/SSu0Sjlje/cSRHzvn0k5MePF1orwPKE3OejTdVgkAS8s9O0tm1pIRFvajI58G63+WuimMt2BdZFuGWzVNhKiIBXAUvdxCuX9cHFQfX1JJgeDwks91159lkS2e5uEvLTp0ncQ6HC12GhzsIGm0W0K5XpP/YY2Tlc9s+DmFnIOzroHKxCvlFl+rJZKmw1RMCrgOVu4rEIFtsrSpFPXltLx+wsReWxmJnX2d1NBTwDA1SZWSzkdlBe4/14NV6Nf8mvs5DXtY8gNtwKgLJjnnySov/hYfLCBwcpCu/uNkLOXRA3Sshls1TYaoiAVwEr3cSrZK8ARsRZyD0e2vw8fZqslbNn6efwMAm5ddwbALwGDwAA3ocv4Av4H/n1meE2AEAHzmIYHQDIi3/mGTq6u8lSOXOG0g27u8mTb2sju6e5mbxxm42Ev9xnWG1ks1TYaiyYB66U2gngDgBtAHIAvq61/opS6uMAbgQwPvfQD2utfzTfa0keeGVW25stnhbEcEEQTwkaGCARHxgwvciPH6cIuhh/z0m8cN/9+OHd7y+57yX4GR7CS0rWOzooK6a9nSLzPXvogtHZaTY8XS5Tpr/WQi4euLAZWXYhj1KqHUC71vp3SikfgCMAXgvgTQBmtNZfWOxJiICvP9ZBE8WiHouRjz0zY6be9/VRVefgIP3s7TXPcXVE0Hr9IShbDp2RcfzqW28veb8P4dP4LD5Ust7aSpZNWxv9vns3ReRdXSTiwWBhmT5n30i5viCsYiWmUup+AF8FcDVEwDcVlWZ5JhJ0jI+ThTI0RNbHiRNkrfT3UwoiQCJe0xXC7EAQqaEAvIghhrqS93oYL8AL8XDJen09DV7miHzXLorG+WdDAwm5200izgIuQi5sZ1ZFwJVSuwE8AuBCAO8F8DYA0wAOA3if1jpS5jk3AdQar6ur67L+/v5lnL6wmpSzV3hq0MwMCTiL+Zkz5GlzD5YTJwqfbyg/hBkozFxhamupPJ9Hy+3cSYLe3U1ROXdALNfKVsRc2G6sWMCVUnUA/gPAp7TW31dKtQKYAKABfAJks7xjvteQCLy6qCTkuRwV/AwPUwXnyAj9/vTTZK0MD5O1Yh0uYaU4l5wpJ+ReL1krvMHJnQ+5ZL+lRYRcEFbUC0Up5QRwH4C7tdbfBwCt9ajl/m8AeHCVzlVYJ8qlIfJ6IECphzMzptdKRwdF5b//Pf0+PEzWSjJZ+PylFAXF48BTT9HFYf9+isTb2ijab2sjceciIbebRJyHMPNFRynjmQvCdmJBAVdKKQDfAnBMa/0ly3q71np47uZ1AJ5am1MU1ppymR/sP9fXmxREblzV0kIWy7FjJLI8gDkWK3rdJQh5LkfFRidPkojv3Uvvc/Ys+eVdXeST79pFEbnNZgYxa23EXKJyYTuxmAj8agA3AHhSKXV0bu3DAP5UKXUQZKH0AfjzNTg/YZ1hAbQOY+Yug93dJLTjOoIzsyHsuSKIc88N4NQpEl5OSVyJkGezpndLZycJeWurSW/cs4dEfO9eurCk0yYq5+cDJPCVyvYFYauwoIBrrX8BlDU15835FjY3xfYKC/nhvgg++vNDSGdzcNhseM/BHlzVEsC+fSTiLL7Dw6VTglioo6hDHYzKs5DbkC3YCOXUxs5OyoTZsYNa2J48SR58Zydw7rnUMoAbZ7lc9NM64UiicmGrIpWYwrwU2yu/GwwhnaVy9Gwuh2l3CC+6OICBAcrnPnCAPO2hIRLaUKi0la0PNG3iTrwZb8bd+fUc6I0aEUIEjfl1FvIzZ+jisHMnCfnx4yZSP+88sngAuui4XMYXl6hc2KqIgAuLxm4HrtoXhOvnphz9efuDaGmh3it79lCJfiBABUKPP24KgkZGqGe5lRtwF27AXSVDmMOgEvdLcQSP4dL8+sgIHTx9iCPyU6dIyHfsKBRy3tzkARMSlQtbDRmpJiwZLkd/7p4gLtlJ5ejxOPnRiQSJN6cbTk1RRN7ba3LJiyNy5jIcxmFcUbL+Bvx/uA9vKFlvbqaNzs5OEu+WForOW1pIyJuazKQgwPjkViQqFzYDMhNTWDN4KLPWJOCpFGWtTE1RhHzmDGWtHD9OQj40NL+QVxrC/Dl8ALfgcyXrDQ1m8DIXBHEO+Xnn0brPZyJxp7O8aEtULlQrIuDCmlDcHCqbNeX5mUyhkA8MmL7hp04Zb7uSkNuQRbaMy3cv3og/wb0l6z4fCTg3yerqohTE9nbKWunsJKvH46HHOxzl88clKheqDRFwYdWZb0CCVchTKTrCYYrEBwdpc7O/n3K/udKzuJWtlXLVnWEEEESp+vv9JNrBIAn5vn30s6ODBH7PHhJ7bmdbKSIHJCoXqgOZSi+sOvMNSLDmjyeTJOBOJ9kdTU1mtmZXF0XmzzxTaLcUxxXlcskbESmbSz49TYfPR4VGIyMk4Lt3U8TPG6D795PYO53kj7PFYhVzKRASqhkRcGHZLGZAAg9fdrnocLvpp99Pgx0mJ8mz7uqiTJJnn6VIvb+frJdUqvD1llIUFI3S4fWSkA8Pky++Zw+Jen8/vfeBA3Qudrs5z2Ibxdo3RsRcqBbEQhFWxFIHJGQylK3CG52xGIn41BRZKxMTFCH39lIkPjBAIhyPl3+9pTTOqq2lSLyhgYR81y6yVJqa6Pa+fWSz2GxkrzgcJiovhwi5sF6IhSKsCUudJs8dBe12isZraihCb2ykDcZwmAR2/37a6GxtJVEfHDTWiBUW6m/jLXgL7syvm4g8By4kjsXoqKmhnPTxcXqP3btJyIeGSMi7umjT02ajc7Tb6TnFSDMtYaORCFzYMHijEyBhjcfp9sgIievICG2Ccq+VcJg8bI7Yy/HX+DK+jPeWrLuQRBqFieBOJwl2czM17dqxg+wVvt3VRaX63MqWfXKHo9SjZyQqF9YCyUIRqharzx2NkrWSTlPUHQ7Thmc0SnYKl+ezkBdXdzLX4sf4MV5Rst6EcYTQVLBmt5N14vdTFWl7O9kpTU2mle6BA2TBKFXY1lap8mIuUbmwmoiFIlQtLheJYDpNmSM+H1kl+/aZToThMDDtisDbEEIwGkTziQAmJsx9xUL+E7wcChoHcAzHcH5+nQuEduM0+rEbAEX9Z87Q/Z2d9FoDAyTk3d0m9bGxEbjgAhJ1bpzFvcldrsKNTmlxK6wHIuBCVcCCmE6T+Pn9tO50UuT7aG8Ed/z2ENI56oL42uf2YH84gOPHScDHxigyHx8vfN1ncB4UNAII53usAEAf9gAAevBr/AY9+XUuLmpvp4vI0BBF4vv308VkbIwE/MABsl9SKYrGs1mKuB2O0qhcvHJhrRABF6oKzvpgW8XvJzE8/rsQMjoHDSCrc4jWhPD8iwNobibhPnmSIuexMSoImpgobF4VQSMUNOzIIAOTWnIIVwEArsd3cQ+uz68PD5u0w5kZet1gkDY8d+yg96yvp9u7dlF2DWev8DAMicqFtUYEXKhKXC4S4EyGItcXHAji9kM26kNut+GlB4OoTZAQtrZSlDw+TrnkbW0kuMPDJOrptHndLBxlc8m/hz/F9/Cn+Bv8HT6Jv8mvj43RUV9vhPzkSRLxXbtoM/XkSfLJ9++n87WKNxczARKVC6uPbGIKVQ9H448NRPCbvhAu3xnE+a0BTE2Zzc1olCo+x8fpOH2axHV0lI7JScpoKUe5XPJ78CZcj3tK1hsa6ALh8VDaY1cXifnOnbTW1ARcdBH9brebKk+73ZTtW6NyKxKVC5WQLBRh01NclZlOk2jPzJBlMjxMv6dS5F1PT1NEzh45C3nxyDcmjAACmCxYO4W96Mapksd6vSZzxeulvwJ27SJBr6ujiP3CC+l3Tj/kFES2WCohzbSEYiQLRdj0sC3BESyLotttioHYOqmpIXH3+Sjy7u01RUHj43NZLUVFQY2gVJZ/w0vxUvw/AMA+9JYt04/HyTqpqSEhj8fpvXt7KZecs1nq6iiTpb2dxNvtJoHmqNxuL930lMETwmIRARc2FSx61mjc6aSI1+MhgWxupmj77FmKigESzFTKeOShkMlcKU5BfBl+BgD4PN6P9+OL+fVyQj47S6JdU0MXiJkZKkBqaqJqzuZmulDU1pLQd3ebbJviqLxc1C3j4IT5EAtFWHeW0j9lvsdy7rgVu52i4elpirx5DBtH7gMDdP/ZsyTeLOSjo5WrO9+Jb+KbuLFkvXgIM0AXkYYGEu7aWrJYurpIvLnCs7GRBk1Yy/Q5jXIhe4U/o7C9EA9cqArm6yG+3MdmMoW2A0BCGIuRkMdiJiMllSIhP3uW1rmBFg+d4ArPcrwID+EhvKRk3YsYEvCWrDc3U7Tv95OVwiPfWltJyH0+Kgxyu8lHd7vpeVZ7ZT4kKt8+iAcurCpL7ULIzNdDfLmPffwsncvlO4O4pIvuz+UoAvZ6ydbweilTZWSEfu7ZQxF6bS2J7Pg4RcacV37mTKmQ/zteDAWNc/EMnsF5+fU4agEAbRjGKNry65wR09hI7zE9TZF+QwO9f2srvYffT3ZLMEgRfC5XmEvOQq11oWCLVy6IgAtLZilRdDGL6SG+lMcWn8td7+zBxR1GxAGKdr1ek/o3PU1C7nCQgEejJKI8aGJsjIQ8HCbPvFjIn8UBKOiS2Z0jaAcAXIQn8BQuyq+Hw3Q0NZGYBwJ0Dh4PtQtoa6O/DurrTTaL3U7nyh442yws1MViLl759kQEXFgyS4mii7lsVwB3v6tnUdH7Yh5bfC6/OR3C5bsDBbYKV0k2NVFmistFwj01RWJtt5v+Kw0NJKLj4xQtB4P0uN7e0tmdITRBQcOFJJIw/WafxMUAgJfjR/gJXp5fn5igo7GRjoYG8uOPH6fNzZYWeo/BQXrf7m66yPj9tFkKmB7llYRcovLtxYICrpTaCeAOAG0AcgC+rrX+ilKqEcA9AHYD6APwJq11hd5wwlZiKVF0OZbSQ3yhx1Y6F275ypuc/LvTSQKdSJDn7PORaE5M0HPq6ykK9/vpcaOjphJzepoKhCYmCs8hBfdcZoou2NTkbog343/hNtycX+eI3OczI+YSCYrI29qoMGhykt67qYnsFq/XXHgyGXodj6ewipOtF0ai8q3PgpuYSql2AO1a698ppXwAjgB4LYC3AQhrrT+rlLoFQEBr/cH5Xks2MbcOy/XAN+JcrLnjDDegisfJI+eMlWiUNjpzORLQmRk6WMijUbI7+vtLG2dZKVfdeRv+AjfjtpJ1r5dsFW5fy4VA55xDFxKfjyydHTtI8PlCY7PReS5kr1iRqHxzsmpZKEqp+wF8de64Rms9PCfyD2utz53vuSLgwkZSnK1izSmPRikrJZmkjJVolAQ+l6OqzmTSbIKOj5vqz6EhWivOgmHKCfmvcBWuxq9K1nnABNsrDQ30V8L555OQ+/1m0EQwaDo1ut0k2rzpaRVp7sdSjETlm4tVEXCl1G4AjwC4EMCA1rrBcl9Ea10S/iilbgJwEwB0dXVd1t/fv+STF4TVghtkWWEhn50lvzseJ7vl7Fn6PZMhAR8bM2I/NkZCHouRgI+OmnzzchzFc/AcPFGwloEdTmRKHmu3m/TDpiYSaY+HWtg2N1PE7vdTbnlrKwk5b9LyXxZsr7BI8z/zcqItUXn1s2IBV0rVAfgPAJ/SWn9fKTW5GAG3IhG4UC0U2yo2m/GPrbYKdyBMpeiYmSELJZcz3vnoKAn82bMk6sPDpRcJ5na8HW/HP5WslxvCDFBEzj653092CZfq19SYKUJdXUbEa2rMkIxyxUGVonIR8uplRQKulHICeBDAT7XWX5pbexZioQibGO7PXc5WyWaNrZJKmWERNhuJczhM92cyJOI8FSgaJSGfmqJc8uIGXMz78AV8Af+jZN06hNlKczOJNf+sqaHf2SdnMd+9m366XEbIHQ7jlVtFWms6itvZir1SfSxbwJVSCsC3QRuWf21Z/zyAkGUTs1Fr/YH5XksEXKhGiqNx7tFt9cdnZkgEx8ZMyX06TeLNNgsL+dQUHSzkg4MmDbCY1+CH+CGuK1l3IlUweIJpajKbmizc9fXUwra+3qQd7tpFkbndThYMt7UFjE9enEdeHIFLr/LqYSUC/gcA/hPAk6A0QgD4MIDfALgXQBeAAQBv1FqHy77IHCLgQjVTLOTcw1spEmCOyHM5isanpymCTSbJ/85k6AiFSMjZbuHhzCMjJPbluBRHcAQl/z4RQBiTKHUm2Tpha4V98vPPp3Wvl7JZOjvpMTU1dFgbaPGwCatIV/LKJSrfWKQXiiAsgmyWRKycraI1iXg8ToKezZLfPTtLj49GKQq32WiNbRUW8uFhUyAUjZZ//04MYhA7S9b34SR6sa9kva6OPPKWFhLx+nqKsM85hzY4a2vpCAQoDdHrJdFmMWePnAXdSrl0RInKNwbphSIIi4BFzBqNW6NyjnY5Gu/sNBkqAAkoV2zu3EnCOTVFzwsGScBbWsyotuKe5GexAwoatZjBDHz59VPoBgA8H4/gF3h+fp3z1KenTcMsrtx0u+kcurroXEMhEvDdu8mGsdspardODWJBB8pnsFjnekpUvvGIgAtCGayDFsplrDQ0mAZZ8TgJ5cwMiXcgQEc4TJG530+PD4dJ4INBEtrRURLVoSF6rpUY6qCgYUMWWcs/0//EHwIA3ow7cTfenF+fnqYjGjUVnoEAnVt/P11oOjrITonHzSCKYJBE2+2mKJyrVVnQy5XqK0UH/5UiUfnGIRaKICxAsa3CkSdH69EoRbyJBHng09MUdXPr2nCY7teaxHNy0hQC8bi3UIg2PYuF3Eq5oqBP4CP4KD5Rss6j3rh5Fpfit7dTjnlHh2nu1dREa263Gf9mt9NtFnPOWMmfi2SwrCvigQvCCmABL97k5J9ak3DPzpKlkkqZXuSpFK1zGmIyaYqGEgmTrTI+Tl453654LmWE/Id4Da7DD0vW6+roCAQoKvd4jLDv2GHyzL1eI+S1tcb35w1PHl3H5ftWrFE5I1H56iIeuCCsAGvEDRhbhUXLZiN7hNvVJhIUufp8FFXHYqY3+cQECarbTRF5bW1hIVBjoxHycsMluOhnCO1oxwgA4LW4HxoKZ7ADXTiTfyx75JOT9P7skUej9F6BANk/zc10UZmYoMe0ttLjczm6ADkcppMjZ7IUR+HW2+KVrw8i4IKwBIoLYcpF5YEAecyxGAkXD12eniYxY5FPpUg4Z2aML71jB6Udjo2RP83ZK+ynW+nAMADgAbwKr8KDAICdGKw4u3N2ls6pro7e1+czlk5jI+BsiyDuC+E5kSD2jwfy0bnfT58xnTZiztaKy2X8cGsULl75+iACLgjLwG43IlWu26HHQwf747zG+eQARd7hMAl8XR2tJ5P0uK4uquQcHS2NyIvf69X4FwDAJ3ErbsWn8+vlhDyZNI25amtNB8SQimC0/RB0LIff9trwhqkeXBAP5Od6NjbSBYWrVLm1AGeucGTOFzVrVG69zecuZfurg3jggrBCitMN2TKwjkKbmjLiyc2xYjEj2qGQyfiYmjIRbzRKg5i5AyJvdk5NVe63cgPuwB14a8m6HRnkUKicLhdZP/6ek8ic9yxV8WugO34urmroRmOjmenJPnpDA/214JgL/7ggiEXc7TaRt1VerJvAjETli0M8cEFYI3gTEygtAuL7GxpMWX46TYLHWR6zsxR1x+Mk1H6/EXJ+biRCEXltLUXDExNG1GOxwve7E2/BnXgLno9H8AhekF/ndMRazOTneKZS9DqJJ4NoPMcGZctBaRtSg0Ec7StsX8vzO30+I+Zsr2Qy5nMlk/S5uPeKROVrh0TggrCKFEfj1giTBSoeN9kpmQw9ljc+OQ2RKzptNrqdyZDwhUKU1x0KmYg8HDY54OXoxgmcwDkl650YxBA687ddHRHU7w/BlwgikA3A5zPNsGprabOzvZ02N+vrjZBz3rnDYXxvruzkhlrOubYui4nKZdOzFEkjFIR1pFjIWbyttsrMjNlc5Fzz6WkS+FTKNMVi3zmRMNOCwmES8okJuhhMTFCEHI2Wz1wBqK9KGKXj7w7iMTyOg4WPnYuwg0GyT9xuU7nZ0UGZKnyfy2X6l/t8JvLO5crbK5ylYv1LxRqVW8VbonJCLBRBWEdYdNlCKN545EHKvNGZTpOQc6l+LGbSECMRsiXq6uhxXC3Z1ETCfeYM3ceVoJwmGCmaUBtBIxQ0nEghBXd+/SguAQBcjt/mG2pFIuZgH7y+ns4jHqfMmOZmOofWVpMOyYVDdXV0O5czefEOB31Ga+UnYMSbxZqjdJtNUhEXQgRcENYIa18VFvRstlCYHA4SvNlZk9HCYud2k5DX1JBAjo2ZrA+u7OzsJCEdGwP6+kg4OWuFBzFzbxYmDVfZIcyHcQUA4Bw8m7dcJieN9x0MmtFumQxdMIaGTDTe2mp6rnCFJ2ex2GwmDZELmzh7xToGrtgrt+baA7LpWYxYKIKwDlgtg/lslXjc9BfnYRCJhJnROTVlcsIdDhJRvjik0ySop0+T6MbjJKbRqPHVK/EELsJFeKpg7Qo8mhd1xuMhsQ4GSZy9XmOt1NeTiAcCZvRbba3py1JXZyo7M5lSe6WmpnSjk7FG5dbiqe0SlYsHLghVAIt3Oc/XGoWyP85C7nCYhlVKGc+bI9J43Dw/kaBioNOnjaceCtE6l+xXGsL8RtyLe/EnBWsvxb/h/+GlBWsej5nZ6febroZuNwl3Wxv9JdDWZh7r89Eal/OzTw4U9innniyMdeOTvzdrJL4donIRcEGoEuaLxq2/c654Om3S9AAS92iUHjc8TLet8zxdLuM99/XRZienG4ZCdEEIh+moNPLtRXgID+ElBWvX47u4B9cXrPEot+ZmEt76epMe6fFQ1gpH5ByN80+vl35qbTohWguDOLK3trW1XnjK+eNbNSoXAReEKsO6scleLwux1VZJJEptFa3J504kSPzOnqXfGfaY2XM+dYo2O7kydHycXoMnB1VqnlVuUtBf4h/wVfxlwZrHUyjOPl9h5kpTE4k4Z7dw6qHPR8/x+Uj0eR+Ah0xwGqPHU76NQfHv/L1ttahcBFwQqpRiW6VcNJ7LkYgnEvT4ZNJshkYiJMo2G1knXKHJUThntWQywDPPkNhbhdzhMN56ceYKUy6X/JO4FX+DT8A6hNnlMn43/+S0Qp7f2dZmZnp6PPQ4awpiLdUY5QdIs73CB1d6MlaLpVz+/VKi8iP9ERzqDaFnbxCX7SodZbdRiIALQhVTToTKCXk6bYqAcjkSYa5+jERMhD4wUNhIKp021Z7JJPDss7ThyUVCoRCJ78yMGdxcbhBzG4bxGC5BG0bza9/Au/Bu/J+CMn2HgzY6PR7yvTmXnEe5eb1krzQ2GvuF0xVra40dw6+VydDn4SZa/JNTEYGVR+VH+iP4s28eQiqTg8thw93v6qkaEZc8cEGoYqxl5db0QxYf/t3pJGHjAiCAxMrppMh2dpbEeNcuEvPBQbMhmMmYqfUeD7B/P/D00xSFu90k8l4vCerkJPnrU1OFg5hH5lrY1mMSD+MaHMTjuBHfxI34Jv4Fr8Qb8M9IwY1MhhpxWdvQBoMk4HV1xt93u03mSjxuslkaG40NxPYKQJ+JS/VZwFnQbbbCv1g46rZG5XyRLBeVH+oNIZXJIaeBdCaHQ72hqhHwSoiAC0IVYe0dYi0CsmZssK/sdpOlwjnkuRytdXSQOE5NAd3ddD8LeSZDz3E6zdDjeJyslbExU97vdhe2s+VsFhbDKTTgEhxFDRL4Aa7DtfgpXoUHkUQNHsUVeDEeys/0HB42PV/cbtNu1+83OeEjIyT4wSBdiCIROjfuU+520+Hzme8nFjNDJ6yRucNRmI7IWFv/cuMwa1TeszcIl8OGdCYHp8OGnr2lVavVhlgoglClzLfJCZhoM5Mx1gjng/PknKkpk6XCKYROp/HJWfg4xbCvjyJ47s2STpPYTUyQwE9OmqlDVuzI4Ha8A2/Bnfm1fnThCvwW42iB220iZx584fMZa4VHvjkcFH03NlLvFc4db2mh371euu3303uwEPNfIVYh501RhoW/OJvF2j3ydwPigVdEBFwQlk6xkFu9cWu2SipFoptMmtucnjc5aQZMcPMrtk20Nul87IcfP06PC4VM+1ul6PboKN3H7XEL0fgcPogP4PP5lSRcOIBn0Ic9eSHnClS/3/jl3DSLNyq5D3lTEwk4++JNTWZj1Ost3NTkv1ysAye4sZYVq5CzBFpbAFdbBosIuCBsYsoVs5Tb5MxmC+dyAkbIUykSYBbdsTFjp3CE39JCtyMRsjVOnCCx5kES6bRppjUxYQqEym14vhdfxBfx/oK1i/E4nsTFqKuj57E1wgLe2GjG0yllslI4KuduiOyd8+8uFz3OunGrdWG5vsNhOiYyi4nKqyGvXARcELYAxdF4pe592azpasgFQGybJJMkwNzKNhw23RB5Gn1jI90fiZB/3ttrhJy9ZI7sWcg5zbGYN+NO3Im3FKw9H4/gF3h+vkGXx0OfhfPHAwG63dhIj+epRRyVt7TMjYGbs1esrW+9XpOdwqLM4s1FRpstKl+2gCulbgfwSgBjWusL59Y+DuBGAONzD/uw1vpHC52ECLggrJxyFYmVbBXetGR/nP+58yagVZAnJkjYuMSfc7rTabpvYICOSMRMBNLapDBydWcsVpi5wrwC/4p/xSsL1l6LH+B+vDZf9MPC29hoeqmwR86RtstFlg+X6gcCply/vt7kkXs8pqLTWrVpzSfn9yyOsvm7qpaofCUC/ocAZgDcUSTgM1rrLyzlJETABWH1sEbj1oiRsUbjbKmw1cGWSS5HRT2xmBH8eJxEmyN89qPTabJdzpyh8nzuzcL++OysKQgKh+l1ywl5D36NX+N5BWvvxDdxO96Zj5BZWP1+ev/aWhLkmhpTlcl/KXi9ZKfwRqc199yaZuhw0Gfk74qHM1vfszgqZxEvHkTBwzbWKypfkYWilNoN4EERcEGoLhaKxoHK/rhSJNhc0clzO1nIp6cL0+64UjKZpNTA4WHKWolG6eCInDNaJieNoJcT8vPwNJ7GBQVrH8an8Bl8ON+dkDNJamvpqKkxGSsNDfTZuTe612sKgzgqtxYR8cXIOh2Ivx8Weo7K7fbSKJuF3Pp9c0S+1lH5Wgj42wBMAzgM4H1a67JFuEqpmwDcBABdXV2X9ff3L+8TCIJQkeKBEda+2kCprTI7a/pzs4CzAHHmCQ+P4FFvfHHw+0kEk0myVM6eJZ88FjNVogCJNkfp7JXH4ybFkdmJATyFC+GHmQn3Ffx3vAdfhs1ug4/SyeH10nM5Guec8Pp6k3kSCNDjgkHjlXMJf10dHUoV2idWe4Vfh6N8u700yt6IqHy1BbwVwAQADeATANq11u9Y6HUkAheEtaXYVimOCq3ReDpNIszdDq1RZCpFvjbbL9ksibN1U5DHqc3OUuvas2fpmJmh53PaIkfkMzNGyK19zJkgJvArPA/n4ER+7R68CW/GXcjZnPD5zEZnJmPmcHKuuM9nNie5F4vPR2Ju7YbI4s8ROfvibK+wCHP/Fqu9slFR+aoK+GLvK0YEXBDWnuLIcCFbhS0Vzgm3lqOz+FqFPBo1IuVymWh1dpa6Ho6Oms6HvFmqNb1+JGK6H7KQc4ojQK/r0TH8BNfi+fhF/nwfxgvwCvwISZsXtbX0vnV1RsiVIsFWis6HUwrdbmOvNDTQpmd9Pf0V4fWai5DVd+e9AaWMaHMfFxb2clF2Nrt2Ufmq9kJRSrVrrYfnbl4HFI3yEARhw7D2VQGMIFsjct6gtNuBY2MR/OpUCJftCOKCtkBeTLlQpqaGRJh9bhbc2VnTrpZ96ec8h0S6pYWaZQ0Pm0g7GiUBTSbNcIfJSfLIp6ZMRWjKUYsXZP8TDp3Cd/Bf8Abch2vwH4ijFsdyB3B19JeYcjTm+4dbhVdrEmfuq8J2kc1G7x0Ok4BzGqLXS4/nromcbsjCnU6bXjSzs2ZyEJfrW6NyPhduf1AcnVuHVKwWi8lC+S6AawA0ARgF8LG52wdBFkofgD+3CHpFJAIXhPWl3GgyazT4u4EIbrjddOC74209eRFnW4WfpzUJLtsunGvOQprLFXYQjEQoEh8ZITGPRuk1YjEzRWh6mm5PT1OZfzRKIhmPWypFczn8A/47bsZt+fOOoAGX2J/EQG5HPoqurze2iFImunY4zH0cqVt7l+/YYdIQ2S/nzof809rZkC+INTXGgrGuW+EUTS4qWi5SyCMI25hKm5xf+4+T+NLPnkVOA3YFvPdl5+K/vqA7PzwimTTzN6250Vz8w9klMzMmUlWKottslp4XDlPa4dAQiXkyadIVrR757KzJJ5+eptdNJo2Q53Iat+JT+CT+puCzXOQ4huO2A3C5SFR9PhMNW8vteZOTLRC3m7xx3uBsbSURb2gofB23m97H6pMDJurnqNy66bnaGSnSTlYQtjFWzxswkfVz95R24GNvm60JjrgBk0vNXQI5g4Uj1ESCxCsSMf1O6uuBSy+l5lR9fRRpDw/Te3DaosdDz62poedMTdFrTE6az+DxKHwu/RF8JvcRvCv3f/F/8W4AwJOZ8wAAz9eH8Nv0c/P2CeeEJxJ0Maqpoc/BZffZLGXPjI6STz45aeZ5BoN0UeINT5fL7A+wQFsHZ3ADMZ4Nyo8pF5WvJhKBC8I2o3ij7Uh/BI/2hfDcPUFc2hUoKBtnP5ejcY6w2W9mS4TTB1mQYzFT1cm52zyJfniYrJXhYSoM0trkp3PqIkfk09OmKAigtZoaM/D5dbgP9+ENBZ/v1c4f46fqWihlNio5h1xr+uly0We0TgziplrcKbG5mbxyq71SW2uyVgBjr/BfNFxJypu71myUlWxkioUiCEIBxbZKMVYrIJczueFsqxTnlsdiprEVF/XE48Yf5+wRbsR15gxFwENDJNI8YWh21vzOlZ7RqInIOeLlzoa5HHB15mE8jBcWnP/bnXfhbvxZXqi5oIezV5xO+j2XKxTpbNakIfIGZ2cnnT8Pbfb7zQBmLiZie0kpY69w73YuFFouIuCCIJRQvMlZDqsNwPnjfHDeNPvfPKPT6p/zwAmO2Bsa6LX4fQcGTMZKOEyP5Yk97MVPT9P7zcxQ+mE0alIbeSiF1sB5qaM4iksKzv+Dri/jq46/BmBSDHnTk+XPKrJcccopiZxP7vWSkPt85jZfFFjA2WJhy8k6KSi4gvkQIuCCIFSkOBovrpYsZ6twJSdvbvJzuMFVNGoidrZh7HbT3raxke6z20mo+/rMRufUlOmvwpWg/JqpFP2cnDQROW928kVkZ+oUfp89Fw6YD/ZF5y34mPPTgFL5BlncLIvTAb1eE1mz0HPPlMZGU9Lf2WnK9VnM2f/mvzjcbvNXi8tFrXCXiwi4IAgLYhVy9rrLdeQDjK3C4mqdjsObfImEmRaUShlbxeUyNkggQBE2e9vWjJXpaRJFru5Mp42ox+MUpXOJPl9Q3G7Tk6U5O4JDmcvRibP5z3CX4224ueabyCl7fvBDU5MRaqVMcQ+nI/p8xs9nK6WurrDKky0YznrhYcy8IdzcvPz/LiLggiAsinJtVIttlnK2CtsdQGEaHZfhW4UcMLMueehxUxNF3pwH3t9PGSKjo2Z9etpE/eyXJ5N0P/ckt871zOXmMlCSU/hp9sW4TB/Jn9dP7S/HDd7vI5atybeqDQRIjLn4iIWY+6NwH5hMhkScUw4bG2kWKeeRc0Ru7XTY1rb8/yYi4IIgLIlyZeGLsVW4yMf6ON54jEYLs03YWuGBy5x/zRPrJydpo3NsjDxyLgKamTH55MmkqQrl9MOZGbOZao3IndlZ3JN5Ha7N/Th/fr9Tl+L1gZ9jPOnPCy73TPF4jLfNEbk1tTCXo0icRdvvp8Ig9se5Ha7XC+zevfz/FiLggiAsmXLRODC/rZLNmmicszK4HwsL+cxM6SBmviCwKNbUmFzsiQnKWhkfp4icN0itG528ccqFQdxAiwdacNXk7CyQS2fxv9M34m36H/Of4yw68EfBIzibbcv3P7HO6mRriPPk2SfnqUKc6cL55zt3ktCz5XJ5ifwuHhFwQRCWTXE0XtxJkNcYFnGOsFn8OP2QM0h4U5IzVTh7gx/b0ECPTyRISIeHScTHxkjIi0v6edOUxdzaj9ya3sibqTqn8ZHErbhFf8acO2zoCRzHidw+ACTGLS2mLzl/FpfLVGLW15NY818TPp/xxNvbadPzZS9b/vcvAi4IwopZaJOznK3CG50cZVunB6XTZpYmWyKAEWyO7Hl4A2etDA2RiLO1wq/LQyi433kySQcXBLGQc+46V5rmcsC7Z/8nvpB9T8HnfWHDYziKg/mIvL6eBN3rNefG037YXmloME2t/H4S/aYm4D3vWX6JvQi4IAirQjlbxdojhNfK2SocMbPgsmWSyRi7g7NNOPukpsZYMY2N9Bi+3d9Pm5cckXNK4vS0yXixTiKaniaPPJk093FRTzJJ5/u6xN34x/SbCz7za/w/x69c1+Q3NjmdkPuKAyYVke0VzifPZKjPyt///fK/cxFwQRBWlXIWymJtFWpOZewUFsHZWZMeyIVC7JuzGHIv8ETC3D8wQBueIyPkl7NFYhVyzk+Px00eeSJhXodfmytBr5n9CX6YfHnB53lr7T/jp3Wvz+d2e70mKueCHf7MDocR+/Z24EtfWv53LQIuCMKqs5hNzkq2Clsi1h7lnKLHlZhsgfBrJpOmr4rTaXp/AyTEw8MUYY+MUGTudJryfp4SBJgURBZyvs2FQHa7ObeL4r/Bw7M9BZ/7fbX/B9/1/zlSKXoPHhjB0TdQWNBja4ngle8IoWdvEJftCiz5exYBFwRhzSi3yVkuDbGcrcI/2Vbh9rVctBOLGQHObz7O9dfOZk15PI9wi8XII5+YMMVAPCwiHjd9WwATkcdiJPxstbBvz8MrbDagK3YMh+PnF3zuz3j+Dv+r/iNIZ1Q+IucUQ05fTPkjGD3/EJSdeq7f/a6eJYt4JQFfg/GbgiBsN4oHGhRPAwKMaPPjuQrS6TRDGABTwcjzLrmpFI9Js9nITuGCoFTKRNs1NfSc/fuB888HLrgAOHDAlLk3NpoCHK6Y5En2XV2Uw93QQK/h9dLrc1OqQd956LgojIM9jyBiawAAfCjxUQyN2PDZmZuBXA5TU3TRGB2lC0ImAyTrQ9Aqh5wG0pkcDvWGVu17l37ggiCsGpx9wdG31d9mW4U3Ma3zJflx7I9be25zcY3HYzJWEglzAUgk6HU4qq6rM38B+P0kpHV1FImPjNB6Y6OpBOUURs4u4eZYoRC9NleZZgMROF92CBFbDgev/g5cD5yLB3tfiXOzx/DO2dvwzpHb8IDr9bi58btIJp15r93rDEJ12GBzmJ7rq4UIuCAIqwpHrBxtWwcEs7BbxZ3br1oHAHMJvPWCwHM3eZQZ2x+8cZhIkAjHYnS7tpaex0MlJidJ0KemyCvn3uDsf/NGp9drjtlZI+TJHSHkbDkoG6CRw+yeHF448zTsyTi+G/1j/EH6Ybw6dR9ePeLCIccf4M1NP8FsuhaTJwNoQA/+4uPL98ArIQIuCMKaUCka50gbMFE5R+LW+7mCk+0Ufg4PZ4jHTfofi7fWpqFVNGpGqmltSuQnJkjQQyEqCuILiNNphi/zhSKZNFN9xmaDmNZ0dVE5GzAWpFz4Wi+u9/wc2dk0vjZzA16bvAc9mV/g5EgdTtn34zXNvwZCQfy3F66ecDOyiSkIwppTbpOzOIOlXLYKj2vjzU67nSwNLpzhKJxtFe6nwuX13NCKo3bOfonHqbBnYoIslvFxUzXKzbK4b4vLRY+Px4GIPYKQPYTMUBDZ0UD+LwW+CGQyQC6Tw0cn34t3J7+S/2xnHbvQmTq97EoeyUIRBGFDKZdyyJWcC2WrsJBbS+05F5sbVXG6IHvh3CBLazPL0+2miJpFembGCHk4bIY1Fws5j4PjC0Y8TsLPvVi4fJ/7pHB73Zujn8GHZz5Mn//4Caj93cv67kTABUGoCspF48DCRUB8aF0o5FyAk82WCjl3EeTSfI78PR5TtMMl+JOTJiKPROi1ucdKJmMGLDscJvUwmTTdD1n42dpxOMwGaXN9CkeedC17yLFMpRcEoSpgm4QFm/3uYlulXLaK1R9na4VzxrnZlNtNgs0ZKrOzhf643W7yynmgsc1mOgc2NhohB8yGKkfWk5OmwjIWM+mJ4TDd5mEVfNFwu7GygZjzIAIuCMKGUGmTczHZKhyN22wk4izu6TQ9n/uQsJhzgY7HU2jJxOP0mjypvqWFxNjno1ayY2OUtcKl/dZJO5OT9Hp1dRSB80ZrOEy33W4j5F7vyqbSV2JBAVdK3Q7glQDGtNYXzq01ArgHwG4AfQDepLWOrP7pCYKwlSmXcgiU2irzZaswLN42m9nA5PzxWMxUYvKmJvcr5wZb1sKipiZT9DM9TWmHU1N0IWABdzrpuZGIEXKOwFMpI+Sc0ZLLYdkWSiUWE4H/E4CvArjDsnYLgIe01p9VSt0yd/uDq3tqgiBsF8rZJxyNL9ZWcTrpMdyfhC0WFnK32+SPz8wYf5wfx69v7WfS3GzmX05NUYVlNGrskWzW/IxEzICHWMykIbLlstriDSxCwLXWjyildhctvwbANXO/fxvAwxABFwRhBRQX+1ij7sXaKrmcyUzhlrQszpw/PjND61y8w1YLV4FaR6jxUVNDRUA+nxHyWKywPwtH5lNTpi84D2v2+80giNVkuR54q9Z6GAC01sNKqZZVPCdBELYxlTY5i9fL2SpWf5s9aH5uMmmm53D/cbfbZJN4veZCwSmEnBbodJro2uejg0vz+SLBeei82cniXV9Pz10L1nwTUyl1E4CbAKCrq2ut304QhC3CfJucC9kqLOJc8MMRMhf/sLCyVz47S4LO/nguZ3LQ+b2dTnptv5+iay7ND4XI73Y6zfvU1JhqUbZb1oLlvuyoUqp9LvpuBzBW6YFa668D+DpAeeDLfD9BELYhlfqqLCZbhR+by5G48tAGLpnn6Trc+pXTDjl/nKfOcx9yjq7ZH6+vJwH3emnTc2yMxJytFO6amEhQ5L4WLFfAHwDwVgCfnft5/6qdkSAIQhHz9VUBFm+rsChzOT73Ifd4zCanNX/cWvzDE32s0bjdTumGXOUZDJqqTs5GsdkoYl9t/xtYXBrhd0Eblk1KqUEAHwMJ971KqXcCGADwxtU/NUEQBMNCm5zlslj4OVZbhbsQsj9u3YSsrTVpfxyNc9tZnnlpndnJTbAAyljhronNzSb1kCP8tWAxWSh/WuGuF6/yuQiCICwID1GuJNbFAl/JVuFMk2Sy1B/ntEMut4/FTJTNgs0NtrjXCpfzNzcbeyaRIGuluE3AaiGVmIIgbDrmi8aXYqvwaDYus+cGWJzFwimEnH7ITapcLtMgixtZcZTORUJtbST+tbVrU4UJiIALgrCJmS/lcCm2Cmef8O3ijBJOJeRonO0X7qXCHQm5h7nTWTj+bS2KeAARcEEQtgCVNjkXa6twNM4bmlygA5Awc/MqLvrhboTceZArOrNZejy3unU6C/8qWG1EwAVB2BLM11dlsbYKYDxtm40ibs5c4TTCujpjq3BEzkJenHqYTpsIvpoqMQVBEKqShayTpdgqbrfxuTl/nAXe2vFwZobEOh43HQv5sbGY2TxdbUTABUHYcsy3yblYW4V7qORyZKuk06a6k/PIXS6zcRmLGSHnTU3ONecLh0TggiAIi2S+Tc6FbBXOSAFMgytOK2R/3Jp9wsMbEgmTsQKQ5cKbmquNCLggCFuehTY5K9kqnCrItgq3mwUKbRXOXPF6CwdJsJhLHrggCMIKmK+vylJtlWyWhFprU3LPAyXsdpOx4vFQ/3B+XbFQBEEQVsB80fhSbRWtKermnidcas+phTYbReOyiSkIgrBKzLfJyT8Xa6vwxiY3wbJ2PnS5zPPWgjVKLxcEQah+uGKT4dQ/wIg8iy974Hyfw2F6qHAFJvvfWptBD9Zxb6t+/mvzsoIgCJuHSkINlBd5q0fOUTbbL9yRkIc4OBwmI2W1EQEXBEHA/EINFFZ0soXC0Tq3lWUhZ8+7ttZE6GuBeOCCIAgW5tvkXKhAqDhbhScArVUzK4nABUEQimChZlZqq6xVHrgIuCAIQgWKOwlabZPi++ezVSQLRRAEYQOYLxtlofut2SprgQi4IAjCIlhok3O++yUCFwRBqALms00Wc/9qIgIuCIKwRMptclqj8XK2ylogaYSCIAjLZL6UQ6AwEl8LJAIXBEFYAQttcvJj1gIRcEEQhFVgoU3ONXnPtX15QRCE7YU1GmdbZa1YkQeulOoDEAWQBZDRWl++GiclCIKwmSk3ym0tyulXYxPzhVrriVV4HUEQhC0Fb3Ku1SamZKEIgiCsIWvZjXClHrgG8G9KqSNKqZtW44QEQRCExbHSCPxqrfWQUqoFwM+UUs9orR+xPmBO2G8CgK6urhW+nSAIgsCsKALXWg/N/RwD8AMAV5Z5zNe11pdrrS9vbm5eydsJgiAIFpYt4EqpWqWUj38H8DIAT63WiQmCIAjzsxILpRXADxS58w4A39Fa/2RVzkoQBEFYkGULuNa6F8BzVvFcBEEQhCUglZiCIAibFBFwQRCETYrSa1UiVO7NlBoH0L9ub1hKE4BqrxqVc1wd5BxXTrWfH7B9znGX1rokjW9dBXyjUUodrvZ+LXKOq4Oc48qp9vMD5BzFQhEEQdikiIALgiBsUrabgH99o09gEcg5rg5yjiun2s8P2ObnuK08cEEQhK3EdovABUEQtgwi4IIgCJuULS3gSqk+pdSTSqmjSqnDc2uNSqmfKaVOzP0MrPM53a6UGlNKPWVZq3hOSqkPKaVOKqWeVUr90Qad38eVUmfnvsejSqlXbNT5zb3nTqXUz5VSx5RSv1dK/dXcejV9j5XOsSq+S6VUjVLqUaXU43Pn97dz69X0HVY6x6r4DovO1a6Uekwp9eDc7fX5HrXWW/YA0AegqWjt7wHcMvf7LQA+t87n9IcALgXw1ELnBOB8AI8DcAPYA+AUAPsGnN/HAby/zGPX/fzm3rcdwKVzv/sAHJ87l2r6HiudY1V8lwAUgLq5350AfgOgp8q+w0rnWBXfYdF7vxfAdwA8OHd7Xb7HLR2BV+A1AL499/u3Abx2Pd9c08CL8CLP6TUAvqe1TmqtTwM4iTI919fh/Cqx7ucHAFrrYa317+Z+jwI4BqAT1fU9VjrHSqzrOWpiZu6mc+7QqK7vsNI5VmJD/n9USu0A8McAvll0Lmv+PW51AS838q1Vaz0M0D8yAC0bdnaGSufUCeCM5XGDmF8E1pKblVJPzFks/Ofghp+fUmo3gEtA0VlVfo9F5whUyXc592f/UQBjAH6mta6677DCOQJV8h3O8T8BfABAzrK2Lt/jVhfwq7XWlwJ4OYD/ppT6w40+oSVSbhTqRuR9fg3APgAHAQwD+OLc+oaen1KqDsB9AP5aaz0930PLrK3LeZY5x6r5LrXWWa31QQA7AFyplLpwnodvyHdY4Ryr5jtUSr0SwJjW+shin1JmbdnnuKUFXJcf+TaqlGoHgLmfYxt3hnkqndMggJ2Wx+0AMLTO5wat9ejcP6QcgG/A/Mm3YeenlHKChPFurfX355ar6nssd47V+F1qrScBPAzgWlTZd1juHKvsO7wawKuVUn0AvgfgRUqpu7BO3+OWFXBVeeTbAwDeOvewtwK4f2POsIBK5/QAgOuVUm6l1B4A+wE8ut4nx/8jznEdzOi8DTk/pZQC8C0Ax7TWX7LcVTXfY6VzrJbvUinVrJRqmPvdA+AlAJ5BdX2HZc+xWr5DANBaf0hrvUNrvRvA9QD+XWv9ZqzX97geO7QbcQDYC9rtfRzA7wHcOrceBPAQgBNzPxvX+by+C/qzLw26Gr9zvnMCcCtop/pZAC/foPO7E8CTAJ6Y+x+wfaPOb+49/wD0Z+cTAI7OHa+osu+x0jlWxXcJ4GIAj82dx1MAPjq3Xk3fYaVzrIrvsMz5XgOThbIu36OU0guCIGxStqyFIgiCsNURARcEQdikiIALgiBsUkTABUEQNiki4IIgCJsUEXBBEIRNigi4IAjCJuX/B3jcUMt7T7/+AAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "for m, b in res.bootstrap_distribution.T[::10]:\n", " plt.plot(x, m*x + b, color='b', alpha=0.01)\n", "plt.plot(x, y, '.', label='original data')\n", "plt.plot(x, res_lr.intercept + res_lr.slope*x, 'r', label='fitted line')" ] }, { "cell_type": "markdown", "id": "a38780c7-38c8-44fe-93be-7e5347e3aea4", "metadata": {}, "source": [ "A major advantage of the bootstrap is that it can produce standard errors and confidence intervals even in more general regression models that have no simple analytical solutions, such as when the regression function is nonlinear in the parameters and when using fitting methods other than least squares." ] }, { "cell_type": "markdown", "id": "1a886e39-f2b5-4ea4-8dff-110b7ddb5b45", "metadata": {}, "source": [ "### Gotchas\n", "\n", "Our final example will show yet another application of the `bootstrap` chosen to illustrate common pitfalls.\n", "\n", "[An Introduction to the Bootstrap](https://books.google.com/books?id=MWC1DwAAQBAJ&printsec=frontcover) presents a study about whether regular doses of aspirin can prevent heart attacks. Subjects were randomly assigned to two groups: 11,037 received aspirin pills, and the remaining 11,034 received placebos. The subjects were instructed to take one pill every other day, and the scientists recorded the number of subjects who experienced a heart attack during the study period: 104 in the aspirin group, and 189 in the placebo group. The statistic to assess the effectiveness of aspirin was the relative prevalence of heart attacks in the aspirin group versus the placebo group." ] }, { "cell_type": "code", "execution_count": 16, "id": "f4153813-8a78-43c6-bd28-2cc6f89d7a33", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0.5501149812103875" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "x = np.zeros(11037) # 11037 subjects in the aspirin group\n", "x[:104] = 1 # 104 experience a heart attack\n", "y = np.zeros(11034) # 11034 subjects in the placebo group\n", "y[:189] = 1 # 189 experience a heart attack\n", "def statistic(x, y):\n", " return (np.sum(x)/len(x))/(np.sum(y)/len(y))\n", "statistic(x, y)" ] }, { "cell_type": "markdown", "id": "d1b65fcb-6f4b-480c-a163-085e272e3249", "metadata": {}, "source": [ "The risk of heart atttack for aspirin-takers seemed to be approximately half that of placebo-takers.\n", "\n", "Suppose we wish to generate a 95% confidence interval to quantify our uncertainty." ] }, { "cell_type": "code", "execution_count": 17, "id": "2b1bcd63-2a11-4643-b036-bce7121afb4f", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "TypeError: bootstrap() takes 2 positional arguments but 3 positional arguments (and 1 keyword-only argument) were given\n" ] } ], "source": [ "try:\n", " stats.bootstrap(x, y, statistic, confidence_level=0.95)\n", "except Exception as e:\n", " print(f\"{type(e).__name__}: {e}\")" ] }, { "cell_type": "markdown", "id": "f0b5caf3-3434-41a2-b8ea-da8d1ccfe679", "metadata": {}, "source": [ "This reminds us that all the data needs to be passed in as a single sequence, not two separate arguments `x` and `y`." ] }, { "cell_type": "code", "execution_count": 18, "id": "8d2738da-e79f-4ed3-968d-6533dadb7769", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "ValueError: `method = 'BCa' is only available for one-sample statistics\n" ] } ], "source": [ "data = (x, y) \n", "try:\n", " stats.bootstrap(data, statistic, confidence_level=0.95)\n", "except Exception as e:\n", " print(f\"{type(e).__name__}: {e}\")" ] }, { "cell_type": "markdown", "id": "aebd581b-d7df-4179-a6f1-a8d4feab9043", "metadata": {}, "source": [ "`bootstrap` offers a `method` argument that selects how the confidence interval is to be estimated from the `bootstrap` distribution; the three methods `{'BCa', 'percentile', 'basic'}` vary in their performance and accuracy. `BCa` is the most computationally intensive but tends to be the most accurate, so it is the default. However, it is currently only available when our data has only one independent sample, whereas our data consists of two independent samples `x` and `y`. Let's try another option, `percentile`." ] }, { "cell_type": "code", "execution_count": 19, "id": "33a84978-c534-4069-8f7e-3171614e0e20", "metadata": {}, "outputs": [], "source": [ "try:\n", " stats.bootstrap(data, statistic, method='basic', confidence_level=0.95)\n", "except Exception as e:\n", " print(f\"{type(e).__name__}: {e}\")" ] }, { "cell_type": "markdown", "id": "47a2f909-b392-4738-ade4-4a80e07f7071", "metadata": {}, "source": [ "Unlike `permutation_test`, `bootstrap` expects `statistic` to be vectorized by default. We can solve this by passing `vectorized=False`." ] }, { "cell_type": "code", "execution_count": 20, "id": "15046441-1e6a-45c8-bdf6-2efa274f127c", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "ConfidenceInterval(low=0.4324405646124769, high=0.6926221426325354)" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "res = stats.bootstrap(data, statistic, method='percentile', confidence_level=0.95, vectorized=False)\n", "res.confidence_interval" ] }, { "cell_type": "markdown", "id": "09cc4689-4037-4ddb-a457-449015ed5256", "metadata": {}, "source": [ "Alternatively, we can vectorize our statistic by making it accept a parameter `axis` and having it work along the specified axis-slice of N-dimensional arrays `x` and `y`." ] }, { "cell_type": "code", "execution_count": 21, "id": "7588b203-8de2-4b52-9ced-64ad374c82a7", "metadata": {}, "outputs": [], "source": [ "def statistic(x, y, axis=0):\n", " return (np.sum(x, axis=axis)/x.shape[axis])/(np.sum(y, axis=axis)/y.shape[axis])\n", "\n", "try:\n", " res = stats.bootstrap(data, statistic, method='percentile', confidence_level=0.95)\n", " res.confidence_interval\n", "except Exception as e:\n", " print(f\"{type(e).__name__}: {e}\")" ] }, { "cell_type": "markdown", "id": "7800e793-92cb-4871-815b-06e3da89bb18", "metadata": {}, "source": [ "Depending on your computer's hardware, you may run into a MemoryError there. Vectorized computations require a lot of memory. The default value of `n_resamples` is $9,999$, and there are a total of $11,037 + 11,034 = 22,071$ observations. Therefore, the resampled data arrays will contain a total of $9,999 \\cdot 22071 = 220,687,929$ elements. Each element is stored in double precision (8-bytes), so at least 1.7GB will be used during the calculation. To relax the memory requirement, we'll process the data in batches of 100 resamples rather than all at once." ] }, { "cell_type": "code", "execution_count": 22, "id": "f6769306-af22-42b3-850b-377e40e3b703", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "ConfidenceInterval(low=0.4309040629913994, high=0.6944371938603752)" ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "res = stats.bootstrap(data, statistic, method='percentile', confidence_level=0.95, batch=100)\n", "res.confidence_interval" ] }, { "cell_type": "markdown", "id": "19a3d153-fb2f-4f32-aa10-a347799c33e9", "metadata": { "tags": [] }, "source": [ "## Conclusion\n", "\n", "The resampling approaches in SciPy can be used not only to replicate the results of most of SciPy's hypothesis tests, but to\n", "\n", "- improve the accuracy of statistical tests for small sample sizes and in the presence of ties,\n", "- provide standard errors and confidence intervals for arbitrary statistics, and\n", "- easily implement statistical tests that SciPy does not yet offer." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.5" } }, "nbformat": 4, "nbformat_minor": 5 }