lectures/24_leftovers-conclusion.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CPSC 330 Lecture 20"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Outline:\n",
    "\n",
    "- Announcements\n",
    "- Gradient boosted trees, `Catboost` (15 min)\n",
    "- Model comparison (15 min)\n",
    "- Break (5 min)\n",
    "- Big data sets: `SGDClassifier` and `SGDRegressor` (10 min)\n",
    "- Multi-class classification (15 min)\n",
    "- Combining multiple tables (10 min)\n",
    "- True/False questions (15 min)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reminder to self: **turn on recording!**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.linear_model import Ridge, LogisticRegression, SGDClassifier, SGDRegressor\n",
    "from sklearn.ensemble import RandomForestRegressor, RandomForestClassifier\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.dummy import DummyClassifier, DummyRegressor\n",
    "import sklearn.datasets\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import confusion_matrix, classification_report\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.feature_selection import RFE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import plot_confusion_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pandas_profiling import ProfileReport"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- These next 3 cells could just be replaced with `import load_process_data`.\n",
    "- But this way I can make changes to that file and not need to restart my kernel each time.\n",
    "- So it's convenient for development."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The autoreload extension is already loaded. To reload it, use:\n",
      "  %reload_ext autoreload\n"
     ]
    }
   ],
   "source": [
    "%load_ext autoreload"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [],
   "source": [
    "%autoreload 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [],
   "source": [
    "%aimport load_process_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [],
   "source": [
    "from plot_classifier import plot_classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.rcParams['font.size'] = 16"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "New dependency: `lightgbm`. Install with\n",
    "\n",
    "`pip install lightgbm`\n",
    "\n",
    "or \n",
    "\n",
    "`conda install -c conda-forge lightgbm`\n",
    "\n",
    "(conda worked better for me in this case)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [],
   "source": [
    "import lightgbm as lgb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "New dependency: `xgboost`. Install with\n",
    "\n",
    "`pip install xgboost`\n",
    "\n",
    "or\n",
    "\n",
    "`conda install -c conda-forge xgboost` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [],
   "source": [
    "import xgboost as xgb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "New dependency: `catboost`. Install with\n",
    "\n",
    "`pip install catboost`\n",
    "\n",
    "or\n",
    "\n",
    "`conda install -c conda-forge catboost` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [],
   "source": [
    "from catboost import CatBoostClassifier, CatBoostRegressor, Pool"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [],
   "source": [
    "def show_scores_factory(X_train, y_train, X_test, y_test):\n",
    "    def show_scores(model, **fit_kwargs):\n",
    "        model.fit(X_train, y_train, **fit_kwargs);\n",
    "        return model.score(X_test, y_test)\n",
    "    return show_scores"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Announcements\n",
    "\n",
    "Same as last class:\n",
    "\n",
    "- hw7 has been posted, due Sunday evening.\n",
    "  - I tried to make it shorter than previous assignments.\n",
    "- Tutorials are still happening as scheduled, on Collaborate Ultra.\n",
    "- Change to course grading scheme per Dean's directive; see [here](https://piazza.com/class/k1gx4b3djbv3ph?cid=319)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Today's class:\n",
    "\n",
    "- A bunch of random things I wanted to cover at some point.\n",
    "- Next week: communication and ethics (hopefully)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gradient boosted trees (15 min)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recently a lot of the winning models have been one of the following:\n",
    "\n",
    "|  Name   |  GitHub |  web/docs | Year | GitHub stars |\n",
    "|---------|---------|---------|---------|--------------|\n",
    "|  XGBoost | [link](https://github.com/dmlc/xgboost) | [link](https://xgboost.ai/) | 2016 | 19k\n",
    "| LightGBM |  [link](https://github.com/microsoft/LightGBM) | [link](https://lightgbm.readthedocs.io/en/latest/) |  2017 | 11k |\n",
    "| CatBoost | [link](https://github.com/catboost/catboost) | [link](https://catboost.ai/) | 2017 | 5k |\n",
    "\n",
    "When I checked, all repos updated in the last 24 hrs (i.e., active development)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- All of these implement [gradient boosting](https://en.wikipedia.org/wiki/Gradient_boosting), which is beyond the scope of the course.\n",
    "  - And not covered in 340, but it probably should be!\n",
    "- These 3 packages are fairly similar. Today I'll focus on CatBoost.\n",
    "- I believe CatBoost and LightGBM seems to be performing better than XGBoost these days, but they are similar."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- I avoided these earlier in the course because I thought they'd be hard to install.\n",
    "- It seems a lot of issues were solved since I tried XGBoost 1-2 years ago - it was easy!\n",
    "- I think next time I might put these a lot earlier and use CatBoost instead of sklearn random forests everywhere, because I think it's a better choice. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Categorical variables\n",
    "\n",
    "- Of these 3, CatBoost is special because of how it handles categorical variables - hence \"Cat\".\n",
    "- Thinking back to decision trees, we talked about splits like `Humidity > 1.2` or `Longitude > 90`. \n",
    "- Because we OHE categorical variables, it will give splits like `Location_Sydney > 0.5`, where that number 0.5 could be anywhere between 0 and 1. \n",
    "- (First off, there is no reason sklearn's decision trees or random forests need to require OHE; they could have easily learned rules like `Location == 'Sydney'`, which is equivalent to `Location_Sydney > 0.5` after OHE. I believe they didn't do this to be consistent with other sklearn estimators. However, I wonder if this makes performance worse, in particular for random forests.)\n",
    "- However, is OHE the best encoding? Do we want 100 columns if there are 100 categories? What about `drop='first'` and `handle_error='ignore'`. It seems like a lot of hassle.\n",
    "- CatBoost uses a more sophisticated encoding from categorical to numeric; see [here](https://catboost.ai/docs/concepts/algorithm-main-stages_cat-to-numberic.html#algorithm-main-stages_cat-to-numberic).\n",
    "  - From the documentation: \"Before each split is selected in the tree (see Choosing the tree structure), categorical features are transformed to numerical. This is done using various statistics on combinations of categorical features and combinations of categorical and numerical features.\"\n",
    "  - This makes me think the conversion is done separately _at each split_.\n",
    "  - So we can't just do this transformation in advance, it is entangled deeply with the algorithm. \n",
    "  - And also this makes sense, since the popularity of each value will change as your split the data.\n",
    "  - Importantly, if there are 100 possible categories, you may not end up with 100 columns.\n",
    "  - (From my reading of the documentation it looks like you end up with some smaller number of columns, depending on the type of _targets_: one per class for classification, and for regression this is controlled by a hyperparameter $k$.)\n",
    "  - (Interestingly, the number of columns does not seem to depend on the number of possible categories of the categorical variable!)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_housing, X_valid_housing, X_test_housing, y_train_housing, y_valid_housing, y_test_housing = load_process_data.load_housing()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>LotArea</th>\n",
       "      <th>OverallQual</th>\n",
       "      <th>OverallCond</th>\n",
       "      <th>YearBuilt</th>\n",
       "      <th>YearRemodAdd</th>\n",
       "      <th>MasVnrArea</th>\n",
       "      <th>BsmtFinSF1</th>\n",
       "      <th>BsmtFinSF2</th>\n",
       "      <th>BsmtUnfSF</th>\n",
       "      <th>...</th>\n",
       "      <th>Functional_Maj2</th>\n",
       "      <th>Functional_Min1</th>\n",
       "      <th>Functional_Min2</th>\n",
       "      <th>Functional_Mod</th>\n",
       "      <th>Functional_Typ</th>\n",
       "      <th>Fence_?</th>\n",
       "      <th>Fence_GdPrv</th>\n",
       "      <th>Fence_GdWo</th>\n",
       "      <th>Fence_MnPrv</th>\n",
       "      <th>Fence_MnWw</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1341</th>\n",
       "      <td>-0.165978</td>\n",
       "      <td>0.338699</td>\n",
       "      <td>-0.062802</td>\n",
       "      <td>-0.505587</td>\n",
       "      <td>1.048406</td>\n",
       "      <td>0.927764</td>\n",
       "      <td>-0.577947</td>\n",
       "      <td>0.791412</td>\n",
       "      <td>-0.284437</td>\n",
       "      <td>-0.622612</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>459</th>\n",
       "      <td>-0.026237</td>\n",
       "      <td>-0.354633</td>\n",
       "      <td>-0.764983</td>\n",
       "      <td>-1.415471</td>\n",
       "      <td>-0.731683</td>\n",
       "      <td>-1.679272</td>\n",
       "      <td>0.294327</td>\n",
       "      <td>-0.544487</td>\n",
       "      <td>-0.284437</td>\n",
       "      <td>-0.120404</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>367</th>\n",
       "      <td>1.464330</td>\n",
       "      <td>-0.133037</td>\n",
       "      <td>-0.062802</td>\n",
       "      <td>-0.505587</td>\n",
       "      <td>-0.328644</td>\n",
       "      <td>-1.099931</td>\n",
       "      <td>1.074497</td>\n",
       "      <td>-0.149451</td>\n",
       "      <td>-0.284437</td>\n",
       "      <td>0.336963</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>894</th>\n",
       "      <td>-0.259139</td>\n",
       "      <td>-0.354322</td>\n",
       "      <td>-0.764983</td>\n",
       "      <td>-0.505587</td>\n",
       "      <td>0.242328</td>\n",
       "      <td>-0.279197</td>\n",
       "      <td>-0.577947</td>\n",
       "      <td>-0.937398</td>\n",
       "      <td>-0.284437</td>\n",
       "      <td>-1.295212</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>672</th>\n",
       "      <td>-0.026237</td>\n",
       "      <td>0.084927</td>\n",
       "      <td>-0.062802</td>\n",
       "      <td>0.404297</td>\n",
       "      <td>0.175155</td>\n",
       "      <td>-0.375754</td>\n",
       "      <td>-0.577947</td>\n",
       "      <td>0.691592</td>\n",
       "      <td>-0.284437</td>\n",
       "      <td>-0.306490</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 288 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      LotFrontage   LotArea  OverallQual  OverallCond  YearBuilt  \\\n",
       "1341    -0.165978  0.338699    -0.062802    -0.505587   1.048406   \n",
       "459     -0.026237 -0.354633    -0.764983    -1.415471  -0.731683   \n",
       "367      1.464330 -0.133037    -0.062802    -0.505587  -0.328644   \n",
       "894     -0.259139 -0.354322    -0.764983    -0.505587   0.242328   \n",
       "672     -0.026237  0.084927    -0.062802     0.404297   0.175155   \n",
       "\n",
       "      YearRemodAdd  MasVnrArea  BsmtFinSF1  BsmtFinSF2  BsmtUnfSF  ...  \\\n",
       "1341      0.927764   -0.577947    0.791412   -0.284437  -0.622612  ...   \n",
       "459      -1.679272    0.294327   -0.544487   -0.284437  -0.120404  ...   \n",
       "367      -1.099931    1.074497   -0.149451   -0.284437   0.336963  ...   \n",
       "894      -0.279197   -0.577947   -0.937398   -0.284437  -1.295212  ...   \n",
       "672      -0.375754   -0.577947    0.691592   -0.284437  -0.306490  ...   \n",
       "\n",
       "      Functional_Maj2  Functional_Min1  Functional_Min2  Functional_Mod  \\\n",
       "1341              0.0              0.0              0.0             0.0   \n",
       "459               0.0              0.0              0.0             0.0   \n",
       "367               0.0              0.0              0.0             0.0   \n",
       "894               0.0              0.0              0.0             0.0   \n",
       "672               0.0              0.0              0.0             0.0   \n",
       "\n",
       "      Functional_Typ  Fence_?  Fence_GdPrv  Fence_GdWo  Fence_MnPrv  \\\n",
       "1341             1.0      1.0          0.0         0.0          0.0   \n",
       "459              1.0      1.0          0.0         0.0          0.0   \n",
       "367              1.0      1.0          0.0         0.0          0.0   \n",
       "894              1.0      1.0          0.0         0.0          0.0   \n",
       "672              1.0      1.0          0.0         0.0          0.0   \n",
       "\n",
       "      Fence_MnWw  \n",
       "1341         0.0  \n",
       "459          0.0  \n",
       "367          0.0  \n",
       "894          0.0  \n",
       "672          0.0  \n",
       "\n",
       "[5 rows x 288 columns]"
      ]
     },
     "execution_count": 122,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_housing.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train_housing_log = np.log(y_train_housing)\n",
    "y_valid_housing_log = np.log(y_valid_housing)\n",
    "y_test_housing_log  = np.log(y_test_housing)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [],
   "source": [
    "show_scores_housing = show_scores_factory(X_train_housing, y_train_housing_log, X_valid_housing, y_valid_housing_log)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8771971926959883"
      ]
     },
     "execution_count": 125,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "show_scores_housing(Ridge())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8451189779530107"
      ]
     },
     "execution_count": 129,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "show_scores_housing(RandomForestRegressor())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8720378299606011"
      ]
     },
     "execution_count": 126,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "show_scores_housing(xgb.XGBRegressor())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8642147899492177"
      ]
     },
     "execution_count": 127,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "show_scores_housing(lgb.LGBMRegressor())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9051122179357415"
      ]
     },
     "execution_count": 128,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "show_scores_housing(CatBoostRegressor(), verbose=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now we try with proper handling of categorical variables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_housing_cat, X_valid_housing_cat, X_test_housing_cat, y_train_housing, y_valid_housing, y_test_housing, categorical_features = load_process_data.load_housing(ohe=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>LotArea</th>\n",
       "      <th>OverallQual</th>\n",
       "      <th>OverallCond</th>\n",
       "      <th>YearBuilt</th>\n",
       "      <th>YearRemodAdd</th>\n",
       "      <th>MasVnrArea</th>\n",
       "      <th>BsmtFinSF1</th>\n",
       "      <th>BsmtFinSF2</th>\n",
       "      <th>BsmtUnfSF</th>\n",
       "      <th>...</th>\n",
       "      <th>MSSubClass</th>\n",
       "      <th>MoSold</th>\n",
       "      <th>SaleType</th>\n",
       "      <th>RoofMatl</th>\n",
       "      <th>LandContour</th>\n",
       "      <th>BsmtExposure</th>\n",
       "      <th>BsmtFinType1</th>\n",
       "      <th>BsmtFinType2</th>\n",
       "      <th>Functional</th>\n",
       "      <th>Fence</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1341</th>\n",
       "      <td>-0.165978</td>\n",
       "      <td>0.338699</td>\n",
       "      <td>-0.0628021</td>\n",
       "      <td>-0.505587</td>\n",
       "      <td>1.04841</td>\n",
       "      <td>0.927764</td>\n",
       "      <td>-0.577947</td>\n",
       "      <td>0.791412</td>\n",
       "      <td>-0.284437</td>\n",
       "      <td>-0.622612</td>\n",
       "      <td>...</td>\n",
       "      <td>20</td>\n",
       "      <td>7</td>\n",
       "      <td>WD</td>\n",
       "      <td>CompShg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>No</td>\n",
       "      <td>GLQ</td>\n",
       "      <td>Unf</td>\n",
       "      <td>Typ</td>\n",
       "      <td>?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>459</th>\n",
       "      <td>-0.0262374</td>\n",
       "      <td>-0.354633</td>\n",
       "      <td>-0.764983</td>\n",
       "      <td>-1.41547</td>\n",
       "      <td>-0.731683</td>\n",
       "      <td>-1.67927</td>\n",
       "      <td>0.294327</td>\n",
       "      <td>-0.544487</td>\n",
       "      <td>-0.284437</td>\n",
       "      <td>-0.120404</td>\n",
       "      <td>...</td>\n",
       "      <td>50</td>\n",
       "      <td>7</td>\n",
       "      <td>WD</td>\n",
       "      <td>CompShg</td>\n",
       "      <td>Bnk</td>\n",
       "      <td>No</td>\n",
       "      <td>LwQ</td>\n",
       "      <td>Unf</td>\n",
       "      <td>Typ</td>\n",
       "      <td>?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>367</th>\n",
       "      <td>1.46433</td>\n",
       "      <td>-0.133037</td>\n",
       "      <td>-0.0628021</td>\n",
       "      <td>-0.505587</td>\n",
       "      <td>-0.328644</td>\n",
       "      <td>-1.09993</td>\n",
       "      <td>1.0745</td>\n",
       "      <td>-0.149451</td>\n",
       "      <td>-0.284437</td>\n",
       "      <td>0.336963</td>\n",
       "      <td>...</td>\n",
       "      <td>80</td>\n",
       "      <td>12</td>\n",
       "      <td>WD</td>\n",
       "      <td>Tar&amp;Grv</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>Gd</td>\n",
       "      <td>GLQ</td>\n",
       "      <td>Unf</td>\n",
       "      <td>Typ</td>\n",
       "      <td>?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>894</th>\n",
       "      <td>-0.259139</td>\n",
       "      <td>-0.354322</td>\n",
       "      <td>-0.764983</td>\n",
       "      <td>-0.505587</td>\n",
       "      <td>0.242328</td>\n",
       "      <td>-0.279197</td>\n",
       "      <td>-0.577947</td>\n",
       "      <td>-0.937398</td>\n",
       "      <td>-0.284437</td>\n",
       "      <td>-1.29521</td>\n",
       "      <td>...</td>\n",
       "      <td>90</td>\n",
       "      <td>6</td>\n",
       "      <td>WD</td>\n",
       "      <td>CompShg</td>\n",
       "      <td>Bnk</td>\n",
       "      <td>?</td>\n",
       "      <td>?</td>\n",
       "      <td>?</td>\n",
       "      <td>Typ</td>\n",
       "      <td>?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>672</th>\n",
       "      <td>-0.0262374</td>\n",
       "      <td>0.0849273</td>\n",
       "      <td>-0.0628021</td>\n",
       "      <td>0.404297</td>\n",
       "      <td>0.175155</td>\n",
       "      <td>-0.375754</td>\n",
       "      <td>-0.577947</td>\n",
       "      <td>0.691592</td>\n",
       "      <td>-0.284437</td>\n",
       "      <td>-0.30649</td>\n",
       "      <td>...</td>\n",
       "      <td>20</td>\n",
       "      <td>6</td>\n",
       "      <td>WD</td>\n",
       "      <td>CompShg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>No</td>\n",
       "      <td>ALQ</td>\n",
       "      <td>Unf</td>\n",
       "      <td>Typ</td>\n",
       "      <td>?</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 79 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     LotFrontage    LotArea OverallQual OverallCond YearBuilt YearRemodAdd  \\\n",
       "1341   -0.165978   0.338699  -0.0628021   -0.505587   1.04841     0.927764   \n",
       "459   -0.0262374  -0.354633   -0.764983    -1.41547 -0.731683     -1.67927   \n",
       "367      1.46433  -0.133037  -0.0628021   -0.505587 -0.328644     -1.09993   \n",
       "894    -0.259139  -0.354322   -0.764983   -0.505587  0.242328    -0.279197   \n",
       "672   -0.0262374  0.0849273  -0.0628021    0.404297  0.175155    -0.375754   \n",
       "\n",
       "     MasVnrArea BsmtFinSF1 BsmtFinSF2 BsmtUnfSF  ... MSSubClass MoSold  \\\n",
       "1341  -0.577947   0.791412  -0.284437 -0.622612  ...         20      7   \n",
       "459    0.294327  -0.544487  -0.284437 -0.120404  ...         50      7   \n",
       "367      1.0745  -0.149451  -0.284437  0.336963  ...         80     12   \n",
       "894   -0.577947  -0.937398  -0.284437  -1.29521  ...         90      6   \n",
       "672   -0.577947   0.691592  -0.284437  -0.30649  ...         20      6   \n",
       "\n",
       "     SaleType RoofMatl LandContour BsmtExposure BsmtFinType1 BsmtFinType2  \\\n",
       "1341       WD  CompShg         Lvl           No          GLQ          Unf   \n",
       "459        WD  CompShg         Bnk           No          LwQ          Unf   \n",
       "367        WD  Tar&Grv         Lvl           Gd          GLQ          Unf   \n",
       "894        WD  CompShg         Bnk            ?            ?            ?   \n",
       "672        WD  CompShg         Lvl           No          ALQ          Unf   \n",
       "\n",
       "     Functional Fence  \n",
       "1341        Typ     ?  \n",
       "459         Typ     ?  \n",
       "367         Typ     ?  \n",
       "894         Typ     ?  \n",
       "672         Typ     ?  \n",
       "\n",
       "[5 rows x 79 columns]"
      ]
     },
     "execution_count": 131,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_housing_cat.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CentralAir</th>\n",
       "      <th>Utilities</th>\n",
       "      <th>GarageType</th>\n",
       "      <th>PavedDrive</th>\n",
       "      <th>Neighborhood</th>\n",
       "      <th>MasVnrType</th>\n",
       "      <th>BldgType</th>\n",
       "      <th>GarageFinish</th>\n",
       "      <th>Heating</th>\n",
       "      <th>HouseStyle</th>\n",
       "      <th>...</th>\n",
       "      <th>MSSubClass</th>\n",
       "      <th>MoSold</th>\n",
       "      <th>SaleType</th>\n",
       "      <th>RoofMatl</th>\n",
       "      <th>LandContour</th>\n",
       "      <th>BsmtExposure</th>\n",
       "      <th>BsmtFinType1</th>\n",
       "      <th>BsmtFinType2</th>\n",
       "      <th>Functional</th>\n",
       "      <th>Fence</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1341</th>\n",
       "      <td>Y</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>Detchd</td>\n",
       "      <td>Y</td>\n",
       "      <td>SawyerW</td>\n",
       "      <td>None</td>\n",
       "      <td>1Fam</td>\n",
       "      <td>Unf</td>\n",
       "      <td>GasA</td>\n",
       "      <td>1Story</td>\n",
       "      <td>...</td>\n",
       "      <td>20</td>\n",
       "      <td>7</td>\n",
       "      <td>WD</td>\n",
       "      <td>CompShg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>No</td>\n",
       "      <td>GLQ</td>\n",
       "      <td>Unf</td>\n",
       "      <td>Typ</td>\n",
       "      <td>?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>459</th>\n",
       "      <td>Y</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>Detchd</td>\n",
       "      <td>Y</td>\n",
       "      <td>BrkSide</td>\n",
       "      <td>BrkCmn</td>\n",
       "      <td>1Fam</td>\n",
       "      <td>Unf</td>\n",
       "      <td>GasA</td>\n",
       "      <td>1.5Fin</td>\n",
       "      <td>...</td>\n",
       "      <td>50</td>\n",
       "      <td>7</td>\n",
       "      <td>WD</td>\n",
       "      <td>CompShg</td>\n",
       "      <td>Bnk</td>\n",
       "      <td>No</td>\n",
       "      <td>LwQ</td>\n",
       "      <td>Unf</td>\n",
       "      <td>Typ</td>\n",
       "      <td>?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>367</th>\n",
       "      <td>Y</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>Basment</td>\n",
       "      <td>Y</td>\n",
       "      <td>NAmes</td>\n",
       "      <td>BrkFace</td>\n",
       "      <td>1Fam</td>\n",
       "      <td>RFn</td>\n",
       "      <td>GasA</td>\n",
       "      <td>SLvl</td>\n",
       "      <td>...</td>\n",
       "      <td>80</td>\n",
       "      <td>12</td>\n",
       "      <td>WD</td>\n",
       "      <td>Tar&amp;Grv</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>Gd</td>\n",
       "      <td>GLQ</td>\n",
       "      <td>Unf</td>\n",
       "      <td>Typ</td>\n",
       "      <td>?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>894</th>\n",
       "      <td>Y</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>Attchd</td>\n",
       "      <td>Y</td>\n",
       "      <td>SawyerW</td>\n",
       "      <td>None</td>\n",
       "      <td>Duplex</td>\n",
       "      <td>Unf</td>\n",
       "      <td>GasA</td>\n",
       "      <td>1Story</td>\n",
       "      <td>...</td>\n",
       "      <td>90</td>\n",
       "      <td>6</td>\n",
       "      <td>WD</td>\n",
       "      <td>CompShg</td>\n",
       "      <td>Bnk</td>\n",
       "      <td>?</td>\n",
       "      <td>?</td>\n",
       "      <td>?</td>\n",
       "      <td>Typ</td>\n",
       "      <td>?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>672</th>\n",
       "      <td>Y</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>Attchd</td>\n",
       "      <td>Y</td>\n",
       "      <td>Veenker</td>\n",
       "      <td>None</td>\n",
       "      <td>1Fam</td>\n",
       "      <td>RFn</td>\n",
       "      <td>GasA</td>\n",
       "      <td>1Story</td>\n",
       "      <td>...</td>\n",
       "      <td>20</td>\n",
       "      <td>6</td>\n",
       "      <td>WD</td>\n",
       "      <td>CompShg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>No</td>\n",
       "      <td>ALQ</td>\n",
       "      <td>Unf</td>\n",
       "      <td>Typ</td>\n",
       "      <td>?</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 37 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     CentralAir Utilities GarageType PavedDrive Neighborhood MasVnrType  \\\n",
       "1341          Y    AllPub     Detchd          Y      SawyerW       None   \n",
       "459           Y    AllPub     Detchd          Y      BrkSide     BrkCmn   \n",
       "367           Y    AllPub    Basment          Y        NAmes    BrkFace   \n",
       "894           Y    AllPub     Attchd          Y      SawyerW       None   \n",
       "672           Y    AllPub     Attchd          Y      Veenker       None   \n",
       "\n",
       "     BldgType GarageFinish Heating HouseStyle  ... MSSubClass MoSold SaleType  \\\n",
       "1341     1Fam          Unf    GasA     1Story  ...         20      7       WD   \n",
       "459      1Fam          Unf    GasA     1.5Fin  ...         50      7       WD   \n",
       "367      1Fam          RFn    GasA       SLvl  ...         80     12       WD   \n",
       "894    Duplex          Unf    GasA     1Story  ...         90      6       WD   \n",
       "672      1Fam          RFn    GasA     1Story  ...         20      6       WD   \n",
       "\n",
       "     RoofMatl LandContour BsmtExposure BsmtFinType1 BsmtFinType2 Functional  \\\n",
       "1341  CompShg         Lvl           No          GLQ          Unf        Typ   \n",
       "459   CompShg         Bnk           No          LwQ          Unf        Typ   \n",
       "367   Tar&Grv         Lvl           Gd          GLQ          Unf        Typ   \n",
       "894   CompShg         Bnk            ?            ?            ?        Typ   \n",
       "672   CompShg         Lvl           No          ALQ          Unf        Typ   \n",
       "\n",
       "     Fence  \n",
       "1341     ?  \n",
       "459      ?  \n",
       "367      ?  \n",
       "894      ?  \n",
       "672      ?  \n",
       "\n",
       "[5 rows x 37 columns]"
      ]
     },
     "execution_count": 133,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_housing_cat[categorical_features].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CentralAir, Utilities, GarageType, PavedDrive, Neighborhood, MasVnrType, BldgType, GarageFinish, Heating, HouseStyle, LotShape, LotConfig, Electrical, KitchenAbvGr, Condition1, MiscFeature, Street, Alley, LandSlope, RoofStyle, Exterior2nd, Condition2, Exterior1st, MSZoning, SaleCondition, BedroomAbvGr, Foundation, MSSubClass, MoSold, SaleType, RoofMatl, LandContour, BsmtExposure, BsmtFinType1, BsmtFinType2, Functional, Fence\n"
     ]
    }
   ],
   "source": [
    "print(\", \".join(categorical_features))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Learning rate set to 0.039377\n",
      "0:\tlearn: 0.3951997\ttotal: 46.5ms\tremaining: 46.5s\n",
      "100:\tlearn: 0.1249415\ttotal: 5.66s\tremaining: 50.4s\n",
      "200:\tlearn: 0.1002285\ttotal: 11.7s\tremaining: 46.5s\n",
      "300:\tlearn: 0.0892388\ttotal: 18.7s\tremaining: 43.5s\n",
      "400:\tlearn: 0.0809382\ttotal: 23.7s\tremaining: 35.4s\n",
      "500:\tlearn: 0.0736061\ttotal: 30.2s\tremaining: 30.1s\n",
      "600:\tlearn: 0.0665125\ttotal: 38.5s\tremaining: 25.6s\n",
      "700:\tlearn: 0.0609587\ttotal: 44.1s\tremaining: 18.8s\n",
      "800:\tlearn: 0.0561925\ttotal: 50.1s\tremaining: 12.4s\n",
      "900:\tlearn: 0.0522277\ttotal: 56.7s\tremaining: 6.23s\n",
      "999:\tlearn: 0.0483357\ttotal: 1m 1s\tremaining: 0us\n"
     ]
    }
   ],
   "source": [
    "cat = CatBoostRegressor(cat_features=categorical_features)\n",
    "cat.fit(X_train_housing_cat, y_train_housing_log, verbose=100);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9034631808902006"
      ]
     },
     "execution_count": 136,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cat.score(X_valid_housing_cat, y_valid_housing_log)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case the code ran slower and it didn't seem to help, but I suspect it will in some cases!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Feature importances"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The syntax for getting feature importances is a bit clunky.\n",
    "- You first need to create a `catboost.Pool` object storing the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data_pool = Pool(X_train_housing_cat, y_train_housing_log, cat_features=categorical_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And then you can get feature importances:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {},
   "outputs": [],
   "source": [
    "importances = cat.get_feature_importance(train_data_pool)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we can display them as usual:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Importance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>OverallQual</th>\n",
       "      <td>17.132160</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>GrLivArea</th>\n",
       "      <td>15.718205</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TotalBsmtSF</th>\n",
       "      <td>6.045116</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ExterQual</th>\n",
       "      <td>3.995476</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>GarageCars</th>\n",
       "      <td>3.933389</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Street</th>\n",
       "      <td>0.003121</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LowQualFinSF</th>\n",
       "      <td>0.001376</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>BsmtHalfBath</th>\n",
       "      <td>0.000309</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>PoolQC</th>\n",
       "      <td>0.000113</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Utilities</th>\n",
       "      <td>0.000019</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>79 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "              Importance\n",
       "OverallQual    17.132160\n",
       "GrLivArea      15.718205\n",
       "TotalBsmtSF     6.045116\n",
       "ExterQual       3.995476\n",
       "GarageCars      3.933389\n",
       "...                  ...\n",
       "Street          0.003121\n",
       "LowQualFinSF    0.001376\n",
       "BsmtHalfBath    0.000309\n",
       "PoolQC          0.000113\n",
       "Utilities       0.000019\n",
       "\n",
       "[79 rows x 1 columns]"
      ]
     },
     "execution_count": 139,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "importances = pd.DataFrame(data=importances, index=X_train_housing_cat.columns, columns=[\"Importance\"])\n",
    "importances.sort_values(by=\"Importance\", ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CentralAir, Utilities, GarageType, PavedDrive, Neighborhood, MasVnrType, BldgType, GarageFinish, Heating, HouseStyle, LotShape, LotConfig, Electrical, KitchenAbvGr, Condition1, MiscFeature, Street, Alley, LandSlope, RoofStyle, Exterior2nd, Condition2, Exterior1st, MSZoning, SaleCondition, BedroomAbvGr, Foundation, MSSubClass, MoSold, SaleType, RoofMatl, LandContour, BsmtExposure, BsmtFinType1, BsmtFinType2, Functional, Fence\n"
     ]
    }
   ],
   "source": [
    "print(\", \".join(categorical_features))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- You can also get SHAP values, both through the SHAP package or built-in to CatBoost.\n",
    "- With proper handling of categorical variables, we can see an importance for the entire categorical variable, not just one possible value of it, which is kind of nice."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model comparison (15 min)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Very rough rule of thumb, when working on a problem with **tabular data**, try the following, in order: \n",
    "\n",
    "1. `DummyRegressor` or `DummyClassifer`\n",
    "2. `HuberRegressor` or `LogisticRegression`\n",
    "3. `CatBoostRegressor` or `CatBoostClassifier`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll now do a \"bake-off\" between different models and different datasets in the course, following the outlier scavenger hunt from L19."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Census data (lecture 7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(26048, 100)"
      ]
     },
     "execution_count": 141,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_adult, y_train_adult, X_test_adult, y_test_adult, _ = load_process_data.load_adult()\n",
    "X_train_adult.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>fnlwgt</th>\n",
       "      <th>education.num</th>\n",
       "      <th>capital.gain</th>\n",
       "      <th>capital.loss</th>\n",
       "      <th>hours.per.week</th>\n",
       "      <th>workclass_Federal-gov</th>\n",
       "      <th>workclass_Local-gov</th>\n",
       "      <th>workclass_Never-worked</th>\n",
       "      <th>workclass_Private</th>\n",
       "      <th>...</th>\n",
       "      <th>native.country_Portugal</th>\n",
       "      <th>native.country_Puerto-Rico</th>\n",
       "      <th>native.country_Scotland</th>\n",
       "      <th>native.country_South</th>\n",
       "      <th>native.country_Taiwan</th>\n",
       "      <th>native.country_Thailand</th>\n",
       "      <th>native.country_Trinadad&amp;Tobago</th>\n",
       "      <th>native.country_United-States</th>\n",
       "      <th>native.country_Vietnam</th>\n",
       "      <th>native.country_Yugoslavia</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>17064</th>\n",
       "      <td>-1.364769</td>\n",
       "      <td>-0.745073</td>\n",
       "      <td>-0.027403</td>\n",
       "      <td>-0.146669</td>\n",
       "      <td>-0.216807</td>\n",
       "      <td>-0.845065</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18434</th>\n",
       "      <td>-1.217915</td>\n",
       "      <td>0.708967</td>\n",
       "      <td>-0.416178</td>\n",
       "      <td>-0.146669</td>\n",
       "      <td>-0.216807</td>\n",
       "      <td>3.204805</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3294</th>\n",
       "      <td>0.911476</td>\n",
       "      <td>1.397806</td>\n",
       "      <td>-0.416178</td>\n",
       "      <td>0.451354</td>\n",
       "      <td>-0.216807</td>\n",
       "      <td>1.179870</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31317</th>\n",
       "      <td>1.058330</td>\n",
       "      <td>-0.736111</td>\n",
       "      <td>1.138922</td>\n",
       "      <td>-0.146669</td>\n",
       "      <td>-0.216807</td>\n",
       "      <td>-0.440078</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4770</th>\n",
       "      <td>0.984903</td>\n",
       "      <td>-1.316034</td>\n",
       "      <td>1.138922</td>\n",
       "      <td>-0.146669</td>\n",
       "      <td>-0.216807</td>\n",
       "      <td>1.179870</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 100 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            age    fnlwgt  education.num  capital.gain  capital.loss  \\\n",
       "17064 -1.364769 -0.745073      -0.027403     -0.146669     -0.216807   \n",
       "18434 -1.217915  0.708967      -0.416178     -0.146669     -0.216807   \n",
       "3294   0.911476  1.397806      -0.416178      0.451354     -0.216807   \n",
       "31317  1.058330 -0.736111       1.138922     -0.146669     -0.216807   \n",
       "4770   0.984903 -1.316034       1.138922     -0.146669     -0.216807   \n",
       "\n",
       "       hours.per.week  workclass_Federal-gov  workclass_Local-gov  \\\n",
       "17064       -0.845065                    0.0                  0.0   \n",
       "18434        3.204805                    0.0                  0.0   \n",
       "3294         1.179870                    0.0                  0.0   \n",
       "31317       -0.440078                    0.0                  0.0   \n",
       "4770         1.179870                    0.0                  0.0   \n",
       "\n",
       "       workclass_Never-worked  workclass_Private  ...  \\\n",
       "17064                     0.0                1.0  ...   \n",
       "18434                     0.0                1.0  ...   \n",
       "3294                      0.0                1.0  ...   \n",
       "31317                     0.0                1.0  ...   \n",
       "4770                      0.0                0.0  ...   \n",
       "\n",
       "       native.country_Portugal  native.country_Puerto-Rico  \\\n",
       "17064                      0.0                         0.0   \n",
       "18434                      0.0                         0.0   \n",
       "3294                       0.0                         0.0   \n",
       "31317                      0.0                         0.0   \n",
       "4770                       0.0                         0.0   \n",
       "\n",
       "       native.country_Scotland  native.country_South  native.country_Taiwan  \\\n",
       "17064                      0.0                   0.0                    0.0   \n",
       "18434                      0.0                   0.0                    0.0   \n",
       "3294                       0.0                   0.0                    0.0   \n",
       "31317                      0.0                   0.0                    0.0   \n",
       "4770                       0.0                   0.0                    0.0   \n",
       "\n",
       "       native.country_Thailand  native.country_Trinadad&Tobago  \\\n",
       "17064                      0.0                             0.0   \n",
       "18434                      0.0                             0.0   \n",
       "3294                       0.0                             0.0   \n",
       "31317                      0.0                             0.0   \n",
       "4770                       0.0                             0.0   \n",
       "\n",
       "       native.country_United-States  native.country_Vietnam  \\\n",
       "17064                           1.0                     0.0   \n",
       "18434                           1.0                     0.0   \n",
       "3294                            1.0                     0.0   \n",
       "31317                           1.0                     0.0   \n",
       "4770                            1.0                     0.0   \n",
       "\n",
       "       native.country_Yugoslavia  \n",
       "17064                        0.0  \n",
       "18434                        0.0  \n",
       "3294                         0.0  \n",
       "31317                        0.0  \n",
       "4770                         0.0  \n",
       "\n",
       "[5 rows x 100 columns]"
      ]
     },
     "execution_count": 142,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_adult.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [],
   "source": [
    "show_scores_adult = show_scores_factory(X_train_adult, y_train_adult, X_test_adult, y_test_adult)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7538768616612928"
      ]
     },
     "execution_count": 144,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "show_scores_adult(DummyClassifier(strategy=\"prior\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8510670965760786"
      ]
     },
     "execution_count": 145,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "show_scores_adult(LogisticRegression(max_iter=1000))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8711807154920927"
      ]
     },
     "execution_count": 146,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "show_scores_adult(xgb.XGBClassifier())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8745585751573776"
      ]
     },
     "execution_count": 147,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "show_scores_adult(lgb.LGBMClassifier())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8730231844004299"
      ]
     },
     "execution_count": 148,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "show_scores_adult(CatBoostClassifier(), verbose=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_adult_cat, y_train_adult, X_test_adult_cat, y_test_adult, cat_features_adult = load_process_data.load_adult(ohe=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>fnlwgt</th>\n",
       "      <th>education.num</th>\n",
       "      <th>capital.gain</th>\n",
       "      <th>capital.loss</th>\n",
       "      <th>hours.per.week</th>\n",
       "      <th>workclass</th>\n",
       "      <th>education</th>\n",
       "      <th>marital.status</th>\n",
       "      <th>occupation</th>\n",
       "      <th>relationship</th>\n",
       "      <th>race</th>\n",
       "      <th>sex</th>\n",
       "      <th>native.country</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>17064</th>\n",
       "      <td>-1.36477</td>\n",
       "      <td>-0.745073</td>\n",
       "      <td>-0.0274029</td>\n",
       "      <td>-0.146669</td>\n",
       "      <td>-0.216807</td>\n",
       "      <td>-0.845065</td>\n",
       "      <td>Private</td>\n",
       "      <td>Some-college</td>\n",
       "      <td>Never-married</td>\n",
       "      <td>Adm-clerical</td>\n",
       "      <td>Own-child</td>\n",
       "      <td>Asian-Pac-Islander</td>\n",
       "      <td>Female</td>\n",
       "      <td>United-States</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18434</th>\n",
       "      <td>-1.21791</td>\n",
       "      <td>0.708967</td>\n",
       "      <td>-0.416178</td>\n",
       "      <td>-0.146669</td>\n",
       "      <td>-0.216807</td>\n",
       "      <td>3.2048</td>\n",
       "      <td>Private</td>\n",
       "      <td>HS-grad</td>\n",
       "      <td>Never-married</td>\n",
       "      <td>Other-service</td>\n",
       "      <td>Own-child</td>\n",
       "      <td>Black</td>\n",
       "      <td>Male</td>\n",
       "      <td>United-States</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3294</th>\n",
       "      <td>0.911476</td>\n",
       "      <td>1.39781</td>\n",
       "      <td>-0.416178</td>\n",
       "      <td>0.451354</td>\n",
       "      <td>-0.216807</td>\n",
       "      <td>1.17987</td>\n",
       "      <td>Private</td>\n",
       "      <td>HS-grad</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Exec-managerial</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>United-States</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31317</th>\n",
       "      <td>1.05833</td>\n",
       "      <td>-0.736111</td>\n",
       "      <td>1.13892</td>\n",
       "      <td>-0.146669</td>\n",
       "      <td>-0.216807</td>\n",
       "      <td>-0.440078</td>\n",
       "      <td>Private</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Other-service</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>United-States</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4770</th>\n",
       "      <td>0.984903</td>\n",
       "      <td>-1.31603</td>\n",
       "      <td>1.13892</td>\n",
       "      <td>-0.146669</td>\n",
       "      <td>-0.216807</td>\n",
       "      <td>1.17987</td>\n",
       "      <td>Self-emp-inc</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Sales</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>United-States</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            age    fnlwgt education.num capital.gain capital.loss  \\\n",
       "17064  -1.36477 -0.745073    -0.0274029    -0.146669    -0.216807   \n",
       "18434  -1.21791  0.708967     -0.416178    -0.146669    -0.216807   \n",
       "3294   0.911476   1.39781     -0.416178     0.451354    -0.216807   \n",
       "31317   1.05833 -0.736111       1.13892    -0.146669    -0.216807   \n",
       "4770   0.984903  -1.31603       1.13892    -0.146669    -0.216807   \n",
       "\n",
       "      hours.per.week     workclass     education      marital.status  \\\n",
       "17064      -0.845065       Private  Some-college       Never-married   \n",
       "18434         3.2048       Private       HS-grad       Never-married   \n",
       "3294         1.17987       Private       HS-grad  Married-civ-spouse   \n",
       "31317      -0.440078       Private     Bachelors  Married-civ-spouse   \n",
       "4770         1.17987  Self-emp-inc     Bachelors  Married-civ-spouse   \n",
       "\n",
       "            occupation relationship                race     sex native.country  \n",
       "17064     Adm-clerical    Own-child  Asian-Pac-Islander  Female  United-States  \n",
       "18434    Other-service    Own-child               Black    Male  United-States  \n",
       "3294   Exec-managerial      Husband               White    Male  United-States  \n",
       "31317    Other-service      Husband               White    Male  United-States  \n",
       "4770             Sales      Husband               White    Male  United-States  "
      ]
     },
     "execution_count": 150,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_adult_cat.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Learning rate set to 0.041445\n",
      "0:\tlearn: 0.6486984\ttotal: 80.6ms\tremaining: 1m 20s\n",
      "100:\tlearn: 0.2916250\ttotal: 16.4s\tremaining: 2m 25s\n",
      "200:\tlearn: 0.2780044\ttotal: 29.3s\tremaining: 1m 56s\n",
      "300:\tlearn: 0.2681981\ttotal: 38.4s\tremaining: 1m 29s\n",
      "400:\tlearn: 0.2609732\ttotal: 45.9s\tremaining: 1m 8s\n",
      "500:\tlearn: 0.2558415\ttotal: 54.1s\tremaining: 53.9s\n",
      "600:\tlearn: 0.2517539\ttotal: 1m 9s\tremaining: 46s\n",
      "700:\tlearn: 0.2480964\ttotal: 1m 20s\tremaining: 34.5s\n",
      "800:\tlearn: 0.2447137\ttotal: 1m 28s\tremaining: 22s\n",
      "900:\tlearn: 0.2418840\ttotal: 1m 36s\tremaining: 10.6s\n",
      "999:\tlearn: 0.2387305\ttotal: 1m 43s\tremaining: 0us\n"
     ]
    }
   ],
   "source": [
    "cat = CatBoostClassifier(cat_features=cat_features_adult)\n",
    "cat.fit(X_train_adult_cat, y_train_adult, verbose=100);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8728696453247351"
      ]
     },
     "execution_count": 152,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cat.score(X_test_adult_cat, y_test_adult)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TODO\n",
    "\n",
    "compare on AUC as well, accuracy might not be as meaningful"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Movie review data (lecture 5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(8000, 2699)"
      ]
     },
     "execution_count": 153,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_imdb, y_train_imdb, X_test_imdb, y_test_imdb = load_process_data.load_movie()\n",
    "X_train_imdb.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "scipy.sparse.csr.csr_matrix"
      ]
     },
     "execution_count": 155,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(X_train_imdb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "metadata": {},
   "outputs": [],
   "source": [
    "show_scores_movie = show_scores_factory(X_train_imdb, y_train_imdb, X_test_imdb, y_test_imdb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.4935"
      ]
     },
     "execution_count": 157,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "show_scores_movie(DummyClassifier(strategy=\"prior\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.838"
      ]
     },
     "execution_count": 158,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "show_scores_movie(LogisticRegression(max_iter=1000))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8415"
      ]
     },
     "execution_count": 159,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "show_scores_movie(xgb.XGBClassifier())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "LightGBM seems to kill my kernel in this case... too many features?? Or maybe it can't handle sparse matrices as input? That seems surprising though."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# show_scores_movie(lgb.LGBMClassifier())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Learning rate set to 0.025035\n",
      "0:\tlearn: 0.6858406\ttotal: 125ms\tremaining: 2m 4s\n",
      "100:\tlearn: 0.4950686\ttotal: 11.6s\tremaining: 1m 43s\n",
      "200:\tlearn: 0.4394911\ttotal: 22.5s\tremaining: 1m 29s\n",
      "300:\tlearn: 0.4038004\ttotal: 33.2s\tremaining: 1m 17s\n",
      "400:\tlearn: 0.3718700\ttotal: 42.8s\tremaining: 1m 3s\n",
      "500:\tlearn: 0.3415765\ttotal: 52.7s\tremaining: 52.5s\n",
      "600:\tlearn: 0.3163704\ttotal: 1m 5s\tremaining: 43.2s\n",
      "700:\tlearn: 0.2956700\ttotal: 1m 13s\tremaining: 31.6s\n",
      "800:\tlearn: 0.2776943\ttotal: 1m 24s\tremaining: 20.9s\n",
      "900:\tlearn: 0.2614404\ttotal: 1m 33s\tremaining: 10.2s\n",
      "999:\tlearn: 0.2472331\ttotal: 1m 49s\tremaining: 0us\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.8595"
      ]
     },
     "execution_count": 160,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "show_scores_movie(CatBoostClassifier(), verbose=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are no categorical features here (they are all word counts), so no need to do another experiment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Rain in Australia data (lecture 16)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(107502, 125)"
      ]
     },
     "execution_count": 161,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_rain, y_train_rain, X_test_rain, y_test_rain, _ = load_process_data.load_rain()\n",
    "X_train_rain.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "metadata": {},
   "outputs": [],
   "source": [
    "show_scores_rain = show_scores_factory(X_train_rain, y_train_rain, X_test_rain, y_test_rain)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 163,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7781845435415525"
      ]
     },
     "execution_count": 163,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "show_scores_rain(DummyClassifier(strategy=\"prior\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 164,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8454354155256406"
      ]
     },
     "execution_count": 164,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "show_scores_rain(LogisticRegression(max_iter=1000))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 165,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8550632728949872"
      ]
     },
     "execution_count": 165,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "show_scores_rain(xgb.XGBClassifier(verbosity=1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 166,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8536219768816119"
      ]
     },
     "execution_count": 166,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "show_scores_rain(lgb.LGBMClassifier())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 167,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Learning rate set to 0.075919\n",
      "0:\tlearn: 0.6417575\ttotal: 52.8ms\tremaining: 52.7s\n",
      "100:\tlearn: 0.3397858\ttotal: 6.78s\tremaining: 1m\n",
      "200:\tlearn: 0.3249774\ttotal: 14.3s\tremaining: 57s\n",
      "300:\tlearn: 0.3139556\ttotal: 21.8s\tremaining: 50.6s\n",
      "400:\tlearn: 0.3049408\ttotal: 28.3s\tremaining: 42.3s\n",
      "500:\tlearn: 0.2972943\ttotal: 36.2s\tremaining: 36s\n",
      "600:\tlearn: 0.2904276\ttotal: 44.7s\tremaining: 29.7s\n",
      "700:\tlearn: 0.2841649\ttotal: 51.3s\tremaining: 21.9s\n",
      "800:\tlearn: 0.2783169\ttotal: 1m\tremaining: 15s\n",
      "900:\tlearn: 0.2727251\ttotal: 1m 8s\tremaining: 7.51s\n",
      "999:\tlearn: 0.2675276\ttotal: 1m 14s\tremaining: 0us\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.8593006831743103"
      ]
     },
     "execution_count": 167,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "show_scores_rain(CatBoostClassifier(), verbose=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 168,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(107502, 26)"
      ]
     },
     "execution_count": 168,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_rain_cat, _, X_test_rain_cat, _, cat_features_rain = load_process_data.load_rain(ohe=False)\n",
    "X_train_rain_cat.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 171,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MinTemp</th>\n",
       "      <th>MaxTemp</th>\n",
       "      <th>Rainfall</th>\n",
       "      <th>Evaporation</th>\n",
       "      <th>Sunshine</th>\n",
       "      <th>WindGustSpeed</th>\n",
       "      <th>WindSpeed9am</th>\n",
       "      <th>WindSpeed3pm</th>\n",
       "      <th>Humidity9am</th>\n",
       "      <th>Humidity3pm</th>\n",
       "      <th>...</th>\n",
       "      <th>Rainfall_lag1</th>\n",
       "      <th>Rainfall_lag2</th>\n",
       "      <th>Rainfall_lag3</th>\n",
       "      <th>Humidity3pm_lag1</th>\n",
       "      <th>Location</th>\n",
       "      <th>WindGustDir</th>\n",
       "      <th>WindDir9am</th>\n",
       "      <th>WindDir3pm</th>\n",
       "      <th>RainToday</th>\n",
       "      <th>Month</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.204302</td>\n",
       "      <td>-0.0271117</td>\n",
       "      <td>-0.205323</td>\n",
       "      <td>-0.140641</td>\n",
       "      <td>0.160729</td>\n",
       "      <td>0.298612</td>\n",
       "      <td>0.666166</td>\n",
       "      <td>0.599894</td>\n",
       "      <td>0.115428</td>\n",
       "      <td>-1.43351</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.274971</td>\n",
       "      <td>-0.274913</td>\n",
       "      <td>-0.274905</td>\n",
       "      <td>0.021742</td>\n",
       "      <td>Albury</td>\n",
       "      <td>W</td>\n",
       "      <td>W</td>\n",
       "      <td>WNW</td>\n",
       "      <td>No</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-0.741037</td>\n",
       "      <td>0.287031</td>\n",
       "      <td>-0.275008</td>\n",
       "      <td>-0.140641</td>\n",
       "      <td>0.160729</td>\n",
       "      <td>0.298612</td>\n",
       "      <td>-1.12562</td>\n",
       "      <td>0.373275</td>\n",
       "      <td>-1.31493</td>\n",
       "      <td>-1.288</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.205283</td>\n",
       "      <td>-0.274913</td>\n",
       "      <td>-0.274905</td>\n",
       "      <td>-1.43362</td>\n",
       "      <td>Albury</td>\n",
       "      <td>WNW</td>\n",
       "      <td>NNW</td>\n",
       "      <td>WSW</td>\n",
       "      <td>No</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.125523</td>\n",
       "      <td>0.372706</td>\n",
       "      <td>-0.275008</td>\n",
       "      <td>-0.140641</td>\n",
       "      <td>0.160729</td>\n",
       "      <td>0.450132</td>\n",
       "      <td>0.55418</td>\n",
       "      <td>0.826513</td>\n",
       "      <td>-1.63279</td>\n",
       "      <td>-1.04548</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.274971</td>\n",
       "      <td>-0.205215</td>\n",
       "      <td>-0.274905</td>\n",
       "      <td>-1.28808</td>\n",
       "      <td>Albury</td>\n",
       "      <td>WSW</td>\n",
       "      <td>W</td>\n",
       "      <td>WSW</td>\n",
       "      <td>No</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.457435</td>\n",
       "      <td>0.701128</td>\n",
       "      <td>-0.275008</td>\n",
       "      <td>-0.140641</td>\n",
       "      <td>0.160729</td>\n",
       "      <td>-1.2166</td>\n",
       "      <td>-0.341712</td>\n",
       "      <td>-1.09975</td>\n",
       "      <td>-1.26195</td>\n",
       "      <td>-1.72454</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.274971</td>\n",
       "      <td>-0.274913</td>\n",
       "      <td>-0.205207</td>\n",
       "      <td>-1.04552</td>\n",
       "      <td>Albury</td>\n",
       "      <td>NE</td>\n",
       "      <td>SE</td>\n",
       "      <td>E</td>\n",
       "      <td>No</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.850283</td>\n",
       "      <td>1.31513</td>\n",
       "      <td>-0.158867</td>\n",
       "      <td>-0.140641</td>\n",
       "      <td>0.160729</td>\n",
       "      <td>0.0713304</td>\n",
       "      <td>-0.789657</td>\n",
       "      <td>0.146656</td>\n",
       "      <td>0.698167</td>\n",
       "      <td>-0.899969</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.274971</td>\n",
       "      <td>-0.274913</td>\n",
       "      <td>-0.274905</td>\n",
       "      <td>-1.72469</td>\n",
       "      <td>Albury</td>\n",
       "      <td>W</td>\n",
       "      <td>ENE</td>\n",
       "      <td>NW</td>\n",
       "      <td>No</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 26 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    MinTemp    MaxTemp  Rainfall Evaporation  Sunshine WindGustSpeed  \\\n",
       "0  0.204302 -0.0271117 -0.205323   -0.140641  0.160729      0.298612   \n",
       "1 -0.741037   0.287031 -0.275008   -0.140641  0.160729      0.298612   \n",
       "2  0.125523   0.372706 -0.275008   -0.140641  0.160729      0.450132   \n",
       "3 -0.457435   0.701128 -0.275008   -0.140641  0.160729       -1.2166   \n",
       "4  0.850283    1.31513 -0.158867   -0.140641  0.160729     0.0713304   \n",
       "\n",
       "  WindSpeed9am WindSpeed3pm Humidity9am Humidity3pm  ... Rainfall_lag1  \\\n",
       "0     0.666166     0.599894    0.115428    -1.43351  ...     -0.274971   \n",
       "1     -1.12562     0.373275    -1.31493      -1.288  ...     -0.205283   \n",
       "2      0.55418     0.826513    -1.63279    -1.04548  ...     -0.274971   \n",
       "3    -0.341712     -1.09975    -1.26195    -1.72454  ...     -0.274971   \n",
       "4    -0.789657     0.146656    0.698167   -0.899969  ...     -0.274971   \n",
       "\n",
       "  Rainfall_lag2 Rainfall_lag3 Humidity3pm_lag1 Location WindGustDir  \\\n",
       "0     -0.274913     -0.274905         0.021742   Albury           W   \n",
       "1     -0.274913     -0.274905         -1.43362   Albury         WNW   \n",
       "2     -0.205215     -0.274905         -1.28808   Albury         WSW   \n",
       "3     -0.274913     -0.205207         -1.04552   Albury          NE   \n",
       "4     -0.274913     -0.274905         -1.72469   Albury           W   \n",
       "\n",
       "  WindDir9am WindDir3pm RainToday Month  \n",
       "0          W        WNW        No    12  \n",
       "1        NNW        WSW        No    12  \n",
       "2          W        WSW        No    12  \n",
       "3         SE          E        No    12  \n",
       "4        ENE         NW        No    12  \n",
       "\n",
       "[5 rows x 26 columns]"
      ]
     },
     "execution_count": 171,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_rain_cat.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 169,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Learning rate set to 0.143349\n",
      "0:\tlearn: 0.5970566\ttotal: 336ms\tremaining: 2m 47s\n",
      "50:\tlearn: 0.3339990\ttotal: 15.8s\tremaining: 2m 18s\n",
      "100:\tlearn: 0.3198865\ttotal: 28.8s\tremaining: 1m 53s\n",
      "150:\tlearn: 0.3101000\ttotal: 41.6s\tremaining: 1m 36s\n",
      "200:\tlearn: 0.3027833\ttotal: 54.8s\tremaining: 1m 21s\n",
      "250:\tlearn: 0.2968515\ttotal: 1m 47s\tremaining: 1m 47s\n",
      "300:\tlearn: 0.2914525\ttotal: 1m 58s\tremaining: 1m 18s\n",
      "350:\tlearn: 0.2869166\ttotal: 2m 9s\tremaining: 55.1s\n",
      "400:\tlearn: 0.2821816\ttotal: 2m 23s\tremaining: 35.5s\n",
      "450:\tlearn: 0.2779684\ttotal: 2m 46s\tremaining: 18.1s\n",
      "499:\tlearn: 0.2739957\ttotal: 3m 17s\tremaining: 0us\n"
     ]
    }
   ],
   "source": [
    "cat = CatBoostClassifier(iterations=500, cat_features=cat_features_rain)\n",
    "cat.fit(X_train_rain_cat, y_train_rain, verbose=50);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 170,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8617508863970482"
      ]
     },
     "execution_count": 170,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cat.score(X_test_rain_cat, y_test_rain)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Discussion\n",
    "\n",
    "- Overall CatBoost looks like a good bet. \n",
    "- We didn't do any hyperparameter tuning, this might change the results. Maybe they just have better defaults.\n",
    "  - In fact, we did not discuss hyperparameters at all for these fancy models.\n",
    "  - There are a ton of them!\n",
    "  - For CatBoost, a main hyperparameter controlling the fundamental tradeoff is `iterations`. \n",
    "    - This is similar to `n_estimators` in the sklearn random forests: larger = more complex models.\n",
    "  - Another main hyperparameter is `learning_rate` but its interpretation is a bit beyond our scope.\n",
    "  - These models should all be compatible with `RandomizedSearchCV`, etc.\n",
    "  - But CatBoost, at least, does not work out of the box for feature selection with `RFE`/`RFECV`\n",
    "- But I think it's pretty good. A bit slow though.\n",
    "  - Now that speed is an issue, we should note that increasing complexity via `iterations` also increases runtime.\n",
    "  - This is not always true, e.g. `gamma` in SVM and (for the most part) `C` in `LogisticRegression`.\n",
    "  - But in fact often we did get increased runtime with more complexity, e.g. more features from `CountVectorizer`. \n",
    "- I'm surprising that feeding in the raw categorical features doesn't seem to help in many cases. \n",
    "  - Isn't that the point of CatBoost?!\n",
    "  - It does seem to help with the Rain data.\n",
    "  - I will need to look into this more at some point.\n",
    "    - Maybe the code needs to be run for longer in these cases?\n",
    "    - Hyperparameter tuning should be done in each case, at least a bit."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Break (5 min)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Big data sets: `SGDClassifier` and `SGDRegressor` (10 min)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Something we haven't discussed in this course is huge data sets.\n",
    "- There could be a couple problems arising from huge data sets:\n",
    "\n",
    "1. The code is too slow.\n",
    "2. The dataset doesn't fit in memory - I can't even load it with `pd.read_csv`. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- \"SGD\" (stochastic gradient descent) can help with both of these problems.\n",
    "- But we'll focus on using it to solve problem (1).\n",
    "- There is a fancy way to implement `fit` that can be a lot faster for big data sets.\n",
    "  - You can think of it as quickly finding \"approximately\" the best coefficients when calling `fit`.\n",
    "  - That is not quite true but it may be a useful way of thinking.\n",
    "  - Much more on this in CPSC 340 and much, much more on this in CPSC 440/540."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- SGD can be used in many contexts.\n",
    "- In sklearn, it's built in as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 172,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import SGDClassifier, SGDRegressor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- `SGDRegressor` is basically equivalent to `Ridge`.\n",
    "- `SGDRegressor(loss='huber')` is basically equivalent to `HuberRegressor`.\n",
    "- `SGDClassifier(loss='log')` is basically equivalent to `LogisticRegression`, except the parameter is called `alpha` instead of `C` (like `Ridge`).\n",
    "- With other settings they are equivalent to other models, but this is good enough."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- For regular sized datasets, use the original functions, as these ones can be a bit more finicky. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's load the [Sentiment140 dataset](http://help.sentiment140.com/home), which contains tweets labeled with sentiment associated with a brand, product, or topic. (I don't think we've looked at this dataset before - using it here because it's large.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 173,
   "metadata": {},
   "outputs": [],
   "source": [
    "tweets_df = pd.read_csv('data/training.1600000.processed.noemoticon.csv', \n",
    "                        encoding = \"ISO-8859-1\",\n",
    "                        names=[\"label\",\"id\", \"date\", \"no_query\", \"name\", \"text\"])\n",
    "tweets_df['label'] = tweets_df['label'].map({0: 'neg', 4: 'pos'})\n",
    "tweets_df = tweets_df[tweets_df['label'].str.startswith(('pos','neg'))]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 174,
   "metadata": {},
   "outputs": [],
   "source": [
    "tweets_df_train, tweets_df_test = train_test_split(tweets_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 175,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>id</th>\n",
       "      <th>date</th>\n",
       "      <th>no_query</th>\n",
       "      <th>name</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>623519</th>\n",
       "      <td>neg</td>\n",
       "      <td>2229582799</td>\n",
       "      <td>Thu Jun 18 15:55:25 PDT 2009</td>\n",
       "      <td>NO_QUERY</td>\n",
       "      <td>Asmcqu01</td>\n",
       "      <td>@mcquage Yep I'm coming home...what's sad thou...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1210683</th>\n",
       "      <td>pos</td>\n",
       "      <td>1989074133</td>\n",
       "      <td>Mon Jun 01 00:15:38 PDT 2009</td>\n",
       "      <td>NO_QUERY</td>\n",
       "      <td>DaWaBZ94</td>\n",
       "      <td>@bb_nation Hi 9h in paris, started gravity - w...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1385171</th>\n",
       "      <td>pos</td>\n",
       "      <td>2052722086</td>\n",
       "      <td>Sat Jun 06 01:46:04 PDT 2009</td>\n",
       "      <td>NO_QUERY</td>\n",
       "      <td>Desmond_Ong</td>\n",
       "      <td>@ianternet haha...play online games ian.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1488275</th>\n",
       "      <td>pos</td>\n",
       "      <td>2068458374</td>\n",
       "      <td>Sun Jun 07 14:06:50 PDT 2009</td>\n",
       "      <td>NO_QUERY</td>\n",
       "      <td>TheWendyHouse</td>\n",
       "      <td>Evening all! That didn't take long did it? Ha ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1306867</th>\n",
       "      <td>pos</td>\n",
       "      <td>2012075176</td>\n",
       "      <td>Tue Jun 02 20:01:44 PDT 2009</td>\n",
       "      <td>NO_QUERY</td>\n",
       "      <td>sarasmile13</td>\n",
       "      <td>@nakeddmblauren I will go back to my self-pity...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        label          id                          date  no_query  \\\n",
       "623519    neg  2229582799  Thu Jun 18 15:55:25 PDT 2009  NO_QUERY   \n",
       "1210683   pos  1989074133  Mon Jun 01 00:15:38 PDT 2009  NO_QUERY   \n",
       "1385171   pos  2052722086  Sat Jun 06 01:46:04 PDT 2009  NO_QUERY   \n",
       "1488275   pos  2068458374  Sun Jun 07 14:06:50 PDT 2009  NO_QUERY   \n",
       "1306867   pos  2012075176  Tue Jun 02 20:01:44 PDT 2009  NO_QUERY   \n",
       "\n",
       "                  name                                               text  \n",
       "623519        Asmcqu01  @mcquage Yep I'm coming home...what's sad thou...  \n",
       "1210683       DaWaBZ94  @bb_nation Hi 9h in paris, started gravity - w...  \n",
       "1385171    Desmond_Ong          @ianternet haha...play online games ian.   \n",
       "1488275  TheWendyHouse  Evening all! That didn't take long did it? Ha ...  \n",
       "1306867    sarasmile13  @nakeddmblauren I will go back to my self-pity...  "
      ]
     },
     "execution_count": 175,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tweets_df_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 176,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1200000, 6)"
      ]
     },
     "execution_count": 176,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tweets_df_train.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Holy cow!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 177,
   "metadata": {},
   "outputs": [],
   "source": [
    "vec = CountVectorizer(stop_words='english')\n",
    "\n",
    "X_train = vec.fit_transform(tweets_df_train['text']) \n",
    "y_train = tweets_df_train['label']\n",
    "\n",
    "X_test = vec.transform(tweets_df_test['text']) \n",
    "y_test = tweets_df_test['label']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 178,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1200000, 563585)"
      ]
     },
     "execution_count": 178,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 179,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "scipy.sparse.csr.csr_matrix"
      ]
     },
     "execution_count": 179,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is the fraction of elements that are nonzero. Having a sparse matrix really helps!!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 180,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.2346648390807657e-05"
      ]
     },
     "execution_count": 180,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.nnz/np.prod(X_train.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's train a classifier. I'll use `time` instead of `%timeit` because I want to keep the output, and it gets lost with `%timeit`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 181,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = LogisticRegression()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 182,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1min 2s, sys: 2.53 s, total: 1min 5s\n",
      "Wall time: 54.6 s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/mgelbart/opt/anaconda3/lib/python3.7/site-packages/sklearn/linear_model/_logistic.py:940: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "                   intercept_scaling=1, l1_ratio=None, max_iter=100,\n",
       "                   multi_class='auto', n_jobs=None, penalty='l2',\n",
       "                   random_state=None, solver='lbfgs', tol=0.0001, verbose=0,\n",
       "                   warm_start=False)"
      ]
     },
     "execution_count": 182,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "lr.fit(X_train, y_train);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 183,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8160875"
      ]
     },
     "execution_count": 183,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr.score(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 184,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7780825"
      ]
     },
     "execution_count": 184,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 185,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr_sgd = SGDClassifier(loss=\"log\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 186,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 5.71 s, sys: 125 ms, total: 5.84 s\n",
      "Wall time: 6.97 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "SGDClassifier(alpha=0.0001, average=False, class_weight=None,\n",
       "              early_stopping=False, epsilon=0.1, eta0=0.0, fit_intercept=True,\n",
       "              l1_ratio=0.15, learning_rate='optimal', loss='log', max_iter=1000,\n",
       "              n_iter_no_change=5, n_jobs=None, penalty='l2', power_t=0.5,\n",
       "              random_state=None, shuffle=True, tol=0.001,\n",
       "              validation_fraction=0.1, verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 186,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "lr_sgd.fit(X_train, y_train);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 187,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7"
      ]
     },
     "execution_count": 187,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_sgd.n_iter_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 188,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7695383333333333"
      ]
     },
     "execution_count": 188,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_sgd.score(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 189,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.76628"
      ]
     },
     "execution_count": 189,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_sgd.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- `SGDClassifier` was about 10x faster than `LogisticRegression`, but the accuracy is slightly lower.\n",
    "- In fact, we can control the speed vs. _train_ accuracy tradeoff in both cases using the hyperparameters.\n",
    "  - The main ones are `max_iter` (higher is slower) and/or `tol` (lower is slower)\n",
    "  - (This is the same for both `LogisticRegression` and `SGDClassifier`)\n",
    "- In general, `LogisticRegression` will get slightly higher _train_ accuracy (may or may not correspond to better validation/test)\n",
    "- But in some cases your dataset is so big that `LogisticRegression` is not feasible, and then `SGDClassifier` can save the day.\n",
    "\n",
    "Random comment: last time I tried this was with sklearn 0.21.3 and scipy 1.3.1 a few months ago, now with skearln 0.22.1 and scipy 1.4.1. I wonder if `LogisticRegression`, or the underlying optimizer in scipy, was improved at all in these recent upgrades, because it seems to be faster now than last time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multi-class classification (15 min)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- I intended to cover this at some point but it didn't really come up naturally.\n",
    "- We briefly talked about it in Lecture 14 on neural networks & computer vision.\n",
    "- Most of the classification problems we looked at were binary classification (2 possible labels)\n",
    "- There are plenty of multi-class classification problems as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 190,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = sklearn.datasets.load_wine()\n",
    "X = pd.DataFrame(data['data'], columns=data[\"feature_names\"])\n",
    "y = data['target']\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 191,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>alcohol</th>\n",
       "      <th>malic_acid</th>\n",
       "      <th>ash</th>\n",
       "      <th>alcalinity_of_ash</th>\n",
       "      <th>magnesium</th>\n",
       "      <th>total_phenols</th>\n",
       "      <th>flavanoids</th>\n",
       "      <th>nonflavanoid_phenols</th>\n",
       "      <th>proanthocyanins</th>\n",
       "      <th>color_intensity</th>\n",
       "      <th>hue</th>\n",
       "      <th>od280/od315_of_diluted_wines</th>\n",
       "      <th>proline</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>109</th>\n",
       "      <td>11.61</td>\n",
       "      <td>1.35</td>\n",
       "      <td>2.70</td>\n",
       "      <td>20.0</td>\n",
       "      <td>94.0</td>\n",
       "      <td>2.74</td>\n",
       "      <td>2.92</td>\n",
       "      <td>0.29</td>\n",
       "      <td>2.49</td>\n",
       "      <td>2.65</td>\n",
       "      <td>0.960</td>\n",
       "      <td>3.26</td>\n",
       "      <td>680.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>106</th>\n",
       "      <td>12.25</td>\n",
       "      <td>1.73</td>\n",
       "      <td>2.12</td>\n",
       "      <td>19.0</td>\n",
       "      <td>80.0</td>\n",
       "      <td>1.65</td>\n",
       "      <td>2.03</td>\n",
       "      <td>0.37</td>\n",
       "      <td>1.63</td>\n",
       "      <td>3.40</td>\n",
       "      <td>1.000</td>\n",
       "      <td>3.17</td>\n",
       "      <td>510.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>134</th>\n",
       "      <td>12.51</td>\n",
       "      <td>1.24</td>\n",
       "      <td>2.25</td>\n",
       "      <td>17.5</td>\n",
       "      <td>85.0</td>\n",
       "      <td>2.00</td>\n",
       "      <td>0.58</td>\n",
       "      <td>0.60</td>\n",
       "      <td>1.25</td>\n",
       "      <td>5.45</td>\n",
       "      <td>0.750</td>\n",
       "      <td>1.51</td>\n",
       "      <td>650.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>89</th>\n",
       "      <td>12.08</td>\n",
       "      <td>1.33</td>\n",
       "      <td>2.30</td>\n",
       "      <td>23.6</td>\n",
       "      <td>70.0</td>\n",
       "      <td>2.20</td>\n",
       "      <td>1.59</td>\n",
       "      <td>0.42</td>\n",
       "      <td>1.38</td>\n",
       "      <td>1.74</td>\n",
       "      <td>1.070</td>\n",
       "      <td>3.21</td>\n",
       "      <td>625.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>13.75</td>\n",
       "      <td>1.73</td>\n",
       "      <td>2.41</td>\n",
       "      <td>16.0</td>\n",
       "      <td>89.0</td>\n",
       "      <td>2.60</td>\n",
       "      <td>2.76</td>\n",
       "      <td>0.29</td>\n",
       "      <td>1.81</td>\n",
       "      <td>5.60</td>\n",
       "      <td>1.150</td>\n",
       "      <td>2.90</td>\n",
       "      <td>1320.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>13.83</td>\n",
       "      <td>1.65</td>\n",
       "      <td>2.60</td>\n",
       "      <td>17.2</td>\n",
       "      <td>94.0</td>\n",
       "      <td>2.45</td>\n",
       "      <td>2.99</td>\n",
       "      <td>0.22</td>\n",
       "      <td>2.29</td>\n",
       "      <td>5.60</td>\n",
       "      <td>1.240</td>\n",
       "      <td>3.37</td>\n",
       "      <td>1265.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>62</th>\n",
       "      <td>13.67</td>\n",
       "      <td>1.25</td>\n",
       "      <td>1.92</td>\n",
       "      <td>18.0</td>\n",
       "      <td>94.0</td>\n",
       "      <td>2.10</td>\n",
       "      <td>1.79</td>\n",
       "      <td>0.32</td>\n",
       "      <td>0.73</td>\n",
       "      <td>3.80</td>\n",
       "      <td>1.230</td>\n",
       "      <td>2.46</td>\n",
       "      <td>630.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>70</th>\n",
       "      <td>12.29</td>\n",
       "      <td>1.61</td>\n",
       "      <td>2.21</td>\n",
       "      <td>20.4</td>\n",
       "      <td>103.0</td>\n",
       "      <td>1.10</td>\n",
       "      <td>1.02</td>\n",
       "      <td>0.37</td>\n",
       "      <td>1.46</td>\n",
       "      <td>3.05</td>\n",
       "      <td>0.906</td>\n",
       "      <td>1.82</td>\n",
       "      <td>870.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>71</th>\n",
       "      <td>13.86</td>\n",
       "      <td>1.51</td>\n",
       "      <td>2.67</td>\n",
       "      <td>25.0</td>\n",
       "      <td>86.0</td>\n",
       "      <td>2.95</td>\n",
       "      <td>2.86</td>\n",
       "      <td>0.21</td>\n",
       "      <td>1.87</td>\n",
       "      <td>3.38</td>\n",
       "      <td>1.360</td>\n",
       "      <td>3.16</td>\n",
       "      <td>410.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>13.05</td>\n",
       "      <td>1.65</td>\n",
       "      <td>2.55</td>\n",
       "      <td>18.0</td>\n",
       "      <td>98.0</td>\n",
       "      <td>2.45</td>\n",
       "      <td>2.43</td>\n",
       "      <td>0.29</td>\n",
       "      <td>1.44</td>\n",
       "      <td>4.25</td>\n",
       "      <td>1.120</td>\n",
       "      <td>2.51</td>\n",
       "      <td>1105.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>133 rows × 13 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     alcohol  malic_acid   ash  alcalinity_of_ash  magnesium  total_phenols  \\\n",
       "109    11.61        1.35  2.70               20.0       94.0           2.74   \n",
       "106    12.25        1.73  2.12               19.0       80.0           1.65   \n",
       "134    12.51        1.24  2.25               17.5       85.0           2.00   \n",
       "89     12.08        1.33  2.30               23.6       70.0           2.20   \n",
       "12     13.75        1.73  2.41               16.0       89.0           2.60   \n",
       "..       ...         ...   ...                ...        ...            ...   \n",
       "51     13.83        1.65  2.60               17.2       94.0           2.45   \n",
       "62     13.67        1.25  1.92               18.0       94.0           2.10   \n",
       "70     12.29        1.61  2.21               20.4      103.0           1.10   \n",
       "71     13.86        1.51  2.67               25.0       86.0           2.95   \n",
       "37     13.05        1.65  2.55               18.0       98.0           2.45   \n",
       "\n",
       "     flavanoids  nonflavanoid_phenols  proanthocyanins  color_intensity  \\\n",
       "109        2.92                  0.29             2.49             2.65   \n",
       "106        2.03                  0.37             1.63             3.40   \n",
       "134        0.58                  0.60             1.25             5.45   \n",
       "89         1.59                  0.42             1.38             1.74   \n",
       "12         2.76                  0.29             1.81             5.60   \n",
       "..          ...                   ...              ...              ...   \n",
       "51         2.99                  0.22             2.29             5.60   \n",
       "62         1.79                  0.32             0.73             3.80   \n",
       "70         1.02                  0.37             1.46             3.05   \n",
       "71         2.86                  0.21             1.87             3.38   \n",
       "37         2.43                  0.29             1.44             4.25   \n",
       "\n",
       "       hue  od280/od315_of_diluted_wines  proline  \n",
       "109  0.960                          3.26    680.0  \n",
       "106  1.000                          3.17    510.0  \n",
       "134  0.750                          1.51    650.0  \n",
       "89   1.070                          3.21    625.0  \n",
       "12   1.150                          2.90   1320.0  \n",
       "..     ...                           ...      ...  \n",
       "51   1.240                          3.37   1265.0  \n",
       "62   1.230                          2.46    630.0  \n",
       "70   0.906                          1.82    870.0  \n",
       "71   1.360                          3.16    410.0  \n",
       "37   1.120                          2.51   1105.0  \n",
       "\n",
       "[133 rows x 13 columns]"
      ]
     },
     "execution_count": 191,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 1, 2, 1, 0, 2, 1, 1, 0, 0, 2, 1, 1, 1, 0, 0, 0, 2, 2, 2, 1, 2,\n",
       "       0, 2, 2, 1, 0, 2, 1, 0, 1, 2, 0, 2, 0, 2, 1, 1, 2, 0, 2, 2, 0, 0,\n",
       "       0, 0, 1, 1, 0, 0, 1, 2, 2, 1, 2, 1, 2, 0, 0, 1, 1, 2, 2, 1, 0, 1,\n",
       "       1, 0, 2, 0, 2, 1, 0, 1, 1, 1, 2, 1, 1, 0, 0, 1, 0, 1, 0, 1, 0, 0,\n",
       "       2, 2, 0, 2, 1, 1, 2, 1, 2, 2, 0, 0, 0, 2, 0, 0, 1, 0, 0, 1, 1, 0,\n",
       "       0, 0, 1, 1, 2, 0, 2, 1, 1, 0, 1, 1, 2, 1, 1, 1, 2, 1, 0, 1, 1, 1,\n",
       "       0])"
      ]
     },
     "execution_count": 192,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Note that we have 3 classes here: 0, 1, 2.\n",
    "- For some models, like a random forest, we don't have to think about anything differently at all:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 193,
   "metadata": {},
   "outputs": [],
   "source": [
    "rf = RandomForestClassifier()\n",
    "rf.fit(X_train, y_train);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 194,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0, 0, 1, 2, 1, 2, 0, 0, 0, 1, 1, 2, 0, 1, 1, 1, 0, 2, 2, 1, 0,\n",
       "       0, 2, 1, 1, 0, 1, 1, 2, 2, 1, 2, 0, 1, 2, 0, 2, 1, 1, 1, 0, 1, 2,\n",
       "       1])"
      ]
     },
     "execution_count": 194,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 195,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Importance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ash</th>\n",
       "      <td>0.011616</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>nonflavanoid_phenols</th>\n",
       "      <td>0.012012</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>proanthocyanins</th>\n",
       "      <td>0.012043</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>alcalinity_of_ash</th>\n",
       "      <td>0.016882</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>magnesium</th>\n",
       "      <td>0.029298</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>malic_acid</th>\n",
       "      <td>0.045506</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>total_phenols</th>\n",
       "      <td>0.045682</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hue</th>\n",
       "      <td>0.083974</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>color_intensity</th>\n",
       "      <td>0.107607</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>od280/od315_of_diluted_wines</th>\n",
       "      <td>0.119953</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>alcohol</th>\n",
       "      <td>0.151987</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>proline</th>\n",
       "      <td>0.161745</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>flavanoids</th>\n",
       "      <td>0.201696</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                              Importance\n",
       "ash                             0.011616\n",
       "nonflavanoid_phenols            0.012012\n",
       "proanthocyanins                 0.012043\n",
       "alcalinity_of_ash               0.016882\n",
       "magnesium                       0.029298\n",
       "malic_acid                      0.045506\n",
       "total_phenols                   0.045682\n",
       "hue                             0.083974\n",
       "color_intensity                 0.107607\n",
       "od280/od315_of_diluted_wines    0.119953\n",
       "alcohol                         0.151987\n",
       "proline                         0.161745\n",
       "flavanoids                      0.201696"
      ]
     },
     "execution_count": 195,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf_importances = pd.DataFrame(data=rf.feature_importances_, index=X_train.columns, columns=[\"Importance\"])\n",
    "rf_importances.sort_values(by=\"Importance\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Everything is the same here.\n",
    "- For our linear classifier, on the other hand, things are a bit different:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 196,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = LogisticRegression(max_iter=10000)\n",
    "lr.fit(X_train, y_train);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 197,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0, 0, 1, 2, 1, 2, 0, 0, 0, 1, 1, 2, 0, 1, 1, 1, 0, 2, 2, 1, 0,\n",
       "       0, 2, 1, 1, 0, 1, 1, 2, 2, 1, 2, 0, 1, 2, 0, 2, 1, 1, 1, 0, 1, 2,\n",
       "       1])"
      ]
     },
     "execution_count": 197,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 204,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[9.83121095e-01, 1.30404370e-02, 3.83846763e-03],\n",
       "       [9.95736967e-01, 1.69179383e-03, 2.57123874e-03],\n",
       "       [9.99913071e-01, 4.17255306e-06, 8.27563492e-05],\n",
       "       [2.21058877e-03, 9.97237440e-01, 5.51971365e-04],\n",
       "       [1.55535649e-03, 1.51962649e-03, 9.96925017e-01],\n",
       "       [1.40295571e-04, 9.99655016e-01, 2.04688647e-04],\n",
       "       [1.77145916e-03, 1.22119079e-02, 9.86016633e-01],\n",
       "       [9.99998530e-01, 1.01172179e-10, 1.46997299e-06],\n",
       "       [9.99957179e-01, 1.40043113e-05, 2.88168294e-05],\n",
       "       [9.96393323e-01, 2.52352829e-04, 3.35432433e-03]])"
      ]
     },
     "execution_count": 204,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr.predict_proba(X_test)[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 205,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0, 0, 1, 2, 1, 2, 0, 0, 0, 1, 1, 2, 0, 1, 1, 1, 0, 2, 2, 1, 0,\n",
       "       0, 2, 1, 1, 0, 1, 1, 2, 2, 1, 2, 0, 1, 2, 0, 2, 1, 1, 1, 0, 1, 2,\n",
       "       1])"
      ]
     },
     "execution_count": 205,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.argmax(lr.predict_proba(X_test), axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 206,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0, 0, 1, 2, 1, 2, 0, 0, 0, 1, 1, 2, 0, 1, 1, 1, 0, 2, 2, 1, 0,\n",
       "       0, 2, 1, 1, 0, 1, 1, 2, 2, 1, 2, 0, 1, 2, 0, 2, 1, 1, 1, 0, 1, 2,\n",
       "       1])"
      ]
     },
     "execution_count": 206,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 198,
   "metadata": {},
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "Shape of passed values is (3, 13), indices imply (13, 1)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/pandas/core/internals/managers.py\u001b[0m in \u001b[0;36mcreate_block_manager_from_blocks\u001b[0;34m(blocks, axes)\u001b[0m\n\u001b[1;32m   1653\u001b[0m                 blocks = [\n\u001b[0;32m-> 1654\u001b[0;31m                     \u001b[0mmake_block\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mblocks\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mplacement\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mslice\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0maxes\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1655\u001b[0m                 ]\n",
      "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/pandas/core/internals/blocks.py\u001b[0m in \u001b[0;36mmake_block\u001b[0;34m(values, placement, klass, ndim, dtype)\u001b[0m\n\u001b[1;32m   3040\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 3041\u001b[0;31m     \u001b[0;32mreturn\u001b[0m \u001b[0mklass\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mndim\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mndim\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mplacement\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mplacement\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   3042\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/pandas/core/internals/blocks.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, values, placement, ndim)\u001b[0m\n\u001b[1;32m    124\u001b[0m             raise ValueError(\n\u001b[0;32m--> 125\u001b[0;31m                 \u001b[0;34mf\"Wrong number of items passed {len(self.values)}, \"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    126\u001b[0m                 \u001b[0;34mf\"placement implies {len(self.mgr_locs)}\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mValueError\u001b[0m: Wrong number of items passed 13, placement implies 1",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-198-123942b7d40a>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mlr_coefs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mDataFrame\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mlr\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcoef_\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mindex\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mX_train\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcolumns\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcolumns\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"Coefficient\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0mlr_coefs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msort_values\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mby\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"Coefficient\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/pandas/core/frame.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, data, index, columns, dtype, copy)\u001b[0m\n\u001b[1;32m    462\u001b[0m                 \u001b[0mmgr\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0minit_dict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m{\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mdata\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mindex\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcolumns\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdtype\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    463\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 464\u001b[0;31m                 \u001b[0mmgr\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0minit_ndarray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mindex\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcolumns\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdtype\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcopy\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcopy\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    465\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    466\u001b[0m         \u001b[0;31m# For data is list-like, or Iterable (will consume into list)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/pandas/core/internals/construction.py\u001b[0m in \u001b[0;36minit_ndarray\u001b[0;34m(values, index, columns, dtype, copy)\u001b[0m\n\u001b[1;32m    208\u001b[0m         \u001b[0mblock_values\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    209\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 210\u001b[0;31m     \u001b[0;32mreturn\u001b[0m \u001b[0mcreate_block_manager_from_blocks\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mblock_values\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mcolumns\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mindex\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    211\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    212\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/pandas/core/internals/managers.py\u001b[0m in \u001b[0;36mcreate_block_manager_from_blocks\u001b[0;34m(blocks, axes)\u001b[0m\n\u001b[1;32m   1662\u001b[0m         \u001b[0mblocks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mgetattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mb\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"values\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mb\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mb\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mblocks\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1663\u001b[0m         \u001b[0mtot_items\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msum\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mb\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mb\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mblocks\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1664\u001b[0;31m         \u001b[0mconstruction_error\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtot_items\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mblocks\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0maxes\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1665\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1666\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/pandas/core/internals/managers.py\u001b[0m in \u001b[0;36mconstruction_error\u001b[0;34m(tot_items, block_shape, axes, e)\u001b[0m\n\u001b[1;32m   1692\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mblock_shape\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1693\u001b[0m         \u001b[0;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Empty data passed with indices specified.\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1694\u001b[0;31m     \u001b[0;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"Shape of passed values is {passed}, indices imply {implied}\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1695\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1696\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mValueError\u001b[0m: Shape of passed values is (3, 13), indices imply (13, 1)"
     ]
    }
   ],
   "source": [
    "lr_coefs = pd.DataFrame(data=lr.coef_, index=X_train.columns, columns=[\"Coefficient\"])\n",
    "lr_coefs.sort_values(by=\"Coefficient\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is going on here?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 199,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3, 13)"
      ]
     },
     "execution_count": 199,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr.coef_.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What's happening here is that we have one coefficient per feature _per class_. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 200,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>alcohol</th>\n",
       "      <td>0.660312</td>\n",
       "      <td>-0.691072</td>\n",
       "      <td>0.030761</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>malic_acid</th>\n",
       "      <td>0.540192</td>\n",
       "      <td>-0.791026</td>\n",
       "      <td>0.250834</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ash</th>\n",
       "      <td>0.708200</td>\n",
       "      <td>-0.874055</td>\n",
       "      <td>0.165855</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>alcalinity_of_ash</th>\n",
       "      <td>-0.211434</td>\n",
       "      <td>0.122472</td>\n",
       "      <td>0.088962</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>magnesium</th>\n",
       "      <td>-0.019100</td>\n",
       "      <td>-0.016223</td>\n",
       "      <td>0.035323</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>total_phenols</th>\n",
       "      <td>0.247848</td>\n",
       "      <td>0.154682</td>\n",
       "      <td>-0.402530</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>flavanoids</th>\n",
       "      <td>0.804320</td>\n",
       "      <td>0.425621</td>\n",
       "      <td>-1.229941</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>nonflavanoid_phenols</th>\n",
       "      <td>0.087637</td>\n",
       "      <td>0.024024</td>\n",
       "      <td>-0.111661</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>proanthocyanins</th>\n",
       "      <td>0.076671</td>\n",
       "      <td>0.395400</td>\n",
       "      <td>-0.472071</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>color_intensity</th>\n",
       "      <td>0.186333</td>\n",
       "      <td>-0.948383</td>\n",
       "      <td>0.762051</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hue</th>\n",
       "      <td>0.003049</td>\n",
       "      <td>0.322720</td>\n",
       "      <td>-0.325769</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>od280/od315_of_diluted_wines</th>\n",
       "      <td>0.657446</td>\n",
       "      <td>-0.001666</td>\n",
       "      <td>-0.655780</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>proline</th>\n",
       "      <td>0.008648</td>\n",
       "      <td>-0.008233</td>\n",
       "      <td>-0.000416</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     0         1         2\n",
       "alcohol                       0.660312 -0.691072  0.030761\n",
       "malic_acid                    0.540192 -0.791026  0.250834\n",
       "ash                           0.708200 -0.874055  0.165855\n",
       "alcalinity_of_ash            -0.211434  0.122472  0.088962\n",
       "magnesium                    -0.019100 -0.016223  0.035323\n",
       "total_phenols                 0.247848  0.154682 -0.402530\n",
       "flavanoids                    0.804320  0.425621 -1.229941\n",
       "nonflavanoid_phenols          0.087637  0.024024 -0.111661\n",
       "proanthocyanins               0.076671  0.395400 -0.472071\n",
       "color_intensity               0.186333 -0.948383  0.762051\n",
       "hue                           0.003049  0.322720 -0.325769\n",
       "od280/od315_of_diluted_wines  0.657446 -0.001666 -0.655780\n",
       "proline                       0.008648 -0.008233 -0.000416"
      ]
     },
     "execution_count": 200,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_coefs = pd.DataFrame(data=lr.coef_.T, index=X_train.columns, columns=lr.classes_)\n",
    "lr_coefs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 202,
   "metadata": {},
   "outputs": [],
   "source": [
    "# lr_coefs.sum(axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The interpretation is a feature importance for predicting a certain class. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr_coefs.loc[\"alcohol\", 2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Means that if alcohol is larger, you are more likely to predict class 2. \n",
    "- The specific interpretation depends on the way the logistic regression is implementing multi-class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 208,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ?LogisticRegression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- We can see that there's a `multi_class` parameter, that can be set to `'ovr'` or `'multinomial'`, or you can have it automatically choose between the two, which is the default.\n",
    "  - In CPSC 340 we discuss in detail the difference between these two approaches.\n",
    "  - In CPSC 340 we make an argument for preferring `'multinomial'`, but in short it doesn't matter which one you choose.\n",
    "- If you want a general feature importance irrespective of class, you could try looking for features where the coefficients have large absolute values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 211,
   "metadata": {},
   "outputs": [],
   "source": [
    "# lr_multi = LogisticRegression(max_iter=10000, multi_class='multinomial')\n",
    "# lr_multi.fit(X_train, y_train);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 212,
   "metadata": {},
   "outputs": [],
   "source": [
    "# lr_multi_coefs = pd.DataFrame(data=lr_multi.coef_.T, index=X_train.columns, columns=lr.classes_)\n",
    "# lr_multi_coefs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the output of `predict_proba` now contains a probability for each class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 213,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[9.83121095e-01, 1.30404370e-02, 3.83846763e-03],\n",
       "       [9.95736967e-01, 1.69179383e-03, 2.57123874e-03],\n",
       "       [9.99913071e-01, 4.17255306e-06, 8.27563492e-05],\n",
       "       [2.21058877e-03, 9.97237440e-01, 5.51971365e-04],\n",
       "       [1.55535649e-03, 1.51962649e-03, 9.96925017e-01]])"
      ]
     },
     "execution_count": 213,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr.predict_proba(X_test)[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And you'll see that each row adds up to 1, as expected:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 214,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.])"
      ]
     },
     "execution_count": 214,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.sum(lr.predict_proba(X_test), axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can still make a confusion matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 215,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[14,  0,  0],\n",
       "       [ 0, 19,  0],\n",
       "       [ 0,  0, 12]])"
      ]
     },
     "execution_count": 215,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "confusion_matrix(y_test, lr.predict(X_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 216,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZoAAAFaCAYAAAAn7pvXAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deVQT994G8GfYZImKCloVQdwQxbpda2uLVi21bnWXooLW5fZeNdYNq70WRVvBBXtegyvX9rWIgtdTFa/2ttViba2Xqi1WcXm1Ki7ggnsCsiTz/mFNiQKRwCTD5Pn05BwyM8l8wzQ8fmd++UUQRVEEERGRRBxsXQARESkbg4aIiCTFoCEiIkkxaIiISFIMGiIikpSTrQsoTcvIr2xdQpXYM+s19I/70dZlVNqJmL62LqFKuDgChXpbV0GAco6Fq4R/Qd06TrX4sfm/xldhJZXHjkZCrV6oaesSqAQHwdYV0BM8FvZFlh0NEZHdE5TTBzBoiIjkSFBO28egISKSI3Y0REQkKQV1NMqJTCIikiV2NEREcsRTZ0REJCkFnTpj0BARyRE7GiIikhQ7GiIikhQ7GiIikpSCOhrlRCYREckSOxoiIjniqTMiIpKUgk6dMWiIiOSIHQ0REUmKQUNERJJS0LfDKScyiYhIltjREBHJEU+dERGRpKw46uz48eNYsWIFEhMTMWPGDOTm5gIArl27hvbt2+PTTz812X7w4MGoWbMmAMDHxwcxMTHlPj+DhohIjqzU0SQkJCA1NRVubm4AYAyV+/fvIyIiAvPmzTPZvqCgAACQmJj43PtQTm9GRKQkgmD5rQJ8fX2h0WieWa7RaDBmzBjUr1/fZPmZM2eQn5+P8ePHIyIiAhkZGWb3waAhIpIjwcHim0ajQUBAgPFWWpA80adPHzg5mZ7cun37Ng4fPoyhQ4c+s72rqysmTJiAjRs3Ijo6GrNnz0ZxcXG5L4WnzoiI5KgS12jUajXUarXFj//Pf/6DAQMGwNHR8Zl1/v7+8PPzgyAI8Pf3h6enJ27duoWGDRuW+XzsaIiIyMThw4fRvXv3Utdt374dsbGxAIAbN25Aq9XC29u73Odj0BARyVElTp1V1sWLF9GkSROTZXPmzEF2djaGDx+Ohw8fIiwsDDNmzMCSJUueOfX2NJ46IyKSIysOb/bx8cG2bduM9/fs2fPMNsuWLTP+HBcXV6HnZ9AQEckRP7BJRESS4tcEEBGRpNjREBGRpBQUNMp5JUREJEvsaIiI5IjXaIiISFIKOnXGoCEikiN2NEREJCl2NEREJCl2NEREJCVBQUGjnN6MiIhkiR0NEZEMKamjYdAQEcmRcnKGQUNEJEdK6mh4jaYKvR5YH/9dGFLmek93Z3z/j974e+8WVqyKNu04hM5Do1Hn5Rl4c/wK/PzbBVuXZNc27TiEoLej0fA1Ho/yCIJg8U1uGDRVpL2vJ2JGti+32503sA3qqlysVhMByXvSMTM2GSP7dsHW5RNRS+WO4dPWIOtarq1Ls0tPjkdY/y7YtJTHozwMGjJydnTAu9398dmkrtAbDGVu16N1fbzS0guPivRWrM6+iaKIJev3YOyQV/HBpH54K7gttq58D3Vre2DN1jRbl2d3Sh6Pf7zXD2++yuNRHgYNGQUHeGHi680R99UZbDmcVeo2qhpO+GhwW6zYewaFxWWHEVWtC1du4UrOHfQNbmdc5uzkiDdfa4v9h0/ZsDL7xONhvyQLGoPBgKioKISGhiI8PBxZWaX/Ea7uTl69j7eWHcCWn7IgiqVvM6tfa/x+U4vUX65Ztzg7d/7yTQBAsybeJsubNvbCxau50OsZ+tbE41FBQiVuMiNZ0Ozbtw+FhYVISUnBrFmzEBsbK9WubOrmgwI8fFRc5vqXmtVFv/YNsWjHSStWRQDwUPcIAKBydzVZrnJ3hcEgQpdfYIuy7BaPR8Uo6dSZZMObjx07huDgYABAhw4dcPJk+X9oNRoN4uPjAQCjx03CwnmzpSpNMkV6QC8C55b3BQCIIvDZpK5wdAAOfPg6AOBRMaAOaYmZb7W0YaX2wcnhcYvp6gy4/vF/uqvTn8vdXRyMy0l6JY8H8Ocx4fEonRwDw1KSHVatVguVSmW87+joiOLiYjg5lb5LtVoNtVoNAGgZ+RWSIr+SqjTJ/L13C4wL9ke7ed8CAE7F9kXmtfsYs/a/MPxxXu3g/N7Y8lMW1n13HnpDGefaZOZETF9bl2ARNzc3AMDtBwWoXfvxH7ZHxcA9bQEcHAQ4OrugnGaUqljJ49GgHoy/++p8PKQMRgbNc1CpVNDpdMb7BoOhzJBRKr0ItGlcG7983Mdk+d96t8DferdAu3nVL0yrk+Z/XAu4dC3X5LrApWu5aOnXQFFv5Oqg5PFo48/jYY6Sfh+S/eXv1KkT0tLS0K9fP2RkZKBVq1ZS7Uq2XByBIf9zyGTZZ5O64qvj2fjXz1dsVJX9aO5bH40b1MGeA7+h18uBAICiYj2++TETb77W1sbV2Z+Sx6PfazweZiknZ6QLmpCQEBw6dAjvvPPO4/HzS5ZItSvZchCAU9cemCzTG0TcfFDwzHKqeoIgYPrYEMxZ/i941nJDcMfmWJ38Pe7c12FyWC9bl2d3Sh4PL083dG7XHAnbeDzsgWRB4+DggEWLFkn19ETPZeKI7nhUUIR1yQewdmsaglr6YPuqyWjq42Xr0uzSk+OxPuUANJvTENSKx6MsSjp1JohiWZ/+sJ2W1XAgQGnOLe+riNdSXQcDPO3JYACyPaUcCykHA3i/m2LxY299HlqFlVSefV2dJyKqJpTU0TBoiIjkSDk5w6AhIpIjdjRERCQpJQUNZ28mIrJzx48fR3h4OAAgMzMTwcHBCA8PR3h4OPbu3WuyrSUTJrOjISKSIWt1NAkJCUhNTTVOEXTq1Cm8++67GD9+fKnbl5wwOSMjA7GxsVi7dm25+2BHQ0QkQ5WZvVmj0SAgIMB402g0Ze7H19fXZP3Jkydx4MABjB49Gh9++CG0Wq3J9hWdMBlg0BARyVMlvo9GrVbj7NmzxtuTCYtL06dPH5N5KF988UXMmTMHSUlJaNKkCVavXm2yfVkTJpeHQUNEJEO2+j6akJAQBAUFGX8+dcr0208tmTCZQUNEJEO2CpoJEybgt99+AwAcPnwYbduaTnjaqVMnHDx4EACee8JkDgYgIpIhWw1vXrhwIRYvXgxnZ2d4eXlh8eLFAIA5c+Zg+vTpFk2YzLnOJMS5zuRFKfNrKYFSjoWUc501mbLL4sdeWT2oCiupPHY0RERypJzPazJoiIjkSEkzAzBoiIhkiEFDRESSYtAQEZGkGDRERCQt5eQMP7BJRETSYkdDRCRDPHVGRESSYtAQEZGkFJQzDBoiIjliR0NERJJSUM4waIiI5EhJHQ2HNxMRkaTY0RARyZCCGhoGDRGRHDk4KCdpGDRERDLEjoaIiCSlpMEADBoiIhlSUM4waIiI5EhJHQ2HNxMRkaTY0RARyZCSOhoGDRGRDCkoZxg0RERyxI6GiIgkpaCcYdAQEckROxoiIpKUgnKGw5uJiEha7GiIiGSIp86IiEhSCsoZeQbNiZi+ti6hyijhtdTpMtXWJVSJ/F/jq/1ruXsk3tYlkJWwoyEiIklZM2eOHz+OFStWIDExEadPn8bixYvh6OgIFxcXLF26FF5eXibbDx48GDVr1gQA+Pj4ICYmptznZ9AQEcmQtTqahIQEpKamws3NDQDwySef4KOPPkJgYCCSk5ORkJCAefPmGbcvKCgAACQmJj73PjjqjIhIhgTB8ltF+Pr6QqPRGO+vXLkSgYGBAAC9Xo8aNWqYbH/mzBnk5+dj/PjxiIiIQEZGhtl9MGiIiBRGo9EgICDAeCsZJE/r06cPnJz+PLlVv359AMAvv/yCzZs3Y9y4cSbbu7q6YsKECdi4cSOio6Mxe/ZsFBcXl1sPT50REclQZU6dqdVqqNVqix+/d+9erF27Fhs2bEDdunVN1vn7+8PPzw+CIMDf3x+enp64desWGjZsWObzsaMhIpIha506e9quXbuwefNmJCYmokmTJs+s3759O2JjYwEAN27cgFarhbe3d7nPyaAhIpIhQRAsvllKr9fjk08+gU6ng1qtRnh4OFatWgUAmDNnDrKzszF8+HA8fPgQYWFhmDFjBpYsWWJy6q00PHVGRCRD1vwcjY+PD7Zt2wYA+Pnnn0vdZtmyZcaf4+LiKvT8DBoiIhlS0Oc1GTRERHKkpJkBeI2GiIgkxY6GiEiGFNTQMGiIiORISafOGDRERDKkoJxh0BARyZGDgpKGQUNEJEMKyhkGDRGRHCnpGg2HNxMRkaTY0RARyZCDchoaBg0RkRwp6dQZg4aISIYUlDMMGiIiORKgnKRh0BARyRCv0RARkaSUdI2Gw5uJiEhS7GiIiGRIQQ0Ng4aISI7sYq6z+Pj4ch84derUKi+GiIgeU1DOsKMhIpIjJQ0GKDNoSnYseXl5uHz5Mlq1aoVHjx7B3d3dKsUREdkrBeWM+VFnhw8fxqBBgzB58mTcvn0bPXv2xI8//miN2oiI7JaDIFh8kxuzQbNy5Ups2bIFtWrVgre3N5KSkrBs2TJr1EZERApg9hqNwWCAt7e38X6LFi0kLYiIiKCgCWieI2heeOEFpKWlQRAEPHjwAElJSWjUqJE1aiMisltKGgxg9tTZokWLsHv3buTk5OCNN97A6dOnsWjRImvURkRktxwEy29yY7ajqVevHlauXAmtVgtHR0e4ublZoy4iIrumpI7GbNCcPXsWc+fORXZ2NgCgWbNmWLp0KXx9fSUvjojIXikoZ8yfOluwYAGmT5+O9PR0pKenY/z48fjwww+tURsRkd0SBMHim9yYDZqCggL06NHDeD8kJARarVbSooiISDnKDJrs7GxkZ2ejdevW2LBhA+7cuYP79+9j8+bN+Mtf/mLNGomI7I41BwMcP34c4eHhAICsrCyEhYVh1KhRWLBgAQwGg8m2BoMBUVFRCA0NRXh4OLKyssw+f5nXaMaMGQNBECCKItLT05GcnGxcJwgC5s+fX/FXQ0REz8Vap8ASEhKQmppqHOgVExOD6dOno2vXroiKisL+/fsREhJi3H7fvn0oLCxESkoKMjIyEBsbi7Vr15a7jzKD5rvvvquil0FERBVlrSstvr6+0Gg0mDNnDgAgMzMTL730EgCge/fuOHTokEnQHDt2DMHBwQCADh064OTJk2b3YXbU2aVLl7B582bk5eVBFEUYDAZcvXoVSUlJFr0oIiIyrzJzlmk0GpOvepk6dSrUanWp2/bp0wdXr1413hdF0dhNeXh44OHDhybba7VaqFQq431HR0cUFxfDyansODE7GGDmzJmoVasWTp8+jcDAQGRnZ6Nly5bmHmbXNu04hM5Do1Hn5Rl4c/wK/PzbBVuXZJf6dm+HywdWmCzzqqPC+kURuLh/GS7uX4b/jZ2AJg3r2qhC+7RpxyEEvR2Nhq/x/VEeQbD8plarcfbsWeOtrJApjYPDn7Gg0+lQq1Ytk/UqlQo6nc5432AwlBsywHMETVFREaZNm4bg4GC0adMGCQkJOHLkyHMXbW+S96RjZmwyRvbtgq3LJ6KWyh3Dp61B1rVcW5dmV1560R/roiNMznOLIrBrzTT0ejkQC+N3YeL8z+Hu6oL//HMm6tT2sGG19uPJ+yOsfxdsWsr3R3lsNby5TZs2SE9PBwAcPHjwmcFfnTp1wsGDBwEAGRkZaNWqldnnNBs0bm5uKCwsRNOmTZGZmQlXV1dLarcLoihiyfo9GDvkVXwwqR/eCm6LrSvfQ93aHlizNc3W5dkFF2cnTAt/A6lrp0Gvf2q0jAi0adEI70V9gU07DmH/4dMYE5mA4mI9pke8YaOK7UfJ98c/3uuHN1/l+0OOPvjgA2g0GoSGhqKoqAh9+vQBAMyZMwfZ2dkICQmBi4sL3nnnHcTExGDevHlmn9PsNZq3334bf/vb37BixQqEhobihx9+QIMGDSr/ahTowpVbuJJzB32D2xmXOTs54s3X2mL/4VM2rMx+vNGtDWaMexNRq3aibm0PTBnd27hOBFBcrMf3R84alxUWFeOXU1no/UobLNDsskHF9oPvj4qx5ucufXx8sG3bNgCAv78/Nm/e/Mw2Jb8epqLzXZrtaMaMGYNVq1ahbt26SExMRGhoKFavXv1cT15ybLY9OH/5JgCgWRNvk+VNG3vh4tXcZ/6FTVXv11NZaD9oATakfA9RNF0nAHByckRDr9omy/0aefE6jRXw/VExSvriszI7mpIjFp529uxZk696Ls3TY7PtwUPdIwCAyt309KLK3RUGgwhdfgFqqezn92ELObful7nOQQBy7z7EukURmBmTjFt3tfjryB4IbN4Qzk6OVqzSPvH9UTEyzAuLme1oLPVkbLY9Ef/4J/TT/4M8WV5yNAdZnyAA4ZEJ8GlQB+n/+ggX9i1Fh8Am+GLnT8h7VGjr8hSP74+KUdJcZ2V2NOY6FnOeHpttTslx31OmTMW0ac8/HE8uvGo//tdYUUEBXP/4zbo6PZ4vzsFBQL2aLtXyXyn5v5bd3cpZkR7Qi6b1p/3vTIji4+s1AoDBPduVuh1VvZLvDwDG90h1f39IRUmxa3YwgLWo1WrjWO9HxY9v1U2TRo/PPZ+9nIvGjbzh6vT4dZy/kouWfg1QoK+e76I6XSr3jw5b+WBSP0wd0xt1eswCAOT9Eo8JUYlI3f8rdPl/djBpX8xB9s17GD17g61KfW53j1TfMCz5/mju6218j1fn94erhH9B5diZWEpJoWlzzX3ro3GDOthz4DfjsqJiPb75MRPdu5gfa07SW7MgHD1fDjTe79LOHx0CffGfH07YsCr7wPeH/XquPM7Ly8Ply5cREBCA/Px8uLu7S11XtSQIAqaPDcGc5f+CZy03BHdsjtXJ3+POfR0mh/WydXl2TxCAvd//ho/fHwJRFOHi7IQlM4fhxP9dRfKen21dnuKVfH94ebqhc7vmSNjG90dZ5PiVzJYyGzSHDx9GVFQU9Ho9UlJSMGDAAMTFxeG1114z++Qlx2bbi4kjuuNRQRHWJR/A2q1pCGrpg+2rJqOpj5etSyMAUxZtRszMYVj1j9EwiAZ8/WMmFqzaiaJiva1LswtP3h/rUw5AszkNQa34/iiLkoJGEMWnP21gasSIEVizZg0mTZqEnTt34vz585g5cyZSU1MlK6o6Xp8pzZNrNNVddb1G87T8X+Ph1rF6v5bqfI2mJKW8N6S8RjNr91nzG5UhbmBAFVZSeWZ/TQaDAd7ef37AqkWLFpIWREREyupozAbNCy+8gLS0NAiCgAcPHiApKQmNGjWyRm1ERHZLQYPOzI86W7RoEXbv3o2cnBy88cYbOH36dIXnuSEiooqxiylonqhXrx5WrlxpjVqIiEiBzAZNr169Sv3g0P79+yUpiIiIlPUhR7NBk5iYaPy5uLgY3377LQoLOS8UEZGUZHgGzGJmQ7Nx48bGm5+fHyZOnIh9+/ZZozYiIrtlV9doSn5tsyiKOHfuHAr+mBSPiIikIcO8sJjZoFm1apXxZ0EQUKdOHcTGxkpaFBGRvbOrz9H069cPYWFh1qiFiIj+IMdTYJYye40mKSnJGnUQEZFCPdfMABEREWjfvj1q1KhhXF7ZL0YjIqKyKaihMR80HTp0sEYdRERUgl1co9mxYweGDBnCzoWIyAYEKCdpyrxG88UXX1izDiIiKsFBsPwmNxJ+mwIREVlKjoFhqTKD5ty5c+jdu/czy0VRhCAInOuMiEhCpc0xWV2VGTR+fn7YsGGDNWshIiIFKjNonJ2d0bhxY2vWQkREf7CLU2edOnWyZh1ERFSCgs6clR00UVFR1qyDiIhKUNIUNBx1RkQkQ3Zx6oyIiGxHQQ0Ng4aIyJ59+eWX2LFjBwCgoKAAp0+fxqFDh1CrVi0AwOeff47t27ejbt26AIDo6Gg0a9asQvtg0BARyZCDlaagGTp0KIYOHQrgcYgMGzbMGDIAkJmZiaVLlyIoKMjifZj9mgAiIrI+QbD8ptFoEBAQYLxpNBqz+ztx4gTOnz+P0NBQk+WZmZnYsGEDwsLCsH79eoteCzsaIiIZqsxggMlqNdRqdYUes379ekyZMuWZ5f3798eoUaOgUqkwdepUpKWloWfPnhV6bnY0REQy5CAIFt8q6sGDB7hw4QJefvllk+WiKGLs2LGoW7cuXFxc0KNHD5w6darir6XCjyAiIslV5tRZRR05cgTdunV7ZrlWq8WAAQOg0+kgiiLS09MtulbDU2dERDJkzQ9sXrx4ET4+Psb7u3fvRl5eHkJDQzFjxgxERETAxcUFr7zyCnr06FHh52fQEBHZuYkTJ5rcHzhwoPHnwYMHY/DgwZV6fgYNEZEM8QObREQkKSVdQGfQEBHJkF188RkREdmOcmKGQUNEJEv8mgAiIpKUcmJGWdebiIhIhtjREBHJkILOnDFoiIjkiKPOiIhIUkq6rsGgISKSIXY0REQkKeXEDIOGnsPdI/G2LqHKVPfX0nf1T7YuoUqkvd9NEa8l7f1np9avKkrqaJR0GpCIiGSIHQ0RkQwpqQtg0BARyZCSTp0xaIiIZEg5McOgISKSJQU1NAwaIiI5clBQT8OgISKSISV1NEoa2EBERDLEjoaISIYEnjojIiIpKenUGYOGiEiGOBiAiIgkxY6GiIgkxaAhIiJJKWkwAIc3ExGRpNjREBHJkINyGhoGDRGRHCnp1BmDhohIhjgYgIiIJGXNjmbw4MGoWbMmAMDHxwcxMTHGdd999x1Wr14NJycnDBs2DCNHjqzw8zNoiIhkyFrXaAoKCgAAiYmJz6wrKipCTEwMtm/fDjc3N4SFhaFnz57w9vau0D446oyISIaESvxXEWfOnEF+fj7Gjx+PiIgIZGRkGNf9/vvv8PX1Re3ateHi4oLOnTvj6NGjFX4tDBoiIoXRaDQICAgw3jQaTZnburq6YsKECdi4cSOio6Mxe/ZsFBcXAwC0Wq3xlBoAeHh4QKvVVrgenjojIpKhygwGUKvVUKvVz7Wtv78//Pz8IAgC/P394enpiVu3bqFhw4ZQqVTQ6XTGbXU6nUnwPC92NEREMiRU4lYR27dvR2xsLADgxo0b0Gq1xmswzZs3R1ZWFu7du4fCwkIcPXoUHTt2rPBrYUdDRCRDDlYa3zx8+HDMmzcPYWFhEAQBS5YswVdffYW8vDyEhoZi7ty5mDBhAkRRxLBhw9CgQYMK74NBQ0QkQ9Ya3Ozi4oK4uDiTZZ06dTL+3KtXL/Tq1atS+2DQEBHJET+wSUREUlLSFDQcDEBERJJiR0NEJEOc64yIiCSloJxh0BARyZKCkoZBQ0QkQ0oaDMCgISKSIV6jISIiSSkoZzi8mYiIpMWOhohIjhTU0jBoiIhkiIMBiIhIUhwMQEREklJQzjBoiIhkSUFJw6AhIpIhJV2j4fBmIiKSFDsaCWzacQirEvch++Y9tGvVGB9PH4qXXmxm67LsFo+H7XVtWgdzQlphWEK6cZmLowPC/uKD7i3roY67C7Lv5WPbL9dw8PxtG1YqH0oaDMCOpool70nHzNhkjOzbBVuXT0QtlTuGT1uDrGu5ti7NLvF42F7gCzUxJ6TlM384p/ZohgHtXsDO4zlYtPcMTuY8xLw+AQhuUc82hcqMUImb3DBoqpAoiliyfg/GDnkVH0zqh7eC22LryvdQt7YH1mxNs3V5dofHw7acHQQM79gISwe3hd4gmqwTRSAksD4SDl3C7hPXkXH1Ptb9cBE/X7qLYR0a2ahimVFQ0jBoqtCFK7dwJecO+ga3My5zdnLEm6+1xf7Dp2xYmX3i8bCtv/jVwchOPvjnT5eQeuK6yToRwJ6T1/HLlXsmy6/ey8cLtVytWKV8CZX4T24kuUZTVFSEDz/8ENeuXUNhYSH+/ve/o3fv3lLsSlbOX74JAGjWxNtkedPGXrh4NRd6vQGOjsx2a+HxsK3/u6nFu4nHoCvUY3SXJibrHAQg/vsLzyz7i68nrtzNt2aZssVrNGakpqbC09MTW7ZsQUJCAhYvXizFbmTnoe4RAEDlbvovMpW7KwwGEbr8AluUZbd4PGzrtq4QukL9c28/5qUm8K3rju2/XpOwqupDQWfOpOlo3nrrLfTp08d439HR0exjNBoN4uPjAQBTpkzFtGlqKUqTlJPD4/PQrs6A6x+/WVenP5e7uzgYl5P0lHg80t7vZusSLFKkB/Siaf0lfy42PL45CkDsoEBblEgSkuRt5uHhAQDQarWYNm0apk+fbvYxarUaavXjcHlU/PhW3bi5uQEAbj8oQO3aj/+oPSoG7mkL4OAgwNHZpVq+rupKicej7+qfbF2CRUZ3aYJhHRuh7+rHw5vT3u+Gnv/z+LVMerUphnZohN0ncrDm4EVblllhkga/HFsTC0l2gjonJwcREREYNGgQBg4cKNVuZKX5H9cCLj01dPbStVy09GsAQUknXasBHg95EwDMfqMFhnZohOSjV6tdyEhNSYMBJAma3NxcjB8/HpGRkRg+fLgUu5Cl5r710bhBHew58JtxWVGxHt/8mInuXVrZsDL7xOMhb5NebYreAfWx4ceL2JR+2dblyI4gWH6TG0lOna1btw4PHjzAmjVrsGbNGgBAQkICXF2VPWxREARMHxuCOcv/Bc9abgju2Byrk7/Hnfs6TA7rZevy7A6Ph3wZRGBQ+4b45fI9nL7+EK0bqEzW/d9NrQ2rkwcZ5oXFJAma+fPnY/78+VI8texNHNEdjwqKsC75ANZuTUNQSx9sXzUZTX28bF2aXeLxkCe9AXAQBHTy9UQnX0+TdflFegzdkF7GI+2IgpJGEEVRNL+ZdVW3C7RleXLxmeRBCcejug4GeFrJwQDVmZSDAc7dsPzzRC0buFVhJZXHT6sREZGkqtmnCIiI7IM1Luqbm8Xl888/x/bt21G3bl0AQHR0NJo1q/jM5wwaIiIZssYlmiezuCxfvhx3797FkCFDTIImMzMTS5cuRVBQUKX2w6AhIpIjKySNuVlcMjMzsWHDBty6dQuvv/463nvvPYv2w2s0REQyVJkPbGo0GgQEBBhvGo2m1H14eHhApVKVOYtL//79sXDhQvEvyFEAAAmTSURBVGzatAnHjh1DWpplX6/BjoaISIYqc42m5JRe5uTk5GDKlCkYNWqUySwuoihi7NixqFmzJgCgR48eOHXqFHr27FnhetjREBHJkDVmby5vFhetVosBAwZAp9NBFEWkp6dbfK2GHQ0RkZ0qbRaXESNGID8/H6GhoZgxYwYiIiLg4uKCV155BT169LBoP/zApoSU8AFBJVHC8eAHNuVFyg9sXrr9yOLHNq0nr+m+2NEQEcmQHGdhthSDhohIhuQ4C7OlGDRERDKkoJxh0BARyRE7GiIikphykoafoyEiIkmxoyEikiGeOiMiIkkpKGcYNEREcsSOhoiIJMUPbBIRkbSUkzMMGiIiOVJQznB4MxERSYsdDRGRDHEwABERSYqDAYiISFrKyRkGDRGRHCkoZxg0RERyxGs0REQkKSVdo+HwZiIikhQ7GiIiGVLSqTN2NEREJCl2NEREMqSkjoZBQ0QkQ0oaDMCgISKSIXY0REQkKQXlDAcDEBGRtNjREBHJkYJaGgYNEZEMcTAAERFJioMBiIhIUgrKGQYNEZEsKShpOOqMiEiGhEr897wMBgOioqIQGhqK8PBwZGVlmaz/7rvvMGzYMISGhmLbtm0WvxYGDRGRndq3bx8KCwuRkpKCWbNmITY21riuqKgIMTEx+Oyzz5CYmIiUlBTcunXLov3I8tSZqyyrqjiNRgO1Wm3rMugPSjgeae93s3UJVUKj0SDt/ep9LKTm5mz5YzUaDeLj4433p06dWur/+8eOHUNwcDAAoEOHDjh58qRx3e+//w5fX1/Url0bANC5c2ccPXoUffv2rXA97GgkVPJAk+3xeMgHj4W01Go1zp49a7yV9Q8srVYLlUplvO/o6Iji4mLjupo1axrXeXh4QKvVWlQPg4aIyE6pVCrodDrjfYPBACcnp1LX6XQ6k+CpCAYNEZGd6tSpEw4ePAgAyMjIQKtWrYzrmjdvjqysLNy7dw+FhYU4evQoOnbsaNF+FHI1RJ6mTp1q6xKoBB4P+eCxkIeQkBAcOnQI77zzDkRRxJIlS7B7927k5eUhNDQUc+fOxYQJEyCKIoYNG4YGDRpYtB9BFEWximsnIiIy4qkzIiKSFIOGiIgkxaAhIiJJMWiIiEhSDBoiIpIUg4aIiCTFoJGAuRlRyfqOHz+O8PBwW5dh94qKihAZGYlRo0Zh+PDh2L9/v61LIivgBzYlUHJG1IyMDMTGxmLt2rW2LstuJSQkIDU1FW5ubrYuxe6lpqbC09MTy5cvx927dzFkyBD07t3b1mWRxNjRSKC8GVHJ+nx9faHRaGxdBgF466238P777xvvOzo62rAashYGjQTKmxGVrK9Pnz7GiQLJtjw8PKBSqaDVajFt2jRMnz7d1iWRFTBoJFDejKhE9i4nJwcREREYNGgQBg4caOtyyAoYNBIob0ZUInuWm5uL8ePHIzIyEsOHD7d1OWQl/Ge2BEqbEZWIgHXr1uHBgwdYs2YN1qxZA+DxYA1XV1cbV0ZS4uzNREQkKZ46IyIiSTFoiIhIUgwaIiKSFIOGiIgkxaAhIiJJMWhIElevXkVQUBAGDRqEwYMHo3///nj33Xdx/fp1i5/zyy+/xNy5cwEAkyZNwo0bN8rcdtWqVTh69GiFnj8gIOCZZRqNxuz0Nb169cLVq1efez/P85xESsKgIcnUr18fu3btws6dO7Fnzx4EBARg2bJlVfLcCQkJaNCgQZnrjxw5Ar1eXyX7IqLK4Qc2yWq6du2KlStXAnjcBbz44os4ffo0tmzZgh9++AGbNm2CwWBA27ZtsWDBAtSoUQM7d+7E2rVroVKp0LhxY7i7uxsf/8UXX8Db2xvR0dE4duwYnJ2dMXnyZBQWFuLkyZOYP38+4uPj4erqioULF+LevXtwdXXFRx99hDZt2uDq1auIjIxEXl4e2rdvb7b+zZs3Y9euXcjPz4ezszPi4uLQrFkzAEB8fDzOnDmDGjVqIDo6Gq1bt0Zubi6ioqJw/fp1CIKAWbNmoVu3btL9golkih0NWUVRURG+/vprdOjQwbise/fu+Prrr3Hnzh1s27YNycnJ2LVrF+rVq4eNGzfixo0bWLFiBZKSkpCSkmIyf9wTiYmJyMvLw1dffYXPP/8cq1evRr9+/RAUFISPP/4YAQEB+OCDDxAZGYkdO3Zg8eLFmDFjBgBg8eLFGDp0KHbt2oVOnTqVW79Wq8W+ffuQmJiIf//733j99deRlJRkXO/n54edO3di8uTJxtN7n3zyCYYNG4Yvv/wSa9euRVRUFLRabVX8OomqFXY0JJmbN29i0KBBAIDCwkK8+OKLmDVrlnH9ky4iPT0dWVlZGDlyJIDHodSmTRv8+uuv6NixI7y8vAAAAwcOxH//+1+TfRw5cgQjR46Eg4MDvL29sWfPHpP1Op0OJ0+exLx584zL8vLycPfuXfz888+Ii4sDALz99tuYP39+ma9FpVIhLi4Oe/bswaVLl/DDDz8gMDDQuH7EiBEAgB49eiAyMhIPHjzATz/9hAsXLmDVqlUAgOLiYly5cqUCv0EiZWDQkGSeXKMpS40aNQAAer0effv2Nf6h1+l00Ov1OHz4MErOkFTaDNhOTk4QBMF4PysrCw0bNjTeNxgMcHFxManj+vXr8PT0BADj8wuCAAeHshv8nJwchIeHY8yYMejevTu8vLxw+vRp4/qS36siiiKcnJxgMBiwadMm475u3ryJevXqYd++fWXuh0iJeOqMbK5r16749ttvcfv2bYiiiIULF2LTpk3o3LkzMjIycOPGDRgMBuzdu/eZx3bp0gV79+6FKIq4ffs2xowZg8LCQjg6OkKv16NmzZpo2rSpMWgOHTqE0aNHAwC6deuG1NRUAMA333yDgoKCMms8ceIE/Pz8MG7cOLRr1w779u0zGWywe/duAMC3336L5s2bw93dHS+//DK2bNkCADh//jwGDhyI/Pz8qvmlEVUj7GjI5lq3bo2pU6di7NixMBgMCAwMxF//+lfUqFED8+fPx7hx4+Dm5oYWLVo889hRo0bh448/xttvvw0A+Oijj6BSqRAcHIwFCxZg6dKlWL58ORYuXIh//vOfcHZ2xqeffgpBEBAVFYXIyEikpKQgKCgIHh4eZdb46quvYuvWrejXrx9EUUSXLl1w7tw54/pLly5h0KBB8PDwQGxsLABg/vz5iIqKMn7nyrJly0y+EI/IXnD2ZiIikhRPnRERkaQYNEREJCkGDRERSYpBQ0REkmLQEBGRpBg0REQkKQYNERFJ6v8BF+WgJCMposoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 576x396 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_confusion_matrix(lr, X_test, y_test, display_labels=lr.classes_, cmap=plt.cm.Blues, values_format = 'd');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Precision, recall, etc. don't apply directly.\n",
    "- But if we pick one of the classes as positive, and consider the rest to be negative, then we can."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 217,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00        14\n",
      "           1       1.00      1.00      1.00        19\n",
      "           2       1.00      1.00      1.00        12\n",
      "\n",
      "    accuracy                           1.00        45\n",
      "   macro avg       1.00      1.00      1.00        45\n",
      "weighted avg       1.00      1.00      1.00        45\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(classification_report(y_test, lr.predict(X_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- We can also make the plots we made earlier in the course, but this time with more classes.\n",
    "- I need to train a model with only 2 features for this to work:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 218,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr_2f = LogisticRegression()\n",
    "lr_2f.fit(X_train.iloc[:,:2], y_train);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 219,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdAAAAE8CAYAAABw2+D5AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXxU9bk/8E+2mSSQQAhSCRACGgQFAXdJSKKAgKBSkcVe5Xa11at16aVKb39e2v5uQbGgbX+CVnsrtZVVwYIYBDQJkaLIUgIYErPhBIQw2SeZrL8/JmdyzsyZmTNnljNn5vN+vXiVzHoSap75Pt/n+zxRvb29vSAiIiKvRGt9AURERHrEAEpERKQCAygREZEKDKBEREQqMIASERGpEOvNg09+VR+o61BkTNkOfPWHt4GhaZpeBxERRYbJb/3V5X36XIHW1Wp9BUREFOF0FUArMxdg2ItrbF/U1QLmb7S9ICIiili6CqAAUBc/BtG/34aROROBnm6tL4eIiCKU7gKowDh5iu0vTOcSEZEGdBtAmc4lIiIt6TaAAjLpXK5GiYgoSHQdQAW1D6zsX41yJUpEREEQFgEUsK1G7StRBlEiIgqwsAmggG0lynQuEREFQ1gFUMAWRK964iHbF1yJEhFRgIRdAAVsFbpM5xIRUSCFZQAFmM4lIqLACtsACjikc3lWlIiI/CisAyhgS+ey9R8REflb2AdQAVv/ERGRP0VMAK3MXCBN5xIREfkgYgIo4JDOZRAlIiIfRFQAFTCdS0REvorIAMp0LhER+SoiAyjAdC4REfkmYgOogOlcIiJSI+IDKNO5RESkRsQHUEAmnctASkREHjCAinCSCxERKcUA6qB51GS2/SMiIo8YQB3UxY/pn+TCdC4REbnAAOoC07lEROQOA6gbTOcSEZErDKBuMJ1LRESuMIAqwHQuERE5YgBViOlcIiISYwBViOlcIiISYwD1kiSdyyBKRBSxGEBV4CQXIiJiAPUBJ7kQEUUuBlAfcJILEVHkYgD1EdO5RESRiQHUT5jOJSKKLAygfsJ0LhFRZGEA9aPKzAUY9uIanhUl8lKxqRbPfFIAa7e0UYm1uxvPfFKAYhP/W6LQwwDqZ0LDBftqlIg82l5WhgM15yRBVAieB2rOYXtZmcZXSOSMATRAmkdNtv2Fq1Aij1ZNz0bWiDT7SrS5o8O+8swakYZV07O1vkQiJ1G9vb29Sh988qv6QF5L2BnaXgnDrrfwdWEJEB0DDPmW1pdEFLLk0rVZI9KwNi8XxpgYDa+MItnkt/7q8j6uQANIks5lE3oit4wxMVjtsNJcPT2bwZNCFgNoEDCdS+SZtbsbzxUdlNz2XNFBp8IiolDBABoEdfFjpNW5nCmqC6wMDR7xzzRrRBqKli6W7IkyiFIoYgANEqZz9YeVocGzouigPXiuzctFksGAtXm59iC6wmFlShQKGECDjOlc/WBlaPAszMzEnemjJAVDxpgYrM3LxZ3po7AwM1PjKyRyxipcDbA6Vz9YGUoU2ViFG2Kc0rlcjYYsVoYSkSsMoBpyav1HIYeVoUQRzMPvZQZQjdXFj+mf5MLq3JDCylCiCGX+RtGihgE0BNgnuTCdG1JYGUoUgepqgZ5ujH38Ibx525/cPpQBNEQwnRt6WBlKFEFEq87DP83Hfx3J9fgUVuGGmDFlO/DVH95mdS4RUbD0Bc6xjz/kFDh//3/GunwaV6AhhulcIqIgUbHqFGMADUFM5xIRBZjDXmfJZzVevwQDaIhidS4RUQD4uOoUYwANYUznkq/YEJ9IxA+rTjEG0BDnlM7lapS8wIb4RPDrqlOMAVQHOMmF1GJDfIpYQtD086pTjMdYdGRoeyUu/nx53xdp2l4M6QYb4lPEER1LeSf6YQBQHTjdHWNhANUZSRDlWVFSqLmjA9M3bbF/XbR0MZIMBg2viCgAzN/Ys3SHf5rvl9Umz4GGkbr4MYj+/Tamc0kxNsSniBDAVK0rDKA6xcHcpAQb4lPYC1CBkBIMoDpVFz8Gw15c0/cFq3NJHhviU1jTYNUpxgCqY07pXK5GyQEb4lNYEq06G1dvD+qqU4xFRGHCXlzEwiIiCmduGr8HAouIIkBd/Bhbs4WebqZziSg8hcCqU4wBNIxImi0wnUtE4aKvIYKw17lth1nrKwLAABp2hNZ/ALgSJSL9C7FVpxgDaBhiOpeIdC9EV51iDKBhiulcItKtEF51ijGAhjGmc4lIV/pWnSNzJobsqlOMATTMMZ1LRLogWnX+quNJjS9GGQbQCMB0LhGFLJ2tOsUYQCOEJJ3L1n9EFAp0uOoUi9X6Aih4hNZ/Y8p24Ks/vK315RBRJBJlwUbmTLQFTh2tOsUYQCOQbZLL27b/I3MwNxEFQ1/gHJkzEdun/A4A8GaQm7/7G1O4ESgcJ7m4Gs8lHudFRBpxSNWWfFYT9MkpgcAAGqHCbTD39rIyHKg5JwmiQvA8UHMO28vKNL5Cogik4wIhJRhAI1y4DOZeNT1bMii6uaNDMkh61fRsrS+RKLLovEBICY4zo/5RaICux6HJpWuFQdLCLEwiCjDRXufqpN/AerlO4wvyDceZkVvhks41xsRgtcNKc/X0bAZPomBxWHXqPXh6wgBKdnpP51q7u/Fc0UHJbc8VHXQqLAo1eiiA0sM1koZEe51vz3sv7PY6XWEAJTs9V+eKf5FnjUhD0dLFkj3RUA6ieiiA0sM1kkYibNUpFrNy5cqVSh98sb49gJdCocASm4KouYsx9moD6g+fACzNQGKS1pfl0c8Li1D4tcm+55kYF4eZo9NxxmxGsakWFY2NmJ2RofVlysobNcp+nWfMZuSMHIHlhUX2DwNrcnMQG63tZ109XCMFWV0tYGnGyJyJeHXiazh57LLWVxQQc3NTXN7HIiJyyV5cpIPComJTLbaXlWGVw56ntbsbK4oOYmFmJrJGhG7TCD0UQOnhGilIRKvOcE/XuisiYgAlt9K2rcTXhSW6CKJ619zRgembtti/Llq6GEkGg4ZX5EwP10gBFGYVtkqwCpdUq31gZf84NJ0WF+mBHgqg9HCNFEARvNfpCgMoeVT7wEoO5g4gPRRA6eEaKUAitMJWCQZQUiQSBnOfNJk0ed8VRQftgWltXi6SDAaszcu1B6gVDqs+XiMFTd+qM3X1Gq46ZbAKlxRrvjYPI1pPo6nqG91U5yplamhAVeFBxI9IQ1J8fFDfe2CcAe3dXViTm2MvxomNjsbM0emoaGzEwsxMpCdr+7PWwzWSHzlU2BYe69H6ijTDKlzyq3Bp/Sd26GAxflJRiQ1XjcXtWdO0vhwi7YhWnS/uGKzxxWiPRUTkV0LrP3tKV+dMDQ24vi99O+lrE2obGzW+IqIg69vnFO91Mnh6xgBKqhknT7H9RefVuTUlpzDN2gEAyLJaUV1ySuMrIgoSUdA8/NN8HP5pPvc6vcAASqpVZi7Qbes/gXj1KeAqlCKCQ4FQuAy5DiYGUPKJp3SumsrWYFbDilefAq5CKawxVes3DKDkF3LpXFNDA745dNir1Zya56glt/oURMoqlFNWIgyPpfhVrNYXQOHBls6dbKvOrasFomNQc7ocP7NYsKHkFNIUVrbWlJzy+jlq1Tc348LYMfjSxf3GpiakDRoU0GvQmnjKitDT1jF4hnIPYVJI9MH27XnvwbqDgdMfeIyF/C5t20oc3v1PDP7sOKZZO1BsNKJpzl0eg5GpoQGD8/d69RzyjWOHodXTs/GcQ9MENorXOR5L8QmPsVBQGSdPQU1FjdeVrayGDT5jTIyko9D0TVsYPMOFcDQF4F5ngDCAkt8djJqAyS0WyW2e9hRZDasdY0wMVk/Plty22mEsHOmMaNX55m1/4l5ngDCAkt9VvfYKbm9sktzmaUXJaljtcMpKGOGqM6gYQMmvTOWluP7jvbL3TaqpkV1RshpWO5yyEka46gw6VuGSX9VXV6B2wWKnytaBZf9Ei6kORlMN0gZNkj6H1bCacZyyIuyJCkF1RdFBrM3L1foyyR1W2GqGVbgUNGnbVuLrwhLbF0N5NCIUFJtqsb2sDKsc9jyt3d1YUXQQCzMzI+IYi25/DqywDTh3VbgMoBRU9kkuYTLFhcLDM58U4EDNOclKXJzevjN9VGitxB1XnUzXBgyPsVDIiITB3KQ/q6ZnS/Z+mzs6JHvDz992Gzae6sJLnw/ExlNdaOvs0u5iudcZMrgCJU0wnUuhRq59YdaINCzKzMLaL+5ATfP3ARgBWJGe9Gc8e0shskYEsYxEtOo8/NN8Nn4PEq5AKeTUPrCyf5ILV6IUAuTOw6687ba+4PkobMETAIyoaX4UL3yWE7yVqMOqk8EzNDCAkmaYztVeW2dX6KQmNSZ3HvZHHx1CTfN3ZB9f0/x9bCsL4AWJhlwDtlUnC4VCC4+xkKZqH1iJkehL59bVMp0bRMWmLrzwWY4kNbntrAapyRDgricw8CCAdwHEOzzLiIutgwC0+vdizN/YRwMe/mk+ANhWnFx1hhyuQElztQ+sxFVPPGT7QqeDufWmrVMInhqnJkOE43nYJIMBa/NykZE8HMAeAHKrUCuGDfBzk4+6WqCnW5KqZbo2dDGAUkiozFzgdjA3uaZmpufWs+hbeToLeGoyBC3MzLQfVRHOgRpjYvCXOdlIiJ0M4BGn56Qn/RkPjPPTBZi/YapWhxhAKaTIDeYm98QzPYUgKgTPAzXnsL3MORpetAxG/8rTkZCajByups8MjjfipdwJSE+qBGDtu9WK9KT1ePaWQiTE+iHVLbPqJH2IrI0OCnmVmQtw1RPAV3942z6Ymw0X3Fs1Pdu+0nzmkwKnmZ6rHCpLAWBYYgNsAUEuiAqpSf56AICsEbHYPL8Q28oKcbF1EIYNaMQD4+B78HTY6yzZwcCpNzwHSiHLflbUD4VFum3VplDl5ctYsScfX/b02G9zN9OzrbMLS3YJe6BS6Unrsfke16urts4ubD1rW8UOS2zAonFAQhyDrVf6MixjH38I/3UkhDockROeAyVd8mc6V02aU08unPkS/xQFT8D9TM+EuFg8e0sh0pPWw5vUZLHJFnjXfvEXvH1mPdZ+8Rcs2ZWDYlNkFR2p5rDXyeCpb1yBUkgbU7bDls4FfFqJujum4G6lpjUlqz1TQwMSPszHmo5O7BHdruT7auvswrYyKEpNely1zi/kStQdrjp1ic3kSff80frPVau2UA2ecuc05VrIFRYWYU9VNfYAmAtgVkYGtnd2+P3DwcZTXVj7xV/gat/0Zzd9Fw9fywDqxHGvk0VCusIUboQ7Ubhf60vwmeSsqMpzonKt2tylObWk9JymqaEB22pq7MHzXQC3nD+P5VOn2Jujr3DorqMWK3dV6KuwHfv4Q6ywDUMMoGHOVF6Ki794ErUV+t7jA4DmUZN9Oicq16rtuaKDTucnQ4HSc5o1JaewoqcX30Z/r5wsqxXnz3yJtXm5uDN9FBZmZvrlmvord+UEoKmAnnGvMyIwgIa5qtdewTMXalG5YZ3Wl+KzuvgxttZ/ORMlPUKVcNwDLVq6WDK+KtSCqJLVnqmhAdebTJgD50Zzk7424XJLC9bm5fqtunjROFvzAJsPAdwPoB1Af1MBd80bIgZXnRGDGxZhzFReius/3gsAmPTxXtRWlCFtrH9WI1qqfWAlrprcV1xk/kbROVHHVm3GmBiszcu1/7JfUXQwpAYmKzmnWd/cjAtjx+BLF69hbGpC2iD/pVWFyt0XPgNqmvMB7ASwAKMG3o1nbzmE6KgoSfDU87EgVRz2Ot9k4Ax7MStXrlyp9MEX69sDeCnkbydXPY9vH/8cAJDe3oZ9rS1InzVP46vyj9gBRqRYz6Op6hsgMcnj4wfGGdDe3YU1uTn2Pc/Y6GjMHJ2OisZGLMzMRHqy59cJlnEpPfio2ozGjpud7ktP+hP+e1o1RqSkIG3ECJd/hvkxeNrfOzkaC66uxEBDNM41NaO58yiuSz2Bh65Nx/LCIvuHlDW5OYiNjqAEV10t0NuLsY8/hJein8BFE9PZ4WJuborL+1iFG6ZM5aVIXjoP0xrM9tuKU4agZfOesFiFCsJ5MLfSKlyt6K2qOSBYYRv2eIwlAhUvfww/eW+T0+0b7l+KrBdf1eCKAsd+VjQM2/55c05TC80dHZi+aYv966Kli5FkMGh4RUHEc50RgcdYIox479ORsBcaTnytzg1lCXGxePjaWPzs5lY8fG1sSAVPPVU1+43MkGsGz8gVOv81kt/UV1egdsFil8UlCZXlYZXGrYsfA4gHcwOapXNDreduoK7HXWenZz4pCL80LodckwymcCms+Kv1n1pCb13xXqA42AgzJ/V+PaH2fQYUU7URzV0Kl1W4FFYaUscjau5ijGg9jaYzXymq0PWnvFGjcMZsRrGpFmfMZuSMHKFpdWqgrkdvVc2qmL8BWpsA2Fadmz53XY1J4YtVuBRxtFyJhlp1aqhdjy5w1Ul9WEREEcc2mLuvd64fxqF5I9R67oba9YQ0UQu+xtXbGTzJLQZQCluVmQsQ/ftt/a3/gsSf1amuWg160zIvIqtl1XBowbdth9nzcyiiMYBSSAjkxBh/Dub2xN89d30dBK63HsCa4KqTVGIREWnOVF6Kqie+h4S8WUhKSfX76zekjsfYqw2o/+xfgKU5oIVFPy8sQuHXJvseY2JcHGaOTrcX8lQ0NmJ2Robi1/O1CMjf1xN2HFrwnf6yTesrohDjroiIAZQ0d3LV83jiyKGA9uoNVnWukurUc83NePnoUeSNGiUJftbubvy8sAgD4wz2ClbhuULA+9+SUzjX3Ky4CCgiqmXVEq06/3ggvDpYkf+wCpdClrhnbzB69Wp9ThRQd4Yyolvm+RsrbMkLrMKlkFX12iv2hvdZ9eaAzy3VsjpXsGp6tmQfsrmjQ7JPucqhYpZFQH7EvU7yIwZQ0oxcz95g9OqtzFyAYS+uUTWY2x+EWaRCEJ2+aYvTrFJBIIqA/FHZqzt9/86ssCV/YgAlzYhXn4JgrEIBW//c2gdW9q9Gg0zp2UzHQeBJBoMk+K5wWJkq4Wtlr+5w1UkBwgBKmgiViTHNoybb/hLkVaintKwQ0MYPGWLfExWCq7CCvTN9FBZmer9f7G0KOZACuhruW3WOzJnIVScFBIuISBMl+/eg7VCRy/sTbp+OiTPmBuVahrZXwrDrraBNchEHhyHx8TC3tyPFaES91WoPYMLKM1BN2V2191s0LhPvf1URtGkyAWtKL1p1MnCSLzhQm0gBe4VugAOoOGiIg6UQRFPj43G5vT3g/WrlKnv/+9NDQZ2y4m4smqrvvy9wjsyZiF91POm366TIxWksRArEDjCi9aOPAt5sQXw2c4CoscHZ+gYAQFtXV8CDp7W7G8sLi3Cuudl+29n6evzf7Cycra8P2jQZX8+5ArAFTUuz7Q9sq871pyb77RopsvEcKJFCknRudAwwJDgH7IN5ztPTqk+8KhYoCWi+DO9W/f2LUrVf1g60Dbkm8iOeAyVSSFKd29MdtP65wTzn6amyd+Wnh1RNb1Fb3avq+5cpEGLwpGBjCpdIRkPqeFxxx81IsZ4PaOs/x9Xg3+fNlaRQZ45O9/sAbk/t/e4dOxa/P3bcKb3r7lqKTbU4dfkyhg8cgE9rz9vTvz8rKMSntedxbeoQ/GHGnU7PV/X9i1adTNVSoLlL4XIFSuRCXfyY/kku5m8C8h6BOOfpiat0rDEmBqumZ2NrWZnXjRu2l5Xhk3NfAwBuTxtubxBxqPY8AGBYYqLT+7V1duHfdtv2VzOSh+N/srLcf/88lkIhhitQIjfsk1wOnwhIcVGoNXtXO71FmBrzae15jBw4EF+3tNjvu234lfhdXq5kJVls6sJTH+egsulBAO1osH6EAzUtSE+uxJhBsc7fP1edpBEWERH5SFJcpFET+mDwpRDI2t2Npz7+xL7qFNyeNhwv35Fnf722zi4s2ZWDmuZHnV4jPWk9Ns8vREJcrO0G0bGU1Um/gfVynT++TSLFWEREYedgwT48/dgyWK3SrIjV2o6nH1uGgwX7/Pp+wUjnhgJ36V0htepOj8Pn8VuHX4lDtecl6d+tZ4Ga5u/LPr+m+fvYJtQaiVadv+p4ksGTQg4DKOnS9s0bsX/vLkkQFYLn/r27sH3zRr+/p32SS5Cqc/Xm54VFOHz+guS26Kgo+56osJ950TIYgNHFqxhxsS7Gvtf59rz3uNdJIYsBlHRp9brXkZ07074SbW5usq88s3NnYvW61wPyvk6TXPxE7xNSrN3duGRpA2BL2QrFR0I6N3fUSHvf3mGJDQCsrl4Jw+LrueokXWAAJV0yGuOx7tWN9iCaNTXDHjzXvboRRmN8wN7bX+lccdAUn6Fs6mvu/sm5c7qZkLKi6CBOX76MrBFpePmOPEk17aHa84iJirKnfxeNA9KT/iz7OlelvoWuH7/JVSfpAouISNeam5uQNTXD/nXxsSokJSUH5b3tvXMBVYVFnnriCo3mA93Wzx+8LT4qNnXhhc9y+vZCjQCsGJm+CzfkxmN05oTgfwNELrCZPIUluYKhYKxAxXxp/efYRODX027HA+/vQr21P72ph+CpVtuFc9hWMwCWKzJxZlg2xk+YgDhDgtaXRSTBKlwKuhOF+wP6+uLgmZ07E8XHqiR7oo7VuYHi1PrPC+LK1mJTLWZs3S4JnoCyFnq6VFeLhNgY3Lvlb4id9VNMmnwDgyfpDgMo+Z2pvBQXf/FkQIdiP/f0I5I9z6SkZMme6HNPPxKw95ajdjC3MSbGqe+sWCB74mpC1E2IFbakdwyg5HdVr72CZy7UonLDuoC9x8IlyzDjrvmSdK1QWDTjrvlYuGRZwN5bTl38GGl1rsLiIrlG6ilGI/YtWqiohZ6u9H24SF29hhW2FBa4B0p+ZSovRfLSeZjWYEZxyhC0bN6DtLGZml7TicL9mJwzI2jvp3Qwt3gPVBiiLRQQXZs6BKkJCejq6cGh2vP2QdZKOgKFHHYTIh3jHigFTdVrr2Bagy0tl1VvDugqVIlgpJMdKU3nihvJ/5/bbsWd6aOw89v3IWtEGk5fNqPoaxOA/jOUSkaDhYy+VC1XnRTOYrW+AAofpvJSXP/xXsltkz7ei9qKMs1WoUI6ecOGdUh78dWgvKeQzvVUnSs0FhCOfuSljwIArM3Lxc8Li3DJ0oZDteeRNSIN118xVFKxu8rNvqnmREHzg9rrUfaVBdYdDJwUfpjCJb8pXv4YfvLeJqfbN9y/FFlBCl5ioZBOVprOlSPXhSikj7X4OVVbXVaAkiObMPuBlxEb19/6r6vTivxtT2HiTUsxOjPXp/cg8oQpXAo4udWnQFiFBlsopJPVVucC8hW6IXusJQCp2pIjm1BxZi8+2PQoujptx3u6Oq34YNOjqDizFyVHnD+sEQUTV6DkFyX796DtUJHL+xNun46JM+YG7XrEq0+BVqvQoe2VuPjz5bYvvGi2oIsVaAALhIRgWV1WgNGZuZi96BXkb33S/vXdS9dLVqZEgcBORBQwwa5wVSrU0smAd63/HLsUrZ6ejedERUchEURFq84XdwwOyFuIg6iAwZOCiSlcCggtKlyVCMV0MtA/yQWAx3OiKxyCpbg5u3g0mCZE1bVvz3svYMETAGLjjJi96BXJbbMXvcLgSSGBVbikmhYVrkrUV1egdsFifOni/oTKcs2qguvix2BkzkRbda75G5fpXMcKXaC/9Z9wDlQTjqvOAB9L6eq0In/rk5Lb8rc+yRUohQSmcEmVUKhw1TNfJ7kEnagI6u157wXlPCf3QCkUMIVLfhcKFa565k061xeeBnUfqD6Hjae68NLnA7HxVBfaOrucX0S06nzztj95FTyrywqw+53+KlpBV6cVu9+R7m06yt/2lCRYGuOTcPfS9RidmYvqsgLkb3tK8XUQBULMypUrVyp98MX64Ey4oNBmKi9F+ku/waj2Nvtt1gu1uDxzLpJSUjW8Mn2xxKZgROtpNFV9A7RbgISBfn+Pl48exYGaczhjNmPm6HTERkfbg2fh1yYUmgajyLQb/6q7D4fO34OPqs1IT65EenI0is+cxMtHjyPvW0Ox6Z6dKDzWA8AW/PZseQLG+CQMTs1w+/7Fe19AxZm9uHS+BFddOwfRMbH2lWVV6QF0drZh3KT5ss81xiehs7MNcxf/0b7SjI6JxVXXzoH5Uhkm3rTU4/sT+WpuborL+xhAyWsnVz2Pbx//XHJbensb9rW2IH3WPI2uSp+ar83D2KsNqD98ArA0A4lJfn39vFGjcMZsRrGpFmfMZuSMHIHlhUUoNtUiPuY6tHd/BkCYnRqLxo6bUVJ3CQtSPsUfz1bgwDeX8FF0BjIycrwKfoKx42fh0vkSVJcV4NL5EmRccyc+3PK4fWU5d/EfER0jX4oxODUD4ybNd7o/OiYW4ybNZ/CkoHAXQJnCJa+EaoWrnknSuV5MclHCcebo9E1bUGyqRUbycLR3F6M/eParaf4+ttUMwPUrSuzp0g82PQpre7N0T/KBlz2+f2ycUZJ2ff23U0J+D9OXtDNFFhYRkVdCrWFCuPGl9Z87zR0dmL5pi/3rxeO+hy1n/+zy8VOmbcT0Odl+O4dpbW/G67+dYv/6kV8chzHev6ttf9n9jq3Tkfj7FP8cxk64C/MeXK/1ZVKQsJECkU5Iuhb5KYjKdTTKSB6OqqYzAAbJPQPZczZj6jRbG0Ffg5/emiGw+pfEWIVL5EcHC/bh6ceWwWqV1gRYre14+rFlOFiwT/VrC5NcbF/4ns517GhUtHQxskakoarpPOJjsgA41zUMSt2JiTfdCMD1OUzH9KYrjsHokV8cl6SFlb5OMOkx7UzaYAAlUuBE4X7737dv3oj9e3dJgqgQPPfv3YXtmzf69F518WMQ/fttuOqJh4Cebs9PcMNlR6MrUtHefQoJMbcCEIKYFYNStyD37gGIMyT4Jfjp9SgKOyCREqzCDXMnCvfjytGuUxDkmam8FFVPfA8JebOQlJKKO2bOxZlTJ3CwYB/OnDqBnDtn4z+f+B4OFuxDdu5MvPSHPyM21vcmX7EDjGj96COfqnMHxhnQ3t2FNbk5MDbWAa1NiG1vxcwrr8DXGWMx9pbv4lsjz2LIsOLj5IQAAB1GSURBVBO4+rpTmHHfZAwZNhwAsGfLE6gqPWAPfgZjIq66do69qtZ8qcxjFa5ej6J0dVrx4ZbH0Wiutt9Wd+GM/SgORQ53VbjcAw1jpvJSnP7uQly38T12CfKB0Jhe3IheLl2bnTsT617dCKPRubL1YME+bN+8EavXvS6532ptx3NPP4KFS5YhO3em0/PUTnJx4jDkGgBKPqtx+5RIncfJPVASYxFRhJL7xU/ecdeysLm5CVlTM+yPLT5WhaSkZNnXEdK74iArDsIz7pqPda+6Tv2qrs41fwP0dKOtqxu/G/00yksaMSC5F5NuvhFxhkTvXitCsAqXxFhEFIHE5zV5PlM9Vy0LrdZ2PPvUDyWPffapHzoVFglWr3sd2bkz7QVIzc1N9uCZnTsTq9e97vY6VA3mrqsFerpx8p4fY84Xj2Hbn+bg+KFlKM5finfWn0Z12RnlrxVBJt60FGMn3CVZaQqFRWMn3IWJNy3V+AopVDCAhin2qvWdXNOISR/vRdWXJZLgV3ysShIc5YKo0RiPda9utD8ua2qG/fmu0r5iXlXnmr+xB9rCR97F478EGi8vBiCkHY1ovLwYBR+0orPDovTHETFGZ+Zi3oPOadrYOCPmPbg+LNPWpA4DaBhy9Yufq1DviD+ECLLqzXj+ke9Igl9SUrIkOD739COyr2c0xuOFl9+Q3PbCy294DJ4Cp+pcudVo36pz7OMP4c3b/oTN63eg8fJ9sq/XePk+lBw5qui9icgZA2gYcvWLn6tQ5dy1LHyoqQHTsvIkK0dhhTnjrvlYuGSZ7PO8Tfu6IjvJRbTqPPzTfPzXEdsqqbUpCv0rT0dGtDR59dZEJMJ67DCjpFctK3I9czeUOwXA47dPd1o5CkFUjrhgKDt3Jl54+Q08+9QP7WlfJWlcMclg7r7AOfbxh2yBU1RdOyC5F7ZznnJB1IqB8jVPRKQAq3DDDHvVhiZfq3BdGVO2Aw0jJuOD2utlj6V0dljwzvrTfXugUoNSt+DBR69DnCFB1fdEFAl4jIVIY2rPgXrSYo3CyXMD3Z7prC47g4IPWvv2Qo2wdRzaidy7B2B05gQV3w1R5GAAJQpTSgIoYFuJlhw5ipYmYGAyMPGmG7nyJFLAXQDlHiiFtBOF+zE5Z4bWl6F7cYZE+3QVsunssODk51+gtSkK8YntQFQU2luNbDRBijGAkl8EItCZyktx8RdPopatCF0aaOzF+LQW4JZ0j6tQ6tef1l4K4DSASgDzIKS4S44wxU2e8RgL+cwe6Px8zrTqtVfwzIVaHr/xICXRFkSNqUO1vhRd6Oyw9AXPxQC6AFQDuB9sNEHeYgAlnwUi0LEVoXeioqK0vgTdOPn5F6LmEh8DkK9KZ6MJ8oQBNMKJ51x6c58gUIGOrQiVs1hasemtvfh4Wz6OFhdx1eSBtLlED9hogtRiAI1g7lKvStOygQh0bEWoXHHhSSy+9zBefelefPHxIjaKV6C/uQRg+xXoajA4G02QewygEcxd6lVJWjZQgY6tCJWxWFqx6teXUFO1ENy/U27SzTdiUOrOvq/uALBH9nGDUndi4k03Bu26SH8YQCOUu9TrgU1/UZSWDUSgU9KKkGy2vvMpaqrmy97H/TvX4gyJyL17AAalboHtIEIngL+ifyVqRXT0XzF+iplnZcktBtAI5Sr1aiovheV//gtpHtKygQp09dUVOJI3E68ZjPgHYP+zwWDE0bxZMFeWq3rdcHTxAsD9O3VGZ07Ag49ei9tnboQh3gJbFe4+AO8D2IeenoX48vgQruLJLZ4DjUDuUq+nX/o1VrS1YQuADIf7xGcx3TVbB4CEynJVZzcnzpiL4g//gR93OOxLdVixISqKfXxFhl0JsFG8enGGRETHGNDRvhS2n+E8yf22VfxmNqAglxhAI1DVa6/gJzKp11UvrsS0Alvl7WgAVbAF0ax6MzZsWIe0F1+1P37ijLlAAIIZp8kot+jBadj6zq6+PVAp7t8p449xb+KORuxiFFkYQCOMuwA14JN9yO3qBADcCnhchQZCoFa24SgxcQBWPH8FVv16e99eqLRRfLjs37kLUJ0dFhz/5yFUl9UBve0YnTkCU26fpjiA+TruTdrRiF2MIg2byUcYV+POzA31uGnXduR1ddlvKzAa8fncBRgyOAUAR6GFKoulFds3HcI5Uy+6jfFIHXZV2ARPd5NkAGD/jnq0Ni+y3wfsRuLAGsz8dqaiAObLuDfPz72WK9EwwGks5FHx8sfwk/c2Od2+4f6lyBKlbil0KZ3MEsokDd4HWHH6qBFN5mVOj0seshHojUNT/YMyr7IDyUOa8J3HblAUwNSOeztaXITifGHl6ciK7DncPw0HnMZCbnHfkbTW2WFB4Z7dKD8VjY72qwHcBduvp90AjgGYKnl8kzkJwN0uXm0umsx7UHLkqKIAZqvItaDkyGavxr35Y/+U9I0BlLjvGAThPpZNWDk21XegueFrJA1OR3JKrMuCGvFKs6PjPL6uuBJN9U+hPxW7B7ZStvsB7AAwDsAA0SvEwF3wAqK9CmBqxr35un9K+scASgGrqCWbQI9ls1hasfWdT2GqBboMRgz91lVuU5dqq0ZdPa8/BToewHkAy+CuoEau8MYWME/DttI0AliA/sA5F7YzmuJjJt1wF7yAnoAHsEk334iSIztd7IGyCjoSMIASBZjQFtHxKJA/FBee7Gvnp2z/Tm3VqKvnZc2KRfFHXWi8PA/AR7AFPoHQVnALHnzUgjhDosMosf7HSQOmsNIUB85uyfUkD2kGet91sQe6B8lDmv0awFx9eMi9ewAKPtgiu38aLoVc5BqLiCgs+Ttlqvb1TOWlSF46D9MazChOGYKWzXv8tgq1WFqx+N7DqKmaA9tYrh7YmovdgUGpu52qQNVWjbp7XnziW2i33APgUwCz4KmgxlPhjfNK830As2FboS6A0ircvPnpaKxv9svZTE9FRp0dFpQcOerV/inpB4uIKKL4O2Xqy+uJm1bINaTwha0X7mjYVn5zIU6HNl4e7VREY5uDuVT2tdx13XH3vHbLUtiCnrKxYJ4Kb6QrTStse527AXTCEL8Gmdf1YvrcefYA9fCTFpw4/FdUn70EwIrR40YgZegoFO/t8svZTOmKuRW2n3UPGi8nIn+bCQ89cQUSBw5ltW2EYi9cCjv+HvCt9vUCPZbNdK4dtj3HBRBPY7F9fR6N9dJ2iGqrRj0HPSuUjgWTjhJzfpwtYAp2A2gBMAbAEnS0L8fXVRkA+pNmcYZE3DT9Tiz8wRLc+/AD6O0F9r1XgcbLiQCEM839E2osLXU4WlyEoj0HFc1O7R++fQy24DkLwL0AZsHaNhRvvbwV1WVnUF1WgN3vPIquTun31tVpxe53HkV1WYHb9yF9YgClsOLvAd++vF6gx7JVn9wP28pTzly0NJokt3gKXq6Kbtw/7x+IjvklgNshjAUbiA8BtMNWQfshBqXuxDWTJ+BocRGa6jsQn+h83thmN2zjxaywTUf5HLZfUSbYVn+up8xUl53BO+tP45/7lqGjfTlsge4j2AIf7M99+497UZy/FMcPLVM0O9X24aELQDWcP6jcj66O6/HJ7ss4+dnfUHFmLz7Y1B9Euzqt+GDTo6g4sxclR1x9z6RnDKAUVvw94Fvt6wV6LJupvBSzynbBgCrYgssu2PYLd/V9bUTSoHTp+0rmYEoNSt2Ja66fILs6c/e8WMNv0NN9BrFxMwEMgQF/wA/xQ0RhNoD3EGt4HuOnmLHtjUoU5y/Fyc9+hHbLeERHS8eHDRqyGdfeWI4p07ZhyBUrABgArASwCNJgaERjvVVynZaWOlGa1XElXg0h+AJGWC3XSx7jaXaq7cPDXrj+oLIYTeZkDE9fitGZuaguK8AHmx6Ftb0ZH2yyrTxHZ+Zi9gMvu3h+cHR2WLxaeZMy3AMl3XIs7HGVMv1480bcscS5m40n7lKwnvZCA322tuq1V/Arixmf4Rl8iB/BcQ8UGI7kFOl/3u6qRsdPMWPbm3C5b+jqedmzV6Dk81dQXVaAlKFPYEa3EaX1JvTChJSh1+Pef1+LHX+pdShAuhU9PRNhTFyDayYNR3JKXF/hzc3o7LDgy+O9AJaIHi+u0m1C2UlD396r7Tq+KNqEdst4Fz8pcSWvY4q4FcDHaLwci/ytWzF70SKnQqNJN9+Izwv2oKPd/ZlTS6sBdy9dbw+ar/92CgBgdGYu7l66HrFxrp4feOzXGziswiVdMpWX4vR3F+I6UWGPXDvCSgCbEhJww85PvA5YwWxvKJzlvHjBNqZs0YPTkJg4QPax4sreXRiIhTiCDlwjeUx09F/xvf8cj8SBqU7Pd6waveb6Cdj2ZqXH6lxX1aZCqlK8z3dLdCzG/3gnvqq45FW7O/dVumYg6l2g94cy9+2AbaUq9zN7H7Z9yx2wdThKhG01Ww3xBw9Xx3/279yK018ITR6cvwdgD7LnNGHqtGxY25vtwRMAHvnFcRjjk2SeFxzs1+s7d1W4TOGSLjkW9rhKmX4OYEVbG06t+ZVXrx/oFKxYceFJLL73MH636j789X8X43er7sPiew+juPCk7OPFaeX5aMGdWOX0mJ6exSj9l/zentB1Z/qcbEydlo0vT5zuW1k6E+85Oj5PqISNjTNi9qJXJM/b29OFoYfe9Lpwyf3jPwV6H3Zx31wAn8jcbmuqEJ/4FoDhsAXPVsjtabpK5+bMnYfklHddvG//mdOuTivytz4puTd/65NOhUXB1F8E5czVfjIpxxQu6Y5cYY9cytTcUI9b9uwArFbc+GmhVz19g9Xe0GJp7WuEIJ7paURN1UKs+vV2bHm/VbISlQvs/4FdOIBSh1Wo8l6svvZ07eq04oO//Uhy24MAnig9gE9vuAnetLtz3x6vw+11OjZbAABD/CbcOL0D194wBdveqETj5VthOzMrv6cpd5wnzpCIvPnJ2L/jbfkzp/MyERUVLd3zXPQK8rc+ad8T1SqNy369gcUASrojd7Yy68VXndoRFi9/DDlW26f/3NYWj2cwxXuq3XEG3PzL3/p0nUqaL9jOcsqvEGqq5mP7pvfx8Pdn2m+TG4Y+H5dxJ1bhQ/xFdKsVg4cHfiamkL79uvpzzAXwDmzBcw8AtDXgpsZinEm9SnG7O3ft8WJiv0B31zyX12lMOApr22xIU7JD7SnZ3Lsv9u3jxrp4DcBVUBmdOUH2zOnkW2chzpBgP6oi3vMU74nmb3sK8x5c7+I9A4f9egOLKdwAO1G4X+tL0AWlPyelZyu9PYNpb5ZQUSb5u1pKX+PiBcDdL/MLF3okr+kqrfwf2AUDSu1fjxy9CwsfvkvRtXqqznXXEi9/21OoLivALdGxeBfAIADvwra+2wPg+Jm9uD2rHoNSt0BSdZu6RbbdXZwhEVmzYvtSrv2PB3aguysVtqMuUrZjM7sx+fYrkD1nM6ZM24jsOZvx4KPXSfYzbVNXrsWYa07A9bEc10FFfOZ04Q+W4abpM+zXP/GmpRg74S7JSlMIomMn3IWJN8k3ogg0X/5tybOYlStXrlT64Iv17QG8lPBjKi9F1RPfQ0LeLCSlOBdzhKMThftx5WjXm+5yvPk5nVz1PL59/HPJbentbdjX2oL0WfO8fpz48U8cOYR9rS1oOHLY/ne5xyohfj13r1FeVoFDB6+GfDLIitlzz2LyVNvPs+bY57gwMAnHptxk/7P7igz87tIYFHRMRwMGoQ0ZSM/YiWd+cQXih47FRVOjx2uNiYnD4NRGXDAdh7VNuBYrBqW+i9y7B2DIsOEun2uMT0KbuRozrrkTH4+6Afkjp2D/yCkYnDYRZksDxqTfiNTR12HqXTcgPnEfhgw7gauvO4UZ902Wfd3qsjMoyreitSkHwGsA6gFUAZgJoAG2CS2HYWuuEAsDTuA/sABHcC+GDAduvSMbo69Ox/BR6YiJiZP9XseMH4eyUwdhbbvO6f74xL9hYFIHzJdMGHrlFbKvIWdwagbGTZqP6Bjpv2N0TCzGTZqPwakZil7H33z5tyWbubkpLu9jFW4ACVWckTKUWq4yVgmlPydx9anTa4j6zCp9nNzrFiYPQl1vL+5vblLdu9ab/rf9/WwXOt2XnrEdW/9xGxIS3KdiLZZWbN90CBcu9ODKK6PxwIPT0B09wOvh2lr3dJVWjO6Cc29d4bYu2AqGujEXr+ADHMBcPISWObMUt9ST628bHb0FPT3jANwKpUO19ULrf1s9Yy9cDcgVuoT7TE01U0e8+TkpLezxtgBIvK+Y09SILX23q+1d603/28TEAVjx/BVY9evtqKmaD+GXeXrGLqx4/gqPwVN4DfE+KQC0qCj8VDMT05/6e+62AiiBuDm+7XjKHehvKj8PBpzB4zgOAHgcO7ElY4ns68oRD9Fuqu9E6cnzsFqega1KF5CbJKNnWv/bhisG0AAJZBPxUKT2A4M3Pyelc0u9mW8qt684GrakYQa8//CjpvlCVs4kbHm/Fds3vS9ZRSoJnuHEVjF6GrYjJk/Debj21L7/fRfAPMzAatwN2/935qEZB/75v+i8f43i9xOCytHiIlgtyyG3F+2uyT4Ri4gCINBNxEORmpZ3Sn5OgS7CkutXeyuAz/r+7m07QLX9b4VV5PJf3IWHvz8z4oInAMQntsPW+kKuOb7Qkm8qkoe04JbbX8CTMdslz7/77MdovVTh9fvyqAepxQAaAIFuIh5q1H5g8PRz8kc1rDvuqlqFVSig/MOPqbwUMfm7ZO8L9w9QfhEVBeksULG5AD7CoNQtyJuXimzLCczubpU8YoalHoOLvD8qorbJPhFTuH6mpINNKO+FqhkcLXc20VM6VsnPSc2eqjfEe6WXz1Wiq7ZWcn/H8BEYlZ4BwH3jBOFnVn6oCM3oxUsLlmDIYOfKPVevcbBgH7Zv3ojV616H0Rhvv91qbcdzTz+ChUuWITt3ptPzPBmf1gLcku5VIZGW2luNcLcSTBt9Bvc+fC86G0yYd/aA7KPuPvsx/n6pAgOuUF4J7u7sKY96kDsMoH4WrA42gaBmcLTaDwyefk6WTwtwQ4CLsLzZK3VF/DOL/ddRrLC0YkN0lFdNGLZv3oj9e3fh6ceWYd2rG2E0xsNqbcfTjy3DwYJ9AOB1AB1otBXXj09rQVnqUFgv13n1fC14OvQ/dsIIxBkS0Gauwt9H52BNZRes1rGwNYjvhtFYgRHpsei+XAF4EUDdNdmXO6tKJOAxFrJTc+ymZP8etB0qcnl/wu3TbYFK5bUIlFyTmtWzr4TrXDvzbtx25J+Kjq44EgfL7NyZeOHlN/DsUz+0fy0EVTUa2qLxfmGULgKo58bn1yHOkBCwBuk86kFy3B1jYQAlAN6dXQzmtQg8XZPaM6j+us5PDAZkdHQgo+8+b8/+Oq44AfgcPAF9BVBA/nym43lM9xNbnCe8EPmC01jII38PovbXtQg8XZPjdBZfHSzYh6cfWwarVdp9SxzoxNeZ19Fhr9wFvC8aMhrj8cLLb0hue+HlN3wKnnpjLiu0t9tz15KPVbMUKhhAKaSO3agZIyZ3BtVX4n1JIYgKwXP/3l14+83/5/L8KOD9hxCrtR3PPiWdc/nsUz90CuDhynKxHON2rkDrpQqXY9MErJqlUMEASiF17Ka+ugJH8mbiNYMR/wDsfzYYjDiaNwvmynKn56hdPbs7Y7p63evIzp1pX4k2NzdJ9imXDh3m9vwooDyYO+6BFh+rkrx3JATRQUWv4fmmC4qOobBBOoUKBtAIF8zB0UpMnDEXhqho/LjDinsA+5+fdFgRFxXlVJCkdvVsKi/FsZ/92OXjjMZ4rHt1oz2QZU3NsAe45T/7JW50EXy/ZTTipQVL8Ofv/gSl9y2WDfiOnnv6EUnBUFJSsuS9n3v6EY+voWeWi+X2YylKmiEIVbNKJ7wQBQqPsUS4UDt24+2xGDVnUAHg2JpfI6PejKNrfoW09W/LPkbYl8yammG/7YWX30D1Z8Woc/MzG+Fl5fHCJcsAQHIOVAjgwjnQcDao6DXc2dYAwNYM4YOi9R5b8ol72bJqlrTCKlwKKd4ci/F26orAVF6K8nty8XhnB/4QZ8C43UWyjwtUZWwwNbRFo7e3F/8oig7JSlzLxXJ8580l9gAKAPsTU/D3H2zxqhkCUaBwGgvphjfNDeqrK3A2byZOfrATaR1WmAxGdM5bgAGDbF2AXK2ej635NXI7OwAAN3R2oFBmFerubKa44UGoG5zQg4a2aGRelYiSy1pfjTPx6lOgdBVKpDXugZJmfG0U77hfKuyT3vzL3+LmX/5WNo1qKi9FUuF+TOv7OgvAwE/2Oe2FhtO+ZGy04iRTUIn3Ph2pbQxPFExcgZIm1LQNlHsNb0eHiVefArlVaKTvSwZDl7kKuyYvgHz7fXjdko8o2LgHSppQ0zbQ1Ws4cvWa4r1PR+72QvWuxRqFk+cG6qapPFEoYSciCin+aHyg5vhN+aEiTHTxehOjgLOHCr2+jlAX6HmqRJGMKVwKOvHREyVHTuSoOX6TmjYCB4ZegebzJqfHnxiVgauuTPPqGkKdkCYf8MYOIHaq1pdDFHYYQCmo1OxbylEziqwzKgp3XqhFnsx9Q8x1aBlztVevF+qE/sB//NM6GB7dqPXlEIUdpnApqLRsG1j+5no09/biN9HRitsE6pX4g8qUgnzUnytV/VrmsvBLbRP5AwMoBY27fcvo/F0BbRtoKi/FnNJTuAfAXT09mATPbQL1TPxBJbvBjIR//A8m3pLu9euIm7wTkRRTuBQ0rvYtzQ316Ny7C9ZDhQGrghXvu94KYAtgn90JqEsjhxLxMHG5Dyo3FuXD9INjMKZe41VHIqHJ+8/Y2IDICQMoBYT4F7rA1b5l8fLH8BNLKzac+AL4tx/4/VrkAsowAH8G0CHqXhTsvr/+4nim1lV/4N+/uQ6YvUHx61ouluM7oibvf79UwfZ6RCJM4ZLf2X+hK0jJBmKWpyO5fdc8AAMh7V6k1xSueJi4uzT51MJ8tFw4q/h1HZu8Kxk1RhRJuAIlvxN+oSs5nuKPIy3uuAsowgBsPadvHT+AHJ98IxJcHO+xdgFdF8sRN+IWj68rXn0KuAolkmIAJb+SW1G6Ckz+OtLijuO+a23hAUyu6F+F9SAwgTtYHD+AnDzxBW528X00tEWjojBK0R4om7wTecYULvmVOF3q6XhKMI60TJwx195cPm3pv+NOc51kUPftfY/TYni4r9QOE/eETd6JlOEKlPzGmxWlt4Oz/SHUhof7Su0wcU/Y5J1IGTaTD0NyFbDB4E1zd28GZ5MzNcPEQ324NlEo4kDtCOKPMWFq39ebFaWaVnzUT81qOtSHaxPpDQNomPGmAtafwi09GurUfgAJ1eHaRHrEABpGvKmA9TeuKIko0rAKN4x4UwFLRES+YQANE4E60kBERPIYQMOElmPCiIgiEQNoGFBSAUtERP7FIqIwwApYIqLgYyMFogjSYo1CZzfwZe1AlHxWo/XlEIU8d40UmMIliiADjb1ISezF+LQWGFOHan05RLrGAEoUgaKiorS+BCLdYwAlIiJSgQGUiIhIBQZQIiIiFRhAiYiIVGAAJSIiUoEBlIiISAUGUKIIdc/0Hp4FJfIBAyhRBBqc0IOoqChkXpWo9aUQ6RYDKFGEio1W3MWTiGQwgBIREanAAEpERKQCAygREZEKDKBEREQqMIASERGpwABKFMHGp7Vg4i3pWl8GkS4xgBJFKA7XJvINAyhRhONwbSJ1GECJiIhUYAAlIiJSgQGUiIhIBQZQIiIiFRhAiYiIVGAAJSIiUoEBlIg4XJtIBQZQogjH4dpE6jCAEhGHaxOpwABKRESkAgMoERGRCgygREREKjCAEhERqcAASkREpAIDKBEB4HBtIm8xgBIRh2sTqcAASkR2HK5NpBwDKBERkQpRvb29bEFCRETkJa5AiYiIVGAAJSIiUoEBlIiISAUGUCIiIhUYQImIiFRgACUiIlKBAZSIiEgFBlAiIiIVGECJiIhUYAAlIiJS4f8DgdM1DS5OTFEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 576x396 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_classifier(X_train.iloc[:,:2], y_train, lr_2f);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: the colours aren't very good - the red triangles correspond to the light blue predictions, the black X's correspond to the red predictions, and the blue circles (correctly) correspond to the blue circles."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can do the same with a random forest:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 220,
   "metadata": {},
   "outputs": [],
   "source": [
    "rf_2f = RandomForestClassifier()\n",
    "rf_2f.fit(X_train.iloc[:,:2], y_train);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 221,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdAAAAE8CAYAAABw2+D5AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXxU5b0/8E/WScImSQAJJGxGQYJBRRGSEGRTFi0axHAvYlcVL9ZKSwu1VW7vVaj0Qq3+BOwq0rIIEiq7gJBFXCISiQRMSCAkw5aFkHUm2++PM2fWc2Y5ObPm83698iqZOXPmJKn55HnO9/k+QZ2dnZ0gIiIilwR7+wKIiIj8EQOUiIhIAQYoERGRAgxQIiIiBRigRERECoS6cvDp87Xuug4iu4YVZ+H8W5uB2DhvXwoRdSPJ770v+xxHoERERAowQImIiBRggBIRESnAACUiIlKAAUpERKQAA5SIiEgBBigREZECDFAiIiIFGKBEREQKMECJiIgUYIASEREpwAAlIiJSgAFKRESkAAOUiIhIAQYoERGRAgxQIiIiBRigRERECjBAiYiIFGCAEhERKcAAJSIiUoABSkREpAADlIiISAEGKBERkQIMUCIiIgUYoERERAowQImIiBRggBIRESnAACUiIlKAAUpERKQAA5SIiEgBBigREZECDFAiIiIFGKBEREQKMECJiIgUYIASEREpwAAlIiJSgAFKRESkAAOUiIhIAQYoERGRAgxQIvK6vEotlh47Dl17u8XjuvZ2LD12HHmVWi9dGZE8BigRed3O4mIcLb9kEaJieB4tv4SdxcVevkIiWwxQIvK6VWmpSBkUZxyJ1uv1xpFnyqA4rEpL9fYlEtlggBKR12lCQrB2croxRNO2bjeG59rJ6dCEhHj7EolsMECJyCdoQkKw2mqkuTotleFJPosBSkQ+QdfejuU5uRaPLc/JtSksIvIVDFAiGawM9Rzz72nKoDjkZM63uCfKECVfxAAlksHKUM9ZkZNrcc+zV3i4xT3RFVYjUyJfwAAlksHKUM/JSEzElIR4i4IhsbBoSkI8MhITvXyFRLaCOjs7O509+PT5WndeC5GsYcVZOP/WZiA2zqPvKzVdy8pQou4j+b33ZZ/jCJTIDlaGEpEcBiiRHawMJSI5DFAiGawMJSJ7GKBEMlgZSkT2hHr7Aoh8lVj5ucrsnqdYGboiJ5eVoUTdHAOUSEbKoDikDLKt+hVDlIi6N07hEhERKcAAJSIiUoABSkREpAADlCiAsSE+kfswQIkCGBviE7kPA5QogLEhPpH7MECJApi45EYM0bSt2y2aQ7CnL5FyDFCiAMeG+ETuwQAlCnBsiE/kHgxQogDGhvhE7sMAJQpgbIhP5D7shUsUwNgQn8h9GKBEAYwN8Ynch1O4RERECjBAiYiIFGCAEhERKcAAJSIiUoABSkREpAADlIiISAEGKBERkQIMUCIiIgUYoERERAowQImIiBRggBIRESnAAKWAILc9l/l2XkREamKAUkDYWVyMo+WXLEJUDM+j5Zews7jYy1dIRIGGAUoBYVVaqsVG0fV6vcVG0qvSUr19iUQUYBigFBDE7bnEEE3but1iI2lxL0wiIrUwQClgaEJCsNpqpLnabCNpIiI1MUApYOja27E8J9fiseU5uTaFRb7GHwqg/OEaiTyNAUoBwfwXecqgOORkzre4J+rLIeoPBVD+cI1EnsYApYCwIifX4p5nr/Bwi3uiK6xGpr7EHwqg/OEaiTwtqLOzs9PZg0+fr3XntRDJGlachfNvbQZi4ySfz6vUYmdxMVZZ3fPUtbdjRU4uMhITkTJI+rW+QGoq1NcKoPzhGonUlvze+7LPMUDJLxgDFJANUUWq7Ny7Cw4Bogeo914O1Ov1SNu63fh5TuZ89AoP99j7O8MfrpFITfYCNNSD10GkWFniXIx4AUKI1lxVJ9hqrgIARrywUPJpY2B7gFwBlC+N7vzhGok8ifdAyW/Uxydj8KQkVc85eFISyhLn2nzUxyer+j72+EMBlD9cI5GnMUDJ/3S0C1OvUh+AMLKUe978o8Pyl35B9hHb97I3xasSfyiA8odrJPI03gMlvzOsOEvycV3BKVRkFwKQn5a1VpY4FwBQWXIOZ76fgdGbdiFueCIAILalDOF73hPOqeZ9Vyv+UADlD9dI5A4sIqJuwbzQaMQLC43h6Iy8Zc/juV1bseHxTKS88Y7tOT1cUEREvoFFRNQtiIVG9fHJKIsY5vTrKkvO4a5PDgEAxnxyCNrSYuMo1KJ4iYjIDO+BUkApS5yLKhfCEwAubHwTE2/UAABSamtQtmGdxfOeLCgiIv/BAKVuzXz0KRJHoURE9jBAya0kK1vd8BqlzEefIqlRKBGRNQYouU1lyTlc+/WLLo3mlLxGKanRp0gchca2lOHaL5e5/Vq8hbusECnHIiJymwsb38TSK1ps2LAOcWaVrWq/Rqnai6XQzp2PszLPR5aVoFe7DtcAty5j8SbzXVbEjkLW4cnlKUTSGKDkFvYqW9V8jbhWEwD0c542/ls7b6XDa0yaOhOYOtP+QTJrTgPFqrRUY1guPXYcq9NSsdysaQJ3WSGSxylccgtHla1qvabXpQJUZBeiIrsQ1365zPjvjp/OQ2xLWde+iG5AExJi0VEobet2i45D7HFLJI8BSqpTUtna5WrY2Dih2UFsnHG6VRyNkn2akBCsthpprrbqOEREthigpDolla2qVMOadwoK5i9/Z8ntssIG8UT2MUBJVc5UtqrxGlIHd1khUo5FRKQqZypbrQuDlLwGsNpkmxSx3mVFvCcqhuoKw36fRGSLAUqqcqqyVYXXxLaUQVdwyvAJl1kolZEo/GFivsuKGKLiLivdAXebISUYoOTfeK+zS1IGxUkGgxii3QXXw5ISDFAKWBXZhRiMlQCENaK9LhWYRq0G+jlPyzaf7zZTxOabhnfT0byj9bCvPPAANn3bhmtNt6B/1A08cTsQGcZfn90d/x9AgSl6AFBzFRW5RUBHOwbjPZzPLrQcsRoeh0TThdiWsm61D2jM6jWoXh64LQsdsb73m7Z1OwBh1PlEYgoW7X8Q5fU/BKABoMOO7/6GX92fjZRB/BXanfGnT35lWHGWcRRZYR2I1sTgq7kqHGv+mOFxh7pBeALAPu1dGO/swVV2+uMqGcGK5/Py6FdcDyuGJwCsfOAB/Gh/GsqbFpsfifL6xfj9iTZsS92JyFDeRuiuGKDkN8TCIWNwOvsL1zAa7S5h6BRDaDW3teOD8ih8s3EPzpVq8ETbJacCIWb1GuzT3mXx2Pg/PWQ/XGUMnpQk/EyrtJYjfqlzxcYpeg9n6NrbsfxkocVjPzmQg/Kmv0keX970DNYOHYJpj89yy/WQb7C3GzADlPyPkmlVhqeN0488i5//bxiuVc4CzmpwGDOxuX4fnlw8GEnj7pZ9XfH5JuiyqgCUWzxe+MCfkXR/gsvX8dcvyoEHgNfGHUfp24Z7zlVaDJ6UhJ1j/894nBjQ1o+roVXfgg3/8wQKr1djfOo0/CFSj+WnvkXe9esAFgD4EECE1as0KC68gVsHl9uekALHQ8Nln2KAEnVDzW3thvB8zOxRDa5VPoZN67ZjweJIhIVHuXzewi+Uh8nXgx9DH5iKtppnPY3CLNP5MgwjVevH1bB3y2KUFh3CkMR0TJj3T4xqeQ1rQ0PxZPZpXGjcD+A/IISoOR169lb1MsjPsBMRkZ9TsqfnB+VRwshTQl3191CYf9It1+oUN03R2pM0LhPDR83ArMz1CA0TRpqakBD8Y+Kd6BE+FsAzNq/pE7MbSePu9fCVki/hCJTIzylZw3itJQZCRakUDRpuuveapezIqkHSTw8aP7ceZf63/kUk/TRB9dEnAAxJTMeQRNt1r7dowvH3/xyHZ/c0ova6DmIVbp+Y3Uif1QNh4ZGqXwv5DwYokZ9Tsqdn/4hqAGIgWNNh/ugSLBln2wd3S/BTLk/TJt2fgAUd71s89ruyDOiqq2yP6TA7aJzEyTpkHleR/ptTFhXeU+6IwlNjJuLkoW1ouAn07A0kjbuX4UkMUOoe5Fq1tbS24aXnFyHjyUVITZ/mxStUThMSgl+MTUb15Ss2axjl9vR84u4+2FH+LsqbXrB5LiHqXTxYno3St/NsnluwBPil/j6c/vIrNN4MQo/enRhz371275cu6HjfVBxksBCbMXzJQgBCmEod41VihbdhqVO4pgfunsjNxckSA5T8T0e7y2sH5aY5v795L46cuwgAvhegLtwLvFJwBp91dOAWs8fs7ekZGRaKX034FL//ItSiQUBC1Lv41YRPETkgXvJaDudfxZb1Z1BXnWl8TWG+MJ05JHGU/Yu0WqIiBuaLq5OBCsMx3bQTEvknBij5jaqIYcC8lRiRnAUAQqcgJ9d3Sk5z5hcg73o1UtOnYfW6d919+a4xLNdou2MU/rZPC21VBOJiW/DDWXHoEWE57XqhXItR2Z9hgdUplht2UpEL0ZRBodg2Jxs7vt6Dgjt/gragNqxq2InIkBCgyvb4yFd+hx/MrkRd9XyzRzWoq56P4/u2Y8HiJsmR6Mv56XhtCYTANASx+VKUwqxyvObmaVkjZ/4oYYiTkxig5FfMOxG5QrZVW78YbJ12OyI+Wq32pXZZdutA/PRn/VBx8QcQR3vvHNqDF5f3R8qkMeip6QQAHFv/LPY3NGE/gJkApg8dip2teuMfC/ZCNDIsFE8Nb8Htz8xB4Rfl+Bf+JrmWs/h8E068ussw8rQlVO5uk53mfDk/HUk/fcr4+V+/KAecuZcqBp6jBgtOGr5kIbYEPyX7fMapn5u6VjFIyQEGaDdQkH0EyZOmevsyukxxJyIDqVZtq+8Zg6rPZTbs9vQv0JqrwvQ0DOs0/zwUFeUZZgdoUHExA6+/shNbPmpEW0dPXDl/Djv2ZSEbQnh+COCry5cxfLrw85bd09NOCMkVCTXeDEJXKneVrhEdvmSh3QYLrvjrF+WwbgBhrhAvYt7qaPRZnuFcq0fq1higAa6y5Byu/fpFaDftktyU2m+J90FdCDldezuW5+RaPLb85GmsvSdJeoTm6fZ/He3GEdLHO/ei7NDDkofVXp+D/125DQ88MhfY9EesaGtFDIB/QeiVk6LTYUPRWYd7epq343Mm3Hr07oS9yl13NRVw1GBBbR/lBGP5pCRhIwIiOxigAe7Cxjex9IoWGzasQ9wb73j7crqkK/dAzddFpgyKE+6BHjmCvOvV+G11Df752yWICA+zeI03tjIrfXszFiwBvqm6DOB7MkcJo73aos/wH9/sxxQA1lE7pqIS1Q0Ndvf0rF6+DAsMlbAvQzhOasmJqDFJh+TCfYbuRQcAvAsxtkfEbcGOZ1oQHHwYP3pjI77/cDqmjxvj0tcttzFAn+UZFp9XL1+G1wzX7Q7Wy1iI5AR1dnZ2Onvw6fO17rwWUlllyTn0zpyNiTdqkNc3Gg3b9gfMKDS2pQzhe94TRglOBOjSY8dxtPySxdIO3XUtXi49j8PnLmLqjDlY984mi/Nf++Uyz0/jGqZWN5VqsPbsLsiN9l7/yWu4f+AN1BacNT56s8AwYorsAQDQ3HorkuLjJV5v+V4ii6lSGXnXQ7C28j9wXpsFYDeAhxAfOQPLkw5gXLQeS08WIu96NaYM6Ie19zofoHb7G1vPNHiiU5H5MhbDzMDL+fJ/jPxy7g30qShw/3WRx/Wa+QPZ5xigASxv2fN4btdW4+cbHs9Eip+PQiW3M3MiQCXXgdZcRez4RDydU26xDtQYztmFXiskaW5tw5O7J8qs03wL29IM1bLWlF6vs5tqV2nR3NaOrRci8MGl09A2azFh4EC8kZ5m0bzBXuGSXzJ8f4ZLjHyNI1YKSMl7j8g+xwANUOajT5G/j0Itgk2Nja5rrmJw6ihozTbUHlacZZq69XIVZl5lG37/xSTLdZq9fGcjZ6leuwEZniK190Elv5D8nvQtDYD3QAPWhY1v4jmz8ASAlNqagLgXqkp4SohtKRPC003nd5VxnWZxNq419kH/HnWYdzsQGeob/9lKVjXbad7g9xiSZMU3/kskVVWWnMNdnxySfG7MJ4egLS3221Go2iqyCzEYKy0f9IHwFEWGheKpOwGgEb72n6tkVbOD5g1EgcS3/oskVdReLIV27nyclXk+sqyEAQoIQXmtAhXHvjE9Fty1Hf7keu7q2tuNS0qsd0ZxJ3ddj2RVs+EeqKPmDUSBggEagJKmzgSmzvT2Zfi+mqtAcDBGvGAqDOnq0hUlW4u5k7uuZ4VEwZB5pyfJ5g1EAYYbalO3NnhSEsoS56IscS7q45O7fL5VaalIGRRnHInV6/UWIzWprcXcyV3Xk5GYiCkJ8RYjTTFEpyTEyzZvIAokHIESqUi2566XqlPddT0pg+IkR67i+xF1BxyBEqlMrE41583qVF+7HqJAwQAlUplcdaquvd3lc4lTr9avlVqD6YnrISITBij5hIJs+W4f/sS6OjUnc77FPUhXQ8u8CEh8rfgeR8svYWexzE4ybroeIjJhgJLXGXeMKbUfBv7Aujq1V3g41k5ON4bWCquRoCNdLQJS+3qIyIQBSl4n7hhTtmGdty+ly5ypTnVlWlZ8rRh4aVu3u9RvltWyRO7DACWvMu+aJHZJ8mdywWYehK5Oy3alCMiZ6yEiZRig5FUXNr5pbHifUlsTEKNQR1ydlmUREJFvYoCS10j17LU3CjXuxKKiiuxCxO1YibgdKxG+5z1Vzy3HlWlZdxQBqVHZS0QMUPIi89GnSG4UOqw4y7Q/p1rN3qMHAMEhqMgtMn54ascNZ6dl3VEE1NXKXiISMEDJK5zZMcYjogdYfniIo2lZMdBGRkerXgTkS+0GORomf8ZWfuQV3XnHGPNwiI6IQE1LC/pqNMYwWZWWahx5ApBsjdeVlnn22vs9cXsiVuTkemw3GV9rvk/kiqDOzs5OZw8+fb7WnddCJGtYcZawU0oAbGosTpWKoz0xLPtqNKjV6RATEYHqlha398+t1+stNsPOyZyPVz89Ybw2qUATR8Nqsbctmrf6BxOZS37vfdnnOIVL5GHmazN7m93TrNXpAMAj4Sk3hbxy4gSPTu92dZ0rkTcxQIk8zDocPN3s3V5lrzh9qyTQlN7PZLN78lcMUCIv8/Q6T0eVvSs/PaEo0JRW93KdK/krBiiRF3mj2buj9n6PDh/ucqDlVWrR0dmJCXEDLaZ/f/bJMeRVanFnTLTk9C+b3ZM/Y4ASeZE3mr3ba++3Ki0VHxQXuxxoO4uLcexSBQAYQzRt63ac0F4GAPSPirJ5v+bWNvzn3hzkVWoxtPdAvJaSwmb35FcYoERe5GvN3pUGunjf9IT2MqwL+x8YeCt+PynN4rG8yjY8uWcSSm68A+AxXLhZhEX7H0ReZRub3ZPf4DIW8guBtIzFl+VVarGzuFjROlBdezt+9skx46hTNCFuIP744GTj+ZpbhfAsr19sc46EXuuxbU42IsO4RJ18A5exUMBhBxv36OruLR1Wf4+PH3grTmgvW/ysPvgOKK//oeTry+t/iB1f3+zCV+AmNVeBKq39D+p2+Gce+SV2sPE9v8zOweeXr1g8FhwUZLwnuiInF2snp+NadSgAjcxZNLjW0lcILA+2VnSoox3DlyzEluCnbJ4aGdeAPsszvHBR5G0cgZJf8qV+rmrw9xG1rr0d15uaAQhTtmLxkTidmx4/2Hg/s39ENQCd3JnQP8I3bxWVvr0ZCzret/kYve9Vb18aeQlHoOSX7PVz9ZcONub3G81H1KvSUrHy0xN4dMRwfPBdsV+MqFfk5OJMdbXF99/85zMlId54/U8kNGFX0y6UlWfanCch6l3MG9YCRPvY1xobB1RpUfr2ZvnnqdthgJLfEjvYmPdz9acONtahqWtvR16lFt/btRu1Oh1OXbuOGkNbP18fUYujS/PiIzFExeIjUWRoCH63IhhLfrMdddXfgzCdq0NC1Lv41YRPEdnPR8OIIUlWGKDkt+Q62PjLCHRVWqpxhLYiJxe/mzgB8/69x9gTt8YDPXHVkjIoTnKELLdrzOSU27BgcTgK87eh4SYwf3QJHizPRuSAeE9cLpEqeA+U3KIg+4hbz+9UBxtnKiXtVVXWXHXr12DdSH3qBzuN4SnypxG1q8LCo3D3xFSkPZyKJY8NQ2SAfp0UuLgOlFRXWXIOZ76fgdGbdqm2p6dxHajB0q9O4+jV66YRWl0VdK16LD1ZiLzr1ZgyoB/W3jsGI15YaHyN+evNmR9jzni8m6furLcVM+cvI1CXGP6QGb7E9H0vfXszEBziW5W3RLC/DpQBSqrLW/Y8ntu1FRsez0TKG++odt5hxVnGf3+cfxpvv70Zq8beaQyXwZOS0DlqNH74+434wcx03Lbgt3bPAQBliXNl3y+2pQzhe95DRXah20JUqsq2r0aDDx6dg1c/PRG4W3tZzwYwPMlHMUDJYypLzqF35mxMvFGDvL7RaNi2X7VRqBQxEOvjk1EVMUzymILsI0ieNFXx+c+/5Z7RkXl4iptoi5tq3xkTjZjISLR1dOCE9rKx3Z8zHYGISD3sREQec2Hjm5h4owYAkFJbg7IN69z6fmWJc1GWOFc2PCtLzuHar1+EtlR6Ky1nzj/ihYVAh/q7gpj3nf3tA+MxJSEeux/7HlIGxeFMdQ1yKioBmNZQOrM1GBF5DgOUVFNZcg53fXLI4rExnxxSHF5quLDxTSy9ou1SkNfHJ6t4RSbmjeQnG/63t6F5e3r8YNwZE4MT2sto6+jAXf1i/bpRBFEgYoCSasxHnyJPjELlmAe6t4Ncir2+s28+OBl/f3iGsUI3bev2wL0fKsPfuzNR4GOAkiqkRp8ib4WXGtPJsS1luPbLZWpfmlPERhHmAnlZizXzRhNiiHIam3wJGymQKmovlkI7dz7OyjwfWVbi1mIia/amk125jl6XCnAN8EoXGn9vFNFV5o0mlh47jtVpqVhudt+Y09jkbazCpS7pSoWrO4lLaay5urTGnVW49lg3irAOj+4SolLTtd3p6yfvYxUuuUVXK1zdRc3pZGMBkRuqcO1ZYRWWvQzFReI90RVWI9NA1d2nscm3cQqXFBMrXDdsWIc4FRsmdJWa08nGKdxgz/7CdqU5e5dZNzXwoabp3X0am3wbp3BJEU83TPAm4zSuDwWLqqq0iFm9Bvu0d2H8nx4SHvOBr5XT2OQLOIVLqvN0wwRvctc6UE9wtBTk6MVL2FSqwQ9eLcemdf/EgB8/IRwg1XjfDe9vbykKp7HJ1zFAyWW+2DDBXby5jEUNsktBDh3C0fJLeDm3FmvP7sLhXXORdzATI38SjdOPPAtAnXWYXVmKYt5ownoae0pCvLrT2EQKhKxcuXKlswdfq21x46WQvzi96hU8dupLi8cSWppxuLEBCdNne+w6hhVnIeqTreh15hh6nTmG+jsnq/4eA8qOo/aLb3xiSlOJyfHxKKqpQV6lFkU1NZg0eBCWZecg78o1REaMRXPrZwAiDEeHQtc8GnmnvsO82BN4+7vzOFp+CUU1NZg2JAGhwcHG8MuuqERLexseGjpU2fsbRpZr0ichNFj67/iE3r3w0NChNs+HBgfjoaFDkdC7V9e/QUQO3PrY47LPcQRKLvGVhgmxLWXQFZxCRXYhKnKLUJFdaLPTCtnuOSp2NBraox+aW47BFJ4m1ypnYUd5D6xKS7XYY7Ver3e5naDc+/vyPUx2QCJnsQqXXOJrDROM6zOdvGcnblEGANp5K+0ea70Hqb8Sl4KY7zl6/6A5uPBdH7lX4FpQPDQhjVg7Od0YGuLrXQ0/qff35aUo5tPO4tdpHZ7cCYcABii5KGnqTGDqTG9fhqTzb21G/zfktzUDhGUp57MLAQD955TJHhvbUuaVBgruILUU5IsrBwDUAZAKUR3696gDEKpK+PnbUhR2QCJncQqXAoPhHqU4ulSNRHj60xSf9VKQnMz5SBkUhws3LyMiJAWAbV1DQq+/Yd7tptdLhZ/11+7q+8t9D32BP047k3cwQCkw1Fx16+lPV1Ya/+1PTc7tLQVpaf8WkaEPANAZjtYhodd6/Or+bESGhqoSfv66FIUdkMgZDNAAV5B9xNuX4Bkd7Rg8KcnhfU0lKm/cwNUTn0NbVwcAqhTXeIqjpSCvpdyCn4/7Pp4a9Rx+Pu772PZINlIGCXd21Ag/f12K0tWRN3UP7EQUwCpLzuHM9zMwetOugOsSJBYDVRjuZwLAiBcWoixxrsPXDivOQn28/Xul5us/TxScwXOVV7Bh0K2YkHwnAMOI82Qh8q5XG1+T0i8Ga+9JEoLC6t5pXqUWO4uLLVrziecRW/N5rDDFyYKrPB3cd81S1+AjS4XYAYnM2etExAANYOKOJK7uQOJPxKUrjgJRidiWMlTn7UfDL1bhgfpGnOjdA73/8GsMiR8IAKhrbMLgeUuMx1fseBt9ekQBgKl61xAK4vSu+S9g81/U4ihNsZqrdhveN7e144PyKFxriUH/iGoMXb0FmogeksfOivsG1cuXuS/QqrQYPCkJO8f+n/EhYwtBkSvvrVLXJNHSwmLbn9V1LZbmFyDvejWmDOiHtfeOsX2hj/wBQOqyF6Cswg1Q5us1leyD6S+cGXEqVRUxDHkHCvFcfSMAYMLNRmw4cBodbyyGTteCl55fZHF85ltZWPfOJmg0Eej/RrIwQs4tAqIHuL+ys6Mdw5csxJbgp2yeKsz/GtvWV+Ba5SwAGgA69PnRbqTP6oEhiaNsz3X/XRjftatxqHnW0yjMKjdd4wN/RtL9CQAMYVpz1aXq57rVO3FW27PL15Vx6ufIMMwqWDTyDwJ2/fl1zHhjP0bP+jE+H2cK/JFxDeizPKPL703+hwEaoC5sfBPPmfWq9bUdU/yBXMvCC2cL8caa3yH3+GGkpk/D7//4F/zqZz9G7vHDeOn5RVj3ziZURQxDHCCMCqu00ADK11Q6GF2Kvh78mEUoAUCrvglb1pehrnq+2aMa1FXPx/F927FgcRPCwqOkT6jyyM6Rwi+Ea1+wZCFK397s9PsPnpSEzTnB0FWXOz7YgVlzn0ZKdiFS+sUAtZaFaUXD5yPt4WlAh+laAaA4JhYLAcvrVWs06uSOBKMAAB5+SURBVOGfAbmGU7gByHynFFGg75jiDnKbct8dNxintBVITZ9mHHGKI9Lc44cxdcYcrHtnEwDTFLO4prS+Z1+LNZU5mfPRKzxc/iIMv0DlRpfmzH+pi07m5SDvYCaEkac1HVIf3oa7J9qOfjUxsUgcIROsKpC6VnPiaFSNc7lK7r3tvY/4/TKORlUMUHGnHPKOZx6SH2dyBBqAzEefIo5CXWOvZeHCmzcQlTLZGJ4AoNFEYN07m7D8pWeQ8aRpalecYh486RTOHy90raGAYWnO5z89iL9+UQ7A9aBovBkE6fAEAA0abko/o6uuQmG19HOeoHYouvu9xe+XcTSqon3au7z6/ej2Hhou+xQDNMA406uWo1DH7LUs7AtgyYQ0Y3iKxBCV0tLaZixCsa7sNG8ZZ23wpCRDeCrTo3cnhHWe0iPQnr0Vn5qo22OABhif61Xrp9RuWfjs1oPG8BTD0vye6ArDSNTIyfuejoy5714U5u+2ugcq6BOzG0nj7u3yexB1VwzQAOPLvWq7s0X3j0Zz1U2sSkqExlCcogGwNikRK9pakTEg2qZgZPiShXg5vwtLWwCEhUchfVYPHN+3HXXV34OxCjdGqMINC4/s0vlJBot/ugUWERF5iPV2azcGJdstDlHzvlervgmF+SfRcBPo2RtIGncvw9NN1C7A4v1P7/rTb+XvgTJAyacVZB9B8qSp3r4Mt6htCsKOrBrHB5JbtOqbcPrLr9B4MwgRUS1AUBBaGjXo0bsTY+67V355D3Ur9gKUU7ikCncEXWXJOVz79YvQBmArQvKui8VFOL6vEXXVmQDOACgDMBviFHdhvp1GE0QGbCZPXWYMulJ1dyG5sPFNLL2iRdmGdaqel7q3Vn2TITznA2gDcBHA4zBVKouNJhrRqm/y2nWS72OAUpe5I+ikWhESqeH0l18ZCqoA4BMA0kV3ddXfQ2H+SY9dF/kfBmg3Z2+7M2e2QnNX0F3Y+Kaxk1JKbQ1HoXa06ptwMi8HOftzcTIvh6MmByybS3RASaMJIoAB2q3Zm3p1dlrWHUEn14OWo1BbF4uLsGX9GeQdzMSpE4uQdzATW9afwcXiIm9fms8yNZcAhF+BOpkj2WiC7GOAdmP2pl6dmZZ1V9CZh7KIo1BblvfyeP/OWWPuuxd9YnYbPnsQwH7J49hoghxhgHZT9qZej279h1PTsu4IOmdaEZLA8l6eJd6/kyc2l+gTsx3CQoRWAO/DNBLVITj4fYwcW8O1smQXA7Sbkpt6rSw5h6bXXkacg2lZdwVd7cVS5E+eho3hGnwEGD82hGtwcvJ01JSVKDpvIFLaKJ6AIYmjsGDxnZgwbRPCI5ogVOEeBvBvAIfR0ZGBs6eiOYonu7gOtBuyN/V65g+/w4rmZmwHMNTqOfO1mO7quZs0dSbyDnyEZ/VW96X0OmwIChJaFRIANorvqrDwKASHhEPfIm73NtvieWEUL73dGxHAAO2W5LY7W/XGSkw8LlTeDgFwAUKISm2F5q6eu9xNxnlsFN91aozizTsasYtR98IA7WbsBVSPY4eR3tYKABgPOByFugN3k3Fed2kUby+gWvVNOPXZCVwsrgI6WzAkcRDGTpjodIB1dRRv2dGIXYy6G/bC7WYKj+xH84kcm8drbtRi3J6dmNzWZnzsuEaDL2fORfQtfQEAkRPSOIWqIrV64QZyo3hTQNn+gQAAR7Jq0Vj/hPE5YC+iepZj2mOJTgVYq74JW9afkRnFb8eCxaNlv5eOX3snR6IBgL1wyUhu6jVv2fMW4QkA6TodioKDcN9vXvfU5ZECYeFRAXOfzqLBew8dzpzU4GbNIrMjhGU6x/ZuAjrD0Fi/0OI54HE0NWTh2N5q/MfzTQ4DrCujeKEKOlPyOd4/7R4YoMT7juR1rfomZO/fi5Jvg6FvuQ3ADAi/nvYC+BrA3RbH36zpBWCWzNlm4mbNfhTmn3QqwISK3CYU5m9zaRTPKmhigBLvO3pAIG/LBphGjjdr9ai/UYFetySgd99Q2YIa85GmXn8ZFaW34mbtz2Cait0PoZTtcQBZAG4H0MPsDCGwF15AsEsBpmQUzypoYoCS2ypqSeDubdlcrQJVWjUq9zrTfcqRAC4DWAR7BTVShTdCYJ6BMNLUAJgLU3DOhLBG03yZSTvshRfQ4fYAYxU0MUCJ3Exsi2i9FEgNrlaBKq0alXtdyvRQ5H3chrrq2QA+hhB8IrGt4HYsWCzcj7RsP2g6zjIwxZGmeXC2W1xP7+h6oPND3KxdIHG1+9E7ul7VAJP746E7VEGTPAYoBSS1p0yVnk+qZaJao1BTGInh1QEgGHXVs3F8315jaNkebxle1iEn/z62rzv67/fQ0vQIHG8LJhTU2Cu8sR1pamAaaXYYHhMDKgYAcCRrs2QVbuqMBJz+Ml+VtZmO/uhQcv+UAgMDlAKO2lOmXTmfedMKqYYUXSGE0UgI4TkT5tOhddVDbIpolFaN2ntdS1MmhNBzblswR4U3liNNHYR7nXsBtCI8Yg0SR3cibeZsY0A99WITCj5/Hxe/uw5AhyG3D0Lf2HjkHWpTZW2m5R8PjRD/UKmrjsLBHZVY+EI/RPWMZbVtN8VeuBRw1N7gW+n53L0t281aPYR7jnNhvhuL8Pll1NVatkNUWjXqOPR0cHZbMMutxGyPEwJTtBdAA4BhAJ6EvmUZKi4MBWBauh4WHoVxaVOQ8aMn8ehT89DZCRzeVYq66igA4rIs0w41TQ1VLu2damrY/zWE8JwO4FEA06FrjsV7f/wAF4uLcLH4OPZuWYy2Vsuvra1Vh71bFuNi8XG770P+iQFKAUXtDb67cj53b8vWVHkMctOmwEw01FVaPOIovOSKbuy/7iMEh/wGwASI24L1xAEALRAqaA+gT8xu3JE8CifzcnCzVo+IqK0y59oLYXsxHYTdUb6E8CuqEsLoT36XGXFf1M8OL4K+ZRmEoPsYQvDB+NrNbx9yae9U4Y+HNgAXYfuHyuNo09+FY3urcfqLf6K06BD2bTWFaFurDvu2LkZp0SEU5st9zeTPGKAUUNTe4Fvp+dTerSbp/gSLj7hbG7Gg+gDCcQFCuOyBsJPIHsPnGvTqk2D5vhb7YFrqE7Mbd9w1SnJ0Zu91oeH/g472IoSGTQMQjXC8hR/jxwjCQwB2ITT8FYwcW4MdfylD3sFMnP7iJ2hpGongYMvtw/pEb8Od95Zg7MQdiO63AkA4gJUAnoBlGGpQV6uzuM6mhirJfVGFwLsIMXwBDXRNd1kc42jvVOGPh0OQ/0NlPm7W9MbAhEwMSUzHxeLj2Ld1MXQt9di3VRh5DklMx0Pz/ijzes9o1Te5NPIm57CVH/kt68KeypJz6J0522LUl9c3Gl/94hU8+OQiqVPYJXe+hm37Hd4LlWuZKIqckIZBE4RGAObt/JLuT7A5dmRcA4KCghAabPpP9asVi7Hk31sxE7NwAD+B9T1QYCBSHz5nc29OrjXeyLE1OHsqWrKadEjiKMnXjYjbgtd+dAl/3f8BPs4/jdvjb8PU9ghc0BZiP4C+sXfh0af/H7L+oZVY6tEITdQ63DFmIHr3DTMW3rTqm/CPtV+hpelpie9aFoC+iIi6YLj3KlxHRNRWtDSNhNDB2ZoOpsIk838L1yAUP+kx7I4CPPTEEzaFRq36JvztD/sNo1o5/8bYiTcwYep9xtAUTR83Bv/8zRJEhIfZeb17Hc6/imUbbsV57QKY/+zWPHcF08YN8Np1+YteM38g+xwDlPxSZck5nPl+BkabFfbkLXsez+2ynCorA7A1MhL37D7mcgGQ1PkAYMPjmUhRqRCotikIAFBQGoSzOQdRdS0IA27txNz5KYiMMjUO6Kkx/WdqHux70BMZyIced1icNzj4ffzgFyMR1TPG5j2te+fecdco7PhrmcOerq36JlRfK0VtVQfuGNGKhd/8P0SGhEDX3o6lJwuRd73a+LrxoWGY86fPcGD3aeQdFMPOmg6pD1sWLp3My7FzfA0Q9CHQ+WOJ57IgjFR7SDz3bwj3LbMgdDiKgjCavQjzPzzM/2Awd2T3BzjzldjkwfZrAPYj9eGbwtcRFYa3fmH6Iyhn+iT0CvNerWZzWzuezM1AedMLNs8lRL2Fbak7ERkaIvFKEiXvPSL7HKtwyS9Zr62UmzL9EsCK5masXfPfiFu/2enze6K9YYMuCEFBQfg89xu8ufoaKi6aRne7tu/Bi8v7Y3zqXYajTQFqXtk7Bw2YglU4gH9YnLujYz7OfSNdVWvddedkXo5T1bk9ByZgwfzBCAoSQn8wRqD6V8ugCQnB6rGjkfZxtvF1B9ta8d6BPyAyynxa1Zpt4ZL9gqVPgc6nZJ6TarYAiEtgIqLeM4xSoyCMPMV7mqZrkVvOM2nmbFSUOl5z2taqw76NP7R4dvGNKDz32w8QFh4hc93u9fHOvSg/9LDkc+VNz2Dt0CGY9rhcS0QCgGQ7zzFAye9IFfZItSOsuVGL+/dnATod7v0026XQ80R7w9Z2YOfhJry/6hpqr2eYPaNBxcUMvP7KTjy1ogkZ0yJR2ySE7ZXztsH+X9iDozhnNQp1vhers9W5uuoq7MgCNDGxhuduA2bvwtD4YPzp15b3CBcAeOHwATTcPQamUZ4128Il++3x9Hav07rZAgCER2zFvWl63HnPWOz4SxnqqsfD2TWrorDwKEye01t2zenk2YkICgq2vOf5xJs4+MGLKPzyANb8fA5mZa5HaJjctbtPSWEd7H3Pigtv4NbB5Z68JP/zEHdjoQAitbYy5Y13bNoR5i17HpN0QqFKemODwzWY5vdU28PCu7wLjb3mC+LU7VcHD6H2uvTor/b6HJw8tA2dDanG0Io/sA5PWFX2zkG1xCjU+V6srvZ01VVXGf/d1qrDmj8uxsXiXMwEsAVCeO4HgKZajKvLQ1HMCKfb3dlrjxcS+hXa22bLXqcm8iR0zQ/Bcko21jglmz7rmqFrUKjMOQC5PzyGJI6SXHOaPH46wsIjjUtVhiSmG8NyVuZ6Y6ge3PEzzF6wXuY93Yf9et2LVbhuVpAtP39OJs5+n5xdW+nqGkxjs4TSYot/K2XvHDeag3FW2xM7smpcGv3VFn2GpMMHJI/8L+xBOM4ZP3elF6uj6lx75zm442e4WHwc9weH4kMAfQB8CGF8tx/AqaJDmJBSiz4x22FRdRuzXbLdXVh4FFKmhyIi6j2L44EstLfFQFjqYklYNrMXyRP6IfXhbRg7cRNSH96GBYtHW9zPFLoG3YlhdxSYnduafKiYrznN+NEijEubarz+pHGZGD5qhsVIUwzR4aNmIGmcXPcl9+rKz5YcC1m5cuVKZw++VtvixksJPJUl53DhhR8gcvJ09OprW8wRiAqyj+DWIfJTHlJc+T6dXvUKHjv1pcVjCS3NONzYgITps10+zvz4F/JP4HBjA27kf278t9SxzjA/n/U5WtqCENtLj/PXe+DqhSJcOj8K0pNBOtw2+lsMjBeWrqCmEJd79kbOyPHGj6y+w/Bm9VDktKXhBvqgGUPRJ+ZDpM/qgej+A5261pCQMNwSU4crlaega77NcC06p86jieiF5pqLmHrHFHwSfw8ODh6LI4PH4pa4JNQ03cCwhHsRM2Q07p5xDyKiDiO6fwFuG/0tpn4vWfK8F4uLkHNQh8abkwBsBFAL4AKAaQBuQNih5XMIzRVCEY4C/BfmIh+PInogMP7BVAy5LQED4xMQEmJb+RoSEoZhI29H8be50DWPtnk+Iuqf6NlLj5rrlYi9tZ/kOaTcEjMUt4+Zg+AQy59jcEgobh8zB7fEDHXqPGrrys+WBDPT+8o+xypcNxKrONWs2vRlUpWxznD2+yS1rMR4DrPlJc4eJ3Xe7N59UNXZicfrbzq9ZMXedcqdo0EXhNZ2YMv2CmxZf8ZOBexo3J16B0YNarRYxmIuvzgI2zfs7nIvVuvqXKXnMV+KU/iF8/fXWvVNZt+LPRCqas1H5+JjbQCOAWjHTLyJfTiKmViIhoenO91ST2pZTnDwdnR03A5hOYx8Va4/Uutn2x396be8B+px7mwi7quU7DriyvfJ2cIeVwuAzO+pTrpZh+2Gx5X2rnWm/21PTSdqm4Kc3tEjNLgTbR3CfdNbIjuM52nQBUET0UOVXqxK9sS0pomJNa5b7ezsBO5PcDpETT13GwEUQmyOL3Qn6mH43/0QqmdnIxxFWIJTAIAl2I3tQ590+jrNm8DfrG3FudOXoWtaCqFKF3Cmyb4/UeNnS7Y4AnUT6zWEgT4KdWbUJcXb3yep0ernAAYAGArnGyfYO5/cOfo2laFPRQFezk9Hq74JZ4vOIqnjO8TFtuCHM+MQFWEafRUOeQythiLTsBDTutAGXRBOX+rp0kjP3UxVupYFR47k7M/FqRNjYL0+07S59t0Q1m+WAZiNmXgG+7DJ+Pqfj30crY+vcfl67a89tV2rSt2LvREoi4jcwN1NxH2RkpZ3znyf3F2EJdWvdjyALwz/drUdoLP9b2NbylC9fBlK396MeXOjERYehfdfHY4f645gVmUervzlA5S+vdn4MaZ8F/pGdVqEp6/SVVcZP1wREdUCIRylmuOLLfnuRu/oBtw/4fd4MWSnxetnffcJGq+Xuny9SpvsEzFA3cDdTcR9jdI/GBx9n9SohrXHXrOEIRBKVwDn//ipLDmHkIN7JJ+zd44+yzMwb2606YHYOMsPM74enl0SFATbRgiimQA+Rp+Y7Zg8OwapTQV4qL3R4oipTbW4Jcf1pSJKm+wT8R6oyjzRwcadlGwcbX7PT+To/qEz3ycl91RdYX6vtPpSGdq0Wovn9QMHIT5hKAD7jRPE71nJiRzUoxN/mPskom+xrdyTPEdsHPKKTmPn9Nuxauyd0ISY2qrp2tuxIicXGQOiMULma+ip6cTIuAaX7jX6qpZGDeyNBOOGFOHRpx5F641KzP7uqORRs777BP+6Xooe/ZyvBLe39pRLPcgeBqjKPNHBxl2UbByt9A8GR9+npk+P4x43F2ElTZ1p03zBVebfs9BvTmJFUyM2BAfZbcIQ21KGa780NSffebUGR69ex9KThVg7Y4Zw56+9HUuPHUdepRbQtSDlrc0Y8QJQljjX5nx9o4QQLY6JdXnatKs0MbFIHGFZYFN8vknRdTha9D981CCEhUeiueYC/jVkEtaUtUGnGw5hD9F2aDSlGJQQivbqUsCFAHW2kIvIGouIyEjJshtndh1JUhBSSoqLlIyeu0q8zrXTZuGB/M+cKqKK27ESFdmFxulZ87BMGRSH1WmpWJ6Ta/x87eR0aOqqgI529H9jDaoihtmc80ZzMP6dHeT2ADVfolJ8vgmJI6Islti0dQShqLKHotGw5TIWS+KSHnHHFvvH3amoapZLPUgKl7GQQ0qX3agxirN3LSJH16Rk9NxV5td5T/ZhxOn1AFxY/lJzFYgeAE1ICNZOTjeGaNpWYSGNMTxDQoAO2z6v9lhvi+bsqFBqRCkaGdcAoAFhIUJQjozrBNCA0GDzqmAYp5QB19aBOjsSNC13sSXVy9aV92e1LbmCAUoAnFu76I1rETm6JrXvl+YeP4yd2zZh9bp3odGYdtLQ6Vqw/KVnkPHkIgTt+dB4nZP1emyHsPQFsB/42nkrMSI5C+ffMu0OowkJweq0VGN4AsDqtFQhPKu0GDwpCdp5K+EoApPuTzCsw7QcFY6M68RZre1eo9asX2vNVMQk/3yDLghj4huEdasu3JutKc7GkMRJxvWZciNBVs2Sr2CAkqIRnyevxdE1uaNpxc5tm3Dk0B689PwirHtnEzSaCOh0LXjp+UXIPX4YzQ31WHPmtMVrxMrdoXAc+PXxyQBMAaq7egnLTxZaHLM8J1cYgQLQJI91eM2PpHVAHCH21HSYPSOE3Zj4BofnEAKya5W+cqNRe2rKizDwT79FyS1/R49+w+2OBNkgnXwFl7GQTy27qb1YivzJ07AxXIOPAOPHhnANTk6ejpqyEpvXKFmDCthfY7p63btITZ+G3OOH8dLzi1Bff9MYnqnp05AZ29/u+lHA+eUv5htSp/SLQU7mfKQMikNepRZLjx2Hrt3x9O0tkR0ICxEKiuSWuvTUdDr8UFNPjbBudUx8g8OPnntex4qqCgwt2uTwvGyQTr6CI9BuzteW3SRNnYm8Ax/hWb3Vujy9DhuCgmwKkpSOnitLzuHrnz+LfjLFPhpNBHYtfQILrhTj8PHDSLl7KABg2h1DsPqJFOh+85bkeQdoNPjDzLnGZSxi1XVsSxnC97xnPK4i2zTaXJGTawzPtTNm2NwTXdHWCum4sOSLa0SduabKknO4J1v4GaZ9sQ/vj1pkdxkKq2bJVzBAuzlfW3bjaqAruV8KAF+v+R2G1tbg5Jr/Rtz6zTbPx7aUIajoW7yaMASHz100Pv5qwhCc/fuHCJ7zEM6G9pE89yCJyuNelwpwPttyihbBwprPjAHRgK7FYh2oGKIrDh9GRoJlM4VAY/4znHyzGh/lrHfYks+8ly2rZslbuIyFfIory2Jc3XVFVFlyDiWPpGNJqx5vhYXj9r05ksf1rDuL5559FofyTfc6Z4wbgw0bN6Khz0iXv7ZhxVkWn4tFRIMnJUGTPFb4PDgEiB5gOqhKixEvLJRc/xkIpH6GR6L64l8/2u5SMwQid+EyFvIbriyLqb1Yiu8mT8PpfbsRp9ehMlyD1tlz0aOP5fSpta/X/A7prcKSk3ta9ciWGIXqdC14fukryM0/jdT0afj9H/+CX/3sxzh0/DAWLX3FWFjkCusQ7P9GMnpdKjA+PnjSKWFqt8qyI5JQcBSYpGYQpjbVYp8To1DAtFzH37swkX9igJLXdLXxgc39UsN9UntdgCpLzqFX9hFMNHyeAuDkscM2U8PLX3rGWDAkhuW6dzYZC4mWv/QM1r3juODFnqqIYahKNDVFEJe3mKuPT5ZsnBAI7E3XO9OST9w6DRA2PyPyNAYoeYUajQ+UFBCZjz5FUqPQjCcXAYDFOlAxRMV1oO4QqFO1UuTuv+vaAHRCtiWfOOq02HeUyAt4D5S8QknbQLlzWJM7p/m9T2v27oWS591oDrZpCaiJiTWsdRX2RAVMG5PvyLK9D+4t1l2guorT097Fe6DkU9RofKBk+U3JiRwkyZwvKQgoPJEdcAHqjf7AapDqhPRIWodf7Ic6Mq7BGPBd1doOr2wSQM5hgJLHqdE2UMnym5i4QTga2w/1lyttji+IH4oRtwbWchFv9Ad2N18PT5Fa13mjmb1ufBkDlDxKrbaBSprYtwYFYcoVLSZLPBddU4WGYbe5dD5f5+79VIm6O/55Qx7lzbaBJX9dj/rOTvxPcLDTbQL9ldQ0uVL2Wh4SdWcMUPIYe/ctgw/u6dIveWfe++Fz3+IRADM6OjAGwCOGj+f0OoRJtAn0Z0r7A1szTgO78WdD5K84hUseI3ffsuZGLVoP7YHOjUU85vddxwMWW48B3tt9Ri3mxUJq7q7DaWAieQxQcgup6k+5+5Z5y57Hc02N2FDwFfCfP1L9WqQCpT+AvwHQm3Uv8nTfX7VYFwsp7Q8sdV61t4kjCiScwiXVuTLtp+a9OjlS910nA+gJ0/Ttfb953W+ncMVRYtmGdU4t73HlvGpMAxMFKo5ASXWuTPupsaTFHnuBIm6A7c+jK+s/QE4l34tIFXbX8aVN1ol8FQOUVOXKtJ8nfklb33fVZh9Fcul3xuc74J7g9hTrP0BOF3yF+1T4OtSaBiYKZJzCJVW5Mu3niSUtSVNn4r7fvI77fvM64jKfxpSaKmP17SMAJhiOc9f0sTvZ+wNE7fOqeX6iQMERKKnGlRGlklZ8XeVrm4d3lbtGiYH2fSJyFwZoAPJW/1NXfqF745e0ku5Fvsqdf4AE0veJyJ0YoAHGW/1PXf2Fzl/SXcNRIpH3MUADjLcWvvMXumfxDxAi72OABhBvLnznL3Qi6m5YhRtAuPCdiMhzGKABwl1LGoiISBoDNEB4c5swIqLuiAEaALjwnYjI81hEFABYAUtE5HkM0ADAClgiIs/jFC4REZECDFAiIiIFGKBEREQKMECJiIgUYIASEREpwAAlIiJSgAFKRESkAAOUiIhIAQYoERGRAgxQIiIiBRigRERECjBAiYiIFGCAEhERKcAAJSIiUoABSkREpAADlIiISAEGKBERkQIMUCIiIgUYoERERAowQImIiBRggBIRESnAACUiIlKAAUpERKQAA5SIiEgBBigREZECDFAiIiIFGKBEREQKMECJiIgUYIASEREpwAAlIiJSgAFKRESkAAOUiIhIAQYoERGRAgxQIiIiBRigRERECjBAiYiIFGCAEhERKRDU2dnZ6e2LICIi8jccgRIRESnAACUiIlKAAUpERKQAA5SIiEgBBigREZECDFAiIiIFGKBEREQKMECJiIgUYIASEREpwAAlIiJS4P8D1MYyrFCqyg4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 576x396 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_classifier(X_train.iloc[:,:2], y_train, rf_2f);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or an RBF SVM:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 222,
   "metadata": {},
   "outputs": [],
   "source": [
    "svm_2f = SVC(kernel=\"rbf\", gamma=2, C=100)\n",
    "svm_2f.fit(X_train.iloc[:,:2], y_train);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 223,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdAAAAE8CAYAAABw2+D5AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3de3xU1b0+/mcmlwkhBEiClyEJN6MgQVARKyFABYuA2iiIcA72cmxPq/XS0h809Ca2XwtHe1BbT9XT2lOVVkDQqAgFAUlCpCJF0MjFhIRbxgK5E5JMbvP7Y8+e7Jnsmczs2Xv23jPP+/XiVTO3bBI6z6y1PuuzLC6XywUiIiIKiVXvCyAiIjIjBigREZECDFAiIiIFGKBEREQKMECJiIgUiA/lwZ+daNDqOoiIiAxnwpihfu/jCJSIiEgBBigREZECDFAiIiIFGKBEREQKMECJiIgUYIASEREpwAAlIiJSgAFKRESkAAOUiIhIAQYoERGRAiG18iMiCoV906qIfa+OO76J2qRREft+RBaXy+UK9sHshUtEcuSC8mxJufAf1jjtL6CnGwCQOT23z10MVgpHoF64DFAiCsmoiiI4Dx/yfB0wKNMuj9BVAag/1/c2mWC1TZyE6pyCSF0VmRwDlIgU8x1dygZmJIMyVL7BKglVhin1hwFKRCERQ9N0YRksMVQZptQPBigR9ctvaEZDYAbCMKUAGKBEJCtmQ9Mfhin5YIASkZeM9mqcX7Fc+CLWQ9MfSZhmTs9lNW+MYoASETLaq5G45RUAkhFnhl3HKzKRWgeA3mpehmnsYIASxSjZ0OSIUxmf6d0xjyzl1G4MYIASxSD7plUMTa3Un+PUboxggBLFCNmiIIamdtxTuxyNRi8GKFGUY1GQjjgajWoMUKIoxKIgg+FoNCoxQImiDNc3Dco9Gr3sqac5Eo0SgQKUx5kRmYR0ffMswPVNI0q7HKh1IHHLK8jgdG7U4wiUyMDYKcikOJ0bNTiFS2QC0jVNgKFpeiwuigqcwiUyoH7P1eQUrbm5f3dnS8oxZuJh1OYwQKMNA5RIRb6jSH/OlpTjBOB9TBgDM2qd+P06jHkEnM6NMgxQMjTfw5yNTvb8TDkMy9iRYQfqz8F5+BAysiZyKjeKMEBJN/2Fo1e1qVkwGElO2uU4W1KOTLA6N5owQElzckEZVDgyjCiaZNi5HhplGKCkKrlKUtmgZDgSkckxQCksvqNLVpISBcaCoujBAKWgBTW6ZGAS+ZdhB2odLCiKEgxQCijggcwMS6LQmakojgJigFIffkOTgUlE5MEAJQAMTSKiUDFAYxhDk4yirMaBzRUVWJ0/Dba43ilOZ3c3VpbuxYKcHOQN51mnZCwM0BjV5zxJhibpaHNFBXafPoNle4qxduYM2OLi4OzuxrI9xSirEU42iaYAZVOF6MDTWGKI7NFYDE0yAGlY5g23Y03+NBSW7vV8LYZqVKl18LgzE+BpLDEuo70a51cs791ywuAkg7HFxWHtzBmeEM1fvxEAojc8KSpY9b4A0taoiiKcX7Fc+CLDLgQnw5MMyBYXhzX507xuW+OzJkpkJByBRiHxnEmvI7MYmmRwzu5uFJbu9bqtsHQvR6BkWByBRpGM9mr0PLoQJ36/Dmf3HhWCUxx1UsjKahxYtqcYzu5ur9t9i1sofL5roKWLFyFvuN3v78D0avlvJxowQKMEp2rVJ60MFd/AxTf63afPYHNFhc5XGD1W+hQMDUpMxNqZMzwhutJnZGpq9ecAgAVEUYABanLSUadnxEmqWJ0/zWsUdLGjw2uUtNpnvY6UW5CTg1uzs7yma8XColuzs7AgJ0fnK1RX5vRcXMyaqPdlUJi4jcXERlUUCcEJMDg1Ijddy8pQCkv9OWROG4cO7gE1hUDbWDgCNSGOOiOHlaFE5A8D1GRk1zpJM/4qQ6OuqIWIQsYANZGM9mqOOiMo5ipDKTJ6+O8mWjBATULsJgSAo84IianKUIqMWgcyp+fCsXAV1z+jABspmACLhfQhVn5KTwgRK0PFE0KIQmWbOEnvSyCVMEANTBx1spuQPvKG22VPABFDlCgkbJ4QdRigBjbozGGcBzjqJDI7Nk+ISlwDNSjPtK2V2yWITK+nm80TohBHoAbkVW3LaVsic5MUDlF04QjUYFhtS2piQ3xjYOFQdGKAGsygM4eF/+C6J6mADfF1xsKhqMYANRCv7SpEKmBDfP2xcCh6sZm8QXimbrnuSSpjQ3yduNc+2TTe3NhM3uC47klaYkN8HbDjUExggBoA1z1JS2yIH2HudU8WDkU/BihRFGNDfH1w3TM2MEB1xsIh0hIb4uuDDRNiAxsp6IgNE0hrbIgfYdy2ElMYoEbA8CSNsCF+BLHqNuZwCpeIKFySwiGGZ+zgCJSIKBzuk1aa1mzGQQAJTiDFFvT2ejIxjkCJiMKUOT0Xm4rq8W6pFZ3dQGMb31pjAX/LREQqcdbVYlNRPVwuFxpaLWhxWvS+JNIQA5SISGUcjcYG/maJiDTA0Wj0Y4ASEWmIo9Hoxd8mEZHGOBqNTgxQIqII2VRUj2OOFI5GowR/g0bA9l9EMaN8/2mORqMEA1RHtUmjcNlTTyNzei5DlCjGcDRqfvyt6aw2aVTvuYHujiYUOn/Hc0mP8yIyGulolCNR82GAGkB1TgHGPLIU6OHZjEptrqjA7tNnvEJUDM/dp89gc0WFzldI5J84EmWImgsD1CA85wdyKleR1fnTvA6KvtjR4XWQ9Or8aXpfIkWxsyXleHJyseLnl+8/zelcE7K4XK6gux5/dqJBy2uJeRnt1Ujc8grOlpTzjFAF5KZrxYOkxbMwiTRRfw7o6UbTms3YVFQf1kstLEhDQhwb0hvFhDFD/d7HjzoGUps0Co6Fq3qnczkaDYktLg5rfEaaayQHSRNpRsUPu5zONQ8GqAFV5xR4V+eyuCgozu5uFJbu9bqtsHRvn8IiozFDAZQZrjFacDrXPPjbMSjZ0SjD1C/pG3necDtKFy/yWhMNOUTFn7fcH5V/B2YogDLDNRrB4MIFWFiQFvbrsDrXHLgGagIZ7dUYdOYwAODE79e5b7TreEXGI76RS9c8paF6a3YW1s6c0fsE95pVIGMeWSp7u+d34EvhurVv+K/Jn4bC0r2er42whmuGazSEWgcyp+fiiY7HVHm53CnZGGtv4ZqojgKtgTJATSajvRrnVyzve0eMFx2V1TiwuaICq33WPJ0XHFh58DMsyLYjb1i613P8BSQgTKP7I/1AI+UVrCH+PsxQAGWGa9Rd/TlkThunWoACQoiOG34JQwb0qPaaFDwGaBQaVVHk9XU4b95RRabwSi4oL2ZNRG3SKFW/dbgzBRc7OpC/fqPn69LFizAoMVHVawyXGa5RV+6ZjdEPL8XPDszo//FBsKVn4M78HlgsFoaoDhigMUD2zdtXtAWrn2lYaWBqEZTBCHVLkhlGd2a4RkNQcUuLFLe36IMBGmOCmmKUMkuwyowuM6fn9rZCROCpVz2MqijqdzRqhvVFM1yjobj/rX706HaU7z+tyktyPVQfDFACoCBYpSIZsgH2v2o5ujxcsgsTp89S7fVEXuvWMj/HkAugdGCGazQclQuKAK6H6oEBSgH5C1aR8/AhYSoyguTWLbUcXdZUHseRby3A+Fffgn10jibfw99o1G8BVHc3VpbuxYKcHOQN17fq2gzXaDgaFBRxPTTyGKAUtv5CVk16TMOWLX8I339rPV68ZzHynvqDZt/HMxo1y7Q5KadBQZGI66GRwwAlCqCm8jhSF8/H1MZ6lA1NQ8uGbZqNQgHAvmkVzu49ygCNBRoVFOVOycaErBYGaASwFy5RACdfeg5TG4U3t7yGelS/+Iz237Snm12lYoGGH5LYL1d/DFCKaTWVx3HdBzu8bpvwwQ44qrRrTefVopFiwuDCBcidkq3a64n9crt6GKB6YoCSpg6X7IrIc5SSjj5FkRiFes5/pejnLhhb0vOaqi9bcaJV1dej0DFASTM1lcdx/qePhTSaU/IcpeRGnyKtR6EeOh9Zx1NWzM3lcvHEFh3F630BFL1OvvQclv3LgRdffAb2ICtblTxHqYZTVXAULMIxP/cPqK7UrJioNmkULnvqaaEit/6cbgVF0lNW5PZ3AuD2FINy1tViU5FQkdvitLCgSAcMUNKEdHQnjub6CyMlzwlH7qy5wKy5mr1+f2qTRiFzeq5QkauT1fnTPGG5bE9xnw5Dq30OKCcFNC4WO+ZIwYSsFk2/B8nj2J80oaSyVZdqWB15Kih1LCayxcVh7cwZnrNT89dvZHs+NdU6NNsLSvpjgJLqlFS26lENGwmNbVbZPw2tFnR2A23zvqn3JcIWF4c1PiPNNT4dhyhE9ee8+uFqHZ7c0qIPTuGS6k6+9By+L1PZGmhdU8lz1KZ2MYbL5cIxx0A/1ZIWOOtqscIAve+d3d0oLN3rdVth6V6OQJVyB6dn1KlSM3l/yvefBtw9cgGug0YSA5RUFUxlq++6ppLnhEMuKIWGXC68W6pmiFrgrAvyzbPWEdLZoWoJdMqKtLCIgiA5Xu+jR7fjZY2DU6riRCvGDY/YtyM3tvIjVZXv2oa2faV+7x9wS75QvBPmc4LV4rR4bTYX/7nLBaWzrlbR9wjXwoI0DC5coEuA8pSVMPmcSavXWqctPQN3TXexwbwG2AuXYobv6NLl6juq1Cso/bGlZ2Dpe3drE6B+Dh0XlV2ow+bTDqyedG3fU1YOHcGCbDvyhqULN+oQ8JGi6LQZyVTt69b7AUC1sz9DxQDVTqAA5RQuRYUWp1CU4zsN66yr0+2agmFLz0DhxV/gbDgv0k8zhvQ1T2Or4zrZ++IB3AfgkMx997n/9yMAN/9ujvz3iZJQDWk/rOxUrT7BSfriCJRMSzo9KxTspOg2AlBqRUEj6gqXBxdEAYLyo0e3+71PrZ+JtJfrPPunwnUDURGigdaC84bb8WReHt45EYfzdfG4LKkOy39zF35T/jW9L9uDI1DtcAqXok5jm9UTmhUnWg03LRssrwDtZ7oVkA9KvT405E7JFkamIpMHqVz7wrzhdtybk4e1+/NxuvU/AdgAODE4/W3MmDcQI3LG6Xa9UgxQ7TBAKWr0TtVC1fMV9eIJUDfpepovo46uvf4OJg/Rix0dyF+/0fP1+wvuwQPbZuJ06yN9Hjs4fSOWPHgtEhKTI3mJshig2mGAUlSQjjqNGiZKSKdGI/336uxoxWcf/xOXmi0YmOrChJtuVBQIXqNRa5wpDwuXG4GOHDgMJy9VABgs9wxMu30Drp+qf7tDW3oG7szvgcViYYiqjAFKphZto06jOFVxFMVbL6Gp7utQa2oypDVdA/FaAx2WjjWTxqPw0Ocou1AHYC6ANwEk9XnepKmvIv92/QNUtLAgDQlxYGN5FbEKl0ytq8eCY46Bmo3OFhak4fqzbyl67ieZd5sy1Ds7Wt3huUhyqw1NdYtQvHUjljzYqmgkutVxHW5W7zIjZqVYMDQsHW/98Td4K+k/sKSjHUf+Iw8NtdsA/BuEEJVyIiVVh4sNgI3lI4sBSoYmTtuqeXjwk5OLvb6uKlyHKoWvNX76Ibyb/mtdi5hOVRSj/MB6zFn4LOITbJ7buzqd2L7ph8idvBgjcrw393/28T/RVLdY9vWa6r6O8gPGmJqMlAU5OYCzHasnXYujoxehvEj4sLbgO3/GX9Y+iK6O/+zznMHpbyN38o2RvlQyEAYoGVaL0wKXy6XKCE8Mzarn16HqHz53hrpm595Okjk9F090PAboXAFcfmA9qo7uwNb1D2Le4hcQn2BDV6cTW9c/iFMVwt/bN0AvNVsgTNvKsaGlWdtrNpT6c8izAXk3TkDTms1e/94GJA/FvPseR/HWZjTVOeE71Z2QOEC3y/aVOyUbY+0cfUYSA5QMSVz3POZIAaA8QMXiFq/QDGd9zh2eTWs242WDTN3OWfisJyy3rn8Qc+59DtvfeAynKooxImcG5ix8ts9zBqa6AIiB4Mt4U5Oa8W38LvM7HZEzDksebEX5gQ1oaQZSUoHcyTcaKjxt6RkYa2/B0GSufUYSi4jIkBrbrDhaE96655OTi1H1/DoAQJkTobdq81Xr6B11Gkyz4wg+/NMCVHR1eG4bkTPDMyL11dnRitdfOOKzBioQtmeM9xsQ/ip3vSpxzVBEJPkwZMZ1bCluY9EOi4jIlJSue3qNOt3Ts5vdTdODatUWQNu8bwJFii5LU+n7/g8fd3VgiOS2Ofc+JxueAJCQmIwZ8waieOtG2Spcf+HZW7m72POc8gPCc56cXCysJZskPAONOomCwQClqPLk5GJU/U4YdUrfyFfnT/OE5bI9xX1ata3ON2bBTDD7NFvPV2Lh8V1Y4vPc7W885ncECoQ+NRmocvfzXX/B5+dewYB4Exx95h55fpJ5N3CA4UnKMUApaiwsSENV4TrZoiBbXBzWzpzhCVGx24z0GC+jCTTak+7THFj8B/yuvQnbIOxYHD3hTmxpb/asiQYK0YTE5KCrbQNV7p5wLMGmIZtw/2T/012GUH8OQHRM25L+1Dw9mAzqcMkuvS9Bc7lTsoUzNQG/FbW2uDis8RlprvFZE+1PXeFyLCxIU3ydwfIe7YnhJ+7TvITODmF6u/V8JQ5+vtUTnm8CuPvEXsyYvQIjcmbgVEUxtm/6oSrX1F/l7nlXpirfR1M93cicnqvywekUq/ivKMrVVB7H+Z8+BkdVhd6XopknJxcHVbzi7O5GYeler9sKS/fC2R24gbuH+7XHb30ctvQMRdcaLGG093XZ+4R9mgcBAINLX8LPejpxN3p75cxqbUDGvpcxb/ELGD3ua8idLD9qDEXulGzcO74SQuWuHCcuG9gU9vfRlHvd84mOx0x7+AAZCwM0yp186Tks+5cD1S8+o/elaGJhQZpQaWuN6zc8pcdVlS5ehLzhds+aaNAhao3D2ZJyFF78haYhGsw+zdbzlZj/xW7cjr6N5uZ98QGcjTWYv+SFPntAg5U7JRtPTi72fEC59XQJspP/133v3wHcA6AdAJA96M9YeLV8P1kjacqcqPclUBThGmgUq6k8jus+2AEAmPDBDjiqKmAfnaPzValHmLZ1jzz7aYSwUlIwJK55StdEV5buxdqZQQSN+/ucLSnHUtyt2VpaMPs0u+pPYsvEAmzx8xrddVXAsNEhfd/cKdlY0vMaAKDqd5IOTdY4DLjCjp/c8iH+a388Tl/cDuBtAAXISpmHn0zZB6vFEnJVc8T0c+g4kRIM0Ch28qXn8P1G4c09r6EeL774DOxP/UHnq1LPkp7Xgt42sSBH+OAg3Qcqhqi4DzQkGXag1oHBhQvwuAZ7QyfcdCPKD7ztZ5/m255q2a6xs/2+RjC9EFYUNGLw2cOer31D0/eDSd7weGy4owTrjyfijeNZcFzajsxBn+K6YflYtsegVc2SwiGjNL+g6MBGClGqpvI4UhfPx9TG3jeMsqFpaNmwzRSj0MY2K94psfhdq5I2SdB936Gkmw0AYW+hCtQ+LcWWnoFfjtrs+drz8/MV5M/T3wHUhqtqrj+HzGnjsGaQvj2LtcKjzLTFRgoxSDr6FEXLKDR3Sraw19Mo5066R6NiID2AdaqEabgt5PqMLp+XaZofxocPsapZegB1qFXNFL6cMcmwWC4xPHXAAI1C0rVPX2ZfC/VqF2eE8BRJg8hPmAKhH38WzD5N36AEhLCs+wdQF+g6w+SvqtlwI9AYEG9lD1w9MECjUMOpKjgKFuGYn/sHVFeaIkDvzO/Bu6UZXtNuoax76qGsxuHdc9cdps7ubqw8dAQLsp/D4wtmIPG6Sap8v45PD+FsYXnfoASADHvf63ELqQewDN+qZmlnJ2m7RKJoxgCNQrmz5gKz5up9GWEZMqAHjW1W5IxJRrk7HbzWPQ1qc0WFd8/dDHtv2Jy7ANiSkFdSDpSUq/dNA3yY6HM9CnsA+1KtqpnIxBigZFjSaSnDrXv6EVTP3QiOzLTqAax6VTORCbEKlwxLPBMUQG+bPoNO3UoZrTrVaNcTcVFehZs7JRsTslqQYuM6qBYCVeGyExEZVorNhTGWKozf+rhwgwnCE1Cn5240Xw9RtGCAkqFZLeJ/mOfNPuyeuxL+Wg2G0jJPzeshol4MUDIEfyfGJG55BWfVLLjRmGo9d92kRUDic8Xvsfv0GWyuCHxIgNrXQ8aSOyUbY+0tel9GzGKAku78nRgzqqJICM8Mu6ELh6R8q1MHJSZi7cwZntBa6TMS7M/q/GlegXexo8MrEPsrAlL7esg4bOkZGGtvwdBkF9c/dcIAJd3JnRgzqqIIJ35v7C0rchbk5ODW7CyvAh2xOvXW7CwsyMkJaVpWfK4YePnrN/bZPhLu9ZB5WSyW/h9EmmGAkq7kTozJaK+G8/Ah4QEmKRwS+Qs2aRCGOi0bThFQMNcT9Xo4TU3aYICSrk6+9Jyn4X1eQz2qX3ymd93TRIVDoQh1WpZFQGGodSDTfVpONG5hIX0xQEk3cj17J+58F/u3fmSqdc9QhTItq0URkBqVvWbSNu+bel+CZlwuFxrb+DauF/7kSTfS0afoluZLOHXilE5XFDnBTstqUQQUbmUvGYOzrhabiurhcrnQ4uRaqB4YoKSLgCfGnK+Fo6kpwlcUWf1Ny4qBNjYtTfUioHAre9Wk6Wi4NrpG0v4cc6TofQkxi638SBflu7ahbV+p5+sBXU2Iqz6KlppaYMBA2K64ArlZWTpeoXak4ZCWlIT69nYMtdnQ4HR6AkwceYrhqeU1iPKG23Hv1Tl450SV6qe3+COOeqXT19JrU/z3rz8H9HSjac3mkI6PMyO28tNWoFZ+DFAyhIz2aqF4aO/RqF37FElDQxqWYoimJyWhrr1d8361Fzs6vA7DLl28CI9/uE+bQPMj0LFoYf393cVD0dr/VooBqi32wiUyEOnezFTJmmaD0wkAEQlPf1PIq6beEtHp3XD3ufZR64jJytvObnAdVAcMUKII8w2HSDd7D1TZu7J0r9caaSiBpnQ9U7W/f/05AEDTms14ouOx0J5rYuX7T+OYIwVdPQzQSGOAEuks0vs8+6vsXfXhPkWBprS6V7W/f083Mqfn4t3S2HtbqzjRyi0tOuBPm0hHejR776+9312jR4ccaGU1DvS4XLjFfqXX9O8PP9iDshoHrk1Pk53+VeXvH6PTtlLc0qIPBiiRjvRo9h6ovd/q/Gl4o6Ii5EDbXFGBPWfOAoAnRPPXb8Q+x5cAgMuSk/t8v7bOLvz7e6Uoq3FgZOqVeDIvL/S/v3urSqxN2/pzzJHC9dAIilu1atWqYB98vqFdw0uhWJbc1Yi4Lw6j+XQtMCB29rWlJCSivbsLT8+Y7gmYeKsVs0dko6qpCQtycpCdOihi17OipBQlZ2s8IZuckIDZI7JxtL4eZTUOVDU1Yc7IkX2eNzMrC0fr6/Gh40tkpqTgbEvvEVtfufIK/PfMGYi39n5eL6vpwg8/mI7q5iUA2tHofB+7T7cgO7UaowbH9//3r3UArReROT0Xz1z5JI4ca9Pgp2E+52uaYE0dhrSUDji7rEhKYGVuuC5PG+D3Pm5jIUPwbGMRe+BG+VYWoyqrcWBzRYWifaDO7m788IM9nlGn6Bb7lXj2qzM9r9fW2YX7tkzH6YsP9nmN7EEvYMMdJRiQEN/3G/g0RoiFPZ7hWFiQhoQ4cHtLmLiNhQyvNmkUHAtXIXN6blCnZ8RaP9dICff0lh6fz+M3X3kF9jm+9PpdvfEFcPrif8g+//TF/8AmuVojyVTtR49ux8tf+SPDsx+cztUeA5QMxTZxkvAf/bRhYz9X41lRUoqPvvyX121Wi8WzJiquZ55vHQLA5udVbDh/aXDvl5ICoXXz38KmonqU7z+tzV8gyojbWzq7wepcjfCnSoZSnVOAMY8sFb6odXj29vkyUj9XNZh9RO3s7saFVmEd8hb7lZ7iI3E6d0ZWpqdv72XJjQCc/l4Jlw307oMsFgjFYnVtuMr3n8a7pVa4XC40tFo4GlUZA5QMpzqnANbfbeqdzpUZjarewUYH0tCUjqib3R8G9pw5Y5oR9crSvThSV4e84XY8+9WZXtW0+xxfIs5i8Uz/3ns1kD3oz7Kvkz3oz1h4tfsL9+89Fvd1qknc4sLRqPpYhUuGdfHamRh9VSIa9n8KtF7s/ZMsVGXGW62Ynjkc/1f+uec5f5s/F8kJCXpdckiePXgQu0+fwdH6evy/aXn4oqEBZTUOFFVU4mh9Pfb/6xy+aGhA3nA7np4x3auK1WhCqSZOiLMiO7Ua5bUX0NRxHYB4AE5kD/ojfjKlBKOHxHumbf+Q+xJHnio5X9OEExcG4upsF9o7LejusSBRplaLvLEKl0xvVEURAODE79d5bnN2d2NZeUWfE0XMMgL1bSLwq6m3YOE7Wzw9cQFz/X1C1dbZhU0VwPlLg3HZwCYsvBoYEO9+R3ePPtPXPI2nioboeJXRKXdKNsbaW2CxCFO6Qwb06HxFxsXTWCjiDpfswsTpszR57VEVRWjv6ETBd36Ksgt1yBuWjjWTxqPwSKXppnH7W+MsXbwIgxITI3xVBuEO0czpuUi8bpLXXU2ZExmsYbKlZwAAcsYke8KUQdoXA5QiqqbyOI58awHGv/oW7KNDP/A5GD966BvYtWMLvjZ5Av76i4dR89IGrxGpVudoasH3WDEpM30Y0Iyfimy5YFXDzw6Y49+NmmzpGbgzXwjPocncNyrFAKWIKlv+EL7/1nq8eM9i5D31B02+x97indi84VWseeZ/YbMlAQDsm1bhxAeHsfLQESzItiNv3ARNvrea5EagQ202vHHXHXj8w32mG1FHVD9bncIx+uGlIT3+k8y7o2JfqnRql6NRAQOUIqam8jhSF8/H1MZ6lA1NQ8uGbZqNQuWIa6XOw4eErkYAPrtQhwkGDFNpeIqHaIuHal+bnob0AQPQ1dODfY4vPSPqYDoCUZgUBnMoofu69X7D7mflaNQbA5QiRhx9irQchfZnVEURTp52YOeKNbhm4njYUwYKd2QYI3jELSp5w+24NycH71RVYdXUWzwN5gFhT2ViXBwWXX01Jl9xuSdwzTRFHRPqzwXVQUsq2MDVa0qZhUYCBihFhKrnRNgAACAASURBVHT0KdJjFColBvpfZ+fh6z9+wGtkqneQBuo7u6KkFBda2zx7K9fkT0Ohz8ktnNI1sRADN5iw1WJUy0IjBihFiO/oU6TXKNTfdPKoiqLe7TAGGY3KkVsfjaXwDKexfVQJYUpZDFqtwjQWp3YZoKQ5udGnSK9RaKDpZK/TXwDDBqlvhW4sbWuRTnGLHxqkHyo4je3DZ1Sr1R7aWCs0YoCS5sp3bUPbvlK/9w+4JR+5s+ZG7HqCnU428mg01kegvo0mOI0dAkmYfvTodk1Ho9G+RsoAJc1o2TAhHKFMJxtxNMrwEMT6h4iwadzRSVwjjeapXQYoaSISDROUUDqdbKTRKKcve8XyNLYqJGuoWoxGRdE6tcsAJU1EomGCEuFMJ3uNRnUMURbQCDgCVVEE+gtHY6ERA5RUp3fDBC15RqLWOCDtcr0vJ2ZxGlsDHI2GLFCAGvd8JDK0ky8955kizWuoR/WLz+h8RerxHOod4sZ4I+rvoO7dp87g1c+78NuPU/Dq511o6+yK6PcPdFD4Sp+wlJ4xWlbjwMrSvapea0yQzKrMs3+q2bfxPcg7WvE8UApZTeVxZP/218hqb/Pc5vyXA3Wz52LQ0HQdr0w98QNtuPT++17nj5qR9MzR2SOyEW+1esKr5GwNSmqGoLTmPXxa+3Xs+/JOvH+qHtmp1chOtaKsxoFnDx7EzKwsr7NIxUYPKQmJnjM+lX7/9u4uzBk5Uva5oZwxSkGK4DFx3W2tOHKsDdbUYUhJ6jTt+aOBzgNlgFLIPlv9S9x96GOv27Lb27DzUguyb5uv01WpqzV+KIZ99SYMdX6J5qMnTBuiM7OycLS+HmU1Dhytr8f0zOFYXlKKshoHkuLGo717P4Ak96Pj0dRxE8prL6Dgqmo8f+iQ4vAL5vv3d1B4duogzBk5ss/98VYr5owcyfAMRa0DaL3oOaS85JPITauKB3lflSUc5O3ssqKrB6YJUx6oTaoxYsMELUXDeqjcdOnI1CtxsvkogMFyz8CPJ38Li66xqLIGabZCoKgr4DLQ4eRm3PbCIiJSjdEaJkSCJ0QNsD9UKd+tIIuu/jY2fvFnv4+/f9z38eObLqkWfmbaihI1W4gkBUPr5r8FZ12tjhfTl1holBAHpNiMG6QMUKIwZLRX4/yK5aYNUKUj0PuvFebYwg0/s41Ao6L610CjzkDEbS9GHomyCpdIDe43pXAqSyPNNwxKFy9C3nA7TjZ/iaS4PAB96xqyB/0ZC6/ufX6hT7VrYenePn/3UL+/v5+hEdji4ryqffPXbzRPeNY6PP9O181/y9DhCcAzKjZrpS6LiIj60Ro/FKcSOjH+8ng0Hz2BZ49VhF1cEykrSkpRcrbG8+afnJCA2SOycbS+HtVNVRgQvwVdPd8GEA/AiexBf8RPppRg9JD4PuH3t/lz8UVDg6cgSPy7K/3+ZTUOVDU1GeZnJRVvtWJ65nD8X/nnntv+Nn8ukhMSdLyqfkhGnc+3LEJ3W6vOFxQco1fqsgo3hh0u2YUrRozW+zJMrabyOI7/8BFcMe+r6DlWjZnD0nG0zamosjSiah1IsQDt3T14+rqxsDlbgeRBXltBHrzuSuQO24WrBu/EbSM34PGppzB6iPAOpkb4mXUrirO7G8tLSnHm4kXPbV80NAT1oSHi3BW2gDDqjGSFrVp8K3UHGOhzCqtwY5RRe9WajbRl4dLvfg0nfr8OzqGXh7Sup0llZxCHMn/06HbPfy849OPehvm+ZNZ3o64aNUimWgM1yVpnKIxWXMQiohhl1F61ZuLbsjDh1T/C9sL/ABn2kIprVKnslDlYefTDS/G69X7Zh0vbtHV2tOKzj/+JAcmDMSTDgunzZ8KWNBAAcPPv5ng/0aTFUmoxRRWuwStsw2WknroM0BgUzb1qI8n3WLS/3HUHru9qFd5QyyuCHoEqHtX4hKZ0RAkgqF6mpyqOonjrJTTVfR2ADYATg9Pfxox5AzEiZxwA4VM/4BOmMRqkhh95R+Go0x8jjEYZoDHI942fo9DQ+TuUu+dP/4Nf/bQQO4+fCikM+93O4WdKVgxNJY2/Ozta8foLR9BUt6jPfYPTN2LJg9ciITHZ6/YVBY2oK1wufBGjIWpIEWoEbzR6b3VhgMYYf2/8HIWGxt+h3NfbM3HIcRZ5w9KxdvJE2IbZg57i8zvt635z9J2SDfdN8mBZKcq2L4Yw8vTlxLTbN+D6qdP63GNLz8DS9+7uvYFBqq8YGnXKWViQBkCf6VzuA40x0pNSRNF2YorWaiqP47oPdsjet7S5EVPzZqLoT7+Bzb19Tdw7eGt2FhbkyH9Ikd1TuWsXnOfOABBGFT87MAPl+097/oTrUrMF8uEJADa0NMvf46yrxctf+SPS1zwt3CCz/koaE/d0un/2Hz26PSbDEwA2FdXjmCMFDa0WtDiNs2fUYDtuKFyB3vgnfLADjqoKjkKD0HCqCo6CRTgmc99QAA/fko/OMWMBbBDe4DLsnhCV47UGOiwdayaNR+Ghz1F2oQ4P1A9A/u1rEa/BlNzAVBcAJ/yNQFNSAz//qaIhsM1/SxiNiiEai6PRICqeteA1fR8jU7b+lO8/jQr3dG5Dq0X34iKAU7hRJxZ71eopo70aiVteEbaHBGg4v2zHDuz+13nkDUvHkpdPIiExCYfKKrB1/YM4VVGM0eO+hvlLXlD9+vpfAx2PhET/+9ykon5ttJ9RdqCKZy3EyhqnEgsL0iJWWMQ1UCKNeRrOA33DpdaBsgt1+HtCHDJnrUN8Qu9osKvTie2bfojcyYsxIkebrRHBVOEGK6rWRmUCs2nNZhxzpMg+nIFmHJGszmWAEkWAZzS692if6b6mNZuxqajvEXCR0tnRivIDB9HSDKSkArmTbwx65CnHdKPRIPbQMiDNJVLVuQxQMq3DJbswcfosvS8jaNKRqDiaidY35twp2cbdNxrE6PKTvcfw2cf/xKVmC5KS2wGLBe2XbBiY6sKEm27ss72HjCcS1bkMUNKcFkFn1laEGe3VqOgR+g/rOeqMFN1HozJhmTk9F5sn/bfna98PMd7T2kcAVAOYj3CnuCnyxBDVajqXAUqa0irozN6KsLHNCpfLhXdLrVHXas1XxEejPqHpO7oMNOr3Lqy6BOB9AAV9Huev0QQZj5bTuQxQ0pQWQRctrQhbnBZ0updDjzlSUHGiNarD1Gs0CgSsTA5JgCnZUKfIvZtLbAFwG0JtNEHGo9V0bqAA5T7QGBdo6jWYaVnpvlM195mefOk5fN/dDCKvoR4vvvgM7CYchYpTSi1OCyZktWCsHQDSVJ3aFRvFX2q26L5+91TREOS69y7Os38qhKkYfsGGabBTskX1AEL/OXo3l+iBkkYTZDybiuqxsCANLU5LxPrmMkBjWE3lcZz/6WNwyEy9BrpPSougk2sGYfYmENL/Qze0WrCwIE2Vqd3etTxxROVE+QF91+/EEWE5AoRpP3ynZF9WsZGAd3MJK8JpNEHGcsyRgglZLRH7fgzQGHbypeew7F8O2dALdJ9Iq6CThrLIzKNQX0OTXWhxWtzHNaUpntrt7Gh1h6e0SYINTXWLULx1I5Y82Kr7+p1cmPan4kQrnEW1UDK6DMaEm25E+YG33T+3rwLYBvk10LeRO/lGTa6BtNPZjYiNQtkLN0bJTb2Kdq//i9/7pLTouRtMK8JokGJzYWiyCwlxwISsFs/Zh6H47ON/uqtI+2qq+zrKDxwM9zJVJe3xG+iP1mvECYnJmDFvIAanb4QwhugE8BqEkSgAOGG1voaxk+rD2itLkVe+/zTeLbV66g60xgCNUdLwk4ZeTeVxtD75M9hl7pPSKugaTlXhwMzZeCnRhncBz58XE204OPM21FdXKnpdo0qxuTyflMUzOYOltFE8ASNyxmHJg9filtmvIjGpFcA9AHYCeAfATvT0LMCxQ2no7GjV90IpZOIHsMY27eONU7gxKNDU65Hf/gor29qwEcBIn/uk07KBmq0DwIDqSkXTuLmz5qLs7+/iex1O7zs6nHjRYonaPr5Dk10Ya29BRXpG0COwcBvFx7qExGRY4xLR0S6uH8/3ul8YxUe+CteWnoFfjtrs+fpnB7Rp8RjNIlVQxACNQf7WGFc/tQpTi3cBAEYAOAkhROXWH3NnzQU0CLNYPk3GYgntmCbvtTxvXL8LjhqjeDWqoKWhWfX8OlRJ7nsA6zD64aUAGKahiERBEQM0xgQKqIF7dmJGVycA4Gag31GoFrQa2UYjcS2veOtG2Ubx0bJ+FyigOjtacegf+3CqohZwtWNEznBMumVq0AEW7ihejSpoYe/sd71C06sRRa0DVc8L7SEfwDp89Oj2qG0PqTatC4rYSCHG+DvurL6xAZO3bMbMri7PbcU2Gz6eW4C0IcJGYh6Fpq3GNiveKbEoqsZVs1G8kQQ6SQYAdhU14NLFez33Ae8hOeU0Zt+dE1SAhXPcW//PDdzFSNHJNu5tQKMfXsrRaBDU6FDETkTUL7GbkC+zttEzI6UBGk2ko82kgU4cOWhDc/03+jwuNe1VwJWA5oYlMq9ShNS0ZvzbQzcENRJVetybd0cjX4G7GIXVP9h9uDdHosFZWJCmWYByCpdiet2RjKGzoxUl295D5edWdLRfBeBrEN6e3gPwCYDrvR7fXD8IwDw/rzYXzfXbUH7gYFAFQEJFbivKD2wIaRSvZP1UHHXW/cN9g5KewWmXA7UOofcwQ1RXDFDiumMEmO1YtlCJI8fmhg5cbDyLQUOykTo03m9BjXSk2dHxJc5WXYHmhh+idyp2G4RStnsAFAG4GsBAySvEIVB4AdaQtvEkJCaHXG0b6vqpuNYJIPxm+xl2T4gu4XRuvxrbrBgyIPS91v3hFC6RxoI9rSacNdBQqkCVVo36e17vFOgIAF8CmItAU6FyU6a9gSkdaRZBaPQeD2GPpnSbyVsQRqDy4QVsw7TbmzXdghLK+qnXaTVqnlTD6dygLCxIU3zcGadwiXQUTFtEpUKtAlVaNerveXm3xaPs/S401c1H32PB+rYV9Nd+UHie70hzLnqD07u1TGraRcD1pp810G1ITbuo6jYefx8egqmCtqVnaBOeAKdzg6TVlhZ2IqKodLhklyFeL1DLxHD1hpEYXu8AeB9NdfNRvPVSny463uEljtzEkOv7+GCet/udi2iqmw3gAwiB15e0rWCg9oPC8/dIvrZBCE4nhFNTACGgNmLm/HTMvCMVAwetg7QFH/AmklNOYtrXBuKzjw+gdNteHCwrDauj0KmKo3j9hSMo274Yh/Z9A2XbF+P1F47gVMVRT0ejabdvwKSpr2La7Ruw5MHxng8jKwoaeytttToj1f26S3pe0+b1yS+OQCnqBHuSTCReL9jTasTDt4HgmykIYTQWQnj2TpsC29BUN6JPEY3w+MWyrxWo606g57W3LoYwSgzuWLD+Cm+8R5pOCGud7wHoRGLS08gZ70L+3Pme0d39j7Xi8Eev4dQXFwA4MeLq4RiakYWyHV2qnFDj/eFBPHy7B011ydi+qQZLHxmG5JQM2Z+bLT1DvTVPMiSOQCnqiFOm4TS1V+P1ArVM9OVyuUI+3qy5oQPCmmMBpCND4esv0dTg3Q5Raded/kPPid5jweT0FtT0Ft7IP04ITNF7AFoAjAJwHzral+PsyZEAetexEhKTMTn/Vix44D7cdf9CuFzAzreq0FSXDEDc09w7ym5tqcXBstKgR6a9I+ZPIITnbQDuAnAbnG0ZeOXZN3Cq4ihOVRTjvdcfRFen8HezpWeg8OIv4OzuxrLyCpTVBHeMWziqnl+HFQWNmn8f6sUApaii9pRpOK8X6mk1oRYPtdbsgb9pU2AuWppqvG7pL7z8dd0J/Lx3YY37OYBbIBQCASn4O4B2CBW0f8fg9LdxzcRxOFhWiuaGDiQl991vLHgPwvFiTgino3wM4S2qBsLoz/8pM+I06z92fgMd7cshBN37EIIPnueue36H7FSsP8KHhy4Ap9D3g8o96Oq4Dnveq8Nn+/+KqqM7sHX9g3h07jksfe9unPjgMJaVV2D36TPYXKHxKUIZdsAah7rC5SEfSkDKMUApqvg7ZSbSr6f1sWyt5ytx3/ltSMRJCOGyBcIa6Bb31zYMGuz9RjrhphsxOP1t2dcbnP42rrlunOzoLNDz4hN/jZ7uo4hPmA0gDYn4Pb6D78CCOQDeQnziLzF2Uj02/akaZdsX47P930V761hYrd7Hhw1O24Brb6zEpKmbkDZsJYBEAKsA3AvvMLShqcHpdZ2tLbWya7RC4J2CGL6ADc7W6xDK+q/w4WEH/H9QWYTm+lRcmb0YI3Jm4FRFMb41705c7OzyjDzzhtuxOj8CDenTLvd7V2dHa0gjbwoO10DJtHz3VvqbMv1gw6v46n19u9n0J5wDw7XeWzu49CU82dmAg1iGv+O78F0DBa5E6lDv/3sHqhodO6kem16G33VDf8+bNmclyj9+DqcqijE04xHM6rbheEMNXKjB0IzrcNc316LoLw6fqtub0dOTC1vy07hmwpVIHZrgblxwEzo7WnHskAvAfZLHS6t0m1HxWaJ77VW4jn+Wrkd761g/PylpJa/vFPElAB+gqS4e2994A3PuvbfPdp4JN92Ij4u3oaM98J7T1kuJuPsHG1H+whTsPH4K+e+XAADyhtuxduYM2OLi/Dxffb4VuWr06yV53AdKpiS3t1KuHWE1gPUDBuCGt/eEHFiRbG/oqG3Frx/fGdTezNbzlfi3l+/DrW2N2IIULMABdOAar8dYra/h2//fWCSnpPd5vm/v3GuuG4dNL1f329PVX8/drk4ntq5/EKcqij3Pm2KNx9jvvY0TVRdCancXuD1ePWB5E3B9R+Y+cc/oQJn73oGwblkEocNRMoTR7Cn0t2cVAHa9/QaO/FNs8tD37yDuOX39qVyc/PGPPOEJAKWLF2FQYqLM8zRU6/D0yg23X2+0yJ2SjQlZLarvA+UULpmSb2GPvynTjwGsbGvD508/EdLraz0FK1VW8hm+s+gfQa/NDS59Cbe2CcUid6AFt2J1n8f09CzC8U/lny923cm/fRqunzoNxw4f8bu1RLrm6Ps8sRI2PsGGOfc+5/W8HT1dyNj3csiFS4Ef/yHgut/Pfb5bYETCFpik5FcAXAkhPC9Bbk3T33Tu9LnzkTr0TT/fV9hz+ucnxsCxfBkKD33udW9h6V44u7v9PFd7gbYN+VtPpuBxCpdMR66wR27KtL6xAVO2FQFOJ278sCSknr6Ram/Y2noJT666gLOnF0hu7duAwPP485X4ty92e73GD7AFu3HcZxQa3FmWQPhnYnZ1OrH1r9/1um0JgEeO78aHN0xGKO3uArfH6wh4nb7NFgAgMWk9bszvwLU3TMKmP1Wjqe5m9L9n1XtUnJCYjJl3pGJX0TrZk19uWzIZDb/8GZYdLEfZhTrkDbdjTf40FJbuRVmNA8v2FEd8Glekxnmn5B8DlExHbm9l3lN/6HPAd9nyhzDdKRSqzLjU0m8nIOmaandCIm76+W/Cus5g+t+++uo+nD19l+x9cm/m0tGn6A7U4Vasxt/xF8mt/Z9lKQrnTExx+vbsqY8xF8DrEMJzGwC0NWJyUxmOpo8J+tDvQIeEx8X/E91d8/1ep23AQTjb5sB7SjbDMyU7Y9559zpuvJ/XAPyFyoiccbJ7TqfMW4xfOFdj0aYjnvAUw3LtzBlYtqcYZTUOrCzdi7UzI9+vNtzzTikwTuFqTO2OONEq2J9TsHsrQ9mDKT7+/E8fg6Oqwuu/lQr2NWrPAcG+mbeer8R8n9Gn6AfYgkQc93wtF07+9FedG+h1tm/6IU5VFGOKNR5vAhgM4E0I47ttAA4d3YFb8howOH0jvKpu0zfKHvqdkJiMvNvi3VOu0g5DRejuSoew1cWbsG3mPUy8ZZjfjkAAPF2DRl1zGP635fgPFeme0wUPfAMb//tGfPv9f8fZknIsyJ2AW7OzvEaaYojemp2FBTmRPYxB3BMazu+W+he3atWqVcE++HxDu4aXEn1qKo/j5CPfxoCZt2HQ0L7FHNHocMkuXDFidEjPCeXn9NnqX+LuQx973Zbd3oadl1qQfdv8kB8nffwjB/Zh56UWNB74yPPfco8NhvT1Ar3GkWNV+PjDqyA/GeTEVeM/x5VZwnaUtjMHccaWgu2Zkzx/3kjJxgsXx2Bv93Q0YjDaMBKD09/EjHkDkXbZlUFda1xcAoakN+FfNYfgbBOvxRnU69iSBqGt/hRmXXMrPsi6AdszJ2FX5iQMseeivrURo7JvRPqI8bj+azcgKXkn0i47jKvGf45ZX58o+7qnKo6idLsTl5qnA3gJQAOAkwBmA2iE0HD+IwjNFeKRiMP4AQpwAHch7Urg5q9Ow4irsnFlVjbi4hJk/66jxl6Nis/3wtk2vs/9Scl/RcqgDtRfqEHGFcNkXwMQGiXkvPCA8EWGHdmpgzBn5EjEW73HJPFWK+aMHIns1EF+f4aqSx4EtLei7f3t6L73R+hoP63odxstcqdkY6y9BXFWIFHBnOvlaf6PtWMVrobEKs5YOZQ62FNHfAX7c6qpPI7UxfP7NCcAgLKhaWjZsA320TlBP07udUtSB6PW5cI9F5tlHxsM6ev19xpf1rVh8R370HBhQZ/7fE/08MdfdWyo1HodpbwrRrdAqKqVjs7F27ogFAx1Yy6ew1bsxlwsRcvttwV9+orciTBW60b09FwN4GYEqsoVuwydLSk3dou+WofnlBa9f7d6CuckFoCnsehCrtAl2s/UVHLqSCg/p2ALe0ItAJKuqU5vbsJG9+2BetcGEmz/WwAYkDwQty66HDvXBz7RIxAlZ1lq+TpK9fbcvQSgHEJ/XSuE7kQD3f+7DUL17Hwk4igexiEAwMN4GxtH3if7unKkh2g3N3Ti+Gdfwtm6DEKVLhCokOux/EqcLTR4ePrQ+3erN6Xh2R8GqEZCeRONBko/MITyc8qdNbdPoVA4j/O9btEICJOGIxH6hx8lzRce/V4uho8Zg4M7NkR8hLCioBGDzx42xIHMQsXoEQhbTH4E+bNCR0BYZZ2PWViDeRD+7czHRez+x/+h856ng/5+YqgcLCuFs3U55NaiAzXZJ2IRkQZCLWCJBkpa3gXzc9K6CEuuX+3NAPa7/zvUdoCh9r8dMqAHCXHAgtkDPHssX38qF09N3S/7+FDZ0jPw5ORi2T8P/OO7qCtcjqrn1xmif2pScjuE1hdyzfHFlnzXIzWtBVNu+S88FrfZ6/nzvvgAly5Uhfx9FW/1qNW+QXy4bv7dHEP8bvWysCBN09fnCFQD0lGVKJpHoUpb3vX3c1L7WLJgrlukZBRaU3kccdu3yN4X6DVSbC4Mba3Gk5MPo+PTQzhbWI46AA9gHUY/vDSUv1IfVc+vQ8BIybB7BYEtPQM5Y9TrTBPSAc8WC4SWe3KEut7B6R2YMS8dV336NuZ0X/J6xKzWBmwtfSGkUSgQ+laPp4qGYOGazRhc2Hfd2lDcv9slPa/hZ9B/hiHScqdkw2K5hCEDevp/sEIMUJUF08HGyGuhwexd9KXkA0MwPycla6qhkK6V1p2pRpfDe0TRceVwZGWPBBC4cYL4M6vcV4qLcOG3BfchbUjfwgPf1xhVUQTn4UPYsLkYm087sHrStcIWCPfamvPcGRR8+ydYkG1H3rAwqriDWKubZ/8UmHIdxtpbYLFcQrw1/DWjrh4Lxhak4Zgjxet2f6HafsmGQCNB+4ijuOv+u9DZWON3O8+8Lz7A3y5UYeCw4CvBA+099bfV491SK5YCwocPE62Fxho1/h0HfH1NXz0GRaqDjRaUjPiUfmDo7+fU+mExbtC4CCuUtVJ/pD+z+E8PYmXrJbxotfhtwiCG5tmScpwAAGscNp/5ErvPXcCy8gphHyHQe47kuQuALQl54yaEdZ39GVJzGBOyR7uLLdR603GhxWnBhKwWzy2d3fAKVWmY9jcSHD1uOBISB6Ct/iT+NmI6nq7ugtM5GkKD+G7YbFUYnh2P7roqIIQADdRk318hl7OuFi9/5Y94PPE5oRoXYJAaiLh1RWsMUJWp8aasFyUjPqUfGPr7OZUtf8hrTTWYa1Iyeg6X+DNb+9QT+MqBfwCQD/yM9mqcX7HcE5qwxnmOn1o9a5anY82yPcVebeAidhQWtKlUlHtNMVR9w7SzI7iR4IDRU/H69iFocvo8zgkMvrARS0ZfG/J1SqtyQynkeqLjMSxckyZM53I0ahhj7S1hbV0JFgOUACivotXiA4OSNVWt10v9fU/xOm8o2Ql7RweA3sC/7lfLkbjlFQAIOErxbfuWv17YSKPHUViRIH1T8wrTRZlISGrDhhfewvmaefA3Euzd7tJXOFWzSrd6bCqqh23+W717QwEGqQFoHZ4Aq3DJTe2DqNW6FlF/1+R7Oku49hbvxI8e+gacTu/uW05nO3700Dewt3in13XO7OiAtG524t+LcOihh3C2pBxn9x4VRpwB3lRtcXFY4zPSXJM/Tfvw1LmSNMXmEoqokl34Yt8uLLk7B69unoIfLH8Hs+8pwr3/ub1PSz4jNkh31tXiiY7H0LRmc/8PJk1pXXkrxQAlQ227UXKMmNzoOVybN7yKXTu2eIWoGJ67dmzBupf/x+/+UQC4pbUdp06cFkIz7XLPdK0/zu5uFJbu9bpN86Ow6s8BAMY8shTVOQXafZ8g1FQeR/Pjj6K55gvYM5Lx79+6Db/69Uw8/PB0LFk03OuxvWulcvRtkP5uqfst1UBbXMS+uLFAqLy1YGiy9qNPgAFKUDbi00rDqSocmDkbLyXa8C7g+fNiog0HZ96G+urKPs9ROnoOtMd0zTP/i2kzZntGohcvNntGntNmzMbijMsC7h8FgAm19XA0NfV7Hc7ubs/0bd5wO0oXL0LecLtnTVSTEK0/B/R0I3N6Li5mTVT/9UPkO4MgHZlaLBYsLEjz7Gc086zdYQAACc5JREFUcoN0Z10t1s1/C5nTc4UQ1TtI3bMedYXLo34/qC09A2PtLZpX3koxQGNcJA+ODkburLlItFjxvQ4n7gQ8f77f4USCxSKsuUooHT3XVB7HJz/+nt/H2WxJeOYPr3pCNO/6kZ7wXP7jn2Ny8XbZ510eZ8VvR4/En8ddg+OjR6K+uf/5xJWSgqG1M2dgUGIi1s6c4QnRlT4jU1W4w9OxcBVqk0ap//oh6G8GQWw2MdbegoUFabh+2lh840ejgj7hJdL6TOcaJESj3Z35PREpHJJiEVGMM9q2m1C3xShtWvHJ07/CyIZ6HHz6CdhfWCf7GJstCf/17J+Qd/1Iz23/9eyf0F703zg6NBVHh6bC8xl0QO+b9pVXXIHcrKx+/qa9xKOuVkvWPMXCopWle9U/Csv9hm6bOEnd11UomHaO4puiWHQ01p6D+bPt+H+rIt/+MFgsLooccd0zkuEJ8DQWMpjyXdvQtq/U7/0Dbsn3jEJDPXVFVFN5HJV3zsDDnR34fUIirn6vVPZx0oIhUd6wdKy9IRe2hMR+1zUNyR2eRlj3BOR/h8GegtPYZoXL5cIxR0poHY90sLAgrbdzkR4hKjmZJZrY0jNwZ34PLBaLZh2HeBoLmUYo22IaTlXhi5mz8dnWt2HvcKIm0YbO+QUYOFj4B+9v9PzJ07/CjE5hy8kNnR0okRmFSsNz2ozZWP9IAe578HGUXajzanhgKrUOz7Rttd7X4hZO28shA3rQ4rRgrL0FYwvSsKmo7wcpo+BoVBu907batesLhGugpJtwG8X7rpeK66Q3/fw3uOnnv+mzXgoII55BJbsw1f11HoCUPTv7rLsV/ug/PeH56tpfofbPb2Lt5InarktqyWDTtoA66+++hUa29Ay1L1M1hlsbNTm9pm2lOAIlXajR+EBJwwXp6FMkNwpdcN83AAB/XP1TNP/i5wAA2zA71s68XJt1SS0ZbNpWpOb6uzgavTO/BwBHo9EsEtO2weIaKOmibPlD+P5b6/HiPYuRp7BRvPgavvy9pnTt05e/tdBRFUU48ft15nyDcwenOG0bK8S10XdLrXDW1ep9OQGtKGhEXeFy4Qst/425/y1EwzrowoK0iFbbBloD5RQuRZwajQ+UTP9V7itFrp/Xy7UAX+wr8brNE55m5H7DvHDLxJgKT6B328ud+T0R7UqjxFNFQyKzb9Qdzkt6XtPm9SPECNO2UpzCpYgLZttCf5RM/6Xbh2N3xjBc/LKmz+MPZ43EmCt6RwAZ7dVCeEqavpuCZNT58aQlOPKtBRh/82LDngCkFfENtrHNioUFaYYejTrravEEHsOKNe7RqDREzTjzoQEjTdtKMUApopQevu1LSRP7TosFt/7LgZky96XV16Jl1FUyd5gvPMW1zpPLH9L0PFUzMNPa6FNFQ2Cb/xZyxiRjnv3T3jCN8RAVRp36Vtv6wylciig92wZWvvwCLrpc+LXVGnSbQMMTp/3cW1Ssv9uE6pwCVfsDh1strTezVeqW7z8tP7Ur/RMjbOkZnt62Rpm2leIIlCIm0LqldfsWOL7/I82mGmsqj+P2459jKoDLenpwOYCR4p0dTrwo0ybQ0CRTteLWFGmFrRrT5IA+x8RpxUyjUaB3ajf30d4etl4jU38y7FERsuK0bbyBh3kMUIoYf+uW9Y0N6NyxBc59JZq9SUsD5WYAGyEJUASYRjbiFJrMtpTDJbsw0X3pak2TA8oOWTcyM62NiqRVs+XoneaVc/Pv5nhV3Pp62SQVuN7TtsYbeYq4jYU0cbhkFyZOnxXUY9XY0hKIXLu4PQCqAHRIuhdJ2wQCQiFR4pZXhL16eoeoZEThuy2lpvK4UCzkHiWGur3HH+nPLdj2embS4rSg033QjdFHo6HInZJt2q0qRiwW4jYWiijPtF8Q625anOXpS27ddSaAFHh3L/Kdwq1NGtXbucd9dmZE+axvjnlkKcY8srTPthTpUWBqnq5jpEPWtWCmtdFQmDU8FxakeVrzGSU8+8MpXFJdKNN+aq3V+RMoUMQDsANNb1bnFGDMIxC2tIijQC23tviMNOXWN6V8P4AcmngjBqjQ3UfNaWCjM9vaaLQx4qgzWAxQUpXciNLfG24k3qR9110dJbsxseoLz/096D+4q3MKcNlTEzHozGEAKoap+1BrqWBCU8r3A8hnh/+Jm1T4ABJOk3czEtfZGlotplkbjQZG3qISDK6Bkqp8198CrbuptVYXLKXHn/nKaK/2DtMwjHlkqdfXofSqDecosFBfV83XN7poXRs1GnHkOTTZuEVCAI8zowgJZUQZ6sHZalCreXlt0ijU5owCAK+RaaguZk1EddIoRc8FtBslGu2Q9UjjaFR70mlbwNgBGghHoFEolApYNYUyogzl4GzqK9ZHiZHC0aj6xH62Rt+iIgo0AmWARhnfLQ2R/L58Q48cfgCJrIZWCwBwNBoGcdQJwPDTtlIM0Bii9Z5Kf/iGTtFOHI0ec6SYdquIXnKnZGOsvcU0o04pBmiMiPaN70RGwNFo8Mw66pRiI4UYEe0b34mMYGiyy3PeKPmXOyXb0xjBrOHZHwZolAhUAUtE6hKnIXOnZPfzyNhjS8/AwoI0jLW3GPYUFbUwQKOEnseEEcWiockujLW3RFUbwHDFwqhTivtAo4AeeyqJSAgJsQ3gMYd5m7iHQxyFj7W3AGiJieAUMUCjQKxvfCfSkzhFOdbegrEx1Hiht0BIqK4FzFdhGy5W4RIRqSRWtrqYeVtKqLiNhYgogsStLsccKQDMe8SYr2jYlhIqBigRUYS1OIUQFVsBmnVq15aegZwxyQAQM6NOKTaTJyKKMGnImLXQqHeqtgVAbAVnMDgCJSKKEDN0MfKuqo2dqVp/OAIlIjIA320vAFBxolX3MPXdihKrVbWhYoASEUWQGEoTslrQ1WPBWLtLlzBlaIaPAUpEpAMhqCIbpgxNdTFAiYh01l+YSgUTrHI9ehma6mOAEhEZiG+YSgUKVpFvUPZ9bVILA5SIyKD6Bp58sPb/PNICA5SIyGQYkMbA48yIiIgUYIASEREpwAAlIiJSgAFKRESkAAOUiIhIAQYoERGRAgxQIiIiBRigRERECjBAiYiIFGCAEhERKcAAJSIiUoABSkREpAADlIiISAEGKBERkQIMUCIiIgUYoERERAowQImIiBRggBIRESnAACUiIlKAAUpERKQAA5SIiEgBBigREZECDFAiIiIFGKBEREQKMECJiIgUYIASEREpYHG5XC69L4KIiMhsOAIlIiJSgAFKRESkAAOUiIhIAQYoERGRAgxQIiIiBRigRERECjBAiYiIFGCAEhERKcAAJSIiUoABSkREpMD/D4gfnj8FMLbAAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 576x396 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_classifier(X_train.iloc[:,:2], y_train, svm_2f);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Combining multiple tables (10 min)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Take a look at the [Formula 1 race data set](https://www.kaggle.com/cjgdev/formula-1-race-data-19502017) from Kaggle. \n",
    "- The dataset contains **multiple CSV files**.\n",
    "- Let's read in one of them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 224,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>raceId</th>\n",
       "      <th>driverId</th>\n",
       "      <th>constructorId</th>\n",
       "      <th>number</th>\n",
       "      <th>grid</th>\n",
       "      <th>position</th>\n",
       "      <th>positionText</th>\n",
       "      <th>positionOrder</th>\n",
       "      <th>points</th>\n",
       "      <th>laps</th>\n",
       "      <th>time</th>\n",
       "      <th>milliseconds</th>\n",
       "      <th>fastestLap</th>\n",
       "      <th>rank</th>\n",
       "      <th>fastestLapTime</th>\n",
       "      <th>fastestLapSpeed</th>\n",
       "      <th>statusId</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>resultId</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>18</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>10.0</td>\n",
       "      <td>58</td>\n",
       "      <td>34:50.6</td>\n",
       "      <td>5690616.0</td>\n",
       "      <td>39.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>01:27.5</td>\n",
       "      <td>218.3</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>18</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>8.0</td>\n",
       "      <td>58</td>\n",
       "      <td>5.478</td>\n",
       "      <td>5696094.0</td>\n",
       "      <td>41.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>01:27.7</td>\n",
       "      <td>217.586</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>18</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>7.0</td>\n",
       "      <td>7</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>6.0</td>\n",
       "      <td>58</td>\n",
       "      <td>8.163</td>\n",
       "      <td>5698779.0</td>\n",
       "      <td>41.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>01:28.1</td>\n",
       "      <td>216.719</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>18</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>5.0</td>\n",
       "      <td>11</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>5.0</td>\n",
       "      <td>58</td>\n",
       "      <td>17.181</td>\n",
       "      <td>5707797.0</td>\n",
       "      <td>58.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>01:28.6</td>\n",
       "      <td>215.464</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>18</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>23.0</td>\n",
       "      <td>3</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>4.0</td>\n",
       "      <td>58</td>\n",
       "      <td>18.014</td>\n",
       "      <td>5708630.0</td>\n",
       "      <td>43.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>01:27.4</td>\n",
       "      <td>218.385</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23777</th>\n",
       "      <td>988</td>\n",
       "      <td>842</td>\n",
       "      <td>5</td>\n",
       "      <td>10.0</td>\n",
       "      <td>17</td>\n",
       "      <td>16.0</td>\n",
       "      <td>16</td>\n",
       "      <td>16</td>\n",
       "      <td>0.0</td>\n",
       "      <td>54</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>33.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>01:43.8</td>\n",
       "      <td>192.542</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23778</th>\n",
       "      <td>988</td>\n",
       "      <td>828</td>\n",
       "      <td>15</td>\n",
       "      <td>9.0</td>\n",
       "      <td>19</td>\n",
       "      <td>17.0</td>\n",
       "      <td>17</td>\n",
       "      <td>17</td>\n",
       "      <td>0.0</td>\n",
       "      <td>54</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>36.0</td>\n",
       "      <td>15.0</td>\n",
       "      <td>01:43.6</td>\n",
       "      <td>193.057</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23779</th>\n",
       "      <td>988</td>\n",
       "      <td>840</td>\n",
       "      <td>3</td>\n",
       "      <td>18.0</td>\n",
       "      <td>15</td>\n",
       "      <td>18.0</td>\n",
       "      <td>18</td>\n",
       "      <td>18</td>\n",
       "      <td>0.0</td>\n",
       "      <td>54</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>52.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>01:42.3</td>\n",
       "      <td>195.402</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23780</th>\n",
       "      <td>988</td>\n",
       "      <td>832</td>\n",
       "      <td>4</td>\n",
       "      <td>55.0</td>\n",
       "      <td>12</td>\n",
       "      <td>NaN</td>\n",
       "      <td>R</td>\n",
       "      <td>19</td>\n",
       "      <td>0.0</td>\n",
       "      <td>31</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>26.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>01:43.4</td>\n",
       "      <td>193.41</td>\n",
       "      <td>36</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23781</th>\n",
       "      <td>988</td>\n",
       "      <td>817</td>\n",
       "      <td>9</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>R</td>\n",
       "      <td>20</td>\n",
       "      <td>0.0</td>\n",
       "      <td>20</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>13.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>01:42.8</td>\n",
       "      <td>194.579</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>23777 rows × 17 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          raceId  driverId  constructorId  number  grid  position  \\\n",
       "resultId                                                            \n",
       "1             18         1              1    22.0     1       1.0   \n",
       "2             18         2              2     3.0     5       2.0   \n",
       "3             18         3              3     7.0     7       3.0   \n",
       "4             18         4              4     5.0    11       4.0   \n",
       "5             18         5              1    23.0     3       5.0   \n",
       "...          ...       ...            ...     ...   ...       ...   \n",
       "23777        988       842              5    10.0    17      16.0   \n",
       "23778        988       828             15     9.0    19      17.0   \n",
       "23779        988       840              3    18.0    15      18.0   \n",
       "23780        988       832              4    55.0    12       NaN   \n",
       "23781        988       817              9     3.0     4       NaN   \n",
       "\n",
       "         positionText  positionOrder  points  laps     time  milliseconds  \\\n",
       "resultId                                                                    \n",
       "1                   1              1    10.0    58  34:50.6     5690616.0   \n",
       "2                   2              2     8.0    58    5.478     5696094.0   \n",
       "3                   3              3     6.0    58    8.163     5698779.0   \n",
       "4                   4              4     5.0    58   17.181     5707797.0   \n",
       "5                   5              5     4.0    58   18.014     5708630.0   \n",
       "...               ...            ...     ...   ...      ...           ...   \n",
       "23777              16             16     0.0    54      NaN           NaN   \n",
       "23778              17             17     0.0    54      NaN           NaN   \n",
       "23779              18             18     0.0    54      NaN           NaN   \n",
       "23780               R             19     0.0    31      NaN           NaN   \n",
       "23781               R             20     0.0    20      NaN           NaN   \n",
       "\n",
       "          fastestLap  rank fastestLapTime fastestLapSpeed  statusId  \n",
       "resultId                                                             \n",
       "1               39.0   2.0        01:27.5           218.3         1  \n",
       "2               41.0   3.0        01:27.7         217.586         1  \n",
       "3               41.0   5.0        01:28.1         216.719         1  \n",
       "4               58.0   7.0        01:28.6         215.464         1  \n",
       "5               43.0   1.0        01:27.4         218.385         1  \n",
       "...              ...   ...            ...             ...       ...  \n",
       "23777           33.0  16.0        01:43.8         192.542        11  \n",
       "23778           36.0  15.0        01:43.6         193.057        11  \n",
       "23779           52.0   6.0        01:42.3         195.402        11  \n",
       "23780           26.0  14.0        01:43.4          193.41        36  \n",
       "23781           13.0  12.0        01:42.8         194.579         9  \n",
       "\n",
       "[23777 rows x 17 columns]"
      ]
     },
     "execution_count": 224,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "racing_results_df = pd.read_csv(\"data/formula-1-race-data-19502017/results.csv\", index_col=0)\n",
    "racing_results_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Let's say we want to predict the `milliseconds` column, namely the total length of time it takes a driver to finish a race. \n",
    "- In that case, we should not have access to most of these other columns. \n",
    "- But we would have the `raceId` and `driverId`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 225,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>raceId</th>\n",
       "      <th>driverId</th>\n",
       "      <th>milliseconds</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>resultId</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>18</td>\n",
       "      <td>1</td>\n",
       "      <td>5690616.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>18</td>\n",
       "      <td>2</td>\n",
       "      <td>5696094.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>18</td>\n",
       "      <td>3</td>\n",
       "      <td>5698779.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>18</td>\n",
       "      <td>4</td>\n",
       "      <td>5707797.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>18</td>\n",
       "      <td>5</td>\n",
       "      <td>5708630.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23777</th>\n",
       "      <td>988</td>\n",
       "      <td>842</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23778</th>\n",
       "      <td>988</td>\n",
       "      <td>828</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23779</th>\n",
       "      <td>988</td>\n",
       "      <td>840</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23780</th>\n",
       "      <td>988</td>\n",
       "      <td>832</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23781</th>\n",
       "      <td>988</td>\n",
       "      <td>817</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>23777 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          raceId  driverId  milliseconds\n",
       "resultId                                \n",
       "1             18         1     5690616.0\n",
       "2             18         2     5696094.0\n",
       "3             18         3     5698779.0\n",
       "4             18         4     5707797.0\n",
       "5             18         5     5708630.0\n",
       "...          ...       ...           ...\n",
       "23777        988       842           NaN\n",
       "23778        988       828           NaN\n",
       "23779        988       840           NaN\n",
       "23780        988       832           NaN\n",
       "23781        988       817           NaN\n",
       "\n",
       "[23777 rows x 3 columns]"
      ]
     },
     "execution_count": 225,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "racing_results_df_subset = racing_results_df[['raceId', 'driverId', 'milliseconds']]\n",
    "racing_results_df_subset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 227,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>raceId</th>\n",
       "      <th>driverId</th>\n",
       "      <th>milliseconds</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>resultId</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>18</td>\n",
       "      <td>1</td>\n",
       "      <td>5690616.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23745</th>\n",
       "      <td>987</td>\n",
       "      <td>1</td>\n",
       "      <td>5491730.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22089</th>\n",
       "      <td>898</td>\n",
       "      <td>1</td>\n",
       "      <td>5984506.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20611</th>\n",
       "      <td>349</td>\n",
       "      <td>1</td>\n",
       "      <td>5344268.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23115</th>\n",
       "      <td>957</td>\n",
       "      <td>1</td>\n",
       "      <td>5695831.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23675</th>\n",
       "      <td>983</td>\n",
       "      <td>842</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23776</th>\n",
       "      <td>988</td>\n",
       "      <td>843</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23714</th>\n",
       "      <td>985</td>\n",
       "      <td>843</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23758</th>\n",
       "      <td>987</td>\n",
       "      <td>843</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23739</th>\n",
       "      <td>986</td>\n",
       "      <td>843</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>23777 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          raceId  driverId  milliseconds\n",
       "resultId                                \n",
       "1             18         1     5690616.0\n",
       "23745        987         1     5491730.0\n",
       "22089        898         1     5984506.0\n",
       "20611        349         1     5344268.0\n",
       "23115        957         1     5695831.0\n",
       "...          ...       ...           ...\n",
       "23675        983       842           NaN\n",
       "23776        988       843           NaN\n",
       "23714        985       843           NaN\n",
       "23758        987       843           NaN\n",
       "23739        986       843           NaN\n",
       "\n",
       "[23777 rows x 3 columns]"
      ]
     },
     "execution_count": 227,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "racing_results_df_subset.sort_values(by=\"driverId\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TODO\n",
    "\n",
    "for fun, try to predict milliseconds just based on raceId and driverId"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Now we need some features to predict the race time. \n",
    "- Enter the other tables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 226,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>driverRef</th>\n",
       "      <th>number</th>\n",
       "      <th>code</th>\n",
       "      <th>forename</th>\n",
       "      <th>surname</th>\n",
       "      <th>dob</th>\n",
       "      <th>nationality</th>\n",
       "      <th>url</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>driverId</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>hamilton</td>\n",
       "      <td>44.0</td>\n",
       "      <td>HAM</td>\n",
       "      <td>Lewis</td>\n",
       "      <td>Hamilton</td>\n",
       "      <td>1985-07-01</td>\n",
       "      <td>British</td>\n",
       "      <td>http://en.wikipedia.org/wiki/Lewis_Hamilton</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>heidfeld</td>\n",
       "      <td>NaN</td>\n",
       "      <td>HEI</td>\n",
       "      <td>Nick</td>\n",
       "      <td>Heidfeld</td>\n",
       "      <td>1977-10-05</td>\n",
       "      <td>German</td>\n",
       "      <td>http://en.wikipedia.org/wiki/Nick_Heidfeld</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>rosberg</td>\n",
       "      <td>6.0</td>\n",
       "      <td>ROS</td>\n",
       "      <td>Nico</td>\n",
       "      <td>Rosberg</td>\n",
       "      <td>1985-06-27</td>\n",
       "      <td>German</td>\n",
       "      <td>http://en.wikipedia.org/wiki/Nico_Rosberg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>alonso</td>\n",
       "      <td>14.0</td>\n",
       "      <td>ALO</td>\n",
       "      <td>Fernando</td>\n",
       "      <td>Alonso</td>\n",
       "      <td>1981-07-29</td>\n",
       "      <td>Spanish</td>\n",
       "      <td>http://en.wikipedia.org/wiki/Fernando_Alonso</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>kovalainen</td>\n",
       "      <td>NaN</td>\n",
       "      <td>KOV</td>\n",
       "      <td>Heikki</td>\n",
       "      <td>Kovalainen</td>\n",
       "      <td>1981-10-19</td>\n",
       "      <td>Finnish</td>\n",
       "      <td>http://en.wikipedia.org/wiki/Heikki_Kovalainen</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>838</th>\n",
       "      <td>vandoorne</td>\n",
       "      <td>2.0</td>\n",
       "      <td>VAN</td>\n",
       "      <td>Stoffel</td>\n",
       "      <td>Vandoorne</td>\n",
       "      <td>1992-03-26</td>\n",
       "      <td>Belgian</td>\n",
       "      <td>http://en.wikipedia.org/wiki/Stoffel_Vandoorne</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>839</th>\n",
       "      <td>ocon</td>\n",
       "      <td>31.0</td>\n",
       "      <td>OCO</td>\n",
       "      <td>Esteban</td>\n",
       "      <td>Ocon</td>\n",
       "      <td>1996-09-17</td>\n",
       "      <td>French</td>\n",
       "      <td>http://en.wikipedia.org/wiki/Esteban_Ocon</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>840</th>\n",
       "      <td>stroll</td>\n",
       "      <td>18.0</td>\n",
       "      <td>STR</td>\n",
       "      <td>Lance</td>\n",
       "      <td>Stroll</td>\n",
       "      <td>1998-10-29</td>\n",
       "      <td>Canadian</td>\n",
       "      <td>http://en.wikipedia.org/wiki/Lance_Stroll</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>841</th>\n",
       "      <td>giovinazzi</td>\n",
       "      <td>36.0</td>\n",
       "      <td>GIO</td>\n",
       "      <td>Antonio</td>\n",
       "      <td>Giovinazzi</td>\n",
       "      <td>1993-12-14</td>\n",
       "      <td>Italian</td>\n",
       "      <td>http://en.wikipedia.org/wiki/Antonio_Giovinazzi</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>843</th>\n",
       "      <td>brendon_hartley</td>\n",
       "      <td>39.0</td>\n",
       "      <td>HAR</td>\n",
       "      <td>Brendon</td>\n",
       "      <td>Hartley</td>\n",
       "      <td>1989-10-11</td>\n",
       "      <td>New Zealander</td>\n",
       "      <td>http://en.wikipedia.org/wiki/Brendon_Hartley</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>842 rows × 8 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                driverRef  number code  forename     surname        dob  \\\n",
       "driverId                                                                  \n",
       "1                hamilton    44.0  HAM     Lewis    Hamilton 1985-07-01   \n",
       "2                heidfeld     NaN  HEI      Nick    Heidfeld 1977-10-05   \n",
       "3                 rosberg     6.0  ROS      Nico     Rosberg 1985-06-27   \n",
       "4                  alonso    14.0  ALO  Fernando      Alonso 1981-07-29   \n",
       "5              kovalainen     NaN  KOV    Heikki  Kovalainen 1981-10-19   \n",
       "...                   ...     ...  ...       ...         ...        ...   \n",
       "838             vandoorne     2.0  VAN   Stoffel   Vandoorne 1992-03-26   \n",
       "839                  ocon    31.0  OCO   Esteban        Ocon 1996-09-17   \n",
       "840                stroll    18.0  STR     Lance      Stroll 1998-10-29   \n",
       "841            giovinazzi    36.0  GIO   Antonio  Giovinazzi 1993-12-14   \n",
       "843       brendon_hartley    39.0  HAR   Brendon     Hartley 1989-10-11   \n",
       "\n",
       "            nationality                                              url  \n",
       "driverId                                                                  \n",
       "1               British      http://en.wikipedia.org/wiki/Lewis_Hamilton  \n",
       "2                German       http://en.wikipedia.org/wiki/Nick_Heidfeld  \n",
       "3                German        http://en.wikipedia.org/wiki/Nico_Rosberg  \n",
       "4               Spanish     http://en.wikipedia.org/wiki/Fernando_Alonso  \n",
       "5               Finnish   http://en.wikipedia.org/wiki/Heikki_Kovalainen  \n",
       "...                 ...                                              ...  \n",
       "838             Belgian   http://en.wikipedia.org/wiki/Stoffel_Vandoorne  \n",
       "839              French        http://en.wikipedia.org/wiki/Esteban_Ocon  \n",
       "840            Canadian        http://en.wikipedia.org/wiki/Lance_Stroll  \n",
       "841             Italian  http://en.wikipedia.org/wiki/Antonio_Giovinazzi  \n",
       "843       New Zealander     http://en.wikipedia.org/wiki/Brendon_Hartley  \n",
       "\n",
       "[842 rows x 8 columns]"
      ]
     },
     "execution_count": 226,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "racing_drivers_df = pd.read_csv(\"data/formula-1-race-data-19502017/drivers.csv\", \n",
    "                                encoding='latin-1', index_col=0,\n",
    "                               parse_dates=['dob'])\n",
    "racing_drivers_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Can we use the driver's nationality and age as features?\n",
    "- `pd.merge` can take care of this for us."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 228,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>raceId</th>\n",
       "      <th>driverId</th>\n",
       "      <th>milliseconds</th>\n",
       "      <th>driverRef</th>\n",
       "      <th>number</th>\n",
       "      <th>code</th>\n",
       "      <th>forename</th>\n",
       "      <th>surname</th>\n",
       "      <th>dob</th>\n",
       "      <th>nationality</th>\n",
       "      <th>url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>18</td>\n",
       "      <td>1</td>\n",
       "      <td>5690616.0</td>\n",
       "      <td>hamilton</td>\n",
       "      <td>44.0</td>\n",
       "      <td>HAM</td>\n",
       "      <td>Lewis</td>\n",
       "      <td>Hamilton</td>\n",
       "      <td>1985-07-01</td>\n",
       "      <td>British</td>\n",
       "      <td>http://en.wikipedia.org/wiki/Lewis_Hamilton</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>19</td>\n",
       "      <td>1</td>\n",
       "      <td>5525103.0</td>\n",
       "      <td>hamilton</td>\n",
       "      <td>44.0</td>\n",
       "      <td>HAM</td>\n",
       "      <td>Lewis</td>\n",
       "      <td>Hamilton</td>\n",
       "      <td>1985-07-01</td>\n",
       "      <td>British</td>\n",
       "      <td>http://en.wikipedia.org/wiki/Lewis_Hamilton</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>20</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>hamilton</td>\n",
       "      <td>44.0</td>\n",
       "      <td>HAM</td>\n",
       "      <td>Lewis</td>\n",
       "      <td>Hamilton</td>\n",
       "      <td>1985-07-01</td>\n",
       "      <td>British</td>\n",
       "      <td>http://en.wikipedia.org/wiki/Lewis_Hamilton</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>21</td>\n",
       "      <td>1</td>\n",
       "      <td>5903238.0</td>\n",
       "      <td>hamilton</td>\n",
       "      <td>44.0</td>\n",
       "      <td>HAM</td>\n",
       "      <td>Lewis</td>\n",
       "      <td>Hamilton</td>\n",
       "      <td>1985-07-01</td>\n",
       "      <td>British</td>\n",
       "      <td>http://en.wikipedia.org/wiki/Lewis_Hamilton</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>22</td>\n",
       "      <td>1</td>\n",
       "      <td>5213230.0</td>\n",
       "      <td>hamilton</td>\n",
       "      <td>44.0</td>\n",
       "      <td>HAM</td>\n",
       "      <td>Lewis</td>\n",
       "      <td>Hamilton</td>\n",
       "      <td>1985-07-01</td>\n",
       "      <td>British</td>\n",
       "      <td>http://en.wikipedia.org/wiki/Lewis_Hamilton</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23772</th>\n",
       "      <td>988</td>\n",
       "      <td>842</td>\n",
       "      <td>NaN</td>\n",
       "      <td>gasly</td>\n",
       "      <td>10.0</td>\n",
       "      <td>GAS</td>\n",
       "      <td>Pierre</td>\n",
       "      <td>Gasly</td>\n",
       "      <td>1996-07-02</td>\n",
       "      <td>French</td>\n",
       "      <td>http://en.wikipedia.org/wiki/Pierre_Gasly</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23773</th>\n",
       "      <td>985</td>\n",
       "      <td>843</td>\n",
       "      <td>NaN</td>\n",
       "      <td>brendon_hartley</td>\n",
       "      <td>39.0</td>\n",
       "      <td>HAR</td>\n",
       "      <td>Brendon</td>\n",
       "      <td>Hartley</td>\n",
       "      <td>1989-10-11</td>\n",
       "      <td>New Zealander</td>\n",
       "      <td>http://en.wikipedia.org/wiki/Brendon_Hartley</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23774</th>\n",
       "      <td>986</td>\n",
       "      <td>843</td>\n",
       "      <td>NaN</td>\n",
       "      <td>brendon_hartley</td>\n",
       "      <td>39.0</td>\n",
       "      <td>HAR</td>\n",
       "      <td>Brendon</td>\n",
       "      <td>Hartley</td>\n",
       "      <td>1989-10-11</td>\n",
       "      <td>New Zealander</td>\n",
       "      <td>http://en.wikipedia.org/wiki/Brendon_Hartley</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23775</th>\n",
       "      <td>987</td>\n",
       "      <td>843</td>\n",
       "      <td>NaN</td>\n",
       "      <td>brendon_hartley</td>\n",
       "      <td>39.0</td>\n",
       "      <td>HAR</td>\n",
       "      <td>Brendon</td>\n",
       "      <td>Hartley</td>\n",
       "      <td>1989-10-11</td>\n",
       "      <td>New Zealander</td>\n",
       "      <td>http://en.wikipedia.org/wiki/Brendon_Hartley</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23776</th>\n",
       "      <td>988</td>\n",
       "      <td>843</td>\n",
       "      <td>NaN</td>\n",
       "      <td>brendon_hartley</td>\n",
       "      <td>39.0</td>\n",
       "      <td>HAR</td>\n",
       "      <td>Brendon</td>\n",
       "      <td>Hartley</td>\n",
       "      <td>1989-10-11</td>\n",
       "      <td>New Zealander</td>\n",
       "      <td>http://en.wikipedia.org/wiki/Brendon_Hartley</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>23777 rows × 11 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       raceId  driverId  milliseconds        driverRef  number code forename  \\\n",
       "0          18         1     5690616.0         hamilton    44.0  HAM    Lewis   \n",
       "1          19         1     5525103.0         hamilton    44.0  HAM    Lewis   \n",
       "2          20         1           NaN         hamilton    44.0  HAM    Lewis   \n",
       "3          21         1     5903238.0         hamilton    44.0  HAM    Lewis   \n",
       "4          22         1     5213230.0         hamilton    44.0  HAM    Lewis   \n",
       "...       ...       ...           ...              ...     ...  ...      ...   \n",
       "23772     988       842           NaN            gasly    10.0  GAS   Pierre   \n",
       "23773     985       843           NaN  brendon_hartley    39.0  HAR  Brendon   \n",
       "23774     986       843           NaN  brendon_hartley    39.0  HAR  Brendon   \n",
       "23775     987       843           NaN  brendon_hartley    39.0  HAR  Brendon   \n",
       "23776     988       843           NaN  brendon_hartley    39.0  HAR  Brendon   \n",
       "\n",
       "        surname        dob    nationality  \\\n",
       "0      Hamilton 1985-07-01        British   \n",
       "1      Hamilton 1985-07-01        British   \n",
       "2      Hamilton 1985-07-01        British   \n",
       "3      Hamilton 1985-07-01        British   \n",
       "4      Hamilton 1985-07-01        British   \n",
       "...         ...        ...            ...   \n",
       "23772     Gasly 1996-07-02         French   \n",
       "23773   Hartley 1989-10-11  New Zealander   \n",
       "23774   Hartley 1989-10-11  New Zealander   \n",
       "23775   Hartley 1989-10-11  New Zealander   \n",
       "23776   Hartley 1989-10-11  New Zealander   \n",
       "\n",
       "                                                url  \n",
       "0       http://en.wikipedia.org/wiki/Lewis_Hamilton  \n",
       "1       http://en.wikipedia.org/wiki/Lewis_Hamilton  \n",
       "2       http://en.wikipedia.org/wiki/Lewis_Hamilton  \n",
       "3       http://en.wikipedia.org/wiki/Lewis_Hamilton  \n",
       "4       http://en.wikipedia.org/wiki/Lewis_Hamilton  \n",
       "...                                             ...  \n",
       "23772     http://en.wikipedia.org/wiki/Pierre_Gasly  \n",
       "23773  http://en.wikipedia.org/wiki/Brendon_Hartley  \n",
       "23774  http://en.wikipedia.org/wiki/Brendon_Hartley  \n",
       "23775  http://en.wikipedia.org/wiki/Brendon_Hartley  \n",
       "23776  http://en.wikipedia.org/wiki/Brendon_Hartley  \n",
       "\n",
       "[23777 rows x 11 columns]"
      ]
     },
     "execution_count": 228,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "merged_df = pd.merge(racing_results_df_subset, racing_drivers_df, on=\"driverId\")\n",
    "merged_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The `on` keyword told it which column to use to match up the rows of the two dataframes.\n",
    "- Note that the first 5 rows have the same `driverId`, so they pulled the same data from `racing_drivers_df`.\n",
    "- Now we could keep only the columns we plan to encode:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 229,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>raceId</th>\n",
       "      <th>driverId</th>\n",
       "      <th>milliseconds</th>\n",
       "      <th>dob</th>\n",
       "      <th>nationality</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>18</td>\n",
       "      <td>1</td>\n",
       "      <td>5690616.0</td>\n",
       "      <td>1985-07-01</td>\n",
       "      <td>British</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>19</td>\n",
       "      <td>1</td>\n",
       "      <td>5525103.0</td>\n",
       "      <td>1985-07-01</td>\n",
       "      <td>British</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>20</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1985-07-01</td>\n",
       "      <td>British</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>21</td>\n",
       "      <td>1</td>\n",
       "      <td>5903238.0</td>\n",
       "      <td>1985-07-01</td>\n",
       "      <td>British</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>22</td>\n",
       "      <td>1</td>\n",
       "      <td>5213230.0</td>\n",
       "      <td>1985-07-01</td>\n",
       "      <td>British</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23772</th>\n",
       "      <td>988</td>\n",
       "      <td>842</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1996-07-02</td>\n",
       "      <td>French</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23773</th>\n",
       "      <td>985</td>\n",
       "      <td>843</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1989-10-11</td>\n",
       "      <td>New Zealander</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23774</th>\n",
       "      <td>986</td>\n",
       "      <td>843</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1989-10-11</td>\n",
       "      <td>New Zealander</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23775</th>\n",
       "      <td>987</td>\n",
       "      <td>843</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1989-10-11</td>\n",
       "      <td>New Zealander</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23776</th>\n",
       "      <td>988</td>\n",
       "      <td>843</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1989-10-11</td>\n",
       "      <td>New Zealander</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>23777 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       raceId  driverId  milliseconds        dob    nationality\n",
       "0          18         1     5690616.0 1985-07-01        British\n",
       "1          19         1     5525103.0 1985-07-01        British\n",
       "2          20         1           NaN 1985-07-01        British\n",
       "3          21         1     5903238.0 1985-07-01        British\n",
       "4          22         1     5213230.0 1985-07-01        British\n",
       "...       ...       ...           ...        ...            ...\n",
       "23772     988       842           NaN 1996-07-02         French\n",
       "23773     985       843           NaN 1989-10-11  New Zealander\n",
       "23774     986       843           NaN 1989-10-11  New Zealander\n",
       "23775     987       843           NaN 1989-10-11  New Zealander\n",
       "23776     988       843           NaN 1989-10-11  New Zealander\n",
       "\n",
       "[23777 rows x 5 columns]"
      ]
     },
     "execution_count": 229,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "merged_df_subset = merged_df[['raceId', 'driverId', 'milliseconds', 'dob', 'nationality']]\n",
    "merged_df_subset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can process the `dob` column to get age:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 231,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>raceId</th>\n",
       "      <th>driverId</th>\n",
       "      <th>milliseconds</th>\n",
       "      <th>dob</th>\n",
       "      <th>nationality</th>\n",
       "      <th>age</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>18</td>\n",
       "      <td>1</td>\n",
       "      <td>5690616.0</td>\n",
       "      <td>1985-07-01</td>\n",
       "      <td>British</td>\n",
       "      <td>34.760290</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>19</td>\n",
       "      <td>1</td>\n",
       "      <td>5525103.0</td>\n",
       "      <td>1985-07-01</td>\n",
       "      <td>British</td>\n",
       "      <td>34.760290</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>20</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1985-07-01</td>\n",
       "      <td>British</td>\n",
       "      <td>34.760290</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>21</td>\n",
       "      <td>1</td>\n",
       "      <td>5903238.0</td>\n",
       "      <td>1985-07-01</td>\n",
       "      <td>British</td>\n",
       "      <td>34.760290</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>22</td>\n",
       "      <td>1</td>\n",
       "      <td>5213230.0</td>\n",
       "      <td>1985-07-01</td>\n",
       "      <td>British</td>\n",
       "      <td>34.760290</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23772</th>\n",
       "      <td>988</td>\n",
       "      <td>842</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1996-07-02</td>\n",
       "      <td>French</td>\n",
       "      <td>23.749331</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23773</th>\n",
       "      <td>985</td>\n",
       "      <td>843</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1989-10-11</td>\n",
       "      <td>New Zealander</td>\n",
       "      <td>30.478098</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23774</th>\n",
       "      <td>986</td>\n",
       "      <td>843</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1989-10-11</td>\n",
       "      <td>New Zealander</td>\n",
       "      <td>30.478098</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23775</th>\n",
       "      <td>987</td>\n",
       "      <td>843</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1989-10-11</td>\n",
       "      <td>New Zealander</td>\n",
       "      <td>30.478098</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23776</th>\n",
       "      <td>988</td>\n",
       "      <td>843</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1989-10-11</td>\n",
       "      <td>New Zealander</td>\n",
       "      <td>30.478098</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>23777 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       raceId  driverId  milliseconds        dob    nationality        age\n",
       "0          18         1     5690616.0 1985-07-01        British  34.760290\n",
       "1          19         1     5525103.0 1985-07-01        British  34.760290\n",
       "2          20         1           NaN 1985-07-01        British  34.760290\n",
       "3          21         1     5903238.0 1985-07-01        British  34.760290\n",
       "4          22         1     5213230.0 1985-07-01        British  34.760290\n",
       "...       ...       ...           ...        ...            ...        ...\n",
       "23772     988       842           NaN 1996-07-02         French  23.749331\n",
       "23773     985       843           NaN 1989-10-11  New Zealander  30.478098\n",
       "23774     986       843           NaN 1989-10-11  New Zealander  30.478098\n",
       "23775     987       843           NaN 1989-10-11  New Zealander  30.478098\n",
       "23776     988       843           NaN 1989-10-11  New Zealander  30.478098\n",
       "\n",
       "[23777 rows x 6 columns]"
      ]
     },
     "execution_count": 231,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ages = (pd.Timestamp.now() - merged_df_subset[\"dob\"]).apply(lambda x: x.total_seconds()/3600/24/365)\n",
    "merged_df_age = merged_df_subset.assign(age=ages)\n",
    "merged_df_age"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- So far we got information for each driver.\n",
    "- Likewise, we can get information about the races, and use those as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 232,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>year</th>\n",
       "      <th>round</th>\n",
       "      <th>circuitId</th>\n",
       "      <th>name</th>\n",
       "      <th>date</th>\n",
       "      <th>time</th>\n",
       "      <th>url</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>raceId</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2009</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Australian Grand Prix</td>\n",
       "      <td>2009-03-29</td>\n",
       "      <td>06:00:00</td>\n",
       "      <td>http://en.wikipedia.org/wiki/2009_Australian_G...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2009</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>Malaysian Grand Prix</td>\n",
       "      <td>2009-04-05</td>\n",
       "      <td>09:00:00</td>\n",
       "      <td>http://en.wikipedia.org/wiki/2009_Malaysian_Gr...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2009</td>\n",
       "      <td>3</td>\n",
       "      <td>17</td>\n",
       "      <td>Chinese Grand Prix</td>\n",
       "      <td>2009-04-19</td>\n",
       "      <td>07:00:00</td>\n",
       "      <td>http://en.wikipedia.org/wiki/2009_Chinese_Gran...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2009</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>Bahrain Grand Prix</td>\n",
       "      <td>2009-04-26</td>\n",
       "      <td>12:00:00</td>\n",
       "      <td>http://en.wikipedia.org/wiki/2009_Bahrain_Gran...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2009</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>Spanish Grand Prix</td>\n",
       "      <td>2009-05-10</td>\n",
       "      <td>12:00:00</td>\n",
       "      <td>http://en.wikipedia.org/wiki/2009_Spanish_Gran...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1005</th>\n",
       "      <td>2018</td>\n",
       "      <td>17</td>\n",
       "      <td>22</td>\n",
       "      <td>Japanese Grand Prix</td>\n",
       "      <td>2018-10-07</td>\n",
       "      <td>05:00:00</td>\n",
       "      <td>http://en.wikipedia.org/wiki/2018_Japanese_Gra...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1006</th>\n",
       "      <td>2018</td>\n",
       "      <td>18</td>\n",
       "      <td>69</td>\n",
       "      <td>United States Grand Prix</td>\n",
       "      <td>2018-10-21</td>\n",
       "      <td>19:00:00</td>\n",
       "      <td>http://en.wikipedia.org/wiki/2018_United_State...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1007</th>\n",
       "      <td>2018</td>\n",
       "      <td>19</td>\n",
       "      <td>32</td>\n",
       "      <td>Mexican Grand Prix</td>\n",
       "      <td>2018-10-28</td>\n",
       "      <td>19:00:00</td>\n",
       "      <td>http://en.wikipedia.org/wiki/2018_Mexican_Gran...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1008</th>\n",
       "      <td>2018</td>\n",
       "      <td>20</td>\n",
       "      <td>18</td>\n",
       "      <td>Brazilian Grand Prix</td>\n",
       "      <td>2018-11-11</td>\n",
       "      <td>16:00:00</td>\n",
       "      <td>http://en.wikipedia.org/wiki/2018_Brazilian_Gr...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1009</th>\n",
       "      <td>2018</td>\n",
       "      <td>21</td>\n",
       "      <td>24</td>\n",
       "      <td>Abu Dhabi Grand Prix</td>\n",
       "      <td>2018-11-25</td>\n",
       "      <td>13:00:00</td>\n",
       "      <td>http://en.wikipedia.org/wiki/2018_Abu_Dhabi_Gr...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>997 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        year  round  circuitId                      name        date  \\\n",
       "raceId                                                                 \n",
       "1       2009      1          1     Australian Grand Prix  2009-03-29   \n",
       "2       2009      2          2      Malaysian Grand Prix  2009-04-05   \n",
       "3       2009      3         17        Chinese Grand Prix  2009-04-19   \n",
       "4       2009      4          3        Bahrain Grand Prix  2009-04-26   \n",
       "5       2009      5          4        Spanish Grand Prix  2009-05-10   \n",
       "...      ...    ...        ...                       ...         ...   \n",
       "1005    2018     17         22       Japanese Grand Prix  2018-10-07   \n",
       "1006    2018     18         69  United States Grand Prix  2018-10-21   \n",
       "1007    2018     19         32        Mexican Grand Prix  2018-10-28   \n",
       "1008    2018     20         18      Brazilian Grand Prix  2018-11-11   \n",
       "1009    2018     21         24      Abu Dhabi Grand Prix  2018-11-25   \n",
       "\n",
       "            time                                                url  \n",
       "raceId                                                               \n",
       "1       06:00:00  http://en.wikipedia.org/wiki/2009_Australian_G...  \n",
       "2       09:00:00  http://en.wikipedia.org/wiki/2009_Malaysian_Gr...  \n",
       "3       07:00:00  http://en.wikipedia.org/wiki/2009_Chinese_Gran...  \n",
       "4       12:00:00  http://en.wikipedia.org/wiki/2009_Bahrain_Gran...  \n",
       "5       12:00:00  http://en.wikipedia.org/wiki/2009_Spanish_Gran...  \n",
       "...          ...                                                ...  \n",
       "1005    05:00:00  http://en.wikipedia.org/wiki/2018_Japanese_Gra...  \n",
       "1006    19:00:00  http://en.wikipedia.org/wiki/2018_United_State...  \n",
       "1007    19:00:00  http://en.wikipedia.org/wiki/2018_Mexican_Gran...  \n",
       "1008    16:00:00  http://en.wikipedia.org/wiki/2018_Brazilian_Gr...  \n",
       "1009    13:00:00  http://en.wikipedia.org/wiki/2018_Abu_Dhabi_Gr...  \n",
       "\n",
       "[997 rows x 7 columns]"
      ]
     },
     "execution_count": 232,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "racing_races_df = pd.read_csv(\"data/formula-1-race-data-19502017/races.csv\", encoding='latin-1', index_col=0)\n",
    "racing_races_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- For those who have taken CPSC 304 or some have other database training, you'll recognize this type of multi-table situation, with foreign keys connecting the tables.\n",
    "- `pd.merge` supports several types of joins, see the documentation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 233,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[0;31mSignature:\u001b[0m\n",
       "\u001b[0mpd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmerge\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mleft\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mright\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mhow\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mstr\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m'inner'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mon\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mleft_on\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mright_on\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mleft_index\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mbool\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mright_index\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mbool\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0msort\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mbool\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0msuffixes\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'_x'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'_y'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mcopy\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mbool\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mindicator\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mbool\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mvalidate\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0;34m'DataFrame'\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
       "\u001b[0;31mDocstring:\u001b[0m\n",
       "Merge DataFrame or named Series objects with a database-style join.\n",
       "\n",
       "The join is done on columns or indexes. If joining columns on\n",
       "columns, the DataFrame indexes *will be ignored*. Otherwise if joining indexes\n",
       "on indexes or indexes on a column or columns, the index will be passed on.\n",
       "\n",
       "Parameters\n",
       "----------\n",
       "left : DataFrame\n",
       "right : DataFrame or named Series\n",
       "    Object to merge with.\n",
       "how : {'left', 'right', 'outer', 'inner'}, default 'inner'\n",
       "    Type of merge to be performed.\n",
       "\n",
       "    * left: use only keys from left frame, similar to a SQL left outer join;\n",
       "      preserve key order.\n",
       "    * right: use only keys from right frame, similar to a SQL right outer join;\n",
       "      preserve key order.\n",
       "    * outer: use union of keys from both frames, similar to a SQL full outer\n",
       "      join; sort keys lexicographically.\n",
       "    * inner: use intersection of keys from both frames, similar to a SQL inner\n",
       "      join; preserve the order of the left keys.\n",
       "on : label or list\n",
       "    Column or index level names to join on. These must be found in both\n",
       "    DataFrames. If `on` is None and not merging on indexes then this defaults\n",
       "    to the intersection of the columns in both DataFrames.\n",
       "left_on : label or list, or array-like\n",
       "    Column or index level names to join on in the left DataFrame. Can also\n",
       "    be an array or list of arrays of the length of the left DataFrame.\n",
       "    These arrays are treated as if they are columns.\n",
       "right_on : label or list, or array-like\n",
       "    Column or index level names to join on in the right DataFrame. Can also\n",
       "    be an array or list of arrays of the length of the right DataFrame.\n",
       "    These arrays are treated as if they are columns.\n",
       "left_index : bool, default False\n",
       "    Use the index from the left DataFrame as the join key(s). If it is a\n",
       "    MultiIndex, the number of keys in the other DataFrame (either the index\n",
       "    or a number of columns) must match the number of levels.\n",
       "right_index : bool, default False\n",
       "    Use the index from the right DataFrame as the join key. Same caveats as\n",
       "    left_index.\n",
       "sort : bool, default False\n",
       "    Sort the join keys lexicographically in the result DataFrame. If False,\n",
       "    the order of the join keys depends on the join type (how keyword).\n",
       "suffixes : tuple of (str, str), default ('_x', '_y')\n",
       "    Suffix to apply to overlapping column names in the left and right\n",
       "    side, respectively. To raise an exception on overlapping columns use\n",
       "    (False, False).\n",
       "copy : bool, default True\n",
       "    If False, avoid copy if possible.\n",
       "indicator : bool or str, default False\n",
       "    If True, adds a column to output DataFrame called \"_merge\" with\n",
       "    information on the source of each row.\n",
       "    If string, column with information on source of each row will be added to\n",
       "    output DataFrame, and column will be named value of string.\n",
       "    Information column is Categorical-type and takes on a value of \"left_only\"\n",
       "    for observations whose merge key only appears in 'left' DataFrame,\n",
       "    \"right_only\" for observations whose merge key only appears in 'right'\n",
       "    DataFrame, and \"both\" if the observation's merge key is found in both.\n",
       "\n",
       "validate : str, optional\n",
       "    If specified, checks if merge is of specified type.\n",
       "\n",
       "    * \"one_to_one\" or \"1:1\": check if merge keys are unique in both\n",
       "      left and right datasets.\n",
       "    * \"one_to_many\" or \"1:m\": check if merge keys are unique in left\n",
       "      dataset.\n",
       "    * \"many_to_one\" or \"m:1\": check if merge keys are unique in right\n",
       "      dataset.\n",
       "    * \"many_to_many\" or \"m:m\": allowed, but does not result in checks.\n",
       "\n",
       "    .. versionadded:: 0.21.0\n",
       "\n",
       "Returns\n",
       "-------\n",
       "DataFrame\n",
       "    A DataFrame of the two merged objects.\n",
       "\n",
       "See Also\n",
       "--------\n",
       "merge_ordered : Merge with optional filling/interpolation.\n",
       "merge_asof : Merge on nearest keys.\n",
       "DataFrame.join : Similar method using indices.\n",
       "\n",
       "Notes\n",
       "-----\n",
       "Support for specifying index levels as the `on`, `left_on`, and\n",
       "`right_on` parameters was added in version 0.23.0\n",
       "Support for merging named Series objects was added in version 0.24.0\n",
       "\n",
       "Examples\n",
       "--------\n",
       "\n",
       ">>> df1 = pd.DataFrame({'lkey': ['foo', 'bar', 'baz', 'foo'],\n",
       "...                     'value': [1, 2, 3, 5]})\n",
       ">>> df2 = pd.DataFrame({'rkey': ['foo', 'bar', 'baz', 'foo'],\n",
       "...                     'value': [5, 6, 7, 8]})\n",
       ">>> df1\n",
       "    lkey value\n",
       "0   foo      1\n",
       "1   bar      2\n",
       "2   baz      3\n",
       "3   foo      5\n",
       ">>> df2\n",
       "    rkey value\n",
       "0   foo      5\n",
       "1   bar      6\n",
       "2   baz      7\n",
       "3   foo      8\n",
       "\n",
       "Merge df1 and df2 on the lkey and rkey columns. The value columns have\n",
       "the default suffixes, _x and _y, appended.\n",
       "\n",
       ">>> df1.merge(df2, left_on='lkey', right_on='rkey')\n",
       "  lkey  value_x rkey  value_y\n",
       "0  foo        1  foo        5\n",
       "1  foo        1  foo        8\n",
       "2  foo        5  foo        5\n",
       "3  foo        5  foo        8\n",
       "4  bar        2  bar        6\n",
       "5  baz        3  baz        7\n",
       "\n",
       "Merge DataFrames df1 and df2 with specified left and right suffixes\n",
       "appended to any overlapping columns.\n",
       "\n",
       ">>> df1.merge(df2, left_on='lkey', right_on='rkey',\n",
       "...           suffixes=('_left', '_right'))\n",
       "  lkey  value_left rkey  value_right\n",
       "0  foo           1  foo            5\n",
       "1  foo           1  foo            8\n",
       "2  foo           5  foo            5\n",
       "3  foo           5  foo            8\n",
       "4  bar           2  bar            6\n",
       "5  baz           3  baz            7\n",
       "\n",
       "Merge DataFrames df1 and df2, but raise an exception if the DataFrames have\n",
       "any overlapping columns.\n",
       "\n",
       ">>> df1.merge(df2, left_on='lkey', right_on='rkey', suffixes=(False, False))\n",
       "Traceback (most recent call last):\n",
       "...\n",
       "ValueError: columns overlap but no suffix specified:\n",
       "    Index(['value'], dtype='object')\n",
       "\u001b[0;31mFile:\u001b[0m      ~/opt/anaconda3/lib/python3.7/site-packages/pandas/core/reshape/merge.py\n",
       "\u001b[0;31mType:\u001b[0m      function\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "?pd.merge"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## True/False questions (15 min)\n",
    "\n",
    "1. `CatBoost` is effective because, when increasing `iterations`, you lower the training error and the approximation error. \n",
    "2. `CatBoost` is likely to be popular in 10 years.\n",
    "3. The primary motivation for using `SGDClassifier` or `SGDRegressor` is speed.\n",
    "4. In multi-class logistic regression, if the coefficient for feature 10, class 2 is positive, that means increasing the value of feature 10 _decreases_ the predicted probability of class 1 (a different class).\n",
    "5. If we are dealing with data from multiple sources, our strategy is to first combine them as a preprocessing step, and then build a model.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}