{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Improving a model with Grid Search\n",
    "\n",
    "In this mini-lab, we'll fit a decision tree model to some sample data. This initial model will overfit heavily. Then we'll use Grid Search to find better parameters for this model, to reduce the overfitting.\n",
    "\n",
    "First, some imports."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Reading and plotting the data\n",
    "Now, a function that will help us read the csv file, and plot the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 14652 characters]\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f087fcc5cc0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def load_pts(csv_name):\n",
    "    data = np.asarray(pd.read_csv(csv_name, header=None))\n",
    "    X = data[:,0:2]\n",
    "    y = data[:,2]\n",
    "\n",
    "    plt.scatter(X[np.argwhere(y==0).flatten(),0], X[np.argwhere(y==0).flatten(),1],s = 50, color = 'blue', edgecolor = 'k')\n",
    "    plt.scatter(X[np.argwhere(y==1).flatten(),0], X[np.argwhere(y==1).flatten(),1],s = 50, color = 'red', edgecolor = 'k')\n",
    "    \n",
    "    plt.xlim(-2.05,2.05)\n",
    "    plt.ylim(-2.05,2.05)\n",
    "    plt.grid(False)\n",
    "    plt.tick_params(\n",
    "        axis='x',\n",
    "        which='both',\n",
    "        bottom='off',\n",
    "        top='off')\n",
    "\n",
    "    return X,y\n",
    "\n",
    "X, y = load_pts('data.csv')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Splitting our data into training and testing sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import f1_score, make_scorer\n",
    "\n",
    "#Fixing a random seed\n",
    "import random\n",
    "random.seed(42)\n",
    "\n",
    "# Split the data into training and testing sets\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Fitting a Decision Tree model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.tree import DecisionTreeClassifier\n",
    "\n",
    "# Define the model (with default hyperparameters)\n",
    "clf = DecisionTreeClassifier(random_state=42)\n",
    "\n",
    "# Fit the model\n",
    "clf.fit(X_train, y_train)\n",
    "\n",
    "# Make predictions\n",
    "train_predictions = clf.predict(X_train)\n",
    "test_predictions = clf.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's plot the model, and find the testing f1_score, to see how we did."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following function will help us plot the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_model(X, y, clf):\n",
    "    plt.scatter(X[np.argwhere(y==0).flatten(),0],X[np.argwhere(y==0).flatten(),1],s = 50, color = 'blue', edgecolor = 'k')\n",
    "    plt.scatter(X[np.argwhere(y==1).flatten(),0],X[np.argwhere(y==1).flatten(),1],s = 50, color = 'red', edgecolor = 'k')\n",
    "\n",
    "    plt.xlim(-2.05,2.05)\n",
    "    plt.ylim(-2.05,2.05)\n",
    "    plt.grid(False)\n",
    "    plt.tick_params(\n",
    "        axis='x',\n",
    "        which='both',\n",
    "        bottom='off',\n",
    "        top='off')\n",
    "\n",
    "    r = np.linspace(-2.1,2.1,300)\n",
    "    s,t = np.meshgrid(r,r)\n",
    "    s = np.reshape(s,(np.size(s),1))\n",
    "    t = np.reshape(t,(np.size(t),1))\n",
    "    h = np.concatenate((s,t),1)\n",
    "\n",
    "    z = clf.predict(h)\n",
    "\n",
    "    s = s.reshape((np.size(r),np.size(r)))\n",
    "    t = t.reshape((np.size(r),np.size(r)))\n",
    "    z = z.reshape((np.size(r),np.size(r)))\n",
    "\n",
    "    plt.contourf(s,t,z,colors = ['blue','red'],alpha = 0.2,levels = range(-1,2))\n",
    "    if len(np.unique(z)) > 1:\n",
    "        plt.contour(s,t,z,colors = 'k', linewidths = 2)\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 18408 characters]\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f08751c5ba8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Training F1 Score is 1.0\n",
      "The Testing F1 Score is 0.7\n"
     ]
    }
   ],
   "source": [
    "plot_model(X, y, clf)\n",
    "print('The Training F1 Score is', f1_score(train_predictions, y_train))\n",
    "print('The Testing F1 Score is', f1_score(test_predictions, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Woah! Some heavy overfitting there. Not just from looking at the graph, but also from looking at the difference between the high training score (1.0) and the low testing score (0.7).Let's see if we can find better hyperparameters for this model to do better. We'll use grid search for this.\n",
    "\n",
    "### 4. (TODO) Use grid search to improve this model.\n",
    "\n",
    "In here, we'll do the following steps:\n",
    "1. First define some parameters to perform grid search on. We suggest to play with `max_depth`, `min_samples_leaf`, and `min_samples_split`.\n",
    "2. Make a scorer for the model using `f1_score`.\n",
    "3. Perform grid search on the classifier, using the parameters and the scorer.\n",
    "4. Fit the data to the new classifier.\n",
    "5. Plot the model and find the f1_score.\n",
    "6. If the model is not much better, try changing the ranges for the parameters and fit it again.\n",
    "\n",
    "**_Hint:_ If you're stuck and would like to see a working solution, check the solutions notebook in this same folder.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import make_scorer\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "clf = DecisionTreeClassifier(random_state=42)\n",
    "\n",
    "# TODO: Create the parameters list you wish to tune.\n",
    "parameters = None\n",
    "\n",
    "# TODO: Make an fbeta_score scoring object.\n",
    "scorer = None\n",
    "\n",
    "# TODO: Perform grid search on the classifier using 'scorer' as the scoring method.\n",
    "grid_obj = None\n",
    "\n",
    "# TODO: Fit the grid search object to the training data and find the optimal parameters.\n",
    "grid_fit = None\n",
    "\n",
    "# TODO: Get the estimator.\n",
    "best_clf = None\n",
    "\n",
    "# Fit the new model.\n",
    "best_clf.fit(X_train, y_train)\n",
    "\n",
    "# Make predictions using the new model.\n",
    "best_train_predictions = best_clf.predict(X_train)\n",
    "best_test_predictions = best_clf.predict(X_test)\n",
    "\n",
    "# Calculate the f1_score of the new model.\n",
    "print('The training F1 Score is', f1_score(best_train_predictions, y_train))\n",
    "print('The testing F1 Score is', f1_score(best_test_predictions, y_test))\n",
    "\n",
    "# Plot the new model.\n",
    "plot_model(X, y, best_clf)\n",
    "\n",
    "# Let's also explore what parameters ended up being used in the new model.\n",
    "best_clf"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
