{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predicting Student Admissions with Neural Networks\n",
    "In this notebook, we predict student admissions to graduate school at UCLA based on three pieces of data:\n",
    "- GRE Scores (Test)\n",
    "- GPA Scores (Grades)\n",
    "- Class rank (1-4)\n",
    "\n",
    "The dataset originally came from here: http://www.ats.ucla.edu/\n",
    "\n",
    "## Loading the data\n",
    "To load the data and format it nicely, we will use two very useful packages called Pandas and Numpy. You can read on the documentation here:\n",
    "- https://pandas.pydata.org/pandas-docs/stable/\n",
    "- https://docs.scipy.org/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>admit</th>\n",
       "      <th>gre</th>\n",
       "      <th>gpa</th>\n",
       "      <th>rank</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>380</td>\n",
       "      <td>3.61</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>660</td>\n",
       "      <td>3.67</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>800</td>\n",
       "      <td>4.00</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>640</td>\n",
       "      <td>3.19</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>520</td>\n",
       "      <td>2.93</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1</td>\n",
       "      <td>760</td>\n",
       "      <td>3.00</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>1</td>\n",
       "      <td>560</td>\n",
       "      <td>2.98</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0</td>\n",
       "      <td>400</td>\n",
       "      <td>3.08</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>1</td>\n",
       "      <td>540</td>\n",
       "      <td>3.39</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0</td>\n",
       "      <td>700</td>\n",
       "      <td>3.92</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   admit  gre   gpa  rank\n",
       "0      0  380  3.61     3\n",
       "1      1  660  3.67     3\n",
       "2      1  800  4.00     1\n",
       "3      1  640  3.19     4\n",
       "4      0  520  2.93     4\n",
       "5      1  760  3.00     2\n",
       "6      1  560  2.98     1\n",
       "7      0  400  3.08     2\n",
       "8      1  540  3.39     3\n",
       "9      0  700  3.92     2"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Importing pandas and numpy\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# Reading the csv file into a pandas DataFrame\n",
    "data = pd.read_csv('student_data.csv')\n",
    "\n",
    "# Printing out the first 10 rows of our data\n",
    "data[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plotting the data\n",
    "\n",
    "First let's make a plot of our data to see how it looks. In order to have a 2D plot, let's ingore the rank."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f20c44f7d30>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Importing matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Function to help us plot\n",
    "def plot_points(data):\n",
    "    X = np.array(data[[\"gre\",\"gpa\"]])\n",
    "    y = np.array(data[\"admit\"])\n",
    "    admitted = X[np.argwhere(y==1)]\n",
    "    rejected = X[np.argwhere(y==0)]\n",
    "    plt.scatter([s[0][0] for s in rejected], [s[0][1] for s in rejected], s = 25, color = 'red', edgecolor = 'k')\n",
    "    plt.scatter([s[0][0] for s in admitted], [s[0][1] for s in admitted], s = 25, color = 'cyan', edgecolor = 'k')\n",
    "    plt.xlabel('Test (GRE)')\n",
    "    plt.ylabel('Grades (GPA)')\n",
    "    \n",
    "# Plotting the points\n",
    "plot_points(data)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Roughly, it looks like the students with high scores in the grades and test passed, while the ones with low scores didn't, but the data is not as nicely separable as we hoped it would. Maybe it would help to take the rank into account? Let's make 4 plots, each one for each rank."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 14256 characters]\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f20bfef52e8>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 19420 characters]\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f20bfed25c0>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 17040 characters]\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f20bfdd5630>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 15056 characters]\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f20bfdcf358>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Separating the ranks\n",
    "data_rank1 = data[data[\"rank\"]==1]\n",
    "data_rank2 = data[data[\"rank\"]==2]\n",
    "data_rank3 = data[data[\"rank\"]==3]\n",
    "data_rank4 = data[data[\"rank\"]==4]\n",
    "\n",
    "# Plotting the graphs\n",
    "plot_points(data_rank1)\n",
    "plt.title(\"Rank 1\")\n",
    "plt.show()\n",
    "plot_points(data_rank2)\n",
    "plt.title(\"Rank 2\")\n",
    "plt.show()\n",
    "plot_points(data_rank3)\n",
    "plt.title(\"Rank 3\")\n",
    "plt.show()\n",
    "plot_points(data_rank4)\n",
    "plt.title(\"Rank 4\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This looks more promising, as it seems that the lower the rank, the higher the acceptance rate. Let's use the rank as one of our inputs. In order to do this, we should one-hot encode it.\n",
    "\n",
    "## TODO: One-hot encoding the rank\n",
    "Use the `get_dummies` function in Pandas in order to one-hot encode the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>admit</th>\n",
       "      <th>gre</th>\n",
       "      <th>gpa</th>\n",
       "      <th>rank_1</th>\n",
       "      <th>rank_2</th>\n",
       "      <th>rank_3</th>\n",
       "      <th>rank_4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>380</td>\n",
       "      <td>3.61</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>660</td>\n",
       "      <td>3.67</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>800</td>\n",
       "      <td>4.00</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>640</td>\n",
       "      <td>3.19</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>520</td>\n",
       "      <td>2.93</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1</td>\n",
       "      <td>760</td>\n",
       "      <td>3.00</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>1</td>\n",
       "      <td>560</td>\n",
       "      <td>2.98</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0</td>\n",
       "      <td>400</td>\n",
       "      <td>3.08</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>1</td>\n",
       "      <td>540</td>\n",
       "      <td>3.39</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0</td>\n",
       "      <td>700</td>\n",
       "      <td>3.92</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   admit  gre   gpa  rank_1  rank_2  rank_3  rank_4\n",
       "0      0  380  3.61       0       0       1       0\n",
       "1      1  660  3.67       0       0       1       0\n",
       "2      1  800  4.00       1       0       0       0\n",
       "3      1  640  3.19       0       0       0       1\n",
       "4      0  520  2.93       0       0       0       1\n",
       "5      1  760  3.00       0       1       0       0\n",
       "6      1  560  2.98       1       0       0       0\n",
       "7      0  400  3.08       0       1       0       0\n",
       "8      1  540  3.39       0       0       1       0\n",
       "9      0  700  3.92       0       1       0       0"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# TODO:  Make dummy variables for rank\n",
    "one_hot_data = pd.concat([data, pd.get_dummies(data['rank'], prefix='rank')], axis=1)\n",
    "\n",
    "# TODO: Drop the previous rank column\n",
    "one_hot_data = one_hot_data.drop('rank', axis=1)\n",
    "\n",
    "# Print the first 10 rows of our data\n",
    "one_hot_data[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TODO: Scaling the data\n",
    "The next step is to scale the data. We notice that the range for grades is 1.0-4.0, whereas the range for test scores is roughly 200-800, which is much larger. This means our data is skewed, and that makes it hard for a neural network to handle. Let's fit our two features into a range of 0-1, by dividing the grades by 4.0, and the test score by 800."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>admit</th>\n",
       "      <th>gre</th>\n",
       "      <th>gpa</th>\n",
       "      <th>rank_1</th>\n",
       "      <th>rank_2</th>\n",
       "      <th>rank_3</th>\n",
       "      <th>rank_4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0.475</td>\n",
       "      <td>0.9025</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>0.825</td>\n",
       "      <td>0.9175</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>1.000</td>\n",
       "      <td>1.0000</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>0.800</td>\n",
       "      <td>0.7975</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>0.650</td>\n",
       "      <td>0.7325</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1</td>\n",
       "      <td>0.950</td>\n",
       "      <td>0.7500</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>1</td>\n",
       "      <td>0.700</td>\n",
       "      <td>0.7450</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0</td>\n",
       "      <td>0.500</td>\n",
       "      <td>0.7700</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>1</td>\n",
       "      <td>0.675</td>\n",
       "      <td>0.8475</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0</td>\n",
       "      <td>0.875</td>\n",
       "      <td>0.9800</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   admit    gre     gpa  rank_1  rank_2  rank_3  rank_4\n",
       "0      0  0.475  0.9025       0       0       1       0\n",
       "1      1  0.825  0.9175       0       0       1       0\n",
       "2      1  1.000  1.0000       1       0       0       0\n",
       "3      1  0.800  0.7975       0       0       0       1\n",
       "4      0  0.650  0.7325       0       0       0       1\n",
       "5      1  0.950  0.7500       0       1       0       0\n",
       "6      1  0.700  0.7450       1       0       0       0\n",
       "7      0  0.500  0.7700       0       1       0       0\n",
       "8      1  0.675  0.8475       0       0       1       0\n",
       "9      0  0.875  0.9800       0       1       0       0"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Making a copy of our data\n",
    "processed_data = one_hot_data[:]\n",
    "\n",
    "# TODO: Scale the columns\n",
    "processed_data['gre'] = processed_data['gre']/800\n",
    "processed_data['gpa'] = processed_data['gpa']/4.0\n",
    "processed_data[:10]\n",
    "\n",
    "# Printing the first 10 rows of our procesed data\n",
    "processed_data[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Splitting the data into Training and Testing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to test our algorithm, we'll split the data into a Training and a Testing set. The size of the testing set will be 10% of the total data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of training samples is 360\n",
      "Number of testing samples is 40\n",
      "     admit    gre     gpa  rank_1  rank_2  rank_3  rank_4\n",
      "16       0  0.975  0.9675       0       0       0       1\n",
      "97       0  0.600  0.8925       0       1       0       0\n",
      "361      1  0.675  0.8725       1       0       0       0\n",
      "315      1  0.375  0.7100       0       1       0       0\n",
      "215      1  0.825  0.7275       0       0       1       0\n",
      "109      0  0.600  0.8625       0       1       0       0\n",
      "199      0  0.725  0.9425       0       0       0       1\n",
      "197      1  0.500  0.8075       0       0       0       1\n",
      "354      1  0.675  0.9450       0       1       0       0\n",
      "84       1  0.625  0.9000       0       0       1       0\n",
      "    admit    gre     gpa  rank_1  rank_2  rank_3  rank_4\n",
      "10      0  1.000  1.0000       0       0       0       1\n",
      "11      0  0.550  0.8050       1       0       0       0\n",
      "13      0  0.875  0.7700       0       1       0       0\n",
      "28      1  0.975  0.8050       0       1       0       0\n",
      "34      0  0.450  0.7850       1       0       0       0\n",
      "50      0  0.800  0.9650       0       0       1       0\n",
      "54      0  0.825  0.8350       0       0       1       0\n",
      "63      1  0.850  0.9625       0       0       1       0\n",
      "65      0  0.750  0.8975       0       1       0       0\n",
      "66      0  0.925  0.9050       0       0       0       1\n"
     ]
    }
   ],
   "source": [
    "sample = np.random.choice(processed_data.index, size=int(len(processed_data)*0.9), replace=False)\n",
    "train_data, test_data = processed_data.iloc[sample], processed_data.drop(sample)\n",
    "\n",
    "print(\"Number of training samples is\", len(train_data))\n",
    "print(\"Number of testing samples is\", len(test_data))\n",
    "print(train_data[:10])\n",
    "print(test_data[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Splitting the data into features and targets (labels)\n",
    "Now, as a final step before the training, we'll split the data into features (X) and targets (y)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "       gre     gpa  rank_1  rank_2  rank_3  rank_4\n",
      "16   0.975  0.9675       0       0       0       1\n",
      "97   0.600  0.8925       0       1       0       0\n",
      "361  0.675  0.8725       1       0       0       0\n",
      "315  0.375  0.7100       0       1       0       0\n",
      "215  0.825  0.7275       0       0       1       0\n",
      "109  0.600  0.8625       0       1       0       0\n",
      "199  0.725  0.9425       0       0       0       1\n",
      "197  0.500  0.8075       0       0       0       1\n",
      "354  0.675  0.9450       0       1       0       0\n",
      "84   0.625  0.9000       0       0       1       0\n",
      "16     0\n",
      "97     0\n",
      "361    1\n",
      "315    1\n",
      "215    1\n",
      "109    0\n",
      "199    0\n",
      "197    1\n",
      "354    1\n",
      "84     1\n",
      "Name: admit, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "features = train_data.drop('admit', axis=1)\n",
    "targets = train_data['admit']\n",
    "features_test = test_data.drop('admit', axis=1)\n",
    "targets_test = test_data['admit']\n",
    "\n",
    "print(features[:10])\n",
    "print(targets[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training the 2-layer Neural Network\n",
    "The following function trains the 2-layer neural network. First, we'll write some helper functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Activation (sigmoid) function\n",
    "def sigmoid(x):\n",
    "    return 1 / (1 + np.exp(-x))\n",
    "def sigmoid_prime(x):\n",
    "    return sigmoid(x) * (1-sigmoid(x))\n",
    "def error_formula(y, output):\n",
    "    return - y*np.log(output) - (1 - y) * np.log(1-output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TODO: Backpropagate the error\n",
    "Now it's your turn to shine. Write the error term. Remember that this is given by the equation $$ (y-\\hat{y}) \\sigma'(x) $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: Write the error term formula\n",
    "def error_term_formula(x, y, output):\n",
    "    return (y - output) * sigmoid_prime(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0\n",
      "Train loss:  0.274675763534\n",
      "=========\n",
      "Epoch: 100\n",
      "Train loss:  0.211541227151\n",
      "=========\n",
      "Epoch: 200\n",
      "Train loss:  0.209108941338\n",
      "=========\n",
      "Epoch: 300\n",
      "Train loss:  0.207892062032\n",
      "=========\n",
      "Epoch: 400\n",
      "Train loss:  0.207253836282\n",
      "=========\n",
      "Epoch: 500\n",
      "Train loss:  0.206889105737\n",
      "=========\n",
      "Epoch: 600\n",
      "Train loss:  0.206656034976\n",
      "=========\n",
      "Epoch: 700\n",
      "Train loss:  0.206488123278\n",
      "=========\n",
      "Epoch: 800\n",
      "Train loss:  0.206353568323\n",
      "=========\n",
      "Epoch: 900\n",
      "Train loss:  0.206236831468\n",
      "=========\n",
      "Finished training!\n"
     ]
    }
   ],
   "source": [
    "# Neural Network hyperparameters\n",
    "epochs = 1000\n",
    "learnrate = 0.5\n",
    "\n",
    "# Training function\n",
    "def train_nn(features, targets, epochs, learnrate):\n",
    "    \n",
    "    # Use to same seed to make debugging easier\n",
    "    np.random.seed(42)\n",
    "\n",
    "    n_records, n_features = features.shape\n",
    "    last_loss = None\n",
    "\n",
    "    # Initialize weights\n",
    "    weights = np.random.normal(scale=1 / n_features**.5, size=n_features)\n",
    "\n",
    "    for e in range(epochs):\n",
    "        del_w = np.zeros(weights.shape)\n",
    "        for x, y in zip(features.values, targets):\n",
    "            # Loop through all records, x is the input, y is the target\n",
    "\n",
    "            # Activation of the output unit\n",
    "            #   Notice we multiply the inputs and the weights here \n",
    "            #   rather than storing h as a separate variable \n",
    "            output = sigmoid(np.dot(x, weights))\n",
    "\n",
    "            # The error, the target minus the network output\n",
    "            error = error_formula(y, output)\n",
    "\n",
    "            # The error term\n",
    "            error_term = error_term_formula(x, y, output)\n",
    "\n",
    "            # The gradient descent step, the error times the gradient times the inputs\n",
    "            del_w += error_term * x\n",
    "\n",
    "        # Update the weights here. The learning rate times the \n",
    "        # change in weights, divided by the number of records to average\n",
    "        weights += learnrate * del_w / n_records\n",
    "\n",
    "        # Printing out the mean square error on the training set\n",
    "        if e % (epochs / 10) == 0:\n",
    "            out = sigmoid(np.dot(features, weights))\n",
    "            loss = np.mean((out - targets) ** 2)\n",
    "            print(\"Epoch:\", e)\n",
    "            if last_loss and last_loss < loss:\n",
    "                print(\"Train loss: \", loss, \"  WARNING - Loss Increasing\")\n",
    "            else:\n",
    "                print(\"Train loss: \", loss)\n",
    "            last_loss = loss\n",
    "            print(\"=========\")\n",
    "    print(\"Finished training!\")\n",
    "    return weights\n",
    "    \n",
    "weights = train_nn(features, targets, epochs, learnrate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calculating the Accuracy on the Test Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prediction accuracy: 0.700\n"
     ]
    }
   ],
   "source": [
    "# Calculate accuracy on test data\n",
    "test_out = sigmoid(np.dot(features_test, weights))\n",
    "predictions = test_out > 0.5\n",
    "accuracy = np.mean(predictions == targets_test)\n",
    "print(\"Prediction accuracy: {:.3f}\".format(accuracy))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
