{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PCA is commonly used with high dimensional data.  One type of high dimensional data is images. In this example, I will show you a similar use case using images of numbers.\n",
    "\n",
    "A classic example of working with image data is the MNIST dataset, which was open sourced in the late 1990s by researchers across Microsoft, Google, and NYU.  I have linked more information about the dataset below this video in the workbook:  http://yann.lecun.com/exdb/mnist/\n",
    "\n",
    "To get started, let's import the libraries we will need for the rest of the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.decomposition import PCA\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import confusion_matrix, accuracy_score\n",
    "\n",
    "import matplotlib.image as mpimg\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dataset has 42,000 images in the training dataset, and another 28,000 images in a test set.  Let's read in the datasets and see what they look like.  If you would like to follow along with me, you can open the workspace below this video in another window and work along with me.  Let's take a closer look at the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "train = pd.read_csv('./data/train.csv')\n",
    "train.fillna(0, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loading the data may take a bit of time, but once it is loaded, we will focus on the training dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>pixel0</th>\n",
       "      <th>pixel1</th>\n",
       "      <th>pixel2</th>\n",
       "      <th>pixel3</th>\n",
       "      <th>pixel4</th>\n",
       "      <th>pixel5</th>\n",
       "      <th>pixel6</th>\n",
       "      <th>pixel7</th>\n",
       "      <th>pixel8</th>\n",
       "      <th>...</th>\n",
       "      <th>pixel774</th>\n",
       "      <th>pixel775</th>\n",
       "      <th>pixel776</th>\n",
       "      <th>pixel777</th>\n",
       "      <th>pixel778</th>\n",
       "      <th>pixel779</th>\n",
       "      <th>pixel780</th>\n",
       "      <th>pixel781</th>\n",
       "      <th>pixel782</th>\n",
       "      <th>pixel783</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 785 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   label  pixel0  pixel1  pixel2  pixel3  pixel4  pixel5  pixel6  pixel7  \\\n",
       "0      1       0       0       0       0       0       0       0       0   \n",
       "1      0       0       0       0       0       0       0       0       0   \n",
       "2      1       0       0       0       0       0       0       0       0   \n",
       "3      4       0       0       0       0       0       0       0       0   \n",
       "4      0       0       0       0       0       0       0       0       0   \n",
       "\n",
       "   pixel8    ...     pixel774  pixel775  pixel776  pixel777  pixel778  \\\n",
       "0       0    ...          0.0       0.0       0.0       0.0       0.0   \n",
       "1       0    ...          0.0       0.0       0.0       0.0       0.0   \n",
       "2       0    ...          0.0       0.0       0.0       0.0       0.0   \n",
       "3       0    ...          0.0       0.0       0.0       0.0       0.0   \n",
       "4       0    ...          0.0       0.0       0.0       0.0       0.0   \n",
       "\n",
       "   pixel779  pixel780  pixel781  pixel782  pixel783  \n",
       "0       0.0       0.0       0.0       0.0       0.0  \n",
       "1       0.0       0.0       0.0       0.0       0.0  \n",
       "2       0.0       0.0       0.0       0.0       0.0  \n",
       "3       0.0       0.0       0.0       0.0       0.0  \n",
       "4       0.0       0.0       0.0       0.0       0.0  \n",
       "\n",
       "[5 rows x 785 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>pixel0</th>\n",
       "      <th>pixel1</th>\n",
       "      <th>pixel2</th>\n",
       "      <th>pixel3</th>\n",
       "      <th>pixel4</th>\n",
       "      <th>pixel5</th>\n",
       "      <th>pixel6</th>\n",
       "      <th>pixel7</th>\n",
       "      <th>pixel8</th>\n",
       "      <th>...</th>\n",
       "      <th>pixel774</th>\n",
       "      <th>pixel775</th>\n",
       "      <th>pixel776</th>\n",
       "      <th>pixel777</th>\n",
       "      <th>pixel778</th>\n",
       "      <th>pixel779</th>\n",
       "      <th>pixel780</th>\n",
       "      <th>pixel781</th>\n",
       "      <th>pixel782</th>\n",
       "      <th>pixel783</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>6304.000000</td>\n",
       "      <td>6304.0</td>\n",
       "      <td>6304.0</td>\n",
       "      <td>6304.0</td>\n",
       "      <td>6304.0</td>\n",
       "      <td>6304.0</td>\n",
       "      <td>6304.0</td>\n",
       "      <td>6304.0</td>\n",
       "      <td>6304.0</td>\n",
       "      <td>6304.0</td>\n",
       "      <td>...</td>\n",
       "      <td>6304.000000</td>\n",
       "      <td>6304.000000</td>\n",
       "      <td>6304.000000</td>\n",
       "      <td>6304.0</td>\n",
       "      <td>6304.0</td>\n",
       "      <td>6304.0</td>\n",
       "      <td>6304.0</td>\n",
       "      <td>6304.0</td>\n",
       "      <td>6304.0</td>\n",
       "      <td>6304.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>4.440355</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.139594</td>\n",
       "      <td>0.142291</td>\n",
       "      <td>0.026967</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>2.885613</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>5.099940</td>\n",
       "      <td>5.531089</td>\n",
       "      <td>1.675547</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>4.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>7.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>9.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>253.000000</td>\n",
       "      <td>253.000000</td>\n",
       "      <td>130.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>8 rows × 785 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             label  pixel0  pixel1  pixel2  pixel3  pixel4  pixel5  pixel6  \\\n",
       "count  6304.000000  6304.0  6304.0  6304.0  6304.0  6304.0  6304.0  6304.0   \n",
       "mean      4.440355     0.0     0.0     0.0     0.0     0.0     0.0     0.0   \n",
       "std       2.885613     0.0     0.0     0.0     0.0     0.0     0.0     0.0   \n",
       "min       0.000000     0.0     0.0     0.0     0.0     0.0     0.0     0.0   \n",
       "25%       2.000000     0.0     0.0     0.0     0.0     0.0     0.0     0.0   \n",
       "50%       4.000000     0.0     0.0     0.0     0.0     0.0     0.0     0.0   \n",
       "75%       7.000000     0.0     0.0     0.0     0.0     0.0     0.0     0.0   \n",
       "max       9.000000     0.0     0.0     0.0     0.0     0.0     0.0     0.0   \n",
       "\n",
       "       pixel7  pixel8    ...        pixel774     pixel775     pixel776  \\\n",
       "count  6304.0  6304.0    ...     6304.000000  6304.000000  6304.000000   \n",
       "mean      0.0     0.0    ...        0.139594     0.142291     0.026967   \n",
       "std       0.0     0.0    ...        5.099940     5.531089     1.675547   \n",
       "min       0.0     0.0    ...        0.000000     0.000000     0.000000   \n",
       "25%       0.0     0.0    ...        0.000000     0.000000     0.000000   \n",
       "50%       0.0     0.0    ...        0.000000     0.000000     0.000000   \n",
       "75%       0.0     0.0    ...        0.000000     0.000000     0.000000   \n",
       "max       0.0     0.0    ...      253.000000   253.000000   130.000000   \n",
       "\n",
       "       pixel777  pixel778  pixel779  pixel780  pixel781  pixel782  pixel783  \n",
       "count    6304.0    6304.0    6304.0    6304.0    6304.0    6304.0    6304.0  \n",
       "mean        0.0       0.0       0.0       0.0       0.0       0.0       0.0  \n",
       "std         0.0       0.0       0.0       0.0       0.0       0.0       0.0  \n",
       "min         0.0       0.0       0.0       0.0       0.0       0.0       0.0  \n",
       "25%         0.0       0.0       0.0       0.0       0.0       0.0       0.0  \n",
       "50%         0.0       0.0       0.0       0.0       0.0       0.0       0.0  \n",
       "75%         0.0       0.0       0.0       0.0       0.0       0.0       0.0  \n",
       "max         0.0       0.0       0.0       0.0       0.0       0.0       0.0  \n",
       "\n",
       "[8 rows x 785 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From a header and the describe of the data, we can see that the data has a label column with what digit was created, and the pixels for each part of the image are the features.  The values for the pixels are between 0 and 255.  You can find an article on the pixel values in the article below in the notebook. https://homepages.inf.ed.ac.uk/rbf/HIPR2/value.htm\n",
    "\n",
    "Given what we know about the data, let's split the label off and put the images in their own matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "train.fillna(0, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save the labels to a Pandas series target\n",
    "y = train['label']\n",
    "# Drop the label feature\n",
    "X = train.drop(\"label\",axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 7288 characters]\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f52cfffeb70>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.countplot(y, color = sns.color_palette()[0]); #a very quick look at the data shows that all the labels appear roughly 4k times"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at what a few of the numbers look like with this little helper function. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def show_images(num_images):\n",
    "    '''\n",
    "    This function plots the num_images provided of MNIST dataset. \n",
    "    \n",
    "    INPUT: num_images - int - The number of images you would like to view.\n",
    "           Mod 10 of num_images should be 0 and it should be fewer than 101 images.\n",
    "    OUTPUT: A figure with the images shown for the training data.\n",
    "    '''\n",
    "    if num_images % 10 == 0 and num_images <= 100:\n",
    "        for digit_num in range(0,num_images): \n",
    "            plt.subplot(num_images/10,10,digit_num+1) #create subplots\n",
    "            mat_data = X.iloc[digit_num].as_matrix().reshape(28,28)  #reshape images\n",
    "            plt.imshow(mat_data) #plot the data\n",
    "            plt.xticks([]) #removes numbered labels on x-axis\n",
    "            plt.yticks([]) #removes numbered labels on y-axis\n",
    "    else:\n",
    "        print('That is not the right input, please read the docstring before continuing.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.6/site-packages/ipykernel_launcher.py:12: FutureWarning: Method .as_matrix will be removed in a future version. Use .values instead.\n",
      "  if sys.path[0] == '':\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 39700 characters]\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f52cffc85f8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_images(100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some of these look pretty easy to identify, and others not so much... I wrote another helper function to be able to see each of the first 50 images of any one type of number."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def show_images_by_digit(digit_to_see):\n",
    "    '''\n",
    "    This function plots the digits_to_see provided of MNIST dataset. \n",
    "    \n",
    "    INPUT: digits_to_see - int - A number between 0 and 9 of what you want to see.\n",
    "    OUTPUT: A figure with the images shown for the training data.\n",
    "    '''\n",
    "    if digit_to_see in list(range(10)):\n",
    "        indices = np.where(y == digit_to_see) # pull indices for num of interest\n",
    "        for digit_num in range(0,50): \n",
    "            plt.subplot(5,10, digit_num+1) #create subplots\n",
    "            mat_data = X.iloc[indices[0][digit_num]].as_matrix().reshape(28,28)  #reshape images\n",
    "            plt.imshow(mat_data) #plot the data\n",
    "            plt.xticks([]) #removes numbered labels on x-axis\n",
    "            plt.yticks([]) #removes numbered labels on y-axis   \n",
    "    else:\n",
    "        print('That is not the right input, please read the docstring before continuing.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.6/site-packages/ipykernel_launcher.py:12: FutureWarning: Method .as_matrix will be removed in a future version. Use .values instead.\n",
      "  if sys.path[0] == '':\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 21748 characters]\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f52c83a75c0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_images_by_digit(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some of these ones are pretty wild looking...  One common way to use PCA is to reduce the dimensionality of high dimensionality data that you want to use for prediction, but the results seem to be overfitting (potentially because their is a lot of noise in the data.  Which can certainly be the case with image data).\n",
    "\n",
    "Let's take a first pass on creating a simple model to predict the values of the images using all of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[205   0   0   0   0   0   4   0   0   0]\n",
      " [  0 234   3   0   1   0   1   2   2   0]\n",
      " [  1   4 211   1   3   0   1   7   0   0]\n",
      " [  2   0   5 171   0   6   0   1   4   2]\n",
      " [  0   0   1   0 165   1   4   0   0   7]\n",
      " [  2   1   0   4   1 174   2   1   1   0]\n",
      " [  3   0   1   0   0   2 204   1   0   0]\n",
      " [  0   0   6   1   5   0   0 205   2   6]\n",
      " [  1   1   2   7   0   3   0   1 185   2]\n",
      " [  2   1   0   2  13   2   0   2   3 183]]\n",
      "AxesSubplot(0.125,0.125;0.62x0.755)\n",
      "0.930802498799\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.93080249879865451"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 19440 characters]\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f52c9000208>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#Because we will fit this more than once, might as well make a function\n",
    "def fit_random_forest_classifier(X, y):\n",
    "    '''\n",
    "    INPUT: names are pretty self explanatory\n",
    "    OUTPUT: none - prints the confusion matrix and accuracy\n",
    "    '''\n",
    "    #First let's create training and testing data\n",
    "    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)\n",
    "\n",
    "    #We could grid search and tune, but let's just fit a simple model to see how it does\n",
    "    #instantiate\n",
    "    clf = RandomForestClassifier(n_estimators=100, max_depth=None)\n",
    "\n",
    "    #fit\n",
    "    clf.fit(X_train, y_train)\n",
    "\n",
    "    #predict\n",
    "    y_preds = clf.predict(X_test)\n",
    "\n",
    "    #score\n",
    "    mat = confusion_matrix(y_test, y_preds)\n",
    "    print(mat)\n",
    "    print(sns.heatmap(mat, annot=True, cmap='bwr', linewidths=.5))\n",
    "    acc = accuracy_score(y_test, y_preds)\n",
    "    print(acc)\n",
    "    return acc\n",
    "    \n",
    "fit_random_forest_classifier(X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above model does pretty well on the test set using all of the data, let's see how we well a model can do with a much lower number of features.  Perhaps, we can do as well or better by reducing the noise in the original features.\n",
    "\n",
    "Working with unsupervised techniques in scikit learn follows a similar process as working with supervised techniques, but excludes predicting and scoring, and instead we just need to transform our data.  Often the fit and transform are done all in one step.\n",
    "\n",
    "1. Instantiate\n",
    "2. Fit \n",
    "3. Transform \n",
    "\n",
    "It is worth mentioning that because all of our features are on the same scale from 0 to 255, scaling isn't super necessary here.  At the same time, I usually try to create processes that are extendible without having to think and dive into the details again.  Without standardizing, this process would only work for data that was already scaled.\n",
    "\n",
    "You can use the below function to perform PCA on this dataset, but also any other dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def do_pca(n_components, data):\n",
    "    '''\n",
    "    Transforms data using PCA to create n_components, and provides back the results of the\n",
    "    transformation.\n",
    "    \n",
    "    INPUT: n_components - int - the number of principal components to create\n",
    "           data - the data you would like to transform\n",
    "           \n",
    "    OUTPUT: pca - the pca object created after fitting the data\n",
    "            X_pca - the transformed X matrix with new number of components\n",
    "    '''\n",
    "    X = StandardScaler().fit_transform(data)\n",
    "    pca = PCA(n_components)\n",
    "    X_pca = pca.fit_transform(X)\n",
    "    return pca, X_pca"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "pca, X_pca = do_pca(2, X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(6304, 2)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_pca.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[108   0  46   8   6  10  26   0   5   0]\n",
      " [  0 207   0   4   2   7   4   5   8   6]\n",
      " [ 40   2  67  28  15  22  33   1  16   4]\n",
      " [  9   1  27  22  27  29  25  10  25  16]\n",
      " [  5   4  17  17  50  16  16  12  12  29]\n",
      " [  9   8  31  23  16  20  30   4  34  11]\n",
      " [ 33   6  53  17   8  28  32   4  28   2]\n",
      " [  3   5   1   8  38  10   3  95   8  54]\n",
      " [  7   7  24  29  22  30  30   9  34  10]\n",
      " [  2  12   4  17  30  10   4  53  10  66]]\n",
      "AxesSubplot(0.125,0.125;0.62x0.755)\n",
      "0.336857280154\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.3368572801537722"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 23916 characters]\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f52c86c8eb8>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fit_random_forest_classifier(X_pca, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, let's look at the positives first... that was super fast.  But clearly two components isn't giving us enough information to clearly identify the digits.  Let's consider a few more components, and take a closer look at what we know about the components."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2, 784)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pca.components_.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_components(X, y):\n",
    "    '''\n",
    "    plots the data in a 2 dimensional space to view separation\n",
    "    INPUT: pretty straightforward again\n",
    "    OUTPUT: none\n",
    "    '''\n",
    "    x_min, x_max = np.min(X, 0), np.max(X, 0)\n",
    "    X = (X - x_min) / (x_max - x_min)\n",
    "    plt.figure(figsize=(10, 6))\n",
    "    for i in range(X.shape[0]):\n",
    "        plt.text(X[i, 0], X[i, 1], str(y[i]), color=plt.cm.Set1(y[i]), fontdict={'size': 15})\n",
    "\n",
    "    plt.xticks([]), plt.yticks([]), plt.ylim([-0.1,1.1]), plt.xlim([-0.1,1.1])\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 23868 characters]\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f52cfff0550>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_components(X_pca[:100], y[:100])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
