{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have seen a bit about how the PCA library works within scikit learn library and fit it to some hand written digits, we should take a closer look at what exactly PCA gives us back to work with and what these different parts mean.  Let's pick up where we left off in the last video.  Below I read in the libraries we used before, split off the image from the label, and I plotted the first 30 images in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 25168 characters]\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f0cf357cd68>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.decomposition import PCA\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import confusion_matrix, accuracy_score\n",
    "from helper_functions import show_images, show_images_by_digit, fit_random_forest_classifier2 \n",
    "from helper_functions import fit_random_forest_classifier, do_pca, plot_components\n",
    "\n",
    "import matplotlib.image as mpimg\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "#read in our dataset\n",
    "train = pd.read_csv('./data/train.csv')\n",
    "train.fillna(0, inplace=True)\n",
    "\n",
    "# save the labels to a Pandas series target\n",
    "y = train['label']\n",
    "# Drop the label feature\n",
    "train.fillna(0, inplace=True)\n",
    "X = train.drop(\"label\",axis=1)\n",
    "\n",
    "show_images(30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's fit PCA with 15 components, and take a look at a few of the main features that live on the pca object we get back."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "pca, X_pca = do_pca(15, X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One of the main features to consider is something called the explained_variance_ratio.  I created what's known as a scree_plot to show the explained_variance_ratio for each of the 15 components."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 34780 characters]\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f0cf4299ba8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def scree_plot(pca):\n",
    "    '''\n",
    "    Creates a scree plot associated with the principal components \n",
    "    \n",
    "    INPUT: pca - the result of instantian of PCA in scikit learn\n",
    "            \n",
    "    OUTPUT:\n",
    "            None\n",
    "    '''\n",
    "    num_components = len(pca.explained_variance_ratio_)\n",
    "    ind = np.arange(num_components)\n",
    "    vals = pca.explained_variance_ratio_\n",
    " \n",
    "    plt.figure(figsize=(10, 6))\n",
    "    ax = plt.subplot(111)\n",
    "    cumvals = np.cumsum(vals)\n",
    "    ax.bar(ind, vals)\n",
    "    ax.plot(ind, cumvals)\n",
    "    for i in range(num_components):\n",
    "        ax.annotate(r\"%s%%\" % ((str(vals[i]*100)[:4])), (ind[i]+0.2, vals[i]), va=\"bottom\", ha=\"center\", fontsize=12)\n",
    " \n",
    "    ax.xaxis.set_tick_params(width=0)\n",
    "    ax.yaxis.set_tick_params(width=2, length=12)\n",
    " \n",
    "    ax.set_xlabel(\"Principal Component\")\n",
    "    ax.set_ylabel(\"Variance Explained (%)\")\n",
    "    plt.title('Explained Variance Per Principal Component')\n",
    "    \n",
    "\n",
    "scree_plot(pca)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each of the bars represents the amount of variability explained by each component.  So you can see the first component explains 5.74% of the variability in the image data.  The second explains 4.11% of the variability and so on.  Often the number of components is chosen based on the total amount of variability explained by the components.  You can see that by using 15 components, we capture almost 35% of the total variability in the images.\n",
    "\n",
    "Let's see if we can get a better idea of what aspects of the image the components might be picking up on.  To do this we will work with the components_ attribute of the pca object.  Looking at the shape of components shows us that each component gives the weights for each pixel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(15, 784)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pca.components_.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I created a function below to plot the weights of each component in the form of the images.  The darker portions of the image show"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 4804 characters]\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f0d1c7de390>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "def plot_component(pca, comp):\n",
    "    '''\n",
    "    Plots an image associated with each component to understand how the weighting \n",
    "    of the components \n",
    "    INPUT: \n",
    "          pca - pca object created from PCA in sklearn\n",
    "          comp - int - the component you want to see starting at 0\n",
    "    OUTPUT\n",
    "          None\n",
    "    '''\n",
    "    if comp <= len(pca.components_):\n",
    "        mat_data = np.asmatrix(pca.components_[comp]).reshape(28,28)  #reshape images\n",
    "        plt.imshow(mat_data); #plot the data\n",
    "        plt.xticks([]) #removes numbered labels on x-axis\n",
    "        plt.yticks([]) #removes numbered labels on y-axis   \n",
    "    else:\n",
    "        print('That is not the right input, please read the docstring before continuing.')\n",
    "        \n",
    "#Plot the first component\n",
    "plot_component(pca, 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at this image, it makes sense that only using two components was able to separate zero better than other digits when looking at this image.  The highest weights are associated with pixels that look a lot like a zero.\n",
    "\n",
    "Now, it is your turn!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
