{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hierarchical Clustering Lab\n",
    "In this notebook, we will be using sklearn to conduct hierarchical clustering on the [Iris dataset](https://archive.ics.uci.edu/ml/datasets/iris) which contains 4 dimensions/attributes and 150 samples. Each sample is labeled as one of the three type of Iris flowers.\n",
    "\n",
    "In this exercise, we'll ignore the labeling and cluster based on the attributes, then we'll compare the results of different hierarchical clustering techniques with the original labels to see which one does a better job in this scenario. We'll then proceed to visualize the resulting cluster hierarchies.\n",
    "\n",
    "## 1. Importing the Iris dataset\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import datasets\n",
    "\n",
    "iris = datasets.load_iris()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A look at the first 10 samples in the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 5.1,  3.5,  1.4,  0.2],\n",
       "       [ 4.9,  3. ,  1.4,  0.2],\n",
       "       [ 4.7,  3.2,  1.3,  0.2],\n",
       "       [ 4.6,  3.1,  1.5,  0.2],\n",
       "       [ 5. ,  3.6,  1.4,  0.2],\n",
       "       [ 5.4,  3.9,  1.7,  0.4],\n",
       "       [ 4.6,  3.4,  1.4,  0.3],\n",
       "       [ 5. ,  3.4,  1.5,  0.2],\n",
       "       [ 4.4,  2.9,  1.4,  0.2],\n",
       "       [ 4.9,  3.1,  1.5,  0.1]])"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "iris.data[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```iris.target``` contains the labels that indicate which type of Iris flower each sample is"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "       0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
       "       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
       "       1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,\n",
       "       2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,\n",
       "       2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "iris.target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Clustering\n",
    "Let's now use sklearn's [```AgglomerativeClustering```](http://scikit-learn.org/stable/modules/generated/sklearn.cluster.AgglomerativeClustering.html) to conduct the heirarchical clustering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.cluster import AgglomerativeClustering\n",
    "\n",
    "# Hierarchical clustering\n",
    "# Ward is the default linkage algorithm, so we'll start with that\n",
    "ward = AgglomerativeClustering(n_clusters=3)\n",
    "ward_pred = ward.fit_predict(iris.data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's also try complete and average linkages\n",
    "\n",
    "**Exercise**:\n",
    "* Conduct hierarchical clustering with complete linkage, store the predicted labels in the variable ```complete_pred```\n",
    "* Conduct hierarchical clustering with average linkage, store the predicted labels in the variable ```avg_pred```\n",
    "\n",
    "Note: look at the documentation of [```AgglomerativeClustering```](http://scikit-learn.org/stable/modules/generated/sklearn.cluster.AgglomerativeClustering.html) to find the appropriate value to pass as the ```linkage``` value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Hierarchical clustering using complete linkage\n",
    "# TODO: Create an instance of AgglomerativeClustering with the appropriate parameters\n",
    "complete = AgglomerativeClustering(n_clusters=3, linkage='complete')\n",
    "# Fit & predict\n",
    "# TODO: Make AgglomerativeClustering fit the dataset and predict the cluster labels\n",
    "complete_pred = complete.fit_predict(iris.data)\n",
    "\n",
    "# Hierarchical clustering using average linkage\n",
    "# TODO: Create an instance of AgglomerativeClustering with the appropriate parameters\n",
    "avg = AgglomerativeClustering(n_clusters=3, linkage='average')\n",
    "# Fit & predict\n",
    "# TODO: Make AgglomerativeClustering fit the dataset and predict the cluster labels\n",
    "avg_pred = avg.fit_predict(iris.data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To determine which clustering result better matches the original labels of the samples, we can use ```adjusted_rand_score``` which is an *external cluster validation index* which results in a score between -1 and 1, where 1 means two clusterings are identical of how they grouped the samples in a dataset (regardless of what label is assigned to each cluster).\n",
    "\n",
    "Cluster validation indices are discussed later in the course."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import adjusted_rand_score\n",
    "\n",
    "ward_ar_score = adjusted_rand_score(iris.target, ward_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise**:\n",
    "* Calculate the Adjusted Rand score of the clusters resulting from complete linkage and average linkage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: Calculated the adjusted Rand score for the complete linkage clustering labels\n",
    "complete_ar_score = adjusted_rand_score(iris.target, complete_pred)\n",
    "\n",
    "# TODO: Calculated the adjusted Rand score for the average linkage clustering labels\n",
    "avg_ar_score = adjusted_rand_score(iris.target, avg_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Which algorithm results in the higher Adjusted Rand Score?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scores: \n",
      "Ward: 0.731198556771 \n",
      "Complete:  0.642251251836 \n",
      "Average:  0.759198707107\n"
     ]
    }
   ],
   "source": [
    "print( \"Scores: \\nWard:\", ward_ar_score,\"\\nComplete: \", complete_ar_score, \"\\nAverage: \", avg_ar_score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. The Effect of Normalization on Clustering\n",
    "\n",
    "Can we improve on this clustering result?\n",
    "\n",
    "Let's take another look at the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 5.1,  3.5,  1.4,  0.2],\n",
       "       [ 4.9,  3. ,  1.4,  0.2],\n",
       "       [ 4.7,  3.2,  1.3,  0.2],\n",
       "       [ 4.6,  3.1,  1.5,  0.2],\n",
       "       [ 5. ,  3.6,  1.4,  0.2],\n",
       "       [ 5.4,  3.9,  1.7,  0.4],\n",
       "       [ 4.6,  3.4,  1.4,  0.3],\n",
       "       [ 5. ,  3.4,  1.5,  0.2],\n",
       "       [ 4.4,  2.9,  1.4,  0.2],\n",
       "       [ 4.9,  3.1,  1.5,  0.1],\n",
       "       [ 5.4,  3.7,  1.5,  0.2],\n",
       "       [ 4.8,  3.4,  1.6,  0.2],\n",
       "       [ 4.8,  3. ,  1.4,  0.1],\n",
       "       [ 4.3,  3. ,  1.1,  0.1],\n",
       "       [ 5.8,  4. ,  1.2,  0.2]])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "iris.data[:15]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at this, we can see that the forth column has smaller values than the rest of the columns, and so its variance counts for less in the clustering process (since clustering is based on distance). Let us [normalize](https://en.wikipedia.org/wiki/Feature_scaling) the dataset so that each dimension lies between 0 and 1, so they have equal weight in the clustering process.\n",
    "\n",
    "This is done by subtracting the minimum from each column then dividing the difference by the range.\n",
    "\n",
    "sklearn provides us with a useful utility called ```preprocessing.normalize()``` that can do that for us"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.80377277,  0.55160877,  0.22064351,  0.0315205 ],\n",
       "       [ 0.82813287,  0.50702013,  0.23660939,  0.03380134],\n",
       "       [ 0.80533308,  0.54831188,  0.2227517 ,  0.03426949],\n",
       "       [ 0.80003025,  0.53915082,  0.26087943,  0.03478392],\n",
       "       [ 0.790965  ,  0.5694948 ,  0.2214702 ,  0.0316386 ],\n",
       "       [ 0.78417499,  0.5663486 ,  0.2468699 ,  0.05808704],\n",
       "       [ 0.78010936,  0.57660257,  0.23742459,  0.0508767 ],\n",
       "       [ 0.80218492,  0.54548574,  0.24065548,  0.0320874 ],\n",
       "       [ 0.80642366,  0.5315065 ,  0.25658935,  0.03665562],\n",
       "       [ 0.81803119,  0.51752994,  0.25041771,  0.01669451]])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn import preprocessing\n",
    "\n",
    "normalized_X = preprocessing.normalize(iris.data)\n",
    "normalized_X[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now all the columns are in the range between 0 and 1. Would clustering the dataset after this transformation lead to a better clustering? (one that better matches the original labels of the samples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scores: \n",
      "Ward: 0.885697031028 \n",
      "Complete:  0.644447235392 \n",
      "Average:  0.558371443754\n"
     ]
    }
   ],
   "source": [
    "ward = AgglomerativeClustering(n_clusters=3)\n",
    "ward_pred = ward.fit_predict(normalized_X)\n",
    "\n",
    "complete = AgglomerativeClustering(n_clusters=3, linkage=\"complete\")\n",
    "complete_pred = complete.fit_predict(normalized_X)\n",
    "\n",
    "avg = AgglomerativeClustering(n_clusters=3, linkage=\"average\")\n",
    "avg_pred = avg.fit_predict(normalized_X)\n",
    "\n",
    "\n",
    "ward_ar_score = adjusted_rand_score(iris.target, ward_pred)\n",
    "complete_ar_score = adjusted_rand_score(iris.target, complete_pred)\n",
    "avg_ar_score = adjusted_rand_score(iris.target, avg_pred)\n",
    "\n",
    "print( \"Scores: \\nWard:\", ward_ar_score,\"\\nComplete: \", complete_ar_score, \"\\nAverage: \", avg_ar_score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Dendrogram visualization with scipy\n",
    "\n",
    "Let's visualize the highest scoring clustering result. \n",
    "\n",
    "To do that, we'll need to use Scipy's [```linkage```](https://docs.scipy.org/doc/scipy/reference/generated/scipy.cluster.hierarchy.linkage.html) function to perform the clusteirng again so we can obtain the linkage matrix it will later use to visualize the hierarchy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import scipy's linkage function to conduct the clustering\n",
    "from scipy.cluster.hierarchy import linkage\n",
    "\n",
    "# Specify the linkage type. Scipy accepts 'ward', 'complete', 'average', as well as other values\n",
    "# Pick the one that resulted in the highest Adjusted Rand Score\n",
    "linkage_type = 'ward'\n",
    "\n",
    "linkage_matrix = linkage(normalized_X, linkage_type)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot using scipy's [dendrogram](https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.cluster.hierarchy.dendrogram.html) function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 24628 characters]\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fe693486fd0>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from scipy.cluster.hierarchy import dendrogram\n",
    "import matplotlib.pyplot as plt\n",
    "plt.figure(figsize=(22,18))\n",
    "\n",
    "# plot using 'dendrogram()'\n",
    "dendrogram(linkage_matrix)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Visualization with Seaborn's ```clustermap``` \n",
    "\n",
    "The [seaborn](http://seaborn.pydata.org/index.html) plotting library for python can plot a [clustermap](http://seaborn.pydata.org/generated/seaborn.clustermap.html), which is a detailed dendrogram which also visualizes the dataset in more detail. It conducts the clustering as well, so we only need to pass it the dataset and the linkage type we want, and it will use scipy internally to conduct the clustering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 38584 characters]\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fe68cc13080>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import seaborn as sns\n",
    "\n",
    "sns.clustermap(normalized_X, figsize=(12,18), method=linkage_type, cmap='viridis')\n",
    "\n",
    "# Expand figsize to a value like (18, 50) if you want the sample labels to be readable\n",
    "# Draw back is that you'll need more scrolling to observe the dendrogram\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at the colors of the dimensions can you observe how they differ between the three type of flowers? You should at least be able to notice how one is vastly different from the two others (in the top third of the image)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
