{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# DBSCAN Lab\n",
    "In this notebook, we will use DBSCAN to cluster a couple of datasests. We will examine how changing its parameters (epsilon and min_samples) changes the resulting cluster structure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "dataset_1 = pd.read_csv('blobs.csv')[:80].values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This our first dataset. It looks like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 24052 characters]\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f51a7049c18>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import dbscan_lab_helper as helper\n",
    "    \n",
    "helper.plot_dataset(dataset_1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's cluster it using DBSCAN's default settings and see what happens. We are hoping for it to be able to assign each of the three \"blobs\" into its own cluster. Can it do that out of the box?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "#TODO: Import sklearn's cluster module\n",
    "from sklearn import cluster\n",
    "\n",
    "#TODO: create an instance of DBSCAN\n",
    "dbscan = cluster.DBSCAN()\n",
    "#TODO: use DBSCAN's fit_predict to return clustering labels for dataset_1\n",
    "clustering_labels_1 = dbscan.fit_predict(dataset_1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 37724 characters]\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f5194cc5be0>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot clustering\n",
    "helper.plot_clustered_dataset(dataset_1, clustering_labels_1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Does that look okay? Was it able to group the dataset into the three clusters we were hoping for?\n",
    "\n",
    "As you see, we will have to make some tweaks. Let's start by looking at Epsilon, the radius of each point's neighborhood. The default value in sklearn is 0.5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 57028 characters]\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f5194c8aa20>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot clustering with neighborhoods\n",
    "helper.plot_clustered_dataset(dataset_1, clustering_labels_1, neighborhood=True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the graph, we can see that an Epsilon value of 0.5 is too small for this dataset. We need to increase it so the points in a blob overlap each others' neighborhoods, but not to the degree where a single cluster would span two blobs.\n",
    "\n",
    "**Quiz**: Change the value of Epsilon so that each blob is its own cluster (without any noise points). The graph shows  the points in the datasets as well as the neighborhood of each point:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 121852 characters]\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f51940c2668>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# TODO: increase the value of epsilon to allow DBSCAN to find three clusters in the dataset\n",
    "epsilon=2\n",
    "\n",
    "# Cluster\n",
    "dbscan = cluster.DBSCAN(eps=epsilon)\n",
    "clustering_labels_2 = dbscan.fit_predict(dataset_1)\n",
    "\n",
    "# Plot\n",
    "helper.plot_clustered_dataset(dataset_1, clustering_labels_2, neighborhood=True, epsilon=epsilon)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Were you able to do it? As you change the values, you can see that the points cluster into larger clusters and the number of noise points keeps on decreasing. Then at Epsilon values above 1.6 we get the clustering we're after. But once we increase it to above 5, we start to see two blobs joining together into one cluster. So the right Epsilon would be in the range between those values in this scenario.\n",
    "\n",
    "## Dataset 2\n",
    "\n",
    "Let's now look at a dataset that's a little more tricky"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_2 = pd.read_csv('varied.csv')[:300].values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 48216 characters]\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f51941c5320>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot\n",
    "helper.plot_dataset(dataset_2, xlim=(-14, 5), ylim=(-12, 7))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What happens if we run DBSCAN with the default parameter values?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Cluster with DBSCAN\n",
    "# TODO: Create a new isntance of DBSCAN\n",
    "dbscan = cluster.DBSCAN()\n",
    "# TODO: use DBSCAN's fit_predict to return clustering labels for dataset_2\n",
    "clustering_labels_3 = dbscan.fit_predict(dataset_2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 184836 characters]\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f518eedecf8>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot\n",
    "helper.plot_clustered_dataset(dataset_2, \n",
    "                              clustering_labels_3, \n",
    "                              xlim=(-14, 5), \n",
    "                              ylim=(-12, 7), \n",
    "                              neighborhood=True, \n",
    "                              epsilon=0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This clustering could make sense in some scenarios, but it seems rather arbitrary. Looking at the dataset, we can imagine at least two scenarios for what we'd want to do:\n",
    " * **Scenario** 1: Break the dataset up into three clusters: the blob on the left, the blob on the right, and the central area (even though it's less dense than the blobs on either side).\n",
    " * **Scenario 2**: Break the dataset up into two clusters: the blob on the left, and the blob on the right. Marking all the points in the center as noise. \n",
    " \n",
    "What values for the DBSCAN parameters would allow us to satisfy each of those senarios? Try a number of parameters to see if you can find a clustering that makes more sense.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 358348 characters]\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f518e9c0748>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# TODO: Experiment with different values for eps and min_samples to find a suitable clustering for the dataset\n",
    "eps=2\n",
    "min_samples=50\n",
    "\n",
    "# Cluster with DBSCAN\n",
    "dbscan = cluster.DBSCAN(eps=eps, min_samples=min_samples)\n",
    "clustering_labels_4 = dbscan.fit_predict(dataset_2)\n",
    "\n",
    "# Plot\n",
    "helper.plot_clustered_dataset(dataset_2, \n",
    "                              clustering_labels_4, \n",
    "                              xlim=(-14, 5), \n",
    "                              ylim=(-12, 7), \n",
    "                              neighborhood=True, \n",
    "                              epsilon=eps)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "The following grid plots the DBSCAN clustering results of a range of parameter values. Epsilon varies horizontally, while vertically each row shows a different value of min_samples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 1424116 characters]\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f5194b19748>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "eps_values = [0.3, 0.5, 1, 1.3, 1.5]\n",
    "min_samples_values = [2, 5, 10, 20, 80]\n",
    "\n",
    "helper.plot_dbscan_grid(dataset_2, eps_values, min_samples_values)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Heuristics for experimenting with DBSCAN's parameters\n",
    "Looking at this grid, we can guess at some general heuristics for tweaking the parameters of DBSCAN:\n",
    "\n",
    "| |Epsilon too low|Epsilon too high|\n",
    "|---|---|---|---|\n",
    "|**min_samples too low** |<img src='images/low_epsilon_and_low_min_sample.png'><br>Many small clusters. More than anticipated for the dataset. <br>**Action**: increase min_samples and epsilon| <img src='images/high_epsilon_and_low_min_sample.png'><br>Most points belong to one cluster<br>**Action**: decrease epsilon and increase min_samples|\n",
    "|**min_samples too high**|<img src='images/low_epsilon_and_high_min_sample.png'><br>Most/all data points are labeled as noise<br>**Action**: increase epsilon and decrease min_sample| <img src='images/high_epsilon_and_high_min_sample.png'><br> Except for extremely dense regions, most/all data points are <br>labeled as noise. (Or all points are labeled as noise). <br>**Action**: decrease min_samples and epsilon.|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Quiz\n",
    "* Which values do you believe best satisfy scenario 1? \n",
    "* Which values do you believe best satisfy scenario 2?\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
