{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "### Diabetes Case Study\n", "\n", "You now have had the opportunity to work with a range of supervised machine learning techniques for both classification and regression. Before you apply these in the project, let's do one more example to see how the machine learning process works from beginning to end with another popular dataset.\n", "\n", "We will start out by reading in the dataset and our necessary libraries. You will then gain an understanding of how to optimize a number of models using grid searching as you work through the notebook. " ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
PregnanciesGlucoseBloodPressureSkinThicknessInsulinBMIDiabetesPedigreeFunctionAgeOutcome
061487235033.60.627501
11856629026.60.351310
28183640023.30.672321
318966239428.10.167210
40137403516843.12.288331
\n", "
" ], "text/plain": [ " Pregnancies Glucose BloodPressure SkinThickness Insulin BMI \\\n", "0 6 148 72 35 0 33.6 \n", "1 1 85 66 29 0 26.6 \n", "2 8 183 64 0 0 23.3 \n", "3 1 89 66 23 94 28.1 \n", "4 0 137 40 35 168 43.1 \n", "\n", " DiabetesPedigreeFunction Age Outcome \n", "0 0.627 50 1 \n", "1 0.351 31 0 \n", "2 0.672 32 1 \n", "3 0.167 21 0 \n", "4 2.288 33 1 " ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Import our libraries\n", "import pandas as pd\n", "import numpy as np\n", "from sklearn.datasets import load_diabetes\n", "from sklearn.model_selection import train_test_split, RandomizedSearchCV\n", "from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score\n", "from sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier\n", "from sklearn.model_selection import GridSearchCV\n", "\n", "import matplotlib.pyplot as plt\n", "from sklearn.svm import SVC\n", "import seaborn as sns\n", "sns.set(style=\"ticks\")\n", "\n", "import check_file as ch\n", "\n", "%matplotlib inline\n", "\n", "# Read in our dataset\n", "diabetes = pd.read_csv('diabetes.csv')\n", "\n", "# Take a look at the first few rows of the dataset\n", "diabetes.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Because this course has been aimed at understanding machine learning techniques, we have largely ignored items related to parts of the data analysis process that come before building machine learning models - exploratory data analysis, feature engineering, data cleaning, and data wrangling. \n", "\n", "> **Step 1:** Let's do a few steps here. Take a look at some of usual summary statistics calculated to accurately match the values to the appropriate key in the dictionary below. " ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "# Cells for work\n" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
0
Outcome
00.65
10.35
\n", "
" ], "text/plain": [ " 0\n", "Outcome \n", "0 0.65\n", "1 0.35" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "outcome_count=diabetes.groupby(['Outcome']).size()\n", "pd.DataFrame((outcome_count/outcome_count.sum()).round(2))" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
PregnanciesGlucoseBloodPressureSkinThicknessInsulinBMIDiabetesPedigreeFunctionAgeOutcome
count768.000000768.000000768.000000768.000000768.000000768.000000768.000000768.000000768.000000
mean3.845052120.89453169.10546920.53645879.79947931.9925780.47187633.2408850.348958
std3.36957831.97261819.35580715.952218115.2440027.8841600.33132911.7602320.476951
min0.0000000.0000000.0000000.0000000.0000000.0000000.07800021.0000000.000000
25%1.00000099.00000062.0000000.0000000.00000027.3000000.24375024.0000000.000000
50%3.000000117.00000072.00000023.00000030.50000032.0000000.37250029.0000000.000000
75%6.000000140.25000080.00000032.000000127.25000036.6000000.62625041.0000001.000000
max17.000000199.000000122.00000099.000000846.00000067.1000002.42000081.0000001.000000
\n", "
" ], "text/plain": [ " Pregnancies Glucose BloodPressure SkinThickness Insulin \\\n", "count 768.000000 768.000000 768.000000 768.000000 768.000000 \n", "mean 3.845052 120.894531 69.105469 20.536458 79.799479 \n", "std 3.369578 31.972618 19.355807 15.952218 115.244002 \n", "min 0.000000 0.000000 0.000000 0.000000 0.000000 \n", "25% 1.000000 99.000000 62.000000 0.000000 0.000000 \n", "50% 3.000000 117.000000 72.000000 23.000000 30.500000 \n", "75% 6.000000 140.250000 80.000000 32.000000 127.250000 \n", "max 17.000000 199.000000 122.000000 99.000000 846.000000 \n", "\n", " BMI DiabetesPedigreeFunction Age Outcome \n", "count 768.000000 768.000000 768.000000 768.000000 \n", "mean 31.992578 0.471876 33.240885 0.348958 \n", "std 7.884160 0.331329 11.760232 0.476951 \n", "min 0.000000 0.078000 21.000000 0.000000 \n", "25% 27.300000 0.243750 24.000000 0.000000 \n", "50% 32.000000 0.372500 29.000000 0.000000 \n", "75% 36.600000 0.626250 41.000000 1.000000 \n", "max 67.100000 2.420000 81.000000 1.000000 " ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "diabetes.describe()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Awesome! These all look great!\n" ] } ], "source": [ "# Possible keys for the dictionary\n", "a = '0.65'\n", "b = '0'\n", "c = 'Age'\n", "d = '0.35'\n", "e = 'Glucose'\n", "f = '0.5'\n", "g = \"More than zero\"\n", "\n", "# Fill in the dictionary with the correct values here\n", "answers_one = {\n", " 'The proportion of diabetes outcomes in the dataset': d,\n", " 'The number of missing data points in the dataset': b,\n", " 'A dataset with a symmetric distribution': e,\n", " 'A dataset with a right-skewed distribution': c, \n", " 'This variable has the strongest correlation with the outcome': e\n", "}\n", "\n", "# Just to check your answer, don't change this\n", "ch.check_one(answers_one)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> **Step 2**: Since our dataset here is quite clean, we will jump straight into the machine learning. Our goal here is to be able to predict cases of diabetes. First, you need to identify the y vector and X matrix. Then, the following code will divide your dataset into training and test data. " ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "y = diabetes['Outcome']\n", "X = diabetes.drop(['Outcome'], axis=1)\n", "\n", "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now that you have a training and testing dataset, we need to create some models that and ultimately find the best of them. However, unlike in earlier lessons, where we used the defaults, we can now tune these models to be the very best models they can be.\n", "\n", "It can often be difficult (and extremely time consuming) to test all the possible hyperparameter combinations to find the best models. Therefore, it is often useful to set up a randomized search. \n", "\n", "In practice, randomized searches across hyperparameters have shown to be more time confusing, while still optimizing quite well. One article related to this topic is available [here](https://blog.h2o.ai/2016/06/hyperparameter-optimization-in-h2o-grid-search-random-search-and-the-future/). The documentation for using randomized search in sklearn can be found [here](http://scikit-learn.org/stable/auto_examples/model_selection/plot_randomized_search.html#sphx-glr-auto-examples-model-selection-plot-randomized-search-py) and [here](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html).\n", "\n", "In order to use the randomized search effectively, you will want to have a pretty reasonable understanding of the distributions that best give a sense of your hyperparameters. Understanding what values are possible for your hyperparameters will allow you to write a grid search that performs well (and doesn't break).\n", "\n", "> **Step 3**: In this step, I will show you how to use randomized search, and then you can set up grid searches for the other models in Step 4. However, you will be helping, as I don't remember exactly what each of the hyperparameters in SVMs do. Match each hyperparameter to its corresponding tuning functionality.\n", "\n" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Accuracy score for random forest : 0.7532467532467533\n", "Precision score random forest : 0.6545454545454545\n", "Recall score random forest : 0.6545454545454545\n", "F1 score random forest : 0.6545454545454545\n", "\n", "\n", "\n" ] } ], "source": [ "# build a classifier\n", "clf_rf = RandomForestClassifier()\n", "\n", "# Set up the hyperparameter search\n", "param_dist = {\"max_depth\": [3, None],\n", " \"n_estimators\": list(range(10, 200)),\n", " \"max_features\": list(range(1, X_test.shape[1]+1)),\n", " \"min_samples_split\": list(range(2, 11)),\n", " \"min_samples_leaf\": list(range(1, 11)),\n", " \"bootstrap\": [True, False],\n", " \"criterion\": [\"gini\", \"entropy\"]}\n", "\n", "\n", "# Run a randomized search over the hyperparameters\n", "random_search = RandomizedSearchCV(clf_rf, param_distributions=param_dist)\n", "\n", "# Fit the model on the training data\n", "random_search.fit(X_train, y_train)\n", "\n", "# Make predictions on the test data\n", "rf_preds = random_search.best_estimator_.predict(X_test)\n", "\n", "ch.print_metrics(y_test, rf_preds, 'random forest')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> **Step 4**: Now that you have seen how to run a randomized grid search using random forest, try this out for the AdaBoost and SVC classifiers. You might also decide to try out other classifiers that you saw earlier in the lesson to see what works best." ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Accuracy score for adaboost : 0.7792207792207793\n", "Precision score adaboost : 0.7441860465116279\n", "Recall score adaboost : 0.5818181818181818\n", "F1 score adaboost : 0.6530612244897959\n", "\n", "\n", "\n", "AdaBoostClassifier(algorithm='SAMME.R', base_estimator=None,\n", " learning_rate=0.4, n_estimators=10, random_state=None)\n" ] } ], "source": [ "# build a classifier for ada boost\n", "clf_ada = AdaBoostClassifier()\n", "\n", "# Set up the hyperparameter search\n", "# look at setting up your search for n_estimators, learning_rate\n", "# http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.AdaBoostClassifier.html\n", "param_dist = {\"n_estimators\": [10, 100, 200, 400],\n", " \"learning_rate\": [0.001, 0.005, .01, 0.05, 0.1, 0.2, 0.3, 0.4, 0.5, 1, 2, 10, 20]}\n", "\n", "# Run a randomized search over the hyperparameters\n", "ada_search = RandomizedSearchCV(clf_ada, param_distributions=param_dist)\n", "\n", "# Fit the model on the training data\n", "ada_search.fit(X_train, y_train)\n", "\n", "# Make predictions on the test data\n", "ada_preds = ada_search.best_estimator_.predict(X_test)\n", "\n", "# Return your metrics on test data\n", "ch.print_metrics(y_test, ada_preds, 'adaboost')\n", "\n", "# Print the hyperparams used\n", "print(ada_search.best_estimator_)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "\n", "\n" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Accuracy score for adaboost : 0.7597402597402597\n", "Precision score adaboost : 0.6551724137931034\n", "Recall score adaboost : 0.6909090909090909\n", "F1 score adaboost : 0.6725663716814159\n", "\n", "\n", "\n", "AdaBoostClassifier(algorithm='SAMME.R', base_estimator=None, learning_rate=1,\n", " n_estimators=10, random_state=None)\n" ] } ], "source": [ "ada_grid_search = GridSearchCV(clf_ada, param_dist)\n", "\n", "ada_grid_search.fit(X_train, y_train)\n", "\n", "ada_grid_preds = ada_grid_search.best_estimator_.predict(X_test)\n", "\n", "ch.print_metrics(y_test, ada_grid_preds, 'adaboost')\n", "\n", "print(ada_grid_search.best_estimator_)\n" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Accuracy score for svc : 0.7532467532467533\n", "Precision score svc : 0.6545454545454545\n", "Recall score svc : 0.6545454545454545\n", "F1 score svc : 0.6545454545454545\n", "\n", "\n", "\n", "SVC(C=0.5, cache_size=200, class_weight=None, coef0=0.0,\n", " decision_function_shape='ovr', degree=3, gamma='auto', kernel='linear',\n", " max_iter=-1, probability=False, random_state=None, shrinking=True,\n", " tol=0.001, verbose=False)\n" ] } ], "source": [ "# build a classifier for support vector machines\n", "clf_svc = SVC()\n", "\n", "# Set up the hyperparameter search\n", "# look at setting up your search for C (recommend 0-10 range), \n", "# kernel, and degree\n", "# http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html\n", "param_dist = {\"C\": [0.1, 0.5, 1, 3, 5],\n", " \"kernel\": ['linear','rbf']\n", " }\n", "\n", "\n", "# Run a randomized search over the hyperparameters\n", "svc_search = RandomizedSearchCV(clf_svc, param_distributions=param_dist)\n", "\n", "# Fit the model on the training data\n", "svc_search.fit(X_train, y_train)\n", "\n", "# Make predictions on the test data\n", "svc_preds = svc_search.best_estimator_.predict(X_test)\n", "\n", "ch.print_metrics(y_test, svc_preds, 'svc')\n", "print(svc_search.best_estimator_)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> **Step 5**: Use the test below to see if your best model matched, what we found after running the grid search. " ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Nice! It looks like your best model matches the best model I found as well! It makes sense to use f1 score to determine best in this case given the imbalance of classes. There might be justification for precision or recall being the best metric to use as well - precision showed to be best with adaboost again. With recall, SVMs proved to be the best for our models.\n" ] } ], "source": [ "a = 'randomforest'\n", "b = 'adaboost'\n", "c = 'supportvector'\n", "\n", "best_model = b\n", "\n", "# See if your best model was also mine. \n", "# Notice these might not match depending your search!\n", "ch.check_best(best_model)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Once you have found your best model, it is also important to understand why it is performing well. In regression models where you can see the weights, it can be much easier to interpret results. \n", "\n", "> **Step 6**: Despite the fact that your models here are more difficult to interpret, there are some ways to get an idea of which features are important. Using the \"best model\" from the previous question, find the features that were most important in helping determine if an individual would have diabetes or not. Do your conclusions match what you might have expected during the exploratory phase of this notebook?" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[0 3 4 2 6 7 5 1]\n", "[ 0.04991133 0.32363753 0.08106476 0.05030821 0.05292798 0.17536724\n", " 0.11839917 0.14838379]\n" ] } ], "source": [ "print(np.argsort(random_search.best_estimator_.feature_importances_))\n", "print(random_search.best_estimator_.feature_importances_)" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfMAAAEcCAYAAADTHV0cAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xu8ZmP9//HXNuTQOHxlEoZmnN5iMJiUCBmRElMRGskh/Yik6Cym0kH5IhXpIBRFdPBFDs3XKYcYGsPg3TczU02Og9QwRnvs3x/r2nXb9uHee88+rJn38/HYj33fa13HtWb2576udd1rtbS1tRERERH1tcxQNyAiIiL6J8E8IiKi5hLMIyIiai7BPCIiouYSzCMiImouwTwiIqLmEswjIiJqbtmhbkBEDC1Jc4A1gUUNmze2/XA/ytwZ+Int0f1qXO/qPA+Ya/uEwaqzm7ZMATa0feBQtyWWDgnmEQHwTtu/HepGtJO0rO3WoW5HX0jK39UYdPlHFxFdkvRG4DRgU+DPwEdt31D2HQJ8EhgNPAGcYvscSa8EfgMsL2l+KWpj4Cs0jJw7jt7LDMHZwOTqrV4JvBr4FrAjMB843faZTbR7DDAbOBT4IjAS+AxwF/BDYL1S99El/cHA4cDdwEHAI8BRtqeW/WsD3wV2AJ4qff1+2TcFGAc8D+wFfLb8tEiaBDxke8uujlfjsQBOBz5FNUvyWds/KvtXBE4G9gFWA+4F3mp7QQ/n6GDgRGAUMA84wfaFPR2/qJ9cM4+ITklaB7iSKoisDhwPXCZpVEnyOLAnsApwCHC6pK1tPwvsATxse2T5aXbK/gDgHVQB60Xgf4B7gHWAicCxknbvRTfeAGwE7AecAXwO2BXYDHivpJ06pJ0FrAGcBPxC0upl30+BucDaVAH1K5ImNuTdG7i0tPuHVB9cLi5937Kk6fR4NZTxGmDV0tfDgO9I+q+y71RgG+BNVOfik8CL3Z2j8mHoTGAP2yuXvNN7ceyiRjIyjwiAX0lqn9a+wfYk4EDgKttXle3XSZoGvB043/aVDflvlHQt8Gaq0W1fnWn7rwCS3gCMsv3Fsm+WpO8D+wPXNFnel2w/D1wr6Vngp7YfL+XfDGwF3FjSPg6cYbsNuFjSccA7JN1ANSLfs5Q1XdIPgPcDU0ve22z/qrxeIOllDWnieP0L+GK5vHBVmdWQpDuoZhjeaPtvJe2tpQ/dnaNLqT4QjZP0F9uPUM04xBIowTwiACZ1cs38tcC+kt7ZsG054HoASXtQjWA3pprlW4lq+rc//tqh/rUl/b1h2wjg5l6U91jD6wWdvB/Z8P5vJZC3+zPVSHxt4Cnb/+ywb0IX7e5UE8fryQ7rBJ4r7VsDWAF4qJNiuzxHtp+VtB/VaP2Hkm4BjrP9YE9tjfpJMI+IrvwV+LHtwzvukLQ8cBnV9eVf2/6XpF8BLSVJZ49jfJYqgLV7TSdpGvP9FZhte6O+NL4P1pHU0hDQ1wMuBx4GVpe0ckNAXw/4W0Pejv19yfsmjld35lFdj9+A6pJDoy7PEYDta4BrGq65f59qNiCWMAnmEdGVnwB3lmvUv6Ua8b0R+BPwDLA81UKu1jLq3A24r+R9DHiVpFVtP1O2TQeOk3Qy8Arg2B7qvwP4h6RPUV37fQF4HbCi7TsXUx8bvRo4RtJZwKRS11W2n5R0K/BVScdTjawPo7oM0ZXHgLdKWsb2i1T97e54dcn2i5LOBU6T9P5S9rZU0/PdnaN/Ua0DmEo1CzGfl379MJYgWQAXEZ0q1673plqZ/QTVKPATwDJlhHoMcAnwNPA+qlFse94HqRaNzZL097Ia/MdUI8s5wLXAxT3Uvwh4JzCeamX6POAHVIvEBsLvqRbLzQO+DOxj+8my7wBgDNUo/ZfASbav66asn5ffT0q6u6fj1YTjqabk76Sspqc6D12eo/JzXGnzU8BOwId7UWfUSEtbW2ezYRERS4/yFa4P2t5hqNsS0RcZmUdERNRcgnlERETNZZo9IiKi5rKaPQZM+TrO66luVJFVtBERzRkBrAXcaXthMxkSzGMgvZ7e3eAjIiL+483A75pJmGAeA+kRgAsvvJDXvKaz+4NERERHjz76KJMnT4Ze3H43wTwG0iKA17zmNYwePWiPtY6IWFI0fXkyq9kjIiJqLsE8IiKi5hLMIyIiai7BPCIiouYSzCMiImouwTwiIqLmEswjIiJqLt8zjwE3diy0tg51KyIiBs9gP/YkI/OIiIiaSzCPiIiouQTziIiImkswj4iIqLkE84iIiJpLMI+IiKi5fDVtmJG0JnA68EbgaeAF4Ovl9fG29xzC5kVExDCUkfkwIqkF+BVwk+31bW8D7A/kYeAREdGljMyHl12AF2x/t32D7T8D35K0c/s2SVOA+bZPLe/vA/a0PUfSQcDxQBsww/b7Jb0WOBcYBTwBHGL7L5L2BU4CFgHP2N5R0gjga8DOwPLAd2yfM8D9joiIfsjIfHjZDLi7r5klbQZ8DtjF9pbAR8uubwMX2N4CuBA4s2w/Edi9pN2rbDuMKrC/Hng9cLiksU3UPUVSW+MPMLuvfYmIiOZlZD6MSfoOsAPVdfNPNJFlF+BS2/MAbD9Vtm8HvLu8/jHVNXiAW4DzJF0C/KJs2w3YQtI+5f2qwEb0EJhtTwGmdGj/mJ7yRURE/yWYDy8zgfe0v7F9lKQ1gGkd0rXy0lmVFcrvFqrp9Z60lfKPkPQG4B3AdEnjSxkfsX1N37oQERGDLdPsw8v/AitIOrJh20qdpJsDbA0gaWugfRp8KvBeSa8q+1Yv22+lWkgHMBn4Xdm/ge3f2z4RmAesC1wDHClpuZJmY0mvXDzdi4iIgZCR+TBiu03SJOB0SZ+kWqz2LPCpDkkvAw6SNB24E/hjyT9T0peBGyUtAv4AHAwcA5wr6ROlzENKOd+QtBHVaHwqcA8wAxgD3F1W1z8BTBqYHkdExOLQ0jbYz2mLpUb7NfNZs6bS2ppv10XE0qM/oXXu3LlMnDgRYKztOc3kyTR7REREzSWYR0RE1FyCeURERM0lmEdERNRcgnlERETN5atpMeBmz4bRWcweETFgMjKPiIiouQTziIiImkswj4iIqLkE84iIiJrLArgYcGPHQmvrULciojm5w3XUUUbmERERNZdgHhERUXMJ5hERETWXYB4REVFzCeYRERE1l2AeERFRc/lq2lJA0iLgXqAFWAQcbftWSWOA2cDJtj9f0q4BPAKcY/toSVOA+bZPHZLGR0REjzIyXzossD3e9pbAZ4CvNuybBezZ8H5fYOZgNi4iIvonwXzpswrwdMP7BcADkiaU9/sBlwx6qyIios8yzb50WFHSdGAFYC1glw77fwbsL+lRqmn4h4G1e1NBmY4/qf9NjYiI3kowXzossD0eQNJ2wAWSxjXsvxr4EvAYcHFfKrA9BZjSuK3hmnxERAygTLMvZWzfBqwBjGrY9gJwF3AccNkQNS0iIvoowXwpI2kTYATwZIdd/w18ynbH7RERMcxlmn3p0H7NHKqvp33A9iJJ/05geyZZxR4RUUstbXneXwyQ9mvms2ZNpbV19FA3J6Ip+ZMYQ23u3LlMnDgRYKztOc3kyTR7REREzSWYR0RE1FyCeURERM0lmEdERNRcVrPHgJs9G0Zn/VtExIDJyDwiIqLmEswjIiJqLsE8IiKi5hLMIyIiai4L4GLAjR0Lra1D3Yqos9yVLaJ7GZlHRETUXIJ5REREzSWYR0RE1FyCeURERM0lmEdERNRcgnlERETNJZgvJSS9S1KbpE2Gui0REbF4JZgvPQ4AfgfsP9QNiYiIxSs3jVkKSBoJbA+8BbgcmCJpGeDbwE7AbKoPdufavlTSNsBpwEhgHnCw7UeGpPEREdGjBPOlwyTgatt/lPSUpK2B9YExwObAq4EHgHMlLQd8C9jb9hOS9gO+DBzaXQWSpgAnDVwXIiKiKwnmS4cDgDPK65+V98sBP7f9IvCopOvLfgHjgOskAYwAehyV254CTGncJmkM1ag/IiIGUIL5Ek7Sq4BdgHGS2qiCcxvwyy6ytAAzbW83SE2MiIh+ygK4Jd8+wAW2X2t7jO11qUbL84D3SFpG0prAziW9gVGStgOQtJykzYai4RER0ZwE8yXfAbx8FH4ZsDYwF7gPOAf4PfCM7ReoPgCcIukeYDrwpsFrbkRE9Fam2ZdwtnfuZNuZUK1ytz2/TMXfAdxb9k8HdhzMdkZERN8lmC/drpC0GvAK4Eu2Hx3qBkVERO8lmC/FOhu1R0RE/eSaeURERM0lmEdERNRcptljwM2eDaNHD3UrIiKWXBmZR0RE1FyCeURERM0lmEdERNRcgnlERETNZQFcDLixY6G1dahbEYtDW9tQtyAiOpOReURERM0lmEdERNRcgnlERETNJZhHRETUXIJ5REREzSWYR0RE1FyPX02TtAi4F1gOaAXOB86w/aKkCcBBto/pJv/BwATbRzfbKEmftf2VZtN3yHsesBPwDPAicJTt23qRf77tkZLWBs60vU9f2tGL+qYAhwNPlE1X2/70Yix/EvBH2/eX918EbrL928VVR0REDK1mvme+wPZ4AEmvBi4CVgVOsj0NmDYA7fos0KdgXnzC9qWSdgPOAbbobQG2HwZ6FcgljbC9qLd1AafbPrUP+ZoxCbgCuB/A9okDVE9ERAyRXt00xvbjkj4E3FlGlDsBx9veU9K2wBnAisAC4BDbLlnXlXQ1MBa4yPYXACQdCBwDvAL4PfBh4MvAipKmAzNtT+4iHcAPgQlAG3Cu7dM7NPkmYMNS1wbAd4BRwHPA4bYflDSW6gPKssDV7RkljQGusD1O0krAecAmwAPAGKoR/zRJ84HTgN2B4yQtKO9HAvOAg20/0lX9XR1rSXOoZjTmlRmQU23vXI77esD65fcZts8seQ4Cji/HYwZwNrAXsJOkE4D3AJ8v/bpU0kTg1NL3O4EjbS8sdZ8PvJNqRmbf7toaERFDq9fXzG3PKvle3WHXg8COtrcCTuSlI+ttgcnAeGBfSRMkvQ7YD9i+jPwXAZPLFPMC2+NLIO80XSlrHdvjbG8O/KiT5r6T6hIBwPeAj9jehirgnVW2fxM42/brgUe76PaHgadtbwF8CdimYd8rgftsv4Hqg8a3gH1KPedSfTjprn6Aj0maXn5276INjTah+vCwLXCSpOUkbQZ8DtjF9pbAR23fClxONVMx3vZD7QVIWoHqA8p+5fgtCxzZUMc821tTfSA4vqcGSZoiqa3xB5jdRF8iIqKf+no715ZOtq0KnC9pI6qR4XIN+66z/SSApF8AO1Bdf9+GapQP1Yj+8U7KndhFuv8B1pf0LeBK4NqGPN8oI9EngMMkjQTeBPy8lAGwfPm9PdWIFeDHwCmdtGEHqqCP7fskzWjYtwi4rLwWMA64rtQzAnikh/qh99PsV9peCCyU9DiwJrALcKnteaWdT/VQhoDZtv9Y3p8PHEU1uwLwi/L7LuDdPTXI9hRgyksqqGY3EtAjIgZYr4O5pPWpAtjjwOsadn0JuN72u8of8Rsa9nW8o3Mb1QeC821/pocqu0wnaUuqEepRwHuBQ8uuT9i+tCHdKsDf26/9d6KnO0539uGl3fMN18lbqC4NbNehnT3V35lW/jNzskKHfQsbXi+iOo8t9NyPRt31qbGO9vIjImKY6tU0u6RRwHeBb9vuGDhWBf5WXh/cYd9bJa0uaUWqBVm3AFOBfcqiOsr+15b0/5LUPrLvNJ2kNYBlbF9GdR14667abfsfwGxJ+5YyWsoHAUpb9i+vJ3dRxO+oPiwgaVNg866qAkZJ2q6kXU7SZj3U35U5/Gc6/z3dpGs3FXivpFeVOlYv2/8JrNxJ+geBMZI2LO/fD9zYRD0RETHMNBPMVyzXcmcCv6Wazv5CJ+m+DnxV0i1U08uNfkc1hT0duMz2tPJVqROAa8u09XXAWiX994AZki7sJt06wA1lodx5QE8j/MlUU+73ADOBvcv2jwJHSbqT6gNJZ86iCtIzgE9RLS57pmMi2y9QrYA/pdQznWp6vbv6u/IF4JuSbqYaHXfL9kyq6/M3ljpOK7t+BnxC0h/KIrz29M8Dh1BN/d9L9TW+7/ZUT0REDD8tbXmmYY8kjQCWs/18CYhTgY1L8I4utF8znzVrKq2to4e6ObEY5M9FxMCbO3cuEydOBBhre04zeXIttDkrAdeXqf8Wqq9wJZBHRMSwkGDeBNv/pPo+e0RExLCTe7NHRETUXIJ5REREzSWYR0RE1FyumceAmz0bRmcxe0TEgMnIPCIiouYSzCMiImouwTwiIqLmEswjIiJqLgvgYsCNHQutrUPdinrK7VMjohkZmUdERNRcgnlERETNJZhHRETUXIJ5REREzSWYR0RE1FyCeURERM3lq2kdSFoE3Au0AIuAo23fKmkMcIXtcYuhjhuA421PkzQH+CfwIvAYcJDtR/tbR0RELD0yMn+5BbbH294S+Azw1UGo8y2lvmnAZzvulDRiENow6HVFRMTikZF591YBnu64UdIKwNnABKAV+Ljt67vZviLwI2BT4AFgxS7quwk4ptQxHzgN2B04TtKC8n4kMA842PYjko4Bjij13W97f0k7Ad8sZbYBOwLbUM0G7FnK/zYwzfZ5ZXbgXGA34NuS7gS+A4wCngMOt/1gbw9eREQMjgTzl1tR0nRgBWAtYJdO0hwFYHtzSZsA10rauJvtRwLP2d5C0hbA3V3UvSfVFD/AK4H7bJ8oaTngRmBv209I2g/4MnAo8GlgrO2FklYreY8HjrJ9i6SRwPNN9Pt52zsASJoKHGH7/yS9ATiri+Pwb5KmACc1UU9ERCxmmWZ/ufZp9k2AtwEXSGrpkGYH4McAZcT6Z2DjbrbvCPykbJ8BzOhQ3vXlA8Qq/GdafxFwWXktYBxwXUl3AtD+hPAZwIWSDqQanQPcApxWRu2r2W7mZqoXA5Tg/ybg56Wuc6g+1HTL9hTbLY0/wNgm6o2IiH7KyLwbtm+TtAbVdHOjjsG9p+1QTXd35S2253XY9rztRQ3lzrS9XSd530H1YWEv4POSNrP9NUlXAm8Hbpe0K1Wgb/zwtkKHcp4tv5cB/m57fDftjYiIYSQj826UqfIRwJMddt0ETC5pNgbWA9zk9nHAFr1sioFRkrYrZSwnaTNJywDr2r4e+CSwGjBS0ga277V9CtWiuk2oZgk2lbS8pFWBiZ1WZP8DmC1p31JXi6Qte9neiIgYRAnmL7eipOllivli4AMNI+R2ZwEjJN1b0hxse2E328+mCrIzqILuHb1pkO0XgH2AUyTdA0ynmgofAfyk1PcH4HTbfweOlXRfSbsA+I3tvwKXUKblS/quTAYOK/lnAnv3pr0RETG4WtryjMUYIOW7+bNnzZpKa+vonpJHJ/LfM2LpM3fuXCZOnAjV4uY5zeTJyDwiIqLmEswjIiJqLsE8IiKi5hLMIyIiai7fM48BN3s2jM76t4iIAZOReURERM0lmEdERNRcgnlERETNJZhHRETUXBbAxYAbOxZam3luWw3kjmwRMRxlZB4REVFzCeYRERE1l2AeERFRcwnmERERNZdgHhERUXMJ5hERETWXYF4jkuYv5vLGSLqvvJ4g6czFWX5ERAyOfM88ALA9DZg21O2IiIjeSzCvIUk7A1OAecA44C7gQNttkr4G7AW0AtfaPl7SecAVti8t+efbHtlJmcfb3lPSFGA9YP3y+wzbGbVHRAxTCeb1tRWwGfAwcAuwvaT7gXcBm5TAvlo/yt8EeAuwMmBJZ9v+V1eJyweAk/pRX0RE9FGCeX3dYXsugKTpwBjgduB54AeSrgSu6Ef5V9peCCyU9DiwJjC3q8S2p1DNFvybpDHA7H60ISIimpAFcPW1sOH1ImBZ263AtsBlwCTg6rK/lXKuJbUAr+hL+f1tcEREDIwE8yWIpJHAqravAo4Fxpddc4Btyuu9geUGv3URETFQMtpasqwM/FrSCkAL8LGy/ftl+x3AVODZIWpfREQMgJa2PNMxBkj7NfNZs6bS2jp6qJuzWOS/S0QMtLlz5zJx4kSAsbbnNJMn0+wRERE1l2AeERFRcwnmERERNZdgHhERUXNZzR4DbvZsGL1krH+LiBiWMjKPiIiouQTziIiImkswj4iIqLkE84iIiJrLArgYcGPHQmvrULei73LXt4gY7jIyj4iIqLkE84iIiJpLMI+IiKi5BPOIiIiaSzCPiIiouQTziIiImstX07og6XPA+4BFwIvA/wMuBibYntch7a2239RNWb8ExgIjgVHA7LLrw8BFXZS5F7Cp7a91UeYY4Arb43rfu4iIWJIkmHdC0nbAnsDWthdKWgN4RVfpuwvkZf+7Srk7A8fb3rOhrq7yXA5c3uvGR0TEUifBvHNrAfNsLwRoHzW3B15JKwK/BC6z/X1J822PLMF6CjAPGAfcBRxou6fbjnxE0juB5YB9bT8o6WCqEfvRktYEvgusX9IfCTzcnlnS+sBlwIeAzYC9gJWADYBf2v5kSbcb8AVgeeAh4BDb8yV9reRpBa61fbykfYGTqGYmnrG9Yy+PYUREDJJcM+/ctcC6kv4o6SxJOzXsGwn8D3CR7e93kncr4FhgU6rgu30T9c2zvTVwNnB8J/vPBG60vSWwNTCzfYeqTxiXUQXmO8vm8cB+wObAfpLWLbMLJwC7lrqmAR+XtDrwLmAz21sAJ5cyTgR2L3Xu1VMHJE2R1Nb4w38uJ0RExABKMO+E7fnANlQj3SeAi8tIGeDXwI9sX9BF9jtsz7X9IjAdGNNElb8ov+/qIv0uVIEe24tsP1O2jyrtOdD29Ib0U20/Y/t54H7gtcAbqT5g3CJpOvCBsv0fwPPADyS9G3iulHELcJ6kw4ERPXXA9hTbLY0/VOsEIiJigGWavQu2FwE3ADdIupcq+EEV5PaQdFEX0+cLG14vorlj3J6n2fTtngH+SjX6n9mwvbM2tADX2T6gYyGStgUmAvsDRwO72D5C0huAdwDTJY23/WQv2hYREYMkI/NOqLJRw6bxwJ/L6xOBJ4GzBrFJU6mukyNphKRVyvYXgEnAQZLe10MZtwPbS9qwlLOSpI0ljQRWtX0V1eWB8WX/BrZ/b/tEqjUA6y72XkVExGKRYN65kcD5ku6XNINqenpKw/5jgRUkfX2Q2vNR4C1lhuAuqkVuANh+lmrl/cck7d1VAbafAA4Gflr6dDuwCbAycEXZdiPwsZLlG5LulXQfcBNwz2LvVURELBYtbXm+YwyQ8l342bNmTaW1dfRQN6fP8l8kIgbT3LlzmThxIsBY23OayZOReURERM0lmEdERNRcgnlERETNJZhHRETUXIJ5REREzeWmMTHgZs+G0fVdzB4RMexlZB4REVFzCeYRERE1l2AeERFRcwnmERERNZcFcDHgxo6F1tahbkXf5XauETHcZWQeERFRcwnmERERNZdgHhERUXMJ5hERETWXYB4REVFzCeYRERE1l6+m9YOkRcC9VMfxAeADtp8b2lY1R9Kttt801O2IiIj+y8i8fxbYHm97HPACcETjTkktkoblMU4gj4hYcmRkvvjcDGwhaQzwG+B6YDtgkiQBXwCWBx4CDrE9X9LbgdOAecDdwPq295Q0BVgPWL/8PsP2mQCSfgWsC6wAfNP298r2+cA3gT2BBcDeth+TtCbw3VIWwJG2b5U03/bIkvcTwHtL+35p+yRJrwQuAUYDI4Av2b54IA5cRET0z7AcNdaNpGWBPaim3AEEXGB7K+BZ4ARgV9tbA9OAj0taATgH2MP2DsCoDsVuAuwObAucJGm5sv1Q29sAE4BjJL2qbH8lcLvtLYGbgMPL9jOBG8v2rYGZHdq+G7BRqWc8sI2kHYG3AQ/b3rLMPFzdwzGYIqmt8QeY3cOhi4iIxSAj8/5ZUdL08vpm4IfA2sCfbd9etr8R2BS4pRqg8wrgNqpgPct2e8D7KfChhrKvtL0QWCjpcWBNYC5VAH9XSbMuVSB+kmqa/4qy/S7greX1LsBBALYXAc906MNu5ecP5f3IUubNwKmSTgGusH1zdwfC9hRgSuO2MkuRgB4RMcASzPtnge3xjRtKwH62YVMLcJ3tAzqk26qHshc2vF4ELCtpZ2BXYDvbz0m6gWq6HeBfttsa0zfZhxbgq7bP6bhD0jbA24GvSrrW9hebLDMiIgZRptkH3u3A9pI2BJC0kqSNgQeB9cvoFWC/JspaFXi6BPJNqEb9PZkKHFnqHiFplQ77rwEOldR+/XwdSa+WtDbwnO2fAKdSTdFHRMQwlGA+wGw/ARwM/FTSDKrgvontBcCHgasl/Q54jJdPgXd0NdUIfQbwpVJWTz4KvEXSvVTT75t1aN+1wEXAbSXNpcDKwObAHeUywueAk5uoKyIihkBLW57vOGQkjSyr2luA7wD/Z/v0oW7X4tJ+zXzWrKm0to4e6ub0Wf6LRMRgmjt3LhMnTgQYa3tOM3kyMh9ah5eR70yqKfSXXbeOiIjoSRbADaEyCl9iRuIRETE0MjKPiIiouQTziIiImss0ewy42bNhdH3Xv0VEDHsZmUdERNRcgnlERETNJZhHRETUXIJ5REREzSWYR0RE1FyCeURERM0lmEdERNRcgnlERETN5aYxMZBGADz66KND3Y6IiNpo+Js5otk8CeYxkNYCmDx58lC3IyKijtYCHmomYYJ5DKQ7y+8NgUVD2ZB+mg2MHepG9NOS0AdYMvqRPgwPw7kPI6gC+Z09JWzX0tbWNnDNiaWepDbbLUPdjv5IH4aPJaEf6cPwsCT0oVEWwEVERNRcgnlERETNJZhHRETUXIJ5DLQvDHUDFoP0YfhYEvqRPgwPS0If/i0L4CIiImouI/OIiIiaSzCPiIiouQTziIiImkswj4iIqLkE84iIiJpLMI+IiKi5PGgl+kTS24BvUj0Q4Ae2v9Zh//LABcA2wJPAfrbnlH2fAQ6jevjKMbavGcSmv0Rf+yFpDPAA4JL0dttHDFrDX9rGnvqwI3AGsAWwv+1LG/Z9ADihvD3Z9vmD0+qX6mcfFgH3lrd/sb3X4LT6pZrow8eBDwKtwBPAobb/XPYNi/NQ2tKfftTlXBwBHEX1N2g+8CHb95d9w+bvU29kZB7+CtK1AAAHp0lEQVS9JmkE8B1gD2BT4ABJm3ZIdhjwtO0NgdOBU0reTYH9gc2AtwFnlfIGXX/6UTxke3z5GapA3kwf/gIcDFzUIe/qwEnAG4BtgZMk/ddAt7mj/vShWNBwHoYqeDTThz8AE2xvAVwKfL3kHRbnobSlz/0o6nIuLrK9ue3xVO0/reQdNn+feivBPPpiW+BPtmfZfgH4GbB3hzR7A+2ji0uBiZJayvaf2V5oezbwp1LeUOhPP4aLHvtge47tGcCLHfLuDlxn+ynbTwPXUf0BG2z96cNw0Uwfrrf9XHl7OzC6vB4u5wH614/hopk+/KPh7SuB9runDae/T72SYB59sQ7w14b3c8u2TtPYbgWeAV7VZN7B0p9+AIyV9AdJN0p680A3tgv9OZ7D5Vz0tx0rSJom6XZJkxZv05rW2z4cBvymj3kHUn/6ATU6F5KOkvQQ1cj8mN7kHY4SzKMvOhuZdrwvcFdpmsk7WPrTj0eA9WxvBXwcuEjSKou5fc3oz/EcLueiv+1Yz/YE4H3AGZI2WDzN6pWm+yDpQGAC8I3e5h0E/ekH1Ohc2P6O7Q2AT/Gf9QrD6Vz0SoJ59MVcYN2G96OBh7tKI2lZYFXgqSbzDpY+96NMwz0JYPsu4CFg4wFv8cv153gOl3PRr3bYfrj8ngXcAGy1OBvXpKb6IGlX4HPAXrYX9ibvIOlPP2p1Lhr8DGifRRhO56JXspo9+uJOYCNJY4G/US0YeV+HNJcDHwBuA/YB/td2m6TLqUaxpwFrAxsBdwxay1+qP/0YRRXUF0lan6ofswav6f/WTB+6cg3wlYbFVrsBn1n8TexRn/tQ2v6c7YWS1gC256ULsgZLj32QtBVwDvA224837Bou5wH60Y+anYuNbP9fefsOoP31cPr71CsZmUevlWvHR1P9EXoAuMT2TElflNS+gvWHwKsk/YlqGvrTJe9M4BLgfuBq4Cjbiwa7D6Utfe4HsCMwQ9I9VAvjjrD91OD2oLk+SHq9pLnAvsA5kmaWvE8BX6L643cn8MW69QF4HTCtnIfrga+1f8VouPWBajp6JPBzSdPLB9thcx5KW/rcD+p1Lo6WNFPSdKr/1x8oeYfN36feyiNQIyIiai4j84iIiJpLMI+IiKi5BPOIiIiaSzCPiIiouQTziIiImsv3zCOiS5LmAM8DC4FXAP9t+wdN5JsCjLR9fA/pJgEP276jvJ8AfMz25P61/N/ljwGm2V5jcZTXxzYcS/Vgj8d7TBzRRxmZR0RP9rG9JdV3vM+StPZiLHsSDQ+ysD1tcQXyoSZpmfJQnmOBVw91e2LJlpF5RDTF9n2SnqZ68MTDAJI+SXVnvGWp7rZ1uO1HG/NJ2hw4i+rpVCsA37N9hqTdgb2AXSV9kOoxlH8BTrU9QdIPgRm2v1nKGUd1h64NgJVL+i1KmdcDH+/pBh+S2qjuwz2J6oE5hwO7Uj2lbDlgX9sPSNqZ6nnYdwNbUj27++CGZ15/Cnh/KfZO4CO255cZiQ2pbqqyAdWtQtcGLpX0PNWdyNYCTi7tXhb4su2flXJvKOVtV/JdYvvTZd86wJlUdyUD+Kntr5ZnAvT6WMSSJSPziGiKpO2BecA95f2BVIHrjba3Bq4C/ruTrHOAXUuabYEPSXqd7WuogvPXyvOvL+iQ7zzKnbmKQ4DzbLdRBa8bbW8LjKca+R7aZFf+bvv1VA/Y+DXwu/LAnAuo7jfebotS39ZUz8e+oPR7D6pA/iZgc2AE8PmGfDsCHyzPy/4y1QeffUof76f6gLBDqXNX4NQOzy9fr5SxFfBBSe3B+yfA7ba3KM8S/37Z3p9jEUuIjMwjoieXluniDYB3lWdEQzWqngDcLQmqvyfPdJJ/JeBsSVtSPY98barR7gPdVWr7ZkkrS9qC6vaaB1CNWNvr3lbScQ11zG2yPxeX33cDbbavLO/vAt7dkO5Ptm8sr38MfK+Mgneleub1PwAkfY9qFN/uKtvzuql/FHBuCdKtwOqAqJ4NDvBz2y8Cz0h6ANhA0iNUHx7e2l5IQx39ORaxhEgwj4ie7FOm2PcFfiJpY9uPUT0u8mTb5/aQ/yvAo1TT1K2SrqWaDm7GBVSj8xuAB2z/uWxvASaVp3P11vPl9yKqhX00vG/mb2ILL38sZuP7+T3kP5tqRuLd5aE9f+Slx+P5htfNtKk/xyKWEJlmj4im2P45cC3/edjM5cCH26eIJS1fRt8drQb8tQTyccCbG/b9g+qxsl05n2pE/kHgRw3bLwc+LWlEqXuN8pSsxWlDSe1tfR9wbxmNXwfsX2YNWkrbfttNOR37uBowpwTyt1JdquiW7fnArcDH2reVJ5PB4ByLGOYSzCOiNz4DHCrpNbZ/DFwI3ChpBtU09fad5DkZOFzSnVSLz25q2Pdj4H3l6VsHdcxo+y9UU+w7A79o2HUs1aj1Hkn3Uj3hap3+dq6D6cABku4CjgEOKm36DdX169uAe0vak7sp50zgR6WPm1J9GDpVUvtjdWc02Z4Dge0l3VeeTHZY2T4YxyKGuTw1LSKig7Ka/VTbE4a6LRHNyMg8IiKi5jIyj4iIqLmMzCMiImouwTwiIqLmEswjIiJqLsE8IiKi5hLMIyIiau7/A+c23UD3WGk8AAAAAElFTkSuQmCC\n", "text/plain": [ "" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "# Show your work here - the plot below was helpful for me\n", "# https://stackoverflow.com/questions/44101458/random-forest-feature-importance-chart-using-python\n", "\n", "features = diabetes.columns[:diabetes.shape[1]]\n", "importances = random_search.best_estimator_.feature_importances_\n", "indices = np.argsort(importances)\n", "\n", "plt.title('Feature Importances')\n", "plt.barh(range(len(indices)), importances[indices], color='b', align='center')\n", "plt.yticks(range(len(indices)), features[indices])\n", "plt.xlabel('Relative Importance');" ] }, { "cell_type": "markdown", "metadata": {}, "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> **Step 7**: Using your results above to complete the dictionary below." ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "That's right! Some of these were expected, but some were a bit unexpected too!\n" ] } ], "source": [ "# Check your solution by matching the correct values in the dictionary\n", "# and running this cell\n", "a = 'Age'\n", "b = 'BloodPressure'\n", "c = 'BMI'\n", "d = 'DiabetesPedigreeFunction'\n", "e = 'Insulin'\n", "f = 'Glucose'\n", "g = 'Pregnancy'\n", "h = 'SkinThickness'\n", "\n", "\n", "\n", "sol_seven = {\n", " 'The variable that is most related to the outcome of diabetes' : f,\n", " 'The second most related variable to the outcome of diabetes' : c,\n", " 'The third most related variable to the outcome of diabetes' : a,\n", " 'The fourth most related variable to the outcome of diabetes' : d\n", "}\n", "\n", "ch.check_q_seven(sol_seven)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> **Step 8**: Now provide a summary of what you did through this notebook, and how you might explain the results to a non-technical individual. When you are done, check out the solution notebook by clicking the orange icon in the upper left." ] }, { "cell_type": "markdown", "metadata": {}, "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ " \n", "\n", "In this case study, we looked at predicting diabetes for 768 patients. There was a reasonable amount of class imbalance with just under 35% of patients having diabetes. There were no missing data, and initial looks at the data showed it would be difficult to separate patients with diabetes from those that did not have diabetes.\n", "\n", "Three advanced modeling techniques were used to predict whether or not a patient has diabetes. The most successful of these techniques proved to be an AdaBoost Classification technique, which had the following metrics:\n", "\n", "Accuracy score for adaboost : 0.7792207792207793\n", "\n", "Precision score adaboost : 0.7560975609756098\n", "\n", "Recall score adaboost : 0.5636363636363636\n", "\n", "F1 score adaboost : 0.6458333333333333\n", "\n", "Based on the initial look at the data, it is unsurprising that Glucose, BMI, and Age were important in understanding if a patient has diabetes. These were consistent with more sophisticated approaches. Interesting findings were that pregnancy looked to be correlated when initially looking at the data. However, this was likely due to its large correlation with age.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.3" } }, "nbformat": 4, "nbformat_minor": 2 }