Finished Model Evaluation Metrics
This commit is contained in:
@@ -0,0 +1,354 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Boston Housing Data\n",
|
||||
"\n",
|
||||
"In order to gain a better understanding of the metrics used in regression settings, we will be looking at the Boston Housing dataset. \n",
|
||||
"\n",
|
||||
"First use the cell below to read in the dataset and set up the training and testing data that will be used for the rest of this problem."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from sklearn.datasets import load_boston\n",
|
||||
"from sklearn.model_selection import train_test_split\n",
|
||||
"import numpy as np\n",
|
||||
"import tests2 as t\n",
|
||||
"\n",
|
||||
"boston = load_boston()\n",
|
||||
"y = boston.target\n",
|
||||
"X = boston.data\n",
|
||||
"\n",
|
||||
"X_train, X_test, y_train, y_test = train_test_split(\n",
|
||||
" X, y, test_size=0.33, random_state=42)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **Step 1:** Before we get too far, let's do a quick check of the models that you can use in this situation given that you are working on a regression problem. Use the dictionary and corresponding letters below to provide all the possible models you might choose to use."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"That's right! All but logistic regression can be used for predicting numeric values. And linear regression is the only one of these that you should not use for predicting categories. Technically sklearn won't stop you from doing most of anything you want, but you probably want to treat cases in the way you found by answering this question!\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# When can you use the model - use each option as many times as necessary\n",
|
||||
"a = 'regression'\n",
|
||||
"b = 'classification'\n",
|
||||
"c = 'both regression and classification'\n",
|
||||
"\n",
|
||||
"models = {\n",
|
||||
" 'decision trees': c,\n",
|
||||
" 'random forest': c,\n",
|
||||
" 'adaptive boosting': c,\n",
|
||||
" 'logistic regression': b,\n",
|
||||
" 'linear regression': a\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"#checks your answer, no need to change this code\n",
|
||||
"t.q1_check(models)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **Step 2:** Now for each of the models you found in the previous question that can be used for regression problems, import them using sklearn."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Import models from sklearn - notice you will want to use \n",
|
||||
"# the regressor version (not classifier) - googling to find \n",
|
||||
"# each of these is what we all do!\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **Step 3:** Now that you have imported the 4 models that can be used for regression problems, instantate each below."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Instantiate each of the models you imported\n",
|
||||
"# For now use the defaults for all the hyperparameters\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **Step 4:** Fit each of your instantiated models on the training data."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Fit each of your models using the training data\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **Step 5:** Use each of your models to predict on the test data."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Predict on the test values for each model\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **Step 6:** Now for the information related to this lesson. Use the dictionary to match the metrics that are used for regression and those that are for classification."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# potential model options\n",
|
||||
"a = 'regression'\n",
|
||||
"b = 'classification'\n",
|
||||
"c = 'both regression and classification'\n",
|
||||
"\n",
|
||||
"#\n",
|
||||
"metrics = {\n",
|
||||
" 'precision': # Letter here,\n",
|
||||
" 'recall': # Letter here,\n",
|
||||
" 'accuracy': # Letter here,\n",
|
||||
" 'r2_score': # Letter here,\n",
|
||||
" 'mean_squared_error': # Letter here,\n",
|
||||
" 'area_under_curve': # Letter here, \n",
|
||||
" 'mean_absolute_area' # Letter here \n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"#checks your answer, no need to change this code\n",
|
||||
"t.q6_check(metrics)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **Step 6:** Now that you have identified the metrics that can be used in for regression problems, use sklearn to import them."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Import the metrics from sklearn\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **Step 7:** Similar to what you did with classification models, let's make sure you are comfortable with how exactly each of these metrics is being calculated. We can then match the value to what sklearn provides."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def r2(actual, preds):\n",
|
||||
" '''\n",
|
||||
" INPUT:\n",
|
||||
" actual - numpy array or pd series of actual y values\n",
|
||||
" preds - numpy array or pd series of predicted y values\n",
|
||||
" OUTPUT:\n",
|
||||
" returns the r-squared score as a float\n",
|
||||
" '''\n",
|
||||
" sse = np.sum((actual-preds)**2)\n",
|
||||
" sst = np.sum((actual-np.mean(actual))**2)\n",
|
||||
" return 1 - sse/sst\n",
|
||||
"\n",
|
||||
"# Check solution matches sklearn\n",
|
||||
"print(r2(y_test, preds_tree))\n",
|
||||
"print(r2_score(y_test, preds_tree))\n",
|
||||
"print(\"Since the above match, we can see that we have correctly calculated the r2 value.\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **Step 8:** Your turn fill in the function below and see if your result matches the built in for mean_squared_error. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def mse(actual, preds):\n",
|
||||
" '''\n",
|
||||
" INPUT:\n",
|
||||
" actual - numpy array or pd series of actual y values\n",
|
||||
" preds - numpy array or pd series of predicted y values\n",
|
||||
" OUTPUT:\n",
|
||||
" returns the mean squared error as a float\n",
|
||||
" '''\n",
|
||||
" \n",
|
||||
" return None # calculate mse here\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# Check your solution matches sklearn\n",
|
||||
"print(mse(y_test, preds_tree))\n",
|
||||
"print(mean_squared_error(y_test, preds_tree))\n",
|
||||
"print(\"If the above match, you are all set!\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **Step 9:** Now one last time - complete the function related to mean absolute error. Then check your function against the sklearn metric to assure they match. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def mae(actual, preds):\n",
|
||||
" '''\n",
|
||||
" INPUT:\n",
|
||||
" actual - numpy array or pd series of actual y values\n",
|
||||
" preds - numpy array or pd series of predicted y values\n",
|
||||
" OUTPUT:\n",
|
||||
" returns the mean absolute error as a float\n",
|
||||
" '''\n",
|
||||
" \n",
|
||||
" return None # calculate the mae here\n",
|
||||
"\n",
|
||||
"# Check your solution matches sklearn\n",
|
||||
"print(mae(y_test, preds_tree))\n",
|
||||
"print(mean_absolute_error(y_test, preds_tree))\n",
|
||||
"print(\"If the above match, you are all set!\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"> **Step 10:** Which model performed the best in terms of each of the metrics? Note that r2 and mse will always match, but the mae may give a different best model. Use the dictionary and space below to match the best model via each metric."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"#match each metric to the model that performed best on it\n",
|
||||
"a = 'decision tree'\n",
|
||||
"b = 'random forest'\n",
|
||||
"c = 'adaptive boosting'\n",
|
||||
"d = 'linear regression'\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"best_fit = {\n",
|
||||
" 'mse': # letter here,\n",
|
||||
" 'r2': # letter here,\n",
|
||||
" 'mae': # letter here\n",
|
||||
"}\n",
|
||||
"\n",
|
||||
"#Tests your answer - don't change this code\n",
|
||||
"t.check_ten(best_fit)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# cells for work"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.6.3"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
Reference in New Issue
Block a user