{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "## Title :\n", "Exercise: Hyper-parameter Tuning for Ridge Regression\n", " \n", "## Description :\n", "The goal of this exercise is to perform hyper-parameter tuning and produce a plot similar to the one below:\n", " \n", "\n", "\n", "## Data Description:\n", "- The dataset has a total of 3 columns with names - x,y and f\n", "- \"$x$\" represents the predictor variable\n", "- \"$y$\" is the response variable\n", "- \"$f$\" denotes the true values of the underlying function\n", "\n", "## Instructions:\n", "- Read the dataset `polynomial50.csv` as a dataframe.\n", "- Assign the predictor and response variables.\n", "- Visualize the dataset by making plots using the predictor and response variables along with the true function.\n", "- Split the data into train and validation sets using `random_state=42`.\n", "- For each value of alpha from a given list:\n", " - Estimate a Ridge regression on the training data with the alpha value.\n", " - Calculate the MSE of training and validation data. Append to separate lists appropriately.\n", " - Use the given **plot_functions** function to plot the value of parameters.\n", "- Compute the best hyperparameter for this data based on the lowest MSE\n", "- Make a plot of the MSE values for each value of hyper-parameter alpha from the list above. It should look similar to the one given above.\n", "\n", "## Hints: \n", "\n", "sklearn.Ridge()\n", "Linear least squares with L2 regularization.\n", "\n", "sklearn.train_test_split()\n", "Splits the data into random train and test subsets.\n", "\n", "ax.plot()\n", "Plot y versus x as lines and/or markers.\n", "\n", "sklearn.PolynomialFeatures()\n", "Generates a new feature matrix consisting of all polynomial combinations of the features with degree less than or equal to the specified degree.\n", "\n", "sklearn.fit_transform()\n", "Fits transformer to X and y with optional parameters fit_params and returns a transformed version of X.\n", "\n", "sklearn.Ridge()\n", "Linear least squares with L2 regularization.\n", "\n", "sklearn.predict()\n", "Predict using the linear model. \n", "\n", "mean_squared_error()\n", "Mean squared error regression loss.\n", "\n", "**Note:** This exercise is auto-graded and you can try multiple attempts. " ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "# Import necessary libraries\n", "%matplotlib inline\n", "import numpy as np\n", "import pandas as pd\n", "import seaborn as sns\n", "import matplotlib.pyplot as plt\n", "from sklearn.linear_model import Ridge, Lasso\n", "from sklearn.metrics import mean_squared_error\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.preprocessing import PolynomialFeatures\n", "plt.style.use('seaborn-white')\n", "\n", "# These are custom functions made to help you visualise your results\n", "from helper import plot_functions\n", "from helper import plot_coefficients\n" ] }, { "cell_type": "code", "execution_count": 52, "metadata": {}, "outputs": [], "source": [ "# Open the file 'polynomial50.csv' as a Pandas dataframe\n", "df = pd.read_csv('polynomial50.csv')\n" ] }, { "cell_type": "code", "execution_count": 0, "metadata": {}, "outputs": [], "source": [ "# Take a quick look at the data\n", "df.head()\n" ] }, { "cell_type": "code", "execution_count": 54, "metadata": {}, "outputs": [], "source": [ "# Assign the values of the 'x' column as the predictor\n", "x = df[['x']].values\n", "\n", "# Assign the values of the 'y' column as the response\n", "y = df['y'].values\n", "\n", "# Also assign the true value of the function (column 'f') to the variable f \n", "f = df['f'].values\n" ] }, { "cell_type": "code", "execution_count": 0, "metadata": {}, "outputs": [], "source": [ "# Visualise the distribution of the x, y values & also the value of the true function f\n", "fig, ax = plt.subplots()\n", "\n", "# Plot x vs y values\n", "ax.plot(___,___, '.', label = 'Observed values',markersize=10)\n", "\n", "# Plot x vs true function value\n", "ax.plot(___,___, 'k-', label = 'Function description')\n", "\n", "# Helper code to annotate the plot\n", "ax.legend(loc = 'best')\n", "ax.set_xlabel('Predictor - $X$',fontsize=16)\n", "ax.set_ylabel('Response - $Y$',fontsize=16)\n", "ax.set_title('Predictor vs Response plot',fontsize=16)\n", "plt.show();\n" ] }, { "cell_type": "code", "execution_count": 132, "metadata": {}, "outputs": [], "source": [ "# Split the data into train and validation sets with \n", "# training size 80% and random_state = 42\n", "x_train, x_val, y_train, y_val = train_test_split(x,y, train_size = 0.8,random_state=42)\n" ] }, { "cell_type": "code", "execution_count": 0, "metadata": {}, "outputs": [], "source": [ "### edTest(test_mse) ### \n", "\n", "fig, rows = plt.subplots(6, 2, figsize=(16, 24))\n", "\n", "# Select the degree for polynomial features\n", "degree= ___\n", "\n", "# List of hyper-parameter values \n", "alphas = [0.0, 1e-7,1e-5, 1e-3, 0.1,1]\n", "\n", "# Create two lists for training and validation error\n", "training_error, validation_error = [],[]\n", "\n", "# Compute the polynomial features train and validation sets\n", "x_poly_train = PolynomialFeatures(___).fit_transform(___)\n", "x_poly_val= PolynomialFeatures(___).fit_transform(___)\n", "\n", "# Loop over all the alpha values\n", "for i, alpha in enumerate(alphas):\n", "\n", " # Code to get the plot grid\n", " l, r = rows[i]\n", " \n", " # Initialize a Ridge regression with the current alpha\n", " ridge = Ridge(fit_intercept=False, alpha=___)\n", "\n", " # Fit the model on the transformed training data\n", " ridge.fit(___,___)\n", " \n", " # Predict on the transformed training set \n", " y_train_pred = ridge.predict(___)\n", "\n", " # Predict on the transformed validation set \n", " y_val_pred = ridge.predict(___)\n", " \n", " # Compute the training and validation errors\n", " mse_train = mean_squared_error(___, ___) \n", " mse_val = mean_squared_error(___, ___)\n", " \n", " # Add the error values to the appropriate list \n", " training_error.append(___)\n", " validation_error.append(___)\n", " \n", " # Calling the helper functions plot_functions & \n", " # plot_coefficients to visualise the plots\n", " plot_functions(degree, ridge, l, df, alpha, x_val, y_val, x_train, y_train)\n", " plot_coefficients(ridge, r, alpha)\n", "\n", "sns.despine();\n" ] }, { "cell_type": "code", "execution_count": 136, "metadata": {}, "outputs": [], "source": [ "### edTest(test_hyper) ###\n", "# Find the best value of hyper parameter, which \n", "# gives the least error on the validdata\n", "best_parameter = ___\n", "\n", "# Print the best hyper parameter\n", "print(f'The best hyper parameter value, alpha = {best_parameter}')\n" ] }, { "cell_type": "code", "execution_count": 0, "metadata": {}, "outputs": [], "source": [ "# Plot the errors as a function of increasing d value \n", "# to visualise the training and validation errors\n", "fig, ax = plt.subplots(figsize = (12,8))\n", "\n", "# Plot the training errors for each alpha value\n", "ax.plot(___,___,'s--', label = 'Training error',color = 'Darkblue',linewidth=2)\n", "\n", "# Plot the validation errors for each alpha value\n", "ax.plot(___,___,'s-', label = 'validation error',color ='#9FC131FF',linewidth=2 )\n", "\n", "# Draw a vertical line at the best parameter\n", "ax.axvline(___, 0, 0.5, color = 'r', label = f'Min validation error at alpha = {best_parameter}')\n", "\n", "ax.set_xlabel('Value of Alpha',fontsize=15)\n", "ax.set_ylabel('Mean Squared Error',fontsize=15)\n", "ax.set_ylim([0,0.010])\n", "ax.legend(loc = 'upper left',fontsize=16)\n", "ax.set_title('Mean Squared Error',fontsize=20)\n", "ax.set_xscale('log')\n", "plt.tight_layout()\n", "plt.show();\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.5" } }, "nbformat": 4, "nbformat_minor": 4 }