{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Title\n",
"\n",
"**Exercise: Regularization using L1 and L2 Norm**\n",
"\n",
"# Description\n",
"\n",
"The goal of this exercise is to understand the affect of L1 and L2 regularization in Neural Networks."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"\n",
"NOTE: This graph is only a sample (including the mindchow part). The data will be the same, the model predictions will depend on your regularization parameters."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Instructions:\n",
"\n",
"- Generate the predictor and response data using the helper code given.\n",
"- Split the data into train and test sets.\n",
"- Visualise the split data using the helper code.\n",
"- Build a simple neural network with 5 hidden layers with 100 neurons each. This network has no regularization.\n",
"- Compile the model with MSE as the loss.\n",
"- Fit the model on the training data and save the history.\n",
"- Use the helper code to visualise the MSE of the train and test data with respect to the epochs.\n",
"- Predict on the entire data. \n",
"- Use the helper function to plot the predictions along with the generated data.\n",
"- Repeat steps 4 to 8 with a neural network with L2 regularization. Try with a range of 1e-3 to 1e-1.\n",
"\n",
"# Hints:\n",
"\n",
"Use the Dense layer to regularize using l2 and l1 regularization. More details can be found here.\n",
"\n",
"tf.keras.sequential() : A sequential model is for a plain stack of layers where each layer has exactly one input tensor and one output tensor.\n",
"\n",
"tf.keras.optimizers() : An optimizer is one of the two arguments required for compiling a Keras model\n",
"\n",
"model.add() : Adds layers to the model.\n",
"\n",
"model.compile() : Compiles the layers defined into a neural network\n",
"\n",
"model.fit() : Fits the data to the neural network\n",
"\n",
"model.predict() : Used to predict the values given the model\n",
"\n",
"history() : The history object is returned from calls to the fit() function used to train the model. Metrics are stored in a dictionary in the history member of the object returned.\n",
"\n",
"tf.keras.regularizers.L2() : A regularizer that applies a L2 regularization penalty."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"# Import the libraries\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"import warnings\n",
"warnings.filterwarnings(\"ignore\")\n",
"import tensorflow as tf\n",
"from tensorflow.keras import layers\n",
"from tensorflow.keras import models\n",
"from sklearn.metrics import mean_squared_error\n",
"from tensorflow.keras import optimizers\n",
"from tensorflow.keras import regularizers\n",
"from sklearn.model_selection import train_test_split\n",
"np.random.seed(0)\n",
"tf.random.set_seed(0)\n",
"%matplotlib inline"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"# Use the helper code below to generate the data\n",
"\n",
"# Defines the number of data points to generate\n",
"num_points = 30 \n",
"\n",
"# Generate predictor points (x) between 0 and 5\n",
"x = np.linspace(0,5,num_points)\n",
"\n",
"# Generate the response variable (y) using the predictor points\n",
"y = x * np.sin(x) + np.random.normal(loc=0, scale=1, size=num_points)\n",
"\n",
"# Generate data of the true function y = x*sin(x) \n",
"# x_b will be used for all predictions below \n",
"x_b = np.linspace(0,5,100)\n",
"y_b = x_b*np.sin(x_b)\n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"# Split the data into train and test sets with .33 and random_state = 42\n",
"x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.33, random_state=42)\n"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAmAAAAHgCAYAAAACM9GVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAABSUklEQVR4nO3de1xVVf7/8dfiHERFBREDTQXUzFIQDdPUvKSSlqVZamp5qSabpprfzHxnmpnfTOPMt5lvv7l8m0szlZWlaek4alpZKd7KNBVNwbsmUCaggIDiBc5h//44ehIFRYWzD4f38/HggWuvzT6fII9v1l57LWNZFiIiIiLiO0F2FyAiIiJS3yiAiYiIiPiYApiIiIiIjymAiYiIiPiYApiIiIiIjymAiYiIiPiY0+4CrkRkZKQVGxtrdxkiIiIil7Vly5Y8y7JaVtZXpwJYbGwsqampdpchIiIiclnGmKyq+nQLUkRERMTHFMBEREREfEwBTERERMTH6tQcMBEREYGysjIOHTrE6dOn7S5FgIYNG9KmTRuCg4Or/TUKYCIiInXMoUOHaNq0KbGxsRhj7C6nXrMsi/z8fA4dOkRcXFy1v063IEVEROqY06dP06JFC4UvP2CMoUWLFlc8GqkAJiIiUgcpfPmPq/lZKICJiIgEutxcWJUCixd6PufmXtPl8vPzSUxMJDExkejoaK6//npvu7S09JJfm5qayjPPPHPVr/3WW2/x1FNPXfKcNWvWsH79+qt+DV/QHDAREZFAlroZ0tPA5fK08/IhKwviEyCp51VdskWLFmzbtg2A6dOn06RJE/7rv/7L2+9yuXA6K48YSUlJJCUlXdXrVteaNWto0qQJffr0qdXXuRYaARMREQlUubkVwxcAlqednnbNI2HnmzJlCj/+8Y8ZNGgQzz77LJs2baJPnz50796dPn36sHfvXsATjkaMGAF4wtsjjzzCwIEDad++PX//+98rvfabb75Jp06dGDBgAJ9//rn3+Pvvv0+vXr3o3r07Q4YMITc3l8zMTF555RVefPFFEhMT+eyzzyo9z24aARMREQlUO9MvCF/ncbk9/VFRNfZy+/btIyUlBYfDQXFxMZ9++ilOp5OUlBR++ctfsnDhwou+Zs+ePaxevZrjx49z44038v3vf7/Ccg7Z2dn85je/YcuWLYSFhTFo0CC6d+8OQL9+/fjiiy8wxvD666/zxz/+kb/85S888cQTFUbljh07Vul5dlIAExERCVRFRZfotC7Tf+XGjBmDw+E4+9JFTJ48mf3792OMoaysrNKvufvuuwkJCSEkJITrrruO3Nxc2rRp4+3fuHEjAwcOpGVLz57W48aNY9++fYBnOY5x48aRnZ1NaWlplctAVPc8X9ItSBERkUAVFgZU9YSeOdtfc0JDQ71//vWvf82gQYPYsWMH77//fpXLNISEhHj/7HA4cFUyYlfVU4ZPP/00Tz31FOnp6bz66qtVvkZ1z/MlBTAREZFA1SUenI7K+5wOT38tKSoq4vrrrwc8Ty5erV69erFmzRry8/MpKytjwYIFlb7GrFmzvMebNm3K8ePHL3uenRTARMSjhh9TFxE/EBXledrR6eS7kTDjaccn1Oj8rwv97Gc/4xe/+AV9+/bF7XZf9XVatWrF9OnTue222xgyZAg9evTw9k2fPp0xY8Zw++23ExkZ6T1+zz33sHjxYu8k/KrOs5OxLMvuGqotKSnJSk1NtbsMkcBz4WPqGM9vx9fwmLqI1J7du3dz0003Vf8LcnM9E+6Lijy3HbvE12r4qo8q+5kYY7ZYllXpmhuahC9S313uMfW27fRGLVLXRUXp77Gf0S1IkfquOo+pi4hIjVIAE6nvfPyYuoiIKICJiI8fUxcREQUwEbHxMXURkfpKAUykvrPxMXURkfpKT0GKiGepibbt9Ji6SIDKOpbDusw08koKiQwNp19sAjHNo6/6evn5+QwePBiAnJwcHA6Hd6ugTZs20aBBg0t+/Zo1a2jQoAF9+vS57GvFxsaSmpp6yfW7/vCHP/DLX/7yCv4L7KcAJiIeekxdJCAt37eRtRnbcLldWMDh4jx2HclgQFwiyZ16XdU1W7RowbZt2wDPYqjnb3xdHWvWrKFJkybVCmDVURcDmG5BioiIBKisYzmszdhG2dnwBWBhUeZ2sfbgNrKO5dTYa23ZsoUBAwZwyy23cOedd5KdnQ3A3//+d26++WYSEhJ48MEHyczM5JVXXuHFF1/0rlR/vvz8fJKTk+nevTvTpk3j/AXjR40axS233EKXLl2YMWMGAD//+c85deoUiYmJTJw4scrz/I1WwhcREaljqrsS/twvl5OWvZ/K/qU3GBJadWRi9+RrqmX69OmEhoayePFilixZQsuWLZk/fz6ffPIJM2fOpHXr1mRkZBASEkJhYSHh4eGXHDV75plniIyM5LnnnuPDDz9kxIgRHD16lMjISAoKCoiIiODUqVP07NmTtWvX0qJFC5o0acKJEye816jqvNqklfBFREQEgLySwkrDF3hGwvJLCmvkdc6cOcOOHTsYOnQoAG63m1atWgGQkJDAxIkTGTVqFKNGjbrstT799FMWLVoEwN13303z5s29fX//+99ZvHgxAN988w379++vNFhV9zw7KYCJiIgEqMjQcA4X52FVEsMMhsjQ8Bp5Hcuy6NKlCxs2bLio78MPP+TTTz9l6dKl/Pd//zc7d+687PWMuXhtwjVr1pCSksKGDRto3LgxAwcO5PTp01d9nt00B0xERCRA9YtNwOmofJ0/Z5CDvrEJNfI6ISEhHD161BvAysrK2LlzJ+Xl5XzzzTcMGjSIP/7xjxQWFnLixAmaNm3K8ePHK71W//79mTt3LgAfffQRx44dA6CoqIjmzZvTuHFj9uzZwxdffOH9muDgYMrKyi57nj9RABMREQlQMc2jGRCXSHCQE3N2nT+DITjIyYD2ide0FMX5goKC+M9//sOzzz5Lt27dSExMZP369bjdbh566CHi4+Pp3r07P/rRjwgPD+eee+5h8eLFlU7C/81vfsOnn35Kjx49WL58Oe3atQNg2LBhuFwuEhIS+PWvf03v3r29X/P44497b3Ve6jx/okn4IiIidUx1J+Gfc24dsPySQlrUwDpgcjFNwhcREZEKYppHK3D5Gd2CFBEREfExBTARERERH1MAExEREfExBTARERERH1MAExEREfExBTARERG5Ivn5+SQmJpKYmEh0dDTXX3+9t11aWlojr/HZZ5/RpUsXEhMTOXXqVI1cE+APf/hDhXafPn1q7NpXQuuAiYiI1DFXug5YbapsY22Xy4XTeW0rXT3xxBP06tWLqVOnXmuJFVy4cXdNudJ1wDQCJiIiItdsypQp/PjHP2bQoEE8++yzTJ8+nT//+c/e/q5du5KZmQnAnDlzuPXWW0lMTGTatGm43e4K13r99df597//ze9+9zsmTpzImjVrGDFihLf/qaee4q233gIgNjaW3/zmN/To0YP4+Hj27NkDwIkTJ5g6dSrx8fEkJCSwcOFCfv7zn3Pq1CkSExOZOHEi4Alk4NnP8qc//Sldu3YlPj6e+fPnA569JQcOHMgDDzxA586dmThxIjUxeKWFWEVEROqwyZMn19q1Z82adUXn79u3j5SUFBwOB9OnT6/0nN27dzN//nw+//xzgoODefLJJ5k7dy6TJk3ynvPYY4+xbt06RowYwQMPPMCaNWsu+bqRkZFs3bqVf/3rX/z5z3/m9ddf57//+78JCwsjPT0dgGPHjnH//ffz0ksvsW3btouusWjRIrZt28b27dvJy8ujZ8+e9O/fH4Avv/ySnTt30rp1a/r27cvnn39Ov379ruh7cyEFMBEREakRY8aMwVHF5t/nrFy5ki1bttCzZ08ATp06xXXXXXdNrzt69GgAbrnlFhYtWgRASkoK8+bN857TvHnzS15j3bp1jB8/HofDQVRUFAMGDGDz5s00a9aMW2+9lTZt2gCQmJhIZmamApiIiIj4h9DQUO+fnU4n5eXl3vbp06cBz62+yZMn8z//8z/Vvm5V1zonJCQEAIfDgcvl8r6OMabar3Gp24rnrn/ha1wLBTAREZE67EpvE/pKbGwsH3zwAQBbt24lIyMDgMGDBzNy5Eh+9KMfcd1111FQUMDx48eJiYmp8loxMTHs2rWLM2fOcPr0aVauXHnZEajk5GReeukl/vrXvwKeW5DNmzcnODiYsrIygoODK5zfv39/Xn31VSZPnkxBQQGffvopf/rTn7xzymqaJuGLSI3JOpbD3C+X87d1/2bul8vJOpZjd0kiYpP777+fgoICEhMTefnll+nUqRMAN998M88//zzJyckkJCQwdOhQsrOzL3mttm3bMnbsWBISEpg4cSLdu3e/7Ov/6le/4tixY3Tt2pVu3bqxevVqAB5//HHvdc533333kZCQQLdu3bjjjjv44x//SHR07W1grmUoRKRGLN+3kbUZ23C5XViAweB0OBgQl0hyp152lycSUPxpGQrx0DIUIuJzWcdyWJuxjbKz4QvAwqLM7WLtwW0aCRMRuYDmgInINVuXmYbLXfmkVFe5m3WZacQ0r72hfL+Xmws706GoCMLCoEs8REXZXZWI2EgBLJDpTV98JK+kkKomM1hY5JcU+rIc/5K6GdLT4NxTU3n5kJUF8QmQ1NPe2kTENgpggUpv+uJDkaHhHC7Ow6okhhkMkaHhvi/KH+TmVvx7CIDlaaenQdt2+qVIrtqVLrMgtedq5tNrDlggutybfm6ubaVJYOoXm4CzisUXnUEO+sYm+LgiP7Ez/YK/h+dxuT39IlehYcOG5Ofn18iWOHJtLMsiPz+fhg0bXtHXaQQsEFXnTV+/dUsNimkezYC4RNYe3Iar3I2F5XkKMsjBgPaJ9Xf+V1HRJTqty/SLVK1NmzYcOnSIo0eP2l2K4AnE51bKry4FsECkN32xQXKnXtzYMoZ1mWnklxTSIjScfrEJ9Td8gWfuZV4+VDpDznj6Ra5CcHAwcXFxdpch10ABLBDpTV9sEtM8un4Hrgt1iffMvaxsRNrp8PSLSL2kOWCBqEu85829MnrTF/GdqCjPgy9OJ3BusrTxtOMTNBVApB7TCFggOvemn57mmfN1dl1ynA696Yv4WlJPz9OOWhJGRM6jABao9KYv4j+iovR3T0QqUAALZDXxpq/FXEVERGqcAphUTYu5ioiI1ApNwpfKaTFXERGRWqMAJpXTCt4iIiK1RgFMKqfFXEVERGqNAphULiyM79YtupAWcxUREbkWCmBSOS3mKiIiUmsUwKRyWsFbRESk1mgZCqmaFnMVERGpFQpgcmlawVtERKTG6RakiIiIiI9pBExEApO20RIRP6YAJiKBR9toiYif0y1IEQks2kZLROoA2wKYMaatMWa1MWa3MWanMeaHdtUiIgFE22iJSB1g5y1IF/ATy7K2GmOaAluMMSssy9plY00iPpV1LId1mWnklRQSGRpOv9gEYppH211W3aZttESkDrAtgFmWlQ1kn/3zcWPMbuB6QAFM6oXl+zayNmMbLrcLCzhcnMeuIxkMiEskuVMvu8uru8LCPHO+sCrp1DZaIuIf/GIOmDEmFugObLS5FBGfyDqWw9qMbZSdDV8AFhZlbhdrD24j61iOrfXVadpGS0TqANsDmDGmCbAQ+D+WZRVX0v+4MSbVGJN69OhR3xcoUgvWZabhclc+T8lV7mZdZpqPKwog2kZLROoAW5ehMMYE4wlfcy3LWlTZOZZlzQBmACQlJVV2T0GkzskrKaz0Bhl4RsLySwp9WY7/udY1vLSNloj4OdsCmDHGAG8Auy3L+l+76hCxQ2RoOIeL87AqiWEGQ2RouO+L8hc1tYaXttESET9m5whYX+BhIN0Ys+3ssV9alrXMvpLErwXQyub9YhPYdSSDskpuQzqDHPSNTaj+xQLo+3LZNbzatqu7/20iIuex8ynIdXw3QUPk0gJsZfOY5tEMiEtk7cFtuMrdWFgYDM4gBwPaJ1Z/KYoA+75Uaw0vBTARCQDaikj8X4COiiR36sWNLWNYl5lGfkkhLa50HbBA/L5oDS8RqScUwM7Sgph+LIBHRWKaR1/9/2c++L6Ul5dTXFxMYWEhhYWFlJSUcObMGUpLSzlz5gxnzpzB7XbjcDgqfAQHB9OkSROaNWtG06ZNvZ8bNGhw6RfUGl4iUk8ogKEFMf2eRkUqV0PfF5fLxdGjR8nOziY7O5vDhw+TnZ1Nfn4+RUVFWFbNPXwcHh5OdHR0hY927doRERGBMcYzfy0rq/JgqTW8RCSA1PsAdv6CmOecvyDmjS1jNBJmN42KVO4qvi/l5eXk5uZy8OBBMjIyOHjwIF9//TVlZWW1Xi7gHUnbs2dPheNhYWF06NCB9u3b0yE8gvZ5R2kIeP7bjCd8aQ0vEQkg9T6AVWdBTAUwm2lUpHLV+L5YlsXhw4fZtWsXu3btYs+ePZw8efKKXqZJkyaEh4cTHh5O06ZNCQkJISQkhAYNGhASEoLD4aC8vByXy4Xb7cbtdlNaWsrx48c5fvw4xcXF3j+73e5KX6OoqIitW7eydevWs+U76HRdS+IjIkjo0J7r+96OidbfQxEJHPU+gGlBzDrg3Mrm6WmeuU0aFfGo4vtSYlmkhTRk++LF7Nq1i6Jq3IqMiIigVatWFT6uu+46wsPDCQ4OrpFy3W43eXl55OTkkJubS05ODt9++y2ZmZmcPn26wrkut5td2Tnsys5h/s5dRKxbT7du3ejduzedOnUiKMh3m3hofqiI1AZTk/M7altSUpKVmppao9ec++Vy0rIPVLkgZrdWHZnQPblGX1OuUiCtd1WTcnPJ3/gFW/fv58ujR9mTewR3eXmVpzdt2pT27dsTFxdH+/btad++PU2bNvVhwRWVl5dz+PBhDh48yFdffcX+/fv59ttvqzw/IiKCXr16cdttt9GuXTvP3LFacuH8UIPB6XBofqiIVIsxZotlWUmV9tX3AJZ1LIcZm5ZUuiBmcJCTx3uN1G+74peKi4vZuHEjGzZs4KuvvqryvNDQUG666SZuuukmbr75Zlq1alWroaUmFBQUsGPHDtLS0ti5c2eVt01bt27NHXfcQb9+/WjUqFGN1qD3BhG5Vgpgl7F838YqF8TUb7niT86cOUNqaiobNmxg586dlFcx0hUXF0f37t1JSEggJibGp7fsaprb7Wb//v1s3LiRzZs3c/z48YvOadiwIX369GHw4MG0adOmRl7XMzq+v4pHHAwJrToyUaPjInIJCmDVcG6ex1UtiClSyzIyMli7di0bNmy4aL4UgMPh4Oabb6Z79+50796diIgIG6qsfS6Xi507d/LFF1+wZcsWzpw5c9E5N910E3fffTddu3a9ppG+v637N98WH62yv02zljzTb+xVX19EAt+lAli9n4R/zjUtiClSC06dOsWGDRtYs2YNWVlZlZ7TqVMnbrvtNm699VaaNGni4wp9z+l00q1bN7p168apU6dYv349KSkpHD582HvO7t272b17Nx06dODee++lW7duVxXEtGG6iNQmjYCJ+JmcnBxSUlL47LPPKh3tatWqFf369aN3795ERkbaUKF/sSyLPXv2sHLlSrZs2XLRbdmYmBhGjhxJjx49riiIaQ6YiFwr3YIU8XOWZbFr1y6WL1/O9u3bL1p9Pjg4mJ49ezJw4EA6derk95Po7ZKfn8+HH37I2rVrcV2wPlqHDh2YMGECHTt2rPb1ND9URK6FApiIn3K73XzxxRcsW7aMQ4cOXdR/7im/Pn36EBoaakOFddOxY8dYtmwZq1evvmiV/1tvvZWxY8fSsmXLal1L80NF5GopgIn4mTNnzvDZZ5/x0UcfkZeXd1F/t27dSE5OpkuXLhrtugZFRUUsW7aMlJSUCiNiTqeToUOHcu+999K4cWMbKxSRQKYAJuInTp06RUpKCp988slFyymEhIRw++23M3ToUKK17U6NOnr0KAsWLGDjxo0VjoeHh/PQQw+RlJSkoCsiNU4BTMRmp06dYsWKFXz88ceUlJRU6GvSpAnJyckMHjy4XjzJaKcDBw7wzjvvXLRwbWJiIpMmTaJFixY2VSYigUgBTMQm50a8Pvroo4uCV4sWLRg+fDj9+/cnJCTEpgrrH8uy2LhxI++8806FfTIbNmzI/fffz5AhQ+r0wrUi4j8UwER8rLS0lJUrV/LBBx9w4sSJCn0tW7bk3nvvpU+fPjidWorPLiUlJSxYsIDVq1dXON6hQwemTZtGlPYZFZFrpAAm4iNut5vPPvuMJUuWUFBQUKFPwcs/7du3jzfffLPCYq4hISFMnDiR/v37a26YiFw1BTCRWmZZFqmpqSxcuJDs7OwKfZGRkYwcOVLBy4+VlZXx4YcfsnTpUtxut/d4jx49mDp1Ks2aNbOxOhGpqxTARGrRvn37mDdv3kUTu8PCwhg5ciQDBgxQ8KojMjIyePXVVyuE6LCwMB577DESEhJsrExE6iIFMJFakJOTw4IFC7jw/8lGjRpx9913k5ycrMn1ddCZM2eYP38+K1eurHD8nnvuYfTo0ZqgLyLVpgAmUoNOnDjBe++9x6pVqyrcrjq3uOeIESO0nEQA2L59O2+88UaFJyVvvvlmvv/97+uWpIhUiwKYSA1wuVysWrWKxYsXc/LkyQp9t912G/fff3+1t7eRuqG4uJhXXnmFnTt3eo9FRETwgx/84Ir2lBSR+kkBTOQapaWl8e6771Z4Ug6gSXQEXQbeyt29B2l/wABVXl7Oe++9x5IlS7zHHA4H48ePZ8iQIXpKUkSqpAAmcpVycnJ455132L59e4XjjiYNadqtLQ2uDyfIBOF0OBgQl0hyp142VSq1bfv27bz66qsVFtQdMGAAkyZN0kMWIlIpBTCRK3T69GmWLl3Kxx9/XGGeV0hICA1viiakY0uMo+Jk7OAgJ4/3GqmRsACWl5fHSy+9REZGhvdY586defrppzXvT0QuogAmUk3ntqmZN28ex44d8x43xnD77bdjboxkb/EhKvtbYzAktOrIxO7JvitYfK6srIyZM2eyfv1677GoqCh+9KMf0apVKxsrExF/c6kApuepRc46dOgQL7zwAi+//HKF8NWxY0emT5/Oo48+ynHOVBq+ACws8ksKfVKr2Cc4OJjHH3+cBx54wHssNzeX3/3udxUm64uIXIomLki9d+rUKd577z2WL19OeXm593hYWBjjxo2jT58+3onWkaHhHC7Ow6okhhkMkaHhvipbbGSM4Z577iE6OpoZM2ZQWlrKyZMn+fOf/8yjjz5Kv3797C5RRPycApjUW5ZlsWnTJt555x0KCwu9x4OCgkhOTmbkyJE0bty4wtf0i01g15EMytyui67nDHLQN1arpdcnPXv2pGXLlrz44osUFhZSXl7Oa6+9RnFxMXfddZfd5YmIH9McMH+Umws706GoCMLCoEs8REXZXVVAycnJYfbs2RfdMurcuTMPP/wwbdq0qfJrl+/byNqD23CVu7GwMBicQQ4GtNdTkPVVQUEB//u//8s333zjPTZs2DDGjRunlfNF6jFNwq9LUjdDehq4zo2wGHA6ID4BknraWlogKC0t5YMPPuDDDz/E5fpuFCssLIwHH3yQ2267rVrrOmUdy2FdZhr5JYW0CA2nX2yCnn6s50pKSvjb3/7G3r17vcf69OnDo48+qmUqROopBbC6IjcXln1wXvg6j9MJd43QSNg1SE9PZ/bs2Rw5csR7zBjDkCFDGD169EW3G0WuVGlpKa+88gpbtmzxHouPj+fpp5/WvqAi9ZACWF2xKgW++qqKTgMd2sMdQ3xaUiA4duwY7777Lhs3bqxwvEOHDkyePJmYmBibKpNAVF5ezqxZs1izZo332I033siPfvQjGjVqZF9hIuJzlwpgGhf3J+dt+nsx6zL9cqHy8nJWrlzJwoULOXXqlPd448aNGTt2LAMGDND8HKlxQUFBTJkyhbCwMO/2RXv37uXPf/4zP/nJTzTSKiKAAph/CQuDvHyoYplPwsJ8XVGdlZWVxZtvvllhxXLwzMl5cPBgwr7OhCWL9ZCD1ApjDKNHj6ZRo0bMmzcPgAMHDvDHP/6Rn/70p4SGhtpcoYjYTQHMn3SJh6ysKuaAOTz9ckmnT59m0aJFLF++nPNvr0dHRzN58mRuPlkCn6397nucl+/5nushB6kFw4cPx+l0MmfOHAAyMjJ44YUX+NnPfkbTpk1trk5E7KT7L/4kKsoTBJxO4NyTeMbTjk/QKM1lbN26lV/84hd88skn3vDldDq57777eP7557m5RYsLnjAFsDzt9DTPQxAiNWzo0KFMmTLF2/7666954YUXKC4utq8oEbGdRsD8TVJPaNtO64BdgYKCAt5++222bt1a4fjNN9/M5MmTiY4+uzzEzvTKRxcBXG5Pv77PUgsGDRqEw+Fg5syZWJbFoUOH+H//7//xi1/8Qpt4i9RTCmD+KCpKQaAaysvLSUlJYeHChZw+fdp7vGnTpowfP77CFkKAHnIQW/Xv3x+n08mMGTO8IexPf/oTzz77rCbmi9RDCmBSJ2VmZvLmm2+SmZlZ4Xj//v0ZN25c5aMKeshBbNanTx8sy+K1117DsiwyMzP5y1/+wk9/+lMaNmxod3ki4kMKYFKnnDp1ikWLFrFixYoKk+xbt27NlClTuPHGG6v+Yj3kIH6gb9++lJWV8eabbwKepyNffPFFfvKTn9CgQQObqxMRX1EAkzpjy5YtzJkzh4KCAu+x4OBg7r33Xu66667Lb/dy7iGH9DTPnC8sKmz1pNu+4iMDBw6ktLSUuXPnArBnzx7+/ve/88Mf/pDg4GCbqxMRX1AAE7+Xn5/PnDlzLppk36VLFyZPnkzUlQQnPeQgfiI5OZmysjL+/e9/A56tsl555RV+8IMfaIFgkXpAAUz8lsvlYvny5SxevJjS0lLv8aZNmzJhwoRqb5x9ET3kIH7i7rvvprS0lPfeew+A1NRUZs+ezeTJk6/u/20RqTMUwMQvHThwgLfeeotvvvmmwvEBAwYwduxYPbovAWPUqFGcPHmS5cuXA7B69WrCwsK47777bK5MRGqTApj4lZKSEhYsWMCaNWsqTLK//vrrmTJlCp06dbKxOpGaZ4xh/PjxHD9+nA0bNgDw3nvv0axZMwYPHmxzdSJSWxTAxC9YlsXnn3/OvHnzOH78uPd4gwYNGDVqFHfeeeflJ9mL1FFBQUE89thjnDhxgvT0dADefvttmjVrRs+e2iJLJBDpXzSx3aFDh5g9ezZ79+6tcLxbt248/PDDtGzZ0qbKRHzH6XTy1FNP8cILL5CRkYFlWbzyyis0adKEm266ye7yRKSGmfNv8/i7pKQkKzU11e4ypIacOXOGJUuW8PHHH+N2u73HIyIimDBhAklJSZqILPVOcXExv//978nJyQGgcePG/OpXv+L666+3uTIRuVLGmC2WZSVV2qcAJr5mWRapqam88847Fdb0cjgc3HnnnYwcOVKrgku9dvToUZ5//nkKCwsBiIyM5LnnniNMuzWI1CkKYOI3srOzefvtt9m5c2eF4506dWLy5Mm0adPGpspE/EtWVha///3vOXPmDADt27fn5z//OSEhITZXJiLVpQAmtqvqdmPTpk0ZN24c/fr10+1GkQts27aNv/71r94ngpOSkrRQq0gdcqkApkn4Uqssy+KLL75g/vz5HDt2zHvcGMPgwYMZPXo0oaGhNlYo4r8SExOZOHEic+bMATwLtS5YsIBx48bZXJmIXCsFMKk1WVlZzJkzh3379lU43rFjRyZNmkRMTIxNlYnUHUOHDuXIkSPehVqXLVvGddddx6BBg2yuTESuhQKY1Ljjx4+zcOHCixZTDQsLY8yYMfTt21e3UESuwPjx4zl69Chffvkl4FkjLDo6WstTiNRhmgMmNcblcpGSksKSJUs4efKk97jD4SA5OZmRI0fSqFEjGysUqbvOnDnDH/7wBzIzMwFo0qQJ06dP1zp5In5Mk/ClVlmWxdatW5k/fz65ubkV+uLj45kwYQKtW7e2qTqRwFFQUMD06dMpKioCoE2bNvz617/Wsi0ifkoBTGpNVlYW77zzDnv27KlwPCoqivHjx5OYmKinG0Vq0IEDB/if//kfXC4XALfccgtPPfWUbuuL+CE9BSk1Li8vj4ULF7Jhw4YK87waN27MyJEjGTJkiPZuFKkFHTt2ZMqUKbz++usAbNmyhffee4/Ro0fbXJmIXAn9C1nTcnNhZzoUFUFYGHSJh6gou6uqMSUlJbz//vukpKRQVlbmPR4UFMTgwYMZNWoUTZo0sbFCkcB3++2388033/DJJ58AsGTJEtq2bVu9jbsD/D1KpK5QAKtJqZshPQ3O3hogLx+ysiA+AZKq8cbox0pLS0lJSeGDDz6gpKSkQl9iYiLjxo3TPC+RWpZ1LId1mWnklRQScWMEHTJv4Ku9+wGYMWMG0dHRtG3btuoLBPB7lEhdowBWU3JzK76xAWB52ulp0LZdnfwt0+Vy8emnn7JkyRLvvnTntG/fnnHjxtG5c2d7ihOpR5bv28jajG243C4s4HBxHkHxzWl6NIzjBUWUlpbyj3/8g+nTp9O4ceOLLxCg71EidZUCWE3ZmX7BG9t5XG5Pfx16cysvL2f9+vW89957HD16tEJfVFQUY8aMISkpSRPsRXwg61gOazO2Ueb+7j3GwsLtgMa94ziTsovS0lJyc3OZMWMGzzzzzMWT8gPsPUqkrlMAqylnHwuvnHWZfv9RXl7Opk2bWLJkCYcPH67QFx4ezr333suAAQM0wV7Eh9ZlpuFyVx6eTNMQOg+5lbRl6wD48ssv+eCDD7j33nsrnhgg71EigUL/itaUsDDPfAoqW9bDePr9mNvtZuPGjSxdupTs7OwKfaGhoYwYMYLBgwcTEhJiU4Ui9VdeSWGl7yzgGQkLvj6MYcOG8fHHHwOwaNEi4uLiiI+P/+7EOv4eJRJoFMBqSpd4z2TWyob4nQ5Pvx9yuVx88cUXLF269KJFVBs2bMiwYcMYNmyYVrAXsVFkaDiHi/OwKglPBkNkaDhjxw4mMzOTPXv2YFkWL7/8Mr/97W+/Wym/jr5HiQQqrdxXU6KiPE8SOZ3AuXlRxtOOT/C7uRVnzpxh+fLlPPvss7z22msVwlejRo0YOXIkf/nLX7jvvvsUvkRs1i82AafDUWmfM8hB39gEHA4HTz75JOHh4YBnyZh//OMflJaWek6sY+9RIoFOK+HXND9fY+f48eOkpKSQkpLCiRMnKvQ1btyY5ORkkpOTCQ0NtalCEanM8n0bWXtwG65yNxYWBoMzyMGA9okkd+rlPe/AgQP84Q9/wO12A3DHHXcwefLk7y7k5+9RIoFEWxEJhw8fZvny5Xz++eff/UZ8VtOmTRk6dChDhw6t/PF1EfEL59YByy8ppEVoOP1iE4hpHn3ReStWrGDOnDne9pNPPkmvXr0uOk9Eape2IqqnysvLSUtLY8WKFezYseOi/sjISIYPH87tt9+uyfUidUBM8+hKA9eFhgwZwt69e9m8eTMAM2fOJCYmhujoy3+tiPiGAlgAOnHiBOvWrWPVqlUXTawHaNeuHXfddRe33norjirmlYhI3WWM4ZFHHiErK4sjR45w+vRp/vnPf/LrX/+aBg0a2F2eiKBbkAHDsiz27t3LmjVr2Lx5M64LnnQyxtC9e3eSk5Pp3LmzFlAVqQeysrL43e9+530/GDRoEFOmTLG3KJF6RLcgA1hBQQEbNmzgs88+u2j9LvBMrO/fvz9Dhgz57nF0EakXYmJimDBhArNnzwZg9erV3Hjjjdx22202VyYiCmB10KlTp0hNTWX9+vXs3r2bykYx4+LiGDhwIL1796Zhw4Y2VCki/uCOO+5g7969bNy4EYC33nqLuLg4zQcTsZkCWB1x+vRptm/fTmpqKtu2bbvoSUbwLJzap08fBg4cSExMjA1Vioi/McYwdepUMjMzyc3N5fTp0/zrX//i17/+NcHBwXaXJ1JvKYD5sZKSEr788ktSU1PZsWMHZWVlF51jjOHmm2+mT58+JCUlabRLRC7SqFEjfvCDH3jng2VlZfGf//yH8ePH212aSL2lAOZHysvLycrKIi0tjfT0dA4cOFDp7UWAtm3b0qdPH3r37k1ERISPKxWRuiYmJoZx48Yxd+5cAD7++GO6dOlCQkKCzZWJ1E8KYDayLIvc3Fz27NnD3r172bFjB8XFxVWe37ZtW5KSkkhKSqJNmzY+rFREAsHQoUPZsWMH27dvB+C1117j+eefJ0wbcYv4nJah8KEzZ87wzTffkJGRwb59+9i7dy9FRUVVnm+MIS4ujqSkJG655RZNmhWRa1ZcXMyvfvUr73tPfHw8P/7xjwkK0tbAIjXNb5ehMMYMA/4GOIDXLct6wc56aoplWRw7dozs7GwOHz5MZmYmmZmZHD58mPLy8kt+bdOmTenatSsJCQl07dqVZs2a+ahqEakPmjVrxrRp0/jTn/6EZVmkp6fzySefMHz4cLtLE6lXbAtgxhgH8E9gKHAI2GyMWWpZ1i67aqqu8vJyTp48SUFBAQUFBeTn53s/5+TkkJ2dzenTp6t1rcaNG9OpUyduvPFGOnfuTGxsrH4TFZFa1aVLF+666y4+/PBDABYsWEDnzp2Ji4uzuTKR+sPOEbBbgQOWZR0EMMbMA0YCtgSwo0ePsmzZMlwuF263u8LnsrIyTp48yYkTJzh58iQnT56scnL8pRhjiI6OJjY2lvbt29O5c2fatGmjwCUiPjd69Gh2797NwYMHcbvdvPrqq/z2t7/VvrAiPmJnALse+Oa89iGgl021cOLECVatWlVj12vcuDGtW7emVatWtGnThtjYWGJiYmjUqFGNvYaIyNVyOp088cQTPPfcc5w+fZrs7Gzmz5/PpEmT7C5NpF6wM4BVthnhRcNKxpjHgcfBs4l0bXE6r+xb0ahRI8LDw4mIiKBFixZEREQQERFBVFQUrVq1olmzZtpvUUT8WlRUFBMnTuSNN94AYOXKlXTr1o1u3brZXJlI4LMzgB0C2p7XbgMcvvAky7JmADPA8xRkbRXTvHlzHn74YZxOJ06nE4fD4f2z0+mkcePGhIaGEhoaSuPGjXE4HLVVioiIz9x+++18+eWXbN26FYA33niD559/Xg8AidQyOwPYZuAGY0wc8C3wIDDBrmKaNGnCkCFD7Hp5ERFbGGN45JFH+OqrrygqKqKoqIg333yTZ555RqP4IrXIttnflmW5gKeAT4DdwL8ty9ppVz0iIvVV06ZNeeyxx7ztrVu38umnn9pYkUjgs/XxO8uyllmW1cmyrA6WZf3ezlpEROqzhISECncB5s6dS25uro0ViQQ2rX8gIiIAjB07llatWgGenTtmzJhx2cWjReTqKICJiAgAISEhPPHEE96HjA4cOMBHH31kc1UigUkBTEREvGJjYxk5cqS3vWjRIg4dOmRjRSKBSQFMREQquPvuu73bErlcLmbMmIHL5bK5KpHAogAmIiIVOJ1OHn/8cYKDgwHIyspi6dKlNlclElgUwERE5CKtW7fmgQce8Lbff/99Dh48aGNFIoFFAUx8IzcXVqXA4oWez3q8XcTvJScnc+ONNwJQXl7Oa6+9Rmlpqc1ViQQGBTCpfambYdkH8NVXkJcHXx30tFM3212ZiFxCUFAQjz32GCEhIQAcPnyYhQsX2lyVSGBQAJPalZsL6WlQYQKv5Wmnp2kkTMTPXXfddYwfP97b/uSTT9i/f7+NFYkEBgUwqV070y8IX+dxuT39IuLXBg4cSNeuXQGwLIs33nhDtyJFrpECmNSuoqJLdFqX6RcRf3Buw+6GDRsCkJ2dzeLFi22uSqRuUwCT2hUWBpgqOs3ZfhHxdy1atKhwK/Kjjz7iwIEDNlYkUrcpgEnt6hIPTkflfU6Hp19E6oQBAwbQpUsXwHMr8vXXX9etSJGrpAAmtSsqCuITwOnku5Ew42nHJ3j6RaRO0K1IkZqjACa1L6kn3DUCOrSHyEjP57tGeI6LSJ0SGRnJuHHjvO2PPvqIr776ysaKROomBTDxjagouGMI3He/57NGvkTqrIEDB3LzzTcD392KLCsrs7kqkbpFAUxERK5IUFAQjzzySIUFWt9//32bqxKpWxTARETkirVs2ZIxY8Z42x988AFff/21jRWJ1C0KYCIiclUGDx7MDTfcAIDb7WbmzJm43W6bqxKpGxTARETkqpy7Fel0OgHIyMjgk08+sbkqkbpBAUxERK5a69atGTVqlLe9aNEicrXHq8hlKYCJiMg1GT58OO3atQOgrKyMmTNnUl5ebnNVIv5NAUxERK6J0+nk0UcfJSjI80/Knj17WLt2rc1Vifg3BTAREblmsbGxDB8+3NueN28eBQUFNlYk4t8UwEREpEaMGjWK6OhoAE6fPs3bb79tc0Ui/ksBTEREakSDBg2YOnWqt71161ZSU1NtrEjEfymAiYhIjencuTMDBgzwtt9++21KSkpsrEjEPymAiYhIjRo3bhxhYWEAFBYWsmDBApsrEvE/CmAiIlKjQkNDefjhh73t1atXs3fvXhsrEvE/CmAiIlLjkpKS6N69u7f95ptvUlZWZmNFIv5FAUxERGqcMYZJkybRsGFDALKzs3n//fdtrkrEfyiAiYhIrYiIiGDMmDHe9gcffMC3335rY0Ui/kMBTEREas0dd9xBx44dAXC73bz11lvapkgEBTAREalFQUFBTJkyBYfDAcC+ffv47LPPbK5KxH4KYCIiUqvatm3LsGHDvO358+dTXFxsY0Ui9lMAExGRWjdy5EhatmwJQElJCe+8847NFYnYSwFMRERqXUhICJMnT/a2N2zYwI4dO2ysSMReCmAiIuIT8fHx9O7d29ueNWsWpaWl137h3FxYlQKLF3o+5+Ze+zVFapkCmIiI+MyECRNo3LgxAEeOHGHp0qXXdsHUzbDsA/jqK8jLg68Oetqpm2ugWpHaowAmIiI+ExYWxrhx47ztZcuWcejQoau7WG4upKeBy3XeQcvTTk/TSJj4NQUwERHxqf79+3PDDTcAnrXBZs2adXVrg+1MvyB8ncfl9vSL+CkFMBER8anK1gZbt27dlV+oqOgSndZl+kXsVWUAM8YsM8bE+rAWERGpJ9q0aVNhbbB58+Zx/PjxK7tIWBhgqug0Z/tF/NOlRsDeApYbY/6vMSbYR/WIiEg9MXLkSCIjIwHP2mDz5s27sgt0iQeno/I+p8PTL+KnqgxglmX9G+gONANSjTH/ZYz58bkPn1UoIiIBKSQkhIcfftjbXrduHXv27Kn+BaKiID4BnE6+GwkznnZ8gqdfxE9dbg5YGVAChABNL/gQERG5JomJiSQlJXnbs2bNwlXVxPrKJPWEu0ZAh/YQGen5fNcIz3ERP+asqsMYMwz4X2Ap0MOyrJM+q0pEROqNiRMnsmPHDk6fPs3hw4f56KOPuOeee6p/gaiogBrtyjqWw7rMNPJKCokMDadfbAIxzaPtLktq2KVGwP4vMMayrJ8rfImISG2JiIjg/vvv97aXLFnCkSNHbKzIPsv3bWTGpiWkZe/n2+KjpGUfYMamJSzft9Hu0qSGXWoO2O2WZe30ZTEiIlI/DR48mJiYGADKysqYM2cOlmXZXJVvZR3LYW3GNsrcLs79l1tYlLldrD24jaxjObbWJzVL64CJiIjtHA4HkydPxhjPZPrt27ezZcsWm6vyrXWZabjclc9/c5W7WZeZ5uOKpDYpgImIyJWrhQ2wO3TowMCBA73tuXPncvr06Wu+bl2RV1JIVWN+Fhb5JYW+LEdqmQKYiIhcmVrcAHvMmDE0a9YMgIKCAhYvXnzN16wrIkPDMVUsLGswRIaG+7YgqVUKYCIiUn21vAF2aGgoDz74oLe9fPlyvv7662u6Zl3RLzYBp6PyhWWdQQ76xib4uCKpTQpgIiJSfT7YALtPnz507twZgPLy8qvfrLuOiWkezYC4RIKDnN6RMIMhOMjJgPaJWooiwCiAiYhI9flgA2xjDJMmTfJu1n3gwIGr26y7Dkru1IvHe40koVVH2jRrSUKrjjzeayTJnXrZXZrUsCoXYhUREblIWBjk5UOl08VrbgPs66+/nmHDhvHhhx8CMH/+fLp3707TpoG/EUtM82iNdtUDGgETEZHq8+EG2Odv1n3ixAkWLFhQY9e+SC081SlyKQpgIiJSfT7cADskJISHHnrI2167di0HDhyoset71eJTnSJVUQATEZEr48MNsLt370737t297VmzZuF2u2vuBWr5qU6RqmgOmIiIXDkfboA9ceJEdu7cSWlpKV9//TUrV64kOTm5Zi5enac6A2ijb/EfGgETERG/1rJlS+69915ve+HChRQWFtbMxX3wVKdIZRTARETE7w0bNoxWrVoBcPr0aebNm1czFw4LgypWn6/JpzpFLqQAJiIifi84OJiHH37Y296wYQO7du269gv78KlOkfMpgImISJ3QpUsXevX6bkHS2bNn46pq/lZ1+fCpTpHzKYCJiEidMX78eBo2bAhAdnY2H3300bVf1IdPdYqcowAmIiJ1RvPmzRk9erS3vXTpUvLy8q79wlFRcMcQuO9+z2eNfEktUwATEZE6ZciQIbRt2xaA0tJS5s6da3NFIldOAUxEROoUh8PBpEmTvO2tW7eyfft2GysSuXIKYCIiUud06tSJ22+/3dt+++23KS0ttbEikSujACYiInXS2LFjCQ0NBeDo0aN8+OGHNlckUn0KYCIiUic1a9aMBx54wNv+8MMPydXejVJHKICJiEidNXDgQOLi4gAoKyvj7bffxrIsm6sSuTwFMBERqbOCgoKYPHkyxngWUU1PT2fLli02VyVyeQpgIiJSp8XFxTFo0CBve+7cuZw5c8bGikQuTwFMRETqvAceeICmTZsCUFBQwJIlS2yuSOTSFMBERKTOCw0NZdy4cd72xx9/zOHDh22sSOTSFMBERCQg9O3blxtuuAEAt9vN7NmzNSFf/JYCmIiIBISgoCAmTZpEUJDnn7bdu3fzxRdf2FyVSOUUwEREJGC0a9eOIUOGeNvz5s3j1KlTNlYkUjkFMBERCSijR48mPDwcgMLCQhYtWmRvQSKVUAATEZGA0qhRIx588EFvOyUlha+//trGikQupgAmIiIBp3fv3tx0000AlJeXM3v2bMrLy22uSuQ7tgQwY8yfjDF7jDFpxpjFxphwO+oQEZHAZIxh0qRJOBwOAPbv38/nn39uc1Ui37FrBGwF0NWyrARgH/ALm+oQEZEA1bp1a4YNG+Ztz58/nxMnTthYkch3bAlglmUttyzLdbb5BdDGjjpERCSwjRw5khYtWgBw/Phx/vOf/9hckYiHP8wBewT4yO4iREQk8ISEhDBx4kRve82aNXz11Vc2ViTiUWsBzBiTYozZUcnHyPPO+b+AC5h7ies8boxJNcakHj16tLbKFRGRANWjRw+6desGgGVZzJo1SxPyxXa1FsAsyxpiWVbXSj6WABhjJgMjgInWJfaKsCxrhmVZSZZlJbVs2bK2yhURkQBljOGhhx4iODgYgKysLFavXm1zVVLf2fUU5DDgWeBey7JO2lGDiIjUH9dddx333HOPt/2f//yHoqIiGyuS+s6uOWAvAU2BFcaYbcaYV2yqQ0RE6onhw4cTFRUFwMmTJ5k3b57NFUl9ZtdTkB0ty2prWVbi2Y8n7KhDRETqjwYNGvDwww972+vXr2fPnj02ViT1mT88BSkiIuIT8fHx9OzZ09ueNWsWLpfrEl8hUjsUwEREpF6ZMGECDRs2BODw4cN88sknNlck9ZECmIiI1CsRERHcd9993vZ7771HXl6ejRVJfaQAJiIi9c7QoUNp27YtAKWlpcydW+VylCK1QgFMRETqHYfDwaRJk7ztrVu38uWXX9pYkdQ3CmAiIlIvderUif79+3vbc+bM4cyZMzZWJPWJApiIiNRbY8eOJTQ0FIC8vDzef/99myuS+kIBTERE6q2mTZsyduxYb3vZsmUcPnzYxoqkvlAAExGReq1///507NgRALfbzaxZs7jEFsUiNUIBTERE6rWgoCAmT55MUJDnn8Q9e/awfv16m6uSQKcAJiIi9V67du248847ve13332XEydO2FiRBDoFMBEREWDUqFFEREQAcPz4cRYsWGBzRRLIFMBERESAhg0bVtise82aNezfv9/GiiSQKYCJiIic1aNHDxITE73tt956S5t1S61QABMRETnPww8/TIMGDQA4dOgQK1assLkiCUQKYCIiIueJjIxk1KhR3vbixYvJz8+3ryAJSApgIiIiF7jzzjtp06YNAGfOnGH27NlaG0xqlAKYiIjIBZxOJ5MnT/a2t23bxpYtW2ysSAKNApiIiEglOnXqxMCBA73tOXPmcOrUKfsKkoCiACYiIlKFsWPHEhYWBsCxY8dYuHChzRVJoFAAExERqUJoaCgTJkzwtlNSUjh48KCNFUmgUAATERG5hF69ehEfHw+AZVm89dZbuN1um6uSuk4BTERE5BKMMUyaNIng4GAAsrKytDaYXDMFMBERkcu47rrrKqwNtmjRIvLy8uwrSOo8BTAREZFqGDZsmNYGkxqjACYiIlINTqeTqVOnYowBYPv27WzatMnmqqSuUgATERGppo4dO3LHHXd423PmzOHEiRM2ViR1lQKYiIjIFRgzZgzNmzcHoLi4mPnz59tckdRFCmAiIiJXoFGjRkyaNMnb/vTTT9m9e7eNFUldpAAmIiJyhXr06EFSUpK3/eabb1JaWmpjRVLXKICJiIhchYceeojGjRsDkJuby9KlS22uSOoSBTAREZGr0Lx5c8aOHettL1u2jK+//trGiqQuUQATERG5SgMGDKBTp04AuN1uZs6cqW2KpFoUwERERK5SUFAQU6dOxel0ApCRkcHy5cttrkrqAgUwERGRa9C6deuLtinKzc21ryCpExTARERErtHw4cNp164dAKWlpcycOVPbFMklKYCJiIhcI6fTyaOPPkpQkOef1T179rB27VqbqxJ/pgAmIiJSA2JjYxk+fLi3PW/ePAoKCmysSPyZApiIiEgNGTVqFFFRUQCcOnWKWbNm6VakVEoBTEREpIY0aNCARx55xNvetm0bX3zxhY0Vib9SABMREalBnTt35o477vC258yZQ1FRkY0ViT9SABMREalhY8eOpUWLFgCcOHGC2bNn61akVKAAJiIiUsMaNWpU4VZkamoqmzZtsrEi8TcKYCIiIrWga9euDBgwwNuePXs2xcXFNlYk/kQBTEREpJaMHz+eiIgIwHMrUk9FyjkKYCIiIrVEtyKlKgpgIiIitSg+Pr7Crci3335btyJFAUxERKS2nX8r8vjx47oVKQpgIiIita2yW5Hr16+3sSKxmwKYiIiID8THxzNo0CBve86cOdor8gpkHcth7pfL+du6fzP3y+VkHcuxu6RrogAmIiLiIw8++CAtW7YE4OTJk7z++uu6FVkNy/dtZMamJaRl7+fb4qOkZR9gxqYlLN+30e7SrpoCmIiIiI80bNiQxx9/HGMMADt37mTVqlU2V+Xfso7lsDZjG2VuF+eiqoVFmdvF2oPbrnwkLDcXVqXA4oWez7m5NV5zdSiAiYiI+FCnTp0YPny4tz1v3jxycur27bTatC4zDZfbVWmfq9zNusy06l8sdTMs+wC++gry8uCrg5526uYaqrb6FMBERER87L777uP6668HoLS0lNdeew23221zVf4pr6SQqm7SWljklxRW70K5uZCeBq7zw5zlaaen+XwkTAFMRETExxo0aMC0adNwOBwAHDhwgGXLltlclX+KDA3HYCrtMxgiQ8Ord6Gd6ReEr/O43J5+H1IAExERsUFMTAwjR470thcvXkxGRoaNFfmnfrEJOM8G1Qs5gxz0jU2o3oWKii7RaV2mv+YpgImIiNhkxIgRdOjQAQC3280rr7zCmTNnbK7Kv8Q0j2ZAXCLBQU7vSJjBEBzkZED7RGKaR1fvQmFhUMVIGpiz/b6jACYiImITh8PBtGnTaNiwIQA5OTm8++67Nlflf5I79eLxXiNJaNWRNs1aktCqI4/3Gklyp17Vv0iXeHBWPpKG0+Hp9yGnT19NREREKoiKimLixIm88cYbAKxevZqEhAR69Ohhc2X+JaZ5dPVHuyoTFQXxCWcn4rsBCzCe8BWf4On3IQUwERERm91+++2kpaWxebNnOYSZM2fSvn17wsPD7S0s0CT1hLbtPBPui4o8tx27xPs8fIFuQYqIiNjOGMOUKVNo3rw54Nmw+/XXX6e8vNzmygJQVBTcMQTuu9/z2YbwBQpgIiIifqFJkyYVVslPT08nJSXF5qqktiiAiYiI+Imbb76ZYcOGedvz588nKyvLxoqktiiAiYiI+JH777+fmJgYAFwuF//85z85deqUzVVJTVMAExERe/jJpsj+Jjg4mCeffJKQkBAAcnNzmT17ts1VSU1TABMREd/zo02R/VF0dDSTJ0/2ttevX8+6detsrEhqmpahEBER37rcpsht29n2ZNq1yjqWw7rMNPJKCokMDadfbMJVr13Vt29fdu3a5Q1es2bNon379rRu3bomSxabaARMRER8y882Ra4py/dtZMamJaRl7+fb4qOkZR9gxqYlLN+38aqvOWnSJFq1agVAaWkp//rXvygtLa2pksVGCmAiIuJbfrYpck3IOpbD2oxtlLldWGePWViUuV2sPbiNrGM5V3XdkJAQnnzySYKDgwH45ptvtFVRgFAAExER3/KzTZFrwrrMNFzuykf1XOVu1mWmXfW127Vrx4QJE7ztVatWsWHDhqu+nvgHBTAREfEtP9sUuSbklRR6R74uZGGRX1J4TdcfNGgQPXv29LZnzpzJoUOHrumaYi8FMBER8a1zmyI7nXw3EmY8bRs2Ra4JkaHhmCpG9QyGyNDwa7q+MYZHH32U6GjPhP7S0lJeeuklrQ9WhymAiYiI7yX1hLtGQIf2EBnp+XzXCM/xOqhfbAJOR+Wjes4gB31jE675NRo1asTTTz9NgwYNAMjOzmbmzJlYVlVjb+LPFMBERMQefrIpck2IaR7NgLhEgoOc3pEwgyE4yMmA9olXvRTFhdq0acPUqVO97U2bNrFixYoaubb4ltYBExERqQHJnXpxY8sY1mWmkV9SSItrXAesKn369GH//v2sWrUKgHnz5hEXF8cNN9xQo68jtUsBTEREpIbENI+u8cBVmQkTJpCRkUFGRgZut5uXXnqJ3/72t4SHh9f6a0vN0C1IERGROiY4OJinnnqK0NBQAAoLC/nHP/5BWVmZzZVJdSmAiYiI1EGRkZF8//vfxxjPnLMDBw4we/ZsTcqvIxTARERE6qj4+HjGjRvnbX/66aesXLnSxoqkuhTARERE6rBhw4bRp08fb3vu3Lns3r3bxoqkOhTARERE6jBjDFOnTiUuLg6A8vJyXnrpJY4ePWpzZXIpCmAiIiJ1XIMGDXjmmWcIO7uP5okTJ/jb3/7G6dOnba5MqqIAJiIiEgAiIiJ4+umncZxdkf+bb77h5Zdfpry83ObKpDK2BjBjzH8ZYyxjTKSddYiIiASCG264gSlTpnjb27Zt45133rGvIKmSbQHMGNMWGAp8bVcNIiIigaZ///7cfffd3vaKFStYvny5jRVJZewcAXsR+BmgBUtERERq0AMPPMCtt97qbb/zzjts3brVxorkQrYEMGPMvcC3lmVtr8a5jxtjUo0xqXqiQ0RE5PKCgoL43ve+R8eOHQGwLIuXX36ZjIwMmyuTc2otgBljUowxOyr5GAn8X+C56lzHsqwZlmUlWZaV1LJly9oqV0REJKA0aNCAH/7wh5z7t7O0tJQXX3xRy1P4iVoLYJZlDbEsq+uFH8BBIA7YbozJBNoAW40xtb97qYiISD3SrFkzfvKTn3j3jCwqKuJPf/oTRUVFNlcmPr8FaVlWumVZ11mWFWtZVixwCOhhWVaOr2sREREJdK1ateKZZ57B6XQCkJuby1/+8hdOnjxpc2X1m9YBExERCXCdO3eusHF3VlYWf/3rXyktLbW5svrL9gB2diQsz+46REREAllSUhKPPPKIt713717+9a9/4Xa7bayq/rI9gImIiIhv9O/fn3HjxnnbX375JTNnztRq+TZQABMREalH7rrrrgoLta5bt445c+ZgWVqW05cUwEREROqZMWPG0L9/f2975cqVvP322wphPqQAJiIiUs8YY5g6dSq9evXyHlu5cqVGwnxIAUxERKQeCgoKYtq0aRVCWEpKCnPnzlUI8wGn3QWIiIiIPRwOB9OmTQNg48aNgGfzboCJEydijhyBnelQVARhYdAlHqKibKs3kCiAiYiI1GPnQphlWWzatAnwhDB3TjYPRzQn6NwyFXn5kJUF8QmQ1NPGigODbkGKiIjUcw6HgyeeeIKePb8LVqvSd/DKzl24vEtUWOByQXoa5ObaU2gAUQATERERbwjr3bu399jGvHxe3LWH0+cv1upye25LyjVRABMREREAnE4n06ZNY0hcnPfYjsJC/rhjJyfKys4esTxzwuSaKICJiIiIV1BQEA/d3pdR7dp6j311/AS/T9tBwZkzgPFMyJdrogAmIiIiFZiuCdzXPo5JHeIwZ48dPnWK325P46uTJZ6nIeWaKICJiIhIRVFREJ/A4LZteeLGTjiMJ4YVlpbxP9vT2XDwoM0F1n1ahkJEREQultQT2raj9850mkW24KXNWygpLaXM7eaVV17h22+/ZfTo0QQFaSznapi6tNptUlKSlZqaancZIiIi9U5ubi4vvvgi2dnZ3mM9evRg2rRpNGzY0MbK/JcxZotlWUmV9Sm2ioiIyGVFRUXx3HPPER//3fyvrVu38tvf/pZDhw7ZWFndpAAmIiIi1dK4cWN+/OMfM2zYMO+xw4cPM336dFavXq09JK+AApiIiIhUW1BQEOPHj+d73/seDRo0AKCsrIy33nqLf/7zn5SUlNhcYd2gACYiIiJXrF+/fkyfPp02bdp4j23evJnnnnuOAwcO2FhZ3aBJ+CIiInLVSktLeffdd1m1apX3mDGGIUOGcP/999OoUSMbq7OXJuGLiIhIrWjQoAGTJ0/mqaeeonHjxgBYlsWKFSv45S9/ybZt2+wt0E9pBExERERqRF5eHm+99Rbp6RU367711luZOHEi4eHh9hRmk0uNgCmAiYiISI2xLIsNGzbwzjvvcPz4ce/xhg0bMnz4cIYNG1Zv1g1TABMRERGfOn78OO+++y6ff/55hePNmjXj3nvvZdCgQTidgb0hjwKYiIiI2GLHjh3MnTuXw4cPVzjesmVLRo0aRe/evQM2iCmAiYiIiG3cbjeff/45ixcvpqCgoEJfWFgYgwYNYtCgQQE3R0wBTERERGxXWlrKypUref/99y9asNXhcHDrrbcyePBgOnToEBCbfCuAiYiIiN84efIkKSkprFy5ksLCwov6mzdvzi233MItt9zCjTfeiMPh8H2RNUABTERERGyTdSyHdZlp5JUUEhkaTr/YBGKaR+NyudiyZQvLly+vcvX8Jk2akJCQQMeOHYmLi6Nt27YEBwdfUz2lpaUUFBQQHR19Tde5HAUwERERscXyfRtZm7ENl9uFBRgMToeDAXGJJHfq5T0vIyODVatWsWXLlkvuJ+lwOGjXrh3t2rUjPDycsLAwwsLCaNasGU2aNMHtduNyuSgrK8PtdnPq1CmOHDlCTk4Oubm55OTkUFBQQIMGDZgxYwbGmFr7b1cAExEREZ/LOpbDjE1LKHO7LuoLDnLyeK+RxDSvOArldrvZt28fqampbNmyhWPHjtVafS+++CIRERG1dv1LBbDAfO5TREREbLcuMw1XJeELwFXuZl1m2kUBzOFwcNNNN3HTTTcxceJEMjIy2L9/PwcPHiQjI4MjR45cc11BQUFERkZy4sSJWg1gl6IAJiIiIrUir6SQqu6zWVjklxRe8uuDgoLo0KEDHTp08B47ceIEmZmZZGdnU1xcTFFREUVFRRQXF3Py5EmcTicOh4Pg4GCcTifBwcG0bNmSqKgooqOjiYqKomXLlravPaYAJiIiIrUiMjScw8V5WJXEMIMhMjT8iq/ZpEkTunbtSteuXWugQvvU/UU2RERExC/1i03AWcUSEs4gB31jE3xckf9QABMREZFaEdM8mgFxiQQHOTF4njY0GIKDnAxon3jR/K/6RLcgRUREpNYkd+rFjS1jWJeZRn5JIS3OWwesPlMAExERkVoV0zy63geuC+kWpIiIiIiPKYCJiIiI+JgCmIiIiIiPKYCJiIiI+JgCmIiIiIiPKYCJiIiI+JgCmIiIiIiPKYCJiIiI+JgCmIiIiIiPKYCJiIiI+JgCmIiIiIiPKYCJiIiI+JgCmIiIiIiPKYCJiIiI+JgCmIiIiIiPKYCJiIiI+JgCmIiIiIiPOe0uQERERGpBbi7sTIeiIggLgy7xEBVld1VylgKYiIhIoEndDOlp4HJ52nn5kJUF8QmQ1NPe2gTQLUgREZHAkptbMXwBYHna6WmefrGdApiIiEgg2Zl+Qfg6j8vt6RfbKYCJiIgEkqKiS3Ral+kXX1EAExERCSRhYYCpotOc7Re7KYCJiIgEki7x4HRU3ud0ePrFdgpgIiIigSQqyvO0o9PJdyNhxtOOT9BSFH5Cy1CIiIgEmqSe0Lad1gHzYwpgIiIigSgqSoHLj+kWpIiIiIiPKYCJiIiI+JgCmIiIiIiPKYCJiIiI+JgCmIiIiIiPKYCJiIiI+JgCmIiIiIiPKYCJiIiI+JgCmIiIiIiPKYCJiIiI+JgCmIiIiIiPKYCJiIiI+JgCmIiIiIiPKYCJiIiI+JixLMvuGqrNGHMUyKrll4kE8mr5NeTK6efif/Qz8U/6ufgf/Uz8ky9+LjGWZbWsrKNOBTBfMMakWpaVZHcdUpF+Lv5HPxP/pJ+L/9HPxD/Z/XPRLUgRERERH1MAExEREfExBbCLzbC7AKmUfi7+Rz8T/6Sfi//Rz8Q/2fpz0RwwERERER/TCJiIiIiIjymAnccYM8wYs9cYc8AY83O76xEwxsw0xhwxxuywuxbxMMa0NcasNsbsNsbsNMb80O6a6jtjTENjzCZjzPazP5Pf2l2TeBhjHMaYL40xH9hdi3gYYzKNMenGmG3GmFTb6tAtSA9jjAPYBwwFDgGbgfGWZe2ytbB6zhjTHzgBzLYsq6vd9QgYY1oBrSzL2mqMaQpsAUbp74p9jDEGCLUs64QxJhhYB/zQsqwvbC6t3jPG/BhIAppZljXC7nrEE8CAJMuybF2bTSNg37kVOGBZ1kHLskqBecBIm2uq9yzL+hQosLsO+Y5lWdmWZW09++fjwG7genurqt8sjxNnm8FnP/Tbtc2MMW2Au4HX7a5F/I8C2HeuB745r30I/aMicknGmFigO7DR5lLqvbO3urYBR4AVlmXpZ2K/vwI/A8ptrkMqsoDlxpgtxpjH7SpCAew7ppJj+g1SpArGmCbAQuD/WJZVbHc99Z1lWW7LshKBNsCtxhjdsreRMWYEcMSyrC121yIX6WtZVg9gOPCDs1NdfE4B7DuHgLbntdsAh22qRcSvnZ1ntBCYa1nWIrvrke9YllUIrAGG2VtJvdcXuPfsfKN5wB3GmDn2liQAlmUdPvv5CLAYzxQkn1MA+85m4AZjTJwxpgHwILDU5ppE/M7ZCd9vALsty/pfu+sRMMa0NMaEn/1zI2AIsMfWouo5y7J+YVlWG8uyYvH8e7LKsqyHbC6r3jPGhJ59eAhjTCiQDNjylL0C2FmWZbmAp4BP8Ewq/rdlWTvtrUqMMe8CG4AbjTGHjDGP2l2T0Bd4GM9v9NvOftxld1H1XCtgtTEmDc8vkyssy9KyByIXiwLWGWO2A5uADy3L+tiOQrQMhYiIiIiPaQRMRERExMcUwERERER8TAFMRERExMcUwERERER8TAFMRERExMcUwESkXjLGtDXGZBhjIs62m59tx9hdm4gEPgUwEamXLMv6BngZeOHsoReAGZZlZdlXlYjUF1oHTETqrbNbKm0BZgLfA7pbllVqb1UiUh847S5ARMQulmWVGWN+CnwMJCt8iYiv6BakiNR3w4FsoKvdhYhI/aEAJiL1ljEmERgK9AZ+ZIxpZW9FIlJfKICJSL1kjDF4JuH/H8uyvgb+BPzZ3qpEpL5QABOR+up7wNeWZa042/4X0NkYM8DGmkSkntBTkCIiIiI+phEwERERER9TABMRERHxMQUwERERER9TABMRERHxMQUwERERER9TABMRERHxMQUwERERER9TABMRERHxsf8PkidQJeZsdRgAAAAASUVORK5CYII=\n",
"text/plain": [
""
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"# Plot the train data\n",
"plt.rcParams[\"figure.figsize\"] = (10,8)\n",
"\n",
"plt.plot(x_train,y_train, '.', label='Train data', markersize=15, color='#FF9A98')\n",
"\n",
"# Plot the test data\n",
"plt.plot(x_test,y_test, '.', label='Test data', markersize=15, color='#75B594')\n",
"\n",
"# Plot the true data\n",
"plt.plot(x_b, y_b, '-', label='True function', linewidth=3, color='#5E5E5E')\n",
"\n",
"# Set the axes labels\n",
"plt.xlabel('X')\n",
"plt.ylabel('Y')\n",
"plt.legend()\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Begin with an unregularized NN"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Model: \"Unregularized\"\n",
"_________________________________________________________________\n",
"Layer (type) Output Shape Param # \n",
"=================================================================\n",
"dense_6 (Dense) (None, 100) 10100 \n",
"_________________________________________________________________\n",
"dense_7 (Dense) (None, 100) 10100 \n",
"_________________________________________________________________\n",
"dense_8 (Dense) (None, 100) 10100 \n",
"_________________________________________________________________\n",
"dense_9 (Dense) (None, 100) 10100 \n",
"_________________________________________________________________\n",
"dense_10 (Dense) (None, 100) 10100 \n",
"_________________________________________________________________\n",
"dense_11 (Dense) (None, 1) 101 \n",
"=================================================================\n",
"Total params: 50,601\n",
"Trainable params: 50,601\n",
"Non-trainable params: 0\n",
"_________________________________________________________________\n"
]
}
],
"source": [
"# Building an unregularized NN. \n",
"# Initialise the NN, give it an appropriate name for the ease of reading\n",
"# The FCNN has 5 layers, each with 100 nodes\n",
"model_1 = models.Sequential(name='Unregularized')\n",
"\n",
"# Add 5 hidden layers with 100 neurons each\n",
"model_1.add(layers.Dense(100, activation='tanh', input_shape=(100,)))\n",
"model_1.add(layers.Dense(100, activation='relu'))\n",
"model_1.add(layers.Dense(100, activation='relu'))\n",
"model_1.add(layers.Dense(100, activation='relu'))\n",
"model_1.add(layers.Dense(100, activation='relu'))\n",
"\n",
"# Add the output layer with one neuron \n",
"model_1.add(layers.Dense(1, activation='linear'))\n",
"\n",
"# View the model summary\n",
"model_1.summary()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Compile the model with MSE as loss and Adam optimizer with learning rate as 0.001\n",
"model_1.compile(___)\n",
"\n",
"# Save the history about the model after fitting on the train data\n",
"# Use 0.2 as the validation split with 1500 epochs and batch size of 10\n",
"history_1 = ___"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"# Helper function to plot the data\n",
"# Plot the MSE of the model\n",
"plt.rcParams[\"figure.figsize\"] = (10,8)\n",
"plt.title(\"Unregularized model\")\n",
"plt.semilogy(history_1.history['loss'], label='Train Loss', color='#FF9A98')\n",
"plt.semilogy(history_1.history['val_loss'], label='Validation Loss', color='#75B594')\n",
"plt.legend()\n",
"\n",
"# Set the axes labels\n",
"plt.xlabel('Epochs')\n",
"plt.ylabel('Log MSE Loss')\n",
"plt.legend()\n",
"plt.show()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Use the model above to predict for x_b (used exclusively for plotting)\n",
"y_pred = ___\n",
"\n",
"# Use the model above to predict for x_text \n",
"y_pred_test = ___\n",
"\n",
"# Compute the test MSE\n",
"mse = ___"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Use the helper code to plot the predicted data\n",
"plt.rcParams[\"figure.figsize\"] = (10,8)\n",
"plt.plot(x_b, y_pred, label = 'Unregularized model', color='#5E5E5E', linewidth=3)\n",
"plt.plot(x_train,y_train, '.', label='Train data', markersize=15, color='#FF9A98')\n",
"plt.plot(x_test,y_test, '.', label='Test data', markersize=15, color='#75B594')\n",
"plt.legend()\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Regularize the NN with L2 regularization \n",
"#### The previous graph indicates that there is overfitting\n",
"#### We now initialise a NN with L2 regularization "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# We will now regularise the NN with L2 regularization\n",
"# Initialise the NN, give it a different name for the ease of reading\n",
"model_2 = models.___\n",
"\n",
"# Add L2 regularization to the model\n",
"# Use L2 regularization parameter value of 0.1 \n",
"___\n",
"\n",
"# Add 5 hidden layers with 100 neurons each\n",
"# relu is the activation for all other layers\n",
"# Don't forget to include the regularization parameter from above\n",
"model_2.add(layers.Dense(___, kernel_regularizer=___ , activation=___, input_shape=___)\n",
"___\n",
"___\n",
"___\n",
"___\n",
"\n",
"\n",
"# Add the output layer with one neuron and linear activation\n",
"model_2.add(layers.Dense(___))\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Compile the model with MSE as loss and Adam optimizer with learning rate as 0.001\n",
"model_2.compile(___)\n",
"\n",
"# Save the history about the model after fitting on the train data\n",
"# Use 0.2 as the validation split with 1500 epochs and batch size of 10\n",
"history_2 = ___"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Helper code to plot the MSE of the L2 regularized model\n",
"\n",
"plt.rcParams[\"figure.figsize\"] = (10,8)\n",
"plt.title(\"L2 Regularized model\")\n",
"plt.semilogy(history_2.history['loss'], label='Train Loss', color='#FF9A98')\n",
"plt.semilogy(history_2.history['val_loss'], label='Validation Loss', color='#75B594')\n",
"plt.xlabel('Epochs')\n",
"plt.ylabel('MSE Loss')\n",
"plt.legend()\n",
"plt.show()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Use the regularised model above to predict for x_b (used exclusively for plotting)\n",
"y_l2_regularized_pred = ___\n",
"\n",
"# Use the regularised model above to predict for x_text \n",
"y_l2_pred_test = ___\n",
"\n",
"# Compute the test MSE by predicting on the test data\n",
"mse_l2 = ___"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Use the helper code to plot the predicted data\n",
"\n",
"# Plotting the predicted data using the L2 regularized model\n",
"plt.rcParams[\"figure.figsize\"] = (10,8)\n",
"\n",
"plt.plot(x_b, y_l2_regularized_pred, label='L2 regularized model', color='#5E5E5E', linewidth=3)\n",
"\n",
"# Plotting the predicted data using the unregularized model\n",
"plt.plot(x_b, y_pred, label = 'Unregularized model', color='#005493', linewidth=2)\n",
"\n",
"# Plotting the training data\n",
"plt.plot(x_train,y_train, '.', label='Train data', markersize=15, color='#FF9A98')\n",
"\n",
"# Plotting the testing data\n",
"plt.plot(x_test,y_test, '.', label='Test data', markersize=15, color='#75B594')\n",
"\n",
"# Set the axes labels\n",
"plt.xlabel('X')\n",
"plt.ylabel('Y')\n",
"plt.legend()\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Mindchow 🍲\n",
"\n",
"**Try the same with L1 regularization. Compare the three neural networks models. Which one best reduces overfitting? Why do you think so?**"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"*Your answer here*"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.5"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
![](\"../img/image.png\")
\n",
"\n",
"NOTE: This graph is only a sample (including the mindchow part). The data will be the same, the model predictions will depend on your regularization parameters."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Instructions:\n",
"\n",
"- Generate the predictor and response data using the helper code given.\n",
"- Split the data into train and test sets.\n",
"- Visualise the split data using the helper code.\n",
"- Build a simple neural network with 5 hidden layers with 100 neurons each. This network has no regularization.\n",
"- Compile the model with MSE as the loss.\n",
"- Fit the model on the training data and save the history.\n",
"- Use the helper code to visualise the MSE of the train and test data with respect to the epochs.\n",
"- Predict on the entire data. \n",
"- Use the helper function to plot the predictions along with the generated data.\n",
"- Repeat steps 4 to 8 with a neural network with L2 regularization. Try with a range of 1e-3 to 1e-1.\n",
"\n",
"# Hints:\n",
"\n",
"Use the Dense layer to regularize using l2 and l1 regularization. More details can be found here.\n",
"\n",
"tf.keras.sequential() : A sequential model is for a plain stack of layers where each layer has exactly one input tensor and one output tensor.\n",
"\n",
"tf.keras.optimizers() : An optimizer is one of the two arguments required for compiling a Keras model\n",
"\n",
"model.add() : Adds layers to the model.\n",
"\n",
"model.compile() : Compiles the layers defined into a neural network\n",
"\n",
"model.fit() : Fits the data to the neural network\n",
"\n",
"model.predict() : Used to predict the values given the model\n",
"\n",
"history() : The history object is returned from calls to the fit() function used to train the model. Metrics are stored in a dictionary in the history member of the object returned.\n",
"\n",
"tf.keras.regularizers.L2() : A regularizer that applies a L2 regularization penalty."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"# Import the libraries\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"import warnings\n",
"warnings.filterwarnings(\"ignore\")\n",
"import tensorflow as tf\n",
"from tensorflow.keras import layers\n",
"from tensorflow.keras import models\n",
"from sklearn.metrics import mean_squared_error\n",
"from tensorflow.keras import optimizers\n",
"from tensorflow.keras import regularizers\n",
"from sklearn.model_selection import train_test_split\n",
"np.random.seed(0)\n",
"tf.random.set_seed(0)\n",
"%matplotlib inline"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"# Use the helper code below to generate the data\n",
"\n",
"# Defines the number of data points to generate\n",
"num_points = 30 \n",
"\n",
"# Generate predictor points (x) between 0 and 5\n",
"x = np.linspace(0,5,num_points)\n",
"\n",
"# Generate the response variable (y) using the predictor points\n",
"y = x * np.sin(x) + np.random.normal(loc=0, scale=1, size=num_points)\n",
"\n",
"# Generate data of the true function y = x*sin(x) \n",
"# x_b will be used for all predictions below \n",
"x_b = np.linspace(0,5,100)\n",
"y_b = x_b*np.sin(x_b)\n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"# Split the data into train and test sets with .33 and random_state = 42\n",
"x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.33, random_state=42)\n"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAmAAAAHgCAYAAAACM9GVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAABSUklEQVR4nO3de1xVVf7/8dfiHERFBREDTQXUzFIQDdPUvKSSlqVZamp5qSabpprfzHxnmpnfTOPMt5lvv7l8m0szlZWlaek4alpZKd7KNBVNwbsmUCaggIDiBc5h//44ehIFRYWzD4f38/HggWuvzT6fII9v1l57LWNZFiIiIiLiO0F2FyAiIiJS3yiAiYiIiPiYApiIiIiIjymAiYiIiPiYApiIiIiIjymAiYiIiPiY0+4CrkRkZKQVGxtrdxkiIiIil7Vly5Y8y7JaVtZXpwJYbGwsqampdpchIiIiclnGmKyq+nQLUkRERMTHFMBEREREfEwBTERERMTH6tQcMBEREYGysjIOHTrE6dOn7S5FgIYNG9KmTRuCg4Or/TUKYCIiInXMoUOHaNq0KbGxsRhj7C6nXrMsi/z8fA4dOkRcXFy1v063IEVEROqY06dP06JFC4UvP2CMoUWLFlc8GqkAJiIiUgcpfPmPq/lZKICJiIgEutxcWJUCixd6PufmXtPl8vPzSUxMJDExkejoaK6//npvu7S09JJfm5qayjPPPHPVr/3WW2/x1FNPXfKcNWvWsH79+qt+DV/QHDAREZFAlroZ0tPA5fK08/IhKwviEyCp51VdskWLFmzbtg2A6dOn06RJE/7rv/7L2+9yuXA6K48YSUlJJCUlXdXrVteaNWto0qQJffr0qdXXuRYaARMREQlUubkVwxcAlqednnbNI2HnmzJlCj/+8Y8ZNGgQzz77LJs2baJPnz50796dPn36sHfvXsATjkaMGAF4wtsjjzzCwIEDad++PX//+98rvfabb75Jp06dGDBgAJ9//rn3+Pvvv0+vXr3o3r07Q4YMITc3l8zMTF555RVefPFFEhMT+eyzzyo9z24aARMREQlUO9MvCF/ncbk9/VFRNfZy+/btIyUlBYfDQXFxMZ9++ilOp5OUlBR++ctfsnDhwou+Zs+ePaxevZrjx49z44038v3vf7/Ccg7Z2dn85je/YcuWLYSFhTFo0CC6d+8OQL9+/fjiiy8wxvD666/zxz/+kb/85S888cQTFUbljh07Vul5dlIAExERCVRFRZfotC7Tf+XGjBmDw+E4+9JFTJ48mf3792OMoaysrNKvufvuuwkJCSEkJITrrruO3Nxc2rRp4+3fuHEjAwcOpGVLz57W48aNY9++fYBnOY5x48aRnZ1NaWlplctAVPc8X9ItSBERkUAVFgZU9YSeOdtfc0JDQ71//vWvf82gQYPYsWMH77//fpXLNISEhHj/7HA4cFUyYlfVU4ZPP/00Tz31FOnp6bz66qtVvkZ1z/MlBTAREZFA1SUenI7K+5wOT38tKSoq4vrrrwc8Ty5erV69erFmzRry8/MpKytjwYIFlb7GrFmzvMebNm3K8ePHL3uenRTARMSjhh9TFxE/EBXledrR6eS7kTDjaccn1Oj8rwv97Gc/4xe/+AV9+/bF7XZf9XVatWrF9OnTue222xgyZAg9evTw9k2fPp0xY8Zw++23ExkZ6T1+zz33sHjxYu8k/KrOs5OxLMvuGqotKSnJSk1NtbsMkcBz4WPqGM9vx9fwmLqI1J7du3dz0003Vf8LcnM9E+6Lijy3HbvE12r4qo8q+5kYY7ZYllXpmhuahC9S313uMfW27fRGLVLXRUXp77Gf0S1IkfquOo+pi4hIjVIAE6nvfPyYuoiIKICJiI8fUxcREQUwEbHxMXURkfpKAUykvrPxMXURkfpKT0GKiGepibbt9Ji6SIDKOpbDusw08koKiQwNp19sAjHNo6/6evn5+QwePBiAnJwcHA6Hd6ugTZs20aBBg0t+/Zo1a2jQoAF9+vS57GvFxsaSmpp6yfW7/vCHP/DLX/7yCv4L7KcAJiIeekxdJCAt37eRtRnbcLldWMDh4jx2HclgQFwiyZ16XdU1W7RowbZt2wDPYqjnb3xdHWvWrKFJkybVCmDVURcDmG5BioiIBKisYzmszdhG2dnwBWBhUeZ2sfbgNrKO5dTYa23ZsoUBAwZwyy23cOedd5KdnQ3A3//+d26++WYSEhJ48MEHyczM5JVXXuHFF1/0rlR/vvz8fJKTk+nevTvTpk3j/AXjR40axS233EKXLl2YMWMGAD//+c85deoUiYmJTJw4scrz/I1WwhcREaljqrsS/twvl5OWvZ/K/qU3GBJadWRi9+RrqmX69OmEhoayePFilixZQsuWLZk/fz6ffPIJM2fOpHXr1mRkZBASEkJhYSHh4eGXHDV75plniIyM5LnnnuPDDz9kxIgRHD16lMjISAoKCoiIiODUqVP07NmTtWvX0qJFC5o0acKJEye816jqvNqklfBFREQEgLySwkrDF3hGwvJLCmvkdc6cOcOOHTsYOnQoAG63m1atWgGQkJDAxIkTGTVqFKNGjbrstT799FMWLVoEwN13303z5s29fX//+99ZvHgxAN988w379++vNFhV9zw7KYCJiIgEqMjQcA4X52FVEsMMhsjQ8Bp5Hcuy6NKlCxs2bLio78MPP+TTTz9l6dKl/Pd//zc7d+687PWMuXhtwjVr1pCSksKGDRto3LgxAwcO5PTp01d9nt00B0xERCRA9YtNwOmofJ0/Z5CDvrEJNfI6ISEhHD161BvAysrK2LlzJ+Xl5XzzzTcMGjSIP/7xjxQWFnLixAmaNm3K8ePHK71W//79mTt3LgAfffQRx44dA6CoqIjmzZvTuHFj9uzZwxdffOH9muDgYMrKyi57nj9RABMREQlQMc2jGRCXSHCQE3N2nT+DITjIyYD2ide0FMX5goKC+M9//sOzzz5Lt27dSExMZP369bjdbh566CHi4+Pp3r07P/rRjwgPD+eee+5h8eLFlU7C/81vfsOnn35Kjx49WL58Oe3atQNg2LBhuFwuEhIS+PWvf03v3r29X/P44497b3Ve6jx/okn4IiIidUx1J+Gfc24dsPySQlrUwDpgcjFNwhcREZEKYppHK3D5Gd2CFBEREfExBTARERERH1MAExEREfExBTARERERH1MAExEREfExBTARERG5Ivn5+SQmJpKYmEh0dDTXX3+9t11aWlojr/HZZ5/RpUsXEhMTOXXqVI1cE+APf/hDhXafPn1q7NpXQuuAiYiI1DFXug5YbapsY22Xy4XTeW0rXT3xxBP06tWLqVOnXmuJFVy4cXdNudJ1wDQCJiIiItdsypQp/PjHP2bQoEE8++yzTJ8+nT//+c/e/q5du5KZmQnAnDlzuPXWW0lMTGTatGm43e4K13r99df597//ze9+9zsmTpzImjVrGDFihLf/qaee4q233gIgNjaW3/zmN/To0YP4+Hj27NkDwIkTJ5g6dSrx8fEkJCSwcOFCfv7zn3Pq1CkSExOZOHEi4Alk4NnP8qc//Sldu3YlPj6e+fPnA569JQcOHMgDDzxA586dmThxIjUxeKWFWEVEROqwyZMn19q1Z82adUXn79u3j5SUFBwOB9OnT6/0nN27dzN//nw+//xzgoODefLJJ5k7dy6TJk3ynvPYY4+xbt06RowYwQMPPMCaNWsu+bqRkZFs3bqVf/3rX/z5z3/m9ddf57//+78JCwsjPT0dgGPHjnH//ffz0ksvsW3btouusWjRIrZt28b27dvJy8ujZ8+e9O/fH4Avv/ySnTt30rp1a/r27cvnn39Ov379ruh7cyEFMBEREakRY8aMwVHF5t/nrFy5ki1bttCzZ08ATp06xXXXXXdNrzt69GgAbrnlFhYtWgRASkoK8+bN857TvHnzS15j3bp1jB8/HofDQVRUFAMGDGDz5s00a9aMW2+9lTZt2gCQmJhIZmamApiIiIj4h9DQUO+fnU4n5eXl3vbp06cBz62+yZMn8z//8z/Vvm5V1zonJCQEAIfDgcvl8r6OMabar3Gp24rnrn/ha1wLBTAREZE67EpvE/pKbGwsH3zwAQBbt24lIyMDgMGDBzNy5Eh+9KMfcd1111FQUMDx48eJiYmp8loxMTHs2rWLM2fOcPr0aVauXHnZEajk5GReeukl/vrXvwKeW5DNmzcnODiYsrIygoODK5zfv39/Xn31VSZPnkxBQQGffvopf/rTn7xzymqaJuGLSI3JOpbD3C+X87d1/2bul8vJOpZjd0kiYpP777+fgoICEhMTefnll+nUqRMAN998M88//zzJyckkJCQwdOhQsrOzL3mttm3bMnbsWBISEpg4cSLdu3e/7Ov/6le/4tixY3Tt2pVu3bqxevVqAB5//HHvdc533333kZCQQLdu3bjjjjv44x//SHR07W1grmUoRKRGLN+3kbUZ23C5XViAweB0OBgQl0hyp152lycSUPxpGQrx0DIUIuJzWcdyWJuxjbKz4QvAwqLM7WLtwW0aCRMRuYDmgInINVuXmYbLXfmkVFe5m3WZacQ0r72hfL+Xmws706GoCMLCoEs8REXZXZWI2EgBLJDpTV98JK+kkKomM1hY5JcU+rIc/5K6GdLT4NxTU3n5kJUF8QmQ1NPe2kTENgpggUpv+uJDkaHhHC7Ow6okhhkMkaHhvi/KH+TmVvx7CIDlaaenQdt2+qVIrtqVLrMgtedq5tNrDlggutybfm6ubaVJYOoXm4CzisUXnUEO+sYm+LgiP7Ez/YK/h+dxuT39IlehYcOG5Ofn18iWOHJtLMsiPz+fhg0bXtHXaQQsEFXnTV+/dUsNimkezYC4RNYe3Iar3I2F5XkKMsjBgPaJ9Xf+V1HRJTqty/SLVK1NmzYcOnSIo0eP2l2K4AnE51bKry4FsECkN32xQXKnXtzYMoZ1mWnklxTSIjScfrEJ9Td8gWfuZV4+VDpDznj6Ra5CcHAwcXFxdpch10ABLBDpTV9sEtM8un4Hrgt1iffMvaxsRNrp8PSLSL2kOWCBqEu85829MnrTF/GdqCjPgy9OJ3BusrTxtOMTNBVApB7TCFggOvemn57mmfN1dl1ynA696Yv4WlJPz9OOWhJGRM6jABao9KYv4j+iovR3T0QqUAALZDXxpq/FXEVERGqcAphUTYu5ioiI1ApNwpfKaTFXERGRWqMAJpXTCt4iIiK1RgFMKqfFXEVERGqNAphULiyM79YtupAWcxUREbkWCmBSOS3mKiIiUmsUwKRyWsFbRESk1mgZCqmaFnMVERGpFQpgcmlawVtERKTG6RakiIiIiI9pBExEApO20RIRP6YAJiKBR9toiYif0y1IEQks2kZLROoA2wKYMaatMWa1MWa3MWanMeaHdtUiIgFE22iJSB1g5y1IF/ATy7K2GmOaAluMMSssy9plY00iPpV1LId1mWnklRQSGRpOv9gEYppH211W3aZttESkDrAtgFmWlQ1kn/3zcWPMbuB6QAFM6oXl+zayNmMbLrcLCzhcnMeuIxkMiEskuVMvu8uru8LCPHO+sCrp1DZaIuIf/GIOmDEmFugObLS5FBGfyDqWw9qMbZSdDV8AFhZlbhdrD24j61iOrfXVadpGS0TqANsDmDGmCbAQ+D+WZRVX0v+4MSbVGJN69OhR3xcoUgvWZabhclc+T8lV7mZdZpqPKwog2kZLROoAW5ehMMYE4wlfcy3LWlTZOZZlzQBmACQlJVV2T0GkzskrKaz0Bhl4RsLySwp9WY7/udY1vLSNloj4OdsCmDHGAG8Auy3L+l+76hCxQ2RoOIeL87AqiWEGQ2RouO+L8hc1tYaXttESET9m5whYX+BhIN0Ys+3ssV9alrXMvpLErwXQyub9YhPYdSSDskpuQzqDHPSNTaj+xQLo+3LZNbzatqu7/20iIuex8ynIdXw3QUPk0gJsZfOY5tEMiEtk7cFtuMrdWFgYDM4gBwPaJ1Z/KYoA+75Uaw0vBTARCQDaikj8X4COiiR36sWNLWNYl5lGfkkhLa50HbBA/L5oDS8RqScUwM7Sgph+LIBHRWKaR1/9/2c++L6Ul5dTXFxMYWEhhYWFlJSUcObMGUpLSzlz5gxnzpzB7XbjcDgqfAQHB9OkSROaNWtG06ZNvZ8bNGhw6RfUGl4iUk8ogKEFMf2eRkUqV0PfF5fLxdGjR8nOziY7O5vDhw+TnZ1Nfn4+RUVFWFbNPXwcHh5OdHR0hY927doRERGBMcYzfy0rq/JgqTW8RCSA1PsAdv6CmOecvyDmjS1jNBJmN42KVO4qvi/l5eXk5uZy8OBBMjIyOHjwIF9//TVlZWW1Xi7gHUnbs2dPheNhYWF06NCB9u3b0yE8gvZ5R2kIeP7bjCd8aQ0vEQkg9T6AVWdBTAUwm2lUpHLV+L5YlsXhw4fZtWsXu3btYs+ePZw8efKKXqZJkyaEh4cTHh5O06ZNCQkJISQkhAYNGhASEoLD4aC8vByXy4Xb7cbtdlNaWsrx48c5fvw4xcXF3j+73e5KX6OoqIitW7eydevWs+U76HRdS+IjIkjo0J7r+96OidbfQxEJHPU+gGlBzDrg3Mrm6WmeuU0aFfGo4vtSYlmkhTRk++LF7Nq1i6Jq3IqMiIigVatWFT6uu+46wsPDCQ4OrpFy3W43eXl55OTkkJubS05ODt9++y2ZmZmcPn26wrkut5td2Tnsys5h/s5dRKxbT7du3ejduzedOnUiKMh3m3hofqiI1AZTk/M7altSUpKVmppao9ec++Vy0rIPVLkgZrdWHZnQPblGX1OuUiCtd1WTcnPJ3/gFW/fv58ujR9mTewR3eXmVpzdt2pT27dsTFxdH+/btad++PU2bNvVhwRWVl5dz+PBhDh48yFdffcX+/fv59ttvqzw/IiKCXr16cdttt9GuXTvP3LFacuH8UIPB6XBofqiIVIsxZotlWUmV9tX3AJZ1LIcZm5ZUuiBmcJCTx3uN1G+74peKi4vZuHEjGzZs4KuvvqryvNDQUG666SZuuukmbr75Zlq1alWroaUmFBQUsGPHDtLS0ti5c2eVt01bt27NHXfcQb9+/WjUqFGN1qD3BhG5Vgpgl7F838YqF8TUb7niT86cOUNqaiobNmxg586dlFcx0hUXF0f37t1JSEggJibGp7fsaprb7Wb//v1s3LiRzZs3c/z48YvOadiwIX369GHw4MG0adOmRl7XMzq+v4pHHAwJrToyUaPjInIJCmDVcG6ex1UtiClSyzIyMli7di0bNmy4aL4UgMPh4Oabb6Z79+50796diIgIG6qsfS6Xi507d/LFF1+wZcsWzpw5c9E5N910E3fffTddu3a9ppG+v637N98WH62yv02zljzTb+xVX19EAt+lAli9n4R/zjUtiClSC06dOsWGDRtYs2YNWVlZlZ7TqVMnbrvtNm699VaaNGni4wp9z+l00q1bN7p168apU6dYv349KSkpHD582HvO7t272b17Nx06dODee++lW7duVxXEtGG6iNQmjYCJ+JmcnBxSUlL47LPPKh3tatWqFf369aN3795ERkbaUKF/sSyLPXv2sHLlSrZs2XLRbdmYmBhGjhxJjx49riiIaQ6YiFwr3YIU8XOWZbFr1y6WL1/O9u3bL1p9Pjg4mJ49ezJw4EA6derk95Po7ZKfn8+HH37I2rVrcV2wPlqHDh2YMGECHTt2rPb1ND9URK6FApiIn3K73XzxxRcsW7aMQ4cOXdR/7im/Pn36EBoaakOFddOxY8dYtmwZq1evvmiV/1tvvZWxY8fSsmXLal1L80NF5GopgIn4mTNnzvDZZ5/x0UcfkZeXd1F/t27dSE5OpkuXLhrtugZFRUUsW7aMlJSUCiNiTqeToUOHcu+999K4cWMbKxSRQKYAJuInTp06RUpKCp988slFyymEhIRw++23M3ToUKK17U6NOnr0KAsWLGDjxo0VjoeHh/PQQw+RlJSkoCsiNU4BTMRmp06dYsWKFXz88ceUlJRU6GvSpAnJyckMHjy4XjzJaKcDBw7wzjvvXLRwbWJiIpMmTaJFixY2VSYigUgBTMQm50a8Pvroo4uCV4sWLRg+fDj9+/cnJCTEpgrrH8uy2LhxI++8806FfTIbNmzI/fffz5AhQ+r0wrUi4j8UwER8rLS0lJUrV/LBBx9w4sSJCn0tW7bk3nvvpU+fPjidWorPLiUlJSxYsIDVq1dXON6hQwemTZtGlPYZFZFrpAAm4iNut5vPPvuMJUuWUFBQUKFPwcs/7du3jzfffLPCYq4hISFMnDiR/v37a26YiFw1BTCRWmZZFqmpqSxcuJDs7OwKfZGRkYwcOVLBy4+VlZXx4YcfsnTpUtxut/d4jx49mDp1Ks2aNbOxOhGpqxTARGrRvn37mDdv3kUTu8PCwhg5ciQDBgxQ8KojMjIyePXVVyuE6LCwMB577DESEhJsrExE6iIFMJFakJOTw4IFC7jw/8lGjRpx9913k5ycrMn1ddCZM2eYP38+K1eurHD8nnvuYfTo0ZqgLyLVpgAmUoNOnDjBe++9x6pVqyrcrjq3uOeIESO0nEQA2L59O2+88UaFJyVvvvlmvv/97+uWpIhUiwKYSA1wuVysWrWKxYsXc/LkyQp9t912G/fff3+1t7eRuqG4uJhXXnmFnTt3eo9FRETwgx/84Ir2lBSR+kkBTOQapaWl8e6771Z4Ug6gSXQEXQbeyt29B2l/wABVXl7Oe++9x5IlS7zHHA4H48ePZ8iQIXpKUkSqpAAmcpVycnJ455132L59e4XjjiYNadqtLQ2uDyfIBOF0OBgQl0hyp142VSq1bfv27bz66qsVFtQdMGAAkyZN0kMWIlIpBTCRK3T69GmWLl3Kxx9/XGGeV0hICA1viiakY0uMo+Jk7OAgJ4/3GqmRsACWl5fHSy+9REZGhvdY586defrppzXvT0QuogAmUk3ntqmZN28ex44d8x43xnD77bdjboxkb/EhKvtbYzAktOrIxO7JvitYfK6srIyZM2eyfv1677GoqCh+9KMf0apVKxsrExF/c6kApuepRc46dOgQL7zwAi+//HKF8NWxY0emT5/Oo48+ynHOVBq+ACws8ksKfVKr2Cc4OJjHH3+cBx54wHssNzeX3/3udxUm64uIXIomLki9d+rUKd577z2WL19OeXm593hYWBjjxo2jT58+3onWkaHhHC7Ow6okhhkMkaHhvipbbGSM4Z577iE6OpoZM2ZQWlrKyZMn+fOf/8yjjz5Kv3797C5RRPycApjUW5ZlsWnTJt555x0KCwu9x4OCgkhOTmbkyJE0bty4wtf0i01g15EMytyui67nDHLQN1arpdcnPXv2pGXLlrz44osUFhZSXl7Oa6+9RnFxMXfddZfd5YmIH9McMH+Umws706GoCMLCoEs8REXZXVVAycnJYfbs2RfdMurcuTMPP/wwbdq0qfJrl+/byNqD23CVu7GwMBicQQ4GtNdTkPVVQUEB//u//8s333zjPTZs2DDGjRunlfNF6jFNwq9LUjdDehq4zo2wGHA6ID4BknraWlogKC0t5YMPPuDDDz/E5fpuFCssLIwHH3yQ2267rVrrOmUdy2FdZhr5JYW0CA2nX2yCnn6s50pKSvjb3/7G3r17vcf69OnDo48+qmUqROopBbC6IjcXln1wXvg6j9MJd43QSNg1SE9PZ/bs2Rw5csR7zBjDkCFDGD169EW3G0WuVGlpKa+88gpbtmzxHouPj+fpp5/WvqAi9ZACWF2xKgW++qqKTgMd2sMdQ3xaUiA4duwY7777Lhs3bqxwvEOHDkyePJmYmBibKpNAVF5ezqxZs1izZo332I033siPfvQjGjVqZF9hIuJzlwpgGhf3J+dt+nsx6zL9cqHy8nJWrlzJwoULOXXqlPd448aNGTt2LAMGDND8HKlxQUFBTJkyhbCwMO/2RXv37uXPf/4zP/nJTzTSKiKAAph/CQuDvHyoYplPwsJ8XVGdlZWVxZtvvllhxXLwzMl5cPBgwr7OhCWL9ZCD1ApjDKNHj6ZRo0bMmzcPgAMHDvDHP/6Rn/70p4SGhtpcoYjYTQHMn3SJh6ysKuaAOTz9ckmnT59m0aJFLF++nPNvr0dHRzN58mRuPlkCn6397nucl+/5nushB6kFw4cPx+l0MmfOHAAyMjJ44YUX+NnPfkbTpk1trk5E7KT7L/4kKsoTBJxO4NyTeMbTjk/QKM1lbN26lV/84hd88skn3vDldDq57777eP7557m5RYsLnjAFsDzt9DTPQxAiNWzo0KFMmTLF2/7666954YUXKC4utq8oEbGdRsD8TVJPaNtO64BdgYKCAt5++222bt1a4fjNN9/M5MmTiY4+uzzEzvTKRxcBXG5Pv77PUgsGDRqEw+Fg5syZWJbFoUOH+H//7//xi1/8Qpt4i9RTCmD+KCpKQaAaysvLSUlJYeHChZw+fdp7vGnTpowfP77CFkKAHnIQW/Xv3x+n08mMGTO8IexPf/oTzz77rCbmi9RDCmBSJ2VmZvLmm2+SmZlZ4Xj//v0ZN25c5aMKeshBbNanTx8sy+K1117DsiwyMzP5y1/+wk9/+lMaNmxod3ki4kMKYFKnnDp1ikWLFrFixYoKk+xbt27NlClTuPHGG6v+Yj3kIH6gb9++lJWV8eabbwKepyNffPFFfvKTn9CgQQObqxMRX1EAkzpjy5YtzJkzh4KCAu+x4OBg7r33Xu66667Lb/dy7iGH9DTPnC8sKmz1pNu+4iMDBw6ktLSUuXPnArBnzx7+/ve/88Mf/pDg4GCbqxMRX1AAE7+Xn5/PnDlzLppk36VLFyZPnkzUlQQnPeQgfiI5OZmysjL+/e9/A56tsl555RV+8IMfaIFgkXpAAUz8lsvlYvny5SxevJjS0lLv8aZNmzJhwoRqb5x9ET3kIH7i7rvvprS0lPfeew+A1NRUZs+ezeTJk6/u/20RqTMUwMQvHThwgLfeeotvvvmmwvEBAwYwduxYPbovAWPUqFGcPHmS5cuXA7B69WrCwsK47777bK5MRGqTApj4lZKSEhYsWMCaNWsqTLK//vrrmTJlCp06dbKxOpGaZ4xh/PjxHD9+nA0bNgDw3nvv0axZMwYPHmxzdSJSWxTAxC9YlsXnn3/OvHnzOH78uPd4gwYNGDVqFHfeeeflJ9mL1FFBQUE89thjnDhxgvT0dADefvttmjVrRs+e2iJLJBDpXzSx3aFDh5g9ezZ79+6tcLxbt248/PDDtGzZ0qbKRHzH6XTy1FNP8cILL5CRkYFlWbzyyis0adKEm266ye7yRKSGmfNv8/i7pKQkKzU11e4ypIacOXOGJUuW8PHHH+N2u73HIyIimDBhAklJSZqILPVOcXExv//978nJyQGgcePG/OpXv+L666+3uTIRuVLGmC2WZSVV2qcAJr5mWRapqam88847Fdb0cjgc3HnnnYwcOVKrgku9dvToUZ5//nkKCwsBiIyM5LnnniNMuzWI1CkKYOI3srOzefvtt9m5c2eF4506dWLy5Mm0adPGpspE/EtWVha///3vOXPmDADt27fn5z//OSEhITZXJiLVpQAmtqvqdmPTpk0ZN24c/fr10+1GkQts27aNv/71r94ngpOSkrRQq0gdcqkApkn4Uqssy+KLL75g/vz5HDt2zHvcGMPgwYMZPXo0oaGhNlYo4r8SExOZOHEic+bMATwLtS5YsIBx48bZXJmIXCsFMKk1WVlZzJkzh3379lU43rFjRyZNmkRMTIxNlYnUHUOHDuXIkSPehVqXLVvGddddx6BBg2yuTESuhQKY1Ljjx4+zcOHCixZTDQsLY8yYMfTt21e3UESuwPjx4zl69Chffvkl4FkjLDo6WstTiNRhmgMmNcblcpGSksKSJUs4efKk97jD4SA5OZmRI0fSqFEjGysUqbvOnDnDH/7wBzIzMwFo0qQJ06dP1zp5In5Mk/ClVlmWxdatW5k/fz65ubkV+uLj45kwYQKtW7e2qTqRwFFQUMD06dMpKioCoE2bNvz617/Wsi0ifkoBTGpNVlYW77zzDnv27KlwPCoqivHjx5OYmKinG0Vq0IEDB/if//kfXC4XALfccgtPPfWUbuuL+CE9BSk1Li8vj4ULF7Jhw4YK87waN27MyJEjGTJkiPZuFKkFHTt2ZMqUKbz++usAbNmyhffee4/Ro0fbXJmIXAn9C1nTcnNhZzoUFUFYGHSJh6gou6uqMSUlJbz//vukpKRQVlbmPR4UFMTgwYMZNWoUTZo0sbFCkcB3++2388033/DJJ58AsGTJEtq2bVu9jbsD/D1KpK5QAKtJqZshPQ3O3hogLx+ysiA+AZKq8cbox0pLS0lJSeGDDz6gpKSkQl9iYiLjxo3TPC+RWpZ1LId1mWnklRQScWMEHTJv4Ku9+wGYMWMG0dHRtG3btuoLBPB7lEhdowBWU3JzK76xAWB52ulp0LZdnfwt0+Vy8emnn7JkyRLvvnTntG/fnnHjxtG5c2d7ihOpR5bv28jajG243C4s4HBxHkHxzWl6NIzjBUWUlpbyj3/8g+nTp9O4ceOLLxCg71EidZUCWE3ZmX7BG9t5XG5Pfx16cysvL2f9+vW89957HD16tEJfVFQUY8aMISkpSRPsRXwg61gOazO2Ueb+7j3GwsLtgMa94ziTsovS0lJyc3OZMWMGzzzzzMWT8gPsPUqkrlMAqylnHwuvnHWZfv9RXl7Opk2bWLJkCYcPH67QFx4ezr333suAAQM0wV7Eh9ZlpuFyVx6eTNMQOg+5lbRl6wD48ssv+eCDD7j33nsrnhgg71EigUL/itaUsDDPfAoqW9bDePr9mNvtZuPGjSxdupTs7OwKfaGhoYwYMYLBgwcTEhJiU4Ui9VdeSWGl7yzgGQkLvj6MYcOG8fHHHwOwaNEi4uLiiI+P/+7EOv4eJRJoFMBqSpd4z2TWyob4nQ5Pvx9yuVx88cUXLF269KJFVBs2bMiwYcMYNmyYVrAXsVFkaDiHi/OwKglPBkNkaDhjxw4mMzOTPXv2YFkWL7/8Mr/97W+/Wym/jr5HiQQqrdxXU6KiPE8SOZ3AuXlRxtOOT/C7uRVnzpxh+fLlPPvss7z22msVwlejRo0YOXIkf/nLX7jvvvsUvkRs1i82AafDUWmfM8hB39gEHA4HTz75JOHh4YBnyZh//OMflJaWek6sY+9RIoFOK+HXND9fY+f48eOkpKSQkpLCiRMnKvQ1btyY5ORkkpOTCQ0NtalCEanM8n0bWXtwG65yNxYWBoMzyMGA9okkd+rlPe/AgQP84Q9/wO12A3DHHXcwefLk7y7k5+9RIoFEWxEJhw8fZvny5Xz++eff/UZ8VtOmTRk6dChDhw6t/PF1EfEL59YByy8ppEVoOP1iE4hpHn3ReStWrGDOnDne9pNPPkmvXr0uOk9Eape2IqqnysvLSUtLY8WKFezYseOi/sjISIYPH87tt9+uyfUidUBM8+hKA9eFhgwZwt69e9m8eTMAM2fOJCYmhujoy3+tiPiGAlgAOnHiBOvWrWPVqlUXTawHaNeuHXfddRe33norjirmlYhI3WWM4ZFHHiErK4sjR45w+vRp/vnPf/LrX/+aBg0a2F2eiKBbkAHDsiz27t3LmjVr2Lx5M64LnnQyxtC9e3eSk5Pp3LmzFlAVqQeysrL43e9+530/GDRoEFOmTLG3KJF6RLcgA1hBQQEbNmzgs88+u2j9LvBMrO/fvz9Dhgz57nF0EakXYmJimDBhArNnzwZg9erV3Hjjjdx22202VyYiCmB10KlTp0hNTWX9+vXs3r2bykYx4+LiGDhwIL1796Zhw4Y2VCki/uCOO+5g7969bNy4EYC33nqLuLg4zQcTsZkCWB1x+vRptm/fTmpqKtu2bbvoSUbwLJzap08fBg4cSExMjA1Vioi/McYwdepUMjMzyc3N5fTp0/zrX//i17/+NcHBwXaXJ1JvKYD5sZKSEr788ktSU1PZsWMHZWVlF51jjOHmm2+mT58+JCUlabRLRC7SqFEjfvCDH3jng2VlZfGf//yH8ePH212aSL2lAOZHysvLycrKIi0tjfT0dA4cOFDp7UWAtm3b0qdPH3r37k1ERISPKxWRuiYmJoZx48Yxd+5cAD7++GO6dOlCQkKCzZWJ1E8KYDayLIvc3Fz27NnD3r172bFjB8XFxVWe37ZtW5KSkkhKSqJNmzY+rFREAsHQoUPZsWMH27dvB+C1117j+eefJ0wbcYv4nJah8KEzZ87wzTffkJGRwb59+9i7dy9FRUVVnm+MIS4ujqSkJG655RZNmhWRa1ZcXMyvfvUr73tPfHw8P/7xjwkK0tbAIjXNb5ehMMYMA/4GOIDXLct6wc56aoplWRw7dozs7GwOHz5MZmYmmZmZHD58mPLy8kt+bdOmTenatSsJCQl07dqVZs2a+ahqEakPmjVrxrRp0/jTn/6EZVmkp6fzySefMHz4cLtLE6lXbAtgxhgH8E9gKHAI2GyMWWpZ1i67aqqu8vJyTp48SUFBAQUFBeTn53s/5+TkkJ2dzenTp6t1rcaNG9OpUyduvPFGOnfuTGxsrH4TFZFa1aVLF+666y4+/PBDABYsWEDnzp2Ji4uzuTKR+sPOEbBbgQOWZR0EMMbMA0YCtgSwo0ePsmzZMlwuF263u8LnsrIyTp48yYkTJzh58iQnT56scnL8pRhjiI6OJjY2lvbt29O5c2fatGmjwCUiPjd69Gh2797NwYMHcbvdvPrqq/z2t7/VvrAiPmJnALse+Oa89iGgl021cOLECVatWlVj12vcuDGtW7emVatWtGnThtjYWGJiYmjUqFGNvYaIyNVyOp088cQTPPfcc5w+fZrs7Gzmz5/PpEmT7C5NpF6wM4BVthnhRcNKxpjHgcfBs4l0bXE6r+xb0ahRI8LDw4mIiKBFixZEREQQERFBVFQUrVq1olmzZtpvUUT8WlRUFBMnTuSNN94AYOXKlXTr1o1u3brZXJlI4LMzgB0C2p7XbgMcvvAky7JmADPA8xRkbRXTvHlzHn74YZxOJ06nE4fD4f2z0+mkcePGhIaGEhoaSuPGjXE4HLVVioiIz9x+++18+eWXbN26FYA33niD559/Xg8AidQyOwPYZuAGY0wc8C3wIDDBrmKaNGnCkCFD7Hp5ERFbGGN45JFH+OqrrygqKqKoqIg333yTZ555RqP4IrXIttnflmW5gKeAT4DdwL8ty9ppVz0iIvVV06ZNeeyxx7ztrVu38umnn9pYkUjgs/XxO8uyllmW1cmyrA6WZf3ezlpEROqzhISECncB5s6dS25uro0ViQQ2rX8gIiIAjB07llatWgGenTtmzJhx2cWjReTqKICJiAgAISEhPPHEE96HjA4cOMBHH31kc1UigUkBTEREvGJjYxk5cqS3vWjRIg4dOmRjRSKBSQFMREQquPvuu73bErlcLmbMmIHL5bK5KpHAogAmIiIVOJ1OHn/8cYKDgwHIyspi6dKlNlclElgUwERE5CKtW7fmgQce8Lbff/99Dh48aGNFIoFFAUx8IzcXVqXA4oWez3q8XcTvJScnc+ONNwJQXl7Oa6+9Rmlpqc1ViQQGBTCpfambYdkH8NVXkJcHXx30tFM3212ZiFxCUFAQjz32GCEhIQAcPnyYhQsX2lyVSGBQAJPalZsL6WlQYQKv5Wmnp2kkTMTPXXfddYwfP97b/uSTT9i/f7+NFYkEBgUwqV070y8IX+dxuT39IuLXBg4cSNeuXQGwLIs33nhDtyJFrpECmNSuoqJLdFqX6RcRf3Buw+6GDRsCkJ2dzeLFi22uSqRuUwCT2hUWBpgqOs3ZfhHxdy1atKhwK/Kjjz7iwIEDNlYkUrcpgEnt6hIPTkflfU6Hp19E6oQBAwbQpUsXwHMr8vXXX9etSJGrpAAmtSsqCuITwOnku5Ew42nHJ3j6RaRO0K1IkZqjACa1L6kn3DUCOrSHyEjP57tGeI6LSJ0SGRnJuHHjvO2PPvqIr776ysaKROomBTDxjagouGMI3He/57NGvkTqrIEDB3LzzTcD392KLCsrs7kqkbpFAUxERK5IUFAQjzzySIUFWt9//32bqxKpWxTARETkirVs2ZIxY8Z42x988AFff/21jRWJ1C0KYCIiclUGDx7MDTfcAIDb7WbmzJm43W6bqxKpGxTARETkqpy7Fel0OgHIyMjgk08+sbkqkbpBAUxERK5a69atGTVqlLe9aNEicrXHq8hlKYCJiMg1GT58OO3atQOgrKyMmTNnUl5ebnNVIv5NAUxERK6J0+nk0UcfJSjI80/Knj17WLt2rc1Vifg3BTAREblmsbGxDB8+3NueN28eBQUFNlYk4t8UwEREpEaMGjWK6OhoAE6fPs3bb79tc0Ui/ksBTEREakSDBg2YOnWqt71161ZSU1NtrEjEfymAiYhIjencuTMDBgzwtt9++21KSkpsrEjEPymAiYhIjRo3bhxhYWEAFBYWsmDBApsrEvE/CmAiIlKjQkNDefjhh73t1atXs3fvXhsrEvE/CmAiIlLjkpKS6N69u7f95ptvUlZWZmNFIv5FAUxERGqcMYZJkybRsGFDALKzs3n//fdtrkrEfyiAiYhIrYiIiGDMmDHe9gcffMC3335rY0Ui/kMBTEREas0dd9xBx44dAXC73bz11lvapkgEBTAREalFQUFBTJkyBYfDAcC+ffv47LPPbK5KxH4KYCIiUqvatm3LsGHDvO358+dTXFxsY0Ui9lMAExGRWjdy5EhatmwJQElJCe+8847NFYnYSwFMRERqXUhICJMnT/a2N2zYwI4dO2ysSMReCmAiIuIT8fHx9O7d29ueNWsWpaWl137h3FxYlQKLF3o+5+Ze+zVFapkCmIiI+MyECRNo3LgxAEeOHGHp0qXXdsHUzbDsA/jqK8jLg68Oetqpm2ugWpHaowAmIiI+ExYWxrhx47ztZcuWcejQoau7WG4upKeBy3XeQcvTTk/TSJj4NQUwERHxqf79+3PDDTcAnrXBZs2adXVrg+1MvyB8ncfl9vSL+CkFMBER8anK1gZbt27dlV+oqOgSndZl+kXsVWUAM8YsM8bE+rAWERGpJ9q0aVNhbbB58+Zx/PjxK7tIWBhgqug0Z/tF/NOlRsDeApYbY/6vMSbYR/WIiEg9MXLkSCIjIwHP2mDz5s27sgt0iQeno/I+p8PTL+KnqgxglmX9G+gONANSjTH/ZYz58bkPn1UoIiIBKSQkhIcfftjbXrduHXv27Kn+BaKiID4BnE6+GwkznnZ8gqdfxE9dbg5YGVAChABNL/gQERG5JomJiSQlJXnbs2bNwlXVxPrKJPWEu0ZAh/YQGen5fNcIz3ERP+asqsMYMwz4X2Ap0MOyrJM+q0pEROqNiRMnsmPHDk6fPs3hw4f56KOPuOeee6p/gaiogBrtyjqWw7rMNPJKCokMDadfbAIxzaPtLktq2KVGwP4vMMayrJ8rfImISG2JiIjg/vvv97aXLFnCkSNHbKzIPsv3bWTGpiWkZe/n2+KjpGUfYMamJSzft9Hu0qSGXWoO2O2WZe30ZTEiIlI/DR48mJiYGADKysqYM2cOlmXZXJVvZR3LYW3GNsrcLs79l1tYlLldrD24jaxjObbWJzVL64CJiIjtHA4HkydPxhjPZPrt27ezZcsWm6vyrXWZabjclc9/c5W7WZeZ5uOKpDYpgImIyJWrhQ2wO3TowMCBA73tuXPncvr06Wu+bl2RV1JIVWN+Fhb5JYW+LEdqmQKYiIhcmVrcAHvMmDE0a9YMgIKCAhYvXnzN16wrIkPDMVUsLGswRIaG+7YgqVUKYCIiUn21vAF2aGgoDz74oLe9fPlyvv7662u6Zl3RLzYBp6PyhWWdQQ76xib4uCKpTQpgIiJSfT7YALtPnz507twZgPLy8qvfrLuOiWkezYC4RIKDnN6RMIMhOMjJgPaJWooiwCiAiYhI9flgA2xjDJMmTfJu1n3gwIGr26y7Dkru1IvHe40koVVH2jRrSUKrjjzeayTJnXrZXZrUsCoXYhUREblIWBjk5UOl08VrbgPs66+/nmHDhvHhhx8CMH/+fLp3707TpoG/EUtM82iNdtUDGgETEZHq8+EG2Odv1n3ixAkWLFhQY9e+SC081SlyKQpgIiJSfT7cADskJISHHnrI2167di0HDhyoset71eJTnSJVUQATEZEr48MNsLt370737t297VmzZuF2u2vuBWr5qU6RqmgOmIiIXDkfboA9ceJEdu7cSWlpKV9//TUrV64kOTm5Zi5enac6A2ijb/EfGgETERG/1rJlS+69915ve+HChRQWFtbMxX3wVKdIZRTARETE7w0bNoxWrVoBcPr0aebNm1czFw4LgypWn6/JpzpFLqQAJiIifi84OJiHH37Y296wYQO7du269gv78KlOkfMpgImISJ3QpUsXevX6bkHS2bNn46pq/lZ1+fCpTpHzKYCJiEidMX78eBo2bAhAdnY2H3300bVf1IdPdYqcowAmIiJ1RvPmzRk9erS3vXTpUvLy8q79wlFRcMcQuO9+z2eNfEktUwATEZE6ZciQIbRt2xaA0tJS5s6da3NFIldOAUxEROoUh8PBpEmTvO2tW7eyfft2GysSuXIKYCIiUud06tSJ22+/3dt+++23KS0ttbEikSujACYiInXS2LFjCQ0NBeDo0aN8+OGHNlckUn0KYCIiUic1a9aMBx54wNv+8MMPydXejVJHKICJiEidNXDgQOLi4gAoKyvj7bffxrIsm6sSuTwFMBERqbOCgoKYPHkyxngWUU1PT2fLli02VyVyeQpgIiJSp8XFxTFo0CBve+7cuZw5c8bGikQuTwFMRETqvAceeICmTZsCUFBQwJIlS2yuSOTSFMBERKTOCw0NZdy4cd72xx9/zOHDh22sSOTSFMBERCQg9O3blxtuuAEAt9vN7NmzNSFf/JYCmIiIBISgoCAmTZpEUJDnn7bdu3fzxRdf2FyVSOUUwEREJGC0a9eOIUOGeNvz5s3j1KlTNlYkUjkFMBERCSijR48mPDwcgMLCQhYtWmRvQSKVUAATEZGA0qhRIx588EFvOyUlha+//trGikQupgAmIiIBp3fv3tx0000AlJeXM3v2bMrLy22uSuQ7tgQwY8yfjDF7jDFpxpjFxphwO+oQEZHAZIxh0qRJOBwOAPbv38/nn39uc1Ui37FrBGwF0NWyrARgH/ALm+oQEZEA1bp1a4YNG+Ztz58/nxMnTthYkch3bAlglmUttyzLdbb5BdDGjjpERCSwjRw5khYtWgBw/Phx/vOf/9hckYiHP8wBewT4yO4iREQk8ISEhDBx4kRve82aNXz11Vc2ViTiUWsBzBiTYozZUcnHyPPO+b+AC5h7ies8boxJNcakHj16tLbKFRGRANWjRw+6desGgGVZzJo1SxPyxXa1FsAsyxpiWVbXSj6WABhjJgMjgInWJfaKsCxrhmVZSZZlJbVs2bK2yhURkQBljOGhhx4iODgYgKysLFavXm1zVVLf2fUU5DDgWeBey7JO2lGDiIjUH9dddx333HOPt/2f//yHoqIiGyuS+s6uOWAvAU2BFcaYbcaYV2yqQ0RE6onhw4cTFRUFwMmTJ5k3b57NFUl9ZtdTkB0ty2prWVbi2Y8n7KhDRETqjwYNGvDwww972+vXr2fPnj02ViT1mT88BSkiIuIT8fHx9OzZ09ueNWsWLpfrEl8hUjsUwEREpF6ZMGECDRs2BODw4cN88sknNlck9ZECmIiI1CsRERHcd9993vZ7771HXl6ejRVJfaQAJiIi9c7QoUNp27YtAKWlpcydW+VylCK1QgFMRETqHYfDwaRJk7ztrVu38uWXX9pYkdQ3CmAiIlIvderUif79+3vbc+bM4cyZMzZWJPWJApiIiNRbY8eOJTQ0FIC8vDzef/99myuS+kIBTERE6q2mTZsyduxYb3vZsmUcPnzYxoqkvlAAExGReq1///507NgRALfbzaxZs7jEFsUiNUIBTERE6rWgoCAmT55MUJDnn8Q9e/awfv16m6uSQKcAJiIi9V67du248847ve13332XEydO2FiRBDoFMBEREWDUqFFEREQAcPz4cRYsWGBzRRLIFMBERESAhg0bVtise82aNezfv9/GiiSQKYCJiIic1aNHDxITE73tt956S5t1S61QABMRETnPww8/TIMGDQA4dOgQK1assLkiCUQKYCIiIueJjIxk1KhR3vbixYvJz8+3ryAJSApgIiIiF7jzzjtp06YNAGfOnGH27NlaG0xqlAKYiIjIBZxOJ5MnT/a2t23bxpYtW2ysSAKNApiIiEglOnXqxMCBA73tOXPmcOrUKfsKkoCiACYiIlKFsWPHEhYWBsCxY8dYuHChzRVJoFAAExERqUJoaCgTJkzwtlNSUjh48KCNFUmgUAATERG5hF69ehEfHw+AZVm89dZbuN1um6uSuk4BTERE5BKMMUyaNIng4GAAsrKytDaYXDMFMBERkcu47rrrKqwNtmjRIvLy8uwrSOo8BTAREZFqGDZsmNYGkxqjACYiIlINTqeTqVOnYowBYPv27WzatMnmqqSuUgATERGppo4dO3LHHXd423PmzOHEiRM2ViR1lQKYiIjIFRgzZgzNmzcHoLi4mPnz59tckdRFCmAiIiJXoFGjRkyaNMnb/vTTT9m9e7eNFUldpAAmIiJyhXr06EFSUpK3/eabb1JaWmpjRVLXKICJiIhchYceeojGjRsDkJuby9KlS22uSOoSBTAREZGr0Lx5c8aOHettL1u2jK+//trGiqQuUQATERG5SgMGDKBTp04AuN1uZs6cqW2KpFoUwERERK5SUFAQU6dOxel0ApCRkcHy5cttrkrqAgUwERGRa9C6deuLtinKzc21ryCpExTARERErtHw4cNp164dAKWlpcycOVPbFMklKYCJiIhcI6fTyaOPPkpQkOef1T179rB27VqbqxJ/pgAmIiJSA2JjYxk+fLi3PW/ePAoKCmysSPyZApiIiEgNGTVqFFFRUQCcOnWKWbNm6VakVEoBTEREpIY0aNCARx55xNvetm0bX3zxhY0Vib9SABMREalBnTt35o477vC258yZQ1FRkY0ViT9SABMREalhY8eOpUWLFgCcOHGC2bNn61akVKAAJiIiUsMaNWpU4VZkamoqmzZtsrEi8TcKYCIiIrWga9euDBgwwNuePXs2xcXFNlYk/kQBTEREpJaMHz+eiIgIwHMrUk9FyjkKYCIiIrVEtyKlKgpgIiIitSg+Pr7Crci3335btyJFAUxERKS2nX8r8vjx47oVKQpgIiIita2yW5Hr16+3sSKxmwKYiIiID8THxzNo0CBve86cOdor8gpkHcth7pfL+du6fzP3y+VkHcuxu6RrogAmIiLiIw8++CAtW7YE4OTJk7z++uu6FVkNy/dtZMamJaRl7+fb4qOkZR9gxqYlLN+30e7SrpoCmIiIiI80bNiQxx9/HGMMADt37mTVqlU2V+Xfso7lsDZjG2VuF+eiqoVFmdvF2oPbrnwkLDcXVqXA4oWez7m5NV5zdSiAiYiI+FCnTp0YPny4tz1v3jxycur27bTatC4zDZfbVWmfq9zNusy06l8sdTMs+wC++gry8uCrg5526uYaqrb6FMBERER87L777uP6668HoLS0lNdeew23221zVf4pr6SQqm7SWljklxRW70K5uZCeBq7zw5zlaaen+XwkTAFMRETExxo0aMC0adNwOBwAHDhwgGXLltlclX+KDA3HYCrtMxgiQ8Ord6Gd6ReEr/O43J5+H1IAExERsUFMTAwjR470thcvXkxGRoaNFfmnfrEJOM8G1Qs5gxz0jU2o3oWKii7RaV2mv+YpgImIiNhkxIgRdOjQAQC3280rr7zCmTNnbK7Kv8Q0j2ZAXCLBQU7vSJjBEBzkZED7RGKaR1fvQmFhUMVIGpiz/b6jACYiImITh8PBtGnTaNiwIQA5OTm8++67Nlflf5I79eLxXiNJaNWRNs1aktCqI4/3Gklyp17Vv0iXeHBWPpKG0+Hp9yGnT19NREREKoiKimLixIm88cYbAKxevZqEhAR69Ohhc2X+JaZ5dPVHuyoTFQXxCWcn4rsBCzCe8BWf4On3IQUwERERm91+++2kpaWxebNnOYSZM2fSvn17wsPD7S0s0CT1hLbtPBPui4o8tx27xPs8fIFuQYqIiNjOGMOUKVNo3rw54Nmw+/XXX6e8vNzmygJQVBTcMQTuu9/z2YbwBQpgIiIifqFJkyYVVslPT08nJSXF5qqktiiAiYiI+Imbb76ZYcOGedvz588nKyvLxoqktiiAiYiI+JH777+fmJgYAFwuF//85z85deqUzVVJTVMAExERe/jJpsj+Jjg4mCeffJKQkBAAcnNzmT17ts1VSU1TABMREd/zo02R/VF0dDSTJ0/2ttevX8+6detsrEhqmpahEBER37rcpsht29n2ZNq1yjqWw7rMNPJKCokMDadfbMJVr13Vt29fdu3a5Q1es2bNon379rRu3bomSxabaARMRER8y882Ra4py/dtZMamJaRl7+fb4qOkZR9gxqYlLN+38aqvOWnSJFq1agVAaWkp//rXvygtLa2pksVGCmAiIuJbfrYpck3IOpbD2oxtlLldWGePWViUuV2sPbiNrGM5V3XdkJAQnnzySYKDgwH45ptvtFVRgFAAExER3/KzTZFrwrrMNFzuykf1XOVu1mWmXfW127Vrx4QJE7ztVatWsWHDhqu+nvgHBTAREfEtP9sUuSbklRR6R74uZGGRX1J4TdcfNGgQPXv29LZnzpzJoUOHrumaYi8FMBER8a1zmyI7nXw3EmY8bRs2Ra4JkaHhmCpG9QyGyNDwa7q+MYZHH32U6GjPhP7S0lJeeuklrQ9WhymAiYiI7yX1hLtGQIf2EBnp+XzXCM/xOqhfbAJOR+Wjes4gB31jE675NRo1asTTTz9NgwYNAMjOzmbmzJlYVlVjb+LPFMBERMQefrIpck2IaR7NgLhEgoOc3pEwgyE4yMmA9olXvRTFhdq0acPUqVO97U2bNrFixYoaubb4ltYBExERqQHJnXpxY8sY1mWmkV9SSItrXAesKn369GH//v2sWrUKgHnz5hEXF8cNN9xQo68jtUsBTEREpIbENI+u8cBVmQkTJpCRkUFGRgZut5uXXnqJ3/72t4SHh9f6a0vN0C1IERGROiY4OJinnnqK0NBQAAoLC/nHP/5BWVmZzZVJdSmAiYiI1EGRkZF8//vfxxjPnLMDBw4we/ZsTcqvIxTARERE6qj4+HjGjRvnbX/66aesXLnSxoqkuhTARERE6rBhw4bRp08fb3vu3Lns3r3bxoqkOhTARERE6jBjDFOnTiUuLg6A8vJyXnrpJY4ePWpzZXIpCmAiIiJ1XIMGDXjmmWcIO7uP5okTJ/jb3/7G6dOnba5MqqIAJiIiEgAiIiJ4+umncZxdkf+bb77h5Zdfpry83ObKpDK2BjBjzH8ZYyxjTKSddYiIiASCG264gSlTpnjb27Zt45133rGvIKmSbQHMGNMWGAp8bVcNIiIigaZ///7cfffd3vaKFStYvny5jRVJZewcAXsR+BmgBUtERERq0AMPPMCtt97qbb/zzjts3brVxorkQrYEMGPMvcC3lmVtr8a5jxtjUo0xqXqiQ0RE5PKCgoL43ve+R8eOHQGwLIuXX36ZjIwMmyuTc2otgBljUowxOyr5GAn8X+C56lzHsqwZlmUlWZaV1LJly9oqV0REJKA0aNCAH/7wh5z7t7O0tJQXX3xRy1P4iVoLYJZlDbEsq+uFH8BBIA7YbozJBNoAW40xtb97qYiISD3SrFkzfvKTn3j3jCwqKuJPf/oTRUVFNlcmPr8FaVlWumVZ11mWFWtZVixwCOhhWVaOr2sREREJdK1ateKZZ57B6XQCkJuby1/+8hdOnjxpc2X1m9YBExERCXCdO3eusHF3VlYWf/3rXyktLbW5svrL9gB2diQsz+46REREAllSUhKPPPKIt713717+9a9/4Xa7bayq/rI9gImIiIhv9O/fn3HjxnnbX375JTNnztRq+TZQABMREalH7rrrrgoLta5bt445c+ZgWVqW05cUwEREROqZMWPG0L9/f2975cqVvP322wphPqQAJiIiUs8YY5g6dSq9evXyHlu5cqVGwnxIAUxERKQeCgoKYtq0aRVCWEpKCnPnzlUI8wGn3QWIiIiIPRwOB9OmTQNg48aNgGfzboCJEydijhyBnelQVARhYdAlHqKibKs3kCiAiYiI1GPnQphlWWzatAnwhDB3TjYPRzQn6NwyFXn5kJUF8QmQ1NPGigODbkGKiIjUcw6HgyeeeIKePb8LVqvSd/DKzl24vEtUWOByQXoa5ObaU2gAUQATERERbwjr3bu399jGvHxe3LWH0+cv1upye25LyjVRABMREREAnE4n06ZNY0hcnPfYjsJC/rhjJyfKys4esTxzwuSaKICJiIiIV1BQEA/d3pdR7dp6j311/AS/T9tBwZkzgPFMyJdrogAmIiIiFZiuCdzXPo5JHeIwZ48dPnWK325P46uTJZ6nIeWaKICJiIhIRVFREJ/A4LZteeLGTjiMJ4YVlpbxP9vT2XDwoM0F1n1ahkJEREQultQT2raj9850mkW24KXNWygpLaXM7eaVV17h22+/ZfTo0QQFaSznapi6tNptUlKSlZqaancZIiIi9U5ubi4vvvgi2dnZ3mM9evRg2rRpNGzY0MbK/JcxZotlWUmV9Sm2ioiIyGVFRUXx3HPPER//3fyvrVu38tvf/pZDhw7ZWFndpAAmIiIi1dK4cWN+/OMfM2zYMO+xw4cPM336dFavXq09JK+AApiIiIhUW1BQEOPHj+d73/seDRo0AKCsrIy33nqLf/7zn5SUlNhcYd2gACYiIiJXrF+/fkyfPp02bdp4j23evJnnnnuOAwcO2FhZ3aBJ+CIiInLVSktLeffdd1m1apX3mDGGIUOGcP/999OoUSMbq7OXJuGLiIhIrWjQoAGTJ0/mqaeeonHjxgBYlsWKFSv45S9/ybZt2+wt0E9pBExERERqRF5eHm+99Rbp6RU367711luZOHEi4eHh9hRmk0uNgCmAiYiISI2xLIsNGzbwzjvvcPz4ce/xhg0bMnz4cIYNG1Zv1g1TABMRERGfOn78OO+++y6ff/55hePNmjXj3nvvZdCgQTidgb0hjwKYiIiI2GLHjh3MnTuXw4cPVzjesmVLRo0aRe/evQM2iCmAiYiIiG3cbjeff/45ixcvpqCgoEJfWFgYgwYNYtCgQQE3R0wBTERERGxXWlrKypUref/99y9asNXhcHDrrbcyePBgOnToEBCbfCuAiYiIiN84efIkKSkprFy5ksLCwov6mzdvzi233MItt9zCjTfeiMPh8H2RNUABTERERGyTdSyHdZlp5JUUEhkaTr/YBGKaR+NyudiyZQvLly+vcvX8Jk2akJCQQMeOHYmLi6Nt27YEBwdfUz2lpaUUFBQQHR19Tde5HAUwERERscXyfRtZm7ENl9uFBRgMToeDAXGJJHfq5T0vIyODVatWsWXLlkvuJ+lwOGjXrh3t2rUjPDycsLAwwsLCaNasGU2aNMHtduNyuSgrK8PtdnPq1CmOHDlCTk4Oubm55OTkUFBQQIMGDZgxYwbGmFr7b1cAExEREZ/LOpbDjE1LKHO7LuoLDnLyeK+RxDSvOArldrvZt28fqampbNmyhWPHjtVafS+++CIRERG1dv1LBbDAfO5TREREbLcuMw1XJeELwFXuZl1m2kUBzOFwcNNNN3HTTTcxceJEMjIy2L9/PwcPHiQjI4MjR45cc11BQUFERkZy4sSJWg1gl6IAJiIiIrUir6SQqu6zWVjklxRe8uuDgoLo0KEDHTp08B47ceIEmZmZZGdnU1xcTFFREUVFRRQXF3Py5EmcTicOh4Pg4GCcTifBwcG0bNmSqKgooqOjiYqKomXLlravPaYAJiIiIrUiMjScw8V5WJXEMIMhMjT8iq/ZpEkTunbtSteuXWugQvvU/UU2RERExC/1i03AWcUSEs4gB31jE3xckf9QABMREZFaEdM8mgFxiQQHOTF4njY0GIKDnAxon3jR/K/6RLcgRUREpNYkd+rFjS1jWJeZRn5JIS3OWwesPlMAExERkVoV0zy63geuC+kWpIiIiIiPKYCJiIiI+JgCmIiIiIiPKYCJiIiI+JgCmIiIiIiPKYCJiIiI+JgCmIiIiIiPKYCJiIiI+JgCmIiIiIiPKYCJiIiI+JgCmIiIiIiPKYCJiIiI+JgCmIiIiIiPKYCJiIiI+JgCmIiIiIiPKYCJiIiI+JgCmIiIiIiPOe0uQERERGpBbi7sTIeiIggLgy7xEBVld1VylgKYiIhIoEndDOlp4HJ52nn5kJUF8QmQ1NPe2gTQLUgREZHAkptbMXwBYHna6WmefrGdApiIiEgg2Zl+Qfg6j8vt6RfbKYCJiIgEkqKiS3Ral+kXX1EAExERCSRhYYCpotOc7Re7KYCJiIgEki7x4HRU3ud0ePrFdgpgIiIigSQqyvO0o9PJdyNhxtOOT9BSFH5Cy1CIiIgEmqSe0Lad1gHzYwpgIiIigSgqSoHLj+kWpIiIiIiPKYCJiIiI+JgCmIiIiIiPKYCJiIiI+JgCmIiIiIiPKYCJiIiI+JgCmIiIiIiPKYCJiIiI+JgCmIiIiIiPKYCJiIiI+JgCmIiIiIiPKYCJiIiI+JgCmIiIiIiPKYCJiIiI+JixLMvuGqrNGHMUyKrll4kE8mr5NeTK6efif/Qz8U/6ufgf/Uz8ky9+LjGWZbWsrKNOBTBfMMakWpaVZHcdUpF+Lv5HPxP/pJ+L/9HPxD/Z/XPRLUgRERERH1MAExEREfExBbCLzbC7AKmUfi7+Rz8T/6Sfi//Rz8Q/2fpz0RwwERERER/TCJiIiIiIjymAnccYM8wYs9cYc8AY83O76xEwxsw0xhwxxuywuxbxMMa0NcasNsbsNsbsNMb80O6a6jtjTENjzCZjzPazP5Pf2l2TeBhjHMaYL40xH9hdi3gYYzKNMenGmG3GmFTb6tAtSA9jjAPYBwwFDgGbgfGWZe2ytbB6zhjTHzgBzLYsq6vd9QgYY1oBrSzL2mqMaQpsAUbp74p9jDEGCLUs64QxJhhYB/zQsqwvbC6t3jPG/BhIAppZljXC7nrEE8CAJMuybF2bTSNg37kVOGBZ1kHLskqBecBIm2uq9yzL+hQosLsO+Y5lWdmWZW09++fjwG7genurqt8sjxNnm8FnP/Tbtc2MMW2Au4HX7a5F/I8C2HeuB745r30I/aMicknGmFigO7DR5lLqvbO3urYBR4AVlmXpZ2K/vwI/A8ptrkMqsoDlxpgtxpjH7SpCAew7ppJj+g1SpArGmCbAQuD/WJZVbHc99Z1lWW7LshKBNsCtxhjdsreRMWYEcMSyrC121yIX6WtZVg9gOPCDs1NdfE4B7DuHgLbntdsAh22qRcSvnZ1ntBCYa1nWIrvrke9YllUIrAGG2VtJvdcXuPfsfKN5wB3GmDn2liQAlmUdPvv5CLAYzxQkn1MA+85m4AZjTJwxpgHwILDU5ppE/M7ZCd9vALsty/pfu+sRMMa0NMaEn/1zI2AIsMfWouo5y7J+YVlWG8uyYvH8e7LKsqyHbC6r3jPGhJ59eAhjTCiQDNjylL0C2FmWZbmAp4BP8Ewq/rdlWTvtrUqMMe8CG4AbjTGHjDGP2l2T0Bd4GM9v9NvOftxld1H1XCtgtTEmDc8vkyssy9KyByIXiwLWGWO2A5uADy3L+tiOQrQMhYiIiIiPaQRMRERExMcUwERERER8TAFMRERExMcUwERERER8TAFMRERExMcUwESkXjLGtDXGZBhjIs62m59tx9hdm4gEPgUwEamXLMv6BngZeOHsoReAGZZlZdlXlYjUF1oHTETqrbNbKm0BZgLfA7pbllVqb1UiUh847S5ARMQulmWVGWN+CnwMJCt8iYiv6BakiNR3w4FsoKvdhYhI/aEAJiL1ljEmERgK9AZ+ZIxpZW9FIlJfKICJSL1kjDF4JuH/H8uyvgb+BPzZ3qpEpL5QABOR+up7wNeWZa042/4X0NkYM8DGmkSkntBTkCIiIiI+phEwERERER9TABMRERHxMQUwERERER9TABMRERHxMQUwERERER9TABMRERHxMQUwERERER9TABMRERHxsf8PkidQJeZsdRgAAAAASUVORK5CYII=\n",
"text/plain": [
"