![](\"https://raw.githubusercontent.com/Harvard-IACS/2018-CS109A/master/content/styles/iacs.png\")
CS-109A Introduction to Data Science\n",
"\n",
"\n",
"## Lab 12: Building and Regularizing your first Neural Network \n",
"\n",
"**Harvard University**In class discussion : why do we care about Neural Nets?
\n",
"\n",
"**Buzzwords**: Linearity, Interpretability, Performance"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Part 2: Data Preparation\n",
"\n",
"### Tensors\n",
"\n",
"We can think of tensors as multidimensional arrays of real numerical values; their job is to generalize matrices to multiple dimensions. \n",
"\n",
"- **scalar** = just a number = rank 0 tensor ($a$ ∈ $F$,)\n",
"\n", " \n", "- **vector** = 1D array = rank 1 tensor ( $x = (\\;x_1,...,x_i\\;)⊤$ ∈ $F^n$ )\n", "
\n", " \n", "- **matrix** = 2D array = rank 2 tensor ( $\\textbf{X} = [a_{ij}] ∈ F^{m×n}$ )\n", "
\n", " \n", "- **3D array** = rank 3 tensor ( $\\mathscr{X} =[t_{i,j,k}]∈F^{m×n×l}$ )\n", "
\n", " \n", "- **$\\mathscr{N}$D array** = rank $\\mathscr{N}$ tensor ( $\\mathscr{T} =[t_{i1},...,t_{i\\mathscr{N}}]∈F^{n_1×...×n_\\mathscr{N}}$ ) <-- **Things start to get complicated here...**\n", " \n", "#### Tensor indexing\n", "We can create subarrays by fixing some of the given tensor’s indices. We can create a vector by fixing all but one index. A 2D matrix is created when fixing all but two indices. For example, for a third order tensor the vectors are\n", "
\n", "$\\mathscr{X}[:,j,k]$ = $\\mathscr{X}[j,k]$ (column),
\n", "$\\mathscr{X}[i,:,k]$ = $\\mathscr{X}[i,k]$ (row), and
\n", "$\\mathscr{X}[i,j,:]$ = $\\mathscr{X}[i,j]$ (tube)
\n", " \n", "#### Tensor multiplication\n", "We can multiply one matrix with another as long as the sizes are compatible ((n × m) × (m × p) = n × p), and also multiply an entire matrix by a constant. Numpy `numpy.dot` performs a matrix multiplication which is straightforward when we have 2D or 1D arrays. But what about > 3D arrays? The function will choose according to the matching dimentions but if we want to choose we should use `tensordot`, but, again, we **do not need tensordot** for this class. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Reese Witherspoon as a Rank 3 Tensor\n", "\n", "A common kind of data input to a neural network is images. Images are nice to look at, but remember, the computer only sees a series of numbers arranged in `tensors`. In this part we will look at how images are displayed and altered in Python. \n", "\n", "`matplotlib` supports only .png images but uses a library called `Pillow` to handle any image. If you do not have `Pillow` installed you can do this in anaconda:\n", "```\n", "conda install -c anaconda pillow \n", "\n", "OR \n", "\n", "pip install pillow\n", "```\n", "\n", "This image is from the dataset [Labeled Faces in the Wild](http://vis-www.cs.umass.edu/lfw/person/Reese_Witherspoon.html) used for machine learning training. Images are 24-bit RGB images (height, width, channels) with 8 bits for each of R, G, B channel. Explore and print the array." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The image is a:
\n", "noun: epoch; plural noun: epochs. A period of time in history or a person's life, typically one marked by notable events or particular characteristics. Examples: \"the Victorian epoch\", \"my Neural Netwok's epochs\".
\n", " \n", "-----------------------------------------------------------" ] }, { "cell_type": "code", "execution_count": 178, "metadata": {}, "outputs": [], "source": [ "# type your code here along with instructor\n" ] }, { "cell_type": "code", "execution_count": 179, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Train on 40199 samples, validate on 19801 samples\n", "Epoch 1/50\n", "40199/40199 - 4s - loss: 0.6735 - accuracy: 0.7575 - val_loss: 0.4546 - val_accuracy: 0.8375\n", "Epoch 2/50\n", "40199/40199 - 4s - loss: 0.4866 - accuracy: 0.8241 - val_loss: 0.3951 - val_accuracy: 0.8557\n", "Epoch 3/50\n", "40199/40199 - 4s - loss: 0.4428 - accuracy: 0.8395 - val_loss: 0.3739 - val_accuracy: 0.8642\n", "Epoch 4/50\n", "40199/40199 - 5s - loss: 0.4175 - accuracy: 0.8511 - val_loss: 0.3722 - val_accuracy: 0.8597\n", "Epoch 5/50\n", "40199/40199 - 5s - loss: 0.4018 - accuracy: 0.8535 - val_loss: 0.3595 - val_accuracy: 0.8684\n", "Epoch 6/50\n", "40199/40199 - 4s - loss: 0.3845 - accuracy: 0.8616 - val_loss: 0.3909 - val_accuracy: 0.8491\n", "Epoch 7/50\n", "40199/40199 - 6s - loss: 0.3726 - accuracy: 0.8639 - val_loss: 0.3609 - val_accuracy: 0.8683\n", "Epoch 8/50\n", "40199/40199 - 4s - loss: 0.3656 - accuracy: 0.8669 - val_loss: 0.3398 - val_accuracy: 0.8727\n", "Epoch 9/50\n", "40199/40199 - 4s - loss: 0.3553 - accuracy: 0.8699 - val_loss: 0.3348 - val_accuracy: 0.8793\n", "Epoch 10/50\n", "40199/40199 - 5s - loss: 0.3455 - accuracy: 0.8734 - val_loss: 0.3335 - val_accuracy: 0.8789\n", "Epoch 11/50\n", "40199/40199 - 4s - loss: 0.3428 - accuracy: 0.8736 - val_loss: 0.3289 - val_accuracy: 0.8804\n", "Epoch 12/50\n", "40199/40199 - 6s - loss: 0.3340 - accuracy: 0.8776 - val_loss: 0.3328 - val_accuracy: 0.8804\n", "Epoch 13/50\n", "40199/40199 - 5s - loss: 0.3283 - accuracy: 0.8792 - val_loss: 0.3223 - val_accuracy: 0.8857\n", "Epoch 14/50\n", "40199/40199 - 5s - loss: 0.3231 - accuracy: 0.8800 - val_loss: 0.3196 - val_accuracy: 0.8819\n", "Epoch 15/50\n", "40199/40199 - 5s - loss: 0.3165 - accuracy: 0.8827 - val_loss: 0.3286 - val_accuracy: 0.8830\n", "Epoch 16/50\n", "40199/40199 - 4s - loss: 0.3167 - accuracy: 0.8824 - val_loss: 0.3166 - val_accuracy: 0.8867\n", "Epoch 17/50\n", "40199/40199 - 6s - loss: 0.3106 - accuracy: 0.8849 - val_loss: 0.3464 - val_accuracy: 0.8752\n", "Epoch 18/50\n", "40199/40199 - 4s - loss: 0.3049 - accuracy: 0.8856 - val_loss: 0.3169 - val_accuracy: 0.8876\n", "Epoch 19/50\n", "40199/40199 - 5s - loss: 0.3018 - accuracy: 0.8877 - val_loss: 0.3285 - val_accuracy: 0.8854\n", "Epoch 20/50\n", "40199/40199 - 5s - loss: 0.3014 - accuracy: 0.8900 - val_loss: 0.3283 - val_accuracy: 0.8858\n", "Epoch 21/50\n", "40199/40199 - 3s - loss: 0.2971 - accuracy: 0.8879 - val_loss: 0.3240 - val_accuracy: 0.8874\n", "Epoch 22/50\n", "40199/40199 - 3s - loss: 0.2953 - accuracy: 0.8900 - val_loss: 0.3331 - val_accuracy: 0.8848\n", "Epoch 23/50\n", "40199/40199 - 3s - loss: 0.2918 - accuracy: 0.8892 - val_loss: 0.3301 - val_accuracy: 0.8836\n", "Epoch 24/50\n", "40199/40199 - 3s - loss: 0.2865 - accuracy: 0.8924 - val_loss: 0.3260 - val_accuracy: 0.8875\n", "Epoch 25/50\n", "40199/40199 - 3s - loss: 0.2816 - accuracy: 0.8946 - val_loss: 0.3181 - val_accuracy: 0.8893\n", "Epoch 26/50\n", "40199/40199 - 3s - loss: 0.2789 - accuracy: 0.8959 - val_loss: 0.3138 - val_accuracy: 0.8895\n", "Epoch 27/50\n", "40199/40199 - 3s - loss: 0.2820 - accuracy: 0.8958 - val_loss: 0.3324 - val_accuracy: 0.8891\n", "Epoch 28/50\n", "40199/40199 - 3s - loss: 0.2796 - accuracy: 0.8963 - val_loss: 0.3289 - val_accuracy: 0.8917\n", "Epoch 29/50\n", "40199/40199 - 3s - loss: 0.2771 - accuracy: 0.8977 - val_loss: 0.3287 - val_accuracy: 0.8867\n", "Epoch 30/50\n", "40199/40199 - 3s - loss: 0.2708 - accuracy: 0.8979 - val_loss: 0.3302 - val_accuracy: 0.8877\n", "Epoch 31/50\n", "40199/40199 - 3s - loss: 0.2694 - accuracy: 0.8979 - val_loss: 0.3264 - val_accuracy: 0.8907\n", "Epoch 32/50\n", "40199/40199 - 3s - loss: 0.2695 - accuracy: 0.8995 - val_loss: 0.3165 - val_accuracy: 0.8921\n", "Epoch 33/50\n", "40199/40199 - 3s - loss: 0.2656 - accuracy: 0.9011 - val_loss: 0.3341 - val_accuracy: 0.8877\n", "Epoch 34/50\n", "40199/40199 - 3s - loss: 0.2633 - accuracy: 0.9007 - val_loss: 0.3259 - val_accuracy: 0.8870\n", "Epoch 35/50\n", "40199/40199 - 3s - loss: 0.2648 - accuracy: 0.9012 - val_loss: 0.3300 - val_accuracy: 0.8906\n", "Epoch 36/50\n", "40199/40199 - 3s - loss: 0.2626 - accuracy: 0.9011 - val_loss: 0.3298 - val_accuracy: 0.8922\n", "Epoch 37/50\n", "40199/40199 - 3s - loss: 0.2601 - accuracy: 0.9032 - val_loss: 0.3201 - val_accuracy: 0.8920\n", "Epoch 38/50\n", "40199/40199 - 3s - loss: 0.2553 - accuracy: 0.9054 - val_loss: 0.3226 - val_accuracy: 0.8911\n", "Epoch 39/50\n", "40199/40199 - 3s - loss: 0.2562 - accuracy: 0.9050 - val_loss: 0.3234 - val_accuracy: 0.8909\n", "Epoch 40/50\n", "40199/40199 - 3s - loss: 0.2532 - accuracy: 0.9061 - val_loss: 0.3258 - val_accuracy: 0.8921\n", "Epoch 41/50\n", "40199/40199 - 3s - loss: 0.2521 - accuracy: 0.9068 - val_loss: 0.3406 - val_accuracy: 0.8885\n", "Epoch 42/50\n", "40199/40199 - 3s - loss: 0.2498 - accuracy: 0.9065 - val_loss: 0.3323 - val_accuracy: 0.8930\n", "Epoch 43/50\n", "40199/40199 - 3s - loss: 0.2531 - accuracy: 0.9049 - val_loss: 0.3318 - val_accuracy: 0.8911\n", "Epoch 44/50\n", "40199/40199 - 3s - loss: 0.2465 - accuracy: 0.9088 - val_loss: 0.3234 - val_accuracy: 0.8906\n", "Epoch 45/50\n", "40199/40199 - 3s - loss: 0.2453 - accuracy: 0.9073 - val_loss: 0.3406 - val_accuracy: 0.8888\n", "Epoch 46/50\n", "40199/40199 - 3s - loss: 0.2452 - accuracy: 0.9093 - val_loss: 0.3407 - val_accuracy: 0.8892\n", "Epoch 47/50\n", "40199/40199 - 3s - loss: 0.2421 - accuracy: 0.9096 - val_loss: 0.3328 - val_accuracy: 0.8924\n", "Epoch 48/50\n", "40199/40199 - 3s - loss: 0.2454 - accuracy: 0.9074 - val_loss: 0.3413 - val_accuracy: 0.8896\n", "Epoch 49/50\n", "40199/40199 - 3s - loss: 0.2400 - accuracy: 0.9103 - val_loss: 0.3315 - val_accuracy: 0.8929\n", "Epoch 50/50\n", "40199/40199 - 3s - loss: 0.2407 - accuracy: 0.9095 - val_loss: 0.3336 - val_accuracy: 0.8925\n", "CPU times: user 5min 56s, sys: 1min 44s, total: 7min 40s\n", "Wall time: 3min 1s\n" ] } ], "source": [ "%%time\n", "# the core of the network training\n", "history = model.fit(x_train, y_train, validation_split=0.33, epochs=50, \n", " verbose=2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Save the model\n", "\n", "You can save the model so you do not have `.fit` everytime you reset the kernel in the notebook. Network training is expensive!\n", "\n", "For more details on this see [https://www.tensorflow.org/guide/keras/save_and_serialize](https://www.tensorflow.org/guide/keras/save_and_serialize)" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [], "source": [ "# save the model so you do not have to run the code everytime\n", "model.save('fashion_model.h5')\n", "\n", "# Recreate the exact same model purely from the file\n", "#model = tf.keras.models.load_model('fashion_model.h5')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 5. Evaluate the model on the test set." ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [], "source": [ "# type your code here along with instructor\n" ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Test accuracy=0.8812\n", "Not bad!\n" ] } ], "source": [ "test_loss, test_accuracy = model.evaluate(x_test, y_test, verbose=0)\n", "print(f'Test accuracy={test_accuracy:.4f}')\n", "if test_accuracy>0.8: print(f'Not bad!')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 6. We learn a lot by studying History! Plot metrics such as accuracy. \n", "\n", "You can learn a lot about neural networks by observing how they perform while training. You can issue `kallbacks` in `keras`. The networks's performance is stored in a `keras` callback aptly named `history` which can be plotted. " ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "dict_keys(['loss', 'accuracy', 'val_loss', 'val_accuracy'])\n" ] } ], "source": [ "print(history.history.keys())" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "
![\"shoe\"](\"../fig/kanye_shoe.jpg\")
\n", "
In class discussion : What kinds of images can our model predict?
\n",
"\n",
"**Buzzword**: Generalization\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 108,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(28, 28, 3)"
]
},
"execution_count": 108,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Let'see the tensor shape\n",
"shoe = np.array(Image.open('../fig/kanye_28.jpg'))\n",
"shoe.shape"
]
},
{
"cell_type": "code",
"execution_count": 109,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(28, 28)"
]
},
"execution_count": 109,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# We need to delete the other 2 channels and make the image B&W. \n",
"shoe = shoe[:,:,0]\n",
"shoe.shape"
]
},
{
"cell_type": "code",
"execution_count": 110,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"In class discussion : How did our model perform?
\n",
" \n",
"**Buzzword:** Convolutional Neural Networks!\n",
"\n",
"Let's now try a different boot:"
]
},
{
"cell_type": "code",
"execution_count": 151,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"\n", "\n", "1. Norm penalties: `kernel_regularizer= tf.keras.regularizers.l2(l=0.1)`\n", "2. Early stopping via `tf.keras.callbacks`. Callbacks provide a way to interact with the model while it's training and inforce some decisions automatically. Callbacks need to be instantiated and are added to the `.fit()` function via the `callbacks` argument.\n", "3. Dropout" ] }, { "cell_type": "code", "execution_count": 81, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Train on 40199 samples, validate on 19801 samples\n", "Epoch 1/50\n", "40199/40199 - 5s - loss: 2.6520 - accuracy: 0.7177 - val_loss: 0.9317 - val_accuracy: 0.7763\n", "Epoch 2/50\n", "40199/40199 - 4s - loss: 1.0647 - accuracy: 0.7302 - val_loss: 1.4198 - val_accuracy: 0.6379\n", "Epoch 3/50\n", "40199/40199 - 4s - loss: 1.0388 - accuracy: 0.7340 - val_loss: 1.2246 - val_accuracy: 0.6660\n", "Epoch 4/50\n", "40199/40199 - 4s - loss: 1.0125 - accuracy: 0.7390 - val_loss: 0.9323 - val_accuracy: 0.7721\n", "Epoch 5/50\n", "40199/40199 - 4s - loss: 1.0084 - accuracy: 0.7401 - val_loss: 1.2886 - val_accuracy: 0.6326\n", "Epoch 6/50\n", "40199/40199 - 4s - loss: 1.0013 - accuracy: 0.7385 - val_loss: 1.1757 - val_accuracy: 0.6909\n", "Epoch 7/50\n", "40199/40199 - 4s - loss: 0.9864 - accuracy: 0.7453 - val_loss: 1.0294 - val_accuracy: 0.7437\n", "Epoch 8/50\n", "40199/40199 - 4s - loss: 0.9900 - accuracy: 0.7432 - val_loss: 0.9308 - val_accuracy: 0.7728\n", "Epoch 9/50\n", "40199/40199 - 4s - loss: 0.9877 - accuracy: 0.7418 - val_loss: 1.5178 - val_accuracy: 0.5823\n", "Epoch 10/50\n", "40199/40199 - 3s - loss: 1.0050 - accuracy: 0.7404 - val_loss: 1.1217 - val_accuracy: 0.7037\n", "Epoch 11/50\n", "40199/40199 - 4s - loss: 0.9929 - accuracy: 0.7413 - val_loss: 1.0637 - val_accuracy: 0.6975\n", "Epoch 12/50\n", "40199/40199 - 4s - loss: 0.9952 - accuracy: 0.7441 - val_loss: 1.0190 - val_accuracy: 0.6955\n", "Epoch 13/50\n", "40199/40199 - 3s - loss: 0.9690 - accuracy: 0.7490 - val_loss: 1.0681 - val_accuracy: 0.6973\n", "Epoch 14/50\n", "40199/40199 - 4s - loss: 0.9784 - accuracy: 0.7471 - val_loss: 0.9325 - val_accuracy: 0.7420\n", "Epoch 15/50\n", "40199/40199 - 4s - loss: 0.9663 - accuracy: 0.7455 - val_loss: 0.9807 - val_accuracy: 0.7505\n", "Epoch 16/50\n", "40199/40199 - 4s - loss: 0.9591 - accuracy: 0.7469 - val_loss: 1.0491 - val_accuracy: 0.7014\n", "Epoch 17/50\n", "40199/40199 - 4s - loss: 0.9707 - accuracy: 0.7461 - val_loss: 0.9409 - val_accuracy: 0.7291\n", "Epoch 18/50\n", "40199/40199 - 4s - loss: 0.9733 - accuracy: 0.7467 - val_loss: 1.0317 - val_accuracy: 0.7103\n", "Epoch 19/50\n", "40199/40199 - 4s - loss: 0.9743 - accuracy: 0.7451 - val_loss: 0.9768 - val_accuracy: 0.7537\n", "Epoch 20/50\n", "40199/40199 - 4s - loss: 0.9638 - accuracy: 0.7446 - val_loss: 1.1004 - val_accuracy: 0.6942\n", "Epoch 21/50\n", "40199/40199 - 4s - loss: 0.9686 - accuracy: 0.7440 - val_loss: 1.2571 - val_accuracy: 0.6854\n", "Epoch 22/50\n", "40199/40199 - 4s - loss: 0.9535 - accuracy: 0.7489 - val_loss: 0.9024 - val_accuracy: 0.7748\n", "Epoch 23/50\n", "40199/40199 - 4s - loss: 0.9529 - accuracy: 0.7490 - val_loss: 0.9575 - val_accuracy: 0.7711\n", "Epoch 24/50\n", "40199/40199 - 4s - loss: 0.9643 - accuracy: 0.7444 - val_loss: 0.9397 - val_accuracy: 0.7550\n", "Epoch 25/50\n", "40199/40199 - 4s - loss: 0.9661 - accuracy: 0.7471 - val_loss: 1.7217 - val_accuracy: 0.5002\n", "Epoch 26/50\n", "40199/40199 - 3s - loss: 0.9625 - accuracy: 0.7457 - val_loss: 1.0638 - val_accuracy: 0.7241\n", "Epoch 27/50\n", "40199/40199 - 4s - loss: 0.9605 - accuracy: 0.7429 - val_loss: 0.9840 - val_accuracy: 0.7463\n", "Epoch 28/50\n", "40199/40199 - 4s - loss: 0.9574 - accuracy: 0.7464 - val_loss: 0.9174 - val_accuracy: 0.7680\n", "Epoch 29/50\n", "40199/40199 - 4s - loss: 0.9442 - accuracy: 0.7509 - val_loss: 0.9042 - val_accuracy: 0.7563\n", "Epoch 30/50\n", "40199/40199 - 4s - loss: 0.9364 - accuracy: 0.7510 - val_loss: 0.9537 - val_accuracy: 0.7506\n", "Epoch 31/50\n", "40199/40199 - 4s - loss: 0.9459 - accuracy: 0.7507 - val_loss: 1.2036 - val_accuracy: 0.6440\n", "Epoch 32/50\n", "40199/40199 - 3s - loss: 0.9669 - accuracy: 0.7425 - val_loss: 0.8378 - val_accuracy: 0.7792\n", "Epoch 33/50\n", "40199/40199 - 4s - loss: 0.9670 - accuracy: 0.7424 - val_loss: 0.9400 - val_accuracy: 0.7351\n", "Epoch 34/50\n", "40199/40199 - 4s - loss: 0.9599 - accuracy: 0.7454 - val_loss: 1.1212 - val_accuracy: 0.6596\n", "Epoch 35/50\n", "40199/40199 - 4s - loss: 0.9547 - accuracy: 0.7468 - val_loss: 0.8319 - val_accuracy: 0.7840\n", "Epoch 36/50\n", "40199/40199 - 4s - loss: 0.9484 - accuracy: 0.7481 - val_loss: 0.9867 - val_accuracy: 0.6978\n", "Epoch 37/50\n", "40199/40199 - 4s - loss: 0.9706 - accuracy: 0.7417 - val_loss: 1.0093 - val_accuracy: 0.7357\n", "Epoch 38/50\n", "40199/40199 - 4s - loss: 0.9364 - accuracy: 0.7497 - val_loss: 0.8835 - val_accuracy: 0.7536\n", "Epoch 39/50\n", "40199/40199 - 4s - loss: 0.9294 - accuracy: 0.7500 - val_loss: 0.9051 - val_accuracy: 0.7595\n", "Epoch 40/50\n", "40199/40199 - 4s - loss: 0.9471 - accuracy: 0.7453 - val_loss: 1.0749 - val_accuracy: 0.7112\n", "Epoch 41/50\n", "40199/40199 - 4s - loss: 0.9329 - accuracy: 0.7490 - val_loss: 1.1296 - val_accuracy: 0.6971\n", "Epoch 42/50\n", "40199/40199 - 4s - loss: 0.9469 - accuracy: 0.7468 - val_loss: 0.8840 - val_accuracy: 0.7557\n", "Epoch 43/50\n", "40199/40199 - 4s - loss: 0.9208 - accuracy: 0.7519 - val_loss: 0.9376 - val_accuracy: 0.7614\n", "Epoch 44/50\n", "40199/40199 - 4s - loss: 0.9374 - accuracy: 0.7470 - val_loss: 0.8816 - val_accuracy: 0.7590\n", "Epoch 45/50\n", "40199/40199 - 4s - loss: 0.9442 - accuracy: 0.7456 - val_loss: 0.9593 - val_accuracy: 0.7444\n", "Epoch 46/50\n", "40199/40199 - 4s - loss: 0.9600 - accuracy: 0.7487 - val_loss: 0.8555 - val_accuracy: 0.7755\n", "Epoch 47/50\n", "40199/40199 - 4s - loss: 0.9480 - accuracy: 0.7448 - val_loss: 0.8525 - val_accuracy: 0.7757\n", "Epoch 48/50\n", "40199/40199 - 4s - loss: 0.9248 - accuracy: 0.7484 - val_loss: 0.9819 - val_accuracy: 0.7089\n", "Epoch 49/50\n", "40199/40199 - 3s - loss: 0.9479 - accuracy: 0.7435 - val_loss: 1.1471 - val_accuracy: 0.6752\n", "Epoch 50/50\n", "40199/40199 - 4s - loss: 0.9320 - accuracy: 0.7458 - val_loss: 1.0043 - val_accuracy: 0.7126\n" ] } ], "source": [ "# callbacks\n", "# watch validation loss and be \"patient\" for 50 epochs of no improvement\n", "es = tf.keras.callbacks.EarlyStopping(monitor='val_loss', verbose=1, patience=10)\n", "\n", "model_regular = tf.keras.models.Sequential([\n", " tf.keras.layers.Flatten(input_shape=(28, 28)),\n", " tf.keras.layers.Dense(154, activation='relu', \n", " kernel_regularizer= tf.keras.regularizers.l2(l=0.1)),\n", " tf.keras.layers.BatchNormalization(),\n", " tf.keras.layers.Dropout(0.4),\n", " tf.keras.layers.Dense(64, activation='relu', \n", " kernel_regularizer= tf.keras.regularizers.l2(l=0.1)),\n", " tf.keras.layers.BatchNormalization(),\n", " tf.keras.layers.Dense(10, activation='softmax')\n", "])\n", "\n", "# compile\n", "loss_fn = tf.keras.losses.SparseCategoricalCrossentropy()\n", "optimizer = tf.keras.optimizers.Adam()\n", "\n", "model_regular.compile(optimizer=optimizer,\n", " loss=loss_fn,\n", " metrics=['accuracy'])\n", "# fit\n", "history_regular = model_regular.fit(x_train, y_train, validation_split=0.33, epochs=50, \n", " verbose=2) #, callbacks=[es])\n", " " ] }, { "cell_type": "code", "execution_count": 41, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Test accuracy for regularized model=0.7746999859809875\n" ] } ], "source": [ "test_loss, test_accuracy = model_regular.evaluate(x_test, y_test, verbose=0)\n", "print(f'Test accuracy for regularized model={test_accuracy}')" ] }, { "cell_type": "code", "execution_count": 42, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "