{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "## Title :\n", "Performance comparison of different SOTAs\n", "\n", "## Description :\n", "The goal of this exercise is to compare different architectures on speed, size and performance.\n", "Your final plot may resemble the one below:\n", "\n", "\n", "\n", "To read more about Imagenet Dataset.\n", "\n", "## Instructions :\n", "\n", "- Load the data and choose only the first 100 images from the validation dataset.\n", "- Using the helper code obtain the model statistics for each of the following SOTAs below:\n", " - VGG16 \n", " - VGG19\n", " - InceptionNetV3\n", " - ResNet50 \n", " - MobileNetV2 \n", "\n", "## Hints :\n", "\n", "model.layers Accesses layers of the model\n", "\n", "tf.keras.activations.linear Linear activation function\n", "\n", "model.predict() Used to predict the values given the model\n", "\n", "np.argsort() Returns the indices that would sort an array.\n", "\n", "tf.keras.applications.VGG16 Instantiates the VGG16 model.\n", "\n", "tf.keras.applications.VGG19 Instantiates the VGG16 model.\n", "\n", "tf.keras.applications.ResNet50 Instantiates the ResNet50 architecture.\n", "\n", "tf.keras.applications.InceptionV3 Instantiates the Inception v3 architecture.\n", "\n", "tf.keras.applications.MobileNetV2 Instantiates the MobileNetV2 architecture." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "\n", " \"Open" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Performance comparision on SOTAs\n", "**(Note: This notebook will not run on Ed. Please click the button above to run in Google Colab)**" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "# Import required libraries\n", "import sys, os, time\n", "\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "%matplotlib inline\n", "from matplotlib.colors import ListedColormap\n", "colors = ['k', 'g', 'r','b','c']\n", "plt.style.use('seaborn-whitegrid')\n", "from helper import ellipse\n", "import pickle\n", "\n", "from tensorflow.keras import backend as K\n", "from tensorflow.keras.utils import to_categorical\n", "from tensorflow.keras.layers import Input\n", "import timeit" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Loading the data" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "# Useful dictionary to go from label index to actual label\n", "with open('idx2name.pkl', 'rb') as handle:\n", " keras_idx_to_name = pickle.load(handle)" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "# Loading input image and labels\n", "images = np.load(\"/course/data/x_val.npy\") # loaded as RGB\n", "labels = np.load(\"/course/data/y_val.npy\")\n", "\n", "# Taking only 100 samples for quicker computation\n", "x_val = images[:100]\n", "y_val = labels[:100]\n", "# One hot encoding the labels\n", "y_val_one_hot = to_categorical(y_val, 1000)" ] }, { "cell_type": "code", "execution_count": 0, "metadata": {}, "outputs": [], "source": [ "# Print a sample image and set the label as title\n", "plt.title(___)\n", "plt.imshow(___)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### ⏸ What is the label for the first image in the validation set?\n", "**(Please answer this in quiz)**\n", "\n", "\n", "#### A. Cabbage Butterfly\n", "#### B. Mixing bowl\n", "#### C. Wok\n", "#### D. French horn" ] }, { "cell_type": "code", "execution_count": 0, "metadata": {}, "outputs": [], "source": [ "# Submit an answer choice as a string below (eg. if you choose option C, put 'C')\n", "answer1 = '___'" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Benchmark models" ] }, { "cell_type": "code", "execution_count": 0, "metadata": {}, "outputs": [], "source": [ "# Helper function to get key stats\n", "# (evaluation speed, top-1 % accuracy, total model parameters)\n", "def model_stats(model,x_val,name):\n", " #Time for evaluation\n", " time = timeit.timeit(lambda: model.predict(x_val, verbose=1), number=1)\n", " # Accuracy\n", " y_pred = model.predict(x_val)\n", " top_1 = np.any(np.argsort(y_pred)[:,-1:].T == y_val_one_hot.argmax(axis=1),axis=0).mean()\n", " # Model size \n", " params = model.count_params()\n", " return (time,top_1,params,name)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## SOTA architectures\n", "\n", "For this exercise, we will consider the following SOTAs:\n", "- VGG16\n", "- VGG19\n", "- InceptionV3\n", "- ResNet50\n", "- MobileNet" ] }, { "cell_type": "code", "execution_count": 0, "metadata": {}, "outputs": [], "source": [ "# VGG16 stats\n", "from tensorflow.keras.applications.vgg16 import VGG16, preprocess_input\n", "\n", "# Preprocess step\n", "# We need to call the data because some preprocess steps\n", "# change the value inplace\n", "x_val = np.load(\"/course/data/x_val.npy\") # loaded as RGB\n", "x_val = x_val[:100]\n", "x_val = preprocess_input(x_val)\n", "\n", "# Call the VGG16 model\n", "model = ___\n", "\n", "# Collect stats \n", "vgg16stats = model_stats(model,x_val,'VGG16')" ] }, { "cell_type": "code", "execution_count": 0, "metadata": {}, "outputs": [], "source": [ "# VGG19 stats\n", "from tensorflow.keras.applications.vgg19 import VGG19, preprocess_input\n", "\n", "x_val = np.load(\"/course/data/x_val.npy\") # loaded as RGB\n", "x_val = x_val[:100]\n", "x_val = preprocess_input(x_val)\n", "\n", "# Call the VGG19 model\n", "model = ___\n", "\n", "# Collect stats \n", "vgg19stats = model_stats(model,x_val,'VGG19')" ] }, { "cell_type": "code", "execution_count": 0, "metadata": {}, "outputs": [], "source": [ "# Inception Stats\n", "from tensorflow.keras.applications.inception_v3 import InceptionV3,preprocess_input\n", "\n", "x_val = np.load(\"/course/data/x_val.npy\") # loaded as RGB\n", "x_val = x_val[:100]\n", "x_val = preprocess_input(x_val)\n", "\n", "# Call the InceptionV3 model\n", "model = ___\n", "\n", "# Collect stats \n", "inceptionstats = model_stats(model,x_val,'Inception')" ] }, { "cell_type": "code", "execution_count": 0, "metadata": {}, "outputs": [], "source": [ "# Resnet50 stats\n", "from tensorflow.keras.applications.resnet50 import ResNet50, preprocess_input\n", "\n", "x_val = np.load(\"/course/data/x_val.npy\") # loaded as RGB\n", "x_val = x_val[:100]\n", "x_val = preprocess_input(x_val)\n", "\n", "# Call the ResNet50 model\n", "model = ___\n", "\n", "# Collect stats \n", "resnetstats = model_stats(model,x_val,'Resnet50')" ] }, { "cell_type": "code", "execution_count": 0, "metadata": {}, "outputs": [], "source": [ "# MobileNet stats\n", "from tensorflow.keras.applications.mobilenet_v2 import MobileNetV2, preprocess_input\n", "x_val = np.load(\"/course/data/x_val.npy\") # loaded as RGB\n", "x_val = x_val[:100]\n", "x_val = preprocess_input(x_val)\n", "\n", "# Call the MobielNetV2 model\n", "model = ___\n", "\n", "# Collect stats \n", "mobilestats = model_stats(model,x_val,'MobileNet')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### ⏸ Which SOTA architecture from above has the **highest** number of trainable parameters?\n", "**(Please answer this in quiz)**\n", "\n", "#### A. VGG-16\n", "#### B. VGG-19\n", "#### C. ResNet50\n", "#### D. MobileNet" ] }, { "cell_type": "code", "execution_count": 0, "metadata": {}, "outputs": [], "source": [ "# Submit an answer choice as a string below (eg. if you choose option C, put 'C')\n", "answer2 = '___'" ] }, { "cell_type": "code", "execution_count": 0, "metadata": {}, "outputs": [], "source": [ "# Use the helper code below \n", "# to plot the model stats for each SOTA\n", "fig, ax = plt.subplots(figsize=(10,6))\n", "for i,val in enumerate([vgg16stats, vgg19stats, inceptionstats,resnetstats, mobilestats]):\n", " r = val[2]/10**9 + 0.04\n", " ellipse(val[0]/40,val[1],width=r,height=0.44*r,color = colors[i],ax=ax)\n", " ax.text(val[0]/40 + 0.035, val[1]+r/4+ 0.004, val[3], va='center', ha='center',fontsize=12)\n", " \n", "ax.set_ylim([0.6,0.85])\n", "ax.set_ylabel('Top-1 accuracy [%]',fontsize=20)\n", "ax.set_xlabel('Time for evaluation [s]',fontsize=20)\n", "ax.set_xticklabels(range(0,60,8));\n", "ax.set_yticklabels(range(50,110,10));\n", "for axis in ['bottom','left']:\n", " ax.spines[axis].set_linewidth(3)\n", " ax.spines[axis].set_color('k')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 🍲 Larger dataset\n", "\n", "Go back and take a larger sample of images, do your results remain consistent?" ] }, { "cell_type": "code", "execution_count": 0, "metadata": {}, "outputs": [], "source": [ "# Type your answer within in the quotes given\n", "answer3 = '___'" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.5" } }, "nbformat": 4, "nbformat_minor": 4 }