Key Word(s): Convolutional Neural Network, Neural Networks, CNNs, MPL

Download Notebook

Title :

Image Occlusion

Description :

The aim of this exercise is to understand occlusion. Each pixel in an image has varying importance for the classification of the image. Occlusion involves running a patch over the entire image to see which pixels affect the classification the most.

Instructions:

  • Define a convolutional neural network based on the architecture mentioned in the scaffold.
  • Load the trained model weights given in the occlusion_model_weights.h5 file.
  • Take a quick look at the model architecture using model.summary().
  • Use the helper function occlusion to visualize the delta loss as a mask moves across the image. The output will look similar to the one shown below.

Hints:

MaxPooling2D()Max pooling operation for 2D spatial data.

compile()Configures the model for training.

Conv2D()2D convolution layer (e.g. spatial convolution over images).

flatten()Flattens the input.

Dense()A regular densely-connected NN layer.

NOTE - To run the test, comment the cells that call the occlusion function and then click on Mark.

In [1]:
# Import necessary libraries
import os
import random
import numpy as np
import pandas as pd
import tensorflow as tf
import matplotlib.pyplot as plt
from tensorflow.keras import backend as K
from sklearn.metrics import accuracy_score
from helper import occlusion, load_dataset
from tensorflow.keras.optimizers import SGD
from tensorflow.keras.datasets import cifar10
from tensorflow.keras.utils import to_categorical
from tensorflow.keras.applications import MobileNet
from sklearn.model_selection import train_test_split
from tensorflow.keras.models import Sequential, Model
from tensorflow.keras.applications.mobilenet import preprocess_input
from tensorflow.keras.layers import Dense, Dropout, Flatten, Activation, Input, Conv2D, MaxPooling2D, InputLayer, ReLU

%matplotlib inline
In [2]:
# Initialize a sequential model
model = Sequential(name="Occlusion")

# First convolution layer
model.add(Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same', input_shape=(32, 32, 3)))

# Second convolution layer
model.add(Conv2D(32, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))

# First max-pooling layer
model.add(MaxPooling2D((2, 2)))

# Third convolution layer
model.add(Conv2D(64, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))

# Fourth convolution layer
model.add(Conv2D(64, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))

# Second max-pooling layer
model.add(MaxPooling2D((2, 2)))

# Fifth convolution layer
model.add(Conv2D(128, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))

# Sixth convolution layer
model.add(Conv2D(128, (3, 3), activation='relu', kernel_initializer='he_uniform', padding='same'))

# Third max-pooling layer
model.add(MaxPooling2D((2, 2)))

# Flatten layer
model.add(Flatten())

# Fully connected dense layer
model.add(Dense(128, activation='relu', kernel_initializer='he_uniform'))

# Output layer
model.add(Dense(10, activation='softmax'))

# Compiling the model
model.compile(optimizer=SGD(lr=0.001, momentum=0.9), loss='categorical_crossentropy', metrics=['accuracy'])
In [0]:
# Take a quick look at the model summary
model.summary()
In [4]:
# Load the weights of the pre-trained model
model.load_weights("occlusion_model_weights.h5")

⏸ Call the function occlusion (below) with image numbers 10, 12 and 35. What do you observe based on the occlusion map plotted for each image?

A. The images are blurred more as compared to other images in the set.

B. The images are incorrectly predicted because the model weights the wrong parts of the image to make the prediction.

C. The images are correctly predicted as the network is giving high importance to the most telling features of the images.

In [0]:
### edTest(test_chow1) ###
# Submit an answer choice as a string below (eg. if you choose option C, put 'C')
answer1 = '___'
In [0]:
# Call the helper function occlusion with 
# the trained model, a valid image number within 50, occlusion 
# patch size
img_num = ___
patch_size = ___

occlusion(model, img_num , patch_size)

⏸ Call the occlusion function (below) with images 1, 15 and 30. What do you observe based on the plots?

In [0]:
### edTest(test_chow2) ###

# Type your answer here 
answer2 = '___'