CS-109B Data Science 2: Advanced Topics in Data Science

Lab 7: Autoencoders¶

Harvard University
Spring 2020
Instructors: Mark Glickman, Pavlos Protopapas, and Chris Tanner
Lab Instructors: Chris Tanner and Eleni Angelaki Kaxiras
Content: Eleni Angelaki Kaxiras, Vivek Hv, Cedric Flamant, and Pavlos Protopapas

Learning Goals¶

By the end of this lab, you should be able to:

• Connect the representation that Principal Component Analysis produces to the that of an autoencoder (AE).
• Add tf.keras Functional API into your machine learning arsenal.
• Implement an autoencoder using tf.keras:
  • build the encoder network/model
  • build the decoder network/model
  • decide on the latent/bottleneck dimension
  • train your AE
  • predict on unseen data

Note: To see solutions, uncomment and run the following:¶

# %load solutions/exercise2.py

First time you run will load solution, then you need to run the cell again to actually run the code.

Table of Contents¶

• Part 1: Autoencoders and their connection to Principal Component Analysis.
• Part 2: Denoising Images using AEs.
• Part 3: Visualizing Intermediate Layers of an AE.

Part 1: Autoencoders and the connection to Principal Component Analysis¶

Principal Component Analysis (PCA)¶

PCA decomposes a multivariate dataset in a set of eigenvectors - successive orthogonal coeefficients that explain a large amount of the variance. By using only a number of the highest valued vectors, let's say $N$ of them, we effectively reduce the dimensionality of our data to $N$ with minimal loss of information as measured by RMS (Root Mean Squared) Error.

PCA in sklearn is a transformer that learns those $N$ components via its .fit method. It can then be used to project a new data object in these components. Remember from 109a that we always .fit only to the training set and .transform both training and test set.

from sklearn.decomposition import PCA
k = 2 # number of components that we want to keep

X_train, X_test = load_data()
pca = PCA(n_components=k)

principal_components = pca.fit_transform(X_train)
principal_components = pca.transform(X_test)

Autoencoders (AE)¶

image source: Deep Learning by Francois Collet

An AE maps its input, usually an image, to a latent vector space via an encoder function, and then decodes it back to an output that is the same as the input, via a decoder function. It’s effectively being trained to reconstruct the original input. By trying to minimize the reconstruction MSE error, on the output of the encoder, you can get the autoencoder to learn interesting latent representations of the data. Historically, autoencoders have been used for tasks such as dimentionality reduction, feature learning, and outlier detection.

One type of architecture for an AE is to have the decoder network be a 'mirror image' of the encoder. It makes more sense this way but it is not necessary.

We can say that AEs are self-supervised learning networks!

Understandind the connection between PCA and AEs¶

If the hidden and output layers of an autoencoder are linear, the autoencoder will learn hidden units that are linear representations of the data, just like PCA does. If we have $M$ hidden units in our AE, those will span the same space as the $M$ first principal components. The hidden layers of the AE will not produce orthogonal representations of the data as PCA would but if we add non-linear components in our encoder-decoder networks we can represent a non-linear space/manifold;

Fashion-MNIST¶

We will use the dataset of clothing article images (created by Zalando), consisting of a training set of 60,000 examples and a test set of 10,000 examples. Each example is a 28 x 28 grayscale image, associated with a label from 10 classes. The names of the classes corresponding to numbers 0-9 are:

'T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat','Sandal', 'Shirt', 'Sneaker', 'Bag', and 'Ankle boot'

The creators intend Fashion-MNIST to serve as a direct drop-in replacement for the original MNIST dataset for benchmarking machine learning algorithms. It shares the same image size and structure of training and testing splits. Each pixel is 8 bits so its value ranges from 0 to 255.

Let's load and look at it!

Note: The first two components explain ~19+12 = 31% of the variance.

Part 2: Denoise Images using AEs¶

We will create an autoencoder which will accept "noisy" images as input and try to produce the original images. Since we do not have noisy images to start with, we will add random noise to our Fashion-MNIST images. To do this we will use the image augmentation library imgaug docs.

From this library we will use SaltAndPepper, an augmenter which replaces pixels in images with salt/pepper noise (white/black-ish colors), randomly with probability passed as parameter p. Use the code below to install the library in your virtual environment.

tf.keras.Sequential API¶

This is what we have been using so far for building our models. Its pros are: it's simple to use, it allows you to create models layer-by-layer. Its basic con is: it is not very flexible, and although it includes layers such as Merge, Concatenate, and Add that allow for a combination of models, it is difficult to make models with many inputs or shared-layers. All layers, as the name implies, are connected sequentially.

Intro to tf.keras.Functional API¶

https://www.tensorflow.org/guide/keras/functional.

In this API, layers are built as graphs, with each layer indicating to which layer it is connected. Functional API helps us make more complex models which include non-sequential connections and multiple inputs or outputs.

Let's say we have an image input with a shape of (28, 28, 1) and a classification task:

num_classes = 10

inputs = keras.Input(shape=(h, w, 1))
x = Dense(64, activation='relu')(inputs)
x = layers.Dense(64, activation='relu')(x)
outputs = Dense(num_classes, activation='softmax', name='output')(x)

ae_model = Model(inputs=inputs, outputs=outputs, name='autoencoder')

ae_model.summary()

Create the Encoder¶

• Use the Functional API
• Create a pair of layers consisting of a Conv2D and a MaxPool layer which takes in our input_layer. Choose the number of filters.
• Stack 3 of these layers, one after the other.
• Give this model the name latent_model (it's not your final model).

• repeat the structure of your encoder but in "reverse".
• What is the output layer activation function? What are the dimensions of the output?

Choose an optimizer and a loss function. Use Early Stopping. To get good results you will need to run this about 20 epochs and this will take a long time depending on your machine. For the purposes of this lab run only for 2 epochs.

(Optional: add Tensorboard).

Here is how to connect the two models:

Part 3: Visualizing Intermediate Layers of AE¶

This is our "encoder" model which we have saved as:

encoder  = Model(input_layer, latent_view, name='encoder_model')

possible answers¶

We can have bottleneck layers in convolutional autoencoders that are not dense but simply a few stacked featuremaps such as above. They might have better generalizability due to only using shared weights. One interesting consequence is that without the dense layer you'll force translational equivariance on the latent representation (a particular feature in the top right corner will appear as an activation in the top right corner of the featuremaps at the level of the bottleneck, and if the feature is moved in the original image the activation in the bottleneck will move proportionally in the same direction). This isn't necessarily a problem, but you are enforcing some constraints on the relationships between the latent space directions that you wouldn't be with the presence of a dense layer.

Visualize Samples reconstructed by our AE¶