Classification and PCA Lab¶

$$ \renewcommand{\like}{{\cal L}} \renewcommand{\loglike}{{\ell}} \renewcommand{\err}{{\cal E}} \renewcommand{\dat}{{\cal D}} \renewcommand{\hyp}{{\cal H}} \renewcommand{\Ex}[2]{E_{#1}[#2]} \renewcommand{\x}{{\mathbf x}} \renewcommand{\v}[1]{{\mathbf #1}} $$

Setting up some code¶

In doing homework so far you have probably seen strange behaviours when you run and rerun code. This happens in the jupyter notebook because one is generally using global variables, and you might change a value 10 cells down and then rerun a cell 10 cells before.

To work around such behavior. encapsulate code withon functions and minimize your use of global variables!

A function to plot the points on the training and test set, and the prediction regions associated with a classifier that has 2 features. Adapted from scikit-learn examples.

A function to add contours to such a plot. I use it while showing predictions as opposed to the default "actual test values" in points_plot.

Digits dataset: constructing a classification dataset¶

This problem is adapted from Jake Vanderpls's tutorial at pydata: (http://nbviewer.jupyter.org/github/jakevdp/sklearn_pydata2015/blob/master/notebooks/02.2-Basic-Principles.ipynb)

The classification problem there is a multiway digit classification problem

This code, taken from Jake's notebook above, plots the images against the targets so that we can see what we are dealing with.

YOUR CODE HERE: To create a stripped down problem for this lab, let us take 2 numbers and try and distinguish them between images. Lets take 8 and 9. make a dtaframe called dftwo for this

Logistic Regression¶

YOUR TURN HERE: Carry out an unregularized logistic regression and calculate the score on the set1 test set.

Logistic Regression using Cross Validation and Regularization¶

A function to grid search on parameters while doing cross-validation. Note we return the grid-search meta estimator. Be default GridSearchCV will refit on the entire training set. Note the use of scoring, which will allow for a use of a different scoring function on the cross-validation set than the loss used to train the model on the training set. (Kevin talked about this in class...and the default in sklearn is to use the 1-0 loss for scoring on the validation sets, and the log-loss for example in LogisticRegression, for training and parameter estimation.

I keel these separate in my head as estimation and decision losses. After all, classification requires you to make a decision as to what threshold you will choose.

do_classify is an omnibus function which will take a dataframe, a set of column names to use as features, a name for the target, and do the entire machine learning process for you. For the reason of comparing classifiers, it can take an existing testing and training set as well. If you ask it to, it will standardize as well.

This was what I had earlier and refactored. What more could you do?

So we refactor:

The confusion matrix: comparing classifiers¶

We have written two classifiers. A classifier will get some samples right, and some wrong. Generally we see which ones it gets right and which ones it gets wrong on the test set. There,

• the samples that are +ive and the classifier predicts as +ive are called True Positives (TP)
• the samples that are -ive and the classifier predicts (wrongly) as +ive are called False Positives (FP)
• the samples that are -ive and the classifier predicts as -ive are called True Negatives (TN)
• the samples that are +ive and the classifier predicts as -ive are called False Negatives (FN)

A classifier produces a confusion matrix from these which lookslike this:

IMPORTANT NOTE: In sklearn, to obtain the confusion matrix in the form above, always have the observed y first, i.e.: use as confusion_matrix(y_true, y_pred)

YOUR TURN NOW: Calculate the confusion matrix for the regularized logistic regression

YOUR TURN NOW: As an exercise to do this completely with a new train/test split done directly on a dataframe. Call your classifier/estimator digitstwo_log and your training/test sets dictionary as set2. Compute the confusion matrix for this set2

From the department of not-kosher things to do, (why?) we calculate the performance of this classifier on set1.

Plotting scores against hyper-parameters¶

Finally plot_scores takes the output of a grid search on one parameter, and plots for you a graph showing the test performance against the parameter. You could augment this with a training set diagram if you like.

Feature engineering¶

Our images here are relatively small, but in general you will have as many features as pizels multiplied by the color channels. This is a lot of features! Having too many features can lead to overfitting.

Indeed, it is possible to have more features than data points. Thus there is a high chance that a few attributes will correlate with $y$ purely coincidentally! [^Having lots of images, or "big-data" helps in combatting overfitting!]

We need to do something similar to what happened in the regularized regression or classification here! We will engage in some a-priori feature selection that will reduce the dimensionality of the problem. The idea we'll use here is something called Principal Components Analysis, or PCA.

PCA is an unsupervized learning technique. The basic idea behind PCA is to rotate the co-ordinate axes of the feature space. We first find the direction in which the data varies the most. We set up one co-ordinate axes along this direction, which is called the first principal component. We then look for a perpendicular direction in which the data varies the second most. This is the second principal component. The diagram illustrates this process. There are as many principal components as the feature dimension: all we have done is a rotation.

(diagram taken from http://stats.stackexchange.com/questions/2691/making-sense-of-principal-component-analysis-eigenvectors-eigenvalues which also has nice discussions)

How does this then achieve feature selection? We decide on a threshold of variation; once the variation in a particular direction falls below a certain number, we get rid of all the co-ordinate axes after that principal component. For example, if the variation falls below 10% after the third axes, and we decide that 10% is an acceptable cutoff, we remove all dimensions from the fourth dimension onwards. In other words, we took our higher dimensional problem and projected it onto a 3 dimensional subspace.

These two ideas illustrate one of the most important reasons that learning is even feasible: we believe that most datasets, in either their unsupervized form $\{\v{x\}}$, or their supervized form $\{y, \v{x}\}$, live on a lower dimensional subspace. If we can find this subspace, we can then hope to find a method which respectively separates or fits the data.

The explained variance ratio pca.explained_variance_ratio_ tells us how much of the variation in the features is explained by these 60 features. When we sum it up over the features, we see that 94% is explained: good enough to go down to a 60 dimensional space from a 136452 dimensional one!

We can see the individual variances as we increase the dimensionality:

The first dimension accounts for 35% of the variation, the second 6%, and it goes steadily down from there.

Let us create a dataframe with these 16 features labelled pc1,pc2...,pc60 and the labels of the sample:

Lets see what these principal components look like:

You might be a bit confused: we needed to use 16 components to explain 90% of the variation in the features, but only 1 or 2 components to separate checks from dollars? This is because PCA is unsupervised: the only variation we are explaining is the variation in the 64 dimensional feature space. We are not explaining the variation in the $y$ or the label, and it might turn out, as it does in this case, that with the additional information in $y$, the dimensionality needed for classification is much lower.

We could thus choose just the first few principal components to make our classifier. For the purposes of this lab, since two components can be easily visualized (even though adding some more features may leads to better separability), we'll go with learning a 2-dimensional classifier in the pc1 and pc2 dimensions!

YOUR CODE NOW: Do a regularized Logistic Regression using the first two principal components. Store the classifier in digitspca_log2 and the sets in setf

We use points plot to see misclassification and the decision boundary:

And a probability contour plot to see probabilities

And a scores plot to see the hyper-parameter landscape.

And a confusion matrix...