CS109A Introduction to Data Science

Standard Section 6: PCA and Logistic Regression¶

Harvard University
Fall 2020
Instructors: Pavlos Protopapas, Kevin Rader, and Chris Tanner
Section Leaders: Marios Mattheakis, Hayden Joy

---

For this section, our goal is to get you familiarized with Dimensionality Reduction using Principal Components Analysis (PCA) and to recap Logistic Regression from the last homework. This medium article was referenced extensively while creating this notebook.

Specifically, we will:

• Define the terms big data and high-dimensionality
• Learn what PCA is. What is the motivation to use PCA.
• Learn about the sklearn PCA library and its nuances
• Get familiar with the Linear Algebra of PCA
• Use PCA to visualize high-dimensional data in 2-dimensions
• Meet the MNIST handwritten digit dataset (and hopefully stay friends for a while)
• Use PCA in order to improve model training time and understand the speed-accuracy trade-off
• Discuss when to use PCA and when not to use it
• Assumptions behind PCA

---

---

Motivation¶

Principal Components Analysis helps us deal with high-dimensionality in big-data.

But first...

High-dimensionality is the case when p is large i.e. there are a lot of predictors. This is sometimes a problem because:

1. Our models may be overfit
2. There may be multi-collinearity
3. Matrices may not be invertible (in the case of OLS)

Our challenge is to represent the p dimensions by a smaller number (m) of dimensions without losing much information. Then, we can fit a model using the m predictors, which addresses the three problems listed above. Here's where PCA comes into play.

What is Principal Components Analysis (PCA)?¶

A Framework For Dimensionality Reduction¶

Consider that the data are described by a Linear Regression Model: $$ Y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \dots + \beta_p X_p + \epsilon, $$ where $p$ is very large, i.e. high dimensionality.

LASSO for dimensional reduction:

We can use LASSO: which will drop some of the predictors by forcing $\beta_j=0$. That effectively reduces the dimensions. Are we happy with that? No! Because we totally lose the information kept by the $X_j$ predictors

PCA for dimensional reduction:

Considering a new system of coordinates, namely a new set of predictors, denoted by $Z_1$, $Z_2$,$\dots$, $Z_m$, where $m \leq p$ and where each $Z_i$ is a linear combination of the original $p$ predictors, $X_1, \dots~ X_p$, thus:

$$ Z_i = \sum_{j=1}^{p} c_{ij} X_i $$

for some fixed coefficients $c_{ij}$ (PCA will determine them). For example: $$ Z_1 = 3.3 X_1 + 4 X_2 + 0 X_3 + \dots + 1.2 X_p. $$

Then we can build a linear regression model using the new predictors as follows: $$ Y = \theta_0 + \theta_1 Z_1 + \theta_2 Z_2 + \dots + \theta_m Z_m + \epsilon $$

Notice: For $m=p$ the number of predictors is the same, hence, we have not lost any information. We just transform the coordinate systems of the data-space.

Determine the PCA coefficients $c_{ij}$: PCA identifies a new set of predictors, as linear combinations of the original ones, that captures the 'maximum amount' of variance in the observed data. In other words, PCA determines the $c_{ij}$ such as the data varies most along the new axis $Z_i$, which are called principal components. PCA sorts the axis such as the largest variance goes along $Z_1$, the second largest variance goes to $Z_2$ and so on.

Comments:

• The basic assumption in PCA is that higher variance indicates more importance
• The principal components consist an $m$-dimensional orthonormal system of coordinates

We see that the "best line" is the one where there is maximal variance along the line. Source here.

In principle, we could explore all the rotations, that is, rotating the original coordinate system under all the angles, and find which rotation yields the maximum variance. However, when the dimensionality (p) is large this is an extremely time consuming and inefficient technique. In that case we may use PCA which is systematic way to find the best rotation or the best coordinate system. PCA is a mathematical method based on linear algebra, for more details and rigorous formulation check the advanced section for PCA.

Have we reduced the dimensions yet? No yet... So far we have defined new predictors that linearly depend on the original predictors. Hence, we still have all the information stored in the data.

Reducing the dimensions: Since the $Z_j$ are sorted with respect how much information the carry, the larger the $j$ the less important the $Z_j$. Hence, we can keep just a few of the principal components and drop the rest. For instance, keeping only the first two components $(m=2)$, we obtain two predictors that contain information from all the original $p$ predictors. How can we choose the m? PCA takes care of that too.

Applications of PCA¶

One major application of PCA is to address the issue of high-dimensionality (reduce the number of predictors).

Another major application of PCA is in visualization. Specifically, if we have an N-dimensional dataset (N>3), how do we visualize it?

One option:

A more practical option: use PCA to get the top 2-3 principal components and plot these components on 2D or 3D plots.

PCA for Visualization¶

Data Source: MTCars Dataset

Here are a few resources that use this dataset and apply PCA for visualization. This notebook references this PCA tutorial in R, these lecture notes from CMU, this blog, and this blog which has some nice visualizations of PCA.

Loading in The Cars Dataset and carry out EDA¶

This dataset consists of data on 32 models of car, taken from an American motoring magazine (1974 Motor Trend magazine). For each car, you have 11 features, expressed in varying units (US units), They are as follows (source):

• mpg: Fuel consumption (Miles per (US) gallon): more powerful and heavier cars tend to consume more fuel.
• cyl: Number of cylinders: more powerful cars often have more cylinders
• disp: Displacement (cu.in.): the combined volume of the engine's cylinders
• hp: Gross horsepower: this is a measure of the power generated by the car
• drat: Rear axle ratio: this describes how a turn of the drive shaft corresponds to a turn of the wheels. Higher values will decrease fuel efficiency.
• wt: Weight (1000 lbs): pretty self-explanatory!
• qsec: 1/4 mile time: the car's speed and acceleration
• vs: Engine block: this denotes whether the vehicle's engine is shaped like a "V", or is a more common straight shape.
• am: Transmission: this denotes whether the car's transmission is automatic (0) or manual (1).
• gear: Number of forward gears: sports cars tend to have more gears.
• carb: Number of carburetors: associated with more powerful engines

Note that the units used vary and occupy different scales.

Dropping the categorical variables vs and am and only keeping in the continuous predictors.

Our task is to try to visualize this data in a meaningful way. Obviously we can't make a 9-dimensional plot, but we can try to make several different plots using the pairplot function from seaborn.

But there are numerous variables and numerous more relationships between these variables. We can do better through PCA.

Visualization by using PCA¶

Standardizing Variables¶

Standardization is a crucial step in PCA formulation (more details in advanced section)

Carrying out PCA¶

Sklearn PCA documentation

Let us examine some of the attributes we obtain from PCA.

1. explained_variance_: The amount of variance explained by each of the selected principal components.
2. explained_variance_ratio_: Percentage of variance explained by each of the selected principal components. By default, unless n_components is specified all components will be stored and the sum of the ratios will be 1.0.

We see that over 85% of the variance is explained by the first 2 principal components!

1. components_: This represents the principal components i.e. directions of maximum variance in the data. The components are sorted by explained_variance_.

Let us write the equation for all the principal components using our formulation of the principal components above:

$$ Z_i = \sum_{j=1}^{p} w_{ij} X_i $$

Using the printed equations above, we can create vectors showing where each feature has a high value. Let us do this for the first 2 principal components (using $v$ to denote a vector):

$$ \begin{aligned} v_{carb} = \begin{pmatrix}-0.24 \\ 0.48 \end{pmatrix}, \; v_{cyl} = \begin{pmatrix}-0.4 \\ 0.02 \end{pmatrix}, \; v_{disp} = \begin{pmatrix}-0.4 \\ -0.09 \end{pmatrix}, \\ v_{drat} = \begin{pmatrix}0.31 \\ 0.34 \end{pmatrix}, \; v_{gear} = \begin{pmatrix}0.21 \\ 0.55 \end{pmatrix}, \; v_{hp} = \begin{pmatrix}-0.37 \\ 0.27 \end{pmatrix}, \\ v_{mpg} = \begin{pmatrix}0.39 \\ 0.03 \end{pmatrix}, \; v_{qsec} = \begin{pmatrix}0.22 \\ -0.48 \end{pmatrix}, \; v_{wt} = \begin{pmatrix}-0.37 \\ -0.17 \end{pmatrix} \end{aligned} $$

Checking if our vectors are orthonormal¶

Orthonormal vectors are the vectors which are orthogonal (zero dot product) with length equal to one (unit vectors).

Orthogonal:¶

We use the dot product between two vectors to check if the vectors are orthogonal or not. If the dot product is 0, that means that the two vectors are orthogonal. The dot product between two vectors is (geometrically):

$$ \textbf{a} \cdot \textbf{b} = ||\textbf{a}|| ||\textbf{b}|| \cos(\theta) $$

Where $\theta$ is the angle between the two vectors and $||\cdot||$ denotes the norm of the vector. Since we assume that the norm of a vector is non-zero, the only way the dot product of two vectors to be zero is when the angle between them is 90 degrees ($\cos(90) = 0$). Thus, the dot product is a way to check if two vectors are perpendicular.

Unit vectors¶

In order to calculate the length $||\textbf{a}||$ of a vector we can take the dot product of a vector with itself, namely $$ ||\textbf{a}|| = \textbf{a}\cdot \textbf{a} $$

We see that the first two principal components are orthogonal and the first principal component is also a unit vector. You can check other pairs of principal components in order to convince yourself that all principal components are always pairwise orthogonal unit vectors.

Visualizing PCA results¶

Any patterns of interest? Let us examine the geography more closely. Source: this blog.

What patterns do you see now?

For example, looking at the axes, we see that the American cars are characterized by high values for number of cyl, disp, and wt. Thus they are more powerful and heavier cars. Japanese cars, on the other hand, are characterized by higher fuel efficieny (mpg). European cars are somewhat in the middle and less tightly clustered than either group, but are average more efficient than American cars

We can draw conclusions visually. Not any modeling so far.

Addressing n_components¶

PCA takes in 1 parameter: n_components. This defines the number of principal components that PCA will use. By default, the number of the used principal components is the minimum of the number of rows and the number of columns in the dataset min(p, n). This is the maximum number of dimensions that PCA transform is allowed to use; For more details check the advanced section.

Note: Setting the default parameter for n_components and taking the top k principal components is equivalent to setting n_components=k. The former is computationally more expensive though. Let's check this.

Break Out Room 1¶

Visualizing high-dimensional data in a 2D plot

• Perform PCA
• Explore the "Explained Variance Ratio"
• Keep the first two principal components and visualize the data in this low-dimensional 2D space.
• Explore the "cumulative explained variance ratio"

As a reminder here is a plot from Section 5, where we outlined the data by using two predictors. Think about the difference between LASSO and PCA as dimensional reduction methods.

PCA to speed up classification of Handwritten Digits¶

This example, using the MNIST dataset, was borrowed from this Towards Data Science blog post. In this example, we will be classifying hand-written digits.

Data Loading and EDA¶

IMPORTANT: Unless you have the "TF version 2.0.0" try the following¶

pip install --upgrade pip

pip install tensorflow==2.0.0

pip3 install keras

OR

conda install tensorflow=2.0.0

Verify the installation by displaying the package information:¶

An alternative way to load the MNIST dataset¶

Our training set (x_train) contains 60000 images of size 28 by 28. Our training labels (y_train) are numbers from 0 to 9. Let's examine one of these values below.

Our task is to classify the test set digits as accurately as possible.

The shape of the training set is $6000 \times 28 \times 28$. We have not dealt with 3D arrays before. We will deal with images in greater detail (and not only!!!) in the follow-up course, CS109b, if you are interested in doing more of this kind of stuff you should take this course. For now, we will reshape the array into a 2D array of shape $6000\times 784$.

Normalizing Data¶

Image data is usually between 0 and 255 (0 indicates a black pixel and 255 indicates a white pixel). We can normalize these values by dividing by 255.

Why do we prefer normalization instead of standardization in this case?

Modeling using Logistic Regression¶

We get a high training and test set score but it takes a relatively long time to fit a model. Let us see if we can speed things up when using PCA

Logistic Regression Model after PCA¶

We see that the first 100 principal components hold over 90% of the variance and the first 50 principal components hold over 80% of the variance! We have significantly reduced the dimensionality of our problem! Let us use PCA to find the first 100 principal components of our dataset and transform our x_train and x_test accordingly.

Note: The time taken to fit our model is considerably smaller! Now let us check our accuracy

Not a significant drop in accuracy!! But, since we are losing information by not accounting for all the variance, we are faced with a speed accuracy tradeoff. You should explore the case of keeping less principal components.

Plotting PCA¶

Plotting the Reconstructed Image¶

Uhhh... this is terrible. But we can use PCA to carry out an inverse transform in order to get a reconstructed image. Let's try again, using pca.inverse_transform()! Source: this github repo

Plotting all our points on a 2-dimensional plot given by the first 2 principal components of PCA¶

This towards data science article has a few similar plots that are pretty cool!

Any patterns of interest?

Break Out Room 2¶

Lets perform classification by using reduced dimensional data-space.

Specifically compare the classification score when we use: (a) all the predictors; (b) 2 of the original predictors; (c) 2 principal components as predictors.

Use all the 4 predictors¶

Use the 2 predictors¶

Use all the 2 principal componets as predictors¶

• Perform a PCA transform with 2 principal components
• Use the 2 principal components for a logistic regression
• Print the score for the training and testing sets
• Compare with the previous two cases

Should I always use PCA?¶

PCA is great for:

1. Speeding up the training of a model without significantly decreasing the predictive ability relative to a model with all p predictors.
2. Visualizing how predictive your features can be of your response, especially in the case of classification.
3. Reducing multicollinearity, and thus potentially improving the computational time required to fit models.
4. Reducing dimensionality in very high dimensional settings.

PCA is not so good in certain situations because:

1. Interpretation of coefficients in PCA is completely lost. So do not do PCA if interpretation is important.
2. When the predictors' distribution deviates significantly from a multivariable Normal distribution.
3. When the high variance does not indicate high importance.
4. When the hidden dimensions are not orthonormal.

Assumptions of PCA¶

• Linear change of basis: PCA is a linear transformation from a Euclidean basis (defined by the original predictors) to an abstract orthonormal basis. Hence, PCA assumes that such a linear change of basis is sufficient for identifying degrees of freedom and conducting dimensionality reduction

• Mean/variance are sufficient (data are approximately multi-variate gaussian): In applying PCA to our data, we are only using the means (for standardizing) and the covariance matrix that are associated with our predictors. Thus, PCA assumes that such statistics are sufficient for describing the distributions of the predictor variables. This is true only if the predictors are drawn jointly from a multivariable Normal distribution. When the predictor distributions heavily violate this assumption, PCA components may not be as informative.

• High variance indicates high importance: This fundamental assumption is intuitively reasonable, since components corresponding to low variability likely say little about the data, but this is not always true.

• Principal components are orthogonal: PCA explicitly assumes that intrinsic (abstract) dimensions are orthogonal, which may not be true. However, this allowes us to use techniques from linear algebra such as the spectral decomposition and thereby, simplify our calculations.

---

End of Standard Section¶

---