CS-109A Introduction to Data Science

Lecture 3: EDA and Visualization Demo¶

Harvard University
Fall 2018
Instructors: Pavlos Protopapas and Kevin Rader
Author: Rahul Dave

---

---

EDA and visualization¶

In the last lecture we focussed on the machinery needed to get data into a tabular form. In this lecture we focus on the visualization part of Exploratory Data Analysis. This notebook accompanies the slides. It is strongly recommended that you see the lecture/read the slides and then work on this notebook. The textual descriptions are sparse here and the code will become much clearer in the context of visualization best practices.

The seaborn API is changing, and seaborn.apionly is being deprecated. The default will soon behave like apionly and not change the standard matplotlib color scheme and defaults. Here we choose apionly to make sure we have to do everything explicitly.

Getting the mtcars dataset into shape¶

The documentation for this data is here but I have extracted some relevant parts below:

Description

The data was extracted from the 1974 Motor Trend US magazine, and comprises fuel consumption and 10 aspects of automobile design and performance for 32 automobiles (1973–74 models).

Usage

mtcars
Format

A data frame with 32 observations on 11 variables.

[, 1]   mpg Miles/(US) gallon
[, 2]   cyl Number of cylinders
[, 3]   disp    Displacement (cu.in.)
[, 4]   hp  Gross horsepower
[, 5]   drat    Rear axle ratio
[, 6]   wt  Weight (1000 lbs)
[, 7]   qsec    1/4 mile time
[, 8]   vs  V/S
[, 9]   am  Transmission (0 = automatic, 1 = manual)
[,10]   gear    Number of forward gears
[,11]   carb    Number of carburetors
Source

Henderson and Velleman (1981), Building multiple regression models interactively. Biometrics, 37, 391–411.

There is an ugly poorly named column right here. Lets fix that.

We parse out a maker, which we shall lkater use to group cars.

This is what the dataframe looks like now:

We can construct the av_mpg series by using the "split-apply-combine" paradigm and summarizing within group data by a mean:

Simple EDA¶

(as provided by Chris, a previous head-TF for cs109)

He says:

I'd like to suggest a basic rubric for the early stages of exploratory data analysis in Python. This isn't universally applicable, but it does cover many patterns which recur in several data analysis contexts. It's useful to keep this rubric in mind when encountering a new dataset.

The basic workflow is as follows:

1. Build a DataFrame from the data (ideally, put all data in this object)
2. Clean the DataFrame. It should have the following properties:
   • Each row describes a single object
   • Each column describes a property of that object
   • Columns are numeric whenever appropriate
   • Columns contain atomic properties that cannot be further decomposed
3. Explore global properties. Use histograms, scatter plots, and aggregation functions to summarize the data.
4. Explore group properties. Use groupby and small multiples to compare subsets of the data.

This process transforms your data into a format which is easier to work with, gives you a basic overview of the data's properties, and likely generates several questions for you to followup in subsequent analysis.

So far we have built the dataframe, and carried out very minimal cleaning (renaming) in this dataframe.

Exploring global properties¶

So lets focus on visualizing global properties of the data set below. For now, we shall only focus on mpg to illustrate the concepts, but you want to be doing this for all the columns. It may identify interesting properties and even errors in the data.

While we do this we shall see several examples of the matplotlib plotting experience.

We first use seaborn to set the global matplotlib plotting context. Here we set it to notebook which makes for reasonable sized graphics.

We ask what the defaukt color palette is. Since we imported sns.apionly, we get the default matplotlib color palette.

Now we see maplotlib's default color palette, 'viridis'.

We can choose other palettes. The default palette is for qualitative data. It creates a color cycle we use in our plots. In other words, as we plot multiple things in a plot, it will use these colors one-by-one.

Bar Charts¶

We see that Pandas series very niftily give us bar graphs.

The default, which comes from matplotlib, is chart-junky (although not as much as matlab, or as much as it used to be in earlier versions of matplotlib). But lets clean it.

And more....

Its important to understand what is going on under the hood. Try and write sns.despine on your own by looking up the matplotlib documentation.

Histograms¶

Numerical data leads to distributions, and distributions to histograms. Here is the Pandas default

And matplotlib:

Lets style it a bit with seaborn. The with syntax followed by indentation is a python construct called a "context manager". Here it sets up a while axes style only for the code that is inside the context (ie, indented).

What if I wanted to get the benefit of the default seaborn style and colors?

I can reset_defaults back to the matplotlib defaults.

Here are the most commonly used matplotlib plotting routines.

Here we use a white grid. We also illustrate how a series can be converted to a numpy array with the values method (dfcars.mpg.values) and how x and y limits can be set (the difference in the way xlim and ylim are called is on purpose and illustrates two ways you can set the limits). In the hist function you can also change the number of bins. We also show how to label a plot and obtain a legend from the plot. A vertical line is drawn at the mean.

One can set bins using a list, and also label the histogram (not recommended but just for illustration). We also illustrate how to capture the color used and use it again

Seaborn very handily provides Kernel Density Estimates (KDE) which try to infer a probability distribution from the data. This is more useful when you have lots of data.

And a histogram and a kdeplot together:

Plotting features against other features¶

Sometimes we want to see co-variation amongst our columns. A scatter-plot does this for us.

We can also use plot without lines:

But what if we want to save our figure into a file? The extension tells you how it will be saved..and note that the savefig needs to be in the same cell as the plotting commands. Go look at the files..

Trend¶

The correlation that we saw might suggest a trend. We can capture it with a "regression". We'll learn more about regressions soon, but we show a quadratic fit here with a 1 standard deviation bar to show the graphics aspect of this. Also see sns.regplot.

Group Properties¶

Such "co-variational" plots, and for that matter, even single-variable plots, are even more interesting when we look at them conditional upon the value of a categorical variable.

Such conditionality is behind the notion of grouping, where we group our data by various values of categorical variables, for example, whether our cars have an automatic transmission or not.

Grouping of one outcome variable¶

This notion of grouping based on combinations of factors is used by seaborn to make various easy-to-see exploratory visualizations for us. We shall first see how to make these plots quick-and-dirty using seaborn and then see how we make make these plots ourselves.

First, we make a boxplot of mpg, grouped by transmission style

The box shows the quartiles of the distribution This can also be done using a violin-plot, which uses KDE to show the distribution itself:

Automatics surveyed have a higher range of mpg, with higher fuel efficiency significant at 1 sigma, and we may want to investigate if this is a selection effect or whether there is some true impact of an automatic transmission. Further subgrouping the data is one way to do this.

Split violin plots can be used to do multiple categorical variables

One can see that the difference in mpg is starker between 6 and 8 cylinder cars, for manual transmissions. And that the large-range effect in automatics is coming almost entirely through 4-cylinder cars. What about the better mpg for automatics? Mostly in the 4 and 6 cyclinders. But we ought to see how representative these are in our sample. We'll show how to do this soon but a cross-tabulation, combined with the graph above, gives us some idea that the poor miles per gallon for manual transmission may be coming from the preponderance of 8-cylinder manual cars in our dataset.

Faceting for general grouping¶

Seaborn provides a nice construct: the FacetGrid. You decide what variables to facet over, and then decide the kind of plot you want. Here we want hue to be am, and different columns in the plot grid to be cylinders. We then ask for a facet plot of mpg against wt scatter.

Such plots are often called small multiple plots. They repeat the same plot based on categories, making sure that all plotting parameters are the same so that we have direct comparability.

We can see that the "regression-like" effect is cleanest for automatic transmissions in 4 cylinder cars.

SPLOM, or Scatter Plot Matrix¶

We'd get tired if we had to this on a 2-by-2 basis for every pair of continuously co-varying features. The PairGrid, colorable by transmission type, allows us to do this comparison for 5 continuous features here, with the diagonal being a kernel density estimate.

In many places, for example mpg vs disp, there see to be two separate trends for the different transmissions. This will (as we shall see later in this course) suggest to us the addition of a transmission term as a indicator variable in regressions for mpg against various features. This changes the intercept of the regression. But the trends have different slopes as well, which suggests that disp may interact with am, the transmission indicator to create a varying slope as well.

Correlation¶

The SPLOM seems to suggest correlations. Well, so lets calculate it:

Since correlations range from -1 to 1 through 0, a diverging palette is probably our best bet.

We use maptplotlibs correlation plot. You'll land up doing plots like this for both EDA and do see misclassification from your machine learning algorithms. In other words, EDA is even useful at the analysis stage.

KDE plots and sequential palettes.¶

Here we make a KDE plot of a multivariate normal distribution. Since a probability density is strictly positive, with values near 0 not being so interesting, a sequential palette is our ticket. Seaborn will by default provide such a palette for KDE plots, but you can use your own!

Matplotlib and multiple plots: Small Multiples¶

There are many cases where we want to see plots side by side. The SPLOMS and Facet grids that seaborn makes for us are an example.But we should know how to do this on our own.

Here is a simple example of a plot with one column and 3 rows. It illustrates one way of doing this.

Small multiples, youself¶

Here is another way, which to me is far clearer that the add_subplot way. It basically creates an array of plots and zips this array up with the various data grouped by categories.

The Interest in Science Problem from lecture slides¶

Dataframe construction:

Multiple Pie Charts¶

Before and after bar charts¶

Stacked Charts¶

This is hard to read. We wznt to compare before and after easily. Sometimes the solution is a transpose!!

We have to play some games to get the ordering right...

Now we see that the blue and greens have it. Try and improve this plot. It still has too much chartjunk.

Difference Bar chart¶

Sometimes a small data transformation makes the effect blindingly obvious. Here we just make a bar chart of the change!

Bacteria Dataset¶

For those want to play...