Key Word(s): matplotlib, seaborn, plots, pandas
CS109A Introduction to Data Science
Lab 5: Exploratory Data Analysis, seaborn
, more Plotting¶
Harvard University
Fall 2019
Instructors: Pavlos Protopapas, Kevin Rader, and Chris Tanner
Material Preparation: Eleni Kaxiras.
#RUN THIS CELL
import requests
from IPython.core.display import HTML
styles = requests.get("https://raw.githubusercontent.com/Harvard-IACS/2018-CS109A/master/content/styles/cs109.css").text
HTML(styles)
# import the necessary libraries
%matplotlib inline
import numpy as np
import scipy as sp
import matplotlib as mpl
import matplotlib.cm as cm
import matplotlib.pyplot as plt
import pandas as pd
import time
pd.set_option('display.width', 500)
pd.set_option('display.max_columns', 200)
pd.set_option('display.notebook_repr_html', True)
import seaborn as sns
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
import warnings
warnings.filterwarnings('ignore')
%config InlineBackend.figure_format ='retina'
%%javascript
IPython.OutputArea.auto_scroll_threshold = 9999;
Learning Goals¶
By the end of this lab, you should be able to:
- know how to implement the different types of plots such as histograms, boxplots, etc, that were mentioned in class.
- have
seaborn
as well asmatplotlib
in your plotting toolbox.
This lab corresponds to lecture 6 up to 9 and maps to homework 3.
1 - Visualization Inspiration¶
Notice that in “Summers Are Getting Hotter,” above, the histogram has intervals for global summer temperatures on the x-axis, designated from extremely cold to extremely hot, and their frequency on the y-axis.
That was an infographic intended for the general public. In contrast, take a look at the plots below of the same data published at a scientific journal. They look quite different, don't they?
James Hansen, Makiko Sato, and Reto Ruedy, Perception of climate change. PNAS
2 - Implementing Various Types of Plots using matplotlib
and seaborn
.¶
Before you start coding your visualization, you need to decide what type of vizualization to use. A box plot, a histogram, a scatter plot, or something else? That will depend on the purpose of the plot (is it for performing an inspection on your data (EDA, or for showing your results/conclusions to people) and the number variables that you want to plot.
You have a lot of tools for plotting in Python. The basic one, of course, is matplotlib
and there are other libraries that are built on top of it, such as seaborn
, bokeh
, or altair
.
In this class we will continue using matplotlib
and also look into seaborn
. Those two libraries are the ones you should be using for homework.
Introduction to seaborn
¶
Seaborn
is a Python data visualization library based on matplotlib. It provides a high-level interface for drawing attractive and informative statistical graphics. The library provides a database of useful datasets for educational purposes that can be loaded by typing:
seaborn.load_dataset(name, cache=True, data_home=None, **kws)
For information on what these datasets are : https://github.com/mwaskom/seaborn-data
The plotting functions in seaborn
can be decided in two categories¶
'axes-level' functions, such as
regplot
,boxplot
,kdeplot
,scatterplot
,distplot
which can connect with thematplotlib
Axes object and its parameters. You can use that object as you would inmatplotlib
:f, (ax1, ax2) = plt.subplots(2) sns.regplot(x, y, ax=ax1) sns.kdeplot(x, ax=ax2) ax1 = sns.distplot(x, kde=False, bins=20)
'figure-level' functions, such as
lmplot
,factorplot
,jointplot
,relplot
. In this case,seaborn
organizes the resulting plot which may include several Axes in a meaningful way. That means that the functions need to have total control over the figure, so it isn't possible to plot, say, anlmplot
onto one that already exists. Calling the function always initializes a figure and sets it up for the specific plot it's drawing. These functions return an object of the typeFacetGrid
with its own methods for operating on the resulting plot.
To set the parameters for figure-level functions:
sns.set_context("notebook", font_scale=1, rc={"lines.linewidth": 2.5})
The Titanic dataset¶
The titanic.csv
file contains data for 887 passengers on the Titanic. Each row represents one person. The columns describe different attributes about the person including whether they survived, their age, their on-board class, their sex, and the fare they paid.
titanic = sns.load_dataset('titanic');
titanic.info();
titanic.columns
'embarked', 'who', 'adult_male', 'embark_town', 'alive', 'alone'
# your code here
# your code here
columns = ['embarked', 'who', 'adult_male', 'embark_town', 'alive', 'alone']
titanic = titanic.drop(columns=columns)
titanic
# your code here
# your code here
missing_decks = len(titanic[(pd.isna(titanic['deck']) == True)])
missing_decks
Histograms¶
Plotting one variable's distribution (categorical and continous)¶
The most convenient way to take a quick look at a univariate distribution in seaborn
is the distplot()
function. By default, this will draw a histogram and fit a kernel density estimate (KDE).
A histogram displays a quantitative (numerical) distribution by showing the number (or percentage) of the data values that fall in specified intervals. The intervals are on the x-axis and the number of values falling in each interval, shown as either a number or percentage, are represented by bars drawn above the corresponding intervals.
# What was the age distribution among passengers in the Titanic?
import seaborn as sns
sns.set(color_codes=True)
f, ax = plt.subplots(1,1, figsize=(8, 3));
ax = sns.distplot(titanic.age, kde=False, bins=20)
# bug
#ax = sns.distplot(titanic.age, kde=False, bins=20).set(xlim=(0, 90));
ax.set(xlim=(0, 90));
ax.set_ylabel('counts');
f, ax = plt.subplots(1,1, figsize=(8, 3))
ax.hist(titanic.age, bins=20);
ax.set_xlim(0,90);
# your code here
infants = len(titanic[(titanic.age < 3)])
children = len(titanic[(titanic.age >= 3) & (titanic.age < 10)])
print(f'There were {infants} infants and {children} children on board the Titanic')
Pandas trick: We want to creat virtual "bins" for readability and replace ranges of values with categories.
We will do this in an ad hoc way, it can be done better. For example in the previous plot we could set:
(age<3) = 'infants'
,(3
, (18
See matplotlib colors here.
# set the colors
cmap = plt.get_cmap('Pastel1')
young = cmap(0.5)
middle = cmap(0.2)
older = cmap(0.8)
# get the object we will change - patches is an array with len: num of bins
fig, ax = plt.subplots()
y_values, bins, patches = ax.hist(titanic.age, 10)
[patches[i].set_facecolor(young) for i in range(0,1)] # bin 0
[patches[i].set_facecolor(middle) for i in range(1,3)] # bins 1 and 2
[patches[i].set_facecolor(older) for i in range(3,10)] # 7 remaining bins
ax.grid(True)
fig.show()
Kernel Density Estimation¶
The kernel density estimate can be a useful tool for plotting the shape of a distribution. The bandwidth (bw) parameter of the KDE controls how tightly the estimation is fit to the data, much like the bin size in a histogram. It corresponds to the width of the kernels we plotted above. The default behavior tries to guess a good value using a common reference rule, but it may be helpful to try larger or smaller values.
sns.kdeplot(titanic.age, bw=0.6, label="bw: 0.6", shade=True, color="r");
sns.kdeplot(titanic.age, bw=2, label="bw: 2", shade=True);
# your code here
sns.kdeplot(titanic.fare, bw=0.5, label="bw: 0.5", shade=True);
You can mix elements of matplotlib
such as Axes with seaborn
elements for a best use of both worlds.¶
import seaborn as sns
sns.set(color_codes=True)
x1 = np.random.normal(size=100)
x2 = np.random.normal(size=100)
fig, ax = plt.subplots(1,2, figsize=(15,5))
# seaborn goes in first subplot
sns.set(font_scale=0.5)
sns.distplot(x1, kde=False, bins=15, ax=ax[0]);
sns.distplot(x2, kde=False, bins=15, ax=ax[0]);
ax[0].set_title('seaborn Graph Here', fontsize=14)
ax[0].set_xlabel(r'$x$', fontsize=14)
ax[0].set_ylabel(r'$count$', fontsize=14)
# matplotlib goes in second subplot
ax[1].hist(x1, alpha=0.2, bins=15, label=r'$x1$');
ax[1].hist(x2, alpha=0.5, bins=15, label=r'$x2$');
ax[1].set_xlabel(r'$x$', fontsize=14)
ax[1].set_ylabel(r'$count$', fontsize=14)
ax[1].set_title('matplotlib Graph Here', fontsize=14)
ax[1].legend(loc='best', fontsize=14);
Introduding the heart disease dataset.¶
More on this in the in-class exercise at the end of the notebook.
columns = ["age", "sex", "cp", "restbp", "chol", "fbs", "restecg",
"thalach", "exang", "oldpeak", "slope", "ca", "thal", "num"]
heart_df = pd.read_csv('../data/heart_disease.csv', header=None, names=columns)
heart_df.head()
# seaborn
ax = sns.boxplot(x='age', data=titanic)
#ax = sns.boxplot(x=titanic['age']) # another way to write this
ax.set_ylabel(None);
ax.set_xlabel('age', fontsize=14);
ax.set_title('Distribution of age in the Titanic', fontsize=14);
Two variables¶
# your code here
# two variables seaborn
ax = sns.boxplot(x='class', y='age', data=titanic)
# two variable boxplot in pandas
titanic.boxplot('age',by='class')
f, ax = plt.subplots(1,1, figsize=(10, 5))
sns.scatterplot(x="fare", y="age", data=titanic, ax=ax);
sns.jointplot("fare", "age", data=titanic, s=40, edgecolor="w", linewidth=1)
You may control the seaborn
Figure aesthetics.
# matplotlib
fig, ax = plt.subplots(1,1, figsize=(10,6))
ax.scatter(heart_df['age'], heart_df['restbp'], alpha=0.8);
ax.set_xlabel(r'$Age (yrs)$', fontsize=15);
ax.set_ylabel(r'Resting Blood Pressure (mmHg)', fontsize=15);
ax.set_title('Age vs. Resting Blood Pressure', fontsize=14)
plt.show();
Plotting the distribution of three variables¶
f, ax = plt.subplots(1,1, figsize=(10, 5))
sns.scatterplot(x="fare", y="age", hue="survived", data=titanic, ax=ax);
Plotting the distribution of four variables (going too far?)¶
Use size=
for the fourth variable
# your code here
f, ax = plt.subplots(1,1, figsize=(10, 5))
sns.scatterplot(x="fare", y="age", hue="survived", size="sex", data=titanic, ax=ax);
Pairplots¶
titanic.columns
to_plot = ['age', 'fare', 'survived', 'deck']
df_to_plot = titanic.loc[:,to_plot]
sns.pairplot(df_to_plot);
from pandas.plotting import scatter_matrix
scatter_matrix(df_to_plot, alpha=0.8, figsize=(10, 10), diagonal='kde');
Plotting Categorical Variables¶
titanic = sns.load_dataset('titanic')
f, ax = plt.subplots(figsize=(7, 3));
ax = sns.countplot(y="deck", data=titanic, color="c");
ax.set_title('Titanic');
ax = sns.countplot(x="class", data=titanic)
ax.set_title('Titanic');
fig, ax = plt.subplots(figsize=(10,6)) # Create figure object
sns.set_context("notebook", font_scale=1, rc={"lines.linewidth": 2.5})
ax = sns.countplot(x="deck", data=titanic)
sns.set(style="ticks", palette="muted")
sns.relplot(x="age", y="deck", col="class", data=titanic);
sns.set_context("notebook", font_scale=1.5, rc={"lines.linewidth": 2.5})
sns.pairplot(data=titanic, hue="deck");
Introduction to pandas
plotting.¶
There is plotting functionality built in pandas
. Look for it in the pandas
"encyclopedia", a mere 2883-page pdf from the creator Wes McKinney: pandas documentation (pdf)
Example: The value_counts()
Series method and top-level function computes a histogram of a 1D array of values. It can
also be used as a function on regular arrays.
Reminder: DataFrame: “index” (axis=0, default), “columns” (axis=1)
df = pd.DataFrame(np.random.randn(1000, 4), columns=['A', 'B', 'C', 'D'])
df.head()
# cumulative sum adds column values as it goes
df = df.cumsum()
df.head()
plt.figure();
df.plot();
plt.legend(loc='best');
ts = pd.Series(np.random.randn(1000),
index=pd.date_range('1/1/2000', periods=1000))
df = pd.DataFrame(np.random.randn(1000, 4),
index=ts.index, columns=list('ABCD'))
df = df.cumsum()
plt.figure();
df.plot();
Plotting methods allow for a handful of plot styles other than the default line plot. These methods can be provided as the kind keyword argument to plot(), and include:
- ‘bar’ or ‘barh’ for bar plots
- ‘hist’ for histogram
- ‘box’ for boxplot
- ‘kde’ or ‘density’ for density plots
- ‘area’ for area plots
- ‘scatter’ for scatter plots
- ‘hexbin’ for hexagonal bin plots
- ‘pie’ for pie plots
In addition to these kind s, there are the DataFrame.hist()
, and DataFrame.boxplot()
methods, which use a separate interface. scatter_matrix
in pandas.plotting takes a Series or DataFrame as an argument.
Bar Plots¶
plt.figure();
df.iloc[0].plot(kind='bar');
df2 = pd.DataFrame(np.random.rand(10, 4), columns=['a', 'b', 'c', 'd'])
df2
df2.plot.bar();
# horizontal bar plot
df2.plot.barh(stacked=False);
Histograms¶
df4 = pd.DataFrame({'a': np.random.randn(1000) + 1, 'b': np.random.randn(1000),
'c': np.random.randn(1000) - 1}, columns=['a', 'b', 'c'])
plt.figure();
df4.plot.hist(alpha=0.5, stacked=False, bins=60);
Boxplots¶
color = {'boxes': 'DarkGreen', 'whiskers': 'DarkOrange',
'medians': 'DarkBlue', 'caps': 'Gray'}
df = pd.DataFrame(np.random.rand(10, 5), columns=['A', 'B', 'C', 'D', 'E'])
df.plot.box(color=color );
Area plots¶
You can create area plots with Series.plot.area()
and DataFrame.plot.area()
. Area plots are stacked
by default. To produce stacked area plot, each column must be either all positive or all negative values.
df = pd.DataFrame(np.random.rand(10, 4), columns=['a', 'b', 'c', 'd'])
df.plot.area(stacked=True);
df.plot.area(stacked=False);
Scatterplot¶
Scatter plot can be drawn by using the DataFrame.plot.scatter() method. Scatter plot requires numeric columns for the x and y axes. These can be specified by the x and y keywords.
ax = df.plot.scatter(x='a', y='b', color='DarkBlue', label='Group 1');
df.plot.scatter(x='c', y='d', color='DarkGreen', label='Group 2', ax=ax);
pandas
Tricks¶
The copy() method on pandas objects copies the underlying data (though not the axis indexes, since they are immutable) and returns a new object. Note that it is seldom necessary to copy objects. For example, there are only a handful of ways to alter a DataFrame in-place:
- Inserting, deleting, or modifying a column.
- Assigning to the index or columns attributes.
- For homogeneous data, directly modifying the values via the values attribute or advanced indexing.
To be clear, no pandas method has the side effect of modifying your data; almost every method returns a new object, leaving the original object untouched. If the data is modified, it is because you did so explicitly
4 - Group Exercise: 1/2 hour in the Life of a Cardiologist¶
Try each exercise on your own and then discuss with your peers sitting at your table. Feel free to engage the TFs and instructors as well.
Visualize and explore the data. Use .describe()
to look at your data and also examine if you have any missing values.
What is the actual number of feature variables after converting categorical variables to dummy ones?
List of available variables (includes target variable num
):
- age: continuous
- sex: categorical, 2 values {0: female, 1: male}
- cp (chest pain type): categorical, 4 values {1: typical angina, 2: atypical angina, 3: non-angina, 4: asymptomatic angina}
- restbp (resting blood pressure on admission to hospital): continuous (mmHg)
- chol (serum cholesterol level): continuous (mg/dl)
- fbs (fasting blood sugar): categorical, 2 values {0: <= 120 mg/dl, 1: > 120 mg/dl}
- restecg (resting electrocardiography): categorical, 3 values {0: normal, 1: ST-T wave abnormality, 2: left ventricular hypertrophy}
- thalach (maximum heart rate achieved): continuous
- exang (exercise induced angina): categorical, 2 values {0: no, 1: yes}
- oldpeak (ST depression induced by exercise relative to rest): continuous
- slope (slope of peak exercise ST segment): categorical, 3 values {1: upsloping, 2: flat, 3: downsloping}
- ca (number of major vessels colored by fluoroscopy): discrete (0,1,2,3)
- thal: categorical, 3 values {3: normal, 6: fixed defect, 7: reversible defect}
- num (diagnosis of heart disease): categorical, 5 values {0: less than 50% narrowing in any major vessel, 1-4: more than 50% narrowing in 1-4 vessels}
# load the dataset
heart_df = pd.read_csv('../data/heart_disease.csv', header=None, names=columns)
heart_df.head()
Answer the following question using plots¶
- At what ages do people seek cardiological exams?
- Do men seek help more than women?
- Examine the variables. How do they relate to one another?
- (Variation on 02): What % of men and women seek cardio exams?
- Does resting blood pressure increase with age?
Pandas trick: .replace
The response variable (num) is categorical with 5 values, but we don't have enough data to predict all the categories.
Therefore we'll replace num
with hd
(heart disease): categorical, 2 values {0: no, 1: yes}.
Use the code below (take a minute to understand how it works, it's very useful!):
# Replace response variable values with a binary response (1: heart disease(hd) or 0: not)
#heart_df['num'].replace(to_replace=[1,2,3,4],value=1,inplace=True)
# Rename column for clarity
#heart_df = heart_df.rename(columns = {'num':'hd'})
heart_df.head()
# look at the features
heart_df.info();
heart_df.describe()
At this point you should split in train and test set and work only on the train!!. For simplicity we will not do this in the solutions.
# your code here
# 01. what ages do people seek cardiological exams?
%load solutions/q01.py
# your code here
# 02. do men seek help more than women?
# %load solutions/q02.py
# your code here
# 03. Examine the variables. How do they relate to one another?
# %load solutions/q03.py
# your code here
# 04. What percentage of men and women seek cardio exams?
# %load solutions/q04.py
# your code here
# 05. Does resting blood pressure increase with age?
# %load solutions/q05.py
Bonus: Find the hidden pattern¶
Read the following file into a pandas
Dataframe: '../data/mystery.csv' and plot it. How does it look? You should see a beautiful pattern. If not, think of ways to fix the issue.
mystery = pd.read_csv('../data/mystery.csv', sep=' ', header=None)
mystery.head()
# your code here
# %load solutions/mystery.py