CS-109A Introduction to Data Science

Lab 1: Introduction to Python and its Numerical Stack¶

Harvard University
Fall 2018
Instructors: Pavlos Protopapas and Kevin Rader
Lab Instructor: Rahul Dave
Authors: Rahul Dave, David Sondak, Will Claybaugh, Pavlos Protopapas

---

Programming Expectations¶

All assignments for this class will use Python and the browser-based iPython notebook format you are currently viewing. Python experience is not a prerequisite for this course, as long as you are comfortable learning on your own as needed.

Note though that the programming at the level of CS 50 is a prerequisite for this course. If you have concerns about the prerequisite, please come speak with any of the instructors.

We will refer to the Python 3 documentation in this lab and throughout the course. There are also many introductory tutorials to help build programming skills, which we are listed in the last section of this lab.

Table of Contents¶

0. Learning Goals
1. Getting Started
2. Lists
3. Simple Functions
4. Numpy
5. Scipy.stats and plotting distributions
6. Conclusions

Additional Stuff¶

99. Dictionaries
100. Reading CSVs using pandas

Learning Goals¶

This introductory lab is a condensed introduction to Python numerical programming. By the end of this lab, you will feel more comfortable:

• Writing short Python code using functions, loops, lists, numpy arrays, and dictionaries.

• Manipulating Python lists and numpy arrays and understanding the difference between them.

• Using probability distributions from scipy.stats

• Making very simple plots using matplotlib

• Reading and writing CSV files using pandas

• Learning and reading Python documentation.

Lab 1 relates to material in lecture 0,1,2,3 and homework 0.

Part 1: Getting Started¶

Importing modules¶

All notebooks should begin with code that imports modules, collections of built-in, commonly-used Python functions. Below we import the Numpy module, a fast numerical programming library for scientific computing. Future labs will require additional modules, which we'll import with the same import MODULE_NAME as MODULE_NICKNAME syntax.

Now that Numpy has been imported, we can access some useful functions. For example, we can use mean to calculate the mean of a set of numbers.

to calculate the mean of 1.2, 2, and 3.3.

The code above is not particularly efficient, and efficiency will be important for you when dealing with large data sets. Later and in lab 2 we will see more efficient options.

Calculations and variables¶

The last line in a cell is returned as the output value, as above. For cells with multiple lines of results, we can display results using print, as can be seen below.

We can store integer or floating point values as variables. The other basic Python data types -- booleans, strings, lists -- can also be stored as variables.

(more on types here: http://www.diveintopython3.net/native-datatypes.html)

Here is the storing of a list (more about what a list is later):

Think of a variable as a label for a value, not a box in which you put the value

(image taken from Fluent Python by Luciano Ramalho)

This DOES NOT create a new copy of a. It merely puts a new label on the memory at a, as can be seen by the following code:

Tuples

Multiple items on one line in the interface are returned as a tuple, an immutable sequence of Python objects.

We can obtain the type of a variable, and use boolean comparisons to test these types.

For reference, below are common arithmetic and comparison operations.

.

• The first element is an integer of your choice
• The second element is a float of your choice
• The third element is the sum of the first two elements
• The fourth element is the difference of the first two elements
• The fifth element is first element divided by the second element

Display the output of tup. What is the type of the variable tup? What happens if you try and chage an item in the tuple?

Part 2: Lists¶

Much of Python is based on the notion of a list. In Python, a list is a sequence of items separated by commas, all within square brackets. The items can be integers, floating points, or another type. Unlike in C arrays, items in a Python list can be different types, so Python lists are more versatile than traditional arrays in C or in other languages.

Let's start out by creating a few lists.

Lists in Python are zero-indexed, as in C. The first entry of the list has index 0, the second has index 1, and so on.

What happens if we try to use an index that doesn't exist for that list? Python will complain!

You can find the length of a list using the builtin function len:

Indexing on lists¶

And since Python is zero-indexed, the last element of float_list is

It is more idiomatic in python to use -1 for the last element, -2 for the second last, and so on

We can use the : operator to access a subset of the list. This is called slicing.

Below is a summary of list slicing operations:

You can slice "backwards" as well:

You can also slice with a stride:

We can iterate through a list using a loop. Here's a for loop.

Or, if we like, we can iterate through a list using the indices using a for loop with in range. This is not idiomatic and is not recommended, but accomplishes the same thing as above.

What if you wanted the index as well?

Use the built-in python method enumerate, which can be used to create a list of tuples with each tuple of the form (index, value).

Appending and deleting¶

We can also append items to the end of the list using the + operator or with append.

Go and run the cell with float_list.append a second time. Then run the next line. What happens?

To remove an item from the list, use del.

List Comprehensions¶

Lists can be constructed in a compact way using a list comprehension. Here's a simple example.

And here's a more complicated one, requiring a conditional.

This is entirely equivalent to creating comp_list1 using a loop with a conditional, as below:

The list comprehension syntax

[expression for item in list if conditional]

is equivalent to the syntax

for item in list:
    if conditional:
        expression

1. Using for loops and conditional if statements.
2. (Stretch Goal) Using a list comprehension. You should be able to do this in one line of code, and it may be helpful to look up the function all in the documentation.

Part 4: Simple Functions¶

A function object is a reusable block of code that does a specific task. Functions are all over Python, either on their own or on other objects. To invoke a function func, you call it as func(arguments).

We've seen built-in Python functions and methods. For example, len and print are built-in Python functions. And at the beginning, you called np.mean to calculate the mean of three numbers, where mean is a function in the numpy module and numpy was abbreviated as np. This syntax allows us to have multiple "mean" functions in different modules; calling this one as np.mean guarantees that we will pick up numpy's mean function, as opposed to a mean function from a different module.

Methods¶

A function that belongs to an object is called a method. By "object" here we mean an "instance" of a list, or integer, or floating point variable.

An example of this is append on an existing list. In other words, a method is a function on an instance of a type of object (also called class, in this case, list type).

User-defined functions¶

We'll now learn to write our own user-defined functions. Below is the syntax for defining a basic function with one input argument and one output. You can also define functions with no input or output arguments, or multiple input or output arguments.

def name_of_function(arg):
    ...
    return(output)

We can write functions with one input and one output argument. Here are two such functions.

What if you want to return two variables at a time? The usual way is to return a tuple:

Lambda functions¶

Often we quickly define mathematical functions with a one-line function called a lambda function. Lambda functions are great because they enable us to write functions without having to name them, ie, they're anonymous.
No return statement is needed.

Refactoring using functions¶

In an exercise from Lab 0, you wrote code that generated a list of the prime numbers between 1 and 100. For the excercise below, it may help to revisit that code.

Part 5: Introduction to Numpy¶

Scientific Python code uses a fast array structure, called the numpy array. Those who have worked in Matlab will find this very natural. For reference, the numpy documention can be found here.

Let's make a numpy array.

The shape array of an array is very useful (we'll see more of it later when we talk about 2D and higher dimensional arrays).

Numpy arrays are typed. This means that by default, all the elements will be assumed to be of one type

Numpy arrays are listy (i.e. they act like lists)! Below we compute length, slice, and iterate.

In general you should manipulate numpy arrays by using numpy module functions (np.mean, for example). This is for efficiency purposes, and a discussion about this will happen in Lab2.

You can calculate the mean of the array elements either by calling the method .mean on a numpy array or by applying the function np.mean with the numpy array as an argument.

The way we constructed the numpy array above seems redundant. After all we already had a regular python list. Indeed, it is the other ways we have to construct numpy arrays that make them super useful.

There are many such numpy array constructors. Here are some commonly used constructors. Look them up in the documentation.

Numpy gains a lot of its efficiency from being typed. That is, all elements in the array have the same type, such as integer or floating point. The default type, as can be seen above, is a float of size appropriate for the machine (64 bit on a 64 bit machine).

Often you will want random numbers. Use the random constructor!

You can generate random numbers from a normal distribution with mean 0 and variance 1:

You can sample with and without replacement from an array. Lets first construct a grid:

Without replacement

With replacement:

Numpy supports vector operations¶

What does this mean? It means that instead of adding two arrays, element by element, you can just say: add the two arrays. Note that this behavior is very different from python lists.

On some computer chips this addition actually happens in parallel, so speedups can be high. But even on regular chips, the advantage of greater readability is important.

Numpy supports a concept known as broadcasting, which dictates how arrays of different sizes are combined together. There are too many rules to list here, but importantly, multiplying an array by a number multiplies each element by the number. Adding a number adds the number to each element.

This means that if you wanted the distribution $N(5, 7)$ you could do:

Multiplying two arrays multiplies them element-by-element

You might have wanted to compute the dot product instead:

You can also use the @ operator for this purpose

Part 6: Probabilitiy Distributions from scipy.stats¶

Since we'll be using many distributions, we'll want to access the pdf/pmf functions of these distributions and obtain samples from them. We already saw how to obtain samples from the continuous uniform and normal distributions. But we might want to obtain their pdfs as well.

scipy.stats allows us to obtain the pdf function as well as samples. The programming interface is identical for all the distributions.

To plot samples from and the pdfs of these distributions, we'll first import matplotlib, python's plotting library.

The

%matplotlib inline incantation ensures that plots are rendered inline in the browser.

Lets get the normal distribution namespace from scipy.stats. Docs here.

Lets create 1000 points between -10 and 10

Lets get the pdf of a normal distribution with a mean of 1 and standard deviation 3 and plot it using the grid points computed before...

And you can get random variables using the rvs function.

We can use a more instance based way of getting both the pdf and samples. The documentation calls this instance a "frozen" distribution:

We can now plot a histogram of the samples using matplotlib:

(see docs on plt.hist by typing

?plt.hist

in a cell by itself)

By default the histogram gives us counts. We can re-normalize these counts to get an approximation to the probability distribution from the samples.

Part 7: Conclusions¶

For more practice exercises (with solutions) and discussion, see this page. Some of these exercises are particularly relevant. Check them out!

Don't forget to look up Jake's book.

Finally, we would like to suggest using Chris Albon's web site as a reference. Lots of useful information there.

Additional Stuff¶

Part 1: Dictionaries¶

A dictionary is another storage container. Like a list, a dictionary is a sequence of items. Unlike a list, a dictionary is unordered and its items are accessed with keys and not integer positions.

Dictionaries are the closest container we have to a database.

Let's make a dictionary with a few Harvard courses and their corresponding enrollment numbers.

One can obtain the value corrsponding to a key thus:

Or thus, which allows for the key to not be in the dictionary

All sorts of iterations are supported:

We can iterate over the tuples obtained above: (to read more about how the print formatting works , look at https://docs.python.org/3/library/stdtypes.html#old-string-formatting and https://docs.python.org/3/tutorial/inputoutput.html)

Simply iterating over a dictionary gives us the keys. This is useful when we want to do something with each item:

The above is an actual copy to another part of memory, unlike, second_dict = enroll2017_dict which would have made both variables label the same memory location.

In this example, the keys were strings corresponding to course names. Keys don't have to be strings though.

Like lists, you can construct dictionaries using a dictionary comprehension, which is similar to a list comprehension. Notice the brackets {} and the use of zip, which is another iterator that combines two lists together.

You can also create dictionaries using the constructor function dict.

Part 2: Introduction to Pandas¶

Often data is stored in comma separated values (CSV) files. For the remainder of this lab, we'll be working with automobile data, where we've extracted relevant parts below.

Note that CSV files can be output by any spreadsheet software, and are plain text, hence are a great way to share data.

Importing data with pandas¶

Now let's read in our automobile data as a pandas dataframe structure.

What we have now is a spreadsheet with indexed rows and named columns, called a dataframe in pandas. dfcars is an instance of the pd.DataFrame class, created by calling the pd.read_csv "constructor function".

The take-away is that dfcars is a dataframe object, and it has methods (functions) belonging to it. For example, df.head() is a method that shows the first 5 rows of the dataframe.

A pandas dataframe is a set of columns pasted together into a spreadsheet, as shown in the schematic below, which is taken from the cheatsheet above. The columns in pandas are called series objects.

Let's look again at the first five rows of dfcars.

Notice the poorly named first column: "Unnamed: 0". Why did that happen?

The first column, which seems to be the name of the car, does not have a name. Here are the first 3 lines of the file:

"","mpg","cyl","disp","hp","drat","wt","qsec","vs","am","gear","carb"
"Mazda RX4",21,6,160,110,3.9,2.62,16.46,0,1,4,4
"Mazda RX4 Wag",21,6,160,110,3.9,2.875,17.02,0,1,4,4

Lets clean that up:

In the above, the argument columns = {"Unnamed: 0": "name"} of rename changed the name of the first column in the dataframe from Unnamed: 0 to name.

Lets save this cleaned dataframe out to a CSV file.

The output will look something like this:

name,mpg,cyl,disp,hp,drat,wt,qsec,vs,am,gear,carb
Mazda RX4,21.0,6,160.0,110,3.9,2.62,16.46,0,1,4,4
Mazda RX4 Wag,21.0,6,160.0,110,3.9,2.875,17.02,0,1,4,4
Datsun 710,22.8,4,108.0,93,3.85,2.32,18.61,1,1,4,1
Hornet 4 Drive,21.4,6,258.0,110,3.08,3.215,19.44,1,0,3,1
Hornet Sportabout,18.7,8,360.0,175,3.15,3.44,17.02,0,0,3,2
Valiant,18.1,6,225.0,105,2.76,3.46,20.22,1,0,3,1

To access a series (column), you can use either dictionary syntax or instance-variable syntax.

Dictionary syntax is very useful when column names have spaces: Python variables cannot have spaces in them.

You can get a numpy array of values from the Pandas Series:

And we can produce a histogram from these values

But pandas is very cool: you can get a histogram directly:

We can also get sub-dataframes by choosing a set of series. We pass a list of the columns we want as "dictionary keys" to the dataframe.

Scatter plots¶

We often want to see co-variation among our columns, for example, miles/gallon versus weight. This can be done with a scatter plot.

You could have used plot instead of scatter.

Usually we use plt.show() at the end of every plot to display the plot. Our magical incantation %matplotlib inline takes care of this for us, and we don't have to do it in the Jupyter notebook. But if you run your Python program from a file, you will need to explicitly have a call to show. We include it for completion.

Suppose we'd like to save a figure to a file. We do this by including the savefig command in the same cell as the plotting commands. The file extension tells you how the file will be saved.

And this is what the saved png looks like. Code in Markdown to show this is:

![](images/foo1.png)

Below is a summary of the most commonly used matplotlib plotting routines.

Create a scatter plot showing the co-variation between two columns of your choice. Label the axes. See if you can do this without copying and pasting code from earlier in the lab. What can you conclude, if anything, from your scatter plot?