CS 109A/STAT 121A/AC 209A /CSCI E-109A¶

Lab 0: Introduction to Python¶

Harvard University
Fall 2017
Instructors: Pavlos Protopapas, Kevin Rader, Rahul Dave, Margo Levine

Programming Expectations¶

All assignments for this class will use Python and the browser-based iPython notebook format you are currently viewing. Python experience is not a prerequisite for this course, as long as you are comfortable learning on your own as needed. While we strive to make the programming component of this course straightforward, we won't devote much time to teaching programming or Python syntax.

Note though that the programming at the level of CS 50 is a prerequisite for this course. If you have concerns about the prerequisite, please come speak with any of the instructors.

We will refer to the Python 3 documentation in this lab and throughout the course. There are also many introductory tutorials to help build programming skills, which we are listed in the last section of this lab.

Table of Contents¶

0. Learning Goals
1. Getting Started
2. Lists
3. Strings and Listiness
4. Dictionaries
5. Functions
6. Text Analysis of Hamlet
7. References

Part 0: Learning Goals¶

This introductory lab is a condensed tutorial in Python programming. By the end of this lab, you will feel more comfortable:

• Writing short Python code using functions, loops, arrays, dictionaries, strings, if statements.

• Manipulating Python lists and recognizing the listy properties of other Python containers.

• Learning and reading Python documentation.

Lab 0 relates to material in lecture 0,1,2,3 and homework 0.

Part 1: Getting Started¶

Importing modules¶

All notebooks should begin with code that imports modules, collections of built-in, commonly-used Python functions. Below we import the Numpy module, a fast numerical programming library for scientific computing. Future labs will require additional modules, which we'll import with the same import MODULE_NAME as MODULE_NICKNAME syntax.

Now that Numpy has been imported, we can access some useful functions. For example, we can use mean to calculate the mean of a set of numbers.

to calculate the mean of 1.2, 2, and 3.3.

The code above is not particularly efficient, and efficiency will be important for you when dealing with large data sets. In Lab 1 we will see more efficient options.

Calculations and variables¶

At the most basic level we can use Python as a simple calculator.

Notice integer division (//) and floating-point error below!

The last line in a cell is returned as the output value, as above. For cells with multiple lines of results, we can display results using print, as can be seen below.

We can store integer or floating point values as variables. The other basic Python data types -- booleans, strings, lists -- can also be stored as variables.

Here is the storing of a list:

Think of a variable as a label for a value, not a box in which you put the value

(image taken from Fluent Python by Luciano Ramalho)

This DOES NOT create a new copy of a. It merely puts a new label on the memory at a, as can be seen by the following code:

Multiple items on one line in the interface are returned as a tuple, an immutable sequence of Python objects.

We can obtain the type of a variable, and use boolean comparisons to test these types.

For reference, below are common arithmetic and comparison operations.

EXERCISE: Create a tuple called tup with the following seven objects:

• The first element is an integer of your choice
• The second element is a float of your choice
• The third element is the sum of the first two elements
• The fourth element is the difference of the first two elements
• The fifth element is first element divided by the second element

Display the output of tup. What is the type of the variable tup? What happens if you try and chage an item in the tuple?

Part 2: Lists¶

Much of Python is based on the notion of a list. In Python, a list is a sequence of items separated by commas, all within square brackets. The items can be integers, floating points, or another type. Unlike in C arrays, items in a Python list can be different types, so Python lists are more versatile than traditional arrays in C or in other languages.

Let's start out by creating a few lists.

Lists in Python are zero-indexed, as in C. The first entry of the list has index 0, the second has index 1, and so on.

What happens if we try to use an index that doesn't exist for that list? Python will complain!

A list has a length at any given point in the execution of the code, which we can find using the len function.

Indexing on lists¶

And since Python is zero-indexed, the last element of float_list is

It is more idiomatic in python to use -1 for the last element, -2 for the second last, and so on

We can use the : operator to access a subset of the list. This is called slicing.

Below is a summary of list slicing operations:

You can slice "backwards" as well:

You can also slice with a stride:

We can iterate through a list using a loop. Here's a for loop.

Or, if we like, we can iterate through a list using the indices using a for loop with in range. This is not idiomatic and is not recommended, but accomplishes the same thing as above.

What if you wanted the index as well?

Python has other useful functions such as enumerate, which can be used to create a list of tuples with each tuple of the form (index, value).

This is an example of an iterator, something that can be used to set up an iteration. When you call enumerate, a list if tuples is not created. Rather an object is created, which when iterated over (or when the list function is called using it as an argument), acts like you are in a loop, outputting one tuple at a time.

Appending and deleting¶

We can also append items to the end of the list using the + operator or with append.

Go and run the cell with float_list.append a second time. Then run the next line. What happens?

To remove an item from the list, use del.

List Comprehensions¶

Lists can be constructed in a compact way using a list comprehension. Here's a simple example.

And here's a more complicated one, requiring a conditional.

This is entirely equivalent to creating comp_list1 using a loop with a conditional, as below:

The list comprehension syntax

[expression for item in list if conditional]

is equivalent to the syntax

for item in list:
    if conditional:
        expression

EXERCISE: Build a list that contains every prime number between 1 and 100, in two different ways:

1. Using for loops and conditional if statements.
2. (Stretch Goal) Using a list comprehension. You should be able to do this in one line of code, and it may be helpful to look up the function all in the documentation.

Part 3: Strings and listiness¶

A list is a container that holds a bunch of objects. We're particularly interested in Python lists because many other containers in Python, like strings, dictionaries, numpy arrays, pandas series and dataframes, and iterators like enumerate, have list-like properties. This is known as duck typing, a term coined by Alex Martelli, which refers to the notion that if it quacks like a duck, it is a duck. We'll soon see that these containers quack like lists, so for practical purposes we can think of these containers as lists! They are listy!

Containers that are listy have a set length, can be sliced, and can be iterated over with a loop. Let's look at some listy containers now.

Strings¶

We claim that strings are listy. Here's a string.

Like lists, this string has a set length, the number of characters in the string.

Like lists, we can slice the string.

And we can iterate through the string with a loop. Below is a while loop:

This is equivalent to the for loop:

So strings are listy.

How are strings different from lists? While lists are mutable, strings are immutable. Note that an error occurs when we try to change the second elemnt of string_list from 1 to b.

We can't use append but we can concatenate with +. Why is this?

What is happening here is that we are creating a new string in memory when we do astring + ', pavlos, ' + 'rahul, ' + 'margo'. Then we are relabelling this string with the old lavel astring. This means that the old memory that astring labelled is forgotten. What happens to it? We'll find out in lab 1.

Or we could use join. See below for a summary of common string operations.

To summarize this section, for practical purposes all containers that are listy have the following properties:

1. Have a set length, which you can find using len
2. Are iterable (via a loop)
3. Are sliceable via : operations

We will encounter other listy containers soon.

EXERCISE: Make three strings, called first, middle, and last, with your first, middle, and last names, respectively. If you don't have a middle name, make up a middle name!

Then create a string called full_name that joins your first, middle, and last name, with a space separating your first, middle, and last names.

Finally make a string called full_name_rev which takes full_name and reverses the letters. For example, if full_name is Jane Beth Doe, then full_name_rev is eoD hteB enaJ.

Part 4: Dictionaries¶

A dictionary is another storage container. Like a list, a dictionary is a sequence of items. Unlike a list, a dictionary is unordered and its items are accessed with keys and not integer positions.

Dictionaries are the closest container we have to a database.

Let's make a dictionary with a few Harvard courses and their corresponding enrollment numbers.

Simply iterating over a dictionary gives us the keys. This is useful when we want to do something with each item:

The above is an actual copy to another part of memory, unlike, second_dict = enroll2016_dict which would have made both variables label the same meory location.

In this example, the keys are strings corresponding to course names. Keys don't have to be strings though.

Like lists, you can construct dictionaries using a dictionary comprehension, which is similar to a list comprehension. Notice the brackets {} and the use of zip, which is another iterator that combines two lists together.

You can also create dictionaries nicely using the constructor function dict.

While dictionaries have some similarity to lists, they are not listy. They do have a set length, and the can be iterated through with a loop, but they cannot be sliced, since they have no sense of an order. In technical terms, they satisfy, along with lists and strings, Python's Sequence protocol, which is a higher abstraction than that of a list.

A cautionary word on iterators (read at home)¶

Iterators are a bit different from lists in the sense that they can be "exhausted". Perhaps its best to explain with an example

What happens, you get nothing when you run this again! This is because the iterator has been "exhausted", ie, all its items are used up. I have had answers go wrong for me because I wasnt careful about this. You must either track the state of the iterator or bypass this problem by not storing enumerate(BLA) in a variable, so that you dont inadvertantly "use that variable" twice.

Part 5: Functions¶

A function is a reusable block of code that does a specfic task. Functions are all over Python, either on their own or on objects.

We've seen built-in Python functions and methods. For example, len and print are built-in Python functions. And at the beginning of the lab, you called np.mean to calculate the mean of three numbers, where mean is a function in the numpy module and numpy was abbreviated as np. This syntax allow us to have multiple "mean" functions" in different modules; calling this one as np.mean guarantees that we will pick up numpy's mean function.

Methods¶

A function that belongs to an object is called a method. An example of this is append on an existing list. In other words, a method is a function on an instance of a type of object (also called class, here the list type).

User-defined functions¶

We'll now learn to write our own user-defined functions. Below is the syntax for defining a basic function with one input argument and one output. You can also define functions with no input or output arguments, or multiple input or output arguments.

def name_of_function(arg):
    ...
    return(output)

The simplest function has no arguments whatsoever.

We can write functions with one input and one output argument. Here are two such functions.

Lambda functions¶

Often we define a mathematical function with a quick one-line function called a lambda. No return statement is needed.

The big use of lambda functions in data science is for mathematical functions.

Refactoring using functions¶

EXERCISE: Write a function called isprime that takes in a positive integer $N$, and determines whether or not it is prime. Return the $N$ if it's prime and return nothing if it isn't. You may want to reuse part of your code from the exercise in Part 2.

Then, using a list comprehension and isprime, create a list myprimes that contains all the prime numbers less than 100.

Notice that what you just did is a refactoring of the algorithm you used earlier to find primes smaller than 100. This implementation reads much cleaner, and the function isprime which containes the "kernel" of the functionality of the algorithm can be re-used in other places. You should endeavor to write code like this.

Default Arguments¶

Functions may also have default argument values. Functions with default values are used extensively in many libraries.

We can have multiple default values.

Or change all the default arguments:

Or use the first default argument but change the second one.

Positional and keyword arguments¶

These allow for even more flexibility.

Positional arguments are used when you don't know how many input arguments your function be given. Notice the single asterisk before the second argument name.

In the function above, arguments after the first input will go into a list called siblings. We can then process that list to extract the names.

Keyword arguments mix the named argument and positional properties. Notice the double asterisks before the second argument name.

Putting things together¶

Finally, when putting all those things together one must follow a certain order: Below is a more general function definition. The ordering of the inputs is key: arguments, default, positional, keyword arguments.

def name_of_function(arg1, arg2, opt1=True, opt2="CS109", *args, **kwargs):
    ...
    return(output1, output2, ...)

Positional arguments are stored in a tuple, and keyword arguments in a dictionary.

Functions are first class¶

Python functions are first class, meaning that we can pass functions to other functions, built-in or user-defined.

Finally, it's important to note that any name defined in this notebook is done at the global scope. This means if you define your own len function, you will overshadow the system len.

EXERCISE: Create a dictionary, called ps_dict, that contains with the primes less than 100 and their corresponding squares.

Part 6. Numpy¶

Scientific Python code uses a fast array structure, called the numpy array. Those who have worked in Matlab will find this very natural. Let's make a numpy array.

Numpy arrays are listy. Below we compute length, slice, and iterate. But these are very bad ideas, for efficiency reasons we will see in lab 1.

In general you should manipulate numpy arrays by using numpy module functions (np.mean, for example). You can calculate the mean of the array elements either by calling the method .mean on a numpy array or by applying the function np.mean with the numpy array as an argument.

You can construct a normal distribution with mean0 and standard deviation 1 by doing:

Numpy supports a concept known as broadcasting, which dictates how arrays of different sizes are combined together. There are too many rules to list here, but importantly, multiplying an array by a number multiplies each element by the number. Adding a number adds the number to each element. This means that if you wanted the distribution $N(5, 7)$ you could do:

Part 7: Text Analysis of Hamlet (try it at home)¶

Below we'll walk you through an exercise analyzing the word distribution of the play Hamlet using what we learned in this lab.

First we need to read the file into Python. If you haven't done so already, please download hamlet.txt from the github repo and save it in the data subfolder. (You could just clone and download a zip of the entire repository as well).

EXERCISE: Read the documentation on reading and writing files. Open and read in and save hamlet.txt as hamlettext. Then close the file.

EXERCISE: Before moving on, let's make sure hamlettext actually contains the text of Hamlet. First, what is the type of hamlettext? What is its length? Print the first 500 items of hamlettext.

Each item in hamlettext is a text character. Let's see what we can learn about the distribution of words in the play. To do this, you'll need to split the string hamlettext into a list of strings, where each element of the list is a word in the play. Search the documentation, or refer back to the summary of string operations in the Strings section of this lab.

EXERCISE: Create a list called hamletwords where the items of the hamletwords are the words of the play. Confirm that hamletwords is a list, that each element of hamletwords is a string, and print the first 10 items in hamletwords. Print "There are $N$ total words in Hamlet," where $N$ is the total number of words in Hamlet.

The total number of words above considers "The" and "the" as two different words. So if we're going to find the distribution of the words in the play, we need to first make all the words lower-case. From the lower-case list of words, we then find the set of unique words.

EXERCISE: Using a list comprehension, create a hamletwords_lc which converts the items in hamletwords to lower-case. You will likely want to search the documentation for the string method lower. Then count the number of occurences of "thou", making use of the string method count.

We need to find the unique set of words in hamletwords_lc.

EXERCISE: Use set to determine the set of unique words in hamletwords_lc. Then print "There are $M$ unique words in Hamlet," where $M$ is the number of unique words. As a sanity check, verify that $M < N$.

Next we want a dictionary with the number of occurrences of each unique word, with the unique words as dictionary keys.

EXERCISE: Create this dictionary. To do it you will use either a loop or a dictionary comprehension and the count method.

Finally, we find the 100 most frequently occurring words using the built-in sorted function.

EXERCISE: (Stretch Goal) Search and read the documentation for the sorted function. The documentation has a link to some examples, "sorting HOW TO." Check out the example under the heading "Key Functions." Then create a list called top100words with the 100 most frequently occurring words in Hamlet.

Now we'll plot the top distribution of the top 10 words on this list. To do this you'll need to import a plotting library called matplotlib. The pyplot module in this library contains a set of functions that are especially useful for generating a wide range of plots. Go back to the first cell in this notebook and type

````
import matplotlib
import matplotlib.pyplot as plt
````

In this cell, also type

````
%matplotlib inline
````

Be sure to evaluate the cell before continuing. Below is the code to generate the bar chart.

This is a somewhat boring result because we forgot to remove common words like "the" from the corpus. That's a topic for a later date!

Part 8: References¶

Congratulations! You've completed lab 0. You'll likely very comfortable with some parts of the lab, and less comfortable with other parts. Below are a few tutorials and practice problems if you'd like to do more -- check them out.