CS-109A Introduction to Data Science

Lab 1: Introduction to Python and its Numerical Stack¶

Harvard University
Fall 2019
Instructors: Pavlos Protopapas, Kevin Rader, and Chris Tanner
Lab Instructor: Eleni Kaxiras
Authors: Rahul Dave, David Sondak, Will Claybaugh, Pavlos Protopapas, Chris Tanner, and Eleni Kaxiras

---

Programming Expectations¶

All assignments for this class will use Python and the browser-based iPython notebook format you are currently viewing. Programming at the level of CS 50 is a prerequisite for this course. If you have concerns about this, come speak with any of the instructors.

We will refer to the Python 3 documentation in this lab and throughout the course.

Learning Goals¶

This introductory lab is a condensed introduction to Python numerical programming. By the end of this lab, you will feel more comfortable:

• Learn about anconda environments and setup your own with the necessary dependencies

• Writing short Python code using functions, loops, lists, numpy arrays, and dictionaries.

• Manipulating Python lists and numpy arrays and understanding the difference between them.

• Introducing the stats libraries scipy.stats and statsmodels

Part 1: Set up a Conda Python Environment and Clone the Class Repository¶

On Python installation packages¶

There are two main installing packages for Python, conda and pip. Pip is the Python Packaging Authority’s recommended tool for installing packages from the Python Package Index (PyPI). Conda is a cross platform package and environment manager that installs and manages conda packages from the Anaconda repository and Anaconda Cloud. Conda does not assume any specific configuration in your computer and will install the Python interpreter along with the other Python packages, whereas pip assumes that you have installed the Python interpreter in your computer. Given the fact that most operating systems do include Python this is not a problem.

If I could summarize their differences into a sentence it would be that conda has the ability to create isolated environments that can contain different versions of Python and/or the packages installed in them. This can be extremely useful when working with data science tools as different tools may contain conflicting requirements which could prevent them all being installed into a single environment. You can have environments with pip but would have to install a tool such as virtualenv or venv. You may use either, we recommend conda because in our experience it leads to fewer incompatibilities between packages and thus fewer broken environments.

Conclusion: Use Both. Most often in our data science environments we want to combining pip with conda when one or more packages are only available to install via pip. Although thousands of packages are available in the Anaconda repository, including the most popular data science, machine learning, and AI frameworks but a lot more are available on PyPI. Even if you have your environment installed via conda you can use pip to install individual packages

(source: anaconda site)

Installing Conda¶

- First check if you have conda¶

In MacOS or Linux open a Terminal window and at the prompt type

conda –V

If you get the version number (e.g. conda 4.6.14) you are all set! If you get an error, that means you do not have Anaconda and would be a good idea to install it.

- If you do not have it, you can install it by following the instructions:¶

Mac : https://docs.anaconda.com/anaconda/install/mac-os/

Windows : https://docs.anaconda.com/anaconda/install/windows (Note: #8 is important: DO NOT add to your path. The reason is that Windows contains paths that may include spaces and that clashes with the way conda understands paths.)

- If you do have anaconda consider upgrading it so you get the latest version of the packages:¶

conda update conda

Conda allows you to work in 'computing sandboxes' called environments. You may have environments installed on your computer to access different versions of Python and different libraries to avoid conflict between libraries which can cause errors.

---

NOTE (Sept.6, 2019):¶

If you are still having issues please check the Announcements and the Discussion Forum (Ed) via the 2019-CS109a Canvas site

Also please check the latest version of the cs109a.yml file. We have edited it as of today.

---

What are environments and do I need them?¶

Environments in Python are like sandboxes that have different versions of Python and/or packages installed in them. You can create, export, list, remove, and update environments. Switching or moving between environments is called activating the environment. When you are done with an environments you may deactivate it.

For this class we want to have a bit more control on the packages that will be installed with the enviromnent so we will create an environment with a so called YAML file called cs109a.yml. Originally YAML was said to mean Yet Another Markup Language referencing its purpose as a markup language with the yet another construct, but it was then repurposed as YAML Ain't Markup Language [source:wikipedia]. This is included in the Lab directory in the class git repository.

Creating an environment from an environment.yml file¶

Using your browser, visit the class git repository https://github.com/Harvard-IACS/2019-CS109A

Go to content --> labs/ --> lab1 and look for the cs109a.yml file. Download it to a local directory in your computer.

Then in the Terminal again type

conda env create -f {PATH-TO-FILE}/cs109a.yml

Activate the new environment:¶

source activate cs109a

You should see the name of the environment at the start of your command prompth in parenthesis.

Verify that the new environment was installed correctly:¶

conda list

This will give you a list of the packages installed in this environment.

References¶

Manage conda environments

Clone the class repository¶

In the Terminal type:

git clone https://github.com/Harvard-IACS/2019-CS109A.git

Starting the Jupyter Notebook¶

Once all is installed go in the Terminal and type

jupyter notebook

to start the jupyter notebook server. This will spawn a process that will be running in the Terminal window until you are done working with the notebook. In that case press control-C to stop it.

Starting the notebook will bring up a browser window with your file structure. Look for the 2019-CS109A folder. It should be where you cloned it previously. When you visit this folder in the future, and while in the top folder of it, type

git pull

This will update the contents of the folder with whatever is new. Make sure you are at the top part of the folder by typing

pwd

which should give you /2019-CS109A/

For more on using the Notebook see: https://jupyter-notebook.readthedocs.io/en/latest/

Part 2: Getting Started with Python¶

Importing modules¶

All notebooks should begin with code that imports modules, collections of built-in, commonly-used Python functions. Below we import the Numpy module, a fast numerical programming library for scientific computing. Future labs will require additional modules, which we'll import with the same syntax.

import MODULE_NAME as MODULE_NICKNAME

Now that Numpy has been imported, we can access some useful functions. For example, we can use mean to calculate the mean of a set of numbers.

Calculations and variables¶

The last line in a cell is returned as the output value, as above. For cells with multiple lines of results, we can display results using print, as can be seen below.

We can store integer or floating point values as variables. The other basic Python data types -- booleans, strings, lists -- can also be stored as variables.

Here is the storing of a list

Think of a variable as a label for a value, not a box in which you put the value

(image: Fluent Python by Luciano Ramalho)

This DOES NOT create a new copy of a. It merely puts a new label on the memory at a, as can be seen by the following code:

Tuples

Multiple items on one line in the interface are returned as a tuple, an immutable sequence of Python objects. See the end of this notebook for an interesting use of tuples.

type()¶

We can obtain the type of a variable, and use boolean comparisons to test these types. VERY USEFUL when things go wrong and you cannot understand why this method does not work on a specific variable!

For reference, below are common arithmetic and comparison operations.

• The first element is an integer of your choice
• The second element is a float of your choice
• The third element is the sum of the first two elements
• The fourth element is the difference of the first two elements

• The fifth element is the first element divided by the second element

• Display the output of tup. What is the type of the variable tup? What happens if you try and chage an item in the tuple?

Lists¶

Much of Python is based on the notion of a list. In Python, a list is a sequence of items separated by commas, all within square brackets. The items can be integers, floating points, or another type. Unlike in C arrays, items in a Python list can be different types, so Python lists are more versatile than traditional arrays in C or other languages.

Let's start out by creating a few lists.

Lists in Python are zero-indexed, as in C. The first entry of the list has index 0, the second has index 1, and so on.

What happens if we try to use an index that doesn't exist for that list? Python will complain!

You can find the length of a list using the built-in function len:

Indexing on lists plus Slicing¶

And since Python is zero-indexed, the last element of float_list is

It is more idiomatic in Python to use -1 for the last element, -2 for the second last, and so on

We can use the : operator to access a subset of the list. This is called slicing.

Below is a summary of list slicing operations:

You can slice "backwards" as well:

You can also slice with a stride:

We can iterate through a list using a loop. Here's a for loop.

What if you wanted the index as well?

Use the built-in python method enumerate, which can be used to create a list of tuples with each tuple of the form (index, value).

Appending and deleting¶

We can also append items to the end of the list using the + operator or with append.

Now, run the cell with float_list.append() a second time. Then run the subsequent cell. What happens?

To remove an item from the list, use del.

You may also add an element (elem) in a specific position (index) in the list

List Comprehensions¶

Lists can be constructed in a compact way using a list comprehension. Here's a simple example.

And here's a more complicated one, requiring a conditional.

This is entirely equivalent to creating comp_list1 using a loop with a conditional, as below:

The list comprehension syntax

[expression for item in list if conditional]

is equivalent to the syntax

for item in list:
    if conditional:
        expression

• 2.1 Using for loops and conditional if statements.
• 2.2 (Stretch Goal) Using a list comprehension. You should be able to do this in one line of code. Hint: it might help to look up the function all() in the documentation.

Simple Functions¶

A function object is a reusable block of code that does a specific task. Functions are commonplace in Python, either on their own or as they belong to other objects. To invoke a function func, you call it as func(arguments).

We've seen built-in Python functions and methods (details below). For example, len() and print() are built-in Python functions. And at the beginning, you called np.mean() to calculate the mean of three numbers, where mean() is a function in the numpy module and numpy was abbreviated as np. This syntax allows us to have multiple "mean" functions in different modules; calling this one as np.mean() guarantees that we will execute numpy's mean function, as opposed to a mean function from a different module.

User-defined functions¶

We'll now learn to write our own user-defined functions. Below is the syntax for defining a basic function with one input argument and one output. You can also define functions with no input or output arguments, or multiple input or output arguments.

def name_of_function(arg):
    ...
    return(output)

We can write functions with one input and one output argument. Here are two such functions.

What if you want to return two variables at a time? The usual way is to return a tuple:

Lambda functions¶

Often we quickly define mathematical functions with a one-line function called a lambda function. Lambda functions are great because they enable us to write functions without having to name them, ie, they're anonymous.
No return statement is needed.

Methods¶

A function that belongs to an object is called a method. By "object," we mean an "instance" of a class (e.g., list, integer, or floating point variable).

For example, when we invoke append() on an existing list, append() is a method.

In other words, a method is a function on a specific instance of a class (i.e., object). In this example, our class is a list. float_list is an instance of a list (thus, an object), and the append() function is technically a method since it pertains to the specific instance float_list.

In Exercise 2, above, you wrote code that generated a list of the prime numbers between 1 and 100. Now, write a function called isprime() that takes in a positive integer $N$, and determines whether or not it is prime. Return True if it's prime and return False if it isn't. Then, using a list comprehension and isprime(), create a list myprimes that contains all the prime numbers less than 100.

Introduction to Numpy¶

Scientific Python code uses a fast array structure, called the numpy array. Those who have programmed in Matlab will find this very natural. For reference, the numpy documention can be found here.

Let's make a numpy array:

The shape array of an array is very useful (we'll see more of it later when we talk about 2D arrays -- matrices -- and higher-dimensional arrays).

Numpy arrays are typed. This means that by default, all the elements will be assumed to be of the same type (e.g., integer, float, String).

Numpy arrays have similar functionality as lists! Below, we compute the length, slice the array, and iterate through it (one could identically perform the same with a list).

There are two ways to manipulate numpy arrays a) by using the numpy module's methods (e.g., np.mean()) or b) by applying the function np.mean() with the numpy array as an argument.

A constructor is a general programming term that refers to the mechanism for creating a new object (e.g., list, array, String).

There are many other efficient ways to construct numpy arrays. Here are some commonly used numpy array constructors. Read more details in the numpy documentation.

Numpy gains a lot of its efficiency from being typed. That is, all elements in the array have the same type, such as integer or floating point. The default type, as can be seen above, is a float. (Each float uses either 32 or 64 bits of memory, depending on if the code is running a 32-bit or 64-bit machine, respectively).

Often, you will want random numbers. Use the random constructor!

You can generate random numbers from a normal distribution with mean 0 and variance 1:

You can sample with and without replacement from an array. Let's first construct a list with evenly-spaced values:

Without replacement

With replacement:

Tensors¶

We can think of tensors as a name to include multidimensional arrays of numerical values. While tensors first emerged in the 20th century, they have since been applied to numerous other disciplines, including machine learning. In this class you will only be using scalars, vectors, and 2D arrays, so you do not need to worry about the name 'tensor'.

We will use the following naming conventions:

• scalar = just a number = rank 0 tensor ($a$ ∈ $F$,)
  
  
• vector = 1D array = rank 1 tensor ( $x = (\;x_1,...,x_i\;)⊤$ ∈ $F^n$ )
  
  
• matrix = 2D array = rank 2 tensor ( $\textbf{X} = [a_{ij}] ∈ F^{m×n}$ )
  
  
• 3D array = rank 3 tensor ( $\mathscr{X} =[t_{i,j,k}]∈F^{m×n×l}$ )
  
  
• $\mathscr{N}$D array = rank $\mathscr{N}$ tensor ( $\mathscr{T} =[t_{i1},...,t_{i\mathscr{N}}]∈F^{n_1×...×n_\mathscr{N}}$ )

Slicing a 2D array¶

source:oreilly

Numpy supports vector operations¶

What does this mean? It means that instead of adding two arrays, element by element, you can just say: add the two arrays.

Note that this behavior is very different from python lists where concatenation happens.

On some computer chips, this numpy addition actually happens in parallel and can yield significant increases in speed. But even on regular chips, the advantage of greater readability is important.

Broadcasting¶

Numpy supports a concept known as broadcasting, which dictates how arrays of different sizes are combined together. There are too many rules to list here, but importantly, multiplying an array by a number multiplies each element by the number. Adding a number adds the number to each element.

This means that if you wanted the distribution $N(5, 7)$ you could do:

Multiplying two arrays multiplies them element-by-element

You might have wanted to compute the dot product instead:

Probabilitiy Distributions from scipy.stats and statsmodels¶

Two useful statistics libraries in python are scipy and statsmodels.

For example to load the z_test:

Let's get the normal distribution namespace from scipy.stats. See here for Documentation.

Let's create 1,000 points between -10 and 10

Let's get the pdf of a normal distribution with a mean of 1 and standard deviation 3, and plot it using the grid points computed before:

And you can get random variables using the rvs function.

Referencies¶

A useful book by Jake Vanderplas: PythonDataScienceHandbook.

You may also benefit from using Chris Albon's web site as a reference. It contains lots of useful information.

Dictionaries¶

A dictionary is another data structure (aka storage container) -- arguably the most powerful. Like a list, a dictionary is a sequence of items. Unlike a list, a dictionary is unordered and its items are accessed with keys and not integer positions.

Dictionaries are the closest data structure we have to a database.

Let's make a dictionary with a few Harvard courses and their corresponding enrollment numbers.

One can obtain the value corresponding to a key via:

If you try to access a key that isn't present, your code will yield an error:

Alternatively, the .get() function allows one to gracefully handle these situations by providing a default value if the key isn't found:

Note, this does not store a new value for the key; it only provides a value to return if the key isn't found.

All sorts of iterations are supported:

We can iterate over the tuples obtained above:

Simply iterating over a dictionary gives us the keys. This is useful when we want to do something with each item:

The above is an actual copy of _enroll2017dict's allocated memory, unlike, second_dict = enroll2017_dict which would have made both variables label the same memory location.

In the previous dictionary example, the keys were strings corresponding to course names. Keys don't have to be strings, though; they can be other immutable data type such as numbers or tuples (not lists, as lists are mutable).

Dictionary comprehension: "Do not try this at home"¶

You can construct dictionaries using a dictionary comprehension, which is similar to a list comprehension. Notice the brackets {} and the use of zip (see next cell for more on zip)

Creating tuples with zip¶

zip is a Python built-in function that returns an iterator that aggregates elements from each of the iterables. This is an iterator of tuples, where the i-th tuple contains the i-th element from each of the argument sequences or iterables. The iterator stops when the shortest input iterable is exhausted. The set() built-in function returns a set object, optionally with elements taken from another iterable. By using set() you can make zip printable. In the example below, the iterables are the two lists, float_list and int_list. We can have more than two iterables.