CS109A

• Syllabus
• Schedule
• Materials
• FAQ
• Preparation

Key Word(s): pandas

Download Notebook

---

CS109A Introduction to Data Science

Lab 01: Introduction to Web Scraping

Harvard University
Fall 2021
Instructors: Pavlos Protopapas and Natesh Pillai
Lab Team: Marios Mattheakis, Hayden Joy, Chris Gumb, and Eleni Kaxiras
Authors: Varshini Reddy, Marios Mattheakis and Pavlos Protopapas

---

## RUN THIS CELL TO GET THE RIGHT FORMATTING 
import requests
from IPython.core.display import HTML
styles = requests.get("https://raw.githubusercontent.com/Harvard-IACS/2018-CS109A/master/content/styles/cs109.css").text
HTML(styles)

Lab Learning Objectives

When we're done today, you will approach messy real-world data with confidence that you can get it into a format that you can manipulate.

Specifically, our learning objectives are:

• Understand the tree-like structure of an HTML document and use that structure to extract desired information.

• Use Python data structures such as lists, dictionaries to store and manipulate information.

• Practice using Python packages such as BeautifulSoup, including how to navigate their documentation to find functionality.

• Identify other (semi-)structured formats commonly used for storing and transferring data, such as CSV.

Pre-Requisites

Before you start working on the lab, we expect you to be familiar with Python programming. Following is the list of topics you need to brush up on before attending the lab session. We have provided some quick start references as well.

• Python Data Structures
  • Lists
  • Dictionaries
• Functions in python
• Python classes
• Files and strings

# Importing necessary libraries
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from bs4 import BeautifulSoup
import requests
import json
from IPython.display import HTML
%matplotlib inline 

# Setting up 'requests' to make HTTPS requests properly takes some 
#       extra steps.

requests.packages.urllib3.disable_warnings()

import warnings
warnings.filterwarnings("ignore")

Lab Data Analysis Questions

Is science becoming more collaborative over time? How about literature? Are there a few "geniuses" or lots of hard workers? One way we might answer those questions is by looking at Nobel Prizes winners. We could ask questions like:

• 1) Has anyone won a prize more than once?
• 2) How has the total number of recipients changed over time?
• 3) How has the number of recipients per award changed over time?

To answer these questions, we will need data: who received what award and when.

When possible: find a structured dataset (.csv, .json, .xls)

After a google search we stumble upon this dataset on github. It is also in the lab folder named github-nobel-prize-winners.csv.

We use Pandas to read it. Pandas will be covered next week in more details.

df = pd.read_csv("data/github-nobel-prize-winners.csv")
df.head() 

	year	discipline	winner	desc
0	1901	chemistry	Jacobus H. van 't Hoff	in recognition of the extraordinary services h...
1	1901	literature	Sully Prudhomme	in special recognition of his poetic compositi...
2	1901	medicine	Emil von Behring	for his work on serum therapy, especially its ...
3	1901	peace	Henry Dunant	NaN
4	1901	peace	Frédéric Passy	NaN

Research Question 1: Did anyone recieve the Nobel Prize more than once?

How would you check if anyone recieved more than one nobel prize?

We will be using Python lists for this, which is a pre-requisite for this lab as mentioned earlier. If you have any questions with regards to lists or list comprehensions, refer to the slides from us here.

# Initialize the list storing all the names 
name_winners = []

for name in df.winner:
    
    # Check if we already encountered this name: 
    if name in name_winners:
        
        # (TODO) If so, print the name
        print(___)
    else:
        # (TODO) Otherwise append the name to the list
        name_winners.append(___)
        

We don't want to print "No Prize was Awarded" all the time.

# List storing all the names 
name_winners = []

for name in df.winner:
    
    # (TODO) Check if we already encountered this name and the name is not "No Prize was Awarded": 
    if name in name_winners and name != ___ : 
        # (TODO) If so, print the name
        print(___)
        
    else:
        # (TODO) Otherwise append the name to the list
        name_winners.append(___)
        

we can use .split() on a string to separate the words into individual strings and store them in a list.

Experiment with the .split() below before using it.

UN_string = "Office of the United Nations"
print(UN_string.split())
n_words = len(UN_string.split())
print("Number of words: " + str(n_words));

['Office', 'of', 'the', 'United', 'Nations']
Number of words: 5

Let us only print winners with only two words in their name:

name_winners = []

for name in df.winner:
    
    # (TODO) Check if we already encountered this name and the name consists of no more than 2 words: 
    if name in name_winners and len(___) <= 2: 
        # (TODO) If so, print the name
        print(___)
        
    else:
        # (TODO) Otherwise append the name to the list
        name_winners.append(___)
        

Marie Curie recieved the nobel prize in physics in 1903 and chemistry in 1911. She is one of only four people to recieve two Nobel Prizes.

All questions, such as "did anyone receive the Noble Price more than once?", are easy to answer when the data is present in such a clean tabular form. However, many times (if not most) we do not find the data we need in such a format.

In such cases, we need to perform web scraping and cleaning to get the data we desire. The end result of this lab is to create a pandas dataframe after web scraping and cleaning.

WEB SCRAPING

The first step in web scraping is to understand the HTML structure of the webpage.

But, what is HTML?

HTML stands for Hyper Text Markup Language. It is the standard markup language for documents designed to be displayed in a web browser. It can be assisted by technologies such as Cascading Style Sheets and scripting languages such as JavaScript.

Standard HTML documents

HTML documents generally have the following structure:

**\** **\** **\** **\** **\** **\** **\

Page Heading\

** **\

The first paragraph of page\

** **.** **.** **.** **.** **\** **\**

What does each of these tags indicate?

• The \<!DOCTYPE html> declaration defines that this document is an HTML5 document

• The \ element is the root element of an HTML page

• The \ element contains meta information about the HTML page

• The \