Syllabus

Course Objective

CS205 provides an introduction to parallel programming techniques and programming models for applications in science, engineering as well as data science. The class starts with an overview of parallel "machines" based on Flynn's taxonomy and then progresses closer to the computer hardware where various forms of parallelism are identified and discussed through different programming models and techniques. The class focuses on shared memory and distributed memory programming paradigms which are the most useful paradigms when developing software that is targeted for high performance computing (HPC) platforms. CS205 is further concerned about the scaling and performance analysis of parallel code. Both of these analyses are important when writing proposals for compute hours in HPC centers such as the Argonne or Oak Ridge National Labs. The class will discuss methods for performance analysis on the node level and methods for parallel scaling analysis on both node and cluster levels.

A short pdf version of this syllabus can be downloaded here.

Learning Objective

After completing this class, the student is capable of identifying various forms of parallelism in algorithms and exploit them with appropriate techniques. The student will know about methods to analyze the performance and scaling of parallel programs and knows how to further optimize software such that it can be deployed on large scale HPC architectures.

Prerequisites

The course assumes that the student is comfortable with reading and writing code in the C, C++ or Fortran programming languages. Homework, lab and examples in class will be presented using C++. Only simple C++ language features will be utilized for this class. If your proficiency is in C, you should be getting along without trouble. Lecture material for a C/C++ primer class can be found here. Familiarity with Linux command line tools, ssh, git and tools to edit source code is assumed. Prerequisite classes include CS50, CS61, CS107, CS161 or AC207.

This class is not suited for first-year students. CS205 will not teach basics of programming.

Textbooks

The class does not follow a specific textbook. The following textbooks are suitable for additional reference:

Course Format

The course contains six main components:

  1. Lectures: Deliver the main content of the class. Attendance is mandatory.
  2. Readings: Accommodate lecture material. The reading assignments are discussed in class. Questions to individual students may be asked.
  3. Quizzes: Graded in-class quizzes.
  4. Labs: Lab sessions offer practice on topics addressed in class and help support homework assignments. Attendance is mandatory.
  5. Homeworks: Homework assignments deepen the lecture material and include coding exercises (skeleton codes are provided in C++). Exercises may be of theoretical or practical nature, including applications such as simulations based on particle methods or stencil methods on structured grids.
  6. Projects: The class is accompanied by a project (teams of 3-4 students) to practice the methods learned in class on a real application. Topics are proposed by the teams and may involve research problems of individual team members.

Grading

The following weight table is used for individual components of the class. The class does not have standard midterm or final exams.

Total Weight
Homework (5 Homeworks) 35%
Project 30%
Quizzes (4 Quizzes) 20%
Labs (6 Labs for attendance and completion) 10%
Comunity Contributions 5%

Homework

There are 5 homeworks where each contributes equally to the final grade. The homework is focused on the topics discussed in class and involves programming and theoretical work. Programming tasks may be combined with applications such as particle methods or simple grid based finite difference solvers. Familiarity with such methods is not required and explanations provided in the homework statements should be sufficient to solve the tasks. Coding exercises may be implemented in C or C++. Note that some coding exercises will include skeleton codes which are written in C++. It is expected that answers for assignments involving written or theoretical work are typeset in a LaTeX\mathrm{\LaTeX} document. The teaching staff is determined to return solutions and graded assignments with feedback shortly after the due date. It is your responsibility to check the consistency between your graded work and the assignment solution. You have the option to address possible inconsistencies in office hours or request a regrading for the assignment (see the homework grading inconsistencies section below). Homework will be released on the CS205 class repository. Push notifications for that repository will be distributed through the class mailing list.

Some of the homework problems must be solved using a remote compute cluster. Please keep in mind that when the cluster is under high load with many queued jobs, your submitted job(s) will not execute right away. Depending on the cluster load, your job may execute only in a couple of hours (resources are limited). Make sure you plan ahead.

Homework Submission

Homework must be submitted on Gradescope before the due date. All submissions must be uploaded in a zip archive of the corresponding homework handout directory (completed with your solutions) which is released through the CS205 class Git repository. Submissions must only contain source code, possible text files and written reports in pdf format (you should give it an intuitive name, e.g. hw1_solution.pdf). All your results and visualizations based on computed data that the problem statement is asking for must be communicated in the written report by means of tables, plots or other forms of display (attached data files are not considered for grading). Object files and other binary data generated by compilers must be cleaned out before submission. Other files that were initially included in the handout directory can be left in place. Please do not rename any files in the handout. Example: The homework 1 handout can be found here. To upload your solution to Gradescope, create a zip archive of your hw1 directory with the following contents:

hw1/
├── code
│   └── p4
│       └── openmp_pi.cpp  <-- skeleton code with your solution
├── hw1.pdf
├── hw1_solution.pdf       <-- your solution write-up (new file)
└── README.md

Please do not include any LaTeX\mathrm{\LaTeX} source code in your submission and remove any unrelated meta data such as .DS_Store files or __MACOSX__ directories. Your submission is your signature.

The homework due date is indicated on the problem sheet and displayed in the schedule. Homework submissions will be graded on:

  1. Correctness: your code must compile, run and should produce the correct result. We are not debugging issues when grading submissions.
  2. Efficiency: programs, especially parallel programs, can be implemented in various ways. Homework will be graded based on efficient parallel code. Consider two parallel implementations, both generate the correct result but the second implementation performs unnecessary work or redundant communication. The second implementation will not get full marks.
  3. Presentation: how you present and interpret your results either in a written report or in your code. Presentation means structure and readability. If you present a plot you should have the axes labeled correctly (including correct units). Difficult sections in code should be commented appropriately such that the teaching staff can understand your thought process. Presentation of results also means that unnecessary or superfluous files like editor backup files or object files left over from previous compilation should not be included in the submission. Results and data must be presented in the written report, the teaching staff will not search for this data in other data files submitted with the homework. It is expected that assignment solutions are typeset in a LaTeX\mathrm{\LaTeX} document and submitted as a PDF file.

Homework Late Days

You have three late days at your disposal that you are free to use for late submissions. You can use at most two consecutive late days for any homework assignment. Deadlines on Gradescope are enforced and late submissions (after consumption of late days) will not be graded. The Gradescope submission window is open 2 days past deadline. It is your responsibility not to overdraft your late day budget. Late submissions are counted by the minute. If you have consumed all of your late budget: any submission after the assignment deadline will be considered late and charged with a 20 point late penalty.

Most assignments in this class are due within 3 weeks. It is your responsibility to plan your work ahead and submit on time.

Please note: the job scheduler on the compute cluster can become busy if there is a large volume of job submissions (i.e. all students try to submit their jobs 3 hours before deadline). In such situations, a submitted job may not immediately execute. This is normal on shared compute resources, do not submit your work in the last minute!

Homework Grading Errors

If you believe there is an error in your assignment grading, you can submit a regrade request through Gradescope.

Note:

  1. The entire assignment will be regraded. This may cause your total grade go up or down.
  2. An assignment can only be regraded once.
  3. Regrade requests are due within 2 days after the release of the grades.

Project

Please see the project section for more details.

Quizzes

There are 4 in-class quizzes which are graded. Each quiz addresses topics from the lecture material. Quizzes are open book/www and include multiple choice questions with at most back of the envelope calculations. Quizzes take 15 minutes (10-12 questions) and are hosted on Canvas. Please bring your laptops.

Please see the class schedule as well.

Labs

The purpose of the CS205 labs is to provide extended practice on important topics discussed in the lecture which also support homework assignments. Practice is the key to learn and deepen these topics. The labs exists because time constraints do not allow for this extended practice during the lecture. Labs will be offered on different weekdays and lead by the teaching fellows of CS205. Lab sections will be offered through my.harvard where you should choose your preferred section(s) such that we can perform an optimized sectioning given your input. Your assigned lab day will then be the same throughout the term. There will be 6 labs in total. See the schedule page for more information on lab content and the attendance policy section below.

Lab Submission

Lab exercises must be submitted on Gradescope before the due date. Submissions must be uploaded in a zip archive of the corresponding lab directory (completed with your solutions) which is released through the CS205 class repository. Submissions must only contain source code and text files. Object files and other binary data generated by compilers must be cleaned out before submission. Other binary files that were initially included in the directory can be left in place. You are not required to write a LaTeX\mathrm{\LaTeX} report for labs. Answers can be provided in markdown files (see problem statement). Submissions are similar to homework, see the homework submission section for an example.

The lab due date is indicated on the problem sheet and displayed in the schedule. Lab submissions will be graded on:

  1. Attendance: your attendance will be recorded by the TF who leads the lab. Joining the lab session at the beginning and then leaving 10-15 minutes later will not reward attendance credit. If you need to leave because of another appointment then it is expected that you communicate this before hand and coordinate with your TF. Please see the attendance policy section below as well.
  2. Completion: lab submissions should show effort that the student attempted to solve the tasks. If you experience difficulties in a particular problem and you are not able to complete the task, please indicate the issues you had in your code using comments and we will take that reasoning into account. Just handing in an empty lab skeleton (same as hand-out) does not meet the expected standard and will not award credit for submission.

Community Contributions

The frequency with which you visit and/or post to the class forum can contribute to your final grade. These contributions must have intellectual value and can be in the form of questions and/or answers related to class material. Be mindful about sharing sensitive material such as solution code for homeworks. If in doubt, try to rephrase your question/answer or ask the teaching staff via cs205-staff@g.harvard.edu whether the content is permissible to share. Factors that contribute are frequency of visits (passive), posing questions (active), giving answers (active), commenting (active) and liking posts (passive) as well as creating posts (active). Anonymous post do count to these factors as well.

Each individual is different, you can contribute to the community passively (typically more taking) and actively (typically more giving). Either way is fine but some active contributions are required to keep the community alive. We encourage you to be active.

Reading Assignments

Some lectures have supportive readings assigned (see the schedule). Reading these assignments will help you gain a deeper understanding of the material discussed in class, it will introduce you to new terminology and definitions and it will give you some thoughts and ideas to participate in the class discussion. You are expected to read them but it is not necessary that you thoroughly understand everything. You should typically set aside 1-2 hours for a reading assignment.

Office Hours

The teaching staff holds weekly office hours. Office hour times and locations are listed on the class main page. Office hours provide you with an opportunity to review and discuss course materials as well as provide you with further guidance for your homework.

Attendance Policy

It is expected that when you decide to take CS205 that you also attend the lectures as well as the labs. These are core parts of the class and therefore mandatory to attend.

Labs will be held on weekdays that we determine at the beginning of the class according to a best fit of the students' individual schedules for the term. You are required to attend the labs at the assigned lab day. Rescheduling of a lab to a different day due to an unforeseeable event must be coordinated with the teaching staff by sending an email to cs205-staff@g.harvard.edu.

To be excused from a lecture or a lab, we ask you to follow the Harvard Honor Code and send an email to cs205-staff@g.harvard.edu at least one day before the lecture or lab. Lecture recordings are available only when students are excused for a lecture.

Collaboration Policy

You are welcome to discuss the course material and homework with others in order to better understand it, but the work you turn in must be your own (with exception of the project where collaborative work is permitted). Any work that is not your own, without properly citing the original author(s), is considered plagiarism. Failure to follow the academic integrity and dishonesty guidelines outlined in the Harvard Student Handbook will have an adverse effect on your final grade. This includes the removal of copyright notices in code. You may not submit the same or similar work to this course that you have submitted or will submit to another without permission. The teaching staff may use tools to compute correlations between submitted work.

Accessibility

If you have a documented disability (physical or cognitive) that may impair your ability to complete assignments or otherwise participate in the course and satisfy course criteria, please contact the teaching staff or directly the Accessible Education Office to receive an AEO letter that will authorize us to help you with corresponding accommodations.

Diversity Statement

All participants in this class are expected to foster empathy and respect towards each other. This includes instructors, teaching staff or students. The motivation to take this course shall be to experience the joy of learning in an environment that allows for a diversity of thoughts, perspectives and experiences and honors your identity including race, gender, class, sexuality, religion, ability, etc. Any constructive feedback for improving the class environment is welcome and I encourage you to reach out to the instructor or teaching staff with any concerns you may have. If you prefer to speak with someone outside of the course, you may find helpful resources at the Harvard Office of Diversity and Inclusion.