Schedule

• Class introduction/organization
• Moore's Law
• Transistor density and power limit
• Parallel computing
• Flynn's taxonomy
• Overview of parallelism treated in class: DLP, ILP, TLP, shared memory and distributed memory

• Computer architecture
• von Neumann architecture
• Memory pyramid
• Linux process anatomy
• Introduction to compute cluster: access, job submission
• Reading: Leiserson paper

Select one of the offered lab session days according to your schedule

The "Reading" assignments are relevant for the lecture and due on the day of the lecture! Questions may be asked to individual students.

1. Lab section preferences submitted on my.harvard
   (2023-01-27)

• Cache memories: why are they there, how they work
• Cache lines and the 3 C's
• What is temporal and spatial locality
• Cache associativity: fully, $n$-way, direct mapped
• Memory access patterns (differences row-major / column-major)

• Shared memory introduction
• Examples of concurrency and concurrent memory access
• Why is shared memory programming hard: what is a race condition and why/how does it happen
• Quiz 1

Accessing cluster, SLURM, Linux, compiler and C++ tutorials.

1. HW1 release
   (2023-01-31)

• Memory model for shared memory programming and its implications on compilers
• Sequential consistency
• Mutual exclusion / critical sections / locks
• Overview of thread libraries

• Introduction to OpenMP: why OpenMP and how to use it in new or existing codes
• OpenMP: fork/join parallel regions
• OpenMP: work sharing constructs
• Reading: OpenMP specification 5.1 Chap. 1 (until 1.4 inclusive)

1. Lab 1 due
   (2023-02-10)
2. Project team formation due
   (2023-02-07)

• OpenMP: data environment
• OpenMP: synchronization constructs
• OpenMP: library routines
• OpenMP: environment variables

• OpenMP: data environment
• OpenMP: synchronization constructs
• OpenMP: library routines
• OpenMP: environment variables
• Quiz 2

OpenMP locks, critical sections and atomic clauses.

1. HW1 due
   (2023-02-14)
2. HW2 release
   (2023-02-14)

• UMA/NUMA memory architectures and processor affinity
• What is cache coherency and why is it required in shared memory programming
• Cache coherency protocols (focus on MESI)
• False sharing

• Performance analysis (single node)
• Relationship of compute performance (flop) to memory bandwidth
• Roofline model
• Reading: Williams paper

False sharing and cache thrashing.

1. Lab 2 due
   (2023-02-24)
2. Project high-level description due
   (2023-02-21)

• Introduction to distributed programming (recap Flynn's taxonomy)
• What is the Message Passing Interface (MPI)
• Simple parallel MPI program example

• MPI: blocking point-to-point
• MPI: blocking collective
• Reading: MPI 4.0 Standard 3.1, 3.2, 3.4, 3.5

• MPI: non-blocking point-to-point
• MPI: non-blocking collective
• Reading: MPI 4.0 Standard 3.7, 6.1

• MPI: I/O file management
• MPI: I/O read and write routines
• Parallel I/O for data compression example
• Quiz 3

MPI reductions and scans.

1. HW2 due
   (2023-03-07)
2. HW3 release
   (2023-03-07)
3. Lab 3 due
   (2023-03-10)

1. Lab 4 due
   (2023-03-24)
2. Project proposals due

• Parallel scaling analysis
• Strong scaling / Amdahl's law
• Weak scaling

• Instruction set architecture (ISA) / RISC / CISC
• Processor pipelining (ILP)
• Reading: Hennessy and Patterson Turing lecture

Linking your code with third party libraries. Examples for BLAS and LAPACK.

1. HW3 due
   (2023-03-28)
2. HW4 release
   (2023-03-28)

• Assembly language (x86-64)
• Recap Flynn's taxonomy: SIMD
• Instruction set architecture extensions
• What is vectorization and why is it important
• Floating-point operations in x86-64

• Memory alignment and relation to cache lines
• Manual vectorization
• Intel intrinsics

1. Lab 5 due
   (2023-04-07)

1. Project designs due

• Intel intrinsics
• Compiler auto vectorization
• Examples for vectorization and performance impact (DLP in roofline)

• SPMD programming model
• Intel ISPC compiler
• Reading: Pharr paper
• Quiz 4

Understanding machine instructions by learning how to debug code.

1. HW4 due
   (2023-04-18)
2. HW5 release
   (2023-04-18)

• GPU computing (outlook):
  • Streaming processors
  • Main difference between CPU and GPU architectures
  • SIMD and SIMT
  • Streaming multiprocessor and Little's Law
  • Introduction to CUDA
  • CUDA warps and threads
• Class summary

1. HW5 due
   (2023-04-30)
2. Lab 6 due
   (2023-04-28)

• Project final presentations:
  • 2023-05-04
  • 2023-05-05

1. Project deliverables due
   (2023-05-04 8:00AM)
2. Project final presentations due
   (2023-05-04/05)