Schedule

• Class introduction/organization
• Moore's Law
• Transistor density and power limit
• Parallel computing
• Flynn's taxonomy
• Overview of parallelism treated in class: DLP, ILP, TLP, shared memory and distributed memory

• Computer architecture
• von Neumann architecture
• Memory pyramid
• Linux process anatomy
• Introduction to compute cluster: access, job submissions
• Reading: Leiserson paper

Select one of the offered lab session days according to your schedule

The "Reading" assignments are relevant for the lecture and due on the day of the lecture!

1. Complete the CS 205: C++ Survey
   (2024-01-23)
2. Lab section preferences submitted on this spreadsheet
   (2024-01-26)

• Cache memories: why are they there, how they work
• Cache lines and the 3 C's
• What is temporal and spatial locality
• Cache associativity: fully, $n$-way, direct mapped
• Memory access patterns (differences row-major / column-major)

• Shared memory introduction
• Examples of concurrency and concurrent memory access
• Why is shared memory programming hard: what is a race condition and why/how does it happen

Accessing cluster, SLURM, Linux, compiler and C++ tutorials.

1. Quiz 1
   (2024-02-02)
2. HW1 release
   (2024-01-30)

• Memory model for shared memory programming and its implications on compilers
• Sequential consistency
• Mutual exclusion / critical sections / locks
• Overview of thread libraries

• Introduction to OpenMP: why OpenMP and how to use it in new or existing codes
• OpenMP: fork/join parallel regions
• OpenMP: work sharing constructs
• Reading: OpenMP specification 5.1 Chap. 1 (until 1.4 inclusive)

1. Lab 1 due
   (2024-02-09)
2. Project team formation due
   (2024-02-06)

• OpenMP: data environment
• OpenMP: synchronization constructs
• OpenMP: library routines
• OpenMP: environment variables

• OpenMP: data environment
• OpenMP: synchronization constructs
• OpenMP: library routines
• OpenMP: environment variables

OpenMP locks, critical sections and atomic clauses.

1. HW1 due
   (2024-02-13)
2. HW2 release
   (2024-02-13)
3. Quiz 2
   (2024-02-16)

• UMA/NUMA memory architectures and processor affinity
• What is cache coherency and why is it required in shared memory programming
• Cache coherency protocols (focus on MESI)
• False sharing

• Performance analysis (single node)
• Relationship of compute performance (flop) to memory bandwidth
• Roofline model
Reading: Williams paper

False sharing and cache thrashing.

1. Lab 2 due
   (2024-02-23)
2. Project high-level description due
   (2024-02-20)

• Introduction to distributed programming (recap Flynn's taxonomy)
• What is the Message Passing Interface (MPI)
• Simple parallel MPI program example

• MPI: blocking point-to-point
• MPI: blocking collective
• Reading: MPI 4.0 Standard 3.1, 3.2, 3.4, 3.5

1. Lab 3 due
   (2024-03-01)

• MPI: non-blocking point-to-point
• MPI: non-blocking collective
• Reading: MPI 4.0 Standard 3.7, 6.1

• MPI: I/O file management
• MPI: I/O read and write routines
• Parallel I/O for data compression example

MPI reductions and scans.

1. HW2 due
   (2024-03-05)
2. HW3 release
   (2024-03-05)
3. Quiz 3
   (2024-03-08)

1. Lab 4 due
   (2024-03-15)
2. Project proposals due

• Parallel scaling analysis
• Strong scaling / Amdahl's law
• Weak scaling

• Instruction set architecture (ISA) / RISC / CISC
• Processor pipelining (ILP)
• Reading: Hennessy and Patterson Turing lecture

Linking your code with third party libraries. Examples for BLAS and LAPACK.

1. HW3 due
   (2024-03-26)
2. HW4 release
   (2024-03-26)

• Assembly language (x86-64)
• Recap Flynn's taxonomy: SIMD
• Instruction set architecture extensions
• What is vectorization and why is it important
• Floating-point operations in x86-64

• Memory alignment and relation to cache lines
• Manual vectorization
• Intel intrinsics

1. Lab 5 due
   (2024-04-05)

1. Project designs due

• Main difference between CPU and GPU architectures
• Introduction to CUDA
• CUDA warps and threads

• Kokkos

Understanding machine instructions by learning how to debug code.

1. HW4 due
   (2024-04-16)
2. HW5 release
   (2024-04-17)
3. Quiz 4
   (2024-04-19)

• TBD
  • TBD
• Class summary

1. Lab 6 due
   (2024-04-26)

• Project final presentations:
  • 2024-05-02
  • 2024-05-03

1. HW5 due
   (2024-04-30)
2. Project deliverables due
   (2024-05-02 8:00 AM)
3. Project final presentations due
   (2024-05-02/03)