Heterogeneous parallel systems with a host CPU and various devices provide high performance and energy efficiency and are widely adopted from supercomputers to edge devices. In this course, you will learn common architectures and parallel algorithm patterns for heterogeneous systems, high-level programming interfaces for HPC and AI applications (OpenCL/OpenMP, TensorFlow), and performance optimization techniques, etc. and work on programming projects.

Computer architectures and data structures are recommended prerequisite classes.

Midterm: 15%
Final exam: 20%
Programming assignments: 40%
Project: 20%
Attendance: 5%

Week 1
9/5 Introduction and overview
9/7 Parallel computing basics 1

Week 2
9/12 Parallel computing basics 2
9/14 GPU architectures 1

Week 3
9/19 GPU architectures 2
9/21 Intro to CUDA

Week 4
9/26 CUD programming model, Lab 0 out
9/28 No class (Chusuk holiday)


Week 5
10/3 No class (National foundation day)
10/5 Memory and data locality 1, Lab 0 due

Week 6
10/10 Memory and data locality 2, Lab 1 out
10/12 Thread execution efficiency

Week 7
10/17 Performance considerations 1, Lab 1 due, Lab 2 out
10/19 Performance considerations 2

Week 8
10/24 Parallel patterns: convolution 1, Lab 2 due, Lab 3 out
10/26 Parallel paterns: convolution 2

Week 9
10/31 Lab 3 due
11/2 Midterm 3:30pm-5:30pm

Week 10
11/7 Parallel patterns: histogram, Lab 4 out, Project out
11/9 Parallel patterns: reduction

Week 11
11/14 Parallel patterns: scan, Lab 4 due, Lab 5 out
11/16: Parallel patterns: sparse computation 1

Week 12
11/21 Parallel patterns: sparse computation 2, Lab 5 due, Lab 6 out
11/23 Related GPU programming models 1

Week 13
11/28 Related GPU programming models 2, Lab 6 due
11/30 Multi-GPU

Week 14
12/5 Libraries and framework support
12/7 Efficient deep learning

Week 15
12/12 DL accelerators
12/14 Project final presentations

Week 16
12/21 Final exam 3:30pm-5:30pm
12/23 Project due