Parallel Computing

Overview

The Parallel Computing lecture covers software and hardware-related topics of parallel systems, algorithms, and application design. In addition to learning theoretical foundations, students will get hands-on experience with selected parallel computing topics.

Key information

Learning Objectives

Successfully completing the lecture, students are able to

• define and describe the benefit of parallel computing and identify the role of software and hardware in parallel computing
• specify the Flynn classification of parallel computers (SISD, SIMD, MIMD)
• analytically evaluate the performance of parallel computing approaches (Scaling/Performance models)
• understand the different architecture of parallel hardware and performance improvement approaches (e.g., caching and cache coherence issues, pipeline, etc.)
• know and define the interconnects and networks and their role in parallel computing
• architecture of parallel computing (MPP, Vector, Shared memory, GPU, Many-Core, Clusters, Grid, Cloud)
• design and develop parallel software using a systematic approach
• generate parallel computing algorithms and development environments (i.e. shared memory and distributed memory parallel programming)
• write parallel algorithms/programs using different paradigms and environments (e.g., POSIX Multi-threaded programming, OpenMP, MPI, OpenCL/CUDA, MapReduce, etc.)
• understand and develop sample parallel programs using different paradigms and development environments (e.g., shared memory and distributed models) expose to some applications of Parallel Computing

Prerequisites

None.

Recommended Prerequisites

• Computer architecture
• Basic knowledge of computer networks and topologies
• Data structures and algorithms
• Programming in C(/C++)

Examination

Type: Written exam (90 minutes) or oral examination, 20 minutes, with grade

Expectation on the examinee

Profund knowledge of:

• Parallel programming
• Shared Memory Parallelism
• Distributed Memory Parallelism
• Single Instruction Multiple Data (SIMD)
• Multiple Instruction Multiple Data (MIMD)
• Hypercube
• Parallel interconnects and networks
• Pipelining
• Cache Coherence
• Parallel Architectures
• Parallel Algorithms
• OpenMP
• MPI
• Multi-Threading (pthreads)
• Heterogeneous Parallelism (GPGPU, OpenCL/CUDA).

Recommended Reading

• An Introduction to Parallel Programming, Peter S. Pacheco, Morgan Kaufmann(MK), 2011, ISBN: 978-0-12-374260-5.
• Designing and Building Parallel Programs, Ian Foster, Addison-Waesley, 1995, ISBN 0-201-57594-9 (Available online).
• Advanced Computer Architecture: Parallelism, Scalability, Programmability, Kai Hwang, Int. Edition, McGraw Hill, 1993, ISBN: 0-07-113342-9.
• In addition to the mentioned text book, tutorial and survey papers will be distributed in some lectures as extra reading material.