What is Parallel Computing?

• Parallel computing is type of computation in which many calculation or processes are carried out simultaneously.
• Large problems are divided into smaller ones, which can then be solved at the same time.
• This methods leverages multiple processing element concurrently to solve a problem more quickly than with a single processor.

Concrrency and Parellelism

Concurreny

• Refers to the ability of a system to handle multiple tasks or processes simultaneously.
• Task may not necessarily executes at the exact same time, but they progress in overlapping time periods.
• Multiple tasks can run in overlapping time periods, but not necessarily at the same time. This can create the illusion of parallel tasks running because of the CPU's rapid switching.
• Focuses on managing and coordinating multiple tasks.
• Can be achieved on a single-core processor through context switching.

Parallelism

• Involves actually executing mulitple tasks or parts of tasks simultaneously.
• Requries mutiple processing units (e.g. CPU cores, GPUs) to execute truly in parallel.
• Is a subset of concurrency - all parallel execution is concurrent, but not all concurrent exeution is parallel.

Key distinctions:

1. Execution: Concurrency is about dealing with lots of things at once, while parallelism is about doing lots of things at once.
2. Hardware requirements: Concurrency can be implemented on a single-core processor, while parallelism requires multi-core processors or distributed systems.
3. Goal: Concurrency aims to improve responsiveness and manage multiple tasks efficiently, while parallelism aims to improve performance by splitting tasks into smaller parts that can be processed simultaneously.

‍
​​

​