These days you hear lots of buzzwords like cloud Computing, Parallel computing etc.

We shall try to understand what is parallel computing and what’s possible in C# 4.0.

Parallel Framework, the Need

Parallel Framework is a library of classes which enables you to write your .Net code in a
parallel fashion. It is specifically written to support task-parallelism for multi-core (and
multi-processor) machines.

Parallel LINQ, the Parallel class, the task parallelism constructs, and the concurrent
collections are collectively known as PFX (Parallel Framework) and have been introduced
from .Net 4.0.
The Parallel class along with the task parallelism constructs is called the Task Parallel
Library or TPL.
The Task Parallel Library (TPL) is a set of public types and APIs in the System.Threading
and System.Threading.Tasks namespaces in the .NET Framework version 4. The purpose
of the TPL is to make developers more productive by simplifying the process of adding
parallelism and concurrency to applications.

As the growth of CPU clock speeds have stagnated and manufacturers have shifted their
focus to increasing number of core counts, there seems to be a magical limit around the
2 GHz. We have a real problem to extract the benefit of the new processing technology
as our standard single-threaded code will not automatically run faster as a result of
those extra cores.
By creating more cores, the chip producers enable the OSes to have more threads before
the system slows down, as the OS is able to divide the threads over multiple cores.

Types of Parallel Programming

 Bit-level parallelism
 Instruction-level parallelism
 Data parallelism
 Task parallelism
Bit-level parallelism is a form of parallel computing based on increasing processor
word size. Increasing the word size reduces the number of instructions the processor
must execute in order to perform an operation on variables whose sizes are greater than
the length of the word. (For example, consider a case where an 8-bit processor must add
two 16-bit integers. The processor must first add the 8 lower-order bits from each
integer, then add the 8 higher-order bits, requiring two instructions to complete a single
operation. A 16-bit processor would be able to complete the operation with single
instruction)

Instruction-level parallelism (ILP) is a measure of how many of the operations in a

computer program can be performed simultaneously. Modern processors have multi-
stage instruction pipelines. Each stage in the pipeline corresponds to a different action
the processor performs on that instruction in that stage; a processor with an N-stage
pipeline can have up to N different instructions at different stages of completion.

The canonical example of a pipelined processor is a RISC processor, with five stages:
instruction fetch, decode, execute, memory access, and write back. The Pentium 4
processor had a 35-stage pipeline. A five-stage pipelined superscalar processor, capable
of issuing two instructions per cycle. It can have two instructions in each stage of the
pipeline, for a total of up to 10 instructions (shown in green) being simultaneously
executed. In addition to instruction-level parallelism from pipelining, some processors
can issue more than one instruction at a time. These are known as superscalar
processors. Instructions can be grouped together only if there is no data dependency
between them

Data parallelism (also known as loop-level parallelism) is a form of parallelization of

computing across multiple processors in parallel computing environments. Data
parallelism focuses on distributing the data across different parallel computing nodes. It
contrasts to task parallelism as another form of parallelism.
For instance, consider a 2-processor system (CPUs A and B) in a parallel environment,
and we wish to do a task on some data d. It is possible to tell CPU A to do that task on
one part of d and CPU B on another part simultaneously, thereby reducing the duration
of the execution. The data can be assigned using conditional statements

Task parallelism (also known as function parallelism and control parallelism) is a form
of parallelization of computer code across multiple processors in parallel computing
environments. Task parallelism focuses on distributing execution processes (threads)
across different parallel computing nodes. It contrasts to data parallelism as another
form of parallelism.
In a multiprocessor system, task parallelism is achieved when each processor executes a
different thread (or process) on the same or different data. The threads may execute the
same or different code. In the general case, different execution threads communicate
with one another as they work. Communication takes place usually to pass data from
one thread to the next as part of a workflow.

As a simple example, if we are running code on a 2-processor system (CPUs "a" & "b") in
a parallel environment and we wish to do tasks "A" and "B" , it is possible to tell CPU "a"
to do task "A" and CPU "b" to do task 'B" simultaneously, thereby reducing the runtime
of the execution. The tasks can be assigned using conditional statements

Difference between Multithreaded Applications and Parallel Programing

Multithreaded application can run code concurrently on a single core, two threads can
never execute at the same time on the cpu. This code executes simultaneously, but in
parallel. When you have too many executing threads, your system slows down, as there
is not enough time to be sliced for all the threads to run at full speed.

Parallel programing is executing the threads simultaneously amongst the cores.

The number of threads created by the thread pool is dependent on several parameters
like
 The number of processors (or cores)
 The version of your .Net framework
 The processing load

The number of cores determine the maximum available threads which could be
concurrently available in the thread pool.
The version of .Net framework sets another maximum of available threads in the
pool. For ex the 3.x framework sets it to 256 threads.

So the available maximum is roughly calculated by these two figures. The actual
number of threads used by the process is calculated by load
The ParallelFramework sets the maximum number of threads to twice the maximum of
the threadpool. This is done so that tasks can be queued. This takes more memory, but
is also quicker, because there's always work in the queue for the threadpool. Threads
don't have to wait idle.

Parallel Framework the Basics

Parallel Framework is a library of classes which enables you to write your .Net code in a
parallel fashion. It is specifically written to support task-parallelism for multi-core (and
multi-processor) machines.
Parallel LINQ, the Parallel class, the task parallelism constructs, and the concurrent
collections are collectively known as PFX (Parallel Framework) and have been introduced
from .Net 4.0.
The Parallel class along with the task parallelism constructs is called the Task Parallel
Library or TPL.
The Task Parallel Library (TPL) is a set of public types and APIs in the System.Threading
and System.Threading.Tasks namespaces in the .NET Framework version 4. The purpose
of the TPL is to make developers more productive by simplifying the process of adding
parallelism and concurrency to applications.

Functional Programming and Parallel Programming

Functional programming languages are at heart the best languages for parallelism. This
is because with functional languages you program the computer what to do instead of
how to do it. This gives the compiler the power of finding out the best way to subdivide
the program into tasks.

Microsoft has just launched a new functional programming language called F#. It has all
the advantages of a functional language, but it’s also fully compatible with the .Net
library. But the best thing is, that you are able to write your parallel code in the F#
language, and use those F# classes (!) in any other .Net language like C# or VB.Net.
That way you can combine the power of F# with the power of C#!

In my opinion such a combination is far more preferable than trying to fit in some
functional frameworks into an imperative language. Both languages have their own
power, let’s keep it that way.

Example ForEach Implementation with PFX and PLINQ and contrasting with
traditional methods

// Sequential version
foreach (var item in Items)
{
Process(item);
}
// Parallel equivalent 
Parallel.ForEach(Items, item => Process(item));

// PLINQ equivalent 
var queryA = from item in Items.AsParallel()
select ExpensiveFunction(item); //good for PLINQ
 
var queryB = from item in Items.AsParallel()
where item.No % 2 > 0 
select item; //not as good for PLINQ

Namespaces and classes in TPL

The primary class of the TPL is System.Threading.Tasks.Parallel. This class supports a
number of methods to iterate over a collection of data (specifically, an object
implementing IEnumerable<T>) in a parallel fashion. This class supports two primary
static methods, Parallel.For() and Parallel.ForEach(), each of which defines numerous
overloaded versions.
Both of these methods require an IEnumerable or IEnumerable<T> compatible container
that holds the data needed to process in a parallel manner. The container could be a
simple array, a non-generic collection (such as ArrayList), a generic collection (such as
List<T>) or the results of a LINQ query.

System.Func<T> and System.Action<T> delegates can be used to specify the target

method which will be called to process the data.
Note that that Func<T> represents a method which can have a given return value and a
varied number of arguments. The Action<T> delegate is very similar to Func<T>, in that
it allows to point to a method taking some number of parameters. However, Action<T>
specifies a method which can only return void.

Parallel.For() and Parallel.ForEach() methods can be used to pass a strongly typed

Func<T> or Action<T> delegate object, we can simplify your programming by making
use of a fitting C# anonymous method or lambda expression.

PFX Concepts
There are two strategies for partitioning work among threads:
 data parallelism
 task parallelism

Data Parallelism (Task Parallel Library)

Data parallelism is enabled by several overloads of the For and ForEach methods in the
System.Threading.Tasks.Parallel class. In data parallel operations, the source collection
is partitioned so that multiple threads can operate on different segments concurrently.
TPL supports data parallelism through the System.Threading.Tasks.Parallel class. This
class provides method-based parallel implementations of for and foreach loops (For and
For Each in Visual Basic). You write the loop logic for a Parallel.For or Parallel.ForEach
loop much as you would write a sequential loop. You do not have to create threads or
queue work items.

// Sequential version
foreach (var item in sourceCollection)
{
Process(item);
}

// Parallel equivalent
Parallel.ForEach(sourceCollection, item => Process(item));

Task Parallelism (Task Parallel Library)

The term task parallelism refers to one or more independent tasks running concurrently.
A task represents an asynchronous operation, and in some ways it resembles the
creation of a new thread or ThreadPool work item, but at a higher level of abstraction.
Tasks provide two primary benefits:
 More efficient and more scalable use of system resources.
Behind the scenes, tasks are queued to the ThreadPool, which has been enhanced
with algorithms (like hill-climbing) that determine and adjust to the number of
threads that maximizes throughput. This makes tasks relatively lightweight, and
you can create many of them to enable fine-grained parallelism.
 More programmatic control than is possible with a thread or work item.
The Parallel.Invoke method provides a convenient way to run any number of arbitrary
statements concurrently. Just pass in an Action delegate for each item of work. The
easiest way to create these delegates is to use lambda expressions. The lambda
expression can either call a named method, or provide the code inline. The following
example shows a basic Invoke call that creates and starts two tasks that run
concurrently.

Parallel.Invoke(() => DoSomeWork(), () => DoSomeOtherWork());

Copy

// Create and start the task in one operation.

var taskA = Task.Factory.StartNew(() => DoSomeWork());

ParallelOptions

In the Task Parallel Library, configuration is handled via the ParallelOptions class. All of
the methods of the Parallel class have an overload which accepts a ParallelOptions
argument.
var options = new ParallelOptions();
options.MaxDegreeOfParallelism = Math.Max(Environment.ProcessorCount / 2,
1);

Using LINQ
int maxThreads = Math.Max(Environment.ProcessorCount / 2, 1);
double min = collection
.AsParallel()
.WithDegreeOfParallelism(maxThreads)
.Min(item => item.PerformComputation());

var reversed = collection

.AsParallel()

.WithExecutionMode(ParallelExecutionMode.ForceParallelism)
.Select(i => i.PerformComputation())
.Reverse();