Shared Memory Programming

Pthreads
Dr Matthew Grove
Slides: http://people.cs.vt.edu/~mjeg/

Slide credits: Dr B. Wilkinson and Blaise Barney

August 24, 2011

Tuesday, September 13, 2011

 Lecture Goals
Ø Shared memory, threads and
processes
Ø Basic Pthreads
Ø Thread safety
ØRace conditions
ØCritical sections
ØLocks
Ø More advanced Pthread techniques
August 24, 2011

Tuesday, September 13, 2011

 Shared Memory +
Multi-Processor
Ø Any memory location can be accessed by
any of the processors.
Ø There is a single address space with each
memory location given a unique address.
Ø Because a program can access shared
memory you do not have to use message
passing exchange data.

August 24, 2011

Tuesday, September 13, 2011

 Shared Memory Programming
Ø Generally more convenient although it
does require access to shared data by
different processors to be carefully
controlled by the programmer.
Ø Shared memory systems have been
around for a long time but with the
advent of multi-core systems, it has
become very important to able to
program for them.
http://www.computerworld.com.au/article/400466/parallel_programming_skills_crisis_could_stall_server_evolution
August 24, 2011

Tuesday, September 13, 2011

 Techniques
Ø Heavyweight processes.
Ø Threads. Example Pthreads, Java threads.
Ø New programming language for parallel
programming - not popular. Example Ada.
Ø Modifying the syntax of an existing sequential
programming language to create a parallel
programming language. Example UPC.
Ø Using an existing sequential programming
language supplemented with compiler directives
and libraries for specifying parallelism. Example
OpenMP.
August 24, 2011

Tuesday, September 13, 2011

 Heavyweight Processes
Ø Operating systems are often based upon the
notion of a process.
Ø Processor time is shared between processes,
switching from one process to another. This
might occur at regular intervals or when an
active process becomes delayed.
Ø Concept can be used for parallel
programming. Not much used because of
overhead. Threads are preferred.

August 24, 2011

Tuesday, September 13, 2011

 Threads vs Processes

August 24, 2011

Tuesday, September 13, 2011

 Thread Summary
Ø Exists within a process and uses the process’
resources.
Ø Has own independent flow of control as long as its
parent process exists and the OS supports it.
Ø Duplicates only the essential resources it needs to be
independently schedulable.
Ø May share the process resources with other threads
that act equally independently.
Ø Dies if the parent process dies.
Ø Is "lightweight" because most of the overhead of
creating the thread occurs when the process starts.

August 24, 2011

Tuesday, September 13, 2011

 Thread vs Process - Create

Source: https://computing.llnl.gov/tutorials/pthreads/fork_vs_thread.txt
August 24, 2011

Tuesday, September 13, 2011

 Memory Bandwidth

Source: https://computing.llnl.gov/tutorials/pthreads/
August 24, 2011

Tuesday, September 13, 2011

 Pthreads
Ø Historically vendors provided their own thread
libraries.
Ø The adopted standard for Unix was created by
the IEEE and called POSIX 1003.1c (1995).
Ø Any implementation of this standard is referred
to as POSiX Threads or Pthreads.

August 24, 2011

Tuesday, September 13, 2011

 Simple Pthread Programming

August 24, 2011

Tuesday, September 13, 2011

 Detached Threads

August 24, 2011

Tuesday, September 13, 2011

 Basic Pthread Methods
pthread_create(
thread, /* Unique identifier for thread */
attr, /* Attributes like detached state */
routine, /* Method to execute after creation */
arg) /* Arguments for thread function */

pthread_join(
thread_id, /* Thread to wait for */
status) /* Exit status of thread */

August 24, 2011

Tuesday, September 13, 2011

 Pthreads Hello World
#include <pthread.h>
#include <stdio.h>
void * entry_point(void *arg){
printf("Hello world!\n");
return NULL;
}
int main(int argc, char **argv){
pthread_t thr;
if(pthread_create(&thr, NULL, &entry_point, NULL)){
printf("Could not create thread\n");
return -1;
}
if(pthread_join(thr, NULL)){
printf("Could not join thread\n");
return -1;
}
return 0;
}
August 24, 2011

Tuesday, September 13, 2011

 Compiling Hello World
Ø Out of the box most compilers can build
your Pthread program.
Ø Gcc example:
Ø gcc -Wall -lpthread -o hello hello.c
Ø The -l flag specifies a library to link against.
Because pthreads is a system library, gcc will
automatically find it for you.
Ø -Wall tells gcc to warn you about any non
fatal mistakes.

August 24, 2011

Tuesday, September 13, 2011

 Exiting a Thread
Ø The thread will exit naturally after the thread
start routine returns.
Ø Thread itself calls pthread_exit.
Ø Another thread calls pthread_cancel.
Ø The process itself exits.

August 24, 2011

Tuesday, September 13, 2011

 Execution Order
Ø On a single processor a thread will typically
execute until it is blocked.
Ø For multi-processor the threads execution
can be interleaved in time.
Ø Unless you are guaranteeing execution order
then you can not assume anything about it.

August 24, 2011

Tuesday, September 13, 2011

 Interleaved Execution
Ø Consider two threads which print sentences
to the terminal.
Ø The characters being printed will also be
interleaved making garbled sentences print
to the screen.
Ø T0: -> “The weather is nice.”
Ø T1: -> “I am going to the shops.”
Ø Output: “Th I e wam geatheoing to thr ie
shs nice. ops.”

August 24, 2011

Tuesday, September 13, 2011

 Thread Safe Routines
Ø Standard IO functions are thread safe so the
messages are not printed with interleaved
characters.
Ø Special care can be taken when writing functions
such that even when called by multiple threads
simultaneously they will produce the correct
results.
Ø The programmer must determine if a routine is
thread safe and take care of it. In general,
assume everything is not thread safe.

August 24, 2011

Tuesday, September 13, 2011

 Accessing Shared Data
Ø The programmer must carefully control access
to shared data.
Ø Consider two processes which both increment
the variable X:

August 24, 2011

Tuesday, September 13, 2011

 Race Conditions

August 24, 2011

Tuesday, September 13, 2011

 Critical Sections
Ø Critical sections provide a way to make sure only
one process accesses a resource at a time.
Ø The critical section is the code that accesses the
resource. They must be arranged such that only
one section can be executed at a time.
Ø This mechanism is known as mutual exclusion.

August 24, 2011

Tuesday, September 13, 2011

 Locks
Ø A lock is the simplest mechanism for ensuring
mutual exclusion of critical sections.
Ø It can only have two values:
Ø 1 - a process has entered the critical section.
Ø 0 - there is no process in the critical section.
Ø To enter the critical section you must acquire the
lock.
Ø On leaving the critical section you release the lock.
This is done by changing the value back to 0.

August 24, 2011

Tuesday, September 13, 2011

 Acquiring the Lock
Ø It must be guaranteed that only one thread can
acquire the lock.
Ø This C code is not adequate:
if (lock == 0) {
// lock is free
lock = 1;
}

Ø Multiple threads could be executing the if

statement simultaneously.

August 24, 2011

Tuesday, September 13, 2011

 Pthread Lock Routines
Ø Use mutually exclusive lock variables, also called
mutex variables.
pthread_mutex_lock(&mutex1);
critical section
pthread_mutex_unlock(&mutex1);
Ø A thread will wait until it can acquire the lock. If
multiple threads are waiting, only one will be
picked to receive the lock when the lock is
available. Only one thread will ever be able to
acquire the lock. Only the thread that locked
the mutex can unlock it.
August 24, 2011

Tuesday, September 13, 2011

 Critical Sections Slow Down
Ø Critical sections have the effect of serializing
code execution.
Ø High performance programs should aim to limit
the number of critical sections.
Ø Suppose every process reaches a critical section
at the same time. The processes must wait and
execute the critical sections in sequence.
Ø In this situation you may see parallel code
become slower than the serial version.

August 24, 2011

Tuesday, September 13, 2011

 Serialization Example

August 24, 2011

Tuesday, September 13, 2011

 Deadlock
Ø Deadlock can occur when two processes both
require a resource currently held by the other.

August 24, 2011

Tuesday, September 13, 2011

 Circular Deadlock
Ø Deadlock can also occur in a circular fashion
with several holding a resource required by
another.

August 24, 2011

Tuesday, September 13, 2011

 Pthread Try-Lock Method
Ø Pthreads provides a method to test if a lock is
being held without blocking the thread.

pthread_mutex_trylock()

Ø This method will return an EBUSY status if the

mutex is currently locked. If the mutex is not
locked it will lock it. This might help overcome a
deadlock situation but it is still requires careful
code design.

August 24, 2011

Tuesday, September 13, 2011

 General Semaphores
Ø Pthread ‘primitives’ can be used to create more
complex routines.
Ø The general semaphore (or counting
semaphore) can store an integer in a thread
safe way.
Ø Can be used to track available resources such
as work available in a queue.
Ø Available in thread libraries for languages such
as Java but do not exist in vanilla Pthreads.
Ø You can however write your own using the
Pthread methods.
August 24, 2011

Tuesday, September 13, 2011

 Condition Variables
Ø Often a decision about when to execute a critical
section is made using some kind of global
condition.
Ø For example if a total has reached a certain
point.
Ø This could be achieved by locking and unlocking
a variable in order to check it. This kind of
polling is very costly.
Ø Condition variables offer a solution.

August 24, 2011

Tuesday, September 13, 2011

 Condition Variables
Ø Thread 1:
Ø Lock mutex and check value of a global variable.
Ø Call pthread_cond_wait() to perform a blocking wait for a signal.
pthread_cond_wait() automatically unlocks the associated mutex
variable.
Ø Thread 2:
Ø Lock mutex.
Ø Change global counter.
Ø If counter has reached the value needed by thread 1 it calls
pthread_cond_signal().
Ø Unlock the mutex.
Ø Thread 1:
Ø Wake up when received signal.
Ø Mutex is automatically locked again.
August 24, 2011
Ø Explicitly unlock the mutex.

Tuesday, September 13, 2011

 Condition Variables
Thread 1 Thread 2

August 24, 2011

Tuesday, September 13, 2011

 Breaking Condition Variables
Thread 1 Thread 2

August 24, 2011

Tuesday, September 13, 2011

 Monitor Methods
Ø A programming technique for providing thread
safety.
Ø Wrap shared resources in an access procedure
which uses some kind of lock to prevent
concurrent calls by multiple threads.
monitor_update_foo(var){
lock(c);
...
//Update foo here
...
unlock(c);
}
August 24, 2011

Tuesday, September 13, 2011

 Race Example
#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>

#define NTHREADS 100

#define LOOPS 10000
int sum = 0;
pthread_mutex_t sum_mutex;

void *do_work() {
int i;
pthread_mutex_lock(&sum_mutex);
for (i=0; i<LOOPS; i++)
sum = sum + 1;
pthread_mutex_unlock(&sum_mutex);
pthread_exit(NULL);
}

...

August 24, 2011

Tuesday, September 13, 2011

 ...
int main(int argc, char *argv[]) {
int i;
pthread_t threads[NTHREADS];
pthread_attr_t attr;

pthread_mutex_init(&sum_mutex, NULL);
pthread_attr_init(&attr);
pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_JOINABLE);
for (i=0; i<NTHREADS; i++)
pthread_create(&threads[i], &attr, &do_work, NULL);

for (i=0; i<NTHREADS; i++)

pthread_join(threads[i], NULL);

printf("Thread sum: %d\n", sum);

sum=0;
for (i=0;i<NTHREADS * LOOPS;i++)
sum = sum + 1;
printf("Check sum: %d\n",sum);

pthread_attr_destroy(&attr);
pthread_mutex_destroy(&sum_mutex);
pthread_exit(NULL);
August 24, 2011
}

Tuesday, September 13, 2011

 Demo
Ø ...

August 24, 2011

Tuesday, September 13, 2011

 Homework
Ø Compile hello world for Pthreads.
Ø Assign each thread a unique number and make it
print the number.
Ø Synchronize the threads such that they always
print their number in ascending order.
“I am thread 0”
“I am thread 1”
“I am thread 2”
“I am thread 3”
...

Ø Do not use other libraries - do it with Pthreads!

August 24, 2011

Tuesday, September 13, 2011