05 Ipc
05 Ipc
Coordination and
Communication
Or
IPC
Sources of Concurrency
Sources of Concurrency in Operating
Systems
• Processes
• Threads (sometimes called light weight processes)
• Scheduling
• All general purpose operating systems are preemptive!
• Single CPU systems have concurrency too
• Multiple CPUs add their own flavor to this
• Sometimes we call threads and processes together as “tasks”
• Note – Kernel has a lot of concurrent tasks!
The concept of ‘threads’
and an example with Linux
pthreads library functions.
This is another way (in addition to processes) to create concurrency
Concurrency with Threads (as
opposed to processes)
• Threads share almost everything with each other
• No parent-child relationship (in terms of PIDs etc)
• Shared address space – watch out – this can be dangerous! However this
means they can easily share data.
• Except the call stack!
• They are identified by different thread IDs (not different PIDs)
• Linux calls them as light-weight processes(LWP) see ps -eLF
• They are independently schedulable
• Note – User threads vs Kernel threads
Processes and threads – creating
and using threads in practiceP1
P1 T1
Stack_T1
fork() Stack_T2 Thread_create()
P2 T2
wait() join()
exit() Thread_exit()
C B
Coordination of
Concurrency
… so as to avoid races and
deadlocks
Notion of a Critical Section and
Mutual Exclusion
• Clearly in our example tasks T1 , T2 and T3 had each some code such that it
would work perfectly fine provided they execute their code exclusively (no two
at the same time).
• Such piece of code is called a critical section. (Heard this before?)
• Mutual exclusion (consisting of lock and unlock operations) is the mechanism
used to protect critical sections.
• Thought as a barrier, only one process allowed to pass at any time.
A thread does a lock operation T1 may access like this: oversimplified lock implementation:
simply turn off all interrupts
to enter the critical section lock_the_critical section
and… // This is the critical section code and an unlock restores interrupts
an unlock operation to signal balance= balance + 100
unlock .. Works, ok for a small time! But…
exiting sectionfrom the ciritical not a good idea in general!
Important
• Mechanism vs Usage:
• Lock and unlock are mechanisms to implement a critical section
• In reality only proper use by the programmer ensures mutual exclusion
• Lets take a look at how the lock and unlock operations may be
implemented so that it can be used by the programmer to implement
mutual exclusion to access critical sections.
Implementing the lock()
and unclock() operations
A Proposed locking code – Aim is
that at most one process is in the !
CSImplementation
at any time Usage E D
IL
struct lock { Task T1 ( or T2, r T3, etc..)
FA
int flag=0; (step, flag)
}L1 ;
lock(&L1) T1 T2
5. // We are in the critical section
lock( struct lock * l){ 10
6. // so we can modify data 1 0
1. while (l->flag == 1 )
unlock(&L1) 3 1
2. ; // keep looping if locked 3 1
3. lock->flag=1; // Got it, so 5
// indicate 5
} Can you say what could easily go
unlock( struct lock * l){ wrong?
4. l->flag=0; // Declare free
}
Ref: www.ostep.org chapter 28
Peterson’s Algorithm
What is an issue
with this idea?
Hardware support: Test and Set
• A special machine instruction: LOAD flag,ax • init(int * flag ) { *flag=0; }
STORE 1,flag
TSL ax, flag • lock(int * flag) {
• Copy original value of flag to ax, then set while( test_and_set(flag) == 1 )
flag=1 in a single instruction ;
• int test_and_set( int *flag){ // flag was 0, now is 1
TSL ax, flag }
RET // ax value is returned
} int flag; // global shared
• If original value was 0 then, we got the lock
T1::
(returns 0) and others cant get it (flag=1).
lock(&flag)
• unlock(int * flag) is simple:
// This is the critical section code
STORE 0, flag balance= balance + 100
The implementations here are a model for understanding and
not intended to be compilable assembly/C code! unlock(&flag)
Avoiding the busy loop
• Notice: Our solution has a busy loop! • What if multiple people are waiting for
• Busy loop consumes too many CPU cycles. the lock
• Possible starvation
• Alternate:
• A syscall to give up CPU “yield until a later time”
• One solution
• lock( int * flag) { • If yield could put waiters in a queue
while( test_and_set(flag) == 1 ) • and unlock wakes them up sequentially.
yield • In FCFS order
// here flag was 0, now is 1
}
• These ideas are combined in the
concept called a mutex
• An extended idea is that of a
• In Linux the closest yield is semaphore.
• sched_yield() or pthread_yeld() or sleep() or nanosleep()
A practical example in the pthreads library:
Mutual Exclusion using a mutex.
• We saw how to create threads using pthreads.
• We can use the mutex provided by pthreads to prevent multiple
threads from simultaneously entering the critical section:
… … critical section …
// initialize before use
pthread_mutex_init(&mymutex // to exit the critical section
, NULL); // NULL is attributes pthread_mutex_unlock(&mymutex)
// returns 0 if and only if success ;
See how a mutex is used in
pthreads/update_balance_mutex.c
Mutex ensures locking
but sometimes we want conditional
locking
• Revisit the find_some_primes.c example with two readers.
• See how there can be a race to deal with access to “front” and “rear”
• Let’s see a solution using mutex to make sure only one thread accesses these
variables at a time.
• pthreads/find_some_primes_mutex.c
• Notice how this still had a busy wait even though access was mutually
exclusive!
• We are waiting for a condition that can only come to be if another threads changes
front or rear.
• So we gave up the mutex
• We would prefer to wait for some other process to tell us that perhaps the condition
we are waiting for is met. (slot is free or slot is filled)
See : pthreads/find_some_primes_mutex.c
Condition Variables
• They work in tandem with Now the reader in our example
mutexes • Does a lock_mutex
• You can wait for a condition and • Waits for a condition for a filled
give up the mutex in one shot slot
• You can also signal to other • Once it wakes up it has the
threads that a condition has mutex lock
occurred
while( desired condition1 not met) { while( desired condition2 not met) {
pthread_cond_wait(&desiredcv1, pthread_cond_wait(&desiredcv2,
&mutex); &mutex);
} }
// do CS // do CS
pthread_cond_signal(&desiredcv2); pthread_cond_signal(&desiredcv1);
pthread_mutex_unlock(&mutex); pthread_mutex_unlock(&mutex);
See: pthread/find_some_primes_cv.c
Semaphore – A more general
solution:
counter
• Conceptually a semaphore is
0
two things:
• A counter (of allowed tasks) n ( 1 3375 4175 2100
in our example)
• A queue (of waiting tasks) wq FIFO Queue
• Often initially
• Basically the semaphore is a
• n=MAX (MAX=1 in our example)
barrier except it allows a MAX
• wq= empty list number of threads to pass the
barrier (MAX possibly > 1)
• It behaves like a mutex if MAX is 1
Implementing sem_wait and
sem_post atomically Edsger
• sem_wait( S )
P • sem_post( S )
V Dijkstra
ATOMIC !
if (S.n > 0 ) if ( S.wq is empty )
S.n -- ; S.n ++ ;
Try to get else else Let others know..
in
addq(self, S.wq) t = delq(S.wq) On the way out
blocked() movetoready(t)
Consumers:
MAX number of array slots used circularly keep getting items from the buffer
put(val){
buffer[free]=val;
free=free+1 % MAX;
}
get(){
tmp=buffer[used];
• We only want as many producers to put() as there are free slots used=used+1 % MAX;
return tmp
• We only want as many consumers to get() as there are filled slots }
• Also we want only one thread updating the index at any time!
A PC problem solution that combines
Mutexes and POSIX Semaphores
Too many people Producer(){
accessing free and used, …
so we’ll guard them with
sem_wait(&empty_slot); //wait till empty_slot
locks ie mutex
lock(); put(x); unlock(); // we don’t want race for “free”;
put(val){ sem_signal(&filled_slot) //signal anyone waiting for a filled slot
buffer[free]=val; …
free=free+1 % }
MAX;
}
init(&empty_slot) :: empty_slot.n=MAX
get(){ init(&filled_slot) :: filled_slot.n = 0
tmp=buffer[used];
used=used+1 % Consumer(){
MAX; …
return tmp sem_wait(&filled_slot); //wait till filled slot
} lock(); x=get(); unlock(); // we don’t want race for “used”;
sem_signal(&empty_slot); //signal anyone waiting for an empty slot
See ipc/sem-posix/pc.c
…
}
Aside: Shared memory among
processes
• We will read soon about how P1 P2
processes, not just threads can
share parts of their address spaces.
• The exact techniqes is called “POSIX
shared memory” and we come to
that later.
• However, this means all the stuff
about mtexes etc are valid between
processes too as long as they are
located inside the shared memory.
Processes and semaphores and
Linux
• POSIX supports two types of semaphores
• Unnamed: These are usually used between threads. However they can be
used between processes if they are using shared memory.
• When the processes exit, it is gone
• Named sempahores: These semaphores have a name. In fact they are
created in
• /dev/shm
• It is persistent (exist even after the processes are gone) and needs to be explicitly
removed
• Can also be used between threads
• They can be used between processes to control access to shared memory data
• They can be used between processes even if they don’t share memory
Semaphore can be used as a mutex
too! Producer(){
…
sem_wait(&empty_slot); //wait till empty_slot
sem_wait(&lock); put(val); sem_post(&lock);
sem_signal(&filled_slot) //signal anyone waiting for a filled slot
…
put(val){ }
buffer[free]=val;
free=free+1 % MAX;
} init(&empty_slot) :: empty_slot.n=MAX
init(&filled_slot) :: filled_slot.n = 0 init(&lock) :: lock.n=1
get(){
tmp=buffer[used]; Consumer(){
used=used+1 % MAX; …
return tmp sem_wait(&filled_slot); //wait till filled slot
} sem_wait(&lock); put(val); sem_post(&lock);
sem_signal(&empty_slot); //signal anyone waiting for an empty slot
…
}
Monitors: A high level abstraction
• Locks and Semaphores rely heavily on the programmer to make sure
they use them appropriately.
• With the intent to make life better for the programmer languages
introduce the notion of Monitors in the programming language itself
• A Monitor is like an object in Object Oriented Languages: Think of it as
a qualification to a class!
• With the additional semantics that only one task is allowed to execute
a method.
• It is implemented on top of semaphores/mutexes/condition variables.
Deadlocks
Example: Dining Philosophers’
Problem
Implemented with a
Philosopher(i) { semaphore to avoid
loop { race
P3 think(i)
eat(i)
f2 }
f3 }
P2 P4 eat(i){ A problem remains:
pick_up(fi)
pick_up(fi-1)
Deadlock
f1
f4 // actually eat
put_down (fi)
put_down (fi-1) How to solve?
P5
P1 f5 }
PS: Think of fi as fork to the left of PI
and fi-1 as fork to the right of Pi
Solving deadlock problems in OS
Now it is available for read and write IMPORTANT: the existence of the shared memory block is
like any other part of the memory! not tied to the life of the processes using it! … needs to be
explicitly removed when we don’t need it.
POSIX interface for shared memory
ops
• shm_fd = shm_open(SHM_NAME, O_CREAT|O_RDWR,0644);
• Just like a file open sample name: /example1
• See /dev/shm
• ftruncate(shm_fd, SHARED_MEMORY_SIZE);
• Setting the size
• s = mmap(NULL, sizeof(struct mystruct), PROT_READ | PROT_WRITE,
MAP_SHARED, shm_fd, 0);
• Mapping the area to memory
• You can write into the area pointed by s
• shm_close(shm_fd);
• shm_unlink(SHM_NAME);
See ipc/shm-posix/*.c
Communication without shared
space
• The goal is to share data, • Conceptually common:
without any form of shared
space
• Typically
• Files
• Pipes
• Pipe() – fd pair
• Named pipe (similar to a file)
• Basically a queue with variations in:
• Message queues
• A way for different processes to identify and access it
• Sockets • Send and receive operations
• Data format
Pipes and named pipes
• Pipes: We saw
fd[1] fd[0]
• int fd[2];
• pipe(fd)
Message queues
Message queue ABCDE FGH IJKLMN …. ….
• Unlike a pipe, a message queue typically has a notion of a data unit “message”.
• Any number of processes can open, read or write.
• Messages are stored in an order until anyone reads.
• write() and read() are usually called send() and receive().
• So one talks of “sending” and “receiving” messages..
• Messages also have the notion of a special tag or a priority used to select or wait for an
appropriate message during receive.
• Also note that we need some common way (across processes) to identify a message
queue – some kind of pre-agreed name or key just like we had the name of the fifo in the
previous example.
com
m
• POSIX message queue
• POSIX semaphore
• There is another interface (different calls, but same underying
functionality) offered by Linux and several older UNIX flavors.
• System-V interface
• So when you look up any manual page, please ensure you are looking at the
right pages.
• We studied POSIX because it is modern and simplified.
Sockets
Communication between processes
via sockets
• Sockets provide a mechanism that work
• Between processes inside an OS instance (UNIX sockets)
• Between two different systems also! (INET sockets)
• It is two way.
• It is the basis of communication between nodes in a network and on the
internet
• Communication primitive is based on the notion of send-recv, However…
• identity is established in a manner that is internet wide acceptable.
• Here the notion of the identity of a socket endpoint is a 5-tuple
• Two endpoints together make the socket connection
Socket: Communication by send()
and recv()
• send(sockfd, data_buff, data_length, flags) or write()
• recv(sockfd, data_buff, data_length, flags) or read()
• These are very similar in form to read and write.
• What is sockfd ?
• It is like a file descriptor.
• It identifies a communication endpoint.
The internet socket endpoints
Port-1 Port-2
P1 P2
TCP
OS-1 OS-2
on Computer-1 on Computer-2
Computer-1 Computer-2
See:
netstat –numeric-ports –a (to see what ports are active)
Try running a process waiting on 8080:
Aside: A little experiment to
understand this
• Use netstat --numeric-ports -a
… and see some connection activity
… can you identify some sockets?
• Here is a simple server to listen and respond to a connection :
while : ; do cat README | nc -l -p 8080 ; done
• Use netstat again and see that 8080 has someone in LISTEN
mode
• Now run a wget localhost:8080 and see netstat output again
• Fun aside: run wireshark and see data moving in and out of your PC
• See /etc/services for standardized service-port mappings
Socket: Understanding a socket file
descriptor
• Recall: send(sockfd, data_buff, data_length, flags) …
similarly recv(sockfd,…))
• What is sockfd ? It identifies a socket.
• A sockfd can be used for send recv only after it has both endpoints established.
• In particular it identifies the protocol being used. Common varieties:
• Internet sockets: TCP, UDP UNIX domain sockets
• An internet socket has IP addresses and ports associated with the endpoints:
• { local_ip_address, local_port, remote_ip_address, remote_port,
protocol }
• This is got not all at once, but in stages – connection establishment process
• A Unix domain socket only works within the OS
• Identified by a socket path name (like we saw semaphore and fifo names)
Focus for this study: a TCP
connection
• Initially create a socket (and get an fd) which has not yet established a
connection. (‘half’ socket)
• The connection process identifies the endpoint pair and now both
ends are sure who they are talking to. This is the ‘full’ socket.
• Once connected then it is like having two directed connections
between the end points to send and receive:
P1 P2
Port Port
TCP
TCP
& IP & IP
addr addr
Server Client
• Create one “half” socket • Create one “half” socket
• Set it to listen mode and
wait
• Once you receive a • Establish connection
client request, accept it with a server creating a
and create a new “full” “full” socket.
socket
• Use the new full socket • Use the new full socket
for send() recv() for send() recv()
2. connect(sockfd, &server,
sizeof(server)); close(sockfd);
Simple server vs
concurrent server lsockfd = socket(…);
// setup the address
lsockfd = socket(…); bind(lsockfd,…);
// setup the address listen(lsockfd,…);
bind(lsockfd,…);
listen(lsockfd,…); Loop{
newfd=accept(lsockfd,…);
• Simple tcp server – client : Single • You can see connection states:
message exchange netstat –a ( ..| more)
• Concurrent tcp server – long • Sockets don’t vanish as soon as
running client (5 messages) your program exits… So be
• Simple udp server – client : Single patient.
message exchange • Good practice to unlink Unix
• Simple udp server in a loop that domain sockets at the beginning.
echos any client message
• Simple unix domain socket server
– client : Single message exchange
Sockets… there is more
• Socket programming in practice is a more elaborate topic that is
covered typically in a network programming course
Signals as a way to communicate
P1
• [Despite the name it has nothing to do with condition variables or
semaphores ] P2
See: misc/sig.c
Summary: Communication methods
in OS
• Shared memory between processes too
• Named Pipes
• Message queues
• Sockets – can be used between different systems as well.
• Signals
• We also saw a POSIX standard implementation of several of these.
This ends the lectures on
Concurrency
Coordination and
Communication