Concepts of Distributed Systems

2006/2007

Processes

Johan Lukkien

11-Nov-06 Johan J. Lukkien, [email protected] 1

TU/e Informatica, System Architecture and Networking
 Programme
• Introduction & overview
• Communication
• Distributed OS & Processes
• Synchronization
• Security
• Consistency & replication

• Naming
• Fault Tolerance

11-Nov-06 Johan J. Lukkien, [email protected] 2

TU/e Informatica, System Architecture and Networking
 Processes
• A computer system typically realizes a few
computational activities on some data

• At a given time, such an activity is found to be in a

certain state

• The actual goal of a system is achieved through the

cooperation of several of these activities

• This presentation is about how these activities are

dealt with in a distributed context

11-Nov-06 Johan J. Lukkien, [email protected] 3

TU/e Informatica, System Architecture and Networking
 Processes (in distributed context)
• Threads & processes
• Client & Server architecture
• Migration
• Agents
• Example: load balancing

11-Nov-06 Johan J. Lukkien, [email protected] 4

TU/e Informatica, System Architecture and Networking
 Threads & Processes
• Process (“program in execution”)
– defines a data space
• virtuzalization
– has at least one associated thread
– unit of distribution

• Thread
– unit of concurrency
– unit of scheduling
• though with one thread per process, a process is often said to
be the unit of scheduling
– several threads ‘live’ within shared address space
• i.e., in a process

11-Nov-06 Johan J. Lukkien, [email protected] 5

TU/e Informatica, System Architecture and Networking
 Schematic

Process Thread

11-Nov-06 Johan J. Lukkien, [email protected] 6

TU/e Informatica, System Architecture and Networking
 Process overhead
• Creation
• Switching
– for each Inter Process Call – no shared memory
– TLB, MMU

11-Nov-06 Johan J. Lukkien, [email protected] 7

TU/e Informatica, System Architecture and Networking
 Threads
• Concurrency on shared memory
– no protection
– no expensive IPCs between threads
– easier to construct programs (?)
• exploit platform concurrency
• latency hiding

• State per thread

– control state: `context’
– stack

• Generally, little overhead for switching

– depending on underlying execution model

11-Nov-06 Johan J. Lukkien, [email protected] 8

TU/e Informatica, System Architecture and Networking
 Mode and context
• A mode switch
– changes the processor mode and loads a new program counter
– is needed for any event that requires kernel activity
– must usually be followed by saving some additional processor
state

• A context switch
– changes the memory management settings
– is needed, besides a mode switch, for only some events, e.g. if a
process switch is needed

• Except for the handler, most of the overhead of a mode switch

is performed in hardware
– hence, we would like to restrict the majority of the event handling
to just this

11-Nov-06 Johan J. Lukkien, [email protected] 9

TU/e Informatica, System Architecture and Networking
 Single and multi-threading
• Notes:
– a thread-switch
can be realized
just through a
mode switch
• light-weight
process

– threads that share

an LWP (i.e. user-
level threads):
• share a control
block
• don’t need a
mode switch
from: W. Stallings, Operating Systems, Internals and Design principles

11-Nov-06 Johan J. Lukkien, [email protected] 10

TU/e Informatica, System Architecture and Networking
 POSIX: Thread interface
• Start a function as a new thread
– setting attributes, e.g. stacksize;
– thread terminates when this function returns
• Extensive interface
– detach, join, cancel

pthread_t thread_id;

status = pthread_create (&thread_id, attr, func, arg_of_func);

/* func(arg_of_func) is started as a new thread; when attr == NULL some
* internal defaults are chose */

status = pthread_join (thread_id, &result);

/* wait for thread to terminate */

11-Nov-06 Johan J. Lukkien, [email protected] 11

TU/e Informatica, System Architecture and Networking
 The life of a thread in POSIX
• Main program: own thread
• Additional threads:
– created on demand or on receipt of signal
– thread code: a function in the program

released
ready blocked
pre-empted wait (resource,
scheduled condition, time, event)
initially
running done (function return),
or cancelled

terminated
removed
detached or joined

11-Nov-06 Johan J. Lukkien, [email protected] 12

TU/e Informatica, System Architecture and Networking
 Thread execution model
• Concurrency level: number of “engines” (virtual
processors) actually executing the threaded
program

• Virtual processors are scheduled by the kernel

• Concurrency level choices

– 1: no concurrency, no kernel activity in switching (all
threads are ‘user-level’ threads
– # threads: switching always becomes kernel activity
– in between: only blocking kernel calls and virtual processor
scheduling requires kernel activity

11-Nov-06 Johan J. Lukkien, [email protected] 13

TU/e Informatica, System Architecture and Networking
 Thread execution: Solaris
• Virtual processor: lightweight process
– LWP just executes threads (no memory space)
– blocking calls in user space switch LWP to new thread

11-Nov-06 Johan J. Lukkien, [email protected] 14

TU/e Informatica, System Architecture and Networking
 Thread use
• Client
– hide latencies
– communicate with several partners
• all references on a Web page, while displaying
• multiple RPC’s
• ....ease of programming

• Server
– handle multiple clients
• easy extension to multi-processor box
– hide blocking services (e.g. disk access)
• ...ease of programming

11-Nov-06 Johan J. Lukkien, [email protected] 15

TU/e Informatica, System Architecture and Networking
 Multithreaded Servers
• Dispatch/worker model

11-Nov-06 Johan J. Lukkien, [email protected] 16

TU/e Informatica, System Architecture and Networking
 Processes (in distributed context)
• Threads & processes
• Client & Server architecture
• Migration
• Agents
• Example: load balancing

11-Nov-06 Johan J. Lukkien, [email protected] 17

TU/e Informatica, System Architecture and Networking
 Model-View-Controller architecture
• Several views on same model (‘data’)

• User changes model by interaction with a view; events go to

controller

• Connect controllers of application

– standardize on user
interaction
• drag & drop
• in-place editing

• Note: all arrows

could be distributed

11-Nov-06 Johan J. Lukkien, [email protected] 18

TU/e Informatica, System Architecture and Networking
 The X-Window System

11-Nov-06 Johan J. Lukkien, [email protected] 19

TU/e Informatica, System Architecture and Networking
 Transparencies at client
• Access, location, replication
– in stubs, proxies, adapters

11-Nov-06 Johan J. Lukkien, [email protected] 20

TU/e Informatica, System Architecture and Networking
 Server design
• Contact
– Service coupled to fixed transport port
• e.g., ftp: 21, telnet: 23 etc.
– ‘Superserver’: handle
many ports
• dispatcher, e.g.,
inetd (at process level)

• How to interrupt server?

– out-of-band
communication
– control connection
• e.g. SIP, RTP

11-Nov-06 Johan J. Lukkien, [email protected] 21

TU/e Informatica, System Architecture and Networking
 Inetd (fork upon request)

Picture from ‘multitasking server architecture’,

Grone, Knopfel, Kugel, Schmidt, Potsdam University

11-Nov-06 Johan J. Lukkien, [email protected] 22

TU/e Informatica, System Architecture and Networking
 Server design
• Internal concurrency
– iterative: single request/reply
– concurrent: pass on request
• internal multithreading
• distributed organization

• Client state
– Server side
• preferred: stateless (open files, caches,..)
• performance, complexity, scalability (1 Mb for 1000 clients is
1Gb ....)
– Client side
• cookies, web-page content

11-Nov-06 Johan J. Lukkien, [email protected] 23

TU/e Informatica, System Architecture and Networking
 Workpool with sockets

Picture from ‘multitasking server architecture’,

Grone, Knopfel, Kugel, Schmidt, Potsdam University

11-Nov-06 Johan J. Lukkien, [email protected] 24

TU/e Informatica, System Architecture and Networking
 Apache organization (preforking)

Picture from ‘multitasking server architecture’,

Grone, Knopfel, Kugel, Schmidt, Potsdam University

11-Nov-06 Johan J. Lukkien, [email protected] 25

TU/e Informatica, System Architecture and Networking
 Object servers
• ‘Connect object to network’

• Policies:
– transient
• create upon initialization/1st access
• destroy upon server down/last client away
– memory sharing
– threading
• per object?
– access & activation
• skeleton: (un)marshalling, actual call
• adapter: implements policy; registers object
• request (de)multiplexing

11-Nov-06 Johan J. Lukkien, [email protected] 26

TU/e Informatica, System Architecture and Networking
 Example Object server

11-Nov-06 Johan J. Lukkien, [email protected] 27

TU/e Informatica, System Architecture and Networking
 Processes (in distributed context)
• Threads & processes
• Client & Server architecture
• Migration
• Agents
• Example: load balancing

11-Nov-06 Johan J. Lukkien, [email protected] 28

TU/e Informatica, System Architecture and Networking
 Migration
• Performance: load balancing
– minimize job latency
– notice:
• maximixing resource utilization and minimizing job latency are
conflicting

• Locality:
– run code where data is
– multiply code over data sources

• Postpone decisions
– dynamic configuration
– late binding

11-Nov-06 Johan J. Lukkien, [email protected] 29

TU/e Informatica, System Architecture and Networking
 Migration framework (Fugetta)
• Code and state using resources

• State
– execution segment
– data

• Resources, affected by migration

– memory
– processing

11-Nov-06 Johan J. Lukkien, [email protected] 30

TU/e Informatica, System Architecture and Networking
 Examples
• CS: client-server
• REV: remote
evaluation
• CoD: Code
on Demand
• MA: Mobile
Agents
• State*: new state

11-Nov-06 Johan J. Lukkien, [email protected] 31

TU/e Informatica, System Architecture and Networking
 Terminology
• Weak mobility
– just code & data – only start from initial state
• e.g. JAVA applets

• Strong mobility
– also exchange execution segment

• Sender or receiver initiated (push or pull)

• Cloning
– make exact copy – similar to UNIX fork(), but then child
process is remote

11-Nov-06 Johan J. Lukkien, [email protected] 32

TU/e Informatica, System Architecture and Networking
 Taxonomy for Code Migration

11-Nov-06 Johan J. Lukkien, [email protected] 33

TU/e Informatica, System Architecture and Networking
 Dangling references
• Question: what to do with references to (other)
resources upon migration
– depends on “stickyness” of resource to its host
• fixed (printer), fastened (database) or unattached (file)
– depends on “uniqueness” of binding
• precise (URL), value (library function), type (printer)

11-Nov-06 Johan J. Lukkien, [email protected] 34

TU/e Informatica, System Architecture and Networking
 Migration and Local Resources
• MV: move
• GR: Generate Globally Unique Reference
• CP: Copy value
• RB: re-bind to locally available

Resource-to machine binding

Unattached Fastened Fixed

Process-to-
resource By identifier MV (or GR) GR (or MV) GR
binding By value CP ( or MV, GR) GR (or CP) GR
By type RB (or GR, CP) RB (or GR, CP) RB (or GR)

11-Nov-06 Johan J. Lukkien, [email protected] 35

TU/e Informatica, System Architecture and Networking
 Migration in Heterogeneous Systems
• Problems
– different Instruction Set Architecture (ISA)
– different notion of executable, process, thread (OS)
– different representations of data

• Solutions
– Define virtual machine
• language interpretation
– Limit migration points
• translate into migratable format

11-Nov-06 Johan J. Lukkien, [email protected] 36

TU/e Informatica, System Architecture and Networking
 Processes (in distributed context)
• Threads & processes
• Client & Server architecture
• Migration
• Agents
• Example: load balancing

11-Nov-06 Johan J. Lukkien, [email protected] 37

TU/e Informatica, System Architecture and Networking
 Agent def.: Computing dictionary
• In the client-server model, the part of the system
that performs information preparation and exchange
on behalf of a client or server.

• Especially in the phrase "intelligent agent" it implies

some kind of automatic process which can
communicate with other agents to perform some
collective task on behalf of one or more humans.

11-Nov-06 Johan J. Lukkien, [email protected] 38

TU/e Informatica, System Architecture and Networking
 Agent def.: Learnthat
• Intelligent Bots. Intelligent Agents are "bots" designed to work
for the user in terms of executing transactions (typically search
queries at this point) across the web.

• It is assumed that as Intelligent Agents become more

sophisticated, they can accomplish more complex tasks for
their users. Once they understand their users' requirements,
they can act somewhat independently of user interaction, in
terms of making online purchasing decisions and other tasks
that currently require significant search and user decision
making processes.

• Intelligent Agents will also become more useful as industry

standards are established.

11-Nov-06 Johan J. Lukkien, [email protected] 39

TU/e Informatica, System Architecture and Networking
 Jennings & Woodridge
• Agent: autonomous process capable of
– reaction to
– initiating
changes in their environment, possibly in
collaboration with user, other agents.

• I add: agents are further characterized by

– goal oriented-ness
– learning capability

11-Nov-06 Johan J. Lukkien, [email protected] 40

TU/e Informatica, System Architecture and Networking
 Some goals and classifications
• Collaborative agents (multi-agent systems)
• Mobile agents
• Interface agents (...wizzard...)
– assist user in using applications
• Information agents
– collect relevant information

11-Nov-06 Johan J. Lukkien, [email protected] 41

TU/e Informatica, System Architecture and Networking
 Agent properties
Common to
Property Description
all agents?

Autonomous Yes Can act on its own

Reactive Yes Responds timely to changes in its environment
Proactive Yes Initiates actions that affects its environment

Communicative Yes Can exchange information with users and other agents

Continuous No Has a relatively long lifespan

Mobile No Can migrate from one site to another
Adaptive No Capable of learning

11-Nov-06 Johan J. Lukkien, [email protected] 42

TU/e Informatica, System Architecture and Networking
 Agent Technology

The general model of an agent platform

11-Nov-06 Johan J. Lukkien, [email protected] 43

TU/e Informatica, System Architecture and Networking
 Agent Communication Languages

Message purpose Description Message Content

INFORM Inform that a given proposition is true Proposition
QUERY-IF Query whether a given proposition is true Proposition
QUERY-REF Query for a give object Expression
CFP Ask for a proposal Proposal specifics
PROPOSE Provide a proposal Proposal
ACCEPT-PROPOSAL Tell that a given proposal is accepted Proposal ID
REJECT-PROPOSAL Tell that a given proposal is rejected Proposal ID
REQUEST Request that an action be performed Action specification
Reference to
SUBSCRIBE Subscribe to an information source
source

Examples of different message types in the FIPA ACL @ 2002

11-Nov-06 Johan J. Lukkien, [email protected] 44

TU/e Informatica, System Architecture and Networking
 Example message

Field Value

Purpose INFORM

Sender max@http://fanclub-beatrix.royalty-spotters.nl:7239

Receiver elke@iiop://royalty-watcher.uk:5623

Language Prolog

Ontology genealogy

Content female(beatrix),parent(beatrix,juliana,bernhard)

A simple example of a FIPA ACL message sent between two

agents using Prolog to express genealogy information.

11-Nov-06 Johan J. Lukkien, [email protected] 45

TU/e Informatica, System Architecture and Networking
 Processes (in distributed context)
• Threads & processes
• Client & Server architecture
• Migration
• Agents
• Example: load balancing

11-Nov-06 Johan J. Lukkien, [email protected] 46

TU/e Informatica, System Architecture and Networking
 Special case: Load balancing
• Global (DOS) scheduling for optimal resource use
– system: maximize throughput
– user: minimize response time

• Dynamic
– react to dynamically changing load

• Pre-emptive
– stop process for migration

• No a priori knowledge
– burdens user with additional specification

• Efficient
– quick decision
• precision; real-time
– balance migration & information gathering

11-Nov-06 Johan J. Lukkien, [email protected] 47

TU/e Informatica, System Architecture and Networking
 Load balancing stability
• Definition: A load balancing algorithm is stable if it
is guaranteed to reach a state in which no more
processes are migrated, provided that the total
(global) load does not change.

11-Nov-06 Johan J. Lukkien, [email protected] 48

TU/e Informatica, System Architecture and Networking
 Instability...

n1 n2 n3 n1 n2 n3
load high low high low high low

migrate migrate migrate migrate

11-Nov-06 Johan J. Lukkien, [email protected] 49

TU/e Informatica, System Architecture and Networking
 Scalability and fault tolerance
• Scalability of load-balancing
– no significant effect of scaling the network
• limit information gathering, e.g., probe m out of n machines

• Fault tolerance
– node failure: continue for remaining nodes
– link failure: continue for connected nodes

11-Nov-06 Johan J. Lukkien, [email protected] 50

TU/e Informatica, System Architecture and Networking
 Taxonomy of Load Balancing Algorithms

static dynamic

deterministic stochastic centralized distributed

cooperative non-cooperative

11-Nov-06 Johan J. Lukkien, [email protected] 51

TU/e Informatica, System Architecture and Networking
 Dynamic distributed load-balancing
• Load estimation policy
• Process transfer policy
• Policy for exchange of load information
• Priority assignment policy
• Policy for limiting migration

• Policy: ‘rule’, ‘method’

11-Nov-06 Johan J. Lukkien, [email protected] 52

TU/e Informatica, System Architecture and Networking
 Load estimation
• Useful parameters are, for example:

– # processes on a node (at time of estimation)

– total resource demands of these processes
– instruction mixes of processes
– architecture and speed of resource
– remaining service time
• fixed: then method = #processes
• unknown but varying: time needed is equal to time used
• distribution: assumes a distribution of service times
– CPU utilization = #CPU cycles/unit of time
• observe regularly

11-Nov-06 Johan J. Lukkien, [email protected] 53

TU/e Informatica, System Architecture and Networking
 Process transfer policy
• Idea: transfer processes from heavily loaded nodes to lightly
loaded nodes

• Threshold: criterion to decide whether load is high or low

– dynamic: depends on current total load
• threshold: aw* rppi
• aw = average workload of ‘all’ nodes (computed regularly)
• rppi = relative processing power of node j
– static: fixed, related to capacity

• Source: try to off-load process if load is high

• Destination: accept (ask for) new process if load is low

11-Nov-06 Johan J. Lukkien, [email protected] 54

TU/e Informatica, System Architecture and Networking
 Single & double threshold
• Single threshold: may be unstable
– off-load waiting process to idle high
processor
low
• Double Threshold (high-low policy)
– use ‘high’ and ‘low’ mark

high
dispatch accept
high yes no normal
normal no no
low no yes low

11-Nov-06 Johan J. Lukkien, [email protected] 55

TU/e Informatica, System Architecture and Networking
 Process Transfer Initiative
• Sender initiated
– heavy loaded nodes seek to off-load work
– may work without pre-emptive migration

• Receiver initiated
– lightly loaded nodes look for load
– typically requires pre-emptive migration
• more expensive (need to record execution state)

11-Nov-06 Johan J. Lukkien, [email protected] 56

TU/e Informatica, System Architecture and Networking
 Location policy: where to off-load
• Random
– probe random destinations until threshold allows transfer

• Minimum load
– probe load at N random destinations
– transfer process to node with lowest load that is below threshold;
execute locally otherwise

• Bidding
– each node is seller (manager) and buyer (contractor)
– seller broadcasts a request for bids
– buyers send their bids containing info concerning node
capabilities; e.g. available processor/memory capacity
– manager approaches bidders (from best to worst)

11-Nov-06 Johan J. Lukkien, [email protected] 57

TU/e Informatica, System Architecture and Networking
 State information exchange policy
• Needed to balance load but consumes resources

• Polling
– active checking

• Periodic Broadcast
– heavy network traffic
– poor scalability (#messages increases)
– data may not be needed

• Improvements
– send data only after (significant) load-change
– send data after request (by a high-load node)
– reply only if can help in balancing

11-Nov-06 Johan J. Lukkien, [email protected] 58

TU/e Informatica, System Architecture and Networking
 Process priority policy
• Nodes have local and remote processes. Which has priority?

• Selfish (best local response)

– local > remote

• Altruïstic (best global response)

– remote > local

• Intermediate
– if #local processes > #remote processes
then local > remote
else remote > local

11-Nov-06 Johan J. Lukkien, [email protected] 59

TU/e Informatica, System Architecture and Networking
 Policy for Limiting Migration
• Uncontrolled
– may lead to instability

• Controlled
– set a maximum to number of migrations per process (process
keeps track of migration count)
• 1: irrevocable (typically static)
• k: may be decided (dynamically) based on process characteristics

11-Nov-06 Johan J. Lukkien, [email protected] 60

TU/e Informatica, System Architecture and Networking
 Open ends
• Process selection policy: which process to off-load?

• Compare time to complete on local node to time to complete at

remote node.

• Take transfer time (migration delay) into account

– distance between nodes
– size of process

11-Nov-06 Johan J. Lukkien, [email protected] 61

TU/e Informatica, System Architecture and Networking