OX DI~T~BWTED

COMMUKICATIO~S NET~OP~ Paul Baran

September

1962

ON DISTRIBUTED

COMMUNICATIONS

NETWORKS

Paul Baran* The RAND Corporation~ Santa Monica~ California

INTRODUCTION The previous paper** described how redundancy of coding can be used to build efficient digital data links out of transmission links of variable and often less than presently useful quality. An arbitrarily low over-all

error rate can be purchased with a modest redundancy of coding and clever terminal equipment. But even links with

low error rates can have less than perfect reliability. We should like to extend the remarks of the previous paper and address ourselves to the problem of building

*Any views expressed in this paper are those of the author. They should not be interpreted as reflecting the views of The RAND Corporation or the official opinion or policy of any of its governmental or private research sponsors. Papers are reproduced by The RAND Corporation as a courtesy to members of its staff. This paper was prepared for presentation at the First Congress of the Information Systems Sciences~ sponsored by The MITRE Corporation and the USAF Electronic Systems Division~ November, 1962. The writer is indebted to John Bower for his suggestions that switching in any store-and-forward system can be described by a model of a postmaster at a blackboard. Programming assistance provided by SharI a Boehm, John Derr, and Joseph Smith is gratefully acknowledged. **A prior paper was presented by Paul Rosen and Irwin Lebow of MIT Lincoln Laboratories, discussing redundancy of coding on a single link, "Low Error Efficient Digitial Communications Links," First Congress on the Information Systems Sciences~ McGraw-Hill, New York~ 1962.

-2-

digital communication networks using links with less than perfect reliability. We shall again trade in the currency

of redundancy, but instead of redundancy of coding we shall make use of redundancy of connectivity. This thing called redundancy is a powerful tool. But

the systems planner must choose that form of redundancy so that the form of the "noise" or interference appears to be somewhat statistically independent for each redundant element added. If this goal is completely met, there can

be an exponential payoff for a linear increase of added elements. As an example, we shall consider in some detail

the synthesis of a system where the form of the disturbance or "noisell 1s the simultaneous destruction of many geographically separated installations. The system in particular

is to be a very high-speed digital data transmission network composed of unreliable links, but which exhibits any arbitrarily desired level of system reliability or survivability. DEFINITION OF SURVIVABILITY

This communications network shall be composed of several hundred stations which must intercommunicate with one another. Survivability as herein defined is the percentage

of stations surviving a physical attack and remaining in electrical connection with the largest single group of surviving stations. This criterion is a measure of the ability

of the surviving stations to operate together as a coherent entity after attack.

-3TYPES OF NETWORKS Although one can draw a wide variety of networks, they all factor into two components: centralized (or (Types (a) and

star) and distributed (or grid or mesh). (c) in Fig. 1)

The centralized network is basically vulnerable. Destruction of the central node destroys intercommunication between the end stations. In practice, a mixture of

star and mesh components is used to form communications networks. For example, type (b) in Fig. 1 shows a hier-

archial structure to a set of stars connected in the form of a larger star with an additional link forming a loop. Such a network is sometimes called a lIdecentralizedllnetwork, because complete reliance upon a single point is not always required. But, as destruction of a small number

of nodes in a decentralized network can destroy communications, we shall turn to consider the properties, problems, and hopes of building communications networks that are as "distributed" as possible. The unstandardized terms

centralized, decentralized, and distributed are often and conveniently used as relative adjectives when referring to real-world networks. DEFINITION OF REDUNDANCY LEVEL Figure 2 defines the term IIredundancy level," which is used in this paper as a measure of connectivity. A

minimum span network, one formed with the smallest number

-4-

CJ W f-

::>

If)

CD 0::f-

-u
(r,
..>C

~ ~ ..._
<l.J "'0

Z
<l.J ......
_Q
1-

If)

(3
"'0

W N
_.J

0
"'0 <l.J N

c:

Cl::CO f-Z W U W

er

0
+-Q.J

.....

U QJ

0
"'0
QJ N

..... ......
C
QJ

_oJ

19 u,
0

If)

W N
_j

<l<:l f-

0::-

W U

-5of links possible~ is chosen as a reference point, and is called "a network of redundancy level one.1I If two times

as many links are used in a gridded network than in a minimum span network, the network is said to have a reI

tlundancy level of two.

Figure 2 defines connectivity of Redundancy level is equivalent

l~ l~, 2, 3, 4, 6~ and 8. stations.

to link-to-node ratio in an infinite size arrays of

ASSUMPTION OF PERFECT SWITCHING Each node and link in the array of Fig. 2 has the capacity and the switching flexibility to allow transmission between any ith station and any jth station,
.

provided a path can be drawn from the ith to the jth station. Starting with a network composed of an array of stations connected as in Fig. 3, an assigned percentage of nodes and links are destroyed. If, after this operation,

it is still possible to draw a line to connect the ith station to the jth station, the ith and jth stations are said to be connected. RATIONALE FOR DESTRUCTION PATTERNS Figure 4 indicates network performance as a function of the probability of destruction for each separate node. If the expected IInoise was destruction
ll

caused by con-

ventional hardware failure, the failures would be randomly distributed through the network. But, if the disturbance

-6-

o
o
.0

o o o
R ;: 1.5

o
R :: I

o
R::2

R=3

R=4

o o o
o
o
R':: 3

o o
o o

o
R':: 4

R': 6

R':: 8

FiG. 2 - Definition

of Redundancy Level

-7-

FIG. 3 -

An Array

of Stet ions

-8-

- - - - - - - - - - - - - - - - - r ~-. ---~-.VI C

w o ~
.<::_

"0
!;:!

s
C

...., 0
>. 0.

'" c

O.

II}

":: 0 wu

~E oE
a.
~

z
0
f<I U

.n

""''0 0 '0 c

'"
o
v

0
U

~ ~

::::>

'" o

0'0

o~

E~
0'

Q
c '

I")

EO"

....

...., o

'" ~

'" ::: v.. in

0' Cl: c

" o E n."':

30

....J'O-

'" '" 0 ::) ~

t_

______ L
R=I

>f-(})

-z
:::!O

>(}) a:::
(j)

<If> <I -fu,

co-

---~------r
• I

i
-

::::>

I
I

0 Z 0

:-

f=

u
<I u, f(})

.s:::: 0' 0.'"

DC ~> '" ;: ,..~

"'::::
E
o-,

CT'

a:::

'" ~ - ....
0'"
::J-

0 '" u >- -

C>

cr

"!2'"
We Cl:o
"''0

<l

-'

01

0.2

0.3
SINGLE

04
NODE

0.5
PROBABI LlTY

0.6
OF

0.7
DESTRUCT

08
ION

FIG. 4 - Perfect

Switching in a Distri buted Network Destruction, 100% of Links Operative.

Sensitivity

to Node

-9were caused by enemy attack there are two possible casesll to be considered. requires of kill, direction Pk IIworst

To bisect a 32-link width network of 288 weapons

i

each with a probability

0.5 ,

or 160 with a

Pk = 0.7 , to produce over an 0.9 probabisecting the network. If hidden single 50 If or or

bility of successfully altprnative

command is allowed, then the largest

group would still have an expected percent of the initial stations this raid misjudges complete knowledge

value of almost intact.

surviving

complete availability

of weapons,

of all links in the cross section,

the effects of the weapons against each and every link, the raid fails. highly parallel The high risk o'f such raids against structures causes examination of alteruniform

native attack policies. raid example. Assume

Consider

the following

that 2,000 weapons network.

are deployed are so

against a 1000-station spaced that destruction weapon 1s unlikely. equal 1,000 weapon struction
i

The stations

of two stations

with a single into two of de-

Divide salvos.

the 2,000 weapons

Assume any probability

of a single node from a single weapon 0.5. Each weapon

less than

1.0; for example, a 0.5 probability

on the first salvo has But, each

of destroying

its target.

;weapon of the second salvo has only a 0.25 probability, , since one-half the targets have already attack been destroyed. a worst-case

Thus, the uniform configuration.

is felt to represent

-10-

MONTE CARLO SIMULATION

Such worst-case l2xle-array attacks have been directed against an

ne tw or-kmodel of 324 nodes w i.t h varying probabillevel~ with results shown in Fig.

ity of kill and redundancy

4.

The probability

of kill was varied frem zero to unity while the ordinate marks survi vabili t: used is the percenta~e and remaining in of
c •

along the abscissa The criterion

of survivability

stations not physically communications stations.

destroyed

with the lare:;estsingle group of survivine:: survivability

The curves of Fig. 4 demonstrate

as f'unc t Lon of a ttack level for networks degrees of r-e undancy . d

of var-y ; Ln

The line labeled "be s t possible

line It marks the upper bound of Lo ss due to the phy sical failure component underwent alone.

'.

For examp Le , if' a ne two rk destructlon of each

an attack of 0.5 probability

of its nodes, expected

then only 50 per cent of its nodes would be of how perfect its interested in the of in netof

to survive--regardless
'v-Je

communications.

are primarily

addi tional system degradation comrr.un i cat.Lons.

caused

by . failure

Two key points are to be noticed First, extremely survivable

the curves of Fig. 4.

.works can be built using a moderately connectivity level. Redundancy

Low redundancy

levels on the order of extremely heavy level

only three permit withstanding attacks with negligible Secondly,

additional

Los s to communications.

the survivability

curves have sharp break-p8ints.

-11-

A network

of this type will withstand

an increasing

attack

level until a certain point is reached, network

!

beyond which the degree

rapidly deteriorates.

Thus, the optimum

of redundancy

can be chosen as a function Further

of the exbuys little.

'pected level of attack. The redundancy attacks

redundancy

level required

to survive even very heavy

is not great--on

the order of only three or four span network.

times that of the min~um

SIMULATION RESULTS--LINK
In the previous formance

FAILURE ONLY
network per-

example we have examined of the destruction than links).

as a function

of the nodes

(which are better targets examine the same network, particular, be without work.

We shall now relinks. In

but using unreliable

we want to know how unreliable further degrading the performance

the links may of the net-

Figure 5 shows the results for the case of perfect nodes; only the links fail. radation There is little system degunreliable links-all nodes

caused even using extremely

on the order of 50 percent are working.

down-time--assuming

COMBINATION LINK AND NODE FAILURES

The worst case is the composite of both the links and the nodes. of link failure upon a network nodes destroyed. It appears effect of failures

Figure

6 shows the effect

of its

having 40 percent

that what would today be re-

garded as an unreliable

link can be used in a distributed

-12-

W Zl?

-4

U1w

04 ---'
\_
I_j

cr: Z>

~Z

1.J...4 ill 0 I-

tn 9 IU1 0.5

::J <l U1 4I 0::1-lJ..,-

>z > 0>0:: - Z

t;

wu
:!:<l

<II--

I-Z

FIG. 5 -

Perfect Switching in a Distributed Network - Sensitivity Link Destruction, 100% of Nodes Operative.

to

-13-

•
0
Z

0.8 0.6

40%
destroyed

of

Nodes at outset

~ <:( u

z
~ ~

::> 0
: Z
)-

_j~

coO
<t
-

>~ <:( >1O:::(/')

_(/') 0
Z
:)lL.

0.4

~ u
0:::
u, IU)

<:(

R=4

<[ .....J

w <..? cr

0.2
R=1.5

OO~--~~::::=OI.2~::::~~~0~.4;;~~~;;;;;0~.6~~~~~~0~.
SINGLE LINK PROBABILITY OF DESTRUCTION

FIG. 6

Perfect Switching· in a Distributed Network Destruction After 40% Nodes Are Destroyed.

Sensitivity

to Link

-14network links. almost as effectively Figure 7 examines as perfectly reliable

the result of 100 trial cases density distribution of node and link of cases for 20 per-

in order to estimate of system performance failures.

the probability for a mixture

This is the distribution

cent nodal damage and 35 percent

DIVERSITY OF ASSIGNMENT

link damage.

There redundancy

is another and more common technique than in the method is assumed described

for using

above in which switching ability.

each station

to have perfect

This alternative ment." quired. selected requires

approach

is called "diversity switching

of assignis not re-

In diversity Instead, between reliable

of assignment,

a number

of independent

paths are which

each pair of stations communications.

in a network

But, there are marked switching

differences

in performance of assignment

between distributed as revealed

and redundancy

by the following

Monte Carlo simulation. In the matrix is connected of N separate stations, each ith station but

to every jth station by three shortest independent paths (i=1,2,3, ... ,N;

totally separate

j=l,2,3, ... ,N; ilj).

A raid is laid against separate

the network.

Each of the pre-assigned

paths from the ith station If one or more of the is said to exist of

to the jth station is examined. pre-assigned paths survive,

communication

between the ith and the jth station.

The criterion

-15-

/
/ I

c
°

-r

u._
0

c> ::I'_

CO

e
:J

c c·_ ._ ...J
10 I"-

u~
CO 0

o(f)

tf>:::-!?

zO::

..;:.(0 +- -

olD

o·zl- 0 2z
I

r- Lf):r:

-w

> ._ ..... >

0
+-

(1)0

a ,_
Q.J

"""'-L

-,

r-w (j")z l.L.0

OJ:

~«
zt::

::I c(/)

uu
00

"-

~z
+-:::-!? (,1)0
0

o~
I-

I..D U Z

o 0::_ <fo
l.L.1(/l-

~@ .....
Q)

...JJ
"_(,I)

r-U

°0 0

\ \
\
U"")

wZ l')~
0:::2:

eU

<f:!
...JO

"-; 8 .000::
+._

';;: .... +- to .t!l II

>.
0>

(,I)

au +.--.::
'.0
+-

C:._ Q.J .£:

:>o.~ (/)

o:j

.0.2: o ,_ .... Q) n..n..

F"<.!)

ou

L&.. U1

U"")

S3S'i1J

~o

o
NOI1J';1~j

-16survivability used is the mean number of stations averaged con-

nected to each station, Figure switching

over all stations. perfect

8 shows~ unlike the distributed

case, that there is a marked with even slightly

loss in communiunreliable nodes or

cations capability links.

The difference

can be visualized permits

by remembering the

that fully flexible privilege emphasizes

switching

th~ communicator Figure

of ex post facto decision a key difference

of paths.

between some present distributed

day net-

works and the fully flexible discussing.

network we are

COMPARISON WITH PRESENT SYSTEMS

Present conventional switching systems try only a

small subset of the potential a gridded network. paths tested~ perfect

paths that can be drawn on the percentage of potential of

The greater

the closer one approaches Thus, perfect

the performance provides an

switching.

switching

upper bound of expected network; Between the diversity

system performance of assignment

for a gridded

case, a lower bound. composed of a mixture

these two limits lie systems routes and diversity of assignment

of switched

of assignment.

Diversity .eliminating ability

is useful for short paths, but requires element survivin long haul

the need for switching,

and reliability

for each tandem

circuits passing

through many nodes.

As every component paths

in at least one out of a small number of possible

-17-

___ '_~7'

Perfect Switching

0.1

0.2

0.3

0.4

0.5

0.6

0.7
OF KILL

O.S

0.9

1.0

SINGLE NODE PROBABILITY

FIG. 8 - Diversity of Assignment Network.

vs. Perfect

Switching

In

a Distributed

-18must be simultaneously operative, high reliability margins

and full standby equipment

ON FUTURE SYSTEMS

are usual.

We will soon be living in an era in which we cannot guarantee survivability of any single point. in which However,

we can still design requires the enemy

systems

system destruction n of

to pay the price of destroying large,

n stations.

If n is made sufficiently survivable

it can be can be

shown that highly built--even

system structures era.

in the thermonuclear and systems

In order to build

such networks number

we will have to use a large in kriow.l ng how the

of elements.

We are interested

Lnexpe ns Lve these elerr.entsmay be and still permit syste[!ito operate reliably. between element There is a strong reliability.

relationship To design of

cost and element

a sy::::tem thE.t must antiCipate both enemy attack, combine together problems, and normal failures

a worst-case

destructi:m one can

system failures, expected

the expected

by enemy attack reliability

with the failures provided

caused by norm2l

the enemy does not know which elements Our future systems design problem is

are inoperative. that of building cribed

very reliable

systems

out of the descost. In digital low

set of unreliable

elements

at lowest

'choosing the corrununications linkS links appear increasingly

of the future,

attractive

by permitting For example,

cost switching

and low cost linkS.

if "perfect

-19switching" tandem without Further, flexible FUTURE is used, digital of many errors links are mandatory connected to permit links magnitude.

connection cu~ulative

separately reaching

an irreducible

the signalling

measures

to implement require

highly

swl t ch Lng doctrines

always

digits.

LO\<[ COST ALL-DIGITP.L

COM1v'JUNICATIONS LINKS system optimized for digits tech-

When one designs and high redundancy, niques

an entire

certain new communications-link than those common

appear more attractive

today.

A key attribute formation

of the new media cheaply,

is that it permits transmission

of new routes

yet allows

on the order of a million enough to be economic, with existing

or so bits per second, high to be inexpensively techniques at

but yet low enough digital computer

processed

the relay station nodes. are secondary.

Reliability

and raw error rates

The network

must be built with the expectaPowerful error removal

tion of heavy damage, methods exist.

anyway ,

Some of the communication look attractive

construction

methods

that

in the near future "poor-boy"

include pulse regenera.TV broadcast satellites.

tive repeater station

line,

microwave,

digital

transmission,and

non-synchronous

Pu..se Re cene r-a tive Repea ter Line I--------~~----------~~~~~~~ Samuel B. Morse's regenerative amplifying weak telegraphic

repeater

invention been

for

signals has recently

-20-

resurrected

and transistorized.

Morse's electrical

relay

,permits amplification

of weak binary telegraphic Experiments by various

signals organizahave

above a fixed threshold. tions

(primarily the Bell Telephone

Laboratories)

shown that digi tal da ta rate s on the orde r of 1. 5 mi IIi on bits per second can be transmitted line at repeater over ordinary telephone

spacings on the order of 6,000 feet for copper pairs. At present,

#22 gage pulp paper insulated

up to 20 tandemly connected amplifiers without retiming synchronization

have been used There appears

problems.

to be no fundamental with correspondinb resynchronization distances be desired

reason why either lines of lower loss spacing, or more powerful link w8uld

further repeater methods

cannot be used to extend

to in excess of 100 miles. for a possible national

Such distances

distributed transist8r

netw8rk. amplifier

Power to energize is transmitted

11

the miniature

over the copper circuit itself.

Poor-Bov"

Mi crowave equipment has been declinIn an analo~ and

While the price of microwave ing, there are still untapped

major savings.

signal network we require a high degree very low distortion

of reliability However,

for each tandem repeater. together with perfect considerations

using digital modulation we minimize planning.

switching from our

these two expensive We would envision

the use of almost mass-produced

-21-

microwave mounted Relay

crystal

receiverjklystron

oscillator commercial

units power.

on t!telegraph polest! carrying station spacing would probably

be on the order of by only a of fading

10+ miles. minimal margins.

Further

economies

can be obtained and reduction

use of standby The ability

equipment

to use alternate normally

paths permits troubled by rain

consideration attenuation problem. While preliminary cheapest

of frequencies problems reducing

the spectrum

availability

this technique indications

has not been fully examined, that this may be the of the type to

suggest

way of building

large networks

be described. T.

v.

Stations siting of receiving might antennas, broadcast

With-proper television

stations

be used to form additional

high data rate links in emergencies. Non-Synchronous The problem synchronous building Satellites of building a reliable network using non-

satellites

is somewhat network

similar

to that of links. When

a communications

with unreliable

When a satellite ; a satellite Thus, system

is overhead,

the link is operative.

is not overhead,

the link is out of service. compatible with the type of

such links are highly to be described.

-22-

VARIABLE DATA RATE LINKS

In a conventional circuit switched system each of the transmission bandwidths. But,

tandem links require matched in the previous

paper,* it was seen that in order to make link the post-error-removal data

fullest use of a digital

rate would have to vary as it is a function The problem then is to build a communication

of noise level. network made

up of links of variable

data rate to use the communication

resource most efficiently.

VARIABLE DATA RATE USERS

Not only will the links of a digital data transmission operate at a variable digital transmission data rate, so will the users. applications Many

are highly intermittent user varying if one over his

in nature, with each potential demand from instant to instant. transmitted a high-data

network

For example,

one line of a 60 w.p.m. teletype message "express routell operating at 1,500,000

bits per

second, a 1/3 millisecond seconds.

burst would be sent every 12 links serve

Where high data rate transmission on a time division

many subscribers and the network highly variable

basis, both the user at a

links will appear to be operating data rate.

*See footnote,

p.

1.

-23-

COMMON USER

In cornmunications~ as in transportation, it is most economic for many users to share a common resource rather than each to build his own system--particularly when supplying intermittent or occasional service. This inter-

mittency of service 1s highly characteristic of digital communication requirements. Therefore, we would like to

consider the interconnection, one day, of many all digital links to provide a resource optimized for the handling of data for many potential intermittent users--a new commonUSer system. Figure 9 demonstrates the basic notion. A wide

mixture of different digital transmission links 1s combined to form a common resource divided among many potential users. But, each of these communications links How can links

could possibly have a different data rate. of different data rates be interconnected?
USE GF STANDARD MESSAGE BLOCK

Present common carrier communications networks, used for digital transmission, use links and concepts originally designed for another purpose--voice. These systems

are built around a frequency division multiplexing linkto.link interface standard. is that of data rate. The standard between links

Time division multiplexing appears

so natural to data transmission that we might wish to

-24-

....... o
<V ::J

><

...

.._

o
r.n

"'0 Cl.J

02 -0 lIl_
We::!
_j"_

o E
o

0..

:; .,_ m«1 ' 9 C{

Wlll I-

o .....

o
<r
I Q)

en

<..9
-::J

o
e::!

l1..

Ww
1->

0t_?

~~-.~
0..2
LU

IW
~

W_j..-J

I-

-25-

consider an alternative approach--a standardized message block as a network interface standard. While a standard-

ized message block is common in many computer-communications applications, no serious attempt has ever been made to use it as a universal standard. A universally standardized

message block would be composed of perhaps 1024 bits. Most of the message block would be reserved for whatever type data is to be transmitted, while the remainder would contain housekeeping information such as error detection and routing data, as in Fig. 10. As we move to the future, there appears to be an increasing need for a standardized message block for our all-digital communications networks. As data rates in-

crease, the velocity of propagation over long links becomes an increasingly important consideration.

* We

soon reach a pOint where more time is spent setting the switches in a conventional circuit switched system for short holding-time messages transmission of the data. Most importantly, standardized data blocks permit !many simultaneous users each with widely different bandwidth requirements to economically share a broadband network made up of varied data rate links. than is required for actual

*3000 miles at ~ 150,000 miles/sec. ~ 50 milliseconds transmission time, T. 1024-bit message at 1,500,000 bits/sec. ~ 2/3 millisecond message time, M.
~T>~

-26-

-----0 Q)
(l)

.::.: u

(/')

r-

a::
u;

CD V N

u,

rW

ex:::
u, u,
::)

r~

C\I

Ol
til
(l)

en

<

::) Q)

r-

:::?:

CD

r-

:::!

I
0
t.:>
u;

<

>

1
}
}

~lf)

WW

(/')I.fl

4'

If)t:.
wCD ~O
If)

-27-

The standardized message block simplifies construction of very high speed switches. Every user connected to the net-

work can feed data at any rate up to a maximum value. :The user's traffic is stored until a full data block is received by the first station. This block is rubber

stamped with a heading and return address, plus additional housekeeping information. the network. SWITCHING In order to build a network with the survivability properties shown in Fig. 4, we must use a switching scheme able to find any possible path that might exist after heavy damage. The routing doctrine should find the or "ringThen, it is transmitted into

shortest possible path and avoid self-oscillatory around-the-rosey" switching.

We shall explore the possibilities

of building a

"real-time" data transmission system using store and forward techniques. The high data rates of the future carry and cir-

us into a hybrid zone between store-and-forward cuit switching. store-and-forward node singularly.

The system to be described is clearly if one examines the operations at each But, the network user who has called up

a "Virtual connection" to an end station and ha-stransmitted messages across the United States in a fraction of °a second might also view the system as a black box providing

-28-

an apparent circuit connection across the U.S.

There are

,two requirements that must be met to build such a quasireal time system. First, the in-transit storage at each

node should be minimized to prevent undesirable time delays. Secondly, the shortest instantaneously available

path through the network should be found with the expectation that the status of the network will be rapidly changing. Microwave would be subject to fading inter-

ruptions and there would be rapid moment-to-moment variations in input loading. These problems place difHowever, the

ficult requirements upon the switching.

development of digital computer technology has advanced so rapidly that it now appears possible to satisfy these requirements by a moderate amount of digital equipment. What is envisioned i6 a network of unmanned digital switches implementing a self-learning policy at each node so that overall traffic is effectively routed in a changing environment--without control point. need for a central and possibly vulnerable One particularly simple routing scheme
ll

examined is called the "hot-potato

heuristic routing

doctrine and will be described in detail. Torn-tape telegraph repeater stations and our mail system provide examples of conventional store-and-forward switching systems.

In these systems, messages are reand stacked until the I1best" The key feature of store-and-

layed from station-to-station outgoing link is free.

-29forward transmission is that it allows a high line occupancy 'factor by storing so many messages at each node that there is a backlog of traffic awaiting transmission. But, the

price for link efficiency is the price paid in storage capacity and time delay. However, it was found that most switching could be

of the advantages of store-and~forward

obtained with extremely little storage at the nodes. Thus, in the system to be described, each node will attempt to get rid of its messages by choosing alternate routes if its preferred route is busy or destroyed. Each

message is regarded as a "hot potato," and the nodes are not wearing gloves. Rather than hold the "hot potato,1I

the node tosses the message to its neighbor, who will now try to get rid of the message. THE POSTMAN The switching process in any store-and-forward is analogous to a postman sorting mail. at each switching node. from all links. system

A postman sits

Messages arrive simultaneously

The postman records bulletins describing

the traffic loading status for each of the outgoing links. With proper status information, the postman is able to determine the best direction to send out any letters. So

far, this mechanism is general and applicable to all storeand-forward communication systems.

-30-

HOT-POTATO HEURISTIC

ROUTING DOCTRINE

To achieve real-time operation it is desirable to respond to change in network status as quickly as possible so we shall seek to derive the network status information directly into each message block. Each standardized message block contains a "toU address, a "from" address, a handover number tag, and error detecting bits together with other housekeeping data. The message block is analogous to a letter. The
11

from"

address is equivalent to the return address of the letter. The handover number is a tag in each message block set to zero upon initial transmission of the message block into the network. Every time the message block is passed The handover

on, the handover number is incremented.

number tag on each message block indicates the length of time in the network or path length. This tag is somewhat

analogous to the cancellation date of a conventional letter. INDUCTIVE DETERMINATION OF BEST PATH links, the post-

Assuming symmetrical bi-directional

man can infer the "best" paths to transmit mail to any station merely by looking at the cancellation time or the equivalent handover number tag. If the postman sitting

in the center of the United States received letters from San Francisco, he would find that letters from San Francisco arriving from channels to the west would come in with later cancellation dates than if such letters had

-31-

arrived in a roundabout manner from the east.

Each letter

carries an implicit indication of its length of transmisSion path. The astute postman can then deduce that the

best channel to send a message to San Francisco is 'probably the link associated with the latest cancellation dates of messages ~ San Francisco. By observing the

cancellation dates for all letters in transit~ information 1s derived to route future traffic. The return address

and cancellation date of recent letters is sufficient to determine the best direction to which to send subsequent letters.
THE HANDOVER NUMBER TABLE

While cancellation dates could conceivably be used on digital messages~ it is more convenient to think in terms of a simpler digital analogy--a tag affixed to each message and incremented every time the message is relayed. Figure 11 shows the handover table located in the memory of a single node. A row is reserved for each major station A column is As

of the network allowed to generate traffic.

assigned to each separate link connected to a node.

it was shown that redundancy levels on the order of four can create extremely "toughll networks and additional redundancy brought little, only about eight columns are really needed.

l ..

-32-

I -W U ,1-

s:

u r(/)

"""u
0::0 WI ClJU v

"Ow

II"'

en

------r<l
"U

"0 "-

20 ::.c:: (/)

~ =>z

~~-r
N W
I

I
I

". ,
,

_-

",
00

'':
I

.0·

I\_
i\
I~

",

00

10

V
r<l

!
I
I

to

OJ

I"-

I"i

-- f--~~
10

I
I

i
I

I
I
(

I"i

N 1---

If)

ZU

::i~
I"-

I"-

10

I"-

,.,.,

:-10

._
OJ .D :J

E .....

Z
OJ

OJ

r<l

w
r0::

(/)

CO
en

a:: w
z
::.c:: 2
...J

:2: :::>

en

a::

ill

en

V N

z
w
0 Z

~ :::>

0
N

Hffi-~~r-=-+ - 1-; ~~t--~--_N I- I r<l INN

--1-

I0 10

> o u

:c

c o

CO

I
I

en

i-V IN-

et:

> r

ro

-----l~
eniN!N
:

I,- r<l
t

~j-~~
)I
\
I

1-i-+--.-+-~-.---~~"
!

I'-

c:r

~
N N

I"i

CO

r<l N

I I

-I
I

0
N

10

I'-

I'-

I 10 N

c:r

CO

-33PERFECT LEARNING If the network used perfectly reliable, error free links, we might fill out our table in the following manner. Initially, set entries on the table to high values. amine the handover number of each message arriving on each line for each station. If the observed handover Ex-

number is less than the value already entered on the handover number table, change the value to that of the observed handover number. If the handover number of the message After

is greater than the value on the table, do nothing.

a short time this procedure will shake down the table to indicate the path length to each of the stations over each of the links connected to neighboring stations. table can now be used to route new traffic. This

For example,

if one wished to send traffic to station C, he would examine the entries for the row listed for station C based on traffic from C. Select the link corresponding to the This is the

column with the lowest handover number. shortest path to C.

If this preferred link is busy, do

not wait, choose the next best link that is free.

-34-

DIGITAL

SIMULATION

OF PERFECT LEARNING was tested by a Monte All tables

This basic routing procedure Carlo simulation

of a 7x7 array of stations.*

were started completely starting

blank to simulate a worst-case

condition where no station knew the location of Within ~ second of simulated real of all

any other station.

world time, the network had learned the locations connected manner. favorably

stations and was routing traffic in an efficient The mean measured to the absolute path length compared very shortest possible path length Preliminary

under various traffic loading conditions. results indicate that network loadings

on the order of

50 per cent of link capacity could be inserted without

undue increase in the network, restrained of path length. When local busy spots occur traffic is intermittently

locally generated

from entering

the busy points while the potential appears is

traffic jams clear.

Thus, to the user, the network If the network

to be a variable data rate system. carrying light traffic,

any new input line into the network

would accept full traffic up to 1.5 million bits per second. But, if every station had heavy traffic and the loaded, the total allowable input

network became heavily

data rate from any single station in the network might

Baran and Sharla Boehm, Simulation of a Hot Potato Routing Doctrine (U), The RANIT Corporation, RM-3103, (In preparation).

* Paul

-35-

drop to perhaps minimum station

0.5 million bits per second.

data capacity of the location

The absolute from any in of

guaranteed

into the network

is a function redundancy

of the station

the network,

level, and the mean path length The "choking in the network noted.
II

transmi tted traffic input procedure

in the network.

of and no

has been simulated under overload

signs of instability

It was found transmission little memory of all traffic

that most of the advantage can be provided capacity.

of store-and-forward

in a system having relatively IIguaranteesll delivery

The network

that it has accepted FORGETTING

from a user. LEARNING network behavior when all in deter-

AND IMPERFECT

We have briefly links are working. mining network destroyed,

considered

But, we are also interested with real world

behavior

links--some The network of destruction,

while others

are being repaired. to the effects

can be made rapidly repair,

responsive

and transmission

fades by a slight modification the values on the handover the lowest handor

of the rules for computing number table.

In the previous

example,

over number ever encountered IIfromll station, in the handover

failed.

for a given origination,

and over each link., was the value recorded number table. But, if some links had
to the change.

our table would not have responded

Thus., we must be more responsive

to recent

measurements

-36-

than old ones. .. the following by

This effect can be included policy. number;

in our calculation measured value found add

Take the most recently subtract the previous

value of handover in the handover a fractional

table; if the difference

is positive,

part of this difference table value.

to the table value merely

to form the updated implements

This procedure

a "forgetting"

procedure--placing

more belief

upon more recent measurements

and less on old measurements. damage, automatically

This device would, in the case of network modify the handover and asymptotically If the difference value is negative, only a fractional ference. This implements Learning a form of sceptical

number table entry so as to exponentially approach the true shortest path value. value minus the table

between measured

the new table value would change by portion of the recently measured dif-

learning. errors.

will take place even with occasional

Thus, by the simple device of using only two separate lIlearning constants,tI depending is greater a mechanism whether the measured value

or less than the table value, we can provide that permits the network routing to be reThis for

sponsive to varying

loads, breaks, technique

and repairs.

learning and forgetting

has been simulated

a few limited cases and was found to work well.

-37ADAPrATION TO ENVIRONMENT

This simple slrnultaneous learning and forgetting mechanism implemented independently at each node causes the entire network to suggest the appearance of an adaptative system responding to gross changes of environment in several respects, without human intervention. For example, consider self-adaptation to station location. A station, Able, normally transmitted from one location
in the network, as shown in Fig. 12 (a).

If Able moved

to the location shown in Fig. 12 (b), all he need do to announce his new location is to transmit a few seconds of dummy traffic. The network will quickly relearn the new

location and direct traffic toward Able at his new location. The links could also be cut and altered, yet the network would relearn. Each node sees its environment through

myopic eyes by only having links and link status information to a few neighbors. There is no central control;

only a Simple local routing policy is performed at each node, yet the overall system adapts.
LOWEST COST PATH

We seek to provide the lowest cost path for data to be transmitted between users. When we consider complex

networks, perhaps spanning continents, we encounter the problem o£ building networks with links o~ widely different data rates. How can paths be taken to encourage

-38-

:
.... N

....

w ~

co

-_j

0 0

c:

--

:::>

Q..> V>

....

0
Q.J

c; 0 .c; U
+-

0'>

>. .._

:..c 0
+-

CL

<l
I :

-0

....w :!

<l

l::>
u,

r-

,-39-

most use of the least expensive links?

The fundamentally Instead

,Simple adaptation technique can again be used.

of incrementing the handover by a fixed amount, each time ia message is relayed, set the increment to correspond to link costjDit of the transmission link. ThUS, instead of

the "instantaneously shortest non-busy path" criterion, the path taken will be that offering the cheapest transportation cost from user to user that is available. The

technique can be further extended by placing priority and cost bounds in the message block itself, permitting certain users more of the communication resource during periods of heavy network use. WHERE WE STAND TODAY Although it is premature at this time to know all the problems involved in such a network and understand all costs, there are reasons to suspect that we may not wish to build future digital communication networks exactly the same way the nation has built its analog telephone plant. There is an increasingly repeated statement made that :one day we will require more capacity for data transmission than needed for voice. If this statement is correct, then

it would appear prudent to broaden our planning consideration to include new concepts for future data network directions. Otherwise, we may stumble into being boxed in with the

-40uncomfortable switches restraints of communications links and analog using reusable.

originally

designed

for high quality techniques

transmission. dundancy

New digital

computer

make cheap unreliable network

links potentially compatible

Some sort of a switched links appears appropriate for digital service.

with these demand

to meet this new upcoming

Of course, we could use our existing techniques. But, a system with greater

circuit

switching than the for

capacity

long lines of telephone such data transmission

plants might best be designed and survivability permit

at the outset. switching of very

Such a system should economically

short blocks of data from a large number taneously with intermittent Considering large volumes

of users simulamong a smaller there

set of points.

the size of the market small amount primarily

appears to be an incommensurately about a national

of thinking for data.

data plant designed

Is it time now to start thinking possibly non-existant public utility,

about a new and a common user digspecifically for

ital data communication the transmission , scribers?

plant designed

of digital

data among a large set of sub-

Is it time to consider standard message

the Euture?

the detailed

format of a of

block as a possible

new data standard