Network Outline
Network Outline
Synchronization is the precise coordination of multiple events, events like causing a set of
files to remain identical in different locations. Time is measured by clocks; therefore, Clock
synchronization is that coordination of independent clocks.
Clocks differ in time even if they are set accurately due to clock drift caused by slightly
different clocks counting time rates, clock drift will eventually affect accuracy and therefore
synchronization must be repeated frequently.
Time in networking devices is the only element of reference between all other devices
on the network, this is because every aspect of managing, securing, planning and
debugging a network involves determining when every event happens.
Time in networking devices allows tracking of security breaches, network usage, or
problems affecting a large number of components, this can be nearly impossible if
timestamps in logs are inaccurate. Time is often the critical factor that allows an event
on one network node to be mapped to a corresponding event on another.
To reduce confusion in shared filesystems, it is important for the modification times
to be consistent, regardless of what machine the filesystems are on.
Billing services and similar applications must know the time accurately.
Some financial services require highly accurate timekeeping by law.
Time synchronization methods and standards have been employed but they have mostly been
limited by accuracy or distance. The shorter the distance the more accurate the time become
in networking devices, that is, time synchronization between networking devices over very
short distances can easily achieve a high level of accuracy, but maintaining the required
accuracy over longer distances maybe difficult.
Some of the following example shows the problems with unsynchronized clocks:
(i) In a distributed banking system, if the timing and ordering of financial transactions
are not tracked, it may raise inconsistent state in the system.
(ii) A distributed online reservation system in which the last available seat may get
booked from multiple nodes if their local clocks are not synchronized.
(iii) There is a need to transmit a message from one node to another at any time.
This will become difficult if sender and receiver clocks are not synchronized with
each other.
Type of synchronization
Time synchronization or Time of day (TOD): Getting clocks or a timer to run at the same
time of day, hour, minute, second, microsecond etc. Time is measured by clocks; a clock is a
device with a stable source frequency and a counter.
Frequency synchronization: Getting clocks to run in the same rate. Frequency is the number
of events in a second (Hertz)
Phase synchronization: Phase synchronization is when two separate repeating events happen
at the same point in time.
Therefore, Accurate time synchronization in any application involves the distribution of the
time of day, frequency, and phase between devices.
Networking devices contain components that count time based on crystal oscillators that
output an electrical signal with a precise frequency. When synchronized time between
devices is important, clocks can therefore track the time with good accuracy. However, when
precise time synchronization is required, it is found that even identical devices still lose
synchronization over time. This reality is due to slight physical differences between crystal
oscillators and temperature variations that affect the exact output frequency and therefore the
clock time. To maintain accurate time between network devices requires continuous
synchronization from a reference time source that has a more accurate and reliable clock.
Existing Solutions
The following table lists some of the more common timing signals and standards that are in
use, as well as current network time protocols.
1PPS 1 Pulse Per Second- an electrical pulse signal to the start of each
second
10MHz a precise refence frequency signal used for synchronization
IEEE 1588 Precision Time Protocol (PTP)- precision time synchronization over
networks. Accuracy ranges from 10ns to 100ns
Time synchronization method used depends mostly on the distance between networking
devices that requires synchronization. A connection distance between these devices involves
delay and degradation of signal quality. For this reason, time synchronization method largely
depends on the distance between the equipment that require synchronization. For instance:
1PPS, 10 MHz, and TOD, are limited to equipment connections in the same
rack/room.
BITS signals are used for connections within the same building.
NTP and IEEE 1588 (PTPV2) are both protocols that provide time synchronization
over Ethernet networks, which can be a connection between two or more building or
over a whole range of distances
GPS time has become the standard for providing precise synchronization between
distant locations. a GPS device serves as a reference master clock at a location and
provides time synchronization signals to other equipment and the local network.
GPS Satellite provide time syn for two sites, siteA and SiteB
1. Network Time Protocol (NTP)
Network Time Protocol (NTP) is a networking protocol for clock synchronization between
computer systems over packet switched. This is the most prominent protocol for time-
synchronization across networks. A subset of NTP is the Simple Network Time Protocol
(SNTP) which is the protocol-compatible with NTP. The intended use of NTP is to
synchronize computer clocks in the global Internet. NTP relies on sophisticated mechanisms,
the intersection algorithm to select accurate time servers and is designed to mitigate the
effects variable network latency. SNTP, on the other hand, does not implement the full set of
NTP algorithms, but rather focuses on simpler synchronization objectives, such as those in a
local control system network. These protocols were developed for the office automation field,
where a precision of several milliseconds is sufficient.
The Network Time Protocol (NTP), established as an Internet Standard protocol, is used to
organize and maintain a set of time servers and transmission paths as a synchronization
subnet. NTP is built on the Internet Protocol (IP) and User Datagram Protocol (UDP) which
provide a connectionless transport mechanism; however, it is readily adaptable to other
protocol suites. It is evolved from the Time Protocol and the ICMP Timestamp Message, but
is specifically designed to maintain accuracy and reliability, even when used over typical
Internet paths involving multiple gateways and unreliable networks.
The purpose of the IEEE 1588 Precision Time Protocol (PTP) is to synchronize time between
different nodes on an Ethernet network. NTP allows time synchronization up to 100
milliseconds, IEEE 1588 PTP is required to achieve tighter synchronization. IEEE 1588 PTP
protocol has been revised and the current version is IEEE 1588v2 or PTPv2 which is more
accurate, providing a potential accuracy down to the nanosecond level.
PTPv2 (IEEE1588 v.2) is a network-based time synchronization standard that is designed for
distributing precise time, frequency, and phase from a clock source over packet-based
networks. With PTPv2, synchronization is achieved by exchanging PTPv2 timing messages.
To ensure clock synchronization, PTPv2 sends synchronization messages between the time
source and the receiver to determine the accurate measurement of the path delay.
PTPv2 is based on ethernet, it is embedded on the physical layer, this feature makes it
hardware-based time stamping for precise time synchronization for all participating devices
in a network.
The revised version PTPv2 protocol provides fault-tolerant synchronization among clocks
embedded in devices across a network. In PTPv2, one participant is selected to work as the
master clock, which delivers time sync messages to all slave clocks in the network.
It uses Best Master Clock algorithm (BMCA) to determine the most accurate clock in a
network, and then synchronizes all other clocks to the grandmaster clock. The grandmaster
clock (which acts as reference point for all other clocks referred to as “slave”) then sends
sync packets with embedded timestamps to slave clocks across the network. If a grandmaster
clock is no longer available on the network, the Best Master Clock algorithm defines the new
grandmaster clock and adjusts other clocks accordingly.
Network switches and other networking devices that support IEEE 1588v2 have the ability to
timestamp packets as they ingress and egress network ports. To prevent uncertain delays
within a switch from causing inaccuracies in the time synchronization, the timestamps are
added to packets between the MAC and PHY layer, exactly when a packet enters or leaves a
port. For devices where there is a delay between the software sending a packet and it leaving
a port, the extra delay time is sent in a follow-up packet. Once all network delays have been
determined from the packet timestamps, a slave clock time can be precisely adjusted to the
grandmaster clock time.
There is delay in time when messages are sent from the master clock to all participants or
slave clocks, messages take time to reach its destination. For this reason, network latency
needs to be compensated which is achieved through a series of messages exchanged between
the master clocks and the slave clocks. The messaging processes is illustrated in the figure
below.
Example of how PTP message advertising takes place between the master clock and the slave
clock.
(i) The master clock sends the Sync message. The time that the Sync message leaves the
master is time stamped as t1, which can be embedded in the Sync message itself (one-
step operation) or sent in the Follow_Up message (two-step operation).
(ii) The slave receives the Sync message; t2 is the time that the slave receives the Sync
message.
(iii) The slave sends the Delay_Request message, which is time stamped as t3
when it leaves the slave and time stamped as t4 when the master receives it.
(iv) The master responds with a Delay_Response message that contains time-stamp t4.
For networks that have a larger PTP domain with many switches and connected devices, the
IEEE 1588v2 protocol also defines a hierarchy of clock types that help ensure accurate time
synchronization across the network.
i. Ordinary Clock: A clock device that has a single port connection to the network.
This clock type can function as grandmaster or slave in the PTP domain.
ii. Boundary Clock (BC): A clock that provides multiple connections to the
network. One slave port will synchronize time with an upstream PTP clock, and
other ports may serve as master ports to other downstream slave clocks. The
connected slave clocks synchronize time directly with the boundary clock rather
than the PTP domain grandmaster clock.
iii. End-to-end transparent clock (TC)—TCs measure the residence time of PTP event
message and updates in the correction field (CF) of event message or associated
follow-up message before forwarding them. Ports do not have a specific state.
Synchronous Ethernet (SyncE) is a physical layer technology that supports frequency transfer
from hop to hop that can be made traceable to an external source such as a network clock. It
distributes synchronization over the physical layer. SyncE can deliver stable frequency
synchronization but it cannot deliver time or phase synchronization, therefore, a separate
protocol is required for delivery of time or phase synchronization.
SyncE uses the physical layer (Ethernet interfaces) to distribute frequency from the primary
reference clock (PRC) to downstream devices. It supports frequency transfer from hop to hop
and is used to provide frequency synchronization in networks. Because SyncE operates at the
physical level, all Ethernet devices along the synchronization path must be SyncEcapable.
The advantage of SyncE for network synchronization over PTPv2 is that it provides high-
quality frequency synchronization irrespective of the network load. When SyncE is enabled
on the physical Ethernet interface and switch, frequency recovery from the received signal on
the physical line occurs, and frequency synchronization is performed in the hardware.
SyncE uses the Synchronization Status Message (SSM) and the Ethernet Synchronization
Message Channel (ESMC) for clock selection, traceability, and failover.
Clock quality helps a synchronous Ethernet node derive timing from the most reliable source
and prevent timing loops by selecting the clock source with the best QL.
Selects the reference clock when there are multiple clock sources
Provides fast recovery when a part of the network fails by switching to the next best
available clock source
(ESMC) is the logical channel that uses an Ethernet PDU (protocol data unit) to exchange
SSM information over the SyncE link.
ESMC packets are received and processed on the SyncE-enabled ports that are configured as
clock sources. The clock selection algorithm uses the QL values to select the best clock
source.
5. Hybrid mode
PTPv2 and SyncE can be combined to address the synchronization needs of the network.
SyncE is used for accurate frequency synchronization; PTPv2 is used for Time of Day (ToD)
synchronization. Configuring hybrid clocking (PTPv2 and SyncE) allows a switch to obtain
frequency synchronization by using SyncE and to obtain ToD and phase synchronization by
using PTPv2.
Absolute time is needed when measurement data needs to be mapped to a certain real-life
event or when two or more Data Acquisition systems are not on the same network.
On the other hand, Relative time is mostly used in test and measurements applications or
processes
Homogeneous Distributed Systems (HDS): It is a distributed system such that all nodes
have identical hardware, the same type of architecture and operating system.
Heterogeneous Distributed Systems (HeDS): It is a distributed system such that each node
has their own operating system and machine architecture.
Each node in a distributed system can share their resources, e.g. using examples in producer-
consumer and client-server processes:
(i) the producer-consumer processes and the client-server processes, sharing of printer or
scanner. But the resources can be limited hence, they can be shared in either
cooperative or competitive modes. Resources like a printer and scanner cannot be
used by multiple processes simultaneously, so it must wait for one process to
complete and then give chance to the next process.
(ii) producer-consumer and client-server processes which work in cooperative mode. So
there is a need of proper allocation of available resources, to preserve the state of
resources and coordination between processes. To resolve these conflicts, clock
synchronization is important.
Synchronization in Distributed System can be achieved by using physical clock of the node.
For synchronization purpose, each node in the system needs to share their local clock time
with another node in the system. During this transaction (message passing of current clock
time value) some factors like a communication link failure, fault tolerance, propagation time,
non-receipt of acknowledgment, congestion in a network, the bandwidth of the
communication link and routing mechanism affect and it may raise communication delay
during this message passing which directly affects clock synchronization.
The term multilayer switch (switch/router) refers to a networking device that performs both
Open Systems Interconnection (OSI) network reference model Layer 2 and Layer 3
forwarding of packets.
The Layer 3 forwarding functions are typically based on the Internet Protocol (IP) which is
responsible for forwarding an IP packet from one subnetwork, network or VLAN to another
subnetwork, network, or VLAN.
The IP subnetwork could be created based on well-known IP sub networking rules and
guidelines or as a VLAN. (A VLAN is a logical group of devices that can span one or more
physically separate network segments that are configured to intercommunicate as if they were
connected to one physical Layer 2 broadcast domain). Even though the devices may be
located on a number of different physical or geographically separate network segments, the
devices can intercommunicate as if they are all connected to one physical broadcast domain.
For the Layer 3 forwarding functions to work, the routing functions in the multilayer switch
learn about other networks, paths to destination networks and destinations, through dynamic
IP routing protocols or static/manual configuration information
They allow routers and switch/routers to communicate and distribute network topology
information between themselves and provide updates when the network topology changes
occur. The routers and switch/routers through the routing protocols learn about the network
topology to try to select the best loop-free path on which to forward a packet from its source
to its destination IP address.
Layer 2 functions are based on Ethernet. The Layer 2 forwarding function is responsible for
forwarding packets (Ethernet frames) within a Layer 2 broadcast domain or Virtual Local
Area Network (VLAN).
The Layer 3 and Layer 2 forwarding functions can each be split into subfunctions:
The control plane and data (or forwarding) plane functions. The Layer 2 functions in an
Ethernet switch and switch/router involve simple control and data plane operations. The data
plane operations in Layer 2 switches involve MAC address learning (to discover the ports on
which new addresses are located), frame flooding (for frames with unknown addresses),
frame filtering, and frame forwarding (using a MAC address table showing MAC address to
port mappings). The corresponding control plane operations in the Layer 2 devices involve
running network loop prevention protocols such as the various variants of the Spanning Tree
Protocol (STP), link aggregation-related protocols, device management and configuration
tools etc.
In a router or switch/ router, the entity that performs the control plane operations is referred
to as the routing engine, route processor, or control engine
The entity that performs the data (or forwarding) plane operations is referred to as the
forwarding engine or forwarding processor.
(i) it runs the operating system software that has modules that include the routing
protocols, system monitoring functions, system configuration and management
tools and interfaces, network traffic engineering functions, traffic management
policy tools etc.
(ii) The control engine runs the routing protocols that maintain the routing tables from
which the Layer 3 forwarding table is generated to be used by the Layer 3
forwarding engine in the router or switch/router. In addition to running other
protocols such as PIM (Protocol Independent Multicast), IGMP (Internet Group
Management Protocol), ICMP (Internet Control Messaging Protocol), ARP
(Address Resolution Protocol), BFD (Bidirectional Forwarding Detection), and
LACP (Link Aggregation Control Protocol)
(iii) the control engine is responsible for maintaining sessions and exchanging protocol
information with other router or network devices. The control engine typically is
the module that provides the control and monitoring functions for the entire router
or switch/router, including controlling system power supplies, monitoring and
controlling system temperature (via cooling fans), and monitoring system status
(power supplies, cooling fans, line cards, ports and interfaces, primary/secondary
router processors, primary/secondary forwarding engines, etc.).
(iv) The routing engine also controls the router or switch/router network management
interfaces, controls some chassis components (e.g., hot-swap or OIR (online
insertion and removal) status of components on the backplane), and provides the
interfaces for system management and user access to the device. In high-
performance platforms, more than one routing engine can be supported in the
router switch/router. If two routing engines are installed, one typically functions
as the primary (or master) and the other as the secondary (or backup). In this
redundant routing engine configuration, if the primary routing engine fails or is
removed (for maintenance/repairs) and the secondary routing engine is configured
appropriately, the latter takes over as the master routing engine. Typically, a
router or switch/router supports a set of management ports (e.g., serial port,
10/100 Mb/s Ethernet ports). These ports, generally located on the routing engine
module, connect the routing engine to one or more external devices (e.g., terminal,
computer) on which a network administrator can issue commands from a
command-line interface (CLI) to configure and manage the device. The routing
engine could support one or more USB ports that can accept a USB memory
device that allows for the loading of the operating system and other system
software. The management plane is considered a sub plane that supports the
functions used to manage the router or switch/router via some connections to
external management devices (a terminal or computer). Examples of protocols
supported in the management plane include Simple Network Management
Protocol (SNMP), Telnet, File Transfer Protocol (FTP), Secure FTP, and Secure
Shell (SSH). These management protocols allow configuring, managing, and
monitoring the device as well as CLI access to the device.
A console port (which is an EIA/TIA-232 asynchronous serial port) could allow the
connection of the routing engine to a device with a serial interface (terminal, modem,
computer, etc.) through a serial cable with an RJ-45 connector. An AUX (or auxiliary) port
could allow the connection of the routing engine (through a serial cable with an RJ-45
connector) to a computer, modem, or other auxiliary device. Furthermore, a 10/100 Mb/s
Ethernet interface could connect the routing engine to a management LAN (or a device that
has an Ethernet connection) for out-of-band management of the router or switch/router. The
routing table (also called the Routing Information Base (RIB)) maintains information about
the network topology around the router or switch/router and is constructed and maintained
from information obtained from the dynamic routing protocols, and static routes configured
by the network administrator. The routing table contains a list of routes to particular IP
network destinations Each route is associated with a metric that is a “distance” measure used
by a routing protocol in performing the best path computation to a destination. The best path
to a destination is determined by a routing protocol based on metric (quantitative value) it
uses to “measure” the distance it takes to reach a destination. Different routing protocols use
different metrics to measure the distance to a given destination. Then the best path to a
destination selected by a routing protocol is the path with the lowest metric. Usually, the
routing protocol selects the best path by evaluating all the possible multiple paths available to
the same destination and selects the shortest or optimum path to reach that network.
Whenever multiple paths from the router to the same destination exist, each path uses a
different output or egress interface on the router to reach that destination.
It performs the actual forwarding of data. It consists of hardware and/or software elements
Software and/or hardware processing module or engine, which provides the route
(best path) lookup function in a forwarding table.
Switch fabric interface modules, which use the results of the forwarding table
lookup to guide and manage the transfer of packet data units across the switch
fabric to the outbound interface(s). The switch interface module will be responsible
for prepending internal routing tags to processed packets. The internal routing tag
would typically carry information about the destination port, priority queuing,
packet address rewrite, packet priority rewrite, and so on.
Layer 2/Layer 3 processing modules, which perform Layer 2 and Layer 3 packet
decapsulation and encapsulation and manage the segmentation and reassembly of
packets within the router or switch/router.
Queuing and buffer memory processing modules, which manage the buffering of
(possibly, segmented) data units in the memory as well as any priority queuing
requirements. As discussed above, the forwarding table is constructed from the
routing and the ARP cache maintained by the routing engine. When an IP packet is
received by the forwarding engine, a lookup is performed in the forwarding table
(and adjacency table) for the next hop destination address and the appropriate
outbound port, and the packet is sent to the outbound port. The Layer 3 forwarding
information and ARP information can be implemented logically as one table or
maintained as separate tables that can be jointly consulted when forwarding
packets. The router also decrements the IP TTL field and recomputes the IP header
checksum. The router rewrites the destination MAC address of the packet with the
next hop router’s MAC address, and also rewrites the source MAC address with
the MAC address of the outgoing Layer 3 interface. The router then recomputes
the Ethernet frame checksum and finally delivers the packet out the outbound on
its way to the next hop.
The design and implementation of software for switch/routers and routers often requires
addressing some important technical issues, that include:
The processing requirements and stability of the Layer 3 (routing) and Layer 2
(bridging) protocols and algorithms
Performance of the packet forwarding mechanisms
Interactions between the Layer 2 and Layer 3 functions
Network management requirements.
Designers of switch/routers and routers nowadays use mostly licensed or opensource real-
time kernel software and protocol software. They generally adopt high-quality simulation,
development, and testing methods when developing product-grade networking devices. To
minimize time, effort, and cost associated with device and network management,
switch/routers and routers support a combination of command-line interface (CLI) features,
scripts, web-based interfaces and tools, and menu-driven configuration programs and tools,
which are carefully integrated into the overall system software. The ultimate design goal of
the system software and hardware is to obtain high system performance and maximized
packet forwarding rate while minimizing system latency for packets.
Kernel Software
Currently, there is a large range of developed and refined/enhanced real-time kernel software,
both licensed and open-source, to address the real-time software design constraints seen in
switch/routers. An equipment vendor may choose to develop a common switch/router
interface model that permits different kernels to be used to create specific platforms as
required. In many cases, a vendor would use a common portable kernel that permits quick
retargeting of the total switch/router software for short time-to-market development needs as
well as reuse of already developed software.
In the early years of networking, until the mid-1990s, designers developed inhouse high-
performance implementations of the main routing and bridging protocols (IP, IPX, Integrated
IS–IS, Ethernet, Token Ring, FDDI, etc.). Designers also used specific software kernels and
provided extensions for any special and proprietary networking features required. In addition,
designers enhanced/extended the real-time software kernels available at that time with
software interfaces commonly used in public domain kernel software. The addition of these
software interfaces facilitated and accelerated the addition of new software from external
sources. In today’s practices, a vendor would use a common switch/router and router
software across its many internetworking platforms. A majority of the vendor’s routing and
bridging software is independent of the underlying hardware and is developed to support the
evolving networking standards and also to allow portability. When designing each platform,
the vendor would customize the performance-intensive and hardware-specific software code
to optimize the design and maximize performance for each instance of a switch/router or
router architecture.
The following are some of the most significant technical issues encountered during
switch/router software design:
Stability of the Layer 3 (routing) and Layer 2 (bridging) protocols and algorithms
Requirements for network management
Switch/router performance
Interactions between Layer 3 (routing) and Layer 2 (bridging) functions and
The amount of memory available ( A switch/router design may include an automatic
shutdown mechanism that takes effect should the device run out of memory in which
the routing information is stored. This mechanism, when used, can help prevent
routing loops in the network)
Another design issue is the mechanisms needed to handle internal control plane and
data plane congestion. To implement congestion control for traffic generated by
protocols such as TCP, a router determines whether or not a packet experiences
congestion by calculating the average queue length at interface over a given time
period and comparing this with predefined thresholds.
Stability of the routing protocols
Closed systems are created and owned by a single company while Open systems require that
technology be specified in standards documents that allow multiple companies to build
products
Open network architecture (ONA) is the overall design of a communication carrier's basic
network facilities and services to permit all users of the basic network to interconnect to
specific basic network functions and interfaces on an unbundled, equal-access basis.
It uses multiplexer at the transmitter to join several signals together and a demultiplexer at the
receiver to split them apart.
WDM are popular with telecommunication companies because they allow them to expand the
capacity of the network without laying more fibre
Normal (WDM)
Coarse (CWDM)
Dense(DWDM)
one that transmits through fibre optic networks (point to point) that include:
communication networks, mesh networks etc
beyond point to point ( free space optical networks) use the same principles as a fiber-
optic network but transmit their signals across open space without the use of fiber and
include; the satellite and terrestrial networks.
Synchronous Optical Networking (SONET) and Synchronous Digital Hierarchy (SDH) that
are used for delivery of a variety of protocols such as Asynchronous Transfer Mode (ATM),
Ethernet, TCP/IP etc.
Summary
Time synchronization is necessary for the ordering of events and to preserve the state of
resources in any network. It is very essential in a large network to reduce confusion in a
shared filesystem and important for the modification times to be consistent regardless of what
machine the filesystems are running on. Existing solutions like NTP developed by D.L Mills,
network devices can maintain synchronized time down to the sub-microsecond level and
IEEE 1588 or PTPv2 which is a more accurate synchronization standard providing a potential
accuracy down to the nanosecond level.
References
Johannessen, S. (May 2004). Time Synchronization in Lacal Area Network. Article in IEEE
control systems · , 4-8.
Ramaswami, R., and Sivarajan, K., Optical Networks: A Practical Perspective, Second
Edition, 2001, Elsevier, Philadelphia, PA, ISBN 0080513212, 9780080513218