An Overview of Networking Infrastructure for Massively

Multiplayer Online Games

Ahmad Fariz Ali Abdul Samad Ismail Abdullah Bade
Dept. of Comp. Graphics & Multimedia Dept. of Comp Sys & Comm. Dept. of Comp. Graphics & Multimedia
Fac. of Comp. Science & Info Sys Fac. of Comp. Science & Info Sys Fac. of Comp. Science & Info Sys
Universiti Teknologi Malaysia Universiti Teknologi Malaysia Universiti Teknologi Malaysia
+607-5532326 +607-5532003 +607-5532324
[email protected] [email protected] [email protected]

ABSTRACT Massively multiplayer online games (MMOGs) differentiate

A massively multiplayer online game (MMOG) is a networked themselves from other multiplayer online games by allowing
computer or video game in which tens of thousands to hundreds thousands of players to share a persistent virtual world (online
of thousands of players may interact with one another in real-time virtual worlds that exist 24 hours a day) at the same time.
in a shared environment, even though these users may be According to Dolbier (2007) [3], a persistent world is a simulation
separated by vast geographic distances. In the last decade the that keeps running whether players are logged in or not. The
popularity of MMOGs has exploded. Unfortunately, the demand integration of players into persistent virtual world creates a new
has far outweighed the resources commercial game providers can social phenomena. Players therefore even spend more money in
provide. Many MMOGs are suffering from scalability issues, order to play in these persistent online virtual worlds as long as
resulting in game world partitioning, service down time and server they are rich in content and highly interactive. With the
crashes. The centralized server architecture of most modern day widespread availability of broadband internet connection, the
MMOGs is unable to cope with this increase in the number of number of players are increasing rapidly and further contribute the
participating players. Hence, there is a need for a scalable network popularity of these games. As a result, MMOGs are becoming the
architecture which can support these large number of players dominating online game type, because they offer players the
without affecting the overall gaming experience for each player. crucial game depth and social interaction.
In this paper, we provide a review on the existing networking Organization of this paper is as follows: Section 2 highlights
infrastructure solutions for MMOGs. This includes description challenges on current networking infrastructure of MMOGs.
and comparison of different implementation techniques for the Section 3 discussed in brief about distributed virtual environment,
deployment of massive multiplayer on-line games, which work as MMOG as an example of distributed virtual environment,
a client/server and peer-to-peer paradigms. networking requirement s for MMOGs and the role of interest
management in virtual environment applications. Section 4
Categories and Subject Descriptors described in general about data transmission approaches for
C.2.1 [Computer-Communication Networks]: Network supporting multiplayer online games, followed by game state
Architecture and Design – distributed networks, network processing for multiplayer online games in section 5. Section 6
topology. discussed about workload distribution techniques for MMOGs.
Section 7 described a generic communication architecture for
General Terms MMOGs. Finally, conclusion is drawn in section 8.
Algorithms, Performance, Experimentation.

Keywords
Online games, peer-to-peer, client-server, MMOG. 2. CHALLENGES
An online multiplayer game is a virtual environment generated by
1. INTRODUCTION computers, in which users may be geographically distributed, see
Online games are an increasingly popular form of entertainment a graphical representation of the virtual world and controls their
on the Internet, with the worldwide online market predicted to be virtual character known as avatar. The avatars communicate with
worth over US$10 billion in 2009 [1]. Massively multiplayer each other to exchange information and do various actions. Some
online games (MMOGs) distinguish themselves from other online basic of such actions are moving the avatar, picking up objects, or
games by allowing thousands of players to share a single virtual communicating with other players. The scalability problems that
world at the same time. As an example of popular MMOG, money MMOGs face rise from the fact that they have to handle a massive
making online game, World of Warcraft reached a record of 11 amount of connected players, presenting them with a consistent
million active subscribers worldwide as of October 2008 [2] who view of the virtual world, providing good gaming performance
are paying monthly fee and purchasing expansions. and an enjoying experience.
Most current commercial MMOGs are based on the client-server
communication model where every packet data flows via a
centralized server. The benefits of this client-server model include
a tight control of the system, which makes data persistence, user

619
authentication and secured communication channel possible. In a virtual environments (NVEs), and (3) multiplayer computer
single session of typical MMOG, there are two main types of games (MCGs). The focus of scientific research has shifted from
processes participating: (1) the server manages and regularly the military simulations (the 1980s) through NVEs (the 1990s)
updates the game state depending on user inputs and the game- and currently to MCGs (see Figure 1). Military terminology
specific logic, and (2) the client regularly visualizes the updated prefers the word ‘simulation’, because they can be more than
game state to the user and forwards user inputs to the server. NVEs (e.g., logistical simulations). Moreover, the entertainment
Several popular multiplayer online game types, also known as industry is investing seriously on MCGs, mobile gaming and
game genres, like First Person Shooter (FPS) or Real-Time online gaming in general. Thus, we survey the literature and
Strategy (RTS) have evolved using this communication model and research regarding distributed military simulation, networked
usually are played with a few dozens of players [3]. Aided by the virtual environments, and commercial online gaming in search of
increasing broadband connection bandwidth of the Internet and patterns for network architectures which are applicable to
existing processing power at servers, the new type of Massively massively multiplayer online games.
Multiplayer Online Games (MMOG) has evolved, which allows a
high number of users to participate in a single game session.
Especially the genre of Massively Multiplayer Online Role
Playing Games (MMORPG) is played with possibly thousands of
concurrent users in a single session. The highest number of
concurrent users in a unique game world was recorded in January
2009 by ‘EVE Online’ to be near 45, 200 [4].While this
centralized communication model is good for administrative
control, security and simplicity, increasing complex game play
and artificial intelligence computation has turning the server into Figure 1. History of distributed virtual environments.
computational and bandwidth bottleneck.
Some of the very popular and large MMOGs nowadays have
managed to scale with a centralized server cluster system. For The relationship between games and simulations is not
examples, in order to serve thousands of players at one time, straightforward (see Figure 2). There are games that are
popular MMOG providers like Sony Online Entertainment simulations (e.g., football manager games) and games that are
(developer for EverQuest, a fantasy MMOG) and Linden Lab especially experience in a virtual environments (e.g., flight
(producer of Second Life, a 3-D social virtual environment) have simulators or first person shooters). While virtual environments
invested a huge number of dedicated servers (between 1,500 to simulate possibly real-world environments, computer games do
2,600 servers) interconnected by high-speed networks to form a not necessarily belong to simulations or virtual environments. As
cluster of servers [5] [6]. However, server-clusters demand an example, as the games get more abstracted from collaborative
investment of millions of dollars to develop, deploy and maintain, interactions among players, they are less and less simulations like
but the total amount of resources is still limited at any given time. puzzles and board games which are personally played on a single
While more servers can be added, the demand placed upon the machine.
system by the players has exponential grow, resulting in a
performance bottleneck [7]. Although game providers put extra
cost by running more server to support more players, MMOGs
still face huge scalability problems in order to be able to support
hundreds of thousands of simultaneous players in a persistent
virtual world.

3. DISTRIBUTED VIRTUAL
ENVIRONMENT
Virtual environments, in particular multiplayer online games
played over the Internet, have become a very popular class of
applications used by an increasing number of users worldwide.
Technically, these applications are distributed systems consisting
of several clients and servers. There have been different names Figure 2. Relationship of simulations, virtual environments
used for these applications – distributed interactive applications and computer games.
(DIAs) [8], networked interactive entertainment [9], distributed
virtual environments [10], networked virtual environments
(NVEs) [11], collaborative virtual environment (CVE) [12, 13, 3.1 Distributed Virtual Environment Classes
14], such that we use their common trait of distributed virtual All applications belonging to the class of distributed virtual
environments for referring to this application class. environment systems have a common feature that for each user a
client software graphically presents a virtual environment. The
users can move and interact in real-time in this environment in a
According to Smed, Kaukoranta, and Hakonen (2002)[15], over
very responsive way. Besides online games, there are other
the past twenty years three distinct classes of virtual environment
application classes which belong to this type of distributed
have become important: (1) military simulations, (2) networked
systems. Distributed Virtual Environment (DVE) systems like

620
DIVE [16] or MASSIVE [17] target at Computer Supported latency, distributed consistency, scalability, and others. One of the
Collaborative Work (CSCW) scenarios like virtual conferences biggest challenge is to scale the game state while providing
and distributed cooperative work. As a different application, life distributed consistency (i.e., the shared sense of space) to a
simulations like Second Life [18] allow to build virtual massive number of concurrent users. In order to provide a shared
communities, chat with other users, develop the user’s avatar as sense of space among players, each player must maintain a copy
its personal alter ego; even building and furnishing a virtual estate of the (relevant) game state on his computer. When one player
is possible. There has been some work on graphical distributed performs an action that affects the world, the game state of all
operating systems in which the user interface is realized as an other players affected by that action must be updated. As the
virtual environment. For example, the user in Croquet [19] as such number of objects that can change and the number of players in
a system moves an avatar around and uses portals to start and the world increase, the amount of information that must be
interact with different applications. exchanged between players also increases.
However, the amount of information that can be exchanged
between computers in an MMOG is bounded by at least two
3.2 Massively Multiplayer Online Games technical limitations: network bandwidth and processing power
Massively multiplayer online games (also known as MMOGs) [11] [28]. Another important requirement of an MMOG is that
are a variation of the standard multiplayer online game, but with the information about the game state must not only be exchanged
the addition that they can support thousands of simultaneous between players, but it must be exchanged in a timely manner.
players in a persistent game world. Regarding multiplayer online Game state changes must be propagated within a delay that does
games as a main application type of virtual environments, there not affect the game play, usually below human perception.
are several different game design types called genres. There are Delivery of information in a timely manner must deal with the
three main genres of multiplayer online games currently played technical limitation of network latency [11] [28].
over the Internet:

3.3.1 Network Bandwidth

3.2.1 Role-Playing Games The bandwidth is the amount of data that can be transmitted over
Role-Playing Games (RPG) for small sessions of about up to a a network in a fixed amount of time. The bandwidth is limited by
dozen players, respectively their counterpart for huge sessions – the underlying hardware that connects the computers. On the
Massively Multiplayer Online Role-Playing Games (MMORPG). Internet many factors can affect the available bandwidth: the
MMORPGs like World of Warcraft [2], Ultima Online [20] or user’s connection to its Internet Service Provider (ISP), the ISP’s
EverQuest II [21] can be played by thousands of concurrent users hardware and software infrastructures, traffic congestion, and
in a single game world and usually require the user to subscribe to others. Furthermore, each user of an MMOG is likely to have
the game service on a monthly basis. different bandwidth capabilities. Ultimately, the amount of
information that can be exchanged between computers in order to
keep the game state consistent in an MMOG is bounded by the
3.2.2 First Person Shooter network bandwidth capabilities of the users. In order to scale an
First Person Shooter (FPS) allow users to play a fast-paced action MMOG it must be possible to add new users without dramatically
game in a collaborative or competitive way. In such games like increasing the bandwidth need of other users. Interest
Counter-Strike: Source [22], Quake 4 [23] or Unreal Tournament management is a technique that has been widely used by
, the user observes the game directly through the in-eye view from developers to cope with high bandwidth demand of multiplayer
the avatar. FPS games are usually played with 2 to 32 users, the online game.
maximum number of players in most games currently is 64 users
[3]. Although there are some few examples of games featuring
more simultaneous users in a single session like Joint Operations 3.3.2 Server Processing Power
[24] (up to 150 users) or PlanetSide [25] (up to 400 users), the The processing power is the amount of calculations that can be
FPS genre cannot be considered as being playable in a large-scale performed by a game server in a fixed amount of time. Every
setting in currently available titles. client computer’s CPU has a fixed processing power capacity. In a
game, processing power is required for multiple concerns such as
physics, collision detection, graphic rendering. In networked
3.2.3 Real-Time Strategy games such as MMOGs, processing power is also required to send
Real-Time Strategy (RTS) games are usually played by 2 to 12 and receive network messages. The amount of information
players. Each of these players can control an army consisting of exchanged between client computers and servers does not only
several different avatars, usually called units. The number of units affect network bandwidth, but it also consumes CPU resources.
each player can control ranges from several tens as in Command
and Conquer Generals [26] over hundreds to even thousands as in
Cossacks [27], for example. However, the number of human 3.3.3 Network Latency
players is limited in existing RTS games, which do not allow Latency is the amount of time it takes a message to travel over a
large-scale sessions. network from source to destination. Although latency is ultimately
bounded by the physical limitation of the underlying hardware,
3.3 Networking Requirement for MMOGs congestion caused by a large amount of information sent over the
The development of MMOGs comprises many technical network also affects latency. When one network application
challenges: limited network bandwidth, computational power,

621
transmits data over a network, the network’s latency determines if messages are sent to all users in the virtual environment, the
when the data becomes available for consumption by the amount of transmission and processing grows at O(n2), which is
receiving application. As players expect to interact with each clearly not scalable. Real-world observation tells us that each
other and with the environment in real-time [29, 30, 31], the individual only has a limited visibility or ‘area of interaction’. We
MMOG developer wishes to give the user the illusion that the cannot look around corners, through doors or walls, into pockets,
entire game world is located on their local machine. Network listening on conversations in a busy room and so on. In other
latency means that incoming data about the state of the game words, our interest is localized [37]. Interest management
environment is already somewhat out-of-date by the time it therefore deals with relevant information filtering, to reduce
arrives for processing. network resource consumption while maintaining adequate
interactivity.
Interest management in virtual environment can be generally
3.3.4 Distributed Consistency classified into three groups [15]:
Latency and distributed communications make it difficult to
maintain consistent world interpretations for each participant. The • Area-of-Interest Management
MMOG developer wishes to present users with a single shared • Multicast Routing
environment, which is challenging if the messages communicating
changes in the environment arrive at different times at each node. • Subscription-based Aggregation
Further, if participants act based on differing views of the
Area-of-Interest Management and Multicast Routing are within
environment, some means must be provided to determine which
the scope of this research and are described in the following
resulting version of the environment is correct.
paragraphs.
A lack of consistency hampers the level of immersion of the
player, taking away from their enjoyment of the game and
decreasing their desire to play. Fortunately, the military, 3.4.1 Area-of-Interest Management
academic, and gaming communities have all found that human Area-of-Interest (AOI) Management is a mechanism that filter
players are often satisfied given the appearance of consistency. messages such that users ideally do not receive and process
Overall, maintaining consistency in an online game has proven to messages that they do not need in order to correctly proceed in the
be a difficult task for game developers; players commonly virtual environment. Thus this mechanism provide MMOGs with
discover defects that violate the consistency of the game world, intelligent information dissemination through the network that has
many of which may be exploited to gain an unfair advantage over the potential to considerably reduce the amount of information
other players [32, 33, 34]. These consistency violations tend to that must be exchanged between users and the resources it
stem from implementation or design errors. Achieving true consumes. . It can be distance-based (by geography), class-based
consistency in a distributed environment is often quite complex, (by object or user attributes), or some combination of both [37].
having subtle failure modes and timing or ordering considerations.

3.4.2 Multicast Routing

3.3.5 Scalability Multicast Routing is a network protocol technique that realizes the
The challenge for the massively multiplayer online game area-of-interest mechanism described previously. Multicast
developer is to design and implement a system that addresses each routing is an efficient network data dissemination protocol
of the other technical challenges in the presence of large numbers compared to unicast and broadcast. Some distributed applications
of concurrent users. The required scalability aspects of a particular require that widely separated processes work together in groups,
distributed virtual environments application depend on the actual for example a group of users participating in virtual meeting via
application category and design. For example, in multi-user video conferencing system [38]. Multicast routing is ideally used
training simulations for natural disaster recovery scenarios, when groups are large but small compared to the entire
physical simulations of earthquake or behavior simulation of communication network. By applying multicasting in this scenario
virtual crowds in panic are usually required to be scaled [35, 36]. the problems associated with broadcasting and unicasting
On the other hand, the MMOG developer seeks to partition and messages in large groups can be avoided. Broadcasting sends out
distribute computational tasks and game state such that; a) each messages to all the nodes in the network regardless whether they
participant is aware of any and all game information which is are interested or not. Unicasting requires the sender to send one
relevant to that participant, b) each computing device participating message per receiver and this is expensive when the groups are
in the game is not over-burdened, c) the capacity of the network large.
resources available to the game is not exceeded and, d) the size of
the concurrent users and the virtual environment can be increased
without requiring architectural changes to the game.
4. DATA TRANSMISSION TECHNIQUES
Data communication in larger multiplayer games can be
performed using a variety of packet delivery methods. This
3.4 Interest Management in Virtual includes basic unicast as the most popular current choice, but also
Environment broadcast and multicast approaches[39]:
Messages are generated by user actions (for example, moving to a
particular location) and are communicated among participating
players of MMOG to maintain distributed consistency. However,

622
4.1 Unicasting including the artificial intelligence of NPCs and the virtual world
Unicasting is communication between a single sender and a single simulation, to the current game state . As the result of this
receiver, which allows to control and direct the traffic from point calculation, the states of several dynamic entities have changed.
to point. If the same message is intended for multiple receivers, (3) The final step transfers the new game state back to the clients.
unicasting wastes bandwidth by sending redundant messages.

4.2 Broadcasting
Broadcasting is communication between a single sender and all
recipients, which means that every node has to receive and
process every broadcast message. Obviously, this leads to
problems as the number of participants grows, which is why
broadcast transmissions are not guaranteed on the WANs.

4.3 Multicasting
Multicasting is communication between a single sender and
multiple receivers, which allows receivers to subscribe to groups
that interest them. The sender does not need to know all the
subscribers but sends only one message, which is received by Figure 3: Typical steps in state update loop processing
multiple receivers belonging to the group. Because no duplicate
messages are sent down the same distribution path, multicasting
provides an efficient way to transmit information among a large There exist several possible network architectures which provide
number of nodes. the actual placement of this state update loop component inside
Multicast transmission is not quite as efficient as basic broadcast, the distributed system [28]. In order to support game applications
but is usually much more efficient than multiple unicast with distributed clients, several architectural solutions have been
operations. Interest management systems in games often specify presented in the research literature and in commercial products.
multicasting as a mean of efficiently implementing interest groups All architectures are classified as either Centralized (as illustrated
and associated network communication [40]. in Figure 4 (a) client-server), Distributed (P2P) (as illustrated in
Figure 4 (b) peer-to-peer), or Hybrid - depending on where the
game state is stored. Centralized and distributed architectures
store the game state on centralized servers and client machines
5. GAME STATE PROCESSING FOR respectively. Hybrid architectures store the game state on both
MULTIPLAYER ONLINE GAMES centralized and client machines.
From the application level point of view, real-time MMOGs
typically simulate a spatial virtual world where a particular player
moves and acts through an avatar, which represents the player
within the game. The state of the virtual world can be
conceptually separated into a static part and a dynamic part. The
static part covers the elements of the game world that never
change, e. g., environmental properties like the landscape,
buildings and other non-changeable objects. Since the static part is
pre-established, no information exchange about it is required
between the game participants. The dynamic part of the game
state covers objects like the players' avatars, non playing
characters (NPCs) controlled by the computer, items that can be
collected by players or, generally, objects that can change their
state. These objects are called entities and the sum of all entities is
the dynamic part of the game state, such that dynamic information
need to be exchanged between the game participants. Figure 4: Network architectures

Typical multiplayer online real-time games implement an endless 5.1 Client-server Architecture
loop, called state update loop, which repeatedly updates the game In this traditional architecture, a single server takes care of the
state in real time. Figure 3 shows one iteration of the server state overall state processing and incorporates the application logic.
update loop for multiplayer games based on the client-server Figure 5 illustrates the client-server architecture for game states
architecture. An iteration of state update loop consists of three processing. The clients run on the users’ desktop computers, send
steps: (1) At first the clients process the users' input and transmit user inputs to the server and present the received application state
them to the server. (2) The server then calculates a new game graphically. For a typical MMOG sessions running over the
state by applying the received user actions and the game logic,

623
Internet, the server usually runs at a dedicated server host extremely difficult [46]. The clients do not alter their copies of the
connected to the network over a high-bandwidth link. entity directly, but send user inputs to the server and receive and
visualize the according state change.
Another very important feature of the client-server approach is
that all user inputs can be verified by the server before applying
them to the overall state and that clients only need to know the
state of the particular region of interest of the user. This concept
of only transferring state information concerning the particular
region of interest to the clients saves communication bandwidth
[47, 48].
Additionally, especially in highly competitive online games,
clients must not be considered trustworthy, because malicious
users can hack and alter the client application to provide unfair
functionality or sabotage the session. This so-called cheating [49,
50] in online games is a major problem which has to be actively
countered by implementing cheat-secure mechanisms for
communication and game state processing.
Figure 5: Game states processing in client-server architecture The client-server approach naturally implements the region of
interest concept by sending the clients only limited information
about the overall virtual environment, which prevents hacked
One major drawback of the architecture is that this server
clients to reveal information to the user which should be hidden,
constitutes a single point of failure for the complete session,
like the positions of hidden opponents in an action game.
because it is the only component of the distributed architecture
Additionally, the server verifies each user action before executing
knowing the complete application state: if the server fails, then
it, which allows to implement particular anti-cheating mechanisms
there is no other component able to take over the state
like plausibility checks at the central server host. Often, these
computation and the complete session is lost.
advantages push game middleware providers to adopt this
The server has to receive the user actions from all clients, architectural solution [51].
compute the new overall state of the application and send each
client the new state information according to its region of interest
of the virtual environment. Therefore, the computational and 5.2 Peer-to-peer Architecture
communication load of the server grows at least linearly with the In the common, fully connected peer-to-peer (P2P) architecture
number of connected clients. If this load exceeds the there is no particular dedicated process computing the state
computational or communication resources of the server host, updates. Instead, each process running at a user’s desktop
then the server will become congested and not be able to finish the computer has its own built-in server that processes the complete
games state processing within the particular deadlines, thus virtual world. All peers consisting of the two components are
violating the real-time constraints of the MMOG. Furthermore, interconnected with each other and have to synchronize their
this architecture is not scalable with the number of increasing world states as illustrated in Figure 6.
players [41, 3, 42, 43, 5, 6] and generally support between 8 and
64 players [44]. It will continue to be the most common
architecture for games with small numbers of players such as First
Person Shooters (FPS) and cannot handle more than a few
hundred players.
To handle large numbers of concurrent players developers use
clusters of computers, also known as Server-Cluster. This is the
approach used by EVE Online [4], which holds the record for the
greatest number of concurrent players in one shard to be near 45,
200 and was recorded in January 2009. Each computer in the
cluster is responsible for managing a different portion of the
virtual world. Load balancing transfers players between
computers in the cluster, keeping every machine’s processing and
bandwidth requirements from reaching saturation.
Figure 6: Game states processing in P2P architecture.

Although the client-server architecture exhibits a single point of

failure for the online gaming session and has a limit of the This peer-to-peer architecture approach does not require a central
maximum number of participating users, it is widely used by server and, therefore, allows users to run virtual environment
commercial online games, such as Quake [23], Torque Network sessions without accessing a server machine. Additionally, the
Library [45], due to several advantages. The architecture is easy to absence of a central server means that there is no single point of
implement and consistency of the game state is automatically failure, such that when a particular peer fails, the remaining
achieved, because only the single server is allowed to alter the clients are able to continue the application session. Each peer has
games state and thus illegal manipulations of the game state are to receive and process all actions of remote users and send the

624
actions of its own user to all other peers. The distributed
processing at the server-components has to realize a particular
consistency model which ensures that the distributed application
state at all peers is the same. In online game implementations,
user actions are commonly timestamped and appropriately
ordered at the peers before they are executed, as for example
described in [52] for the real-time strategy game Age of Empires
[53].
All peers have to execute the same computations and send and
Figure 7: Partitioning a virtual world into several zones.
receive the same amount of state and action of game data.
Therefore, the maximum number of connected peers is
determined by the host with the lowest computation power and
communication bandwidth. If this particular peer is not able to Figure 7 depicts an example of a virtual environment partitioned
maintain the processing update rate or communicate state into four zones, where each zone-server handles its entities
information in time anymore, then the complete session will independently of the rest of the environment. The overall
become unresponsive. Consequently, the fully connected peer-to- workload partitioning of the entity processing is done in an
peer architecture has only been implemented for small-scale independent, side-by-side manner based on the virtual geography,
online action and strategy games. i.e., the large virtual world. Communication between servers only
occurs when a user moves the avatar from one zone into another:
The use of the peer-to-peer architecture for virtual environments The client has to establish a new connection to the server
has been discussed in detail during the last decade [54] [55]. responsible for the new zone and the servers have to migrate the
However, with Internet-based dedicated servers becoming less client’s state from one to the other. Zoning is used especially in
costly, the advantage of peer-to-peer approach lost its importance. Massively Multiplayer Online Role-Playing Games like
Recent online games do not use the fully connected peer-to-peer EverQuest 2 [21] or World of Warcraft [2] which feature a very
concept anymore. However, different heterogeneous concepts large game world suitable to be partitioned this way. Figure 8
using concepts related to the peer-to-peer approach have been shows the world map of the MMORPG Ragnarok Online split up
presented, as for example [56, 57, 58] that resort to P2P protocols into 80 independent zones.
which dynamically build overlay networks to support
communication among peers.

6. PARALLELIZATION OF MMOGS
WORKLOAD DISTRIBUTION
In MMORPGs, which incorporate a huge virtual world, the
scalability aspect of increasing the total player number is mostly
happened. This aspect is made scalable by using the zoning
approach: the world is subdivided into independent regions, so-
called zones, which are then processed in parallel on several
servers. This results in a very good scalability if the virtual world
is very large and thus can be subdivided in many zones. Besides
zoning, instancing is another commonly used approach which
creates multiple, independent copies of particular areas of the
virtual world, called instances. Instancing limits the user
interactivity in the application, because users residing in a
particular instance cannot interact with those in other instances.

6.1 Zoned Virtual World

Zoning [59, 60] partitions the virtual world into disjoint parts, Figure 8: Virtual world of the MMORPG Ragnarok Online
called zones, and which are processed in parallel on different partitioning.
servers, as illustrated by Figure 7. A particular client is connected
to the server responsible for the region in which the single avatar
of the user is located and has to change the server connection if 6.2 Instanced Virtual World
the user moves the avatar into another zone. Although clients can The instancing approach creates several independent copies of
move between zones, no interaction between clients in different particular regions of the virtual environment, which can be
zones is possible. The zoning approach fits best for games where processed in parallel [61]. Instances allow to run independent
the players are reasonably distributed in a very large virtual world. copies of such spaces or regions on several servers as illustrated in
Figure 9. Ultima Online refers to these instances of regions as
shards [20]. Usually, some particular regions of virtual worlds are
more frequently visited than others, as for example city areas to
meet and interact or dungeon areas to adventure in By creating

625
independent copies of a same region of the virtual world, the Reckoning’ [29, 30], the missing network packet during a running
instancing concept allows to handle congestion of particular areas. online game session is computed with an approximation
If for example, a lot of users move into the same dungeon in an techniques. Based on previously received packets, the node
MMORPG, independent instances of that dungeon allow to predicts movement of particular object. The predicted state is used
parallelize the processing of that area. In role-playing games like until new information is received from the sender node.
World of Warcraft, users arrange in teams (so-called groups) to Depending on the prediction accuracy, the approximated location
play together in a particular instance of a certain region. This way, can be some distance from its actual location, as illustrated in
a high number of teams are able to adventure in the same region Figure 11.
without congesting the real-time processing of a single server,
because the independent instances can be operated on several
servers. Instancing also allow the effects of server failure to be
isolated to the players who are using the failed server.

Figure 10: Communication architecture for (massively)

multiplayer online games.

Figure 9: Independent instanced copies of particular virtual

world regions.

7. GENERIC MMOGs COMMUNICATION

ARCHITECTURE
A generic multiplayer online game core components can be
generalized by a layered communication architecture as illustrated
in Figure 10. At the top level are the virtual world or scenarios
with which the players interact. This layer is composed of virtual
world objects (as described previously) and events (for
representing user input and game world actions). The level below Figure 11: Dead Reckoning Example
is the game state management component, which continuously
processes user actions and virtual world events, according to the
rules of the game logic, to compute a new game state. Network The first three layers, as illustrated in Figure 10, are fundamental
Bandwidth Management is responsible to control network packet components for any client-server-based multiplayer online game.
transmission of data messages generated from previous two upper However, in the case of massively multiplayer online games, an
layer, in a bandwidth efficient manner. This layer may comprises additional layer of ‘Parallelization & Distribution’ components is
of ‘Interest Management’, ‘Packet Compression & Aggregation’ required for achieving high scalability.
and ‘Dead Reckoning’ techniques. The Interest Management
component implements Area-of-Interest concept which assigns
each avatar in the game world a specific area where dynamic
game information is relevant and thus has to be transmitted to the 8. CONCLUSION
avatar client. This component optimizes network bandwidth by With scalability in the area of distributed virtual environment, we
omitting irrelevant information in the communication. The refer the capability of the MMOG application to sustain the real-
purpose of ‘Packet Compression is to reduce the size of network time processing constraints for an increasing application demand.
packet (number of bits) needed to represent particular game There are several different aspect of scalability in which the
generated messages. Thus the compression of network packet size processing demand of the distributed interactive applications
helps to minimize overall network traffic. In addition, ‘Packet increases; the common characteristic of these aspects is that they
Aggregation’ [28] further reduces high bandwidth multiplayer require more computing resources, such as computational
online games requirement by merging several packets and processing power, communication bandwidth, storage space, etc.
transmitting their content into one larger packet. In ‘Dead when being increased. The underlying architecture of such
distributed system must be expandable for adding such resources

626
in order to scale at a particular level and maintain timely virtual [16] Frecon, E. and M. Stenius, DIVE: A Scalable Network
world state computations. Architecture for Distributed Virtual Environments.
Distributed Systems Engineering Journal, 1998. 5:p. 91-100.
[17] Greenhalgh, C., J. Purbrick, and D. Snowdon. Inside
9. REFERENCES Massive-3: Flexible Support for Data Consistency and World
[1] Worldwide Market Forecasts for the Video Game and Structuring. in Proceedings of the Third ACM Conference on
Interactive Entertainment Industry. DFC Intelligence 2008 Collaborative Virtual Environments (CVE 2000). 2000:
June 30 [cited 2008 16 December]; Available from: ACM.
http://www.dfcint.com. [18] Linden Research. Second Life. [Online]. Available at:
[2] World of Warcraft Surpasses 11 Million Subscribers http://secondlife.com
Worldwide. Blizzard Entertainment (Press Releases) 2008 [19] Smith, D.A., et al. Croquet - A Collaboration System
October 28 [cited 2008 16 December]; Available from: Architecture. in Proceedings of First Conference on Creating,
http://www.blizzard.com/us/press/081028.html. Connecting and Collaborating through Computing. 2003.
[3] Dolbier, G., ‘Massively multiplayer online games, Part 1: A Kyoto, Japan.
performance-based approach to sizing infrastructure’, IBM [20] Electronic Arts. Ultima Online. [Online]. Available at:
developerWorks (Technical Report), 10 Apr. 2007. http://www.uo.com
[4] Welsh, O. EVE Breaks Concurrent User Record. EuroGamer [21] Sony Entertainment. Everquest 2. [Online]. Available at:
News. 2009 January 6. [Online]. Available at: http://everquest2.station.sony.com.
http://www.eurogamer.net/articles/eve-breaks-concurrent-
user-record [22] Valve Software. Counter-Strike: Source . [Online]. Available
at: http://www.valvesoftware.com
[5] Kushner, D., Engineering EverQuest: Online Gaming
Demands Heavyweight Data Centers. Spectrum, IEEE, 2005. [23] ID Software. Quake 4. [Online]. Available at:
42(7): p. 34-39. http://www.idsoftware.com/games/quake/quake4
[6] Terdiman, D. 'Second Life': Don't worry, we can scale. [24] NovaLogic. Joint Operations: Combined Arms. [Online].
ZDNet News (Personal Technology) 2006 June 6. [Online]. Available at: http://www.novalogic.com
Available at: http://news.zdnet.com/2100-1040_22- [25] Sony. PlanetSide. [Online]. Available at:
148320.html. http://planetside.station.sony.com
[7] Brandt, D. Networking and Scalability in EVE Online. in [26] Electronic Arts. Command & Conquer Generals. [Online].
Proceedings of 4th ACM SIGCOMM workshop on Network Available at: http://www.commandandconquer.com
and system support for games. 2005. Hawthorne, New York,
[27] CDV Software Entertainment AG and GSC Game World.
USA: ACM.
Cossacks game series. [Online]. Available at:
[8] Delaney, D., T. Ward, and S. McLoone, On Consistency and http://www.cossacks2.de
Network Latency in Distributed Interactive Applications: A
[28] Smed, J., T. Kaukoranta, and H. Hakonen. Aspects of
Survey. Presence: Teleoperators & Virtual Environments,
Networking in Multiplayer Computer Games. in
2006. 15(2): p. 218–234.
International Conference on Application and Development of
[9] Capps, M., McDowell, P., & Zyda, M. A Future for Computer Games in the 21st Century. 2001.
Entertainment-Defense Research Collaboration. IEEE
[29] Aronson, J., Dead Reckoning: Latency Hiding for Networked
Computer Graphics and Applications, 2001. 21(1) :p. 37–43.
Games. 1997, Gamasutra. [Online]. Available at:
[10] Stytz, M. R. Distributed Virtual Environments. IEEE http://www.gamasutra.com/features/19970919/aronson_01.ht
Computer Graphics and Applications, 1996. 16(3):p. 19–31. m
[11] Singhal, S. and Zyda, M., Networked Virtual Environments: [30] Lincroft, P., The Internet Sucks: Or, What I Learned Coding
Design and Implementation (Siggraph Series). 1999, New X-Wing vs. TIE Fighter. 1999, Gamasutra. [Online].
York: ACM Press. Available at:
[12] Park, K. S., & Kenyon, R. V. Effects of Network http://www.gamasutra.com/features/19990903/lincroft_01.ht
Characteristics on Human Performance in a Collaborative m
Virtual Environment. In Proceedings of Virtual Reality (VR [31] Pantel, L. and L.C. Wolf. Network Issues for Video and
’99), 1999. Games: On the Impact of Delay on Real-Time Multiplayer
[13] Vaghi, I., Greenhalgh, C., & Benford, S. Coping with Games. in 12th International Workshop on Network and
Inconsistency Due to Network Delays in Collaborative Operating Systems Support for Digital Audio and Video.
Virtual Environments. In Proceedings of Symposium on 2002. Miami, FL: ACM Press.
Virtual Reality Software and Technology (VRST ’99), 1999. [32] IMG News, DiabloII.Net Recaps Real Issues. in Inside Mac
[14] Benford, S., et al., Collaborative Virtual Environments. Games. 2002. [Online]. Available at:
Communication of the ACM, 2001. 44(7): p. 79-85. http://www.insidemacgames.com/news/story.php?ArticleID=
4741
[15] Smed, J., T. Kaukoranta, and H. Hakonen, A Review on
Networking and Multiplayer Computer Games, in TUCS [33] Kirmse, A. and C. Kirmse, Security in Online Games. 1997.
Technical Report No 454. 2002. [Online]. Available at:
http://www.gamasutra.com/features/19970707/security.htm

627
[34] Shachtman, N., 'Blizzard' of Cheaters Banned. Wired News. [49] Jenkins, K. Designing Secure, Flexible, and High
[Online]. Available at: Performance Game Network Architectures, December 2004,
http://www.wired.com/gaming/gamingreviews/news/2002/09 Gamasutra. [Online]. Available at:
/55092 http://www.gamasutra.com/view/feature/2174/designing_sec
[35] Sung, M., M. Gleicher, and S. Chenney. Scalable Behaviors ure_flexible_and_.php
for Crowd Simulation. in Computer Graphics Forum 23, [50] Yan, J. and Randell, B. A Systematic Classification of
(Eurographics ’04). 2004. Cheating in Online Games. In NetGames ’05: Proceedings of
[36] [56] Treuille, A., S. Cooper, and Z. Popovic, Continuum 4rd ACM SIGCOMM Workshop on Network and System
Crowds. ACM Transactions on Graphics, 2006. 25(3): p. Support for Games. ACM Press, 2005.
1160-1168. [51] Bauer, D., Iliadis, I., and Scotton, P., Communication
[37] Morse, K.L., Bic, L. and Dillencourt, M., Interest Architectures for Massive Multi-Player Games. Multimedia
Management in Large-Scale Virtual Environments. Presence: Tools and Applications, 23:47–66, 2004.
Teleoperators & Virtual Environments, 2000. 9(1): p. 52-68. [52] Bettner, P. and Terrano, M. 1500 Archers on A 28.8:
[38] Tanenbaum, A. and M.v. Steen, Distributed Systems: Network Programming in Age of Empires and Beyond. In
Principles and Paradigms (2nd Edition) 2007, Upper Saddle Game Developer Conference Proceedings, San Jose,
River, New Jersey: Pearson Education Inc. California, USA, March 2001.

[39] Forouzan, B.A., TCP/IP Protocol Suite (3rd Edition). 2007: [53] Ensemble Studios. Age of Empires. [Online]. Available at:
McGraw-Hill. http://www.microsoft.com/games/empires

[40] Zou, L., M.H. Ammar, and C. Diot. An Evaluation of [54] Diot, C. and Gautier, L. A Distributed Architecture for
Grouping Techniques for State Dissemination in Networked Multiplayer Interactive Applications on the Internet. IEEE
Multiuser Games. in Proceedings of the ninth Internation Networks magazine, 13(4), July/August 1999.
Symposium on Modeling, Analysis and Simulation of [55] Gautier, L. and Diot, C. Design and Evaluation of MiMaze, a
Computer and Telecommunication Systems. 2001. Multi-player Game on the Internet. In International
[41] Cecin, F. R. et. al., Freemmg: A Hybrid Peer-to-Peer and Conference on Multimedia Computing and Systems, pages
Client-Server Model for Massively Multiplayer Games. In 233–236, 1998.
Proceedings of ACM SIGCOMM 2004 workshops on [56] Knutsson, B., et al., Peer-to-peer Support for Massively
NetGames ’04, 2004. Multiplayer Games. In Proceedings of IEEE Infocomm,
[42] Jenkins, K. Designing Secure, Flexible, and High March 2004.
Performance Game Network Architectures, December 2004, [57] Boukerche, A. Araujo, R.B. and Laffranchi, M. Multiuser 3D
Gamasutra. [Online]. Available at: Virtual Simulation Environments Support in the Gnutella
http://www.gamasutra.com/view/feature/2174/designing_sec Peer-to-Peer Network. Journal of Parallel and Distributed
ure_flexible_and_.php Computing, 65(11):1462–1469, 2005.
[43] Zhuang, S. Q., Zhao, B. Y., and Joseph, A. D. Bayeux: An [58] Yu, A. and S.T. Vuong. MOPAR: A Mobile Peer-to-Peer
Architecture for Scalable and Fault-tolerant Wide-area Data Overlay Architecture for Interest Management of Massively
Dissemination. In Proceedings of NOSSDAV’01, Jun. 2001. Multiplayer Online Games. in Proceedings of the
[44] Zaichen, Z. and V.O.K. Li, Network-Supported Layered International Workshop on Network and Operating System
Multicast Transport Control for Streaming Media. Parallel Support for Digital Audio and Video, 2005.
and Distributed Systems, IEEE Transactions on, 18(9): p. [59] Wentong, C., et al. A Scalable Architecture for Supporting
1332-1344, 2007. Interactive Games on the Internet. in Proceedings of the 16th
[45] GarageGames Inc., Torque Network Library (TNL), Parallel and Distributed Simulation, 2002.
[Online]. Available at: http://www.opentnl.org [60] Dibbell, J., What's in a Shard?, in Terra Nova "Blogs". 2004
[46] Mauve, M., Fischer, S., and Widmer, J., A Generic Proxy April 2. [Online]. Available at:
System for Networked Computer Games, in Proceedings of http://terranova.blogs.com/terra_nova/2004/04/whats_in_a_s
the 1st Workshop on Network and System Support for har.html
Games, 2002. [61] Frank, G., et al., RTF: A Real-Time Framework for
[47] Lambright, R., Distributing Object State for Networked Developing Scalable Multiplayer Online Games, in
Games Using Object Views. September 2002, Gamasutra. Proceedings of the 6th ACM SIGCOMM workshop on
[Online]. Available at: Network and system support for games. 2007.
http://www.gamasutra.com/resource_guide/20020916/lambri
ght_01.htm
[48] Masa, M. and Zara, J. Interest Management for
Collaborative Environments Through Dividing Their Shared
State. In Cooperative Design, Visualization, and
Engineering, volume 3190 of Lecture Notes in Computer
Science, pages 156–163. Springer-Verlag, 2004.

628