TheTRIBESEngineNetworkingModel

orHowtoMaketheInternetRockforMultiplayerGames
by
MarkFrohnmayerandTimGift
[email protected]@Dynamix.com

Abstract
Thispaperdiscussesthenetworkingmodeldevelopedtosupporta
"realtime"multiplayergamingenvironment.Thismodelisbeing
developedforTRIBESII,andwasfirstimplementedinStarsiege
TRIBES,amultiplayeronlineteamgamepublishedinDecember
'98.Thethreemajorfeaturesofthismodelare:supportfor
multipledatadeliveryrequirements,partialobjectstateupdates
andapacketdeliverynotificationprotocol.
Overview
StarsiegeTRIBESsupportstwomodesofplay:singleplayeror
multiplayeroveraLANortheInternet.Themultiplayermode
supportsupto128humanorAIcontrolledplayersinasingle
game.PerformanceovertheInternetdrovethedesignofthe
networkingmodel.Themodelsupportslowendmodemconnections
andisdesignedtodealwithlowbandwidth,highlatencyand
intermittentpacketloss.
Themodeldealsprimarilywiththedeliveryofdataandakey
conceptistheclassificationofdeliveryrequirements.
Alldataisclassifiedintooneofseveralrequirementcategories
andthedesignofeachcomponentinthemodelcentersaround
meetingthoserequirements.Weorganizetransmitteddataas
follows:
1.
2.
3.

Nonguaranteeddataisdatathatisneverretransmittedif
lost.
Guaranteeddataisdatathatmustberetransmittediflost,
anddeliveredtotheclientintheorderitwassent.
MostRecentStatedataisvolatiledataofwhichonlythe
latestversionisofinterest.

4.

GuaranteedQuickestdataisdatathatneedstobedeliveredin
thequickestpossiblemanner.

Thenetworkingmodelisdividedintothreemajorcomponentsas
showninFigure1:
1.

AConnectionLayerthatdealswithnotificationanddelivery
ofpacketsbetweenclientandserver.Thefeaturesofthis
layeralongwithastreamclassprovidethegeneral
infrastructureonwhichtheotherlayersarebuilt.

2.

AStreamLayerwhichprovidespacketstreammanagement.This
layeremploysfivestreammanagerstodealwithevents,object
mirroring,inputmovemanagement,staticdataandstring
compression.Eachofthefivestreammanagersprovides
differentdatadeliveryguarantees.

3.

ASimulationLayerwhichmanagesallobjectsinthe
simulation.Afulldescriptionofthislayerisoutsidethe
scopeofthisarticlebutseveralitemsarerelevant:the
advancementoftime,objectscopingandclientprediction.

ThoughStarsiegeTRIBESemploysaclientserverconnectionmodel,
onlyafewofthestreammanagersareasymmetrical.Nothing
preventsthismodelfrombeingusedinpeertopeerormulti
serverarchitectures.

SimulationLayer
Ghost
Manager

Move
Manager

Datablock
Manager
EventManager

String
Manager
StreamLayer

StreamManager
ConnectionManager

ConnectionLayer

PlatformPacketModule
Figure1

PersistentObjects
Thoughpolymorphicobjectpersistenceisnotanintegralpartof
thenetworkingmodel,itisafeaturethatisusedextensively.A
largepercentageofdatatransmissionisdoneusingpersistent
objectsandthepolymorphicnatureoftheobjectsusedtohide
thetypeandcontentofthedatafromthenetworkingcode.

Eachclassthatisdeclaredpersistentregistersa"class
representative"(ClassRep)objectwiththePersistmanagerandis
assignedauniqueClassID.TheClassRepisusedbythePersist
managertoconstructobjectsofthetypeitrepresents.By
mappingassignedClassIDstoClassRepsthePersistmanagercan
constructanypersistentobjectgivenitsclassID.This
constructionbyClassIDalongwithvirtualreadandwrite
methodsprovidesbasicpolymorphicIO.
Writinganobjectintoastreamisatwostepprocess.Firstits
classIDiswritten,thentheobject'svirtualwritemethodis
invoked.ReadinganobjectinvolvesreadingtheClassID,
constructingtheobjectusingthePersistmanager,andthen
invokingitsvirtualreadmethod.Inthiswayboththeobject's
class,andthedataitreadsandwritesarehiddenfromthe
streammanagers.
ConnectionLayer
TheConnectionlayerprovidestransmissionofpacketsbetween
hostmachinesandisdividedintotwomodules,aplatformpacket
moduleandtheConnectionmanager.Theplatformpacketmodule
providesbasicconnectionlessunreliablepacketdelivery,
currentlyimplementedusingstandardUDPsockets.TheConnection
managerprovidesavirtualconnectionbetweentwohostsandwhile
itdoesnotprovidedeliveryguarantees,itdoesprovidepacket
deliverystatusnotifications.
Thisnotificationguaranteeisveryimportanttothe
architecture;apacketlayerthatsupportsonlyguaranteedor
nonguaranteedpacketswouldnotbesufficienttosupportthe
fourbasicdatadeliverymodalitiesoutlinedabove.Howeach
deliverymodeishandledisdelegatedtoahigherlevelthe
connectionmanageronlyguaranteesthecorrectnotificationofa
sentpacket'sstatus.I.e.,ifapacketisnotifiedasdroppedit
waseitherdroppedordeliveredoutoforder(andsubsequently
dropped),andifapacketisnotifiedasdelivered,ithasbeen
delivered.
TheConnectionmanagernotifiestheStreamlayerofthestatusof
eachpacketintheorderthattheyweresent.Anoverviewofthis
relationshipisshowninFigure2.Droppedpacketsareneverre
transmittedbytheConnectionmanager;theStreamlayer,andits
associatedmanagers,handlealldataguaranteemechanisms.Since
packetsareneverretransmitted,theyarefreedimmediatelyafter
transmission.

StreamManager

Packet
Data

PacketNotifyEvent
DeliveryStatus

Packet
Data

ConnectionManager

Packet
Address
Data

Packet
Address
Data

ConnectionLayer

PlatformPacketModule
Figure2

TheConnectionmanageremploysaslidingwindowprotocolinorder
totrackthedeliveryofpackets.Whenthewindowisfull,
transmissionstopsuntilanAcknowledgmentisreceived.
Acknowledgmentofpacketsisonlyusedtoadvancethewindowand
generatenotificationevents.Thisprotocolimposesanaverage
overheadof3bytesperpacket.
Thoughnotpartoftheprotocol,animportantfeatureofthe
architectureisbitpackingprovidedbyacustombitstream
class.Thisclassprovidesbitlevelreadandwritefunctions,
includingread/writefunctionsfor:asinglebit,variablelength
integers,variablelengthnormalizedfloats,andHuffman
compressedstrings.Allpacketdata,includingtheheaderandthe
slidingwindowprotocol,areaccessedthroughthisstream.These
packingfeaturesareusedextensivelyandvirtuallyalldatais
transmittedusingthesmallestnumberofbitspossible.
Examplesofbitpackingusingthebitstream.
Writingintothepacket:
if(stream>writeBool(updateDamage)){
//Uses1bit
stream>writeInt(mDamageState,2); //Uses2bits
if(mDamageState!=Dead)
stream>writeInt(mDamageLevel,6);
if(stream>writeBool(mRepairActive))

stream>writeInt(mRepairRate,4);
}
Thematchingreadmethod:
if(stream>readBool()){
mDamageState=stream>readInt(2);
if(mDamageState!=Dead)
mDamageLevel=stream>readInt(6);
mRepairActive=stream>readBool();
if(mRepairActive)
mRepairRate=stream>readInt(4);
}
StreamLayer
TheStreamlayeriscomprisedofaStreammanagerandtheEvent,
Ghost,Move,DatablockandStringmanagers.TheConnection
managerdealswiththefrequencyofpackettransmissionaswell
aspacketsizeandstreammanagerordering.Theindividual
streammanagersdealwiththepackinganddeliveringofdata
includingthevariousguaranteemechanisms.TheStringmanager,
dealingprimarilywiththecompressionofstringdata,isnot
coveredinthisarticle.
TheStreammanagerallocatesandtransmitspacketstoits
counterpartonaremotehost.Tocontrolbandwidth,eachStream
managerhasapacketupdaterateandsize.Theseparametersare
setbytheremotehostsStreammanagerandrepresenttheamount
ofdataitscapableofreceiving.Clientsconnectingtoa
dedicatedserversetthesevaluestorepresentthebandwidthof
theirInternetconnection.IfaclientisconnectingtoanISP
witha28.8modemthenitcouldsetarateof10packetsper
secondwithasizeof200toproduceabout2Kofdatapersecond.
Theclientmaychangetheseparametersontheflyinresponseto
changesinlinequality.Tocontrolitsownoutgoingbandwidth,
theserverimposesamaximumbandwidthperclientbasedonits
ownconnectionquality.
PacketsareallocatedbytheStreammanagerandfilledbythe
Move,Event,andGhostmanagers.Sincetheopportunitytowrite
intothestreamisgiventothemanagersinafixedorder,this
orderformsafixedpriorityamongthem.Oncetherequested
packetsizeisexceededorallthemanagersaredone,thepacket
ishandedofftotheConnectionmanagerfortransmission.Figure
3showsanexamplepacket.

Header

StreamManagerData

Move

MoveManagerData

Packet1
Event1
Event2

EventManagerData

Event3
GhostState1

GhostManagerData

GhostState2
Figure3:ExamplePacket
WhenapacketisdeliveredtotheStreamlayerforreading,it
simplyhandsoffthepackettoeachstreammanagerinthesame
fixedorderandeachmanagerisresponsibleforreadingthedata
writtenbyitscounterpart.
TheStreammanageralsoprovidesaTransmissionRecordforeach
packetthatitconstructs(showninFigure4).Whenastream
managerstoresdataintoapacket,itstoresinformation
regardingthatdataintheTransmissionRecord.Whena
ConnectionmanagerNotifyEventoccursforapacket,the
TransmissionRecordforthatpacketisprocessedbytheStream,
EventandGhostmanagers,andisusedtoprovidedelivery
guarantees.SincetheConnectionmanageralwaysproducesNotify
Eventsforeachpacketintheordertheyweresent,the
TransmissionRecordsarestoredinasimpleFIFO.
EventStreamManager
TheEventStreammanagerisresponsibleforprovidingguaranteed
andnonguaranteeddeliveryofeventobjectsfromonehostto
another.Guaranteedeventsarealsoguaranteedtoprocessinthe
ordertheyweresent.Aslidingwindowisusedtotrackthe
statusofevents.Allwindowmanagementisperformedusingthe
TransmissionRecords.
WhentheEventmanagerisgivenapackettowriteinto,itpops
eventsoffitsoutgoingqueueandwritesthemintothestream
untileitherthepacketsizeisexceeded,thequeueisempty,or
theeventwindowisfull.Eventsarepersistentobjectsandare
writtenusingthemethodsdiscussedearlier.
Onceeventsarewrittenintoastream,theyarelinkedtogether
andattachedtotheStreammanagersTransmissionRecord.When

theStreammanagerisnotifiedofapacketsstatus,the
TransmissionRecordforthatpacketispassedtotheEvent
manager.IftheConnectionmanagersignalsthesuccessful
deliveryofthepacketthentheeventsattachedtotherecordare
deletedandtheslidingwindowupdated.IftheConnectionmanager
signalsnondeliverythentheeventsassociatedwiththelost
packetaresimplypushedontotheheadoftheeventqueueforre
transmission.

Transmission
Record1
NextRecord
EventList
GhostList

Event1
NextEvent

Event2
NextEvent

TR2

TR3

Event3
NextEvent

Figure4:Transmissionqueueincludingrecordwith
events
Whenthemanagerisgivenapacketstreamtoread,itunpacksand
constructsalltheevents.Iftheeventismarkedasguaranteed,
itisaddedtoanorderedqueueusedtoprovideordered
processing.Processingeitherhappensimmediatelyuponreceipt
(fornonguaranteedevents)orastheorderedqueueisadvanced.
SinceneitherthetransmittingnorthereceivingEventmanagers
storedeliverystatusinformationintothepacketstream,this
protocolimposesverylittleoverheadabovethatalreadyimposed
bytheConnectionmanager,normally3bitsperpacketand1bit
perevent.Ifapacketisdroppedanadditional714bitsper
packetmaybewritten.
Inpacking,deliveringandguaranteeingEventobjects,theEvent
managerprovidesafundamentalserviceusedbymanyother
subsystemsintheTRIBESIIengine.Sinceeventsarepacked,
unpackedandprocessedusingvirtualfunctionsandthePersist
manager,theEventmanageritselfhasnoknowledgeofthetypeor
contentsoftheseevents.
Anexampleofasimpleevent:
classSignal:publicEvent{
enum{

SignalBits=4,
};
U8signal;
public:
Signal(U8s){
guaranteed=true;
signal=s;
}
voidpackData(Bitstream*stream){
stream>writeInt(signal,SignalBits);
}
voidunpackData(Bitsream*stream){
signal=stream>readInt(SignalBits);
}
voidprocess(NetConnection*ps){
printf("Recievedsignal%dfromhost%s",
signal,ps>getObjectName());
}
};
GhostStreamManager
TheGhostmanagerprovidestwokeyfunctions:the"ghosting"of
objectsfromonehosttoanother,andthetransferringofstate
informationbetweentheoriginalobjectanditsghost.Aghost
isacopyofanobjectpersistedandtransmittedtoaremote
host.AnobjectmayonlyhaveoneghostperGhostmanager(and
thusperremotehost),butmaybeghostedbyseveralGhost
managersatonce(todifferentclients,forexample).Ghostsare
createdusingaformofguaranteeddeliveryalsousedtosupport
partialstateupdatesbetweenanobjectanditsghost.State
dataisconsideredvolatileandtransferredusinga"MostRecent
State"algorithm.
Sinceghostinganobjectinvolvesnetworkoverhead,theghost
managerdoesnotghostallobjectsinthesimulation,butinstead
hasaconceptof"scope."Objectsmaycomeinandoutofscope
foramanagerforanumberofdifferentreasons(thisprocessis
managedintheSimulationlayer).Whenanobjectcomesinto
scope,itsghostistransferredtotheremotehost;whenan
objectgoesoutofscopeitsghostisdeleted.Whileanobject
isinscope,statedataistransferredbetweentheobjectandits
ghostandisupdatedataratebasedonitspriorityandstate
mask.Themanageralsosupportsthenotionof"ghostalways"
objects,whicharealwaysinscope.
Whenanewobjectcomesintoscope,theobjectistaggedwitha
GhostRecordcontainingaGhostIDandaStateMask.TheGhostID
isassignedfromalimitedrangeandisusedbythelocalmanager
toaddressghostobjectsontheremotehost.TheremoteGhost
managermaintainsadictionary,whichtranslatesGhostIDsinto
localghostobjects.Whenthemanagertransfersinformationfrom
anobjecttoitsghost,itsGhostIDisembeddedinthestreamso
thattheremotemanagercanproperlydeliverthedata.TheGhost

RecordsStateMaskrepresentsstatedatathatanobjectis
interestedintransferringandistheheartofthe"MostRecent
State"algorithm.EachbitintheStateMaskrepresentsaclass
dependantsetofrelateddata,orstate,thatwillbe
transferred.Anobjectmayrepresentitspositionandvelocityas
onestatebit,changesinrotationasanother,andpossiblya
changeinanimationstateasathird.(TRIBESsimulationobjects
typicallyhaveupwardsof20stateflags.)Eachstateistracked
andtransferredindependentlyoftheothers,providingthe
abilitytoperformpartialupdatesofanobjectstotalstate.
Whentheghostmanagerisgivenapackettofillbythestream
manager,itperformstwobasicactions.First,itbuildsan
updatelistwhichincludeseveryobjectwithastatuschangeor
nonzeroStateMask.Thisupdatelistisorderedfirstbystatus
change,thenbyobjectpriority.Statuschangesincludethe
transferringordeletingofghostsfromtheremotehost;an
object'spriorityisavalueassignedbytheSimulationlayeras
partofthescopingprocess.Next,theupdatelististraversed
inorderwritingtheGhostID,statusandobjectstate
informationintothepacketuntilthepacketisfilledorthe
listisempty.Foreachobjectthatcontainsdatainthestream,
atransmissionstructureisconstructedcontainingthestatus
changerequestedandtheStateMask.Thisstructureisattached
totheTransmissionRecordforthepacketasshowninFigure5.
TheStateMaskbitsintheTransmissionstructurerepresentthe
statedatawrittenintothestreambytheobject.

Transmission
Record1
NextRecord
EventList
GhostList
TR2

Object1GTR1
NextRecord
StateMask[111]
NextObjectTR

Object2GTR2
NextRecord
StateMask[101]
NextObjectTR

Object1GTR3
NextRecord
StateMask[010]
NextObjectTR

TR3

Figure5:Transmissionqueueincludingghosttransmissionrecordsfor
objects1&2

Whenaghostmanagerreceivesapacket,itreadseachsetof
GhostIDandstatusflagsinorder.Ifastatusindicatesanew
ghost,thePersistmanageriscalledtoconstructtheobjectand
thenewghostisenteredintotheGhostIDdictionary.Ifthe
statuschangeisadeletionrequest,theghostobjectisobtained

fromthedictionaryanddeleted.Ifthereisnostatuschange,
thepacketcontainsonlystatedata.Theghostobjectis
obtainedfromtheIDdictionaryanditsunpackmethodiscalled
withthecurrentpacketbitstream.Whichstatemaskbitswere
usedbythesourceobjecttopackdataisnottransmittedtothe
ghost.Theobjectisresponsibleforencodingsuchinformation
intothebitstream.
Theguaranteeofobjectstateinformationhappensasfollows;
StateMaskbitsrepresentstatechanges,soifanobject's
positionchanges,itsetsits"positionstate"bitintheState
Mask.Iftherearemultiplemanagersghostingthisobject,the
bitissetintheGhostRecordrelatedtoeachmanager.Thisset
bitnowrepresentsstatedatathattheGhostmanagerneedsto
transfertotheobject'sghost.Whenthemanagerisfillinga
packet,objectswithanonzeroStateMasksareaskedtowrite
intothepacketstreamgiventhecurrentmaskforthatGhost
manager.Intheexampleofthe"positionstate",theobject
writesitscurrentposition.AtthispointtheStateMask
representswhichstateswerewrittenintothepacket,andthe
maskisstoredinatransmissionstructurewhichislinkedtothe
TransmissionRecord.Oncethisisdone,theStateMaskis
cleared.Ifthepacketisnotifiedaslost,thentheStateMask
foreachobjectwhichhaddatainthatpacketisupdatedto
includelostbits.Statebitsareonlyconsideredlostifno
subsequentlysentpacketcontainedthesamestatebitforthat
object.Sinceweguaranteeonlythelateststate,ifalater
packethasalreadybeensentwithalaterversionofthatstate,
thenthisrequirementhasbeenmetandnootheractiontakes
place.
InFigure5,ifthepacketforTransmissionRecord1islostthen
bits[101]forobject1and2areconsideredlost;whereasbit
[010]forobject1isnot,becauseGTR3,alaterupdate,has
thatbitset.Ifalostbitisaddedtoanobject'sStateMask,
thenthatstatewillgetretransmittedatthenextopportunity.
Statuschanges,suchasobjectconstructionanddeletion,are
handledinasimilarfashion.
Exampleofstatemasks:
Anobject'spositionchangesanditsetsitsPositionState
bittotrue.Theghostmanagerdecidestopacktheobject's
stateandcallstheobjectspackmethodwiththestate
mask.Theobjectseesthatthemanagerwantstotransmit
itspositionandwritesitscurrentpositionintothe
packetbitstream.Themanagerstoresthestatemask
containingthePositionStatebitinthenotification
structureforthatpacketandtheStateMaskisthen
cleared.
Firstpossibility:thepacketisdeliveredandthestate
hasbeensuccessfullytransferred.Theobject'sposition
hasnotchangedsincethepacketwassentandtheState
Maskisstill0sonofurtheractiontakesplace.

Secondpossibility:thepacketisnotdeliveredandinthe
meantimetheobject'spositionhasnotchanged.Thepacket
isnotifiedaslost.Sincenopacketshavebeensentwith
newPositionStatedatafortheobject,thePositionState
bitissetintheobjectsStateMask.Inthiscasethe
samepositionwillgetresentthenexttimetheobjectis
packed.
Thirdpossibility:thepacketisnotdelivered,butinthe
meantime,theobject'spositionhaschanged.Before
notificationofnondelivery,anotherpacketwasalready
sentwiththelatestposition.Inthiscase,themanager
knowsthatPositionStatedatahasalreadybeen
retransmittedandthelossofthestateisignored.Thisis
possiblebecausetheGhostmanageronlyguaranteesthatthe
latestversionofthestatedataistransferredandnotany
particularversionorcopyofthatstate.Inthisexample,
thisisexactlythebehaviorwewanttheposition
informationcontainedinthelostpacketisstaleandwe
don'twanttoretransmitit.Unlessthesecondpacketis
notifiedaslost,wealsodon'twanttoretransmitthe
currentposition,asithasn'tchangedsincethesecond
packetwassent.
Fourthpossibility:thepacketisnotdelivered,andinthe
meantime,theobject'spositionhaschanged.Butthe
positionchangewasrecentandnosecondpackethasbeen
sentyet.Sincenonewpacketshavebeensentwith
PositionStatedatafortheobject,theobject'sPosition
StatebitissetbytheGhostmanager.Theobjectwill
havealreadysetthebititself,sothiswillnotchange
theobjectsbehavior.Thenexttimetheobjectispacked
itwillwriteitscurrentposition,notthepositionsent
inthelostpacket.Similartothethirdpossibility,the
positiondatalostinthedroppedpacketisneverresent.
LiketheConnectionmanager,Event,andotherstreammanagers,a
serverwillhaveaGhostmanagerforeachclientconnectedtoit.
IfthereareseveralGhostmanagersghostinganobject,the
objectwillhaveaGhostRecordforeachmanager.Eachmanager
willhaveadifferentGhostIDandStateMaskforthatobject
representingthestateofthatobjectonitsremotehost.When
anobjectsetsoneofitsstatebits,itupdatestheStateMask
foreveryGhostmanagercurrentlyghostingitandeachmanager
tracksandupdatesstateinformationindependently.
Thisprocessoftrackingstatemasksandguaranteeingthe
deliveryofstateinformationbetweenobjectandtheirghostsis
akeyfeatureofTRIBESnetworking.Gamesimulationobjectsrely
almostexclusivelyonthismechanismtotransferinformation.

MoveStreamManager
TheMovemanagerguarantees"soonestpossible"deliveryofclient
inputmovestotheserverand"soonestpossible"deliveryof
controlobjectstatedata.Aslidingwindowisusedtotrackthe
deliveryofmovesandalsotosynchronizemoveprocessingbetween
controlobjectsandtheirghosts.Thismanagerisasymmetricalin
thatmovesareonlysentfroma"client"connectiontoa"server"
connectionandControlObjectstatedatafromserverobjectto
clientghost.
Inputmovesareusedtocontrolsimulationobjectssuchas
vehicles,cameras,andplayers.Inputisgatheredontheclient
byanInputManager,whichcollectsamoveevery32milliseconds.
Movesconsistofx,yandztranslations,yaw,pitchandroll
rotationsaswellasanarrayoftriggerstates.Thesemovesare
deliveredtoobjectsbytheMovemanagerandarethesolemeansof
usercontrolledobjectmovement.
TheMovemanager,similartotheEventandGhostmanagers,writes
informationintoeverypacketstreamallocatedbytheStream
manager.Unliketheothermanagers,itmakesnouseofthe
TransmissionRecordortheConnectionmanager'snotification
system.Instead,theMovemanagerprovides"soonestpossible"
deliveryofmovestotheserverbywritingmovesintoeverypacket
sent.Eachmoveistransmittedthreetimesinconsecutive
packets.Eachpackettransmittedfromtheservercontainsthe
lastmovereceived,whichtheclientusestoadvanceasliding
windowofoutstandingmoves.Ifthewindowbecomesfull,the
clientsimulationishalteduntiltheserveradvancesthewindow.
If3packetsaredroppedinarowthenanymovesuniquetothose
packetsarelost.
InStarsiegeTRIBES,everymoveinthecurrentwindowwas
transmittedineverypacketguaranteeing100%deliveryofall
moves,butthiswassometimesthecauseofanegativefeedback
loop.Adroppedpacketwouldincreasethesizeofthenextpacket
bywideningtheslidingwindowandcausingmoremovestobe
transmitted.OnlowbandwidthInternetconnectionstheincreased
packetsizecouldexceedtheconnection'sbandwidth,causingmore
packetstobedropped.Thoughthismeansmovesarenolonger
guaranteeddelivery,theclientrespondsbetterwhenlargenumbers
ofpacketsaredropped.Losingmovescancausetheclientand
servercontrolobjecttogetoutofsyncwhichmayproducea
momentarywarpingfortheclient.
TheControlObjectforaclientistheobjectthatreceivesthe
currentmovebeingproduced.BoththeControlObjectonthe
serveranditsghostontheclientreceiveeverymove.Control
Objectshavetwomajorrequirementsthatdifferentiatethemfrom
otherobjects.Thefirstisthedeterministicprocessingof
moves.Boththeoriginalserverobjectanditsghostaregiven
movesgeneratedbytheplayerandboth,giventhesamestarting
state,mustproducethesameendstate.Thesecondisthatthe
controlobjectmustbeabletotransferitscurrentstateon

demand.ThoughControlObjectsareghosted,andthuscantransfer
informationusingtheghostStateMask,theMovemanagerprovides
aseparate"Soonestpossible"deliveryofaControlObject'sstate
toitsghost.Thisstateinformationissimilarbutnotidentical
toinformationtransmittedthroughtheGhostmanager.State
informationsenttoaghostisnormallylimitedtothatneededto
rendertheobject,ortoreasonablypredictitsmotion.Control
Objectstateinformationmustincludeeverythingneededto
deterministicallyprocessinputmoves."Soonestpossible"
deliveryofthiscontrolstateinformationisachievedbyhaving
theControlObjectwriteitscurrentstateintoeverypacketsent
fromtheserver.Thiscombinationofstatetransferand
deterministicmoveprocessingisusedtokeeptheControlObject
anditsghostsynchronized,andtoallowtheclienttopredictthe
motionofthecontrolobjectaheadofserverupdates.
DeliveringmovesandControlObjectstateupdatesassoonas
possiblealongwithclientpredictionofmoveprocessingallows
smoothandimmediateresponsetoplayerinput,whileproviding
fullvalidationofallplayermovesbytheserver.
DatablockStreamManager
TheDatablockmanagerprovides"lateststate"deliveryof
datablocksfromtheservertotheclient.Datablocksareobjects
whichcontainrelativelystaticdataandthemanagerguarantees
deliveryonlyofthelateststateataverylowfrequency.Like
theGhostmanager,theDatablockmanagerisasymmetric,and
datablocksareonlytransmittedfromservertoclient.Themost
commonuseofdatablocksistotransmitinitializationor
referencedataforghosts.
Datablocksarecopiedtotheclientinastreamofguaranteed
events.Sincedatablocksareconsideredstatictheyarenot
updatedduringthenormalcourseofasession,butonlyatfixed
timessuchaswhenfirstconnectingtoaserverorduringamap
change.Agloballastmodifiedkeyisusedtotrackchangesto
datablocks.Eachtimeadatablockismodifieditisassignedthe
currentvalueofthiskey,afterwhichthekeyisincremented.
Themanagerkeepsthehighestkeyvaluetransmittedtoitsremote
host.Sincethemanageronlyguarantees"lateststate",ifa
Datablockismodifiedseveraltimesbetweenupdates,this
informationislosttotheclient.Further,sincethechangesto
datablocksarelowinfrequency,theGhostmanagersoverheadof
partialstateupdatesisnotnecessaryachangeddatablockis
retransmittedinitsentirety.
UnliketheotherConnectionstreammanagers,theDatablock
managerdoesnotwriteinto,orreadfromthepacketbitstream
producedbytheStreammanager.Updatesareinvokeddirectly
fromthescriptinglanguageandinvolveloopingthroughallthe
existingdatablocksandtransmittingeverydatablockwitha
modifiedkeygreaterthanmanager'skey.Theactualtransferof
datablocksisperformedbysendingguaranteedDatablockEvents

usingtheEventmanager.ADatablockEventcontainsthedatablock
objecttotransmit,andthedatablockispackedaspartofthe
event.
Asanexample,theVehicleclasshasanassociatedVehicleData
Datablockclass.TheDatablockclasscontainsnumerousvehicle
propertyattributessuchasshape,maximumspeed,maximum
acceleration,suspensionpropertiesforwheels,etc.Whena
Vehicleobjectisconstructed,itisassignedaninstanceofthe
VehicleDataDatablock.EachinstanceoftheVehicleData
Datablockrepresentsadifferentsetofattributesdescribinga
differenttypeofvehiclesuchasatank,scout,transport,etc.
TheVehicleclassitselfrepresentstheinstanceofthatvehicle
typeinthesimulationandprovidesallthecodeanddynamic
attributesnecessarytosimulatethevehicle.

VehicleDataDatablock
Instancewithstatic
vehicleattributes
Server

VehicleDataDatablock
Copyofstaticvehicle
attributes
Client

VehicleClassObject
Datablock
Dynamicvehicle
attributes

VehicleClassGhost
Datablock
Dynamicvehicle
attributes

Figure5:VehicleDatablockrelationshipmirroredonclient

Theadvantageofthisseparationofstaticattributedataand
instancedataisthatDatablocksarenormallyonlytransmittedto
theclientonceatconnecttime.WhenaVehicleobjectis
ghostedtotheclient(aprocessthatcanhappenoften)itsends
onlyitsdatablock'sIDalongwithotherdynamicdatarelevantto
theghost.Thisresultsinhugesavingsinbandwidth,whilestill
providingtheflexibilityofdefiningstaticattributedataon
theserver.AsshowninFigure5,theclientwillmirrorthe
samerelationshipofVehicleinstancetoVehicleDatadatablock.
Sincedatablocksareonlytransferredattimeswhenthe
simulationiseffectivelystopped,morebandwidthisavailable
anddatablockscancontainmoreinformationthanwouldnormally
bepossible.ThisalsomeanstheDatablockmanagerimposesno
extraoverheadtothebitstreamduringthenormalcourseof
simulation.

SimulationLayer
TheSimulationlayerperformsanumberoffunctionsofwhichonly
afewarerelevanttoadiscussionofthenetworkingmodel.
Thesearetheadvancementoftime,scopingfortheGhostmanager,
andclientsideprediction.
Advancementofsimulationtimeiscenteredontheprocessingof
moves.Movesaregatheredevery32milliseconds,andthe
simulationisadvancedin32millisecondincrementswithamove
foreachincrementortick.Allobjectsontheclientandserver
areadvancedinthesamemannerandifaMovemanagerisnot
controllinganobject,thenitispassedNULLmoves.
Thoughclientobjectsareadvancedinfixedincrements,the
elapsedtimebetweenframesmaynotfallonevenincrements.To
presentasmoothview,especiallyonamachinewithahighframe
rate,timeisalwaysadvancedtothebeginningofthenext
interval,andghostobjectsmaintainbackwardsinterpolation
informationwhichisusedto"tween"framesbetweenticks.
Objectsontheserverareneverrenderedsotheydonothaveto
performthisfunction.
Simulationobjectsmustbescopedandprioritizedfortheghost
manager.Scopingistheprocessofdeterminingwhichobjectson
theserverarerelevanttoaparticularclient.Inthesimplest
sense,thismeansallobjectsthatarepotentiallyvisibletoa
clientfromthatclient'scurrentcontrolobject.Thisformof
scopingisperformedusingaspatialdatabasemaintainedbythe
simulation.Whenanobjectisdeterminedtobeinscope,the
objectitselfdeterminesitsownprioritybasedoninformation
suchascurrentghoststatemask,distancefromthescoping
object,projectedradius(usingthescopingobjectsview
frustrumparameters),relativevelocity,animationstateand
interestmodifiers(projectilesthataremovingtowardsthe
clientaremoreinterestingthanvehicles,vehiclesaremore
interestingthanitems,etc.).
Therearetwoformsofclientsidepredictioninthismodel.
FirstistheControlObjectmovepredictionperformedbytheMove
manager.Thisisbyfarthemorecomplicatedandprovidesthe
foundationforcontrollingandpredictingobjectssuchas
players,vehicles,turretsandcameras.Theotherisprediction
performedbynoncontrolledghostobjectssuchasprojectiles,
items,etc.Thislevelofpredictionisleftentirelytothe
objectsthemselvesandnosupportingstructureisprovided.
AnexampleofapredictionstrategyisthatprovidedbytheItem
class,oneofthesimplerobjects.Thisobjectisdriven
entirelybythephysicsoftheenvironmentanddoesnotprocess
inputmoves.Bysimplysimulatingnormallytheghostwill
correctlypredictinteractionwithallstaticobjects.Ifa
dynamicobjectishitontheserver,thenitsghoststatemaskis
settotransferitslatestpositionandvelocity.Whenthe

clientghostreceivesanupdatefromtheserver,itinterpolates
itscurrentpositionsmoothlytotheserver'supdatedposition
andcontinuesfromthere.Morecomplexstrategiesareusedby
thevehicleandplayerclasses,whichhavetodealwith
predictingmovementfromotherclients.
Summary
Byclassifyingdataaccordingtoitsdeliveryrequirementsand
organizingthestreammanagerstoefficientlymeetthose
requirements,theTribesIInetworkingmodelattemptstomake
optimaluseofavailablebandwidth.Incombinationwithdynamic
objectscopingandprioritizationofpartialobjectstate
updates,theTRIBESIIengineallowssmoothvisualizationof
largevirtualenvironmentsoverlowbandwidthconnections,even
whenasubstantialnumberofobjectsarevisibletoaclient.
Thoughthemodelprovidesasolidfoundationfornetworktraffic,
itisnotsufficienttosimplyprovideafastreliableconnection
betweenhosts;therewillalwaysbebandwidthandlatencyissues
tocontendwith.Notcoveredinthisarticleare:choicesmade
whendecidingwhichobjectbehaviorshouldbemanagedbythe
serverandwhichbytheclients,whichobjectstateinformation
totransfer,howtoorganizeobjectpartialstates,howto
prioritizeobjectstatechangesrelativetoeachother,which
compressionalgorithmtouse,andhowtoperformclientside
predictionofControlObjects.Allofthesecanhaveadramatic
effectonperceivednetworkperformance.
ThismodelwasfirstimplementedinStarsiegeTRIBES,which
shippedinDecemberof98.TheupdatedimplementationinTRIBES
IIhasthefollowingimprovements:
1. ReorganizationandsimplificationoftheConnectionandStream
layers.
2. BetterpacketheadercompressionfortheConnectionmanagers
notificationprotocol.
3. Areductioninthenumberoftimesmovesareretransmittedto
avoidnegativefeedbackloopsduetoincreasedpacketsize.
4. Betterdatacompressionbyindividualobjects.
5. TheadditionoftheStringmanager,whichprovideshighorder
compressionforrepeatedlytransmittedcharacterstrings(not
discussedinthisarticle).
Severalmodificationstothemodelhavebeenconsidered,and
thoughtheywillnotbeimplementedforTRIBESII,areworth
exploring.Theseinclude:
1. VariableproductionofpacketsbytheStreammanager.Updates
arecurrentlyproducedataconstantratedictatedbythe
Streammanagersbandwidthsettings.Packetscouldbe
producedassoonasdataisavailablefortransmissionordata
priorityreachesagiventhreshold.
2. AsynchronousConnectionprocessingontheserver.All

Connectionmanagersarecurrentlyprocessedatthesametime
ontheserverinasinglethread.Theserverarchitecture
couldbechangedtosupportaprimarysimulationthreadwitha
separatethreadforeachclient.Thiswouldprovidethe
ConnectionandStreammanagerswithlowerlatency
communicationwiththeirremotehostaswellasprovidebetter
supportformultiprocessorhardware.
3. Theadditionofnewstreammanagers,includingamultiplexing
audiostreammanager.TRIBESIIwillsupportvoiceusinga
simplerapproach.
4. Dynamicallocationofserverbandwidth.Bandwidthonthe
serveriscurrentlysetatafixedperclient.Theserver
couldbemodifiedtoadjustitsperclientbandwidthbasedon
thenumberofconnectedclients.
5. Determineamethodofautomaticallyandeffectively
calculatingtheavailablebandwidthonaconnection.
Bandwidthsettingsarecurrentlysetmanuallybyaclient.
Acknowledgments
ThoughseveraloftheideaspresentedhereareuniquetoTRIBES,
othershavebeenassimilatedfromothersourcesovercountless
yearsofreadingandtheauthors,beingtoolazytoresearch
theirsources,wouldliketothankeveryoneinonesentence.