DomainSpecificLanguages

AdvancedOperatingSystems Ad dO ti S t (263380000)

TimothyRoscoe Thursday 9thDecember 2010

Systems Group Department of Computer Science ETH Zrich

Outline
Introduction Interfacedefinitionlanguages H d Hardwareinterfacelanguages i f l FiletofFish Hamlet References

Theproblem The problem

C is a pain to write OS code in. CisapaintowriteOScodein. 2classesofproblem:
1. Lackofautomaticresourcemgmt 1 L k f t ti t 2. Hardtoexpresshighlevelsemantics

Highlevellanguagestothe rescue?
Write your OS in Java/Eiffel/C#/etc. WriteyourOSinJava/Eiffel/C#/etc.
Hasbeentried.Severaltimes.

Problems:
Loseallcontroloverresourcemanagement Explicit layout / memory access becomes hard Explicitlayout/memoryaccessbecomeshard Stillcantexpresshighlevelsemantics
(OScodeishighlyspecialized)

Sufficientlyexpressivelanguagestooslowandtooabstract
(e.g.Haskell)

ExtendC? Extend C?
Promisingapproach: Promising approach: NesC:TinyOSs Cdialectwithsupportformodules, events[Gay2003] events [Gay 2003] Deputy:extensionstoCusingtypeinferencefor staticchecks[e.g.Anderson2009] static checks [e g Anderson 2009] Ivy:evolvingCasalanguage[Brewer2005] Sofar,littleuptake(poortoolchain support?)

Domainspecic Domain specic languages

Oldidea Verybroadapplicability(notjustOSes) GuySteele:designyoursystemasifyouweredesigning alanguageanyway. l Buildalittlelanguagetailoredforthetaskathand G GenerateCwhichisthencompiledwiththeOS t C hi h i th il d ith th OS InBarrelfish,weuseDSLs extensively(4sofar,and In Barrelfish we use DSLs extensively (4 so far and counting)

Domainspecic language workow k

DSLcode Ccode
DSL compiler

Forallourexamples:

C compiler

Binary i

Advantages
Highlyspecialized:capturetheexactsemanticsyouwant! Cancheckandenforceusefulinvariants Small,easytolearn Canbeveryfast(fasterthanaprogrammercouldwrite) C b f t (f t th ld it ) Dramaticallyreducesdevel/debugtime

Ofcourse,thereisadownside: Lotofefforttowritethecompiler Complicatestoolchain management Maymakethecodelooksomewhatalien...

ExamplesofDSLs inOperating Systems

Communication interface definition Communicationinterfacedefinition Hardwareregisteraccess S h d li Schedulingalgorithms(seenextweek!) l ih ( k!) Protocol stack design(Click,Prolac) Capabilitytypesystemspecification Error code definitions Errorcodedefinitions ...

Outline
Introduction Interfacedefinitionlanguages H d Hardwareinterfacelanguages i f l FiletofFish Hamlet References

Interfacedenition Interface denition languages

Perhaps oldest DSLs for OS development PerhapsoldestDSLsforOSdevelopment OriginalRPC[BirrellandNelson,1984]

Interfacedenition Interface denition semantics

IDLs are NOT like (Java) RMI! areNOTlike(Java)RMI! AnIDLtypicallydefinesitsowntypesystem. C Concretetypes:integers,structs,etc. i Abstracttypes:interfacereferences IDLcompilermapsthisto(perhapsmany) programminglanguages

Stubfunctionality Stub functionality

Userclientcode Cfunctioncall Transmit stub Marshalproc# M h l # Marshalargs IPCsend IPCsubsystem

Stubfunctionality Stub functionality

Userservercode

Userclientcode Cfunctioncall

CallCuserfn. Unmarshal args

Transmit stub

Marshalproc# M h l # Dispatch Marshalargs

Unmarshal proc#

Receive stub t b

IPCsend IPCreceive IPCsubsystem

MemorymanagementinIDLs Memory management in IDLs

NetworkRPCIDLs needtoworryaboutmemory y y management: E.g.CORBAdefines3parametermodes:
in:Argumentispassedfromclienttoserver(parameter) out:Argumentispassedfromservertoclient(result) inout:Argumentissenttoserver,modified,sentback

Thisisnotenoughlocally,inanOSwith:
Sharedmemorytransport Nogarbagecollection Values (like arrays) bigger than a register or machine word Values(likearrays)biggerthanaregisterormachineword

MemorymanagementinIDLs Memory management in IDLs

NetworkRPCIDLs needtoworryaboutmemory y y management: E.g.CORBAdefines3parametermodes:
in:Argumentispassedfromclienttoserver(parameter) out:Argumentispassedfromservertoclient(result) inout:Argumentissenttoserver,modified,sentback

Thisisnotenoughlocally,inanOSwith:
Sharedmemorytransport Nogarbagecollection Values (like arrays) bigger than a register or machine word Values(likearrays)biggerthanaregisterormachineword

MemorymanagementinIDLs Memory management in IDLs

Basicquestions: Basic questions: Whenshouldmemoryintheclientbefreed? H Howcanmemoryintheserverbeallocated? i h b ll d? Howcanmemoryintheclientbeallocated? Whenshouldmemoryintheserverbefreed? When is it safe to modify client data, if its been sent Whenisitsafetomodifyclientdata,ifit sbeensent totheserver?

MemorymanagementinIDLs
[HamiltonandKougiouris,1994]

TheIDLfortheSpringOSmodifiedCORBAIDLforanOS The IDL for the Spring OS modified CORBA IDL for an OS setting:
copy:Argumentiscopiedtotheserver. py g p consume:Argumentissentfromclienttoserver,and destroyedatclient. produce:Argumentisgeneratedattheserverandsentback (destroyedatserver) borrow:Likeinout,butcan tbemodifiedbyclientinthe borrow: Like inout but cant be modified by client in the meantime.

Performance
FornetworkIDLs (CORBA,ANSA,DCE,SunRPC,etc.) For network IDLs (CORBA, ANSA, DCE, SunRPC, etc.) stubperformancenotcritical
Networklatencydominates y Callsareinfrequent Callsmusttraversenetworkstackanyway

ItsverydifferentforlocalOSstubs:
IPCsystemhighlyoptimized stubperformancecritical Callsarefrequent(particularlyinamicrokernel)

Performance
FornetworkIDLs (CORBA,ANSA,DCE,SunRPC,etc.) For network IDLs (CORBA, ANSA, DCE, SunRPC, etc.) stubperformancenotcritical
Networklatencydominates y Callsareinfrequent Callsmusttraversenetworkstackanyway

ItsverydifferentforlocalOSstubs:
IPCsystemhighlyoptimized stubperformancecritical Callsarefrequent(particularlyinamicrokernel)

Flick
[Eide etal.,1997]

Optimizingstubcompiler:manytechniques,e.g.: Optimizing stub compiler: many techniques, e.g.: Marshallallfixedsizedatainonego

A id h ki b ff i Avoidcheckingbuffersizeeachtime h ti

Inlinemostoperations Usereceivebufferspaceforarguments(e.g.in) Aggregatecopiesintoonebigmemcpy Transportspecificmarshalling

e.g. L4 IPC in registers e.g.L4IPCinregisters

Flickperformance
[Eide etal.,1997]

Showseffectofcompileroptimizationsonmarshallingcode: p p g

Generalpattern General pattern

Consideradomainspecificlanguagewhere: Consider a domain specific language where: Yourewritingthesameboilerplatecodeagainand again(withminorvariations) again (with minor variations) Itseasytomakemistakes Interoperability(commonspecifications)areuseful Itsclearwhatthecompilershoulddo Compileroptimizationswouldbeuseful ...oratleastsomeoftheabove. or at least some of the above

Outline
Introduction Interfacedefinitionlanguages H d Hardwareinterfacelanguages i f l FiletofFish Hamlet References

Hardwareregisteraccess Hardware register access

Accessinghardwareregistersisgenerallyfiddlycode Lotsofbitmanipulation(registershavemanyfields) PoorCsupport
wordsize,signextension,volatilesemantics bitfield structs areimplementationspecific!

Consequencesoferrorsarebad
Veryhardtofindbugs Frequently hangs entire machine Frequentlyhangsentiremachine

Ccodetomanipulateregistersistedioustowrite

DevilExample:LogitechBusmouse
[Mrillon etal.,2000]

Handwritten macros: Hand writtenmacros:

DevilExample:LogitechBusmouse
[Mrillon etal.,2000]

Handwritten macros: Hand writtenmacros:

Programmerusageidioms:

DevilExample:LogitechBusmouse
[Mrillon etal.,2000] DevicespecifiedintheDevilDSL: p

DevilExample:LogitechBusmouse
[Mrillon etal.,2000]

WhatsDevilgenerating?

DevilExample:LogitechBusmouse
[Mrillon etal.,2000]

Whattheprogrammergetstowrite:

OtherDevilfeatures Other Devil features

Pre and postconditions Pre andpost conditions
E.g.indexregistersusedtoaccessotherregisterbanks Semaphores which must be held before writing a register Semaphoreswhichmustbeheldbeforewritingaregister

Variables
valueswhichcombinations(usuallyconcatenations)of registervalues

Mackerel
Barrelshs answertoDevil
Things havechangessomewhatinthemeantime: g g
Lotsofaddressspace Indexregistersarelessfrequent
preconditionslessimportant Registeraddressspacesmoreuseful(PCI,memory,IO)

Registersarewider(32or64bits)
meaningfulvaluesrarelysplitacrosshardwarefields

Most complex devices communicate using Mostcomplexdevicescommunicateusing descriptorrings

Inmemorydatastructuresarejustasimportantasregisters

Mackerelfeatures Mackerel features

Goal:specificationsshouldbeasclosetodatasheet descriptionsaspossible. Basic constructs specify: Basicconstructsspecify:
Individualregisters Registertypes Registerarrays R i Inmemorydatatypes Collectionsofconstantvalues

MakeextensiveofCcompilerstypesystemandinlining CommentsareincorporatedinCprintf likecode Comments are incorporated in C printflike code

Mackerelfeatures Mackerel features

Mackerelgenerates: g Cconstantdefinitionsforallconstantvalues CTypedefinitionsforallregisteranddatatypes C Type definitions for all register and data types Functionstoread/writeallregisters Functionstoread/writeallregisteranddatatypefields Functions to read/write all register and data type fields Functionstosnprintf:
Register values Registervalues Datatypevalues Entiredevicestate!

Mackerel
Example:Intele1000Ethernetcontroller Fragmentshowingaregisterdefinition:
register status rw addr(base, 0x0008) "Device status" { g ( , ) fd 1 "Link full duplex configuration"; lu 1 "Link up"; lan_id 2 "LAN ID"; txoff 1 "Transmission paused"; Transmission paused ; tbimode 1 "TBI mode"; speed 2 type(linkspeed) "Link speed setting"; asdv 2 type(linkspeed) "Auto speed detection val"; phyra 1 "PHY reset asserted"; PHY asserted ; _ 8 mbz; gio_mes 1 "GIO master enable status"; _ 12; };

Mackerel:somegures Mackerel: some gures

Linesofcode(usingDavidWheeler sSLOCCount): Lines of code (using David Wheelers SLOCCount):
2359linesofHaskellfortheMackerelcompiler 1028linesofMackerelforthee1000specification 1028 lines of Mackerel for the e1000 specification 23762linesofCgeneratedfrome1000.dev

IfDSLs aresogood... If DSLs are so good...

HowcomewedontseemoreoftheminOSresearch? Quitehardtodesignagoodone
ExceptMackerel,alltheDSLs inBarrelfish weredesignedafter wehadaninitialCimplementationandunderstoodthe functionality.

Perception:theefforttoimplementDSLusually outweighsthecostofdesigning,building,and implementingit

Withyesterdaystools,thereissometruthinthis But But...

BuildingaDSL:whatdoesit take? k
DSLs are basically simple compilers: arebasicallysimplecompilers:

BuildingaDSL:whatdoesit take? k
DSLs are basically simple compilers: arebasicallysimplecompilers: 1. Parser
U d t b t di Usedtobetedioustowrite t it Gloriouseasythesedays E g combinatorbased Monadic parsing in Haskell E.g.combinator basedMonadicparsinginHaskell

BuildingaDSL:whatdoesit take? k
DSLs are basically simple compilers: arebasicallysimplecompilers: 1. Parser
U d t b t di Usedtobetedioustowrite t it Gloriouseasythesedays E g combinatorbased Monadic parsing in Haskell E.g.combinator basedMonadicparsinginHaskell

2. BackendCcodegenerator
Rather more difficult Rathermoredifficult...

WritingabackendforaDSL Writing a backend for a DSL

ThebackendtakesanASTandgeneratesCcode Basically:concatenateasetofstringsintoaCfile Better:encodesubsetofCsyntaxintofunctionalcombinators easier. easier. Butstill: Writingcodethroughalevelofindirection OnlycapturessyntaxofC,notintendedsemantics. Cantautomatetests Error prone Errorprone Annoyingtodebug Ultimately,noassuranceitworks.

FiletoFish
[Dagand etal.,2009]

FiletoFishis... ToolforwritingCcodegenerators Embedding of a subset of C in Haskell EmbeddingofasubsetofCinHaskell NotationforexpressingDSLsemantics Lib LibraryforcreatingprovablycorrectCcodefrom f ti bl tC d f semanticspecifications UsedinBarrelfish for(todate)2DSLs: Used in Barrelfish for (to date) 2 DSLs:
Fugu defineserrorcodesandanerrorstack Hamlet defines capability type system definescapabilitytypesystem

Hamlet:specifyingthecapability typesystem type system

Yes,Hamletreallyisatypeofsh

RecallthatBarrelfish usestyped,partitionedcapabilities Foreachcapability,wemustspecify: For each capability, we must specify:

Physicallayoutinmemory Whatitcanberetypedtoandfrom Validinvocationsonthecapability Whathappenswhenitispassedbetweendomains etc.

WecaptureallthisinformationinaHamletspecification.

HowFiletoFishcompilesHamlet
[Dagand etal.,2009]

HowFiletoFishcompilesHamlet
[Dagand etal.,2009]

Defining semantics instead of syntax: Definingsemanticsinsteadofsyntax:

WhatdoesFoF looklike? What does FoF look like?

For the previous example, Haskell resembles: Forthepreviousexample,Haskellresembles:
validateRetypeCode destType (srcType validTypes) = (srcType, do srcTypeV <- srcType validTypesP <- sequenceSem validTypes return (srcTypeV, (do returnc $ condition validTypesP)) where condition validTypes = fold orType false validTypes orType acc srcType = acc .||. (destType .==. srcType)

UsingQuickCheck totestDSLs
[Dagand etal.,2009]

Check randomlygenerated ASTs against semantic Checkrandomly generatedASTsagainstsemantic assertions:

AffectingtheOSdesign Affecting the OS design

Hamletmakesiteasytoaddnewcapabilitytypesto a e a es easy o add e capab y ypes o Barrelfish y yp y Ledustoencodemorefunctionalityintothetypesystem
E.g.differentcaptypesforpagetablelevels
(Onallarchitectures)

Typesystemenforcespagetablecorrectness

Can encode multiple physical address spaces etc Canencodemultiplephysicaladdressspaces,etc. Weexpecttopushfurtherfunctionalityintocapability system

Summary
Usedappropriately: pp p y Reducecodecomplexity
Though rarely if never actually evaluated Thoughrarely,ifnever,actuallyevaluated DSLs perhapsseenmoreasameanstoanend...

Reducebugs
Capture(andcheck)highlevelsemanticsofthedomain

Facilitateautomatedtestingand/orcorrectness proofs

References
Birrell,A.D.andNelson,B.J.(1984).Implementingremoteprocedure calls. ACMTrans.Comput.Syst.,2(1):3959. calls ACM Trans Comput Syst 2(1) 39 59 Dagand,P.E.,Baumann,A.,andRoscoe,T.(2009).FiletoFish:Practical andDependableDomainSpecificLanguagesforOSDevelopment.In Proc.5thWorkshoponProgrammingLanguagesandOperatingSystems Proc. 5th Workshop on Programming Languages and Operating Systems (PLOS2009). Eide,E.,Frei,K.,Ford,B.,Lepreau,J.,and Lindstrom,G.(1997).Flick:A flexible,optimizingIDLcompiler.InPLDI,pages4456. Hamilton,G.andKougiouris,P.(1994).TheSpringnucleus:Amicrokernel forobjects.Technicalreport,SunMicrosystemsLaboratories. Mrillon,F.,Rveillre,L.,Consel,C.,Marlet,R.,andMuller,G.(2000). Devil:AnIDLforhardwareprogramming.InProceedingsofthe4th il f h d i di f h h USENIXSymposiumonOperatingSystemsDesignandImplementation.

References
DavidGay,PhilipLevis,RobertvonBehren,MattWelsh,EricBrewer, andDavidCuller.2003.ThenesC l d D id C ll 2003 Th C language:Aholisticapproachto A h li ti ht networkedembeddedsystems.InProceedingsoftheACM SIGPLAN2003conferenceonProgramminglanguagedesignand implementation(PLDI'03).ACM,NewYork,NY,USA,111 p ( ) , , , , ZacharyR.Anderson,DavidGay,andMayur Naik.2009.Lightweight annotationsforcontrollingsharinginconcurrentdatastructures. InProceedingsofthe2009ACMSIGPLANconferenceon g Programminglanguagedesignandimplementation(PLDI'09) EricBrewer,JeremyCondit,BillMcCloskey,andFeng Zhou.2005. Thirtyyearsislongenough:gettingbeyondC.InProceedingsof the10thconferenceonHotTopicsinOperatingSystems Volume h h f l 10(HOTOS'05),Vol.10.USENIXAssociation,Berkeley,CA,USA,14 14.