23/08/2028, 14:12 Effective Scala Fork me on GitHub Effective Scala Marius Eriksen, Twiter Inc marius@twitter. com (@marus Table of Contents + Introduction * Formatting: Whitespace, Naming, Imports, Braces, Pattern matching, Comments + Types and Generies: Return type annotations, Variance, Type aliases, Implicits + Collections: Hierarchy, Use, Style, Performance, Java Collections : tions : ‘Returns, far loops and comprehensions, require and assert + Eunetional programming: Case classes as algebraic data types, Options, Pattern matching, Partial functions, Destructuring bindings, Laziness, Call by name, flaston ol ‘Dependency injection, Traits, Visibility, Structural typing nrror handling: Handling excepti Garbage collection lava compatibility itter’s standard Ii Eutures, Offer/Broker Acknowledgments Other languages BAH Pyecunii Introduction Scala is one of the main application programming languages used at Twitter. Much of our infrastructure is written in Seala and we have several large libraries supporting our use. While bi effective, Scala is also a large language, and our experiences have taught us to practice great care in its, application, What are its pitfalls? Which features do we embrace, which do we eschew? When do we employ “purely functional style”, and when do we avoid it? In other words: what have we found to be an effective use of the language? This guide attempts to distill our experience into short essays, providing a set of best practices. Our use of Scala is mainly for creating high volume services that form distributed systems — and our advice is thus biased — but most of the advice herein should translate naturally to other domains. This is not the law, but deviation should be well justified Scala provides many tools that enable succinct expression. Less typing is less reading, and less reading is often faster reading, and thus brevity enhances clarity. However brevity is a blunt tool that can also deliver the opposite effect: After correctness, think always of the reader Above all, program in Scala. You are not writing Java, nor Haskell, nor Python; a Seala program is unlike one written in any of these. In order to use the language effectively, you must phrase your problems in its terms. There's no use coercing a Java program into Scala, for it will be inferior in most ‘ways to its original. This is not an introduction to Scala; we assume the reader is familiar with the language. Some resources for learning Seala are: + Scala School + Learning Scala + Learning Scala in Small Bites This is a living document that will change to reflect our current “best practices,” but its core ideas are unlikely to change: Always favor readability; write generic code but not at the expensive of clarity; take advantage of simple language features that afford great power but avoid the esoteric ones (especially in the type system). Above all, be always aware of the trade offs you make. A sophisticated language requires a complex implementation, and complexity begets complexity: of reasoning, of semantics, of interaction between features, and of the understanding of your collaborators. Thus complexity is the tax of sophistication — you must always ensure that its utility exceeds its cost. itpsstwitr github. oftfectwoscala 42823/08/2028, 14:12 Effective Scala And have fun, Formatting The specifies of code formatting — so long as they are practical — are of little consequence. By definition style cannot be inherently good or bad and almost everybody differs in personal preference. However the consistent application of the same formatting rules will almost always enhance readability. A reader already familiar with a particular style does not have to grasp yet another set of local conventions, or decipher yet another corner of the language grammar. This is of particular importance to Seala, as its grammar has a high degree of overlap. One telling example is method invocation: Methods can be invoked with *.”, with whitespace, without parenthesis for nullary or unary methods, with parenthesis for these, and so on. Furthermore, the different styles of ‘method invocations expose different ambiguities in its grammar! Surely the consistent application of a carefully chosen set of formatting rules will resolve a great deal of ambiguity for both man and machine. We adhere to the Scala style guide plus the following rules. Whitespace Indent by two spaces. Try to avoid lines greater than 100 columns in length. Use one blank line between method, class, and object definitions. Naming Use short names for small scopes 4s, 48 and ks are all but expected in loops. Use longer names for larger scopes External APIs should have longer and explanatory names that confer meaning, Future.collect not Future.all. Use common abbreviations but eschew esoteric ones Everyone knows ok, ere OF deén whereas s¢r‘ is not so common. Don't rebind names for different uses Use vais Avoid using °s to overload reserved names. typ instead of -type* Use active names for operations with side effects user.activate() not user.setactive() Use descriptive names for methods that return values sre. isbeFined not sre. deFined Don't prefix getters with get ‘As per the previous rule, it's redundant: sive. count not site. getCount Don't repeat names that are already encapsulated in package or object name Prefer: object User ¢ 1 Bef eRCE amt): optontuser] to onject User ¢ def getUser(id: Int): Option(User] ) ‘They are redundant in use: User. getUser provides no more information than user.get Imports Sort import lines alphabetically ‘This makes it easy to examine visually, and is simple to automate, Use braces when importing several names from a package ‘port con.tadtter concurrent. (Broker, Offer) Use wildeards when more than six names are imported 8. import con.twitter.concurre Don't apply this blindly: some packages export too many names When using collections, qualify names by importing scata. collection. imputable and/or scala.collection.mutable itpsstwitr github. oftfectwoscala 212823/08/2028, 14:12 Effective Scala Mutable and immutable collections have dual names. Qualifiying the names makes is obvious to ‘the reader which variant is being used (e.g. "snmutable.¥ap") Do not use relative imports from other packages ‘Avoid Anport con.twitter Apert concuerent in favor of the unambiguous port con-twitter. concurrent Put imports at the top of the file ‘The reader can refer to all imports in one place. Braces Braces are used to create compound expressions (they serve other uses in the “module language”), where the value of the compound expression is the last expression in the list. Avoid using braces for simple expressions; write def square(x: Tot) = x*x but not det square(x: Tot) = { even though it may be tempting to distinguish the method body syntactically. The first alternative has less clutter and is easier to read. Avoid syntactical ceremony unless it clarifies. Pattern matching ‘Use pattern matching directly in function definitions whenever applicable; instead of a ? ? collapse the match ist map ( ease Sonex) => x ? it’s clear that the list items are being mapped over — the extra indirection does not elucidate. Comments Use ScalaDac to provide API documentation. Use the following style; ™ a but not the standard ScalaDoe style: I SenviceButléer butlds services yo Do not resort to ASCII art or other visual embellishments. Document APIs but do not add unnecessary comments. If you find yourself adding comments to explain the behavior of your code, ask first if it can be restructured so that it becomes obvious what it does. Prefer “obviously it works” to “it works, obviously” (with apologies to Hoare). ‘Types and Generics itpsstwitr github. oftfectwoscala 2823/08/2028, 14:12 Effective Scala ‘The primary objective of a type system is to detect programming errors. The type system effectively provides a limited form of static verification, allowing us to express certain kinds of invariants about our ‘code that the compiler can verify. Type systems provide other benefits too of course, but error checking isits Raison d'Etre. Our use of the type system should reflect this goal, but we must remain mindful of the reader: judicious use of types can serve to enhance clarity, being unduly clever only obfuscates. Scala’s powerful type system is a common source of academie exploration and exercise (eg. Type level progtamming_in Scala). While a fascinating academic topic, these techniques rarely find useful application in production code. They are to be avoided. Return type annotations While Scala allows these to be omitted, such annotations provide good documentation: this is especially important for publie methods. Where a method is not exposed and its return type obvious, omit them. ‘This is especially important when instantiating objects with mixins as the scala compiler creates singleton types for these. For example, nake in it Service dee nake() = new Service { def getTd = 123 ? does not have a return type of service; the compiler ereates the refinement type Object with Servico(det getid: Int). Instead use an explicit annotation: def nake(): Service = new servicet) ‘Now the author is free to mix in more traits without changing the public type of wake, making it easier to manage backwards compatibility. Variance Variance arises when generics are combined with subtyping. Variance defines how subtyping of the contained type relates to subtyping of the container type. Because Scala has declaration site variance annotations, authors of common libraries — especially collections — must be prolific annotators. Such annotations are important for the usability of shared code, but misapplication can be dangerous. Invariants are an advanced but necessary aspect of Seala’s typesystem, and should be used widely (and correctly) as it aids the application of subtyping, Immutable collections should be covariant. Methods that receive the contained type should “downgrade” the collection appropriately: st Collection st] { ef add[U >: T]{other: u): collectiontu; » Mutable collections should be invariant. Covariance is typically invalid with mutable collections. Consider it Hashset [oT] ( def ada[U >: T]Cltem: v) and the following type hierarchy If Tnow have a hash set of dogs val dogs: Hashset[00g] treat it as a set of Mammals and add a cat, itpsstwitr github. oftfectwoscala 412323/08/2028, 14:12 Effective Scala val mannalis: Hashset{Marmal] «= dogs Inanmals.acd(new c3E(}) This is no longer a HashSet of dogs! Type aliases Use type aliases when they provide convenient naming or clarify purpose, but do not alias types that are self-explanatory. 02 Int is clearer than type TotMaker = () => Tat Inthaxer since it is both short and uses a common type. However class ConcurrentPoolfky V] ‘ype Queue = ConcurrentLinkedqueuelV] type Hap. = ConcurrenthashMap(K, Queue] is helpful since it communicates purpose and enhances brevity. Don’t use subclassing when an alias will do. serait SocketFactory extends (Sockethddress => Socket) a socketractory isa function that produces a socket. Using a type alias is better. We may now provide function literals for values of type socketractory and also use function ‘composition: val adartotnet: Socketkddress => Long val inetTosocket: Long => Socket val factory: SecketFactory = addsTornet andthen inetTosocket ‘Type aliases are bound to toplevel names by using package objects: package con.tritter package object net { type SocketFactory = (SocketAddress) => Socket > Note that type aliases are not new types — they are equivalent to the syntactically substituting the aliased name for its type. Implicits Implicits are a powerful type system feature, but they should be used sparingly. They have complicated resolution rules and make it difficult — by simple lexical examination — to grasp what is actually happening, It's definitely OK to use implicits in the following situations: + Extending or adding a Scala-style collection + Adapting or extending an object “pimp my library” pattern) + Use to enhance type safety by providing constraint evidence + To provide type evidence (typeclassing) + For manssests Ifyou do find yourself using implicit, always ask yourself if there is a way to achieve the same thing without their help. Do not use implicits to do automatic conversions between similar datatypes (for example, converting alist to a stream); these are better done explicitly because the types have different semanties, and the reader should beware of these implications. itpsstwitr github. ioftfectwoscala 5128,23/08/2028, 14:12 Effective Scala Collections Scala has a very generic, rich, powerful, and composable collections library; collections are high level ‘and expose a large set of operations. Many collection manipulations and transformations can be expressed succinctly and readably, but careless application of these features ean often lead to the opposite result. Every Scala programmer should read the collections design document; it provides great insight and motivation for Scala collections library. Always use the simplest collection that meets your needs. Hierarchy ‘The collections library is large: in addition to an elaborate hierarchy — the root of which being Traversable[T] — there are inrutable and mutable variants for most collections. Whatever the complexity, the following diagram contains the important distinctions for both inmutabie and nutabte hierarchies lerable[T] — Seq{T] Se{T] |_| Map(T] 1serable(1] is any collection that may be iterated over, they provide an iterator method (and thus foreact). eq[t]s are collections that are ordered, set[1]s are mathematical sets (unordered collections of ‘unique items), and sap{t]s are associative arrays, also unordered. Use Prefer using immutable collections. They are applicable in most circumstances, and make programs easier to reason about since they are referentially transparent and are thus also threadsafe by default. Use the mitabte namespace explicitly. Don't import scala.cotlection utable._and refer to set, instead inport seala.colection.mutable val set = aucable.set() ‘makes it clear that the mutable variant is being used. Use the default constructor for the collection type. Whenever you need an ordered sequence (and not necessarily linked list semantics), use the Seq) constructor, and So on val seq = Seq(t, 2, 3) val set = Sex(1, 2, 3) val map = #ap(2’->'“one", 2 two", 3 => “three") This style separates the semantics of the collection from its implementation, letting the collections library use the most appropriate type: you need a sp, not necessarily a Red-Black Tree. Furthermore, these default constructors will often use specialized representations: for example, #ap() will use a 3-ficld object for maps with 3 keys. ‘The corollary to the above is: in your own methods and constructors, receive the most generic collection type appropriate. This typically boils down to one of the above: Tterapie, Sea, Set, oF Map. If your method needs a sequence, use seq(t}, not List(r]. (A word of caution: the default traversabie, reerable and Seq types in scope ~ defined in scals.package ~ are the scaia.collection versions, as opposed to Map and set ~ defined in predet. scala — which are the scala.collection. mutable versions. This means that, for example, the default seq type can be both the immutable and mutable implementations. Thus, if your method relies on a collection parameter being immutable, and you are using traversable, Tterable ‘or seq, you must specifically require/import the immutable variant, otherwise someone may pass you the mutable version.) Style Functional programming encourages pipelining transformations of an immutable collection to shape it to its desired result. This often leads to very succinct solutions, but can also be confusing to the reader itpsstwitr github. oftfectwoscala 828,23/08/2028, 14:12 Effective Scala — it is often difficult to discern the author's intent, or keep track of all the intermediate results that are only implied. For example, let's say we wanted to aggregate votes for different programming languages from a sequence of (language, num votes), showing them in order of most votes to least, we could write: val votes = Seq((*seala", 1), ("Java", 4), ("seala", 18), (*seala", 1), python", 18)) val orderedvotes = votes ‘groupey(_._1) imap { cade (which, counts) => (hich, counts foldlert(@)(. +. yet this is both succinct and correct, but nearly every reader will have a difficult time recovering the original intent of the author. A strategy that often serves to clarify is to name intermediate results and parameters: val votestyLang = votes groupty { case (lang, _) => lang } Val SunByLang = votesByLang sap { case (lang, counts ‘val countsOnly = counts map { case (_, count) => court } |, (lang, countsonly. 548) val orderedvotes = sunsyLang. tose ‘Cease (., count) => count} the code is nearly as suecinet, but much more clearly expresses both the transformations take place (by naming intermediate values), and the structure of the data being operated on (by naming parameters). Ifyou worry about namespace pollution with this style, group expressions with () val onderedvote: val votestyLang ‘ Performance High level collections libraries (as with higher level constructs generally) make reasoning about performance more difficult: the further you stray from instructing the computer directly — in other words, imperative style — the harder it is to predict the exact performance implications of a piece of code. Reasoning about correctness however, is typically easier; readability is also enhanced. With Scala the picture is further complicated by the Java runtime; Scala hides boxing/unboxing operations from you, which can incur severe performance or space penalties. Before focusing on low level details, make sure you are using a collection appropriate for your use. Make sure your datastructure doesn't have unexpected asymptotic complexity. The complexities of the various Scala collections are described here. ‘The first rule of optimizing for performance is to understand why your application is slow. Do not operate without data; profile(1) your application before proceeding. Focus first on hot loops and large data structures. Excessive focus on optimization is typically wasted effort. Remember Knuth’s maxim: “Premature optimisation is the root of all evil.” It is often appropriate to use lower level collections in situations that require better performance or space efficiency. Use arrays instead of lists for large sequences (the immutable vector collections provides a referentially transparent interface to arrays); and use buffers instead of direct sequence construction when performance matters, Java Collections Use scala. collection. JavaConverters to interoperate with Java collections. These are a set of implicits that add aztova and aescala conversion methods. ‘The use of these ensures that such conversions are explicit, aiding the reader: Inport scala.collection.Javaconverters.__ val List: java.util tist 5e9(2,2,3,4).asdava val buffer: scala. collection, mutable, GuFfer(int) ~ 11 itpsstwitr github. oftfectwoscala 2323/08/2028, 14:12 Effective Scala Concurrency Modern services are highly concurrent — it is common for servers to coordinate 10s-100s of thousands of simultaneous operations — and handling the implied complexity is a central theme in authoring robust systems software. Threads provide a means of expressing concurrency: they give you independent, heap-sharing execution contexts that are scheduled by the operating system. However, thread creation is expensive in Java and is a resource that must be managed, typically with the use of pools. This creates additional complexity for the programmer, and also a high degree of coupling: it's difficult to divorce application logic from their use of the underlying resources. This complexity is especially apparent when creating services that have a high degree of fan-out: each incoming request results in a multitude of requests to yet another tier of systems. In these systems, thread pools must be managed so that they are balanced according to the ratios of requests in each tier: mismanagement of one thread pool bleeds into another. Robust systems must also consider timeouts and cancellation, both of which require the introduction of yet more “control” threads, complicating the problem further. Note that if threads were cheap these problems would be diminished: no pooling would be required, timed out threads could be discarded, and no additional resource management would be required. ‘Thus resource management compromises modularity. Futures Use Futures to manage concurrency. They decouple concurrent operations from resource management: for example, Finagle multiplexes concurrent operations onto few threads in an efficient ‘manner. Scala has lightweight closure literal syntax, so Futures introduce little syntactic overhead, and they become second nature to most programmers. Futures allow the programmer to express concurrent computation in a declarative style, are composable, and have principled handling of failure. These qualities have convinced us that they are especially well suited for use in functional programming languages, where this is the encouraged style. Prefer transforming futures over creating your own. Future transformations ensure that failures are propagated, that cancellations are signalled, and free the programmer from thinking about the implications of the Java memory model. Even a careful programmer might write the following to issue an RPC 10 times in sequence and then print the results: val p = nex Pronise(List{Result]] vas Pesults: List[Result] = Mil det collect) ¢ ohc() onsucess ¢ resus => SF (results. ength < 20) collect() PesetValue(results) ) onfatlure (t => pesetexception(t) > } collect() P onsuccess { results => Printf("Got results Xs\n", results.0kString(*, *)) y ‘The programmer had to ensure that RPC failures are propagated, interspersing the code with control flow; worse, the code is wrong! Without declaring resuits volatile, we cannot ensure that resuits holds the previous value in each iteration. The Java memory model is a subtle beast, but luckily we can avoid all of these pitfalls by using the declarative style: det collect(results: List{Result] dope) FlatMap { result => 3 (results, length < 9) (1): Future(List{aesult]] = collect(result 1: results) ese Future.value(result ++ sesults) itpsstwitr github. oftfectwoscala 2823/08/2028, 14:12 Effective Scala } collect() onsuccess { results => printf ("Got results ¥s\n", results.nkString(", ")) ‘We use flavtap to sequence operations and prepend the result onto the list as we proceed. This is a common functional programming idiom translated to Futures. This is correct, requires less boilerplate, is less error prone, and also reads better. Use the Future combinators. ruture.select, Future. join, and Future.cellect codify common patterns when operating over multiple futures that should be combined, Do not throw your own exceptions in methods that return Futures. Futures represent both successful and failed computations. Therefore, it’s important that errors involved in that computation are properly encapsulated in the returned Future, Concretely, return Future.exception instead of throwing that exception: def divide(x: Int, ys Int): Future(Result) = { af (y= 8) return Future-exceptton(new TlLegalargunentxception ‘oivisor 4s @°)) Future. value(x/y) Fatal exceptions should not be represented by Futures. These exceptions include ones that are thrown when resources are exhausted, like OulOfMemoryError, and also JVM-level errors like ‘NoSuchMethodError. These conditions are ones under which the JVM must exit. The predicates scala.vtil.control.nonFatal — oF Twitter’s version con.twitter.util.NonFatel — should be used to identify exceptions which should be returned as a Future.exception, Collections ‘The subject of concurrent collections is fraught with opinions, subtleties, dogma and FUD. In most practical situations they are a nonissue: Always start with the simplest, most boring, and most standard collection that serves the purpose. Don't reach for a concurrent collection before you know that a synchronized one won't do: the JVM has sophisticated machinery to make synchronization cheap, so their efficacy may surprise you. If an immutable collection will do, use it — they are referentially transparent, so reasoning about them in a concurrent context is simple. Mutations in immutable collections are typically handled by updating a reference to the current value (in a var cell or an AtonieReference). Care must be taken to apply these correctly: atomies must be retried, and vars must be declared volatile in order for them to be published to other threads. Mutable concurrent collections have complicated semantics, and make use of subtler aspects of the Java memory model, so make sure you understand the implications — especially with respect to publishing updates — before you use them. Synchronized collections also compose better: operations like getoreiseupdate cannot be implemented correctly by concurrent collections, and creating composite collections is especially error prone. Control structures Programs in the functional style tend to require fewer traditional control structures, and read better when written in the declarative style. This typically implies breaking your logic up into several small methods or functions, and gluing them together with natch expressions. Functional programs also tend to be more expression-oriented: branches of conditionals compute values of the same type, for (..) yield computes comprehensions, and recursion is commonplace, Recursion Phrasing your problem in recursive terms often simplifies it, and if the tail call optimization applies (which can be checked by the etaiirec annotation), the compiler will even translate your code into a regular loop, Consider a fairly standard imperative version of heap FIX-DOWN: itpsstwitr github. oftfectwoscala 92823/08/2028, 14:12 Effective Scala nt): Unit = ¢ def Fxboun(heap: Array(T], 92 Th var ks Int =m Wile (7 >= 24k) ¢ var j= tk 4 GJ = hesp(H)) return ese ( swap(esp, &, 3) ee ? Every time the while loop is entered, we're working with state dirtied by the previous iteration. The value of each variable is a function of which branches were taken, and it returns in the middle of the loop when the correct position was found (The keen reader will find similar arguments in Dijkstra’s “Go ‘To Statement Considered Harmful”) Consider a (tal) recursive implementation(2]} @tatiree ‘Final def Fhooun(heap: Arcay(T], AF} < 192) return J Tmt) « val m= AF (J a» £92 || heap(2tt) © heap(2*iet)) 241 else 241 + ete) «ancy ¢ smap(heap, 4, 9) Fixbounheap, a, 3) > ? here every iteration starts with a well-defined clean slate, and there are no reference cells: invariants abound. It's much easier to reason about, and easier to read as well. There is also no performance penalty: since the method is tal-recursive, the compiler translates this into a standard imperative loop. Returns This is not to say that imperative structures are not also valuable. In many cases they are well suited to terminate computation early instead of having conditional branches for every possible point of termination: indeed in the above Fixooan, a return is used to terminate early if we're at the end of the heap. Returns can be used to cut down on branching and establish invariants. ‘This helps the reader by reducing nesting (how did I get here?) and making it easier to reason about the correctness of subsequent code (the array cannot be accessed out of bounds after this point). This is especially useful in “guard” clauses: def conpare(a: AnyRef, b: AnyRef): Int if (ea 8) val 4 = system. dentitytashcode(a) compare Sys iG l=) return @ 1. SdentityHashCode() 11 stow path ? Use returns to clarify and enhance readability, but not as you would in an imperative language; avoid using them to return the results of a computation. Instead of if def sufFax(4) ie itpsstwiter github. ioftfectwoscala 1012323/08/2028, 14:12 Effective Scala else £F (4 = 2) “nt else if (1 = 3) "re but using a natch expression is superior to either: def suffix(i: Tot) = 4 match ¢ ‘Note that returns can have hidden costs: when used inside of a closure, seq foreach { elem => isLast) 11 process... this is implemented in bytecode as an exception catching/throwing pair which, used in hot code, has performance implications, for loops and comprehensions for provides both suecinet and natural expression for looping and aggregation. It is especially useful when flattening many sequences. The syntax of for belies the underlying mechanism as it allocates and dispatches closures. This can lead to both unexpected costs and semantics; for example for (1vem <- container) { if (item [= 2) return ? ‘may cause a runtime error if the container delays computation, making the return nonlocal! For these reasons, it is often preferable to call foreach, #1atvap, nap, and filter directly — but do use ‘ars when they clarify require and assert require and assert both serve as executable documentation. Both are useful for situations in which the type system cannot express the required invariants. assert is used for invariants that the code assumes (either internal or external, for example Sesert(stream I~ null} resourceasstrean("someclassdata") ‘Whereas require is used to express API contracts: def Fb(n: Int) = ( require(n > @) > Functional programming Value oriented programming confers many advantages, especially when used in conjunction with functional programming constructs. This style emphasizes the transformation of values over stateful mutation, yielding code that is referentially transparent, providing stronger invariants and thus also easier to reason about. Case classes, pattern matching, destructuring bindings, type inference, and lightweight closure- and method-creation syntax are the tools of this trade. Case classes as algebraic data types Case classes encode ADTs: they are useful for modelling a large number of data structures and provide for succinct code with strong invariants, especially when used in conjunction with pattern matching. The pattern matcher implements exhaustivity analysis providing even stronger static guarantees. itpsstwitr github. ioftfectwoscala tt2823/08/2028, 14:12 Effective Scala Use the following pattern when encoding ADTs with case classes: sealed trait tree(t] case elass Node[T] ett: Teee(T], right: Tree(T)) extends case class Leaf{r](value: 1) extends Tree[] rT) ‘The type tree{7) has two constructors: Node and Leaf. Declaring the type seated allows the compiler to do exhaustivity analysis since constructors cannot be added outside the source file. ‘Together with pattern matching, such modelling results in code that is both succinct and “obviously correct”: def Findiin{T <: Ordered{T)}(tree: Tree{T)) = tree match ¢ ‘case Node(left, 1 Seq(Findlin(2eFt), Findin(right)) min ease Leaf value ? While recursive structures like trees constitute classie applications of ADTs, their domain of usefulness is much larger. Disjoint unions in particular are readily modelled with ADTs; these occur frequently in state machines. Options ‘The option type is a container that is either empty (Wone) or full (Seme(value)). It provides a safe alternative to the use of nuit, and should be used instead of null whenever possible. Options are collections (of at most one item) and they are embellished with collection operations — use them! Write var username: Option{Steing] bar" instead of var usernane: String germane = "foobar since the former is safer: the option type statically enforces that usersare must be checked for emptyness. Conditional execution on an option value should be done with ¢oreach; instead of 1 (opt -Asvefined) operate(opt.get) write ‘opt foreach { value => wvoperate(value) The style may seem odd, but provides greater safety (Wwe don’t call the exceptional get) and brevity. If both branches are taken, use pattern matching: opt maven { ‘case Sone(value) => oper: ‘case None => gefaultaction() > but ifall that’s missing is a default value, use getoreise operate(opt getoreise defaulevalue) Do not overuse option: if there isa sensible default —a Null Object — use that instead. option also comes with a handy constructor for wrapping nullable values: option( getClass. getResourceasstrean("oo")) is an option( Tnputstreae) that assumes a value of None should getResourcessstream return null. itpsstwitr github. oftfectwoscala 12323/08/2028, 14:12 Effective Scala Pattern matching Pattern matches (x aaten ( ...) are pervasive in well written Scala code: they conflate conditional execution, destructuring, and casting into one construct. Used well they enhance both clarity and safety. Use pattern matching to implement type switches: obj maven ( case adorn: Socketaddress ~ Pattern matching works best when also combined with destructuring (for example if you are matching case classes); instead of animal natch ( ‘case dog: Dog => “dog (X3)"-Format (dog. breed) case _ > aninal. species } waite animal natch ( ‘ease Dog(breed) => "dog (%s)*.format(breed) cate other => other. species Write custom extractors but only with a dual constructor (apply), otherwise their use may be out of place. Don't use pattern matching for conditional execution when defaults make more sense. The collections libraries usually provide methods that return options; avoid val x ‘ease head care MAL » because val x = list.headoption getorelse default is both shorter and communicates purpose. Partial functions ‘Scala provides syntactical shorthand for defining a partialruncton val pé: partialFunction(int, string] = ‘case © 4F 132 == 8 => “even and they may be composed with orsise val tf: (Int => String) = pf erelse { case _ => “odd") odd” Partial functions arise in many situations and are effectively encoded with Pa example as arguments to methods alfunction, for trait publisher(T) ( Gef subscribe(: Partialfunction[, Unit]) » val publisher: Publishe publisher, subscribe { ‘ease £ if isrine(i) => print case {if 1 count 7 ignore the rest */ > ("found prine”, 3) or in situations that might otherwise call for returning an option itpsstwitr github. oftfectwoscala 132323/08/2028, 14:12 Effective Scala 1/ Atkenpt to classify the the throwable for logeing. type Classifier = Throwable => Option[ java.util. logging. Level] might be better expressed with a partiaifunction fe Classifier = 9: aLfunction[ Throwable, java. 1 Losein Level] as it affords greater composability: val classifier: Classifier val classifier2: Classifier val classifier: Classifier assifiert ontlse classifier? o > java.util Logging. Level. FINEST } Destructuring bindings Destructuring value bindings are related to pattern matching; they use the same mechanism but are applicable when there is exactly one option (lest you accept the possibility of an exception). Destructuring binds are particularly useful for tuples and case classes, val tuple = (°3" val (char, digi » ‘tuple val tweet = Tweet just tweeting”, Tine. now) val Tweet (text, tinestanp) = te Laziness Fields in scala are computed by need when vat is prefixed with 1acy. Because fields and methods are equivalent in Scala (lest the fields are private[shis]) lazy val field = computa a0) is (roughly) short-hand for var _theField = None det Field = if (_therield.isbefined) _therield.get else ¢ Field = Sofe(computation()) weField.get ? i.e. it computes a results and memoizes it, Use lazy fields for this purpose, but avoid using laziness when laziness is required by semantics. In these cases it's better to be explicit sinee it makes the cost model explicit, and side effects can be controlled more precisely. Lazy fields are thread safe. Call by name ‘Method parameters may be specified by-name, meaning the parameter is bound not to a value but to ‘a computation that may be repeated. This feature must be applied with care; a caller expecting by-value semanties will be surprised. The motivation for this feature is to construct syntactically natural DSLs — new control constructs in particular can be made to look much like native language features, Only use call-by-name for such control constructs, where it is obvious to the ealler that what is being passed in is a “block” rather than the result of an unsuspecting computation. Only use call-by-name arguments in the last position of the last argument list. When using call-by-name, ensure that the ‘method is named so that it is obvious to the caller that its argument is call-by-name. ‘When you do want a value to be computed multiple times, and especially when this computation is, side effecting, use explicit functions: lass Ssiconnector(akngine: () => ssLengine) ‘The intent remains obvious and the caller is left without surprises, ‘flathap ‘fiatmap — the combination of sap with flatten — deserves special attention, for it has subtle power and great utility. Like its brethren rap, itis frequently available in nontraditional collections such as Future itpssitwitr github. ioftfectwoscalal 1412323/08/2028, 14:12 Effective Scala and option. Its behavior is revealed by its signature; for some container([A] ‘flavtap[a] (ft A => Container[a]): Container[3] ‘tathap invokes the function ¢ for the element(s) of the collection producing a new collection, (all of) which are flattened into its result. For example, to get all permutations of two character strings that aren't the same character repeated twice: val peres = chars ‘enars #latlap { > => HF (a T= b) Sea( "exe" else sea() ? ? ap (a> format(a, b)) which is equivalent to the more concise for-comprehension (which is — roughly — syntactical sugar for the above): val peres = for ( & chars fea le } yield "eke formata, b) ‘Flavwap is frequently useful when dealing with options — it will collapse chains of options down to one, val host: 9} val port: 0 eon{string) = sen[ nt val adér: Option{inetSocketadéress) = hast Flatmap { port map { p => > ? which is also made more suecinet with for val adér: option[Inetsockethddress) = for { hh c= host P< port } yield new InetSocketAddress(h, p) ‘The use of fiatmap in Futures is discussed in the futures section. ‘Object oriented programming ‘Much of Scala’s vastness lies in its object system. Scala is a pure language in the sense that all values are objects; there is no distinction between primitive types and composite ones. Scala also features mixins allowing for more orthogonal and piecemeal construction of modules that can be flexibly put together at compile time with all the benefits of static type checking. A motivation behind the mixin system was to obviate the need for traditional dependency injection. ‘The culmination of this “component style” of programming is the cake pattern Dependency injection In our use, however, we've found that Scala itself removes so much of the syntactical overhead of “classic” (constructor) dependency injection that we'd rather just use that: itis clearer, the dependencies are still encoded in the (constructor) type, and class construction is so syntactically trivial that it becomes a breeze. It's boring and simple and it works. Use dependency injection for program modularization, and in particular, prefer composition over inheritance — for this leads to more ‘modular and testable programs. When encountering a situation requiring inheritance, ask yourself: how ‘would you structure the program if the language lacked support for inheritance? The answer may be compelling Dependency injection typically makes use of traits, it Tweetsereas ( ef subseribe(#: Tweet => Unit) itpsstwitr github. oftfectwoscala 1912323/08/2028, 14:12 Effective Scala » Class Hosebindstreas extends we class TweetCounter(strean: Tweetstream) { Strean-subseribe { tweet => count += 1) > Itis common to inject factories — objects that produce other objects. In these cases, favor the use of simple functions over specialized factory types. class FilteredtweetCounter(akstrean: Filter => Tweetstrean) { rkStrean(PublicTweets) subscribe { tweet => publiccount += 4} rikstrean(Os).subserabe { tweet => dacount += 1 } Traits ‘Dependency injection does not at all preclude the use of common interfaces, or the implementation of common code in traits. Quite the contrary — the use of traits are highly encouraged for exactly this reason: multiple interfaces (traits) may be implemented by a conerete class, and common code ean be reused across all such classes. Keep traits short and orthogonal: don’t lump separable functionality into a trait, think of the smallest related ideas that fit together. For example, imagine you have an something that can do IO: serait Ter ¢ def wrize(bytes: Arraytsyce]) def reaagn: Int): array[ayte) ? separate the two behaviors: it Reader ( def reaa(n: et): arrayTayte] » (bytes: Array(ayte]) ef writ ? ‘and mix them together to form what was an toer: new Reader with Writer... Interface minimalism leads to greater orthogonality and cleaner modularization, Visibility Scala has very expressive visibility modifiers. It’s important to use these as they define what constitutes the public API. Public APIs should be limited so users don't inadvertently rely on implementation details and limit the author's ability to change them: They are crucial to good modularity. As a rule, i's much easier to expand public APIs than to contract them. Poor annotations can also compromise backwards binary compatibility of your code. private(this} Aclass member marked private, private val x: In is visible to all instances of that class (but not their subclasses). In most cases, you want private this]. private[this] val x: ant which limits visibility to the particular instance. The Scala compiler is also able to translate private( this} into a simple field access (since access is limited to the statically defined class) which can sometimes aid performance optimizations, Singleton class types It's common in Seala to create singleton class types, for example itpsstwitr github. oftfectwoscala 1812323/08/2028, 14:12 Effective Scala def Fool) = new Foo with Bar with Baz { In these situations, visibility can be constrained by declaring the returned type: def F00(): Foo with Bar = new Foo with Bar with Baz { where callers of fo0() will see a restricted view (Foo with sar) of the returned instance, Structural typing Do not use structural types in normal use. They are a convenient and powerful feature, but unfortunately do not have an efficient implementation on the JVM. However — due to an implementation quirk — they provide a very nice shorthand for doing reflection. val 0b}: Anyker obj. asinstanceof[ (det close()}]-close() Error handling Scala provides an exception facility, but do not use it for commonplace errors, when the programmer must handle errors properly for correctness. Instead, encode such errors explicitly: using option or 1.try are good, idiomatic choices, as they harness the type system to ensure that the user is properly considering error handling. For example, when designing a repository, the following API may be tempting: At Reposttory[key, Value] ef get(Key: Key): Value > but this would require the implementor to throw an exception when the key is absent. A better approach is to use an opts ‘trait Repository{Key, value] ¢ ef get(key: Key): optton(vaiue) ? This interface makes it obvious that the repository may not contain every key, and that the programmer ‘must handle missing keys. Furthermore, option has a number of combinators to handle these cases. For example, getoreise is used to supply a default value for missing keys: val repo: Repository[tnt, string] repo.get(123) getorelse “default: Handling exceptions Because Scala’s exception mechanism isn't checked — the compiler cannot statically tell whether the programmer has covered the set of possible exceptions — itis often tempting to cast a wide net when handling exceptions. However, some exceptions are fatal and should never be caught; the cade oneration() y eaten ¢ is almost always wrong, as it would catch fatal errors that need to be propagated. Instead, use the 1.NonFatal extractor to handle only nonfatal exceptions. wy ¢ operation() p eaten ¢ cease NonFatal(exc) > itpsstwitr github. oftfectwoscala se23/08/2028, 14:12 Effective Scala Garbage collection We spend a lot of time tuning garbage collection in production. The garbage collection concerns are largely similar to those of Java though idiomatic Scala code tends to generate more (short-lived) garbage than idiomatic Java code — a byproduct of the functional style. Hotspot's generational garbage collection typically makes this a nonissue as short-lived garbage is effectively free in most circumstances. Before tackling GC performance issues, watch this presentation by Attila that illustrates some of our experiences with GC tuning. In Scala proper, your only tool to mitigate GC problems is to generate less garbage; but do not act without data! Unless you are doing something obviously degenerate, use the various Java profiling tools — our own include heapster and gepraf. Java compatibility When we write code in Scala that is used from Java, we ensure that usage from Java remains idiomatic. Oftentimes this requires no extra effort — classes and pure traits are exactly equivalent to their Java counterpart — but sometimes separate Java APIs need to be provided. A good way to get a feel for your library’s Java APT is to write a unittest in Java Gust for compilation); this also ensures that the Java-view of your library remains stable over time as the Scala compiler can be volatile in this regard. ‘Traits that contain implementation are not directly usable from Java: extend an abstract class with the trait instead. 1/ Mot directly usable fron 2ava trait Animal ( ef eat(other: Aninal) {def eatany(aninals: Seq{aninal) = aninals foreach(est(_)) » 1{ But this is abstract class Javadninal extends Animal ‘Twitter's standard libraries ‘The most important standard libraries at Twitter are Util and Hinagle. Util should be considered an extension to the Scala and Java standard libraries, providing missing functionality or more appropriate implementations. Finagle is our RPC system; the kernel distributed systems components. Futures Futures have been discussed briefly in the concurrency section. They are the central mechanism for coordination asynchronous processes and are pervasive in our codebase and core to Finagle. Futures allow for the composition of concurrent events, and simplify reasoning about highly concurrent ‘operations. They also lend themselves to a highly efficient implementation on the JVM. ‘Twitter's futures are asynchronous, so blocking operations — basically any operation that can suspend the execution of its thread; network 10 and disk IO are examples — must be handled by a system that itself provides futures for the results of said operations. Finagle provides such a system for network 10. Futures are plain and simple: they hold the promise for the result of a computation that is not yet complete. They are a simple container — a placeholder. A computation could fail of course, and this must also be encoded: a Future can be in exactly one of 3 states: pending, failed or completed. Aside: Composition Let's revisit what we mean by composition: combining simpler components into more complicated ones. The canonical example of this is function composition: Given functions f and g, the composite function (g2f)(x) = g(f(x)) — the result of applying fto x frst, and then applying g tothe result ofthat — can be written in Scala: vat f = (1: ant) => ‘.tostring Val g = G String) => ses tpstter git oeecivscal see23/08/2028, 14:12 Effective Scala val h = @ compose f // : Int => String seala> h(223) ese: java-lang. string = 123123123 the funetion h being the composite. It is a new funetion that combines both f and g in a predefined way. Futures are a type of collection — they are a container of either 0 or 1 elements — and you'll find they have standard collection methods (eg. nap, F£1¢er, and foreach). Since a Future’s value is deferred, the result of applying any of these methods is necessarily also deferred; in = result map { 1 => i.tosteing the function { i => i.testring } is not invoked until the integer value becomes available, and the transformed collection resuitstr is also in pending state until that time. Lists ean be flattened; val ListofList: List{Lise{int]] = Val List: List{tne] = Lstortist-iatten and this makes sense for futures, too: coffuture: Future[ Future ure: Future(Int] = futureofFul re. Flaten since futures are deferred, the implementation of f1atten — it returns immediately — has to return a future that is the result of waiting for the completion of the outer future (Future(Future(ot}}) and after that the inner one (Future{Future{int}}). Ifthe outer future fails, the flattened future must also fail Futures (like Lists) also define latmap; ruture(A] defines its signature as ‘latnap(9] (4: A => Future[o]): Future(a) which is like the combination of both rap and flatcen, and we could implement it that way: det Flatmap(e](f: A => Future[]): Future[a] = { ‘val mapped: Future(Future(6}) = this map f val flattened: Future[s] = mapped. #lat% Flattened ? This is a powerful combination! With fiatvap we can define a Future that is the result of two futures sequenced, the second future computed based on the result of the first one. Imagine we needed to do ‘two RPCs in order to authenticate a user (id), we could define the composite operation in the following way: def getuser(id: Int): Future[tser) def authenticate(user: User): Future[Soolean def sstanvthed(id fi igetuser(i¢) #latap { user -e[800lean] = authenticate(user) } an additional benefit to this type of composition is that error handling is built-in: the future returned from istutned(..) will fail if either of getuser(...) or autnerticate(... does with no extra error handling code. Style Future callback methods (respond, onsuccess, onfailure, ensure) return a new future that is chained to its parent. This future is guaranteed to be completed only after its parent, enabling patterns like acquireResource() onsuccess { value => ‘Conputesonetning(value) } ensure { freekesource() > where freekesource() is guaranteed to be executed only after conput the native try .. finatly pattern, itpsstwitr github. ioftfectwoscalal ng, allowing for emulation of 1912323/08/2028, 14:12 Effective Scala Use onsuccess instead of foreach — it is symmetrical to onfaiture and is a better name for the purpose, and also allows for chaining, Always try to avoid creating eronise instances directly: nearly every task can be accomplished via the use of predefined combinators. These combinators ensure errors and cancellations are propagated, and. generally encourage dataflow style programming which usually obviates the need for synchronization and volatility declarations. Code written in tail-recursive style is not subject to stack-space leaks, allowing for efficient implementation of loops in dataflow-style: case class Node(parent: option(Nede], ...) def gethoce(id: Tht): Future[Node) = det getilierareny(id: Int, nodes: List{Node] = NIL): Future[Node] = igetNode( id) FlatMap ( ‘ase niode(Sone(parent), ..) => getiierarchy(parent, nm :: nodes) case n => Future.value({h :2 nodes) reverse) ? Future defines many useful methods: Use Futore.vatue() and Future.exceptton() to ereate pre-satistied futures, Future. collect(), Future. join() and Future. select() provide combinators that turn many futures into one (ie. the gather part ofa scatter-gather operation) Cancellation Futures implement a weak form of cancellation, Invoking Future#cancet does not directly terminate the computation but instead propagates a level triggered signal that may be queried by whichever process ultimately satisfies the future. Cancellation flows in the opposite direction from values: a cancellation signal set by a consumer is propagated to its producer. The producer uses onCance2lation on Pronise to listen to this signal and act accordingly. This means that the cancellation semantics depend on the producer, and there is no default implementation, Cancellation is but a hint. Locals Util’s kacat provides a reference cell that is local to a particular future dispatch tree. Setting the value of a local makes this value available to any computation deferred by a Future in the same thread. They are analogous to thread locals, except their scope is not a Java thread but a tree of “future threads”. In it User ( def ane: String ef incrcost points: Int) ? Val user = new Loca2user] user() = currentuser rpe() ensure ‘ser() inertost(18) user) in the ensure block will refer to the value of the user local at the time the callback was added. As with thread locals, Locals can be very convenient, but should almost always be avoided: make sure the problem cannot be sufficiently solved by passing data around explicitly, even if it is somewhat burdensome. Locals are used effectively by core libraries for very common concerns — threading through RPC traces, propagating monitors, creating “stack traces” for future callbacks — where any other solution would unduly burden the user. Locals are inappropriate in almost any other situation, Offer/Broker Concurrent systems are greatly complicated by the need to coordinate access to shared data and resources. Actors present one strategy of simplification: each actor is a sequential process that itpsstwitr github. oftfectwoscala 2012323/08/2028, 14:12 Effective Scala ‘maintains its own state and resources, and data is shared by messaging with other actors. Sharing data requires communicating between actors Offer/Broker builds on this in three important ways. First, communication channels (Brokers) are first class — that is, you send messages via Brokers, not to an actor directly. Secondly, Offer/Broker is a synchronous mechanism: to communicate is to synchronize, This means we can use Brokers as a coordination mechanism: when process » has sent a message to process b; both a and b agree on the state of the system. Lastly, communication can be performed selectively: a process can propose several different communications, and exactly one of them will obtain. In order to support selective communication (as well as other composition) in a general way, we need to decouple the description of a communication from the act of communicating. This is what an offer does — it is a persistent value that describes a communication; in order to perform that communication {act on the offer), we synchronize via its syne() method. it oFfer{r] { ef syne() ? ref] which returns a Future(r) that yields the exchanged value when the communication obtains, A sroker coordinates the exchange of values through offers — it is the channel of communications: ‘trait sroker(T) { def sena(nsg: 1): oFfer(Unit) Val recy! offer(T] so that, when creating two offers val b: Broker{Tat} val sendof = b.send(1) Val recvof = birecy and sendof and recvof are both synchronized 11 tn process 4 send0Fesyne() 11 tn process 2: reevOFesyne() both offers obtain and the value 1 is exchanged. Selective communication is performed by combining several offers with oFfer.choose def choose[T)(ofs: offer(t]*): offer which yields a new offer that, when synchronized, obtains exactly one of ofs — the first one to become available. When several are available immediately, one is chosen at random to obtain. ‘The offer object has a number of one-off Offers that are used to compose with Offers from a Broker. often tineout (duration): oFFer( Unit] is an offer that activates after the given duration. oFfer.never will never obtain, and oFfer.const(value) obtains immediately with the given value. These are useful for composition via selective communication. For example to apply a timeout on a send operation; offer choose offer. timeout (10. seconds), broker. send("ny value") yesyne() It may be tempting to compare the use of Offer/Broker to SynchronousQueue, but they are different in subtle but important ways. Offers can be composed in ways that such queues simply cannot. For example, consider a set of queues, represented as Brokers: val @ = new Broker( Int) val a1 = new Broker[int] Val a2 = new Broker( at] itpsstwitr github. oftfectwoscala 2112323/08/2028, 14:12 Effective Scala Now let's create a merged queue for reading: val anyq: offer OFfer.chocse(qe.recv, ql.necy, @2.reev) anyg is an offer that will read from first available queue, Note that anyq is still synchronous — we still have the semantics of the underlying queues. Such composition is simply not possible using queues. Example: A Simple Connection Pool Connection pools are common in network applications, and they're often tricky to implement — for example, it's often desirable to have timeouts on acquisition from the pool since various clients have different latency requirements. Pools are simple in principle: we maintain a queue of connections, and wwe satisfy waiters as they come in. With traditional synchronization primitives this typically involves keeping two queues: one of waiters (when there are no connections), and one of connections (when there are no waiters). Using Offer/Brokers, we can express this quite naturally: lass Pool coms: Seq(Conn)) « private[this] val waiters ~ new Broker[Conn] private[this] val returnconn = new Broker [Cann] val get: oFfer{conn] = waiters.recy ef pute: conn) { returnconn |e } private[this] def loop(conna: Queve[conn]) { OFfer.choose(| AF (conng.istapty) Offer.never else { val (head, rest) = conng.dequeve() waiters. send(head) map ( _ => loop(rest) ) returnconn.reev map { € => Loop{conna.enqueue(e)) } syne) } o9p(Queue-crpty ++ conns) ? 1ocp will always offer to have a connection returned, but only offer to send one when the queue is nonempty. Using @ persistent queue simplifies reasoning further. The interface to the pool is also through an Offer, so if a caller wishes to apply a timeout, they can do so through the use of combinators: val conn: Future[option{cona]] = offer. choose ool.get map { conn => Some(conn) }, OFfen-tineout(a.second) map { => None ) desyne) No extra bookkeeping was required to implement timeouts; this is due to the semantics of Offers: if ofter-tincout is selected, there is no longer an offer to receive from the pool — the pool and its caller never simultaneously agreed to receive and send, respectively, on the waiters broker. Example: Sieve of Eratosthenes Itis often useful — and sometimes vastly simplifying — to structure concurrent programs as a set of sequential processes that communicate synchronously. Offers and Brokers provide a set of tools to make this simple and uniform. Indeed, their application transcends what one might think of as “classic” concurrency problems — concurrent programming (with the aid of Offer/Broker) is a useful structuring tool, just as subroutines, classes, and modules are — another important idea from CSP. One example of this is the Sieve of Eratosthenes, which ean be structured as a successive application of filters to a stream of integers. First, welll need a source of integers: def ancegers(fron: In} val b = new Broker[tnt] GeF gen(n: Int): Unit = b.send(n)-sync() ensure gen(n + 2) encom) bereey offertar = {ntegers(n) is simply the offer of all consecutive integers starting at n. Then we need a filter: itpsstwitr github. oftfectwoscala 2212323/08/2028, 14:12 Effective Scala def Filter(in: offer{Int], prime: Int val b = new Sroxer[Int] def LoopC) in.syne() onsuceess ( i => if (i prine I= 0) besend(i).syne() ensure 1o0p¢) “Toop oFfentnt ? ? oop) ? flter(in, p) returns the offer that removes multiples of the prime p from in. Finally, we define our def sieve = ( val b= new Broker[Tnt) GeF Loop(oF: oFFer{int]) { for (prine ¢- oF. sync()5 _ 4 lomeriter(, prise} Toop (sntegers(2)) bereev b.send(prine) syne()) 1oop() works simply: it reads the next (prime) number from of, and then applies a filter to of that excludes this prime. As loop recurses, successive primes are filtered, and we have a Sieve. We can now print out the first 10000 primes: val prives ~ sieve @ until 19ee0 foreach { _ => printin(prives.syne()) > Besides being structured into simple, orthogonal components, this approach gives you a streaming Sieve: you do not a priori need to compute the set of primes you are interested in, further enhancing. ‘modularity. Acknowledgments ‘The lessons herein are those of Twitter's Seala community — hope I've been a faithful chronicler. Blake Matheny, Nick Kallen, Steve Gury, and Raghavendra Prabhu provided much helpful guidance and many excellent suggestions. 1. Yourkit is a good profiler [back] 2. From Finagle's heap balancer [back] Copyright © 2012 Twitter Ine. Tiensed wider COBY 3.0 itpsstwitr github. oftfectwoscala 23123