Lisp Users and Vendors Conference August 10, 1993 Tutorial on Good Lisp Programming Style Peter Norvig Sun Microsystems Labs Inc. Kent Pitman Harlequin, Inc. Portions copyright © 1992, 1993 Peter Norvig. Portions copyright © 1992, 1993 Kent M. Pitman. All Rights Reserved.Outline 1. 2. What is Good Style? Tips on Built-In Functionality « Tips on Near-Standard Tools . Kinds of Abstraction . Programming in the Large . Miscellaneous1. What is Good Style? Good Lisp Programming Style “Elegance is not optional.” — Richard A. O’Keefe Good style (in any language) leads to programs that are: Understandable Reusable Extensible Efficient Easy to develop/debug It also helps correctness, robustness, compatibility Our maxims of good style are: Be explicit Be specific Be concise Be consistent Be helpful (anticipate the reader’s needs) Be conventional (don’t be obscure) Build abstractions at a usable level Allow tools to interact (referential transparency) Good style is the “underware’ that supports a program 31.1. Where does good style come from? What To Believe Don’t believe everything we tell you. (Just most.) Worry less about what to believe and more about why. Know where your “Style Rules’ come from: Religion, Good vs. Evil ‘This way is better.” Philosophy ‘This is consistent with other things.” Robustness, Liability, Safety, Ethics ‘T’ll put in redundant checks to avoid something horrible.” Legality “Our lawyers say do it this way.” Personality, Opinion ‘‘I like it this way.” Compatibility “Another tool expects this way.’”” Portability “Other compilers prefer this way.” Cooperation, Convention “It has to be done some uniform way, SO we agreed on this one.” Habit, Tradition ‘We've always done it this way.” Ability “My programmers aren’t sophisticated enough.” Memory “Knowing how I would do it means I don’t have to remember how I did do it.” Superstition ‘I’m scared to do it differently.” Practicality “This makes other things easier.”1.1. Where does good style come from? It’s All About Communication Expression + Understanding = Communication Programs communicate with: e@ Human readers Compilers Text editors (arglist, doc string, indent) Tools (trace, step, apropos, xref, manual) e e e e Users of the program (indirect communication)1.1. Where does good style come from? Know the Context When reading code: e Know who wrote it and when. When writing code: e Annotate it with comments. e Sign and date your comments! (Should be an editor command to do this) Some things to notice: e@ People’s style changes over time. @ The same person at different times can seem like a different person. e@ Sometimes that person is you.1.2. How do I know if it’s good? Value Systems Are Not Absolute Style rules cannot be viewed in isolation. They often overlap in conflicting ways. The fact that style rules conflict with one another re- flects the natural fact that real-world goals conflict. A good programmer makes trade-offs in programming style that reflect underlying priority choices among var- ious major goals: Understandable Reusable Extensible Efficient (coding, space, speed, ...) Easy to develop/debug1.2. How do I know if it’s good? Why Good Style is Good Good style helps build the current program, and the next one: e Organizes a program, relieving human memory needs e Encourages modular, reusable parts Style is not just added at the end. It plays a part in: Organization of the program into files Top-level design, structure and layout of each file Decomposition into modules and components Data-structure choice Individual function design/implementation Naming, formatting, and documenting standards1.2. How do I know if it’s good? Why Style is Practical: Memory “When I was young, I could imagine a castle with twenty rooms with each room having ten different ob- jects in it. I would have no problem. I can’t do that anymore. Now I think more in terms of earlier experi- ences. I see a network of inchoate clouds, instead of the picture-postcard clearness. But I do write better programs.” — Charles Simonyi “Some people are good programmers because they can handle many more details than most people. But there are a lot of disadvantages in selecting programmers for that reason—it can result in programs no on else can maintain.” — Butler Lampson “Pick out any three lines in my program, and I can tell you where they’re from and what they do.” — David McDonald Good style replaces the need for great memory: e Make sure any 3 (5? 107) lines are self-explanatory Also called “referential transparency” Package complexity into objects and abstractions; not global variables/dependencies e Make it “fractally” self-organizing all the way up/down e Say what you mean e Mean what you say1.2. How do I know if it’s good? Why Style is Practical: Reuse Structured Programming encourages modules that meet specifications and can be reused within the bounds of that specification. Stratified Design encourages modules with commonly- needed functionality, which can be reused even when the specification changes, or in another program. Object-Oriented Design is stratified design that con- centrates on classes of objects and on information hid- ing. You should aim to reuse: e Data types (classes) e Functions (methods) e Control abstractions e Interface abstractions (packages, modules) e Syntactic abstractions (macros and whole languages) 101.2. How do I know if it’s good? Say What You Mean “Say what you mean, simply and directly." — Kernighan & Plauger Say what you mean in data (be specific, concise): e Use data abstractions e Define languages for data, if needed @ Choose names wisely Say what you mean in code (be concise, conventional): e Define interfaces clearly e Use Macros and languages appropriately e Use built-in functions e Create your own abstractions e Don’t do it twice if you can do it once In annotations (be explicit, helpful): e Use appropriate detail for comments @ Documentation strings are better than comments e@ Say what it is for, not just what it does e Declarations and assertions e Systems (and test files, etc.) 111.2. How do I know if it’s good? Be Explicit Optional and Keyword arguments. Tf you have to look up the default value, you need to supply it. You should only take the default if you truly believe you don’t care or if you’re sure the default is well-understood and well-accepted by all. For example, when opening a file, you should almost never consider omitting the :direction keyword argu- ment, even though you know it will default to :input. Declarations. If you know type information, declare it. Don’t do what some people do and only declare things you know the compiler will use. Compilers change, and you want your program to naturally take advantage of those changes without the need for ongoing intervention. Also, declarations are for communication with human readers, too—not just compilers. Comments. If you’re thinking of something useful that others might want to Know when they read your code and that might not be instantly apparent to them, make it a comment. 121.2. How do I know if it’s good? Be Specific Be as specific as your data abstractions warrant, but no more. Choose: 33 More specific 33 More abstract (mapce #’process-word (map nil #’process-word (first sentences)) (elt sentences 0)) Most specific conditional: e@ if for two-branch expression e@ when, unless for one-branch statement @ and, or for boolean value only @ cond for multi-branch statement or expression 33 Violates Expectation: 33 Follows Expectation: (and (numberp x) (cos x)) (and (numberp x) © x 3)) (if (numberp x) (cos x)) (if (numberp x) (cos x) nil) (if (numberp x) (print x)) (when (numberp x) (print x)) 131.2. How do I know if it’s good? Be Concise Test for the simplest case. If you make the same test (or return the same result) in two places, there must be an easier way. Bad: verbose, convoluted (defun count-all-numbers (alist) (cond (null alist) 0) (t (+ Gif (listp (first alist)) (count-all-numbers (first alist)) (if (numberp (first alist)) 1 0)) (count-all-numbers (rest alist)) )) )) e Returns O twice e Nonstandard indentation @ alist suggests association list Good: (defun count-all-numbers (exp) (typecase exp (cons (+ (count-all-numbers (first exp)) (count-all-numbers (rest exp)))) Coumber 1) o7 0))) cond instead of typecase is equally good (less specific, more conventional, consistent). 141.2. How do I know if it’s good? Be Concise Maximize LOCNW: lines of code not written. “Shorter is better and shortest is best.” — Jim Meehan Bad: _ too verbose, inefficient (defun vector-add (x y) (let (@ nil) n) (setq n (min (list-length x) (list-length y))) (dotimes (j n (reverse z)) (setq z (cons (+ (nth j x) (th j y)) z))))) (defun matrix-add (A B) (let ((C nil) m) (setq m (min (list-length A) (list-length B))) (dotimes (i m (reverse C)) (setq C (cons (vector-add (nth i A) (ath i B)) C))))) @ Use of nth makes this O(n?) e Why list-length? Why not length or mapcar? e Why not nreverse? e Why not use arrays to implement arrays? e The return value is hidden 151.2. How do I know if it’s good? Be Concise Better: more concise (defun vector-add (x y) "Element-wise add of two vectors" (mapcar #?+ x y)) (defun matrix-add (A B) "Element-wise add of two matrices (lists of lists)" (mapcar #’vector-add A B)) Or use generic functions: (defun add (&rest args) "Generic addition" Gif (null args) ° (reduce #’binary-add args))) (defmethod binary-add ((x number) Cy number)) Gx y)) (defmethod binary-add ((x sequence) (y sequence)) (map (type-of x) #’binary-add x y)) 161.2. How do I know if it’s good? Be Helpful Documentation should be organized around tasks the user needs to do, not around what your program hap- pens to provide. Adding documentation strings to each function usually doesn’t tell the reader how to use your program, but hints in the right place can be very ef- fective. Good: (from Gnu Emacs online help) next-line: Move cursor vertically down ARG lines. .If you are thinking of using this in a Lisp program, consider using ‘forward-line’ instead. It is usually eas- ier to use and more reliable (no dependence on goal column, etc.). defun: defines NAME as a function. The definition is (lambda ARGLIST [DOCSTRING] BODY...). See also the function interactive. These anticipate user’s use and problems. 171.2. How do I know if it’s good? Be Conventional Build your own functionality to parallel existing features Obey naming conventions: with-something, dosomething macros Use built-in functionality when possible @ Conventional: reader will Know what you mean @ Concise: reader doesn’t have to parse the code e Efficient: has been worked on heavily Bad: non-conventional (defun add-to-list (elt list) (cond ((member elt 1st) 1st) (t (cons elt 1st)))) Good: use a built-in function (left as an exercise) “Use library functions” — Kernighan & Plauger 181.2. How do I know if it’s good? Be Consistent Some pairs of operators have overlapping capabilities. Be consistent about which you use in neutral cases (where either can be used), so that it is apparent when you’re doing something unusual. Here are examples involving let and let*. The first exploits parallel binding and the second sequential. The third is neutral. Get ((a b) (ba)) ...) Clet* (Ca b) (b (# 2 a)) (c Gb 1))) ...) (let (Ca (# x (+ 2 y))) (b (Hy G@ 2 x)))) «..) Here are analogous examples using flet and labels. The first exploits closure over the local function, the second exploits non-closure. The third is neutral. labels ((process (x) ... (process (cdr x)) ...)) ... (flet ((foo (x) (+ (foo x) 1))) ...) (flet ((add3 (x) (+ x 3))) ...) In both cases, you could choose things the other way around, always using let* or labels in the neutral case, and let or flet in the unusual case. Consistency mat- ters more than the actual choice. Most people, how- ever, think of let and flet as the normal choices. 191.2. How do I know if it’s good? Choose the Right Language Choose the appropriate language, and use appropriate features in the language you choose. Lisp is not the right language for every problem. “You got to dance with the one that brung you.” — Bear Bryant Lisp is good for: Exploratory programming Rapid prototyping Minimizing time-to-market Single-programmer (or single-digit team) projects Source-to-source or data-to-data transformation Compilers and other translators Problem-specific languages Dynamic dispatch and creation (compiler available at run-time) Tight integration of modules in one image (as opposed to Unix’s character pipe model) High degree of interaction (read-eval-print, CLIM) User-extensible applications (gnu emacs) “I believe good software is written by small teams of two, three, or four people interacting with each other at a very high, dense level.” — John Warnock 201.2. How do I know if it’s good? Choose the Right Language “Once you are an experienced Lisp programmer, it’s hard to return to any other language."” — Robert R. Kessler Current Lisp implementations are not so good for: Persistent storage (data base) Maximizing resource use on small machines Projects with hundreds of programmers Close communication with foreign code Delivering small-image applications Real-time control (but Gensym did it) Projects with inexperienced Lisp programmers Some kinds of numerical or character computation (Works fine with careful declarations, but the Lisp efficiency model is hard to learn.) 212. Tips on Built-in Functionality Built-in Functionality “No doubt about it, Common Lisp is a big language” — Guy Steele @ 622 built-in functions (in one pre-ANSI CL) e 86 macros @ 27 special forms e 54 variables ¢ 62 constants But what counts as the language itself? e C++ has 48 reserved words e ANSI CL is down to 25 special forms e The rest can be thought of as a required library Either way, the Lisp programmer needs some help: Which built-in functionality to make use of How to use it 222. Tips on Built-in Functionality DEFVAR and DEFPARAMETER Use defvar for things you don’t want to re-initialize upon re-load. (defvar *options* ’()) (defun add-option (x) (pushnew x *options*)) Here you might have done (add-option ...) many times before you re-load the file-perhaps some even from another file. You usually don’t want to throw away all that data just because you re-load this defini- tion. On the other hand, some kinds of options do want to get re-initialized upon re-load... (defparameter *use-experimental-mode* nil "Set this to T when experimental code works.") Later you might edit this file and set the variable to T, and then re-load it, wanting to see the effect of your edits. Recommendation: Ignore the part in CLtL that says defvar is for variables and defparameter is for parame- ters. The only useful difference between these is that defvar does its assignment only if the variable is un- bound, while defparameter does its assignment uncon- ditionally. 232. Tips on Built-in Functionality EVAL-WHEN (eval-when (:execute) ...) Ceval-when (:compile-toplevel) ...) + (eval-when (:load-toplevel) ...) Also, take care about explicitly nesting eval-when forms. The effect is not generally intuitive for most people. 242. Tips on Built-in Functionality FLET to Avoid Code Duplication Consider the following example’s duplicated use of (f © @))). (do (x (£ (g (h))) (f (g (h))))) Gib ...) Every time you edit one of the (f (g (h)))’s, you prob- aby want to edit the other, too. Here is a better mod- ularity: (flet (Cfgh O (f (g (h))))) (do ((x (fgh) (fgh))) (nil) ...)) (This might be used as an argument against do.) Similarly, you might use local functions to avoid dupli- cation in code branches that differ only in their dynamic state. For example, (defmacro handler-case-if (test form &rest cases) (let ((do-it (gensym "DO-IT"))) ‘let ((,do-it © ,form)) Gf test (handler-case (,do-it) ,@cases) (,do-it))))) 252. Tips on Built-in Functionality DEFPACKAGE Programming in the large is supported by a design style that separates code into modules with clearly defined interfaces. The Common Lisp package system serves to avoid name clashes between modules, and to define the in- terface to each module. e There is no top level (be thread-safe) e There are other programs (use packages) e Make it easy for your consumers Export only what the consumer needs e Make it easy for maintainers License to change non-exported part (defpackage "PARSER" (:use "LISP" #+Lucid "LCL" #+Allegro "EXCL") (:export "PARSE" "PARSE-FILE" "'START-PARSER-WINDOW" "DEFINE-GRAMMAR" "DEFINE-TOKENIZER") ) Some put exported symbols at the top of the file where they are defined. We feel it is better to put them in the defpackage, and use the editor to find the corresponding definitions. 262. Tips on Built-in Functionality Understanding Con ns vs Errors Lisp assures that most errors in code will not corrupt data by providing an active condition system. Learn the difference between errors and conditions. All errors are conditions; not all conditions are errors. Distinguish three concepts: e@ Signaling a condition— Detecting that something unusual has happened. e Providing a restart— Establishing one of possibly several options for continuing. e@ Handling a condition— Selecting how to proceed from available options. 272. Tips on Built-in Functionality Error Detection Pick a level of error detection and handling that matches your intent. Usually you don’t want to let bad data go by, but in many cases you also don’t want to be in the debugger for inconsequential reasons. Strike a balance between tolerance and pickiness that is appropriate to your application. Bad: what if its not an integer? (defun parse-date (string) "Read a date from a string. ... (multiple-value-bind (day-of-month string-position) (parse-integer string :junk-allowed t) wed) Questionable: what if memory runs out? (ignore-errors (parse-date string)) Better: catches expected errors only (handler-case (parse-date string) (parse-error nil)) 282. Tips on Built-in Functionality Writing Good Error Messages Use full sentences in error messages (uppercase initial, trailing period). No “Error: "or";;" prefix. The system will sup- ply such a prefix if needed. Do not begin an error message with a request for a fresh line. The system will do this automatically if necessary. As with other format strings, don’t use embedded tab characters. Don’t mention the consequences in the error mes- sage. Just describe the situation itself. Don’t presuppose the debugger’s user interface in describing how to continue. This may cause porta- bility problems since different implementations use different interfaces. Just describe the abstract ef- fect of a given action. Specify enough detail in the message to distinguish it from other errors, and if you can, enough to help you debug the problem later if it happens. 292. Tips on Built-in Functionality Writing Good Error Messages (cont’d) Bad: (error '“%>> Error: Foo. Type :C to continue.") Better: (cerror "Specify a replacement sentence interactively." "An ill-formed sentence was encountered:~% ~A" sentence) 302. Tips on Built-in Functionality Using the Condition System Start with these: error, cerror warn handler-case with-simple-restart unwind-protect Good: standard use of warn (defvar *word* ’?? "The word we are currently working on.") (defun lex-warn (format-str &rest args) “Lexical warning; like warn, but first tells what word caused the warning." (warn "For word “a: ~?" *word* format-str args)) 312. Tips on Built-in Functionality HANDLER-CASE, WITH-SIMPLE-RES TART Good: handle specific errors (defun eval-exp (exp) "If possible evaluate this exp; otherwise return it." 33 Guard against errors in evaluating exp Chandler-case (if (and (fboundp (op exp)) (every #’is-constant (args exp))) (eval exp) exp) (arithmetic-error () exp))) Good: provide restarts (defun top-level (&key (prompt "=> ") (read #’read) (eval #’eval) (print #’print)) "A read-eval-print loop." (with-simple-restart (abort "Exit out of the top level.") (loop (with-simple-restart (abort "Return to top level loop.") (format t "“&"a" prompt) (funcall print (funcall eval (funcall read))))))) 322. Tips on Built-in Functionality UNWIND-PROTECT unwind-protect implements important functionality that everyone should know how to use. It is not just for sys- tem programmers. Watch out for multi-tasking, though. For example, im- plementing some kinds of state-binding with unwind=protect might work well in a single-threaded environment, but in an environment with multi-tasking, you often have to be a little more careful. (unwind-protect (progn form, form ... formn) cleanup, Cleanup ... Cleanupn) e Never assume form, will get run at all. e Never assume form, won’t run to completion. 332. Tips on Built-in Functionality UNWIND-PROTECT (cont'd) Often you need to save state before entering the unwind-protect and test before you restore state: Possibly Bad: (with multi-tasking) (catch ’robot-op (unwind-protect (progn (turn-on-motor) (manipulate) ) (turn-off-motor))) Good: (safer) (catch *robot-op (let (status (motor-status motor))) Cunwind-protect (progn (turn-on-motor motor) (manipulate motor)) (when (motor-on? motor) (turn-off-motor motor)) (setf (motor-status motor) status))))2. Tips on Built-in Functionality I/O Issues: Using FORMAT Don’t use Tab characters in format strings (or any strings intended for output). Depending on what column your output starts in, the tab stops may not line up the same on output as they did in the code! Don’t use "#<~S ~A>" to print unreadable objects. Use print-unreadable-object instead. Consider putting format directives in uppercase to make them stand out from lowercase text sur rounding. For example, "Foo: “A" instead Of "Foo: “a". Learn useful idioms. For example: ~{“A**, ~} and “ip. Be conscious of when to use ~& versus ~%. Also, "2%" and "“2&"' are also handy. Most code which outputs a single line should start with ~& and end with ~%. (format t "“&This is a test.~%") This is a test. Be aware of implementation extensions. They may not be portable, but for non-portable code might be very useful. For example, Genera’s — and — for handling indentation. 352. Tips on Built-in Functionality Using Streams Correctly e *standard-output* and *standard-input* Vs *terminal-io* Do not assume *standard-input* and *standard-output* will be bound to *terminal-io* (or, in fact, to any interactive stream). You can bind them to such a stream, however. Try not to use *terminal-io* directly for input or output. It is primarily available as a stream to which other streams may be bound, or may indi- rect (e.g., by synonym streams). e@ *error-output* VS *debug-io* Use *error-output* for warnings and error mes- sages that are not accompanied by any user inter- action. Use *debug-io* for interactive warnings, error mes- sages, and other interactions not related to the normal function of a program. In particular, do not first print a message on *error-output* and then do a debugging session on *debug-iot, ex- pecting those to be the same stream. Instead, do each interaction consistently on one stream. 362. Tips on Built-in Functionality Using Streams Correctly (cont’d) @ xtrace-output* This can be used for more than just receiving the output of trace. If you write debugging routines that conditionally print helpful information with- out stopping your running program, consider do- ing output to this stream so that if *trace-output* is redirected, your debugging output will too. A useful test: If someone re-bound only one of several 1/O streams you are using, would it make your output look stupid? 373. Tips on Near-Standard Tools Using Near-Standard Tools Some functionality is not built in to the language, but is used by most programmers. This divides into exten- sions to the language and tools that help you develop programs. Extensions e defsystem to define a program e CLIM, CLX, etc. graphics libraries Tools e@ emacs from FSF, Lucid indentation, font/color support definition /arglist/doc/regexp finding communication with lisp e xref, manual, etc. from CMU e Browsers, debuggers, profilers from vendors 383. Tips on Near-Standard Tools DEFSYSTEM Pick a public domain version of defsystem (unfortu- nately, dpANS CL has no standard). e Put absolute pathnames in one place only e Load everything through the defsystem e Distinguish compiling from loading e Optionally do version control (defpackage "PARSER" ...) (defsystem parser (:source "/lab/indexing/parser/*") (:parts utilities " ammar" "tokenizer" “optimize: #+CLIM "clim-graphics" #+CLX "clx-graphics")) @ Make sure your system loads with no compiler warnings (first time and subsequent times) (learn to use (declare (ignore ...))) @ Make sure the system can be compiled from scratch (eliminate lingering bootstrapping problems) 393. Tips on Near-Standard Tools Editor Commands Your editor should be able to do the following: Move about by s-expressions and show matching parens Indent code properly Find unbalanced parens Adorn code with fonts and colors Find the definition of any symbol Find arguments or documentation for any symbol Macroexpand any expression Send the current expression, region or file to Lisp to be evaluated or compiled Keep a history of commands sent to Lisp and allow you to edit and resend them Work with keyboard, mouse, and menus Emacs can do all these things. If your editor can’t, complain until it is fixed, or get a new one. 403. Tips on Near-Standard Tools Emacs: Indentation and Comments Don’t try to indent yourself. Instead, let the editor do it. A near-standard form has evolved. e 80-column maximum width e Obey comment conventions ; for inline comment 33 for in-function comment 333 for between-function comment 3333 for section header (for outline mode) @ cl-indent library can be told how to indent (put ’defvar ’common-lisp-indent-function °(4 2 2)) e@ lemacs can provide fonts, color (hilit: :modes-list-update "Lisp" 2(C"3 3.4" nil hilit2) ...)) 414. Abstraction Abstraction All programming languages allow the programmer to define abstractions. All modern languages provide sup- port for: e Data Abstraction (abstract data types) e Functional Abstraction (functions, procedures) Lisp and other languages with closures (e.g., ML, Sather) support: e Control Abstraction (defining iterators and other new flow of control constructs) Lisp is unique in the degree to which it supports: e Syntactic Abstraction (macros, whole new lan- guages) 424. Abstraction Design: Where Style Begins “The most important part of writing a program is de- signing the data structures. The second most impor- tant part is breaking the various code pieces down.” — Bill Gates “Expert engineers stratify complex designs. ... The parts constructed at each level are used as primitives at the next level. Each level of a stratified design can be thought of as a specialized language with a variety of primitives and means of combination appropriate to that level of detail.” — Harold Abelson and Gerald Sussman “Decompose decisions as much as possible. Untangle aspects which are only seemingly independent. Defer those decisions which concern details of representation as long as possible.” — Niklaus Wirth Lisp supports all these approaches: e Data Abstraction: classes, structures, deftype e Functional Abstraction: functions, methods Interface Abstraction: packages, closures e Object-Oriented: CLOS, closures e Stratified Design: closures, all of above e Delayed Decisions: run-time dispatch 434. Abstraction Design: Decomposition “& Lisp procedure is like a paragraph.” - Deborah Tatar “You should be able to explain any module in one sen- tence.” — Wayne Ratliff Strive for simple designs Break the problem into parts Design useful subparts (stratified) Be opportunistic; use existing tools Determine dependencies Re-modularize to reduce dependencies Design most dependent parts first will cover the following kinds of abstraction: Data abstraction Functional abstraction Control abstraction Syntactic abstraction4.1. Data Abstraction Data Abstraction Write code in terms of the problem’s data types, not the types that happen to be in the implementation. e Use defstruct or defclass for record types e Use inline functions as aliases (not macros) e Use deftype e Use declarations and :type slots for efficiency and/or documentation e Variable names give informal type information Pretty Good: specifies some type info (defclass event () ((starting-time :type integer) (location :type location) (duration :type integer :initform 0))) Better: problem-specific type info (deftype time () "Time in seconds" ’ integer) (defconstant +the-dawn-of-timet 0 "Midnight, January 1, 1900" (defclass event () ((starting-time :type time :initform +the-dawn-of-time+) Clocation :type location) (duration :type time :initform 0))) 454.1. Data Abstraction Use Abstract Data Types Introduce abstract data types with accessors: Bad: obscure accessor, eval (if (eval (cadar rules)) ...) Better: introduce names for accessors (declaim (inline rule-antecedent)) (defun rule-antecedent (rule) (second rule)) (if (holds? (rule-antecedent (first rules))) ...) Usually Best: introduce first-class data type (defstruct rule name antecedent consequent) or (defstruct (rule (:type list)) name antecedent consequent) or (defclass rule () (name antecedent consequent)) 464.1. Data Abstraction Implement Abstract Data Types Know how to map from common abstract data types to Lisp implementations. Set: list, bit-vector, integer, any table type Sequence: list, vector, delayed-evaluation stream Stack: list, vector (with fill-pointer) Queue: tconc, vector (with fill-pointer) Table: hash table, alist, plist, vector Tree, Graph: cons, structures, vector, adjacency matrix Use implementations that are already supported (e.g. union, intersection, length for sets as lists; logior, logand, logcount for sets as integers. Don’t be afraid to build a new implementation if pro- filing reveals a bottleneck. (If Common Lisp’s hash tables are too inefficient for your application, consider building a specialized hash table in Lisp before you build a specialized hash table in C.) aT4.1. Data Abstraction Inherit from Data Types Reuse by inheritance as well as direct use e structures support single inheritance e classes support multiple inheritance e both allow some over-riding e classes support mixins Consider a class or structure for the whole program e Eliminates clutter of global variables e Thread-safe e Can be inherited and modified 484.2. Functional Abstraction Functional Abstraction Every function should have: e A single specific purpose If possible, a generally useful purpose e A meaningful name (names like recurse-aux indicate problems) A structure that is simple to understand An interface that is simple yet general enough As few dependencies as possible A documentation string 494.2. Functional Abstraction Decomposition Decompose an algorithm into functions that are simple, meaningful and useful. Example from comp.lang.lisp discussion of loop vs. map: (defun least-common-superclass (instances) (let ((candidates (reduce #’ intersection (mapcar #? (lambda (instance) (clos:class-precedence-list (class-of instance))) instances) )) (best-candidate (find-class t))) (mapl #? lambda (candidates) let ((current-candidate (first candidates)) (remaining-candidates (rest candidates))) (when (and (subtypep current-candidate best-candidate) (every #? (lambda (remaining-candidate) (subtypep current-candidate remaining-candidate)) remaining-candidates) ) (setf£ best-candidate current-candidate)))) candidates) best-candidate)) 504.2. Functional Abstraction Decomposition Very Good: Chris Riesbeck (defun least-common-superclass (instances) (reduce #’more-specific-class (common-superclasses instances) rinitial-value (find-class *t))) (defun common-superclasses (instances) (reduce #?intersection (superclass-lists instances) )) (defun superclass-lists (instances) (loop for instance in instances collect (clos:class-precedence-list (class-of instance) ))) (defun more-specific-class (classi class2) (if (subtypep class2 classi) class2 class1)) e Each function is very understandable e Control structure is Clear: Two reduces, an intersection and a loop/collect e But reusablity is fairly low 514.2. Functional Abstraction Decomposition Equally Good: and more reusable (defun least-common-superclass (instances) "Find a least class that all instances belong to." (least-upper-bound (mapcar #’class-of instances) #?clos:class-precedence-list #subtypep) ) (defun least-upper-bound (elements supers sub?) "Element of lattice that is a super of all elements." (reduce #? (lambda (x y) (binary-least-upper-bound x y supers sub?)) elements)) (defun binary-least-upper-bound (x y supers sub?) "Least upper bound of two elements." (reduce-if sub? (intersection (funcall supers x) (funcall supers y)))) (defun reduce-if (pred sequence) "E.g. (reduce-if #’> numbers) computes maximum" (reduce #?(lambda (x y) (if (funcall pred x y) x y)) sequence) ) e Individual functions remain understandable e Still 2 reduces, an intersection and a mapcar e Stratified design yields more useful functions 524.2. Functional Abstraction Rule of English Translation To insure that you say what you mean: 1. 2. 3. 4. Start with an English description of algorithm Write the code from the description Translate the code back into English Compare 3 to 1 Example: . “Given a list of monsters, determine the number that are swarms.” 2. (defun count-swarm (monster-list) 3. (apply ’+ (mapcar #? lambda (monster) Gif (equal (object-type (get-object monster) ) swarm) 1 0)) monster-list))) “Take the list of monsters and produce a 1 for a monster whose type is swarm, and a O for the others. Then add up the list of numbers.” 534.2. Functional Abstraction Rule of English Translation Better: 1. “Given a list of monsters, determine the number that are swarms.” 2. (defun count-swarms (monster-names) "Count the swarms in a list of monster names." (count-if #’swarm-p monster-names :key #’get-object)) or (count ’swarm monster-names :key #’get-object-type) or (loop for name in monster-names count (swarm-p (get-object monster))) 3. “Given a list of monster names, count the number that are swarms.” 544.2. Functional Abstraction Use Library Functions Libraries may have access to low-level efficiency hacks, and are often fine-tuned. BUT they may be too general, hence inefficient. Write a specific version when efficiency is a problem. Good: specific, concise (defun find-character (char string) "See if the character appears in the string." (find char string)) Good: _ efficient (defun find-character (char string) "See if the character appears in the string." (declare (character char) (simple-string string)) (loop for ch across string when (eql ch char) return ch)) 554.2. Functional Abstraction Use Library Functions Given build1, which maps n to a list of n x’s: (buildi 4) > (x x x x) Task: Define build-it so that: (build-it ?(40 3)) > (xxx x) O @ x x)) Incredibly Bad: (defun round3 (x) (let ((result °())) (dotimes (n (length x) result) (setq result (cons (car (nthcdr n x)) result))))) (defun build-it (arg-list) Get (result ?O)) (dolist (a (round3 arg-list) result) (setq result (cons (buildi a) result))))) Problems: e round3 is just another name for reverse © (car (athcdr n x)) is (nth n x) e dolist would be better than dotimes here © push would be appropriate here e (mapcar #’buildi numbers) does it all 564.3. Control Abstraction Control Abstraction Most algorithms can be characterized as: Searching (some find find-if mismatch) Sorting (sort merge remove-duplicates) Filtering (remove remove-if mapcan) Mapping (map mapcar mapc) Combining (reduce mapcan) Counting (count count-if) These functions abstract common control patterns. Code that uses them is: Concise Self-documenting Easy to understand Often reusable Usually efficient (Better than a non-tail recursion) Introducing your own control abstraction is an impor- tant part of stratified design. 574.3. Control Abstraction Recursion vs. Iteration Recursion is good for recursive data structures. Many people prefer to view a list as a sequence and use iter- ation over it, thus de-emphasizing the implementation detail that the list is split into a first and rest. AS an expressive style, tail recursion is often considered elegant. However, Common Lisp does not guarantee tail recursion elimination so it should not be used as a substitute for iteration in completely portable code. (In Scheme it is fine.) The Common Lisp do macro can be thought of as syn- tactic sugar for tail recursion, where the initial values for variables are the argument values on the first func- tion call, and the step values are argument values for subsequent function calls. do provides a low level of abstraction, but versatile and has a simple, explicit execution model. 584.3. Control Abstraction Recursion vs. Iteration (cont’d) Bad: (in Common Lisp) (defun any (1st) (cond (Cull 1st) nil) ((car 1st) t) (t (any (cdr 1st))))) Better: conventional, concise (defun any (list) "Return true if any member of list is true." (some #?not-null list)) or (find-if-not #’null 1st) or (loop for x in list thereis x) or (explicit) (do ((list list (rest list))) (@ull list) nil) (when (first list)) (return t)))) Best: efficient, most concise in this case Don’t call any at alll! Use (some p list) instead of (any (mapcar p list)) 594.3. Control Abstraction LOOP “Keep a loop to one topic—like a letter to your Senator.” — Judy Anderson The Common Lisp loop macro gives you the power to express idiomatic usages concisely. However it bears the burden that its syntax and semantics are often sub- stantially more complex than its alternatives. Whether or not to use the loop macro is an issue sur- rounded in controversy, and borders on a religious war. At the root of the conflict is the following somewhat paradoxical observation: e loop appeals to naive programmers because it looks like English and seems to call for less knowledge of programming than its alternatives. @ loop is not English; its syntax and semantics have subtle intricacies that have been the source of many programming bugs. It is often best used by people who’ve taken the time to study and un- derstand it—usually not naive programmers. Use the unique features of loop (e.g., parallel iteration of different kinds). 604.3. Control Abstraction Simple Iteration Bad: verbose, control structure unclear (LOOP (SETQ *WORD* (POP *SENTENCE*) ) 3get the next word (COND 33; if no more words then return instantiated CD form 33 which is stored in the variable *CONCEPT* (CNULL *WORD*) (RETURN (REMOVE-VARIABLES (VAR-VALUE ?*CONCEPT*)))) (T (FORMAT T "~%~%Processing ~A" *WORD*) (LOAD-DEF) ; look up requests under ; this word (RUN-STACK))))) ; fire requests e No need for global variables e End test is misleading e Not immediately clear what is done to each word Good: conventional, concise, explicit (mapc #’process-word sentence) (remove-variables (var-value ’*concept*)) (defun process-word (word) (format t "~2%Processing ~A" word) (load-def word) (run-stack)) 614.3. Control Abstraction Mapping Bad: verbose 3 (extract-id-list ’?l_user-recs) ------------- [lambda] ; WHERE: l_user-recs is a list of user records ; RETURNS: a list of all user id’s in l_user-recs 3 USES: extract-id 3 USED BY: process-users, sort-users (defun extract-id-list (user-recs) (prog (id-list) loop (cond ((null user-recs) 33 id-list was constructed in reverse order 33 using cons, so it must be reversed now: (return (nreverse id-list)))) (setq id-list (cons (extract-id (car user-recs)) id-list)) (setq user-recs (cdr user-recs)) ;next user record (go loop))) Good: conventional, concise (defun extract-id-list (user-record-list) "Return the user ID’s for a list of users." (mapcar #?extract-id user-record-list)) 624.3. Control Abstraction Counting Bad: verbose (defun size 0 (prog (size idx) (setq size 0 idx 0) loop (cond ((< idx table-size) (setq size (+ size (length (aref table idx))) idx (1+ idx)) (go loop))) (return size))) Good: conventional, concise (defun table-count (table) ; Formerly called SIZE "Count the number of keys in a hash-like table." (reduce #?+ table :key #?length)) Also, it couldn’t hurt to add: (deftype table () "A table is a vector of buckets, where each bucket holds an alist of (key . values) pairs." ? (vector cons)) 634.3. Control Abstraction Filtering Bad: verbose (defun remove-bad-pred-visited (1 badpred closed) Returns a list of nodes in L that are not bad 333 amd are not in the CLOSED list. (cond ((null 1) 1) (Cor (funcall badpred (car 1)) (member (car 1) closed)) (remove-bad-pred-visited (cdr 1) badpred closed) ) (t (cons (car 1) (remove-bad-pred-visited (cdr 1) badpred closed))))) Good: conventional, concise (defun remove-bad-or-closed-nodes (nodes bad-node? closed) "Remove nodes that are bad or are on closed list" (remove-if #? (lambda (node) (or (funcall bad-node? node) (member node closed) )) nodes) )4.3. Control Abstraction Control Flow: Keep It Simple Non-local control flow is hard to understand Bad: verbose, violates referential transparency (defun isa-test (x y n) (catch ’isa (isa-test1 x y n))) (defun isa-testi (x y n) (cond ((eq x y) t) (Qmember y (get x ’isa)) (throw ’isa t)) ((zerop n) nil) (t (any (mapcar #? lambda (xx) (isa-test xx y (i- n)) ) (get x isa) ))) ) ) Problems: catch/throw is gratuitous member test may or may not be helping mapcar generates garbage any tests too late; throw tries to fix this result is that any never gets called! 654.3. Control Abstraction Keep It Simple Some recommendations for use of catch and throw: e Use catch and throw as sub-primitives when imple- menting more abstract control structures as macros, but do not use them in normal code. e@ Sometimes when you establish a catch, programs may need to test for its presence. In that case, restarts may be more appropriate. 664.3. Control Abstraction Keep It Simple Good: (defun isa-test (sub super max-depth) "Test if SUB is linked to SUPER by a chain of ISA links shorter than max-depth." (and (>= max-depth 0) (or (eq sub super) (some #? (lambda (parent) (isa-test parent super (- max-depth 1))) (get sub ’isa))))) Also good: uses tools (defun isa-test (sub super max-depth) (depth-first-search :start sub :goal (is super) :successors #’get-isa :max-depth max-depth)) “Write clearly—don’t be too clever.” — Kernighan & Plauger Be Aware: Does “improving” something change the semantics? Does that matter? 674.3. Control Abstraction Avoid Complicated Lambda Expressions When a higher-order function would need a compli- cated lambda expression, consider alternatives: dolist or loop generate an intermediate (garbage) sequence Series Macros or read macros local function — Specific: makes it clear where function is used — Doesn’t clutter up global name space — Local variables needn’t be arguments — BUT: some debugging tools won’t work 684.3. Control Abstraction Avoid Complicated Lambda Expressions Find the sum of the squares of the odd numbers in a list of integers: All Good: (reduce #?+ numbers rkey #? (lambda (x) (if (oddp x) (* x x) 0))) (flet ((square-odd (x) (if (oddp x) (* x x) 0))) (reduce #?+ numbers :key #’square-odd)) (loop for x in list when (oddp x) sum (* x x)) (collect-sum (choose-if #’oddp numbers) ) Also consider: (may be appropriate sometimes) 3; Introduce read macro: (reduce #’+ numbers :key #L(if (oddp _) (* _ _) 0)) 33 Generate intermediate garbage: (reduce #’+ (remove #’evenp (mapcar #’square numbers))) 694.3. Control Abstraction Functional vs. Imperative Style It has been argued that imperative style programs are harder to reason about. Here is a bug that stems from an imperative approach: Task: Write a version of the built-in function find. Bad: _ incorrect (defun i-find (item seq &key (test #’eql) (test-not nil) (start O s-flag) (end nil) (key #?identity) (from-end nil)) Gf s-flag (setq seq (subseq seq start))) (if end (setq seq (subseq seq 0 end))) seed Problems: e Taking subsequences generates garbage e No appreciation of list/vector differences e Error if both start and end are given Error stems from the update to seq 704.3. Control Abstraction Example: Simplification Task: a simplifier for logical expressions: (simp ’(and (and a b) (and (or ¢ (or d e)) f))) => (AND AB (ORC D E) F) Not bad, but not perfect: (defun simp (pred) (cond ((atom pred) pred) ((eq (car pred) ’and) (cons ’and (simp-aux ’and (cdr pred)))) ((eq (car pred) ’or) (cons ’or (simp-aux ’or (cdr pred)))) (t pred))) (defun simp-aux (op preds) (cond ((null preds) nil) (Cand (listp (car preds)) (eq (caar preds) op)) (append (simp-aux op (cdar preds)) (simp-aux op (cdr preds)))) (t (cons (simp (car preds)) (simp-aux op (cdr preds)))))) 714.3. Control Abstraction A Program to Simplify Expressions Problems: e No meaningful name for simp-aux e No reusable parts e No data accessors e (and), (and a) not simplified Better: usable tools (defun simp-bool (exp) "Simplify a boolean (and/or) expression." (cond ((atom exp) exp) ((member (op exp) ’ (and or)) (maybe-add (op exp) (collect-args (op exp) (mapcar #’simp-bool (args exp))))) (t exp))) (defun collect-args (op args) "Return the list of args, splicing in args that have the given operator, op. Useful for simplifying exps with associate operators." (loop for arg in args when (starts-with arg op) nconc (collect-args op (args arg)) else collect arg)) 724.3. Control Abstraction Build Reusable Tools (defun starts-with (list element) "Is this a list that starts with the given element?" (and (consp list) (eql (first list) element))) (defun maybe-add (op args &optional (default (get-identity op))) "If 1 arg, return it; if 0, return the default. If there is more than 1 arg, cons op on them. Example: (maybe-add *progn ’((f x))) ==> (£ x) Example: (maybe-add ’* ’(3 4)) ==> (* 3 4). Example: (maybe-add ’+ ’()) ==> 0, assuming 0 is defined as the identity for +." (cond ((null args) default) (Gength=1 args) (first args)) (t (cons op args)))) (deftable identity sinit ?((+ 0) (* 1) (and t) (or nil) (progn nil))) 734.4. Syntactic Abstraction A Language for Simplifying Task: A Simplifier for all Expressions: (simplify ?(* 1 (+ x (- y y)))) (simplify ?(if (= 0 1) (£ x))) (simplify ’(and a (and (and) b))) Syntactic abstraction defines a new language that is appropriate to the problem. This is a problem-oriented (as opposed to code-oriented) approach. Define a language for simplification rules, then write some: (define-simplifier exp-simplifier (G4 x 0) x) (G+ 0 x) x) ((- x 0) x) ((- x x) 0) (Gif nil x y) (Gf x y y) 744.4. Syntactic Abstraction Design Your Language Carefully “The ability to change notations empowers human beings.” — Scott Kim Bad: verbose, brittle (setq timesO-rule ’( simplify (* @ e1) 0) ° timesO-rule >) (setq rules (list timesO-rule ...)) Insufficient abstraction Requires naming timesO-rule three times Introduces unneeded global variables e e e e Unconventional indentation Sometimes it is useful to name rules: (defrule timesO-rule (* ?x 0) ==> 0) (Although I wouldn’t recommend it in this case.) 754.4. Syntactic Abstraction An Interpreter for Simplifying Now write an interpreter (or a compiler): (defun simplify (exp) "Simplify expression by first simplifying components." (if (atom exp) exp (simplify-exp (mapcar #’simplify exp)))) (defun-memo simplify-exp (exp) "Simplify expression using a rule, or math." 33 The expression is non-atomic. (rule-based-translator exp *simplification-rules* :rule-pattern #’first :rule-response #’third ction #’simplify sotherwise #’eval-exp) ) This solution is good because: Simplification rules are easy to write Control flow is abstracted away (mostly) It is easy to verify the rules are correct ‘The program can quickly be up and running. If the approach is sufficient, we’re done. If the approach is insufficient, we’ve saved time. Tf it is just slow, we can improve the tools, and other uses of the tools will benefit too. 764.4. Syntactic Abstraction An Interpreter for Translating “Success comes from doing the same thing over and over again; each time you learn a little bit and you do a little better the next time.” — Jonathan Sachs Abstract out the rule-based translator: (defun rule-based-translator Cinput rules &key (matcher #’pat-match) (rule-pattern #’first) (rule-response #’rest) (action #identity) (sub #’sublis) (otherwise #?identity)) "Find the first rule that matches input, and apply the action to the result of substituting the match result into the rule’s response. If no rule matches, apply otherwise to the input." (loop for rule in rules for result = (funcall matcher (funcall rule-pattern rule) input) when (not (eq result fail)) do (RETURN (funcall action (funcall sub result (funcall rule-response rule)))) finally (RETURN (funcall otherwise input)))) If this implementation is too slow, we can index better or compile. Sometimes, reuse is at an informal level: seeing how the general tool is built allows a programmer to con- struct a custom tool with cut and paste. 774.4. Syntactic Abstraction Saving duplicate work: defun-memo Less extreme than defining a whole new language is to augment the Lisp language with new macros. defun-memo makes a function remember all computa- tions it has made. It does this by maintaining a hash table of input/output pairs. If the first argument is just the function name, 1 of 2 things happen: [1] If there is exactly 1 arg and it is not a &rest arg, it makes a eql table on that arg. [2] Otherwise, it makes an equal table on the whole arglist. You can also replace fn-name with (name :test ... :size tkey-exp ...). This makes a table with given test and size, indexed by key-exp. The hash table can be cleared with the clear-memo function. Examples: (defun-memo f (x) j (complex-computation x)) eql table keyed on x (defun-memo (f :test #eq) (x) ;; eq table keyed on x (complex-computation x)) (defun-memo g (x y z) 33 equal table (another-computation x y z)) ;; keyed on on (xy . z) (defun-memo (h :key-exp x) (x optional debug?) 33 eql table keyed on x oe) 78