Metaprogramming for Erlang

Abstract Format & Core Salvador Tamarit Mu noz

Universitat Polit` ecnica de Val` encia

Contents
1

Abstract Format Description How to get it Manipulating Abstract Format Get the Abstract Format from an Erlang expression Smerl Evaluating Abstract Format Libraries Core Erlang Description Syntax How to get it From Erlang to Core Manipulating Core Erlang Get the Core Erlang from an Erlang expression Get the Core Erlang from an external module Retrieving Erlang Code from Core Erlang Libraries Conclusions

Abstract Format

Abstract Format

Description

Description
The following Erlang expression: it is represented in abstract format by the following tuple:

The tuple forms a tree structure:

Salvador Tamarit Mu noz (UPV)

Abstract Format & Core Erlang

U. Polit` ecnica de Val` encia

4 / 29

Abstract Format

Description

Description
The following Erlang expression: it is represented in abstract format by the following tuple:

The tuple forms a tree structure:

Salvador Tamarit Mu noz (UPV)

Abstract Format & Core Erlang

U. Polit` ecnica de Val` encia

4 / 29

Abstract Format

Description

Description
The following Erlang expression: it is represented in abstract format by the following tuple:

The tuple forms a tree structure:

Salvador Tamarit Mu noz (UPV)

Abstract Format & Core Erlang

U. Polit` ecnica de Val` encia

4 / 29

Abstract Format

Description

Description
A module is represented with a list [F1 ...Fn ], where each F represents a form. A form is either an attribute or a function declaration. Concretely:
For instance, the abstract format corresponding to the attribute -module(Mod) is {attribute,LINE,module,Mod}. The abstract format corresponding to a function declaration is {function,LINE,Name,Arity,[FC1 ...FCn ]} where each FC is the abstract format of a function clause, wich in turn is represented by {clause,LINE,[P1 ...Pn ],[G1 ...Gn ],[E1 ...En ]} where each P, G and E is the abstract representation of one of its pattern, one of its guards and one of its bodys expressions respectively.

The whole abstract format description is publicly available at: http://www.erlang.org/doc/apps/erts/absform.html

Salvador Tamarit Mu noz (UPV)

Abstract Format & Core Erlang

U. Polit` ecnica de Val` encia

5 / 29

Abstract Format

How to get it

How to get it

One line of code:

Lets see an example: factorial.erl

Salvador Tamarit Mu noz (UPV)

Abstract Format & Core Erlang

U. Polit` ecnica de Val` encia

6 / 29

Abstract Format

How to get it

How to get it

One line of code:

Lets see an example: factorial.erl

Salvador Tamarit Mu noz (UPV)

Abstract Format & Core Erlang

U. Polit` ecnica de Val` encia

6 / 29

Abstract Format

Manipulating Abstract Format

Manipulating Abstract Format

We want to write a program transformation to replace each occurrence of an append operator (++) where its left operand is single element list (e.g. [1]) with a list construction. For instance: [1] ++ [2,3] [1 | [2,3]] Additionally, we will replace append operations where one of their operand is an empty list with its other operand. For instance: [] ++ [2,3] [2,3] [2,3] ++[] [2,3] The code is in: refactorer.erl And a test with some cases in: refactorer test.erl
Salvador Tamarit Mu noz (UPV) Abstract Format & Core Erlang U. Polit` ecnica de Val` encia 7 / 29

Abstract Format

Manipulating Abstract Format

Manipulating Abstract Format

We want to write a program transformation to replace each occurrence of an append operator (++) where its left operand is single element list (e.g. [1]) with a list construction. For instance: [1] ++ [2,3] [1 | [2,3]] Additionally, we will replace append operations where one of their operand is an empty list with its other operand. For instance: [] ++ [2,3] [2,3] [2,3] ++[] [2,3] The code is in: refactorer.erl And a test with some cases in: refactorer test.erl
Salvador Tamarit Mu noz (UPV) Abstract Format & Core Erlang U. Polit` ecnica de Val` encia 7 / 29

Abstract Format

Manipulating Abstract Format

Manipulating Abstract Format

We can write a simpler version using function erl syntax lib:map/2: refactorer simpler.erl We have been accessing directly to the raw abstract format. We can use library erl syntax to avoid having to do so: refactorer simpler alt.erl

Salvador Tamarit Mu noz (UPV)

Abstract Format & Core Erlang

U. Polit` ecnica de Val` encia

8 / 29

Abstract Format

Manipulating Abstract Format

Manipulating Abstract Format

We can write a simpler version using function erl syntax lib:map/2: refactorer simpler.erl We have been accessing directly to the raw abstract format. We can use library erl syntax to avoid having to do so: refactorer simpler alt.erl

Salvador Tamarit Mu noz (UPV)

Abstract Format & Core Erlang

U. Polit` ecnica de Val` encia

8 / 29

Abstract Format

Manipulating Abstract Format

Manipulating Abstract Format

If we want easily reuse our transformations, we can use parse transformations. In order to implement a parse transformation, we need to write in our module a function namedparse transform/2 and export it. refactorer pt.erl Then, we can reuse this transformation via a complier option: c(refactorer test,[{parse transform,refactorer pt}]). The loaded code will have its content changed according to the transformation dened in refactorer pt. When a program transformation is faced for rst time, it is a good idea to take a look in the following module to consult the cases to be treated: erl id trans.erl
Salvador Tamarit Mu noz (UPV) Abstract Format & Core Erlang U. Polit` ecnica de Val` encia 9 / 29

Abstract Format

Manipulating Abstract Format

Manipulating Abstract Format

If we want easily reuse our transformations, we can use parse transformations. In order to implement a parse transformation, we need to write in our module a function namedparse transform/2 and export it. refactorer pt.erl Then, we can reuse this transformation via a complier option: c(refactorer test,[{parse transform,refactorer pt}]). The loaded code will have its content changed according to the transformation dened in refactorer pt. When a program transformation is faced for rst time, it is a good idea to take a look in the following module to consult the cases to be treated: erl id trans.erl
Salvador Tamarit Mu noz (UPV) Abstract Format & Core Erlang U. Polit` ecnica de Val` encia 9 / 29

Abstract Format

Manipulating Abstract Format

Manipulating Abstract Format

If we want easily reuse our transformations, we can use parse transformations. In order to implement a parse transformation, we need to write in our module a function namedparse transform/2 and export it. refactorer pt.erl Then, we can reuse this transformation via a complier option: c(refactorer test,[{parse transform,refactorer pt}]). The loaded code will have its content changed according to the transformation dened in refactorer pt. When a program transformation is faced for rst time, it is a good idea to take a look in the following module to consult the cases to be treated: erl id trans.erl
Salvador Tamarit Mu noz (UPV) Abstract Format & Core Erlang U. Polit` ecnica de Val` encia 9 / 29

Abstract Format

Get the Abstract Format from an Erlang expression

Get the Abstract Format from an Erlang expression

It is very easy to get the abstract format corresponding to any expression:

Salvador Tamarit Mu noz (UPV)

Abstract Format & Core Erlang

U. Polit` ecnica de Val` encia

10 / 29

Abstract Format

Get the Abstract Format from an Erlang expression

Get the Abstract Format from an Erlang expression

It is very easy to get the abstract format corresponding to any expression:

Salvador Tamarit Mu noz (UPV)

Abstract Format & Core Erlang

U. Polit` ecnica de Val` encia

10 / 29

Abstract Format

Smerl

Smerl

Module smerl.erl provides functionality to simplify the metaprogramming task, along with some interesting features. Additionally, with smerl it is possible to create and compile easily a new module allowing to call its functions: smerl test.erl

Salvador Tamarit Mu noz (UPV)

Abstract Format & Core Erlang

U. Polit` ecnica de Val` encia

11 / 29

Abstract Format

Evaluating Abstract Format

Evaluating Abstract Format

We can use the functionality of module erl eval to evaluate abstract format structures:

Salvador Tamarit Mu noz (UPV)

Abstract Format & Core Erlang

U. Polit` ecnica de Val` encia

12 / 29

Abstract Format

Evaluating Abstract Format

Evaluating Abstract Format

We can use the functionality of module erl eval to evaluate abstract format structures:

Salvador Tamarit Mu noz (UPV)

Abstract Format & Core Erlang

U. Polit` ecnica de Val` encia

12 / 29

Abstract Format

Libraries

Libraries
There are several useful modules to work with the abstract form: erl syntax This module denes an abstract data type for representing Erlang source code as syntax trees, in a way that is backwards compatible with the data structures created by the Erlang standard library parser module erl parse. erl syntax lib This module contains utility functions for working with the abstract data type dened in the module erl syntax. erl prettypr Pretty printing of abstract Erlang syntax trees. erl id trans An Identity Parse Transform.
Salvador Tamarit Mu noz (UPV) Abstract Format & Core Erlang U. Polit` ecnica de Val` encia 13 / 29

Abstract Format

Libraries

Libraries
There are several useful modules to work with the abstract form: erl tidy Tidies and pretty-prints Erlang source code, removing unused functions, updating obsolete constructs and function calls, etc. epp dodger Bypasses the Erlang preprocessor - avoids macro expansion, le inclusion, conditional compilation, etc. Allows to nd/modify particular denitions/applications of macros, etc. igor It merges the source code of one or more Erlang modules into a single module, which can then replace the original set of modules.

Salvador Tamarit Mu noz (UPV)

Abstract Format & Core Erlang

U. Polit` ecnica de Val` encia

14 / 29

Core Erlang

Core Erlang

Description

Description

A program operating on source code must handle so many cases as to become impractical in general. Core Erlang was designed to overcome this issue. The compiler uses it as an intermediate representation between the source code and the byte code. Additionally it helps to perform some optimizations. Some interesting features are:
A strict, higher-order functional language. Simple and unambiguous grammar. Human-readable textual representation. Language easy to work with.

Salvador Tamarit Mu noz (UPV)

Abstract Format & Core Erlang

U. Polit` ecnica de Val` encia

16 / 29

Core Erlang

Syntax

Syntax
fname lit fun clause pat pats exprs expr ::= ::= ::= ::= ::= ::= ::= ::= Atom / Integer Atom | Integer | Float | Char | String | [ ] fun(var1 , . . . , varn ) -> exprs pats when exprs1 -> exprs2 var | lit | [ pats | pats ] | { pats1 , . . . , patsn } | var = pats pat | < pat, . . . , pat > expr | < expr, . . . , expr > var | fname | fun | [ exprs | exprs ] | { exprs1 , . . . , exprsn } | let vars = exprs1 in exprs2 | letrec fname1 = fun1 . . . fnamen = funn in exprs | apply exprs ( exprs1 , . . . , exprsn ) | call exprsn+1 :exprsn+2 ( exprs1 , . . . , exprsn ) | primop Atom ( exprs1 , . . . , exprsn ) | try exprs1 of <var1 , . . . , varn > -> exprs2 catch <var1 , . . . , varm > -> exprs3 | case exprs of clause1 . . . clausen end | do exprs1 exprs2 | catch exprs Exception(val m ) lit | fname | fun | [ vals | vals ] | {vals1 , . . . , valsn } | val | < val, . . . , val > var | < var, . . . , var >

val vals vars

::= ::= ::= ::=

Salvador Tamarit Mu noz (UPV)

Abstract Format & Core Erlang

U. Polit` ecnica de Val` encia

17 / 29

Core Erlang

How to get it

How to get it

There are two main ways to get the core representation of a module: In the interpreter, with the following instruction: > c(factorial,to core). In a source code, using the functionality of the module compile: {ok, ,Core} = compile:file("factorial.erl",[to core, binary]) Try it: factorial.erl

Salvador Tamarit Mu noz (UPV)

Abstract Format & Core Erlang

U. Polit` ecnica de Val` encia

18 / 29

Core Erlang

How to get it

How to get it

There are two main ways to get the core representation of a module: In the interpreter, with the following instruction: > c(factorial,to core). In a source code, using the functionality of the module compile: {ok, ,Core} = compile:file("factorial.erl",[to core, binary]) Try it: factorial.erl

Salvador Tamarit Mu noz (UPV)

Abstract Format & Core Erlang

U. Polit` ecnica de Val` encia

18 / 29

Core Erlang

From Erlang to Core

From Erlang to Core

It is important to know some basis of how the translation from Erlang to Core Erlang is done. It is going to be denitely very useful when treating the Core Erlang code. Let see an example: core transformations.erl There is a compiler option that let the resulting Core Erlang closer to the original Erlang source. We should use it when we are facing a problem where the correspondence between Core and Erlang is important. > c(core transformations, [to core, no copt])

Salvador Tamarit Mu noz (UPV)

Abstract Format & Core Erlang

U. Polit` ecnica de Val` encia

19 / 29

Core Erlang

From Erlang to Core

From Erlang to Core

It is important to know some basis of how the translation from Erlang to Core Erlang is done. It is going to be denitely very useful when treating the Core Erlang code. Let see an example: core transformations.erl There is a compiler option that let the resulting Core Erlang closer to the original Erlang source. We should use it when we are facing a problem where the correspondence between Core and Erlang is important. > c(core transformations, [to core, no copt])

Salvador Tamarit Mu noz (UPV)

Abstract Format & Core Erlang

U. Polit` ecnica de Val` encia

19 / 29

Core Erlang

Manipulating Core Erlang

Manipulating Core Erlang

Suppose we need to know all the variables names used in a module. We should only return the names of user dened variables, in other words, we should ignore variables introduces by Core (i.e. cor variables). Function names are also considered as variables names and they are represented by tuples of two elements. We want these function names to be ignored also. The code is in: core vars.erl As in the case of Abstract Format, there are some libraries that can simplify this code. In this case, library cerl trees contain some useful methods. We can get a simpler version of our previous code: core vars simpler.erl

Salvador Tamarit Mu noz (UPV)

Abstract Format & Core Erlang

U. Polit` ecnica de Val` encia

20 / 29

Core Erlang

Manipulating Core Erlang

Manipulating Core Erlang

Suppose we need to know all the variables names used in a module. We should only return the names of user dened variables, in other words, we should ignore variables introduces by Core (i.e. cor variables). Function names are also considered as variables names and they are represented by tuples of two elements. We want these function names to be ignored also. The code is in: core vars.erl As in the case of Abstract Format, there are some libraries that can simplify this code. In this case, library cerl trees contain some useful methods. We can get a simpler version of our previous code: core vars simpler.erl

Salvador Tamarit Mu noz (UPV)

Abstract Format & Core Erlang

U. Polit` ecnica de Val` encia

20 / 29

Core Erlang

Get the Core Erlang from an Erlang expression

Get the Core Erlang from an Erlang expression

We could need to get the Core Erlang corresponding to a single expression. We can build an auxiliary module, and then compile it to core. The following module shows how to get this: core from expression.erl

Salvador Tamarit Mu noz (UPV)

Abstract Format & Core Erlang

U. Polit` ecnica de Val` encia

21 / 29

Core Erlang

Get the Core Erlang from an external module

Get the Core Erlang from an external module

Sometimes we need to load Core Erlang from a module included in the OTP libraries. The following code allow us to get this: core from module.erl Note that this code can also load Abstract Format changing the call to function compile:file/2 in the the last expression to a call to function epp:parse file/3.

Salvador Tamarit Mu noz (UPV)

Abstract Format & Core Erlang

U. Polit` ecnica de Val` encia

22 / 29

Core Erlang

Get the Core Erlang from an external module

Get the Core Erlang from an external module

Sometimes we need to load Core Erlang from a module included in the OTP libraries. The following code allow us to get this: core from module.erl Note that this code can also load Abstract Format changing the call to function compile:file/2 in the the last expression to a call to function epp:parse file/3.

Salvador Tamarit Mu noz (UPV)

Abstract Format & Core Erlang

U. Polit` ecnica de Val` encia

22 / 29

Core Erlang

Retrieving Erlang Code from Core Erlang

Retrieving Erlang Code from Core Erlang

It is not easy to return from Core Erlang to Erlang. There are dierent problems: The language is dierent, and some Core constructions are not in Erlang and vice versa. Providing that the expression that we want to retrieve could be retrieved, there is still a problem. Its line is the only way to identify an expression as well as its type. Therefore, if two expression of the same type share the same line is impossible to know which is the one that we want to retrieve. For example, suppose we have this expression in one line of a Erlang module {{1,2},{3,4}}. During the Core processing, we want to retrieve a tuple from this line. We will have three possibilities.

Salvador Tamarit Mu noz (UPV)

Abstract Format & Core Erlang

U. Polit` ecnica de Val` encia

23 / 29

Core Erlang

Retrieving Erlang Code from Core Erlang

Retrieving Erlang Code from Core Erlang

The problem can be seen using this module: core to abstract bad.erl with this module as test: core to abstract test.erl

Salvador Tamarit Mu noz (UPV)

Abstract Format & Core Erlang

U. Polit` ecnica de Val` encia

24 / 29

Core Erlang

Retrieving Erlang Code from Core Erlang

Retrieving Erlang Code from Core Erlang

We can build a parse transformation to give to each individual expression a unique line, and then compile to Core from the resulting forms. The module implementing this transformation is: line changer pt.erl We can now modify the previous module to retrieve correctly the Erlang expression for a given Core Erlang: core to abstract.erl

Salvador Tamarit Mu noz (UPV)

Abstract Format & Core Erlang

U. Polit` ecnica de Val` encia

25 / 29

Core Erlang

Libraries

Libraries
Interesting modules when manipulating Core Erlang are: cerl This module denes an abstract data type for representing Core Erlang source code as syntax trees. cerl trees Basic functions on Core Erlang abstract syntax trees. cerl clauses Utility functions for Core Erlang case/receive clauses. cerl inline An implementation of the algorithm by Waddell and Dybvig (Fast and Eective Procedure Inlining, 1997), adapted to the Core Erlang.

Salvador Tamarit Mu noz (UPV)

Abstract Format & Core Erlang

U. Polit` ecnica de Val` encia

26 / 29

Core Erlang

Libraries

Libraries

Interesting modules when manipulating Core Erlang are: cerl prettypr Core Erlang prettyprinter. cerl closurean Closure analysis of Core Erlang programs. cerl pmatch Core Erlang pattern matching compiler. cerl typean Type analysis of Core Erlang programs.

Salvador Tamarit Mu noz (UPV)

Abstract Format & Core Erlang

U. Polit` ecnica de Val` encia

27 / 29

Conclusions

Conclusions
Abstract format It is easy to manipulate and understand. We have a lot of libraries to ease our work. We can evaluate its forms in two dierent ways. Core Erlang It is suitable for program analysis. It is simpler that Abstract Format but it stills human-readable. There exists a lot of libraries helping to work with it. It is possible to retrieve the corresponding Erlang expressions using some tricky methods.

Salvador Tamarit Mu noz (UPV)

Abstract Format & Core Erlang

U. Polit` ecnica de Val` encia

28 / 29

Thanks for listening