Software Language Engineering

8. Generators
8.1. Forms of Generators
see also MC Handbook, Ch. 12, 13

Prof. Dr. Bernhard Rumpe

Software Engineering
RWTH Aachen

http://www.se-rwth.de/
Farbe!
Definition: Generator and Code Generator

• A generator transforms a set of input files to a set of output files

Input files Output files

Generator

• A code generator transforms a set of input files to a set of source files in a programming language

Executable
Input files
Code generator source code

• Automatic generation is key

2 Software Engineering | RWTH Aachen

Code Generation

Code generation is the process of translating a source model into artefacts of an executable language.

• Forms of code generation, e.g.:

 Compiler
 Java to virtual machine
 C to byte code
 UML Statecharts to Java
 MatLab to C

• Code generation can be decomposed in several steps

 StateCharts  Java  Virtual Machine  Byte Code

3 Software Engineering | RWTH Aachen

Code Generator: Parameterization + Runtime System

• Generators are usually adaptable to varying target technology stacks

models target system: the product

manually Predefined
Predefined
written code predefined
components
components
components
parameterized
generator generated code
scripts + + included
code snippets runtime
templates
environment
code (RTE)
snippets

4 Software Engineering | RWTH Aachen

 Rep.
Generator – Architecture

• At a glance this is how a generator is internally structured:

templates

control- workflow function

script execution library

models,
models model input output template reports,
loader AST AST engine code, etc.

Frontend: Center: Backend:

read transform generate

5 Software Engineering | RWTH Aachen

Kinds of Transformations

• Text-to-Model transformations Text-to-Model- Model-to-Model- Model-to-Text

 parsing Transformation Transformations Transformation
Data store e.g., optimizations,
• Model-to-Model transformations reduction to simpler form
 model transformation-languages
 Java
Model Model ... Model Text
Text
• Model-to-Text transformations
 written in code directly, e.g. pretty printing conforms to conforms to conforms to
 templates

• “Model” here means tool-internal DSL A DSL B ... DSL Z

representation (e.g. AST)

Figure shows sequence of processing steps

executed by a tool

6 Software Engineering | RWTH Aachen

Template-based Generation

• Template-based generation (Model2Text = M2T)

 template refers to elements of the abstract syntax
 simple pieces of code embedded in templates
 holes are pieces of tool code to be executed when generating
 control structures in the template language e.g. for lists
 beneficial if generated code structure is similar to model

Key:
uses/accesses
input/output CpD
Template

Template
Model Parser AST Java
Engine

7 Software Engineering | RWTH Aachen

Types of Generators

• Pretty printing the AST (Model-to-Text, M2T)

 AST-API allows to build AST of target language
 then last step: pretty print the AST
• Transformational (M2M)
 An explicit transformation language transforms source AST to target AST (sequence of transformations; Model2Model)
• Combinations are possible

Key:
uses/accesses
input/output CpD
transforms to
Transformation Transformation
Engine

Pretty
Model Parser AST AST' Java
Printer
output model AST
input model AST

8 Software Engineering | RWTH Aachen

Pretty Printing

• If the AST is given in the target language

• then AST just needs to be printed Model Text

• Printing is a systematic depth-first process:

 Using visit/endvisit methods conforms to

• Pretty printing also adds layout (Whitespaces)

 e.g. by counting depths and adding a linebreak after “;” Target
language

• Recursive descent (depth-first) is handled by Visitors

 Rather straightforward and schematic
(can almost be generated)

9 Software Engineering | RWTH Aachen

 Rep.
Adapting Visit – Example

• Pretty print: nice formatted output of the model

1 public class PrettyPrint Java
2 implements AutomataVisitor2 { «hc»
3 @Override
4 public void visit(ASTAutomaton node) {
5 print("automaton "+node.getName()+" {"); Result
6 }
@Override
7
public void endVisit(ASTAutomaton node) {
1 automaton A {
8 state S <<initial>>;
9 print("}\n"); 2
10 } 3 state T;
11 @Override 4 // …
12 public void visit(ASTState node) {
print(" state " + node.getName()); 5 }
13
14 if (node.isInitial()) {
15 print(" <<initial>>");
16 } // …
17 print(";\n");
18 } // … transition missing
19 }

10 Software Engineering | RWTH Aachen

Template-based Model-to-Text-Transformations

• Template engines
 transform a DSL model into output files using templates Template
DSL
language
• Properties of template-based conforms to
Model-to-Text-transformation:
+ relatively easy to adapt Model Template
- complex code generations
pollute templates with GPL code

• Tool support: various engines exist: Template-

engine
 Velocity/Freemarker
 MOFScript (JET)
 Xtend
 ....
Output Target
files language

11 Software Engineering | RWTH Aachen

 Software Language Engineering
8. Generators
8.2. Template-Engine FreeMarker

Prof. Dr. Bernhard Rumpe

Software Engineering
RWTH Aachen

http://www.se-rwth.de/
www.freemarker.org
Farbe!
Freemarker – The Principle

• Code generation defined by combination of data model (here: AST) and template

• Example:

SC MyAut FM Result
The name of the The name of the
1
+ automaton is
<b>${ast.name}</b>.
= automaton is
<b>MyAut</b>.
2
Freemarker directive:
fixed content
Pieces of Java are processed
by template engine

The name of the automaton is

• Usage of the result, e.g., as HTML: MyAut.

13 Software Engineering | RWTH Aachen

Freemarker

• is a template engine for Model-to-Text-transformation:

 Source: Apache project; it is robust;
designed for Web-Pages
 www.freemarker.org

• It interprets templates in context of the AST and other

values
Templates
• To learn FreeMarker:
 A) Consult Ref.Man. Ch. 12 or
 B) FreeMarker Manual Builder.ftl
Java Files
Input AST

Template Person
AST
Engine Builder.java

14 Software Engineering | RWTH Aachen

Freemarker – Languages

• Freemarker templates are stored as files with extension .ftl The name of the
FM
automaton is
• A template consists of three languages that are interwoven <b>${ast.getName()}</b>.
• The target language:
 can be any language; Freemarker knows nothing about it
 (e.g. html, Java, etc.)
• The FM directives language:
 controls the output (loops, conditionals, template calls etc.),
 looks like <#if …>, <#assign …>
 usual controls: if, switch, loop, template call + some comfort
• Expression language with callbacks to the underlying data model
 looks like ${expression}

• This is a typical example for interweaving of languages.

• Don’t confuse the sub-languages, in particular Java as target and Java-like callbacks in the FM expression
language!

15 Software Engineering | RWTH Aachen

Freemarker Example: Generating a Factory

1 package ${ast.packageName}; 1 package a.b;

FM Factory.ftl Java
2 2
<#assign cname = ast.name> «gen»
3 3
4 public class ${cname}Factory { 4 public class PersonFactory {
5 5
6 protected static ${cname}Factory f = null; 6 protected static PersonFactory f = null;
7 7
8 protected ${cname}Factory() {} 8 protected PersonFactory() {}
9 9
10 public static ${cname} create() { 10 public static Person create() {
11 if (f == null) { 11 if (f == null) {
12 … 12 …

• ast is a predefined MontiCore variable pointing to the data model (AST)

• ${ast.packageName}:
 Java-like expressions resulting in String
• <#assign cname = ast.name>:
 Definition and assignment to local variable

16 Software Engineering | RWTH Aachen

Some Freemarker Elements … (please consult the Freemarker Ref. Manual!)

• Special variable now produces

 ${.now}  02.04.2032 23:46:06
 ${.now?date}  02.04.2032
 ${.now?time}  23:46:06

• ${object.attribute}
 is shorthand for get-function object.getAttribute()
 as extension: Freemarker can be configured to use direct attribute access if no get method exists

 Freemarker’s expression language is not exactly like Java

17 Software Engineering | RWTH Aachen

To Avoid in a Template Language

• Don’t pollute templates for technical code with business specific knowledge!

1 <#if job.getName()?ends_with("Manager") > FM

2 <#assign salary=5000>
3 <#elseif job.getName()?contains("CEO") >
4 <#assign salary=6500>
5 <#else>
6 <#assign salary=3200>
7 </#if>

 Although: we may use project specific templates for individual adaptation of commonly used templates

• Don’t try to encode complex control or algorithms in templates

(use the underlying Java instead)

18 Software Engineering | RWTH Aachen

Freemarker – Summary

• Freemarker is a standalone product very helpful to produce text pages (Java, HTML, …)
 free choice of target language
 directives for controlling the output
 expression language with built in types
 strong connection to underlying Java data model

• Template based Model-to-Text transformations are easier to write than a simple pretty printer

• MontiCore and many of its descendants use Freemarker

19 Software Engineering | RWTH Aachen

 Software Language Engineering
8. Generators
8.3. MontiCore’s Freemarker API

Prof. Dr. Bernhard Rumpe

Software Engineering
RWTH Aachen

http://www.se-rwth.de/
www.freemarker.org
Farbe!
Understanding Templates

• We look at:
 Template APIs provided by MontiCore
 tc: TemplateController
 ast: Model representation of the abstract syntax

 Generator engine provided by MontiCore

 How to write extensible and adaptable templates

 How to add functions / data on the abstract syntax

 Helpers: handwritten code in the Generator

21 Software Engineering | RWTH Aachen

Templates Use Predefined Objects

• MontiCore provides five core objects with various • GlobalExtensionManagement (glex)

methods:  manages global variables and the handling of extension
points
 (same object in all templates: shares data)
• TemplateController (tc)
 provides typical template operations and template-specific
information • Model Access (ast)
 e.g. include sub-templates  the currently processed node in abstract syntax
 tc refers to a new object in every template execution (i.e., the internal representation of the model)
 ast can refer to different AST in every template: used to
navigate through AST
• TemplateLogger (log)
 provides logging methods • Target structure (cd4c)
 (same object in all templates)
 When mapping a language to an OO language as target:
it helps to build an intermediate structure like a class
diagram (classes, attributes, methods, etc)

22 Software Engineering | RWTH Aachen

Variable ast points to the Abstract Syntax Node

• ast points to the currently handled AST node Automaton AST-CD

(with varying types) String name

• In the Automata language

* *
ast points to one of the three classes State Transition
boolean initial String from
• Available signature, when boolean final String input
ast is a Transition object: String name String to
 ${ast.from}
 ${ast.input}
 ${ast.to}

• and also methods like

 ${ast.getFrom()}
 ${ast.setFrom(…)} // + all methods of class Transition
 ${… ast.iterator_PreComments()} // + all methods of ASTNode

23 Software Engineering | RWTH Aachen

Template Controller tc

• Provides methods for typical template operations: • Aliases for Java methods improve readability
 inclusion of sub templates
 instantiation of helpers (e.g., a file handler)  ${include(ʺmy.Templateʺ)} instead of
 ${tc.include(ʺmy.Templateʺ)}
• Each template call gets its own
TemplateController tc
Example aliases:
 containing template-specific information, such as the
name of the current template include(template)  tc.include(template)
error(message)  log.error(message)
• The following gives an overview of some functions of warn(message)  log.warn(message)
the tc

 Retrieve information getTemplatename,

existsHandwrittenFile
 Create objects of a given class
instantiate

24 Software Engineering | RWTH Aachen

Logging from inside a Template

• MontiCore provides a dedicated logging mechanism for use from within templates:
 delegates log messages to the central logging component
 methods for different log message severities

• error(String msg)
• log.error(String msg)
 announce errors from within templates
 errors abort the generation immediately

• Also: warn(…), info(…), debug(…), trace(…) methods

 but these continue processing

• Errors and warnings

 have a compact message describing the problem to the user

25 Software Engineering | RWTH Aachen

Template Controller: signature(…)

• In FreeMarker templates don't have a "signature"

• signature(...) method mimics such a signature: it defines a list of variables

 signature(List<String> parameterNames) -- for several arguments

 signature(String... parameterName) -- for a single argument

• Example: Typically at the beginning of a template:

 ${signature("className", "package")}

• When calling a template the arguments are bound to these variables

•  for a better and more efficient handling of templates

26 Software Engineering | RWTH Aachen

Template Controller: include(…), write(…)

• Templates may be nested: they include each other

• include(...) methods in various forms

 include(String templateName)
 include(String templateName, ASTNode ast) {
 includeArgs(String templateName, ASTNode node, List<Object> templateArguments)
 includeArgs(String templateName, List<Object> templateArguments) {

• Examples within a template:

 ${tc.include("core.Attribute", attribute)}
 ${tc.include("tpl.StateBody", ast.getStateBody())}

• Similar methods, write(…) and writeArgs(...) execute a template and write result in a fresh file, e.g.:

 write(String templateName, String qualifiedFileName, ASTNode ast)

27 Software Engineering | RWTH Aachen

Global Variables in Templates (glex)

• Global variables can be accessed from within all templates

• GlobalExtensionManagement provides functionality to define and access global variables

• defineGlobalVar(String name, Object value)

• setGlobalValue(String name, Object value)
• addToGlobalVar(String name, Object value)
• boolean hasGlobalVar(String name)
• Object getGlobalVar(String name)

• But: avoid global variables if possible and use explicit arguments

• Usable, e.g., to transport global settings

28 Software Engineering | RWTH Aachen

Intermediate Class Diagram Structure with cd4c

• Often an intermediate representation of the target is useful

 we use the class diagram AST (from MontiCore CD4Code language) for the structure of Java and
 templates for the method bodies

• cd4c object helps to transform the source model to the intermediate CD-AST

Key:
uses/accesses
input/output CpD
transforms to Transformation Template
Transformation
Engine, cd4c

Template
Model Parser AST CD-AST' Java
Engine

Structure of output
input model AST
model AST: form of a class diagram

29 Software Engineering | RWTH Aachen

Intermediate Class Diagram Structure with cd4c

• cd4c object provides methods for

• creating AST nodes of the CD4Code language

 e.g.
 addAttribute(ASTCDClass clazz, String templateName)
 addMethod(ASTCDClass clazz, String templateName, Object... arguments)
 addImport(ASTCDClass clazz, String importStr)

• compact definition of a signature a string (being actually parsed)

 method(String methodSignature)

• Examples:
 ${cd4c.addMethod(ast, "de.monticore.sc2cd.StateSetStateMethod")}
 ${cd4c.method("public void setState("+ast.getName()+"_State k)")}

30 Software Engineering | RWTH Aachen

AST vs. Template Structure (taken from the Dex project)

CD-AST Template-Hierarchy

CompilationUnit FactoryMain.ftl ClassMain.ftl InterfaceMain.ftl

CDDefinition
Factory.ftl Class.ftl Interface.ftl

* *
CDClass CDInterface

Constructor.ftl Attribute.ftl Method.ftl

* * * * *
CDConstructor CDAttribute CDMethod GetterSetter.ftl

• Observations:
 more than one template for the same type of ASTNode (colors)
 hierarchy of AST roughly corresponds to template hierarchy

31 Software Engineering | RWTH Aachen

AST vs. Template vs. File Structure (Example: DEx)

CD-AST Template-Hierarchy Produced

Files

CompilationUnit FactoryMain.ftl ClassMain.ftl InterfaceMain.ftl

NFactory
.java
CDDefinition
Factory.ftl Class.ftl Interface.ftl

* * NImpl.java
CDClass CDInterface

Constructor.ftl Attribute.ftl Method.ftl

* <<interface>>
* * * *
CDConstructor CDAttribute CDMethod GetterSetter.ftl N.java

• Observations here:
 some templates produce full file artefacts
 others describe parts of files corresponding to nonterminals of the target language
 each file-production (e.g. factory) has one main template

32 Software Engineering | RWTH Aachen

AST-, Template-, File-Structure (Example: Statechart’s)

CD-AST Template-Hierarchy

Artifact StatechartBase.ftl State.ftl ConcreteState.ftl

Statechart

* * *
State * Transition
Files = Java-CD
0..1
… …
0..3 0..2 Base <<abstract>> State
Code Action Invariant Event Statement event() state
handleEvent(Base bc)
setState(State s)

…
• Observations here:
 templates correspond to java classes StateA
…
StateB
…
 large templates with inner logic,
e.g. transition information is used inside State templates handleEvent(Base bc) handleEvent(Base bc)

33 Software Engineering | RWTH Aachen

 Software Language Engineering
8. Generators
8.4. Some Templates Explained

Prof. Dr. Bernhard Rumpe

Software Engineering
RWTH Aachen

http://www.se-rwth.de/
www.freemarker.org
Farbe!
Example Templates

• … help to understand how templates work

• … allow to identify reusable assets discuss

• For Statecharts encoded using the design pattern CD for "State Pattern"
“State” (Gamma et.al. 1994)
… …
 Statechart.ftl Main StateClass
 AbstractState.ftl stimulus() state
handleStimulus(Main m)
 ConcreteState.ftl setState(StateClass k)
 …
…

… …
StateA StatebB
handleStimulus(Main m) handleStimulus(Main m)

• Sources:
• github.com:MontiCore/monticore

35 Software Engineering | RWTH Aachen