Module 5: Choosing appropriate

message transformation and

routing patterns

Goal

n (n - 1)
⎯⎯⎯⎯⎯⎯
2

A B

C D

All contents © MuleSoft Inc. 2

 At the end of this module, you should be able to

● Describe reusable ways to transform and process events

● Explain how to simplify and decouple complex data mappings
using common data models
● Design transformations between data models
● Choose the best event transformation, data validation, and event
routing patterns to an integration use case

All contents © MuleSoft Inc. 3

Converting between data

models
 Why are event transformation and routing patterns
needed?
● To identify reusable ways to transform and process events
between particular enterprise systems
– Systems use particular message formats and message schemas
– Systems often define business processes that must execute in certain
sequences

All contents © MuleSoft Inc. 5

Defining REST API data types

● Data types in a REST API can be modeled using

– JSON and XML schema
– REST API Modeling Language (RAML®)
– OpenAPI Specification (OAS)

● Anypoint Platform™ supports interoperability between OAS and

RAML

All contents © MuleSoft Inc. 6

 Choosing message
transformation patterns for
Mule applications

The challenge of schema transformations

● Schemas of different systems typically use fields with different

names or types
– names (address vs. addr)
– types (String vs. Integer)

● To integrate different systems, each system's data model must be

transformed to the other system’s data model

Message Schema 1 Transformer Message Schema 2

address addr

All contents © MuleSoft Inc. 8

 Using the Transform Message component to
implement the transformation
● Converts input data to output data guided by provided metadata
● Metadata can be auto-populated by inbuilt DataSense
● Uses DataWeave language for transforming data
– Allows drag-and-drop definition of mappings
– Optionally, DataWeave code can be typed manually

All contents © MuleSoft Inc. 9

How to access Metadata Editor

● Package Explorer of a project’s menu

● Metadata tab in the properties view of most event

processors

● Transform Message component

● Can be directly edited into the application-types.xml

file stored in the src/main/resources folder

All contents © MuleSoft Inc. 10

 How to populate metadata for Mule components

● Metadata defines the structure of message processors’ input and

output data
● Some connectors use DataSense to provide metadata
– Database, Salesforce, Facebook, etc.

● Supports metadata for input and output payload, input and output
variables, output attributes

All contents © MuleSoft Inc. 11

How to populate metadata for Mule components

● Use an example or schema to

define data structure and data
types
● Supports JSON Schema, XML
Schema, Java classes, CSV, flat file,
simple types, Excel, fixed width,
and Copybook
● RAML DataType fragments can be
used to define metadata for Object
or CSV

All contents © MuleSoft Inc. 12

 DataWeave formats (reader or writer)

MIME Type ID Supported Formats

application/avro avro Avro Format

application/csv csv CSV Format

application/dw dw DataWeave Format (dw) for testing a DataWeave expression

application/flatfile flatfile Flat File Format, COBOL Copybook Format, Fixed Width Format

application/java java Java Format, Enum Format for Java

application/json json JSON Format

application/octet-stream binary Binary Format

application/xlsx excel Excel Format

application/xml xml XML Format, CData Custom Type

application/x-ndjson ndjson Newline Delimited JSON Format (ndjson)

application/x-www-form-urlencoded urlencoded URL Encoded Format

application/yaml yaml YAML Format

multipart/form-data multipart Multipart Format

text/plain text Text Plain Format

text/x-java-properties properties Text Java Properties

All contents © MuleSoft Inc. 13

Leveraging flows, function modules, and Java logic

from within DataWeave
● DataWeave can invoke flows, DataWeave modules, or Java
methods
● Allows work to be offloaded from DataWeave, for example
– XSLT transformation flows that use the XML Module
– Invoke a method on a Java library to execute business mapping logic

All contents © MuleSoft Inc. 14

 Transformation using Java

● Java is generally NOT recommended for data transformation

– Use DataWeave instead

● However, an organization may want to call out to Java to reuse a

Common Object Model and associated transformations that
are written in Java
– DataWeave can invoke static methods of Java classes
– Mule Java module can also create instances and invoke instance methods

All contents © MuleSoft Inc. 15

Transformation using scripting languages

● Transformation using scripting: Groovy, JRuby (Ruby), Jython

(Python), and Nashorn (JavaScript)
● Using supported scripting languages is generally NOT recommended
for data transformation
– Use DataWeave instead

● Excessive use of scripting languages can affect maintainability and

understandability

All contents © MuleSoft Inc. 16

 Implementing the message enrichment pattern
with MuleSoft®
● Mule event processors use data from the incoming event (e.g., key
fields) to retrieve data from an external source
● After the event processor retrieves the required data from the
resource, it appends the data to the payload or other parts of the
message or event
● Typically, the previous event payload must be stored in a variable
and then merged with the new event payload after the retrieving
event processor completes

All contents © MuleSoft Inc. 17

Ways to preserve the previous event payload after

an event processor executes
● Many event processors have a target and targetValue
– When the target is set, the “targetValue” DataWeave expression result is
stored in the target variable
– Message (payload and attributes) from the previous event processor is then
left unchanged

<https:request target="shipAddress" targetValue="#[payload.address]" .../>

All contents © MuleSoft Inc. 18

 Simplifying and reusing data
mappings using common data
models

Critical data is often locked up across an enterprise

in incompatible silos
● When an application is integrated with other applications, it needs
to know how to
– Parse data coming in from other applications
– Format data to be sent to and understood by other applications
XML <account-id>C1234</account-id>
<account-id>C5678</account-id>
JSON {
"account1": "cust-1234",
App 2 App 3 "account2": "cust-5678"
}

App 1 App 4

CSV ACCT_ID
1234
App 6 App 5
5678
All contents © MuleSoft Inc. 20
 The schema mapping explosion problem

● Directly transforming data sent by one application to another

application results in many mappings
– In the worst case, N*(N-1) separate and often custom mappings are
required, which is O(N²) complexity

N*(N - 1) mappings

App 2 App 3
Fixed Width
"4acct141234214acct21245678"

Flat File App 1 App 4

H]abc
L]1345
L]3789
F]xyz App 6 App 5
All contents © MuleSoft Inc. 21

The solution via a common data model (CDM)

● Build a common (aka canonical) data model

– Specify general rule or acceptable procedure
– Map native application data schema to and from the CDM
– At most, 2*N mappings are required to support n systems with disparate
data schemas, which is O(N) complexity

O(N²) O(N)

App App App App JSON {

2 3 2 3 "account1": "cust-1234",
"account2": "cust-5678"
}

App App App App

CDM
1 4 1 4

App App App App

6 5 6 5
22
All contents © MuleSoft Inc.
 A common data model decouples multiple
transformation steps
● Isolates backend systems and supporting Mule applications from
change
● Encourages innovation and reuse
– Drives evolution of legacy systems and business processes toward more
modern, standardized, and cloud-friendly data formats, like JSON (or even
XML)
N changes
Change Change
1 change
App App App App JSON (or XML)
2 3 2 3

App App App App

CDM
1 4 1 4

App App App App

6 5 6 5

All contents © MuleSoft Inc. 23

Pros and cons of CDM

Pros
● Reduces #transformations
● Reusability of CDM-level validations
● Provides common language

Cons
● CDM can become very complex
● A given CDM may lack data sent or required by some applications
– CDM should be extensible and flexible
• E.g., FlexFields, anyType, etc.

All contents © MuleSoft Inc. 24

 Choosing data validation
patterns for Mule applications

The problems addressed by message validation

patterns
● Context
– In a flow, the Mule event may contain invalid or incorrect data that was
generated by humans, by applications, or during transmission
● Especially after a response from an external system via connector

– May also have data that is incompatible with data transformation or

connector requests later in the flow

● Problem
– Incorrect data sent to downstream systems may cause data-related errors
– At a minimum, this could consume unnecessary system resources
– Even worse, unexpected data errors could crash downstream applications

All contents © MuleSoft Inc. 26

 Where data should be tested and checked in Mule
flows
● Validate events as early as possible
– At the beginning of each flow, if the event source has incoming data
● E.g. HTTP Listener, database On Table Row, or JMS On New Message

– Later in a flow, immediately after an event is modified or enriched

● After calls via connector operations

● In other words, your Mule application should fail fast

flow validate validate
request response

Event Event Connector Event

request
source processor operation processor

external
system
All contents © MuleSoft Inc. 27

Implementing data validation patterns in Mule apps

● Common ways to validate

– DataWeave code
● Write Boolean logic to test data or conditions
● In a Transform Message component, set variables based on results of Boolean
logic tests
– Modules and connectors
● Validation module performs data validations
– Validation operations typically raise errors
● XML and JSON modules have schema validators
● APIkit validates query parameter and headers (via APIkit Router)
● HTTP Request operation performs HTTP response validation
– Expression or Bean reference
– Failure status code validator
– Success status code validator
– Choice routers
● Should not be used to handle validation failures
● Are best used for conditional routing
All contents © MuleSoft Inc. 28
 Validating and routing response events

● Data validation results can be stored in variables

● HTTP Listener responses can be set from these data validation
variables

All contents © MuleSoft Inc. 29

Validating conditions in a flow via Validation

module
● Set of validation operations to test (validate)
– Condition is applied to a DataWeave expression of
some part of the Mule event
– Raises an error if the condition fails

● Validation operations also provide an easy way

to raise a specific error type
– Error message
– Error type mapping

All contents © MuleSoft Inc. 30

 Modules that provide validation operations

● Generic validation operations

– Validation module
● Other modules provide validation operations for specific schema or
data types
– JSON - Validate schema
– XML - Validate schema
– APIkit - API request validation
– Java - Validate type
– HTTP Request operation - Validate response
status code
Java - Validate type

All contents © MuleSoft Inc. 31

Combining Validation module operations

● Validation module provides generic validation

operations
● The All scope and Any scope can combine
validation operations

All contents © MuleSoft Inc. 32

 Using Validation operations without raising an error

● Validation operations can be used directly inside

DataWeave code
● In this case, the operation just returns true or
false, it does not throw an error
● You can use this to write Choice router conditions
<choice>
<when expression="#[ Validation::isEmail(payload.email) ]" >
<!-- Send email to payload.email -->
</when>
<otherwise >
<!-- Send letter to payload.address -->
</otherwise>
</choice>

All contents © MuleSoft Inc. 33

Applying message
transformation, validation, and
routing patterns
 Exercise 5-1: Apply message transformation and
routing patterns to a systems integration use case
● Design integration between multiple inbound and outbound
systems
● Apply enterprise integration design patterns efficiently and
effectively to meet the requirements of a specific scenario

All contents © MuleSoft Inc. 35

Exercise 5-1 solution: Without a CDM

#[vars.src == "System1"] • Have to create different

transformations for 2x2 routes
• Must create separate
transformations to Salesforce and
#[vars.src == "System2"]
SAP twice
– These mappings tend to be
complicated and system-specific

All contents © MuleSoft Inc. 36

 Exercise 5-1 solution: With a CDM

• Transformations to Salesforce and SAP are reused

#[vars.src == "System1"]

#[vars.src == "System2"]

All contents © MuleSoft Inc. 37

Exercise 5-1 solution: Adding a third output

endpoint
• Adding a third endpoint is now decoupled from the CDM step

#[vars.src == "System1"]

#[vars.src == "System2"]

All contents © MuleSoft Inc. 38

 Summary

Summary

● DataWeave simplifies translating between common input and

output data types
● Mule events often must be validated and routed within a flow
● Defensive code is implemented with a combination of choice
routers, validator components, and DataWeave expressions, and
error handling

All contents © MuleSoft Inc. 40

 References

● Hohpe and Woolf, 2003

– Describes 65 patterns for enterprise application integration (EAI) and
message-oriented middleware (MoM)
– http://www.eaipatterns.com
– Mule runtime has implemented many of these
patterns

All contents © MuleSoft Inc. 41