Mulesoft ESB

API - Application Programming Interface "how to communicate with a software component".

● Operations (what to call)
● Inputs (what to send with a call)
● Outputs (what you get back from a call)
● Underlying data types
Defines functionality independent of implementations.

Application Network - network of applications, data, and devices connected with API's to make
them pluggable and to create reusable services.
Mule - an Application that processes API's across an ESB.
ESB - Enterprise Service Bus
Anypoint Platform - Web interface for designing, creating, and managing Mule applications
and API's.
Anypoint Studio - IDE for designing, building, creating, and managing Mule Applications.
API-LED CONNECTIVITY
● From the bottom -> up beginning with:
● System API's - SAP customers, Salesforce customers, UPS shipments... (Central I.T.) -
unlock assets and decentralize access.
● Process API's - Order history, Order status, shipment status... (LOB - Line of Business
I.T.) - discover, reuse assets and compose information
● Experience API's - Mobile API, Web app API... (Developers) - discover, self-serve, reuse
and consume.
API Term Usage
● Web service API (definition of specification)
● Web service interface implementing the API
● Web service implementation itself
● Web service proxy to control access to the API
Web Service - A method of communication that allows two software systems to exchange data
over the Internet.
● Defined rules of communication
● How data is requested
● What parameters are required
● The structure of the data returned
Types of API's
● RESTful - The more up-to-date and simpler to use HTTP communication protocols.
● SOAP - communication rules are defined in an XML-based WSDL (Web Services
Description Language) file.
C4E - Center for enablement is a cross functional team:
● Central I.T.
● LOB - Line of Business
● Developers / Innovation teams
RESTful - Representational State Transfer - An architectural style where clients and servers
exchange representations of resources using standard HTTP.
Other systems interact with the web service using HTTP. The HTTP request method indicates
which operation should be performed on the object identified by the URL.

Example RESTful Web Service

JSON - JavaScript Object Notation
● Lightweight data interchange format
● Human readable results
● Supports collections and maps XML -
CSV- Comma Separated Values
DW- Dataweave
 Calling RESTful Web Services
Specify HTTP method, Request Header, and Request body.
Tools:
● API portal with API console Advanced Rest Client
● Postman
● A CURL command-line utility
Unsecured API's
● Public and require no authentication
Secured API's
● Require authentication
● Provide credentials or token Proxy to govern access to API
● Authentication protocols - - OAuth, SAML, JWT etc.
HTTP Status Codes
● 200 - OK - request succeed. Returned by GET, DELETE, PATCH, PUT.
● 201 - Created a new resource or an object in a collection. Returned by POST. 304 - NOT
modified. Returned by PATCH, PUT.
● 400 - Bad Request. Returned by ALL. 401 - Unauthorized. Returned by ALL. 404 -
Resource Not Found - URI not recognized by the server. Returned by ALL.
● 500 - Server error. Returned by ALL.
Anypoint Platform
Web based platform
1. Create Mule Applications using a simple interface for creating Mike integration Flows.
2. Create API specifications to design, document, and test.
3. Can import designed API's into Anypoint Studio to build an entire API Router Kit with an
interface of Flows and error handling based on the methods created in the API
specifications.
Anypoint Design Center - Rapid Development
Anypoint Management Center - Collaboration and self-service
Anypoint Exchange - Visibility and control
Runtime services - Hybrid Cloud - CloudHub
Anypoint Platform
Design once, Deploy anywhere...
Deploy:
● on premisses
● On private cloud
● Fully managed iPaaS

MuleSoft Blueprint:
 Anypoint Platform: Exchange
● Library of assets
● Central repository
● Mulesoft public assets
● Private assets
REST API's - added to Exchange gets an API portal.
API portal - API documentation auto generated.
API console - for consuming and testing API's.
API endpoints - auto generated, use a mocking service for testing API's.
API portals - internal and external users.
RAML 1.0
API Specification - add to Exchange to get a connector created.
Connector - used in Mule applications to make calls to API conversion by REST Connect.
Anypoint Platform: Design Center
Web app for building Flow Designs and API Designs.
Create a Mule Application - create new or import. (Flow Designer)
Create an API Specification - interface for designing, documenting, and testing API
specifications. Create api specification or create fragments. (API Designer)
Mule applications are deployed to a Mule runtime.
● Cloud hosted
● On-Premises hosted
Lightweight Java based ESB
● Connect apps to exchange data
● Connect systems
○ JMS, JDBC, HTTP
Mule ESB becomes the transit system, the bus that drives all messages to their destination
converting and translating the data to the required format.
Anypoint Platform: API Manager
● Manage API's
● Create proxy applications
● Deploy proxies to an API gateway runtime Throttling, security, and other policies Tiers
with different types of access Approve, reject, or revoke access to API's
● Promote managed API
● Define SLA tiers
● Apply runtime policies
● API Gateway enforces the policies
Policies
● Rate limiting
● Spike control
● security
● SLA tiers defines the number of requests that can be made per time frame to an API. Set
to automatic (free) or manual (for tiers that cost $)
● SLA based rate limiting policies require all applications that consume the API to...
Register and pass their client credentials
● Add client ID enforcement to the API spec so the REST connector can enforce the
authentication
CloudHub Worker
● 1 application per CloudHub worker Dedicated instance of Mule that runs an app
● Runs in a separate container from other applications
● Deployed and monitored independent
● Run on a cloud specific to a region in the world
● Different memory capacity and processing power
● Different memory capacity and processing power
○ scaled vertically by changing worker size
○ scaled horizontally by adding multiple workers
Worker Environments
● Design environment (Flow Designer apps only), sandbox, production
Anypoint Studio: Mule Applications
Mule receives events, processes them, and routes them to other endpoints.
Mule Event Source - initiates the execution of the Flow.
Mule Event Processors - transform, filter, enrich, and process the event and it's message
plugged together with a source in a Flow:
● source->processor(s)->connector endpoint
Mule runtime - hosted by MuleSoft called CloudHub worker.
Mule Applications: Mule Events
A Mule event consist of:
● A Mule message:
○ Attributes
● Payload
● Variables
Mule applications receive, process and route events to other endpoints through a series of
event processors plugged together in a Flow using:
● A Mule Event Source - initiates execution of the flow.
● A Mule Event Processor(s) - that transforms, filters, enriches, and processes the event
and its message.

Mule Applications: Mule Events

View event data at design-time and/or run-time.
Design-time:
● the DataSense Explorer in Studio
● the Transform Message component
● auto-completion when writing expressions
DataSense is the ability to proactively discover metadata from internal and external resources.
Provides DataWeave expected input or output.
Run-time :
● in a web client (postman) on requests deployed applications log files
● in the console using logger components Logger is a component added to flows to view
output.
Mule Applications: Debugging
Add breakpoints to processors in Flows
● Watch event properties and values Watch and evaluate DataWeave expressions
● Debugger listens for incoming TCP connections on localhost port 6666, make changes
in run configuration
Anypoint Studio
Desktop IDE for implementing API's and building integration applications.
Based on Eclipse, A Java integrated development environment.
● Edit between visual graphical and XML
● Connect to API's (REST, SOAP), protocols (HTTP, FTP, SMTP), and services
(Salesforce, Workday, etc)
● Data transformation framework and language
● Mule runtime to test applications
● One-click deployment of applications to CloudHub
● Templates for common integrations
● Integration with Maven for continuous build processes
 Anypoint Studio: Mule Palette
● Search Exchange
● Add Modules
● Favorites
● Core
○ Batch
○ Components
○ Endpoints
○ Error Handling
○ Flow Control
○ Scopes
○ Transformers
Console - An embedded Mule runtime to test applications
MUnit - Automated testing of Mule applications.
Connectors -
JDBC relational database
● add database module to the project
● add database operation to a flow
● configure connection to database
Global Configuration - configurations encapsulated in a separate global element. Defines a
connection to a network resource and reusable by many operations.
Transforming Data:
● Visual Graphical Interface
● DataWeave 2.0 Stand-alone scripting language
● add Transform Message to the Flow
Anypoint Studio

The Studio’s Look

 Anatomy of a Flow: Visual

Anatomy of a FLow: XML

Create RESTful interfaces

Manually - have listeners for each resource/method pairing defined by the API.
Automatically - generated from the API specification(definition) imported using the API Router
Kit in Anypoint Studio.
API Proxy - An application that controls access to a web service, restricting access and usage
through the use of an Apl gateway.
API Gateway - Runtime designed and optimized to host an API or to open a connection to an
API deployed to another runtime.
● determines which traffic is authorized to pass through the API to backend services
● meters the traffic flowing through
 ● Logs all transactions
enforce governance; rate limiting, throttling, and caching

DataWeave
DataWeave 2.0 - Mule 4 (2017) - transformation expression language for Mule to access, query,
and transform Mule 4 event data
DW 1.0 - Mule 3.7 (2015)
M.E.L. - Mule Expression Language (early)
DataWeave: Graphical editor
● Transform message in the Flows
● Drag and drop
● Script editors
● Input
● Output
● Preview
DataWeave Expressions: Script editor
Standalone scripts:
● auto-generated when using the graphical editor
● Can be written in manually in code editor
Inline expressions
● Dynamically set the value of properties in an event processor or global configuration
element
● are enclosed in #[]
● offers an expression mode button
DataWeave selectors:
● Single Value selector - #[payload.name], #[attributes.queryParams]
● Array index selector - #[payload[0].name]
● Array multi value selector (key:value pairs) - #[payload.*name]
● Array descendants selector - #[payload..zip]
DataWeave Expressions: Script editor
DataWeave operators
 ● Arithmetic (+, -, /, *) #[payload.age * 2] Equality (==,=, ~=) #[payload.name == "max]
● Relational (>, >=, <, <=, is) #[payload.age > 30]
● Conditional (and, or, not) #[(payload.name == "max") and (payload.age > 30)]
● Type coercison (as) #[(payload.age as Number) * 2]
● Default value default (default) #[payload.type default "student"]
DataWeave conditional logic -
if /else if/else statements
● if (payload.age < 15) group: "child"
else if (payload.age < 25)
group: "youth"
else if (payload.age < 65)
group: "adult"
Else
group: "senior"

Referencing Mule objects in DataWeave expressions MuleSoft

 Accessing event attributes

DataWeave Expressions
DataWeave functions -
● Packaged in modules
● function reference
○ https://docs.mulesoft.com/mule4-user-guide/v/4.1/dataweave
● core dw functions examples:
○ Contains
○ daysBetween
○ distinctBy
○ ends With
○ isLeapYear
○ isOdd
○ orderBy
○ Sum
○ Sqrt
○ Write
○ Upper
 ● functions with 2 parameters
○ #[contains(payload,ax)]
○ #[payload contains "max"]
Anypoint Studio: Set Variable
● Store metadata for the Mule event
● Set Variable is a transformer
● Persist through all Flows unless the event cross a transport boundary
● Expression referenced as vars - #[vars.foo]
Flows -
● Error Handling strategy
● No event source considered private
Subflows -
● Error handling comes from the parent Flow
● Executed exactly as if the processors were still in the calling Flow

Anypoint Studio: Synchronous vs Asynchronous

Synchronous events - are passed between flows using Flow Reference connectors
Asynchronous events - are passed between flows using VM Connectors:
● Parallelism
● Specificity
● distribute across a cluster
● communicate with other applications
● simple queuing, no full JMS broker
● intra and inter application communication
● queues can be transient or persistent
● in-memory queues
● transient queues are faster, but are lost in the case of a system crash
● persistent queues are slower, but reliable
Anypoint Studio: Encapsulation
 ● Separate flows into multiple flows based on functionality.
● Separate configuration files into multiple configuration files.
● Compile global elements in one config file and reference them.
● Separate applications based on functionality.
● Create run configuration to run more than one application at a time.
Domain project - Share global configuration elements between applications, not available on
CloudHub customer hosted Mule run-times.
● create the Mule Domain project
● then associate Mule applications with a domain
● add global element configurations to the domain project.
Anypoint Studio: Properties
.yaml files created in src/main/resources folder (config.yaml).
Define properties in the hierarchical YAML
● File.
db:
port: "3306"
user: "mule"
In configuration properties global element call and/or reference:
● $(db.port) from the YAML file.
In a DataWeave expression reference:
● {port: p('db.port')}
System properties override property values when deploying an application to different
environments.
● Run > Run Configurations > Arguments Command-line standalone Mule instance.
○ mule -M-Ddb.database=training2 -M-Ddb.password=mule2
Anypoint Studio
Mule 4 applications are Maven Projects. Maven is a tool for building and managing any
Java-based project that provides
● a standard way to build projects
● what a project consist of
● publish project information
● Share JARS across several projects
● manages a project's build, reporting, and documentation from a POM (project object
model)
● produces one or more artifacts (compiled JAR)
● artifacts has a group ID (reversed domain name com.example.foo), an artifact ID (just a
name), and a version string
POM file - an XML file contains info about the project and configuration details used by Maven
to build the project.
Define and access Metadata:
● Transform Message Component
● Event processors properties view Metadata tab
● Project's menu in package explorer
Metadata is stored application-tpyes.xml in src/main/resources.
Anypoint Studio: Consuming Web Services
● Call an operation of an API in Exchange
● Call a RESTful web service
● Call a SOAP web service
Connectors in Exchange package an easy way to make calls to APIs

Modules are extensions to the Mule runtime used when building Mule apps:
● HTTP, Database, Salesforce, SAP, Slack
REST CONNECT - A tool that generates connectors for RESTful APIs, RAML 1.0 or OAS API
specifications (Mule 3 and 4)
● Anypoint Studio
● Anypoint Platform Flow Designer
Connectors pre-installed in Studio - HTTP, Database, Salesforce, Validation, Java, search for
others in "Search in Exchange".
Consume RESTful Web Service not in Exchange:
● Add a HTTP connector
● Configure its Request operation
○ operation and/or global element configuration
● Specify headers, query parameters, or URI parameters to pass to the call
Consume RESTful Web Service with API and connector in Exchange:
● create flow to call RESTful web service
● Add REST connector from Exchange
 ● Configure REST connector
● Set headers, query parameters, uri parameters
Consume SOAP Web Service:
● Add Web Service Consumer connector Module
● Configure global element, location of WSDL
● Use Consume operation
● SOAP operation to invoke
● Use Transform message to pass arguments to the SOAP
● Transform before Consume, DataSense creates metadata for input that includes the
arguments
Combining data from multiple services:
● Canonical
● standard
Anypoint Studio: Controlling Event Flow
Routing Events send events to one or more groups of event processors(routes):
● Choice - one route executed based on conditional logic
● First Successful - sequentially until one is successfully executed
● Round Robin - One route executed, which one is selected by iterating through a list
maintained across executions
● Scatter-Gather - All routes executed concurrently and returns a collection of all results
● Collection is an object of objects containing attributes and payload from each Mule event
returned from a flow
● Scatter-Gather then use DataWeave to flatten multiple collections into one collection
Anypooint Studio: Control Event Flow
● Choice Router: Route based on conditional logic written with DataWeave to set router
paths.
● Validation Events: Test that some conditions are met and throw an error if the validation
fails:
○ Add validation module to project
○ Select validation operation (Is true)
○ Return custom error messages if condition is not true or met
Anypoint Studio: Handling Errors
When an event in a Mule flow throws an error:
● Normal flow execution stops
● Event passed to 1st processor in error handler
Mule Default Error Handlers are used when an error handler in the application is not explicitly
defined:
● Implicitly and globally handles messaging errors
● Stops execution and logs information
● cannot be configured
Error Object is created and contains two properties:
● error.description - a string
● error. errorType - an object
 ● identified by a namespace and an identifier
○ HTTP:UNAUTHORIZED
○ HTTP:CONNECTIVITY
○ HTTP:INVALID_BOOLEAN - (NAMESPACE:IDENTIFIER)
Error type hierarchy:
Any Errors
General - can be handled
● Unknown
● Transformation
● Routing
● Connectivity
○ RETRY EXHAUSTED
● Expression Security
○ CLIENT_SECURITY
○ SERVER_SECURITY
Critical Errors - Cannot be handled
● Fatal
● Overload
Error Handling Scopes:
On Error Continue
1. All processors in the error handling scope are executed
2. At the end of the scope
a. the rest of the flow that threw the error is not executed
b. Event passed up to the next level as if flow execution completed successfully 3
HTTP Listener returns a successful response
On Error Propagate
1. All processors in the error handling scope are executed
2. At the end of the scope
a. Flow is not executed
b. Error rethrown up to the next level and handled there
3. HTTP Listener returns an error response
Error Handling Scenarios
1. Default - Error occurs
a. Mule default error handler executed
b. Error gets rethrown
c. Error response returned
2. On Error Continue Scope - error occurs
a. On Error Continue scope executed
b. Subsequent processors NOT executed
c. Success response returned
3. On Error Propagate Scope - error occurs
a. On Error Propagate scope executed
b. Subsequent processors NOT executed
 c. Error gets rethrown
d. Error response returned
4. On Error Propagate Scope - child Flow error occurs
a. O-E-P scope executed
b. Subsequent processors NOT executed
c. Error gets thrown up to calling flow
d. On Error Propagate Scope calling Flow
e. O-E-P scope executed
f. Error gets rethrown
g. Subsequent processors NOT executed
h. Error response returned
5. On Error Propagate Scope - child Flow error occurs
a. O-E-P scope executed
b. Subsequent processors NOT executed
c. Error gets rethrown up to calling flow On Error Continue Scope calling Flow
d. O-E-C scope executed
e. Subsequent processors NOT executed
f. Success response returned
6. On Error Continue Scope - childFlow error occurs
a. O-E-C scope executed
b. Subsequent processors NOT executed
c. Success response returned On Error Continue or On Error Propagate Scope
calling Flow
d. Because Success response was received No Error Handler is called in calling
Flow
e. Subsequent processors are executed
f. Success or error response returned
Application Error Handling - Add error handler outside a flow or typically placed in the global
configuration file and specify the application's default error handler.
Flow Level Error Handling - All flows except subflows can have error handlers, any number of
error scopes can be added to a flow's error handler.
● Flow w/ error handler - 1st error scope evaluated to true, if no scope is true Mule Default
handles the error propagating the error up the execution chain, where there may or may
not be any error handlers.
● Flow wo/ error handler - error is handled by the Application's Global Default handler who
1st evaluates to true, if not the case, the Mule Default handles the error.
Event Processor Error Handling - Try scope provides more specific error handling, unlimited
number of processors can be added to an error handler scope and…
Specify scope execution for specific error types -
● Any - default
● Types - WSC:BAD_REQUEST
● WSC:BAD_RESPONSE
● WSC:CONNECTIVITY
 ● WSC:INVALID_WSDL
Specify scope execution for a specific error condition -
Set the when condition to a Boolean DataWeave expression i.e.
● HTTP:UNAUTHORIZED
○ error.errorType.namespace == 'HTTP'
○ error.errorType.identifier == 'UNAUTHORIZED'
○ error.cause.message contains 'request unauthorized'
○ error.cause.class contains 'http'

Flow Level Error Handling Scenarios

Processor Leveling Error Handling Scenarios
APIKit Error Handling - Default Interfaces contains multiple

DataWeave: Transform Message

● Use visual editor
● input/output payload metadata
● write basic transformation expressions
● DataWeave Code by default is placed inline in the Mule configuration file Live Preview
sample data > src/test/resources
● in component click "Edit current target" and "set source code" to File.
● DWL (DataWeave Language) > src/main/resources
● DWL file referenced TMC "resource" attribute and
● in an element expression property using ${file::filename} syntax
● Can create modules of reusable DW functions
DataWeave: Writing Transformations
● Not dependent upon the types of input/output, just the structures
● Transforms executes against this model
● Produced output consist of 3 types of data
○ Objects: collection of key value pairs
○ Arrays: sequence of comma separated values
○ Simple literals
● Output directive mime types (formats)
○ application/json
○ application//java
○ application/xml
○ text/plain
○ application/dw
○ application/csv
○ and more...
 Example DataWeave transformation expression

DataWeave: Errors
● Scripting Errors - problem with syntax
● Formatting Errors - problem with the written transformation
● application/dw is a good way to test errors
● Comments - // single line, /**multi-line*/
DataWeave: Writing Transformations

Expressions for XML output:

 ● Can only have one top-level value Value must be an object with one property
● Use @(attName: attValue) create an attribute
Expressions for XML Input:
● XML elements not attributes are created as JSON fields or Java object properties
● use @ to reference attributes
Writing expressions for XML input

DataWeave: Writing Transformations

Working with collections (Arrays)
● Use map function to transform array elements
● input array can be JSON or Java
● Returns array of elements
● Lambda is an anonymous function not bound to an identifier
● Transformation Function
○ $$ refers to the index or key
○ $ refers to the value

DataWeave: Writing Transformations

Transforming complex XML data Working with collections (Arrays) Use explicit arguments in
lambda
● payload map {} --> should be
 ● payload map (object, index) -> {} to make code more readable.

Expressions for XML output:

When mapping array elements (JSON or JAVA) to XML, wrap the map operation in {(...)}.
● {} define the object
● () transforming each element as a key/value pair.

DataWeave: Writing Transformations

In Postman use RAW over PRETTY responses.

Expressions for XML Input

● Use .* selector to reference repeated

DataWeave: Writing Transformations

Defining and using Global variables
● Defined above the delimiter (---)
● Use var directive in the header
● Functional Programming Language
● Variables behave like functions
● Can be referenced anywhere in the body
Variables used like Functions

Defining and using Local variables

● Use the using keyword in the body
● using (<variable-name>= <expression>)
● Can only be referenced within the scope

DataWeave: Writing Transformations

Formatting and coercing data

● Use the as operator for type coercion
 ○ price: payload.price as Number
● Defined types
○ Any (top-level type)
○ Null, String, Number, Boolean
○ Object, Array, Range
○ Date, Time, LocalTime, DateTime
○ LocalDateTime, TimeZone Period and more…
● Use format schema property to format numbers and dates
○ price as Number as String {format: "###.00"}
○ someDate as DateTime {format: "yyyyMMddHHmm"}
Define and use Custom Data types
● Use type header directive
○ Name has to be all lowercase letters
○ No special characters, uppercase letters, or numbers

Transforming Objects to POJO's

● Specify inline
○ customer:payload.user as Object {class: "my.company.User") or
○ type user = Object {class: "my.company.User"}
---
customer:payload.user as user
DataWeave: Functions
● Functions are packaged in modules
● Core module functions are imported automatically into DataWeave scripts
● Functions with 2 parameters, alternate syntax
○ #[contains(payload,max)]
○ #[payload contains "max"]
DataWeave Functions examples
● ++(concatenate), abs, contains, distinctBy, filter, filterObject, flatten, map, replace, upper,
orderBy, isEmpty, isOdd, joinBy, log, sqrt, splitBy, sizeOf, scan, etc.
Series of functions, last function in the chain is executed first:
● flights orderBy $.price filter ($.availableSeats > 30)
In this case:
● filter will execute first then…
 ● orderBy will execute
● Expressions using map function must be defined as a local variable
Functions
Non-Core module functions must be imported:
● Import a function -
○ %dw 2.0
output application/xml
import dasherize from dw::core::Strings
---
n: upper(dasherize(payload.firstname ++ payload.lastname))
● Import a module -
○ %dw 2.0
output application/xml
import dw::core::Strings
---
n: upper(Strings::dasherize(payload.firstnam ++ payload.lastname))
Call flows in DataWeave
● Use the lookup function
○ {a: lookup("myFlow", {b: "Hello"})}
■ Argument (a) is the name of the Flow
■ Subflows can not be called
■ Argument (b) is the payload to send to the flow as a map
DataWeave: Synchronize Data
Triggering flows for Files and folders
● Schedule flows
● Persist and share data across flow executions
● Publish and consume JMS Messages Process items in a collections sequentially
● Process records asynchronously in batches
Four connectors for files and folders
● File (local system), FTP, FTPS, SFTP
● Use On New or Updated File listener Triggering flows for Databases
Synchronize Data by using Timestamps called Watermarks stored on each sync:
● Automatic - saving, retrieving, and comparing, available for several connectors,
restricted
● Manual
● Query
○ Statement Select *
From table
Where Timestamp> :watermark
● Fixed Frequency - default poll is every 1000 milliseconds
● Cron - a standard for describing time and date
DataWeave: Triggering Flows
Persist Data
 ● Use the Object Store component to store simple key-value pairs
Publish and consume JMS messages supports two messaging models:
● Queues: PTP (point-to-point or 1:1), single receiver pulls messages off the queue, does
not need to be listening to the queue.
● Topics: Pub-Sub (Publish/Subscribe or 1:tooMany), publisher sends message to a topic
and all active subscribers receive the message
● Subscribers not actively listening will miss the published message, unless messages are
made durable.
● Add JMS module to project
● Connect to any JMS messaging service that implements the JMS spec.
DataWeave: Batch Processing
● Split large messages into records that are processed asynchronously
● Created for processing data sets
● Performs actions upon each record Handles record level failures so batch job is not
aborted
● Reports on the results
● Pushes output to other systems or queues
● Enterprise edition only
Use Cases include:
● Integrating data sets to parallel process records
● Engineering "near real-time" data integration, like syncing contacts between NetSuite
and Salesforce
● Extracting, transforming, and loading (ETL) info into a target system, like uploading data
from a csv flat file to Hadoop
● Handling large quantities of incoming data from an API into a legacy system
DataWeave: Batch Jobs
● Contains one or more batch steps that act upon records as they move through the batch
job
Three phases of a Batch job:
● Load and Dispatch - "behind-the-scenes", splits payload into collection of records,
creates persistent queues and stores each record Process - (required) asynchronously
processes the records, contains one or more batch steps
● On Complete - (optional) reports summary of records processed, provides insight into
which records failed so you can address issues.
● One queue exists
● Each record-keeps track of what steps it has processed through
○ Moves through the processors in the first batch step
○ Is sent back to the queue to be processed by the second step
● Repeats until each record has passed through every batch step
Note: All records do not have to finish processing in one step before any of them are sent to the
next step.
● Batch records are queued and scheduled in blocks of 100
● A threading profile of 16 threads per job
 ○ Each thread processes a block of 100 records
○ Thread iterates through the block processing each record and then the block is
queued back
Batch Job Variables:
● Variables created before a batch job is available to all batch steps
● Variables created inside a batch step are record-specific, record-level data persist across
batch steps in processing. Does a record already exist in a database?
Record-level errors during processing
● Stop processing
● Continue processing
● Continue processing (until a max number) of failed records
DataWeave: Batch Jobs
Filtering - has two attributes to filter
records:
● An accept expression
● An accept policy
Aggregation - a batch step for bulk insert
● Batch Aggregator scope inside the aggregator section of a batch step to accumulate
records
● Use a Batch Aggregator scope to commit records in batches
Important note for Batch Job Processing
● Use the For Each scope to process individual collection elements sequentially and return
the original payload
● Use the Batch Job Scope (EE Only) for complex batch jobs