0% found this document useful (0 votes)
38 views44 pages

Data Mappingand Transformation Optionsin App Connect

Uploaded by

Dary
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
38 views44 pages

Data Mappingand Transformation Optionsin App Connect

Uploaded by

Dary
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 44

Data Mapping and

Transformation Options in
App Connect
Dominic Storey
IBM

November 2020
Session 1AD
What is Integration?

IBM App Connect


What is Integration?

Provides routing and


transformation to enable
integration of any application
and data securely
20 years of
innovation
#1 in market share for application
integration every year since 1999

WebSphere
WebSphere IBM
MQSeries WebSphere
MQ Integrator
Business Integration IBM App Connect
Integrator
Message Broker Message Broker Integration Bus
IBM App Connect

Designer Toolkit Hybrid (callable flows capability)

Strongly typed Loosely typed


Set models Generic model
Smart connectors Highly configurable
nodes Designer flows → Toolkit flows
Simple routing
Complex routing Toolkit flows → Designer flows
All users Expert users
citizen, ad-hoc, specialist integrators
specialist integrators
App Connect Designer
Overview

Integration Technical Conference / © 2020 IBM Corporation


6
IBM App Connect Designer - SaaS

Enterprise
Grade

easy, guided, intuitive for all type of users increase productivity scalable, secured
data-driven tooling citizen, ad-hoc, reduce time to market industry standard &
specialist integrators increase ROI all best practices
baked into the platform
IBM App Connect Designer - Cloud Pak for Integration (CP4i)
Cloud Pak for
Integration

API Messaging
Lifecycle and Events

App and Data


Integration

Container
platform and
operational services

On-premise on your own servers

On your cloud

Installed using Red Hat Operators


IBM App Connect Designer - Cloud Pak for Integration (CP4i)

Grafana metrics

Kibana logging
Connect Applications and Data

Leverage 100+ pre-built connectors


Sales, HR, CRM, HCM, Finance, Marketing, AI…
Built-in optimization
• Metadata and business object aware
• Configured to listen and trigger to change events
• Process data in batch/bulk when moving large volume of data
• Robust and resilient to failure with built-in fail-over and retry mechanism

All SaaS endpoints are included in Enterprise license


Customers get free entitlement to connectors with v11 via a free cloud hosted plan
Contact [email protected] to provision free plan

Can’t find the connector you need…


Contact offering management to help us understand opportunities you are seeing and get
things prioritized
Contact AskApplicationIntegration for a connector
App Connect Connectors, the list goes on and on…

More connectors
added continuously
through SaaS and
with every CP4i
drop…

In fact… this list will


already be out of
date!
Extend the reach of your connectivity

• Simplest means to register new endpoints …. supply an OpenAPI or


WSDL file and give your “connector” a name and a description

• Connect securely to endpoints public/private, in cloud/on premise

• Onboard your org’s APIs, SaaS/ Software or take advantage of online


libraries like apis.guru to expand the org reach to 1000s of endpoints

• Once imported, these APIs can be easily added to any integration


Act on events

• New events are taking place across the organization:

• New sales order “Customers expect


• New marketing lead organizations to
• Customer complaint respond to events
• Reacting to these events, in real-time, enables organizations
as they occur in real
to capture new opportunities and provide enhanced time”
customer experiences
Gartner
• Building workflows ensures that the right-follow on action is
taken

• Capturing information about the event enables teams with


the context to apply it to react quickly
https://www.gartner.com/doc/3719717/use-eventdriven-approach-deliver-superior
Demo 1
Act on events from Salesforce ->
Cloudant

Integration Technical Conference / © 2020 IBM Corporation


14
Build your APIs in minutes with App Connect

• Easily build resource centric APIs through a model-driven


approach, without the need to code

• Extensive connectivity to reach data from any system or


endpoint

• Define the operations for the API:


• define models using connector metadata
• define CRUD and custom operations for your models implemented
as flows
• define nested and array data for your API request/response
• generate the proper metadata on the flow request / response
endpoints to align
• Automatic Open API doc generation

• Expose as a well-formed, REST API – ready to share with


development teams
Manage your APIs in SaaS

• Embedded IBM API Connect functionality allows users to


apply basic management

• For advanced management and socialization, publish


APIs to the IBM Cloud Catalog and utilize IBM API
Connect

• Consume APIs in App Connect from API Connect catalog

• Download the OpenAPI doc to use with development


teams

• Automated sync between flow metadata and model


definition as changes to the model or flow occur
Demo 2
Create an API

Integration Technical Conference / © 2020 IBM Corporation


17
App Connect Designer
Parsing/ Mapping and Transform (JSONata)

Integration Technical Conference / © 2020 IBM Corporation


18
Parsing JSON

JSON String in -> Mappable JSON Structure out


Parsing XML

XML String in -> Mappable JSON


structure of the XML string out

Similar functionality for CSV


https://youtu.be/ZBaK40rtIBM
• Lightweight query and transformation language for JSON data
• Inspired by the location path semantics of XPath 3.1
• Sophisticated query expressions with minimal syntax
• Built in operators and functions for manipulating and combining data
• Create user-defined functions
• Format query results into any JSON output structure
21
The Mapping Challenge

• IBM App Connect required a graphical data mapper to extract and transform
data to and from JSON structures

• Graphical mappers require an expression language to manipulate data


• Equivalent to the expressions in an Excel spreadsheet
• JSON has no ‘standard’ language to select & manipulate contents
• Unlike XML which has XPath – simple to learn yet expressive and powerful
• Needs to run in the browser (client-side) AND server-side (e.g. Node.js)
• Has to be implemented in JavaScript
<script src="lib/jsonata.js"></script> npm install jsonata
22
The Language
The Exerciser
• Declarative
Syntax describes what you want, not how to
do it
• Location path selectors
Similar to XPath
• Functional
In the programming sense – the function is
a first class data type
• Extensible
Large built-in function library can be
extended with user-defined functions
• Turing complete try.jsonata.org
• No limit to complexity of logic • Try out YOUR expressions on
YOUR data
• Hosted on GitHub, distributed on npm • Save and share with the
• ~50,000 downloads/month community
• Hosted in IBM Cloud 23
The Users and Community

• IBM App Connect github.com/jsonata-js/sonata


The instigator – required for a first class, easy to use
graphical data mapper • 18 contributors, 93 forks
• Node-RED • Variety of open issues from different people
The first open-source adopter. Built-in syntax for enhancement requests, usage questions, bug
several nodes
reports
• Watson IOT
• Trickle of stackoverflow questions
Implementing the language in Java
The community are now answering each
• Watson Conversation Service other’s questions
Chat ‘rules’ configured using JSONata
• Active ports to other languages
• Elastic.io
Golang and TypeScript nearing completion
Our competitors!!
• Other projects on GitHub & npm • Third party (non-IBM) users are writing blog
44 dependents on npm posts, tweeting and presenting at
conferences
24
The Future of JSONata

• Growing user base, increasing rate of new feature requests


• More functions
• Date/Time parsing and formatting
• Binary data parsing and processing (for IoT use cases)
• Support for other runtime environments
• e.g. database queries
• New syntax enhancements
• Comments within expressions
• Debugger
• Improved documentation
• Community help always welcome!
25
Mapping Assist

To identify matches, Mapping Assist uses a pre-trained algorithm that applies fuzzy
matching and semantic rules, as well as mapping histories that are stored for started flows.

https://www.youtube.com/watch?v=a3ULbH0MyQk
26
Flow Testing

Can check account and


connector parameters
work in real-time

Edit any input


parameters to check
behavior in real-time

27
Demo 3
Situation Detection Watson Customer
ticket

Integration Technical Conference / © 2020 IBM Corporation


28
App Connect Enterprise
Toolkit
Overview

Integration Technical Conference / © 2020 IBM Corporation


29
App Connect Enterprise Toolkit capability

Built-in nodes encapsulate transports, technologies and applications

• Our intent is always to


make the common tasks
easy, and the rest possible!
We have built-in nodes to serve different purposes:
• Use the built-in nodes to • Connecting with client applications
reduce the amount of • Routing messages
custom code required • Transforming and enriching messages
• Processing events
• This makes best use of the • Handling errors in message flows
built-in facilities like activity
log and resource statistics High performance and scalability
Very feature rich Message Modelling
Developing an integration solution

IBM App Connect Enterprise provides different ways


of creating your integration solutions.

• Using Applications and Libraries


• Using an Integration Service
• Using a REST API
• Using a Policy
• Using Patterns
Message Flow Structures

Asynchronous request-response message

Replies are correlated with the original Use asynchronous flows


requests. to avoid bottlenecks.
Store the original request message id on a
queue and then the reply message flow
retrieves that message id information from
the queue.
Message Flow Structures

Route the message based


Separation of processing on the message type.

• Split out incoming


different requests
received at single
entry point

• Find the message


type upfront quickly
and then route
accordingly.
Message Flow Structures

MongoDB
A message flow deployed to one
integration server can directly WINDOWS
invoke a message flow in a
different integration server,
passing the contents of the
message body and the
Environment folders.
Use asynchronous flows
to avoid bottlenecks.

LINUX
Subflows

Subflows are simply message flows that are invoked


from another flow
• Input and output nodes in the subflow
become terminals in the main flow
• Use subflows to break up large problems into
smaller more manageable chunks

Subflows are directly deployable to the runtime


• Shared subflows deployed just once per
integration server
• Redeployment of a subflow is automatically
picked up by any consumers
IBM App Connect – Parse/ Transform/ Route

• Assemble from dozens of nodes that can parse, serialize, restructure, split, merge, work with an in
memory cache, etc…

• Graphical mapper simplifies working with sources and targets. Includes hierarchical data and
understands those complexities

• Prebuilt functions that work


on the field level to
restructure data items

• Shared libraries and


subflows promote reuse

• Extensible to insert code


so enterprises can reuse
other enterprise assets.
Multiple Terminals on a message flow node Thread switching

The Try / Catch concept Code versus Configuration

Message Domains / Strong typing Immuteable context trees

37
IBM App Connect - Test

Be confident in the integration flows that are


developed through various tools that can help
identify issues and ensure the integrity of the
operations.

• Visual Flow Debugger – built in to the


Toolkit and provides an interface for
debugging a message flow in the same way
you would debug an application

• User Trace - to work out what path a


message followed through an integration
flow and why

• Trace Nodes - special nodes that you can


place in your message flow which log
information to either, the local error log, a
file on the filesystem or User Trace
When it is Right Make a Pattern of It

Patterns provide top-down, parameterized


connectivity solution of common use cases
• e.g. Web Service façades, Message
oriented processing, Queue to File…

They help describe best practice for efficient


message flows
• Reduce common problems in flow
development
• Establish best practice
• Reduces time to develop solutions
• Helps new message flow developers to
come on board more quickly.
Patterns Enhance Developer Re-Use

Patterns, Samples & Documentation


• These are now separately
downloaded, as required
• Allows in-version release of
samples and patterns
• Can be staged to local server

Built-in Unit Test Environment


• Developer tools have built-in unit
test server
• Fixed name of
TESTNODE_<userid>
• Started and stopped with tools
• Can still test / deploy to manually
created local and remote servers
40
Find out more…
• App Connect Designer JSONata course
• https://developer.ibm.com/integration/docs/app-
connect/learning/zero-to-hero/301-content/mapping-301-the-
power-of-jsonata/

• DeveloperWorks
• https://developer.ibm.com
• https://developer.ibm.com/components/app-connect/

• IBM App Connect early eXperience program


• https://epwt-
www.mybluemix.net/software/support/trial/cst/programwebsite.
wss?siteId=440&h=-1602000539&w=JBZQo
Thank you
Dom Storey
IBM Integration

[email protected]

• © Copyright IBM Corporation 2020. All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind,
express or implied. Any statement of direction represents IBM’s current intent, is subject to change or withdrawal, and represent only goals and objectives. IBM, the IBM logo, and ibm.com are
trademarks of IBM Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available
at Copyright and trademark information.

TechCon 2020 / © 2020 IBM Corporation


42
Please submit your session feedback!
• Do it online at http://conferences.gse.org.uk/2020/feedback/1DA

• This session is 1DA


GSE UK Conference 2020 Charity
• The GSE UK Region team hope that you find this presentation and
others that follow useful and help to expand your knowledge of z
Systems.
• Please consider showing your appreciation by kindly donating a small
sum to our charity this year, NHS Charities Together. Follow the link
below or scan the QR Code:
http://uk.virginmoneygiving.com/GuideShareEuropeUKRegion

You might also like