Successful Boomi Projects Ebook
Successful Boomi Projects Ebook
2227
KitePipe E-mail: [email protected]
Web: Kitepipe.com
Successful Boomi
Integration Projects
CONTENTS
I. PREMISE AND AUDIENCE: 2
II. INTRODUCTION: 4
V. DATA TRANSFORMATION: 7
XI. IN CONCLUSION: 19
4 Successful Boomi Projects
Introduction
Why talk about project framework and deliverables? It’s the highest value thing I can
talk about to the Boomi Integration community. Project-level mistakes are the big
mistakes. This document gives you a checklist of things to look for in structuring your
project.
In this updated version of the “Sprints” I’ve captured learnings from the last three
eventful years at Kitepipe. By far the biggest source of project problems were scope
issues – basically disagreement, unawareness or changes of the scope of the effort
that eventually was covered by the project.
Often the seeds of big project problems were baked into the assumptions of the initial
estimate. I’ve paid many times for my own estimating mistakes, and developed a
keen sense of scope and risk in the process. No process will protect you from “chang-
es” in mid-stream, but I’ve found that applying a detailed framework early in the proj-
ect can surface those “changes” earlier, in time to do something about them.
By describing a framework with which to understand the scope of a Boomi Integration
project, I seek to give you a set of tools with which to understand the scope of your
project, develop better estimates, and manage risk along the way.
Larry Cone
Chief Architect, Kitepipe
July, 2017
Successful Boomi Projects 5
We use a services model for Boomi projects with six sprints, and
each may be larger or smaller depending on the project.
The Sprints are:
1. Basic Connectivity: In the first sprint, you develop basic Boomi processes that
demonstrate connectivity to the integration endpoints – this helps manage your
project risk.
2. Data Transformation: In the second sprint, you work on the data transforma-
tions and lookups– the meat and potatoes of the project.
3. Business Process Support: In the third sprint, the focus is on supporting the
larger business process – Alerts, notifications, post-backs, and migration issues.
5. UAT and Go Live: The fifth sprint is often part of a larger project, and supports
overall testing and go live goals.
6. Clean-up and Tech Transfer: The last sprint is where you clean up, document,
and hand-off.
Basic Connectivity:
In the first sprint, you develop basic Boomi processes that demon-
strate connectivity to the integration endpoints.
This can take a few hours for simple endpoints with existing con-
nectors, or a month where custom development is needed to drive
a complex API.
Goals:
To demonstrate basic connectivity to the endpoints.
Deliverables:
-Develop a simple process that shows basic read write activity to the endpoints, or a listener that accepts
calls from a source system or web hook.
-Confirm that connectors, sandbox environments, account logins, and connectivity is in place to pass re-
cords from one endpoint to another
-Prototype and demonstrate the technique that the process will use to select records.
Risks:
-Are connectors available? For a RESTful API without a pre-built connector, that is implemented with se-
curity signatures that require custom coding, it can take several weeks to push your first record from one
endpoint to another.
-Local data base connections can require local atom ionstallation, networking, account creation, and per-
missions updates.
-Some EDI connections such as AS2 connectors require opening of ports on firewalls. It is good to identify
these issues early as there can be lead times involved.
-Is the design of the record selection or handshaking appropriate? Simple schemes rely on last updated
fields, so make sure these are available on the records that you plan to use. More complex handshakes
use custom fields that flag records for interface, and are then cleared by the process. Validate the hand-
shake or selection design, and get any custom fields in place.
-This is a good time to verify the number of Objects that the integration will touch on either end, as this is
a key measure of complexity.
Estimation Guidelines:
-Key factor here is the availability of connectors for the endpoints, and local atoms, if not already installed.
-Other factors include the complexity of the handshake scheme, Batch vs. real-time transactions, the
number of objects handled by the process, and any connectivity components that are out of your direct
control, like firewalls, Atom servers, or third party EDI infrastructure
Successful Boomi Projects 7
Data
Transformation:
In the second sprint, you work on data transformations and lookups. The rest of the project should be
far enough along to have significant amounts of test data available, and in a series of iterations you
push over test data, identify quality issues (missing or invalid data) and add logic, lookups, and transfor-
mations.
This is also where you handle differences in instantiation – that is, for example, when an order comes in
as one document, but must inserted into the target as a series of line items. Or, when shipping instruc-
tions come in as multiple line fields (looking at you, SAP), but must be consolidated into a single field in
the target.
This is the bulk of the Boomi development effort, and is the place where you design and develop the
basic structure of the Boomi processes. Most of this work is done in the Boomi Build environment with
small data volumes. The deliverable from this sprint is a set of basic Boomi Processes that are unit test-
ed, and ready for additional development.
At this point you can say that you are “Moving data end-to-end”, which is comforting to everybody.
Goals:
To build out the source queries, listeners, transformations, and lookups required to accept data from the
source, transform to the format of the target, handle differences in instantiation, and post transactions to
the target (or publish to multiple targets).
Deliverables:
-A set of Processes that move transactions between the source system and the target system.
-The Maps, scripts, profiles, and connector objects required to drive the processes
-A set of test data sufficient to exercise all of the branches and conditions in the processes
-An executed set of unit tests, with formal or informal documentation, that demonstrates that the process
moves data “end to end” and applies selection and transformation logic.
Risks:
-Is there good documentation that covers required fields, date formats, record dependencies, etc?
-Are the Objects accessible as planned? Some RESTful APIs may require three calls to change a relation-
ship (GET the old one, DELETE it, PUT the new one)
-Are there complex transformations of EDI or XML documents? Pulling order or shipment items out of a
complex EDI or XML document can be tricky
-Have you accounted for all status combinations? Things like Adds, Updates, Cancellations, and Deletes
may require separate handling and separate processes
Estimation Guidelines:
-Number of Processes, number of connectors in the processes, number of Maps, number of Profiles.
-Complex EDI or XML profiles can require fussy trial and error to transform from one format to another
8 Successful Boomi Projects
Business Process
Support:
In the third sprint, you work on fitting the integration process (a piece
of implementation) into the overall business process. This is an area
where we have had some notable misses and surprises, and where
we deliver significant business value.
Business process support means this: how does the Boomi integration
process fit into the overall business process, and what features and
functions are required to make the fit seamless, and to drive business
value?
Here are some examples of integration features, and how they support the business
process:
Completion notifications – when users want to know that the transaction was success-
fully processed
Status notification – when a multi-step process in in progress, and users need to know
Handshakes – when a complex multi-state integration must be managed from the end-
points, and the integration be kept stateless
Awareness – prevent the integration from being a “black box” in the users’ mind by
providing notifications and logging
Certainly, there are many integration scenarios that don’t require these types of fea-
tures: Extract, Transform and Load into data marts is about volume of completed
transactions, and status and quality are not issues. Most Pub/Sub scenarios don’t care
about the completion status, or recovery/retry.
But, increasingly, integration processes are called upon to handle live data not yet
complete from a business process standpoint. In these cases, the Architect or Designer
must understand the business process context, and build in features that support the
overall business process.
In an ideal, waterfall world, you would figure out the details of how the integration fits
onto the overall business process up front, in a requirements document, and build any
implementation components in during the build.
One of our significant discoveries in building Boomi integrations is that this discussion
can be left to later in the project. This may sound counter-intuitive, but it works.
-You can quickly get started on the more technical parts of the system (connectivity, mapping) while
you gain an understanding of the business context
-Looking at the data and the endpoints will inform you about what questions to ask. Key fields are
“status” type fields.
-Often, no-one on the business side has a full view of the overall process – teams tend to concentrate
on their platform and process. The Sales guys in Salesforce live in a country very distant from the Fi-
nance team in Netsuite. So the more context you have, the better the questions you can ask to elicit
issues and needs.
-Often, no one has thought about “migration issues” – what happens when we first turn this on. You,
the architect, know that integrations are designed to work in a steady state, with endpoints synched,
and only deltas being passed. You, the architect know that often one-time configurations are needed
to handle the initialization. You can ask better questions later in the project.
So, you can get started with the configuration tasks, learn something about the data,
and can ask good questions and design a complete solution. In my opinion, this phase
is the difference between technicians building an IT solution, and consultants helping to
solve business process problems. This is the phase where you build in features that are
visible to users and management, and create a strong perception of value in the man-
agement team for the solution.
Being able to handle business process support in this way is one of the reasons that
Dell-Boomi Atomsphere is the best integration platform available.
10 Successful Boomi Projects
At Kitepipe, we have built our teams and tuned our processes to maximize the
business value of our integration projects. Our developers are highly trained, ex-
perienced, and productive at building Boomi processes. They are also experienced
IT Consultants who can ask the right questions about the overall business process.
With this combination, we quickly deliver Boomi integrations that solve business
problems.
Goals:
Why Dell-Boomi Atom- To understand the business context of the integration, and to
sphere is the best platform build in features that minimize transaction friction and maximize
for Agile Integration: velocity, transparency, and business value.
It is a hyper-productive Deliverables:
development platform, so -A high level flow chart that puts the integration in the context
that you have project hours/ of the business process, and communicates the structure of the
budget to do more than integration process.
basic mapping and transfor- -Notifications, alerts and logs that communicate status and er-
mations. rors.
It is an agile product, where -Boomi process features and additions that support the business
you can build the basic pro- process
cess, then revise and add BP -A migration plan that describes how the newly linked systems
support features will get from the current state to the target steady state.
It is a hyper-productive -An inventory of views and reports in the endpoints that will
product that frees up devel- show transaction status, summary counts, and errors
opers to be business process
analysts Risks:
It is a full-featured product -Do you have access to knowledgeable business process owners
that supports all types of that can explain the context?
business process features, -Are you knowledgeable enough about business processes to
across the full life cycle. identify potential problems and solutions?
-Do you have a library of Boomi patterns from with to draw solu-
tions from?
-Are there business process owners that can make decisions
about statuses, notifications and recovery?
-Are parts of the overall business process missing from the
discussion? These can include Returns, Reissues, special prod-
ucts or offers, additional status fields, out of stock, custom-
er-sub-partner situations, etc. This risk here is that you find out
about these very late in the project.
Estimation Guidelines:
-Number of Processes, number of endpoints, number of depart-
ments.
-Watch out for missing business processes, and ask questions like
“Are these orders ever cancelled? How does that work?”
Successful Boomi Projects 11
Transaction Quality:
In the fourth sprint, you work on quality and completeness checks for
the transactions. Some integrations don’t have these requirements
– in Pub/Sub models, it is the target system’s responsibility to do vali-
dation, for instance.
-High value transactions – B-to-B sales transactions, for instance, where many people care about the
status of the transaction moving from Salesforce to Netsuite
-When the target is the source of master data – when “subsidiary” systems push data into a master
system, you should validate the transaction against the target before attempting to create/Update
-When logic is complex – complex validation logic is best handled in the integration layer, insulated
from changes on either end
-When endpoints are out of your control – either because of organizational issues, or they are part of
another organization
-When the data is time sensitive – often transaction, HR or user data is time or date sensitive, and
should be validated before it is passed on to other systems
-When the target systems don’t provide much/any validation upon input. Some systems will happily
accept an invalid transaction and not error, or will provide cryptic error messages. You can do better
with validation in Boomi
Goals:
How to do Validation/quality To understand and detect transaction data quality problems,
checks with Boomi: and to handle them in a way that speeds the resolution of the
error.
-Use a Try-Catch to handle an
error in the Boomi Process. Deliverables:
Try-catch is great, but should -A list of fields that require quality checks, and the require-
be reserved for un-anticipated ments for those checks.
problems, like connector or end- -The Maps, business rules, lookups and flows required to do
point failures. Better and more the checks
effective tests can be done with -A set of test data sufficient to exercise all of the branches and
other techniques. conditions in the processes
-Some design around error handling – we often try to get all
-Business Rules Shape: the best errors and messages into a single cache, and dump it at the
tool for applying complex tests end of the process to a notification scheme (email, post-back,
or business logic. Not so easy to etc)
use – check out our 5 part video
series about using Business Rules Risks:
Here http://www.kitepipe.com/ -Is there good documentation that covers required fields,
instructional-videos logic, data transforms, lookups, etc?
-Can your team or others access the source system and build
-Cleanse shape: A quick way to error data?
apply required value logic across -Has anyone thought about who is in the best role to fix trans-
many fields. If you have tried action data problems?
this shape and are mystified, un- -Who should be notified for platform / endpoint problems,
derstand that it uses metadata and how?
in a profile to provide the rules. -Resistance to the concept of putting data quality logic in the
integration layer. Pub/Sub and ESB practitioners typically
-Returning connector errors: Set trust the subscriber to do its own validation. So check the
your outbound connector calls to situations list above.
return errors (instead of failing)
and test for and process the Estimation Guidelines:
error -Number of Processes, number of Maps, number of mapped
fields.
-Lookups against data sources -Watch out for objects with very different validation require-
(internal to the process, or exter- ments in different status states – these multiply the build and
nal systems) test effort
Successful Boomi Projects 13
Goals:
To deliver a set of integration processes running live in production. This includes system
testing and deployment, and may include conversion tasks if the integration processes
are to be used for data conversion, as well.
Deliverables:
-Comprehensive system tests that are designed and executed from a business process
point of view, not just from an integration technical point of view.
-Updates or fixes to the Processes driven by a UAT punchlist.
-Deployment documentation, as noted above
-A set of deployed processes, with appropriate tracking and environment extension prop-
erties, running in production.
-If in scope, data conversion processes and load cycles that load an empty target system
from the source system as part of the project. This is often a separate deployment and
UAT cycle.
Risks:
-Are there well-populated sandbox environments, or test accounts, that can support ex-
tensive system and UAT testing?
-Is there a business process-level system test plan which will exercise all aspects of the in-
tegration? Testing to interface specs can miss important questions like: Who will update
this field, and how and when will they do it?
-Is the integration going to be used to do data conversion as well? That is, accomplish the
initial load of the target system? This is a separate project all in itself, even starting with a
tested interface, and can add significantly to the scope of the overall project.
-Are the runtime UAT and Prod atom environments in place?
Estimation Guidelines:
-Number of Processes, number of connectors in the processes, number of Extensions.
-Scope of system test plan
-Requirement to do data conversion with the newly developed integration processes –
volume and variation of that data.
Successful Boomi Projects 15
In the heat of battle when bringing a big project live, you can end up with multiple
versions of Profiles, logic, and processes. Boomi has great versioning and where-used
tools, but it is very easy to end up with three identical record profiles in use in five differ-
ent processes. You want to delete and consolidate the “dead objects” and not leave it
for the next guy to try to figure out what is the difference, if any, between these two five
hundred element XML profiles.
We use specialist tech writers to capture mapping and logic detail from the Boomi envi-
ronment.
Plus, you will inevitably be doing some fine tuning of the Interface as un-planned opera-
tional and data situations occur.
16 Successful Boomi Projects
Goals:
To make the integration processes “Maintenance-Ready”, and hand off the processes
to the maintenance team.
Deliverables:
-A set of “cleaned-up” boomi processes, where redundant components have been
removed, and naming and process structure is consistent from Process to Process
-Documentation that describes the overall context, and specifics about what is differ-
ent and unique about each process. Documentation need not describe each process
in detail, but should identify process and document properties, environment exten-
sions, any extensive scripting, and error handling, at a minimum
-Exported main maps, in excel form, identifying the version number.
-Go live documentation, as described above.
-Handoff to the maintenance team – often done as a walk-through of the processes
in the build environment, and review of recent Process Reporting logs
Risks:
-Have multiple developers worked on the project without clear guidelines and stan-
dards? If so, there may be a good bit of cleanup to do
-Are there trained Analysts to take over the maintenance? You don’t need to be at
a developer level to keep an eye on a set of Boomi processes in production, but it
helps to have a playbook containing common error situations, and what to do.
-Are the environments stable – source, target, and atom environments?
Estimation Guidelines:
-Number of Processes, number of connectors in the processes, overall hours esti-
mate for the project.
Successful Boomi Projects 17
Integration Analysis
– the “Zero Sprint”:
An Integration process in production can be thought of as a technical means for spread-
ing a single business process over multiple technical environments.
It often appears that through the integration, one process is handing off transactions to
another, or sharing data with another that is unrelated. It is common to think of the two
ends of the integration in isolation, and just to focus on the data or transactions that are
passing between them.
You can miss important aspects of the project when you treat the endpoints as indepen-
dent. A classic situation is order fulfillment, where order information is passed from one
system to another, but product master file information that is needed in fulfillment (like
addition of new products) is not passed.
A way to get a better view of the dependencies that might otherwise be missed in the
integration is to do and Integration Analysis up front, and ask some key questions:
-Is there a higher level business process that includes both ends of the integration en-
vironment? For instance, you might identify Product Management and Customer Ser-
vice as higher level business processes that need to be supported by the integration.
-What are the needs of these higher-level processes? Product Management may want
to insure consistent product master information. Customer Service may want to have
order fulfillment status and history information available.
-What business events impact both sides of the process? These are often the gotchas
that only become clear once things are in production. For instance, cancelled orders
must often be rolled back on both sides.
In Conclusion:
It is my hope that this White Paper has given
you a framework with which to think about,
structure, scope, and estimate integration proj-
ects. I’ve distilled my many learnings over an
extensive IT services career in hopes that you
might benefit, and execute integration projects
more successfully.
Larry Cone
About Larry Cone
Larry Cone, Founder and Chief Solutions Architect at KitePipe, has been active in the IT Services
sector for most of his career, first at Accenture, where he learned the basics of professional IT ser-
vices. Later as founder of Cone Software, he built a custom software development practice focused
on field and shop-floor computing in the Pharma industry. Along the way he led the software de-
velopment team for the Emmy and Academy-award-winning Skycam 3D volumetric camera motion
system.
Currently Leader at Kitepipe, he is applying business analysis, cloud architecture, and quality princi-
ples to deliver outstanding results in Cloud Integration Projects.
Larry manages the Boomi Services Team at Kitepipe, and is involved in all aspects of Boomi integra-
tion delivery. He consults with Dell-Boomi on Architect-Level training materials. He writes a popular
cloud applications blog at ittoolbox.com/coneblog. He spoke on Boomi integration architecture at
Gartner Data and Analytics 2017 and Boomi World 2017.”
Phone: 844.232.2227
E-mail: [email protected] Kitepipe
Web: Kitepipe.com Boomi Integration Services