Process Automation and

Apache Kafka

March 2022 Mai 2022

camunda.de
2 Contents

Contents

Microservice orchestration 3

Event-driven architecture and emergent behavior 5

Event streaming and workflow engines 5

What’s next? 7

camunda.de

3 Microservice orchestration

Apache Kafka can help users implement a Let’s look at an example based on an actual
variety of use cases. Business processes customer project: You place an order in an
are often automated as a result of online store. The order is accepted via a
these implementations – consciously or checkout service. The system can now write an
unconsciously – and Apache Kafka is frequently event to Kafka stating that an order has been
combined with workflow engines. To explore placed (“order placed”). An order fulfillment
the automation of business processes in the service can respond to this event. This service
context of Apache Kafka further, this article will is then responsible for processing the order,
touch on the following topics: which requires it to communicate with other
services, such as a payment service that is used
1. Microservice orchestration to process the payment. This can also take
2. Event-driven architecture with “emergent place via Kafka, whereby a command is written
behavior” to Kafka and is then processed by the payment
3. Combining event streaming with workflow service. This sequence is shown in Figure One.
engines
The order fulfilment service now orchestrates
other services. The term “orchestrate” can be
Microservice orchestration understood in the same way as “coordinate”,
which you are perhaps more familiar with.
Many modern architectures rely on
microservices to split functionality into smaller, Of course, the necessary orchestration logic
manageable pieces. These microservices could be implemented in a programming
need to communicate with each other in order language of your choice. But orchestration in
to implement business functionality. This distributed systems has special requirements,
communication can take place via Kafka, where mostly because it is long-running. After all,
Kafka is used here in place of a message bus, you never know how long you’ll have to wait,
often in order to be able to scale accordingly. either for a payment to be received, or for

Figure One: Microservices

can communicate via Kafka
and exchange events and
commands.

camunda.de

4 Microservice orchestration

Figure Two: Possible architecture of order fulfilment service

a package labelling service to be available. This is why a workflow engine is used within the
This is why a workflow engine is an obvious order fulfillment microservice. Modern products
solution for implementing orchestration logic, allow a great deal of flexibility to be hooked
since it provides its own persistence and is in various architectures. Figure Two shows
therefore very good at waiting even long times. an example where Camunda SaaS is being
In addition, those tools allow the graphical used, while microservices are written in Java
representation of orchestration logic – not with Spring Boot. The microservices contain
just during development, but also during source code for combining the workflow engine
runtime. Diagnosing problems is one example with Apache Kafka as sketched in [1]. Different
of graphical representation during runtime. architectures are, of course, possible. If you
This is very helpful as such logic often needs want to dive deeper into process automation
to be discussed and understood by various architectures you might want to look into [2].
stakeholders such as business or domain
specialists, infrastructure people, and, of course, 1 https://github.com/berndruecker/kafka-camunda-spring-simple
developers. 2 https://processautomationbook.com/

camunda.de

5 Event-driven architecture and emergent behavior

Event-driven architecture and an order: No one is monitoring whether an

order has actually been delivered within a
emergent behavior specific time (SLA); if an order gets stuck, it
is not easy to find out why this has happened
Describing an orchestration engine as part of and how to get things moving again; there isn’t
your microservices architecture typically leads anyone actually responsible for the end-to-end
to a discussion about if you need orchestration process who considers how it can be optimized.
and a workflow engine at all. Couldn’t you just Perhaps a significant number of COD orders are
build an event-driven architecture leveraging canceled if shipping takes longer than three
what is called choreography? The latter is days - nobody knows!
connected to the expectation of designing a
better microservices architecture in which the It is important to think about who is responsible.
individual services are optimally decoupled In a purely event-driven architecture,
from each other. responsibility for end-to-end business
processes is not specifically allocated. In a
In order to better understand this argument, number of customer projects, however, this
let’s take another look at our example. You has turned out to be an anti-pattern. In
could get along without a dedicated order microservice architectures you should better
fulfillment service if the payment service try to find a balance between orchestration and
responds directly to an “order placed” event. event-driven communication. This is explained
Once the payment has been received, a further in [3].
“payment received” event is created. The
inventory service listens to this event in order
to have the goods ready from the warehouse. Event streaming and workflow
Thus, the actual business process is created
from a chain of responses to specific events by
engines
the various microservices. This event-chain was
never designed with the big picture in mind, but Speaking of Apache Kafka and workflow
it is called “emergent behavior.” engines, there is another combined use case
that is very exciting. I’ll explain this one using
Emergent behavior has its advantages, namely a real-world example: Vehicles in a customer’s
that behaviors do not have to be designed and fleet continuously gather sensor data and send
implemented in advance. A good use for this it via Kafka. This could be that the current oil
approach is sending customer notifications pressure in vehicle xyz is 6 bar, for example.
related to their order. Instead of the team This information on its own is pretty useless – is
developing the payment service having to spend 6 bar good or bad? Stream processors respond
a long time thinking about when and how the to these events to provide insights, such as
customer will be informed that their payment the oil pressure being too high. These insights
has been received, a separate service can are collected in a new event stream. Finding
simply react to the “payment received” event. insights from sensor data is pretty complex.
This notification service could be implemented Maybe high oil pressure only becomes a
without even consulting the team responsible problem after five minutes, perhaps even
for the payment service. This is exactly the type depending on the ambient temperature. But
of decoupling that we want. those requirements can be covered easily with
Apache Kafka Streams.
However, emergent behavior can be very risky
if there is a logical flow or even an end-to-end Now, however, a simple list of insights is not yet
business process that needs to be implemented. enough. Insights must also trigger an action. In
This is because you cannot see the the real-world project, process instances were
implementation of those business processes started in a workflow engine for new insights,
anywhere. It cannot be designed or optimized
according to a specific purpose. This can then 3 https://learning.oreilly.com/library/view/practical-process-
lead to a number of issues when processing automation/9781492061441/

camunda.de

6 What’s next?

Figure Three: Executable process model for implementing responses to insights

for example, to arrange for vehicle maintenance This model works very well and connects the
or even to stop the vehicle immediately in an world of event streaming, often in the context of
emergency. The workflow engine is relevant in IoT and mass data, with processes to organize
those processes whenever they become long responses that are then often long-running.
running, for example if humans are involved. We will be seeing this model a lot more in the
Figure Three shows a simplified version of a future.
process model that can be executed directly in
the workflow engine.

To wire together Apache Kafka and the

Camunda workflow engine, a dedicated
connector was developed in the project. This
connector starts process instances for new
insights or influences existing process instances
if one insight occurs. In the process model, for
example, the maintenance scheduling was
canceled when the oil pressure returned to
normal, but the cancellation still had to be
accepted by the mechanic for safety reasons.
Figure Four shows a schematic overview of the
streaming plus workflow architecture.

camunda.de

7 What’s next?

Figure Four: Schematic interaction between the event stream and workflow engine

What’s next?
This article explored two typical models for
linking Apache Kafka with a workflow engine:
microservice orchestration and responding
to insights in event streams via long-running
processes. This was a high-level overview, your
next step could be to look at the technical
details. These can depend on a number of
parameters, including the architecture and
stack, the specific tools or the program
language. Unfortunately, there isn’t enough
time to explore that in this blog post, but I
hope this has piqued your interest, and if so,
I recommend reading the book “Practical
Process Automation” or taking a look at the
book’s website. These resources explore the
topic further and also provide links to specific
code examples to get you started.

Finally, I think the best advice I can offer you

now is: if you want to get started with a new
theme, start with a proof of concept. You
can also find unbureaucratic advice in the
Confluent4 or Camunda5 forums. Enjoy!

4 https://forum.confluent.io/
5 https://forum.camunda.io/

camunda.de

 Bernd Rücker
Chief Technologist and Co-Founder of
Camunda

Bernd Rücker has been developing software

to help customers automate their core
processes for 15 years, including ordering
processes for Zalando, order processes
for T-Mobile and patent applications in
Switzerland. He has been actively involved
in the development of several open
source workflow engines and is the co-founder of Camunda,
a company that is reinventing process automation. He is the
author of “Practical Process Automation” and co-author of
“Real-Life BPMN”, speaks regularly at conferences and also
writes for a number of magazines. For some time, he has been
working on process automation paradigms that fit into modern
architectures involving microservices, domain-driven design,
event-driven architecture, and reactive systems.

About Camunda
Camunda is the leader in process orchestration software. Our software helps orchestrate
complex business processes that span people, systems, and devices. With Camunda,
business users collaborate with developers to model and automate end-to-end processes
using BPMN-powered flowcharts that run with the speed, scale, and resiliency required to
compete in today’s digital-first world. Hundreds of enterprises such as Allianz, ING, and
Vodafone design, automate, and improve mission-critical business processes with Camunda
to drive digital transformation. To learn more visit camunda.com.