A Cloud- and Container-Based Approach to Microservices-Powered Workflows (CodeOne 2018, San Francisco)

A Cloud- and
Container-Based
Approach to
Microservices-
Powered
Workflows
A Cloud- and Container-Based Approach to Microservices-Powered Workflows 1
Lucas Jellema, CTO of AMIS
CodeOne 2018, San Francisco, USA

Lucas Jellema
Architect / Developer
1994 started in IT at Oracle
2002 joined AMIS
Currently CTO & Solution Architect

Overview
• Definitions – what is workflow | microservice
• And what are objectives & requirements
• Bare essence of workflow
• Challenges with workflows – especially in a microservices world
• Approaches
• Orchestrated
• Choreographed
• Hybrid
• Required components
• Tools & technologies

Defining Workflow
• Cross domain cutting concern
• Composite transaction
• Multi-step chain
• Long running process
• System initiated human participation

Objectives
• Flows completed
• In time
• Following the plan
• Handling of non-happy situations
• Efficient execution
• Regarding resource usage (compute and human)
• Agility – easily adaptable

Examples
• From Webshop Order to Fulfillment and Invoice
• CI/CD Pipeline – from source code to running container
• Composite Order (“concert ticket, flight ticket, hotel reservation, car rental”)
• Travel Approval process
• CQRS-style refresh of query-stores upon update of command-database
• Bid management process
• From Blood Sample to Lab Results and Notification
• Nightly job to process data [in several steps]
• Process complex incoming message into
multiple domains [and data stores]
• From “order food to deliver meal”
• ChatBot – conversation flow
• Synchronous Service Call Retry
• …

Source: https://thenewstack.io/5-workflow-automation-use-cases-you-might-not-have-considered/

Constituents of a Workflow
• Activities (& actor roles)
• Flow logic
• Sequence
• Conditional
• Events (including time)
• Loop
• Parallellism
• Deadlines
• Events and Signals that trigger or influence
• Transaction boundaries
• Succeed or rollback together
• Exceptions, non-happy-flow, compensation handlers
• State – data associated with an instance of a workflow
• Including the progress & status of the instance (where are we at?)
• Business indicators (per instance and across instances) & business monitoring

Workflow Design | Blueprint
• Communication with business
• Input for implementation of workflow
• Input for implementing business monitoring & reporting
• Input for a workflow engine – to execute
• Examples of formal notation methods:
• BPMN
• BPEL
• CMMN
• Harel Statecharts
• State Diagrams
• Petri net

Examples of workflow designs

At the core
• The logical definition of the workflow
combined with
the current state of the instance
and the current time (or other external conditions)
• To produce
one or more “To Do items” for activity types in the context of the workflow instance
• Including non-happy exception items (for example when previous to-do item timed out)
• The To Do items should be made available to actors (for example microservices)
• Including (reference to the) state
• Actors can ‘take on’ a To Do item – typically exclusively
• They can read and extend the state of the workflow instance
(Including completing | failing | returning the task)

Tasks go through states
• Task == Workflow Step == Activity
• Each state change requires reevaluating the workflow-instance

Workflow instances go through states
Business state and Operational state
• New
• Running
• Waiting on
• Actors
• Events
• Failed
• Completed

Microservices
• Business Agility
• Functionality: quick, cheap, effortless and risk free
• IT Agility
• Non-functionality: scale, resilience, infrastructure & location
• Independent components
• Asynchronous communication – whenever possible
• Encapsulated
• Location does not matter
• Strictly within one domain, owned by one team
• Not too big or complex
• Horizontally Scalable (multiple instances )
• Ephemeral, Stateless
• Enabling Automated DevOps

• Manage data context for Workflow instance
• Persist
• Share
• Update
• Derive next state
• from data context
• Event (time)
• Workflow definition
To make workflows work in a microservices arena…
• Ensure workflow instances are completed according to plan
• Agile evolution of workflow definitions (and data structure)
• Short & Long-running workflow
• Provide business insight in status and progress
Workflow in Microservices Land • Prevent multiple actors from working
on the same task
• Detect tasks not picked up or
abandoned
• Act on events & functional timeouts
• Microservice (& Lambda [Stateless] Functions)
• Assume that ownership for each workflow lies in one domain
• Someone cares – because of what that team tries to achieve
• Preferably Asynchronous communication
• Multiple instances of actor
• Distributed/unknown | changing location
• Poly-tech implementations
• Frequent scale & redeploy & replace (plus A/B and Canary)
• Smart Endpoints, Dumb Pipes
• Cloud – wide area network
• Containers – ephemeral (stateless), restart, multi-instance

Approaches
• Orchestration
• Choreography
• Hybrid
• Coordinated | Facilitated Choreography
• Mixing orchestration and choreography

Orchestration
• Central coordination
• Flow logic
• Actor invocation (synchronous?) and communication
• Transaction
• Exception, Timeout and Event & Signal handling
• Workflow instance state & data content
• Within and/or across domains
Workflow Orchestration Engine
actor
actor
actor

Orchestration – challenges & undesirabilities
• Hard dependency on actors: what, where, how to invoke
• Any change in actor may impact central orchestrator
• Monolithic orchestrator may become bottleneck
• Physically (defying Ops – scale, patch, …)
• Mentally (god service, omniscient )
• (in the past?) very not agile:
• running instances, changing data structure & workflow definitions
• Several products provide(d) this capability – and have sometimes made life hard and given
workflow orchestration a bad name
• Oracle BPM Suite, Camunda, jBPM, Activiti, Pega Systems, Tallyfy, Bizagi, Oracle Integration
Cloud (PCS), IBM Business Process Manager, Red Hat Process Automation Manager

Choreography
• No one is owner of the workflow instance
• Full independence between all actors
• Microservices know nothing about each other
• Events trigger them into action
• Their end state is published through an event
• The workflow does not explicitly exist
• Arises as sequence of independent microservice actions
• Microservice need to know about the event that should trigger them
• Highly flexible
• As long as actors are acting on events – they can be anywhere, scaled in
anyway, doing anything and be implemented in any technology

Example of Choreography – Flowing Retail – Process Order
Source: Bernd Rücker, https://github.com/berndruecker/flowing-retail
ShipmentInventoryPayment
Shop &
Checkout

Example of Event Driven Choreography – Flowing Retail
Order
Completed

Pure Choreography - demo

Code in Actor
Inventory

Choreography Challenges
• How to share the workflow state (“data context”)
• Hard to implement flow logic – e.g. conditional actions or loops
• Hard to handle parallel activities on same “instance”
• State is payload of event
• Changing the implicit workflow requires changing the microservices
• The way they respond to events
• Tracing the workflow is hard
• Detecting and fixing stuck and failing
workflow instances is hard
• Who determines if
the Order is completed?
Order
Completed

Event Driven Pitfall regarding workflows and pure
choreography

Guided Choreography
• Workflow definition exists
• Workflow definition is instantiated as routing slip that is included in events
• Available to each actor
• Actors determine if the routing slip for an instance allows | prompts them to act
• If so, they perform work then update routing slip and publish an event
• And so on
• Extremely flexible
• Deploy and redeploy actors as desired
• A/B and Canary testing
• Modifying workflow definitions
• Potentially even for running instances

Workflow
[Event]
Monitor
Guided Choreography with Routing Slip
Workflow Initiator
Workflow
definitions
Workflow event
EventBus
Actor
Actor
Actor
Check if the routing slip
contains a task that this
actor can perform and
that is released for
execution
execution
execution
execution
execution
execution
Payload
Instance Identifier
Workflow Definition Identifier
State (associated data)
Tasks
- Activity type
- Identifier
- Status (new | done | waiting | failed)
- Conditions/Dependencies
Audit
If it is, then perform the
task and publish a new
workflow event with
updated state, task &
audit
Workflow event

Routing Slip
Retrieve
Payment
checkout
Retrieve
Goods
Ship
Goods
Cancel
Order

Challenges with Guided Choreography – Routing Slip
• Prevent multiple actors picking up the same task in an instance (concurrency)
• Exclusively claim a task
• Handle state updates by actors on parallel tasks (split brain state of routing slip)
• Perhaps store state in distributed cache
• Potentially inefficient as each actor evaluates all workflow events
• For all workflows and all instances
• Detect failing instances
• Handle timers and signals
A Cloud- and Container-Based Approach to Microservices-Powered Workflows
done

Best of Both Worlds: Hybrid – Coordinated Choreography
• Asynchronous communication based on queues | commands | events
• Distributed, stateless, horizontally scalable workflow engine
• Data context (“state”)
• State transition (workflow logic)
• Communication (Event) handling
• Publish tasks, receive task updates
• Handle external and time triggers
• Detect abandoned tasks, failing workflows
• Publish metrics for monitoring workflow instances
• Governance on [definitions of] Workflows and Events
• Ensure that events are understood and an actor is available for each task
[event]

Commands are Events that express what should happen
• Example: the Retrieve Payment event

Decider – state engine
• Take workflow definition
• Take state & state change [events]
• Take context (time)
• Derive new state
• Status of workflow instance
• Status of activities
• Update persisted state
• Inform task dispatcher
Workflow
Instance State
Workflow
Definitions
Task
Queue/Dispatcher
Signals &
Events
Decider

Workflow definition management
• Hold definitions of multiple workflows
• And the associated data structures
and event messages
• Manage multiple versions of each workflow definition
• and validity period for each version
• Aid upgrade of running instances
Workflow
Definitions

Task dispatcher & Actors
• Publish task & data context
• Allow actors to pick up [and claim] task
• Detect unclaimed tasks and <do something>
• Detect timed out tasks | failing actors
• Detect regularly completed tasks
• Task dispatcher & Actors publish metrics
A Cloud- and Container-Based Approach to Microservices-Powered Workflows 35A Cloud- and Container-Based Approach to Microservices-Powered Workflows 35
Task
Queue/Dispatcher
Actor
ActorActor
Topic/
Queue
Publish tasks to be
performed; with exactly
once delivery
Task
Heart
beat
Task
Update
Detect failed
actor/reschedule task
Interrupt running
task (send signal)

Facilitated Choreography
or: Orcheography
Workflow
Instance State
Workflow
Definitions
Task
Queue/Dispatcher
Actor
ActorActor
Topic/
Queue
SweeperSignals &
Events
Decider
Derive new state (status & actions
to release) from workflow definition,
current state, context (e.g. time)
Publish tasks to be
once delivery
Detect abandoned
| stuck workflow
instancesHandle external events and
signals that could impact running
instances (note: event from one
workflow instance can be signal to
other)
Produce
time(out) events
for workflow
instances
Task
Heart
beat
Detect failed
Deploy minor and
major versions of
workflow definitions
Hold data context for
workflow instance
Interrupt running
task (send signal)

Orcheography

Additionally in our workflow execution toolset
• Human participants
• Allocate
• Notify
• Provide multichannel Task UI
• Task Management
• Business indicators
• Find WIP, Waste, Bottlenecks
• Monitoring
• Individual instances & Aggregates per Workflow
• Technical/IT perspective & Business Activity
• Rule Engine (for business logic inside the workflow)

Involving Human Actors in Workflow
Human ParticipantWorkflow
allocate
Who should
perform this
task?
notify
How to inform task-
holder about new |
expiring todo item?
multi-channel
task specific user interface
task management (todo list,
claim | reject | delegate task)
Enable human
to perform task
(data, status)?
Enable human
to manage all
her tasks

Microservice is actor as far as workflow engine should know –
it decides if and how to involve a human
µ
Workflow
Task
Queue/Dispatcher
Notification
Service
Multi Channel Facilities
(chatbot, portal, mobile)
Generic Task
Management
application
Task
specific
UI
Directory

Microservice is actor – proxy for the human contributor(s)
µ
Notification
Service
Multi Channel Facilities
(chatbot, portal, mobile)
Generic Task
Management
application
Task
specific
UI
Directory

Facilities for
Workflow Management
Workflow &
Task Metrics
Workflow
Instance State
Workflow
Definitions
Task
Queue/Dispatcher
Monitoring
& Reporting
Actor
Actor
Topic/
Queue
SweeperSignals &
Events
Decider
Derive new state (status & actions
to release) from workflow definition,
current state, context (e.g. time)
Publish tasks to be
once delivery
Detect abandoned
| stuck workflow
instancesHandle external
events and signals
that could impact
running instances
Produce
time(out) events
for workflow
instances
Task
Heart
beat
Detect failed
Deploy minor and
major versions of
workflow definitions
Hold data context for
workflow instance
Interrupt running
task (send signal)
Actor
Allocation,
Notification,
Task Mgt
Rule
Evaluation

Orchestration with proxy actors for
decoupled (microservice) actors
A Cloud- and Container-Based Approach to Microservices-Powered Workflows
Orchestration engine
Proxy
Actor
Proxy
Actor
Proxy
Actor
Proxy
Actor
µ µ λ SOAP
API Gateway
Enterprise
Service Bus
Multi-instance,
Distributed,Scalable
Flexible workflow definition
Shared, flexible instance
state
Workflow
Instance State
µµµ
SOAP
SOAP

Hybrid (2)
• Embrace (or at least allow) synchronous orchestration within a domain or bounded context
• For (parts of) flows that are the responsbility within a domain – and a team
• And use (facilitated) choreography for flows stretching across domains
• To retain strong decoupling and flexibility between domains
• Perhaps a Proxy service can consume the “choreographed” event and turn it in locally
orchestrated logic

Cross Bounded Context |
Domain
Orchestrator
Orchestrator
Facilitated
Choreography
Workflow & Task
Metrics Workflow
Instance
State
Workflow
Definitions
Task
Queue/Dispatcher
µ
µ µ
λ
µ
λ
µ
Proxy
Actor

Cross Domain
Orchestrator
Orchestrator
Facilitated
Choreography
Workflow & Task
Metrics Workflow
Instance
State
Workflow
Definitions
Task
Queue/Dispatcher
µ
µ µ
λ
µ
λ
µ
Proxy
Actor

Demo Hybrid Orchestration (Orcheography) – Flowing Retail
Place
Order

Demo Orcheographed Workflow
Orchestrator
Proxy
Actor
Order Context
Checkout Context
OrderPlaced
Event
Event Bus
Payment Context
Inventory ContextShipment Context
GoodsShipped
Event GoodsFetched
Event
PaymentReceived
Event
Proxy
Actor
Proxy
Actor
Retrieve
Payment
Fetch Goods Ship Goods
µ
µ
µ
µ
µ
µ
Ship Goods Fetch Goods
Retrieve
Payment
Orchestrator

Orchestrator
Proxy
Actor
Order Context
Checkout Context
OrderPlaced
Event
Event Bus
Payment Context
Inventory ContextShipment Context
GoodsShipped
Event GoodsFetched
Event
PaymentReceived
Event
Proxy
Actor
Proxy
Actor
Retrieve
Payment
Fetch Goods Ship Goods
µ
µ
µ
µ
µ
µ
Ship Goods Fetch Goods
Retrieve
Payment
Orchestrator

Event Bus

Summary
• Workflows exist – also in microservice environments
• Short running composite transactions  Long running business process
• Responsibility for running workflow instances can be a cross cutting concern – outside the
scope of any individual microservice
• All generic workflow components need to be agile, scalable, distributed, cloud-enabled
• For resilience, scale, flexible evolution, optimal use of resources
• A lot can happen over the lifetime of a workflow instance – that need to be catered for
• Events, changes in data context, modification of workflow definition Scenarios
• Workflows within single bounded context could be pure orchestration
• Workflows across bounded contexts should use decoupled, choreographed workflow
coordination [between bounded contexts]
• That can span across technologies, physical locations, vendors and clouds

Summary
• Several frameworks, services and tools are available for supporting workflow
management (e.g. AWS SWF, Zeebe, Camunda, Baker, Cadence, Conductor, Project
Fn Flow , Azure Logic Apps)
• Born from real life needs
• Microservice oriented and [hybrid] cloud enabled
• At heart pre-configured combinations of queue, event bus, NoSQL data store, rule
engine, …
• Roll your own can be fun –
and also quite challenging

Source: https://www.infoq.com/articles/events-workflow-automation

Thank you
Dank je wel
• Blog: technology.amis.nl
• Email: lucas.jellema@amis.nl
• : @lucasjellema
• : lucas-jellema
• : www.amis.nl, info@amis.nl

Challenges
• Exactly once delivery of task to actor
• Lock? Queue? Direct call?
• Detect failed | abandoned task execution (& reschedule)
• Heartbeat? Timeout?
• Compensation (for failed transaction)
• Timer events
• Handle Signals/Events to impact running instance
• Correlation (tags/indexes) to locate impacted instances
• Communicate with/interrupt actors
• Monitor individual instances and across instances
• Deal with peak load and high priority instances and tasks
• Distributed, scaled & Ephemeral actors and workflow engine
• How to design workflow in a way that users understand, IT-staff can create
and workflow engines can process

A Cloud- and Container-Based Approach to Microservices-Powered Workflows (CodeOne 2018, San Francisco)

More Related Content

What's hot (20)

Similar to A Cloud- and Container-Based Approach to Microservices-Powered Workflows (CodeOne 2018, San Francisco) (20)

More from Lucas Jellema (20)

Recently uploaded (20)

A Cloud- and Container-Based Approach to Microservices-Powered Workflows (CodeOne 2018, San Francisco)

Editor's Notes