Process Miing
Process Miing
Julia Rudnitckaia
Abstract. At last decades people have to accumulate more and more data in different areas.
Nowadays a lot of organizations are able to solve the problem with capacities of storage
devices and therefore storing “Big data”. However they also often face to problem getting
important knowledge from large amount of information. In a constant competition
entrepreneurs are forced to act quickly to keep afloat. Using modern mathematics methods
and algorithms helps to quickly answer questions that can lead to increasing efficiency,
improving productivity and quality of provided services. There are many tools for
processing of information in short time and moreover all leads to that also inexperienced
users will be able to apply such software and interpret results in correct form.
One of enough simple and very powerful approaches is Process Mining. It not only allows
organizations to fully benefit from the information stored in their systems, but it can
also be used to check the conformance of processes, detect bottlenecks, and
predict execution problems.
This paper provides only the main insights of Process Mining. Also it explains the key
analysis techniques in process mining that can be used to automatically learn process models
from raw event data. Generally most of information here is based on Massive Open Online
Course: "Process Mining: Data science in Action".
Data science is the profession of the future, because organizations that are unable
to use (big) data in a smart way will not survive. Clive Humby even offered to use
metaphor “Data is new oil” to emphasize what important role the data plays nowadays
[7]. It is not sufficient to focus on data storage and data analysis. The data scientist
also needs to relate data to process analysis. Process mining bridges the gap between
traditional model-based process analysis (e.g., simulation and other business process
management techniques) and data-centric analysis techniques such as machine
learning and data mining.
Figure 1. Positioning of PM
Process mining (PM) techniques are able to extract knowledge from event logs
commonly available in today's information systems. These techniques provide new
means to discover, monitor, and improve processes in a variety of application
domains. There are two main drivers for the growing interest in process mining:
1) more and more events are being recorded, thus, providing detailed information
about the history of processes;
2) there is a need to improve and support business processes in competitive and
rapidly changing environments.
Process Mining provides an important bridge between BI and BPM, Data Mining
and Workflow. It includes (automated) process discovery (i.e., extracting process
models from an event log), conformance checking (i.e., monitoring deviations by
comparing model and log), social network/ organizational mining, automated
construction of simulation models, model extension, model repair, case prediction,
and history-based recommendations.
In the table below there are pointed various directions of analysis and significant
questions that PM can answer to. [6]
Figure 2. The three basic types of process mining explained in terms of input and
output
Play-
Out
Play-In
Replay
1. Play-Out
Input is finished process model. Next you may simulate different scenarios of a
process (according to the model) for filling the event log by data recorded during the
simulation events.
Event Logs – collection of cases, where each element refers to a case, an activity
and a point in time (timestamps).
You can find sources of event data everywhere. For instance, in database system,
transaction log (e.g. a trading system), business suite/ ERP system (SAP, Oracle…),
message log (e.g. from IBM middleware), open API providing data from websites or
social media, CSV (comma-separated values) or spreadsheet etc.
When extracting event log, you can face the next challenges:
1) Correlation. Events in an event log are grouped per case. This simple
requirement can be quite challenging as it requires event correlation, i.e., events need
to be related to each other.
2) Timestamps. Events need to be ordered per case. Typical problems: only
dates, different clocks, delayed logging.
3) Snapshots. Cases may have a lifetime extending beyond the recorded period,
e.g. a case was started before the beginning of the event log.
4) Scoping. How to decide which tables to incorporate?
5) Granularity. The events in the event log are at a different level of
granularity than the activities relevant for end users.
Additionally event logs without preprocessing have so called noise and
incompleteness. The first one means the event log contains rare and infrequent
behavior not representative for the typical behavior of the process. And
incompleteness - the event log contains too few events to be able to discover some of
the underlying control-flow structures. There are many methods to “clean” data and
use only useful data as filtering and data mining techniques.
Every event log must have some certain fields, without that PM will be impossible.
Figure 7 clearly shows the basic attributes of the events in the logs:
Case ID - instances (objects), which are arranged sequence of events log.
Activity name - actions performed within the event log.
Timestamp - date and time of recording log events.
Resource - holds the key actors log events (those who perform actions in the
event log).
Figure 7. Example of Event log (the selection of the attributes depends on the
purpose of analysis)
Using Figure 7 we can list some assumptions about event logs:
• A process consists of cases
• A case consists of events such that each event relates to precisely one case.
• Events within a case are ordered.
• Events can have attributes
• Examples of typical attribute names are activity, time, costs, and resource.
This type of PM can help to find out what actual process model is. Based just on an
event log, a process model is constructed thus capturing the behavior seen in the log.
The most popular algorithms used for this goal are:
• Alpha Miner;
• Alpha+, Alpha++, Alpha#;
• Fuzzy miner;
• Heuristic miner;
• Multi-phase miner;
• Genetic process mining
• Region-based process mining (State-based regions and Language based
regions);
• Classical approaches not dealing with concurrency (Inductive inference
(Mark Gold, Dana Angluin et al.) and Sequence mining).
Also it’s necessary to have a good notification for represent ready process model to
end-user. As a rule commonly used Workflow Nets, Petri Nets, Transition Systems,
YAWL, BPMN, UML, Causal nets (C-nets) and Event-Driven Process Chain (EPCs).
But any discover technique requires such a representational bios. It helps limiting
the search space of possible candidate models. Also it can be used to give preference
to particular types of models. It’s important to observe process discovery is, by
definition, restricted by the expressive power of the target language. Therefore
representational bias – the selected target language for presenting and constructing
process mining results. Because of every notification isn’t universal and has its
limitations (e.g. silent steps, work with duplicate activities, with concurrency and
loops etc.) and benefits, it recommends to use different variants to correct
interpretation of process.
The following are the main characteristics of process discovery algorithms:
1) Representational bias:
• Inability to represent concurrency
• Inability to deal with (arbitrary) loops
• Inability to represent silent actions
• Inability to represent duplicate actions
• Inability to model OR-splits/joins
• Inability to represent non-free-choice behavior
• Inability to represent hierarchy
2) Ability to deal with noise
3) Completeness notion assumed
4) Used approaches - direct algorithmic approaches (α-algorithm), two-phase
approaches (TS, Markov model→ WF-net), computational intelligence approaches
(genetic algorithm, neural networks, fuzzy sets, swarm intelligence, reinforcement
learning, machine learning, rough sets), partial approaches (mining of sequent
patterns, discovery of frequent episodes), etc.
Completeness and noise refer to qualities of the event log and do not say much
about the quality of the discovered model. m. In fact, there are four competing quality
dimensions:
• Fitness. The discovered model should allow for the behavior seen in the event
log. A model with good fitness allows for most of the behavior seen in the
event log. A model has a perfect fitness if all traces in the log can be replayed
by the model from beginning to end.
• Precision. The discovered model should not allow for behavior completely
unrelated to what was seen in the event logs (avoid underfitting). Underfitting
is the problem that the model overgeneralizes the example behavior in the log
(i.e., the model allows for behaviors very different from what was seen in the
log).
• Generalization. The discovered model should generalize the example behavior
seen in the event logs (avoid overfitting). Overfitting is the problem that a
very specific model is generated whereas it is obvious that the log only holds
example behavior (i.e., the model explains the particular sample log, but a
next sample log of the same process may produce a completely different
process model).
• Simplicity. The discovered model should be as simple as possible.
In the Figure 8 it’s presented the key criteria for evaluating the quality of process
models with appropriate answers that explain meaning of every criteria.
Is the event log possible according to
the model? Is this the simplest model?
Is the model not overfitting (only allow Is the model not underfitting (allow
for the “accidental” examples)? for too much)?
PM is not limited only by discovery techniques. When one gets appropriate process
model, the most interesting and necessary part (for stakeholders) of analyzing begins.
The model may have been constructed by hand or may have been discovered.
Moreover, the model may be normative or descriptive. Conformance checking relates
in the event log to activities in the process model and compares both. It needs an
event log and a model as input. The goal is to find commonalities and discrepancies
between the modeled behavior and the observed behavior. This type of PM is relevant
for business alignment and auditing. The different types of models can be considered:
conformance checking can be applied to procedural models, organizational models,
declarative process models, business rules/policies, laws, etc.
Generally, conformance checking is used for:
• improving the alignment of business processes, information systems and
organizations;
• auditors;
• repairing models;
• evaluating process discovery algorithm;
• connecting event log and process model.
Following description of the technique for conformance checking will be mainly
focused on the one of quality criteria – fitness, because the other three quality
criteria are less relevant.
2.6.1. Conformance checking using Token-based play
The idea of this method is counting tokens while replaying, i.e. to simply count the
fraction of cases that can be “parsed completely” (the proportion of cases
corresponding to firing sequences leading from <start> to <end>). While replaying
on top of, for example, WF-net, we have four counters: p (produced tokens), c
(consumed tokens), m (missing tokens – consumed while not there) and r (remaining
tokens – produced but not consumed). Initially, p=c=0, then the environment
produces a token for place <start>. At the end, the environment consumes a token
from place end. For instance, if we try replay trace Ϭ= <a,d,c,e,h> on the top of given
model, then final state of replay will look like in the figure 9. For this case, we will
have following counters: p=6, c=6, m=1, r=1.
Figure 9. Replaying trace Ϭ= < a,d,c,e,h > on top of WF-net (final state)
Fitness for trace will be computed by formula:
(1)
,where Ϭ– trace from event log, N – process model.
Then fitness for trace Ϭwill be equal to:
(2)
The fitness of an event log L on WF-net N is defined as:
(3)
By help this method we can analyze and detect problem in compliance as it’s
shown in figure below.
This method allows for log-to-model comparisons, i.e. it can be checked whether
and model and event log “agree” on the ordering of activities. However, the same
approach can be used for log-to-log and model-to-model comparisons.
It should provide a "closest matching path" through the process model for any trace
in the event log. Also required for performance analysis. In figure 12 the first string
shows possible trace in model and second one shows move in log only. Cost of
alignment are differences between model and log. The goal is to find the most optimal
alignment (with minimal cost).
Today many data are updated in real-time and sufficient computing power is
available to analysis events when they occur. Therefore, PM should not be restricted
to off-line analysis and can also be used for online operational support.
Figure 14 shows refined PM Framework (can be extended). Provenance refers to
the data that is needed to be able to reproduce an experiment. Data in event logs are
portioned into “pre mortem” and “post mortem”. “”Post mortem” – information about
cases that have completed and can be used for process improvement and auditing, but
not for influencing the cases. “Pre mortem” – cases that have not yet completed and
can be exploited to ensure the correct or efficient handling the cases.
The refined PM Framework also distinguish between two types of models: “de jure
models” and “de facto models”. The first one is normative, and second one is
descriptive.
2.9.1 Detect
Figure below illustrates type of operational support. Users are interacting with
some enterprise information system. Based on their actions, event are recorded. The
partial trace of each case is continuously checked by the operational support system,
which immediately generates an alert if a deviation is detected.
2.9.2 Predict
We again consider the setting in which users are interacting with some enterprise
information system (viz. Figure 16). The events recorded for cases can be sent to the
operational support system in the form of partial traces. Based on such a partial trace
and some predictive model, a prediction is generated.
Figure 16. Both the partial trace of a running case and some predictive model are
used to provide a prediction.
2.9.3. Recommend
The setting is similar to prediction. However, the response is not a prediction but a
recommendation about what do next (viz. Figure 17). To provide such a
recommendation, a model is learned from “post mortem” data. A recommendation is
always given with respect to a specific goal. For example, to minimize the remaining
flow time or the total cost, to maximize the fraction of cases handled within 4 weeks,
etc.
Figure 17. A model based on historic data is used to provide recommendations for
running cases.
2.10. Tools
All techniques described above were realized in such software as PROM. ProM is
an extensible framework that supports a wide variety of process mining techniques in
the form of plug-ins.
The main characteristics of PROM:
Aims to cover the whole process mining spectrum.
Notations supported: Petri nets (many types), BPMN, C-nets, fuzzy models,
transition systems, Declare, etc.
Also supports conformance checking and operational support.
Many plug-ins are experimental prototypes and not user friendly.
This is extremely powerful instrument, but confusing for someone. Nowadays
already exist 600 plug-ins and this amount grows up.
There is also commercial software Disco that has following characteristics:
Focus on discovery and performance analysis (including animation).
Powerful filtering capabilities for comparative process mining and ad-hoc
checking of patterns.
Uses a variant of fuzzy models, etc.
Does not support conformance checking and operational support.
Easy to use and excellent performance.
Disco can be used by unexperienced people, has intuitive user-friendly interface.
Tools are available (Figure 18), but process mining is still a relatively young
discipline. New tools will appear in coming years and process mining functionality
will be embedded in more BI/BPM/DM suites.
In table 2 described popular tools for process mining with specifying benefits and
lacks.
Lasagna processes are relatively structured and the cases flowing through such
processes are handled in a controlled manner. Therefore it’s possible to apply all of th
PM techniques presented in the preceding chapters. Definition of Lasagna process is:
a process is a Lasagna process if with limited efforts it is possible to create an agreed-
upon process model that has a fitness of at least 0.8, i.e., more than 80% of the events
happen as planned and stakeholders confirm the validity of the model. In figure below
presented example of Lasagna process by help Petri net and BPMN.
Easy to discover, but it is less interesting to show the "real" process. (close to
expectation)
Whole process mining toolbox can be applied.
Added value is predominantly in more advanced forms of process mining based
on aligning log and model.
Spaghetti processes are less structured than Lasagna processes, only some of
process mining techniques can be applied. Figure below shows why unstructured
processes are called Spaghetti processes. There are different approaches to get
valuable analyze from such kind of processes. For example, method Divide and
Conquer (by clustering of cases) or showing only the most frequencies paths and
activities (Disco).
Figure 21. Applications of Spaghetti (violet cells), Lasagna (blue cells) processes
and both (pink cells).
Nevertheless, Spaghetti processes are very interesting from the viewpoint of PM as
they often allow for various improvements. A highly-structured well-organized
process is often less interesting in this respect; it’s easy to apply PM techniques but
there is also little improvement potential.
3. Conclusions
- There are no negative examples (i.e., a log shows what has happened but
does not show what could not happen).
- Preprocessing of Event log (problems with Noise and Incompleteness)
- Concept drift [13, 14]
- There is no clear how to correct recognize attributes of event log[12]
- Due to concurrency, loops, and choices the search space has a complex
structure and the log typically contains only a fraction of all possible
behaviors.
- There is no clear relation between the size of a model and its behavior (i.e., a
smaller model may generate more or less behavior although classical
analysis and evaluation methods typically assume some monotonicity
property).
- Improving the Representational Bias Used for Process Discovery
- Balancing Between Quality Criteria such as Fitness, Simplicity, Precision,
and Generalization
- Improving Usability and Understandability for NonExperts
- Cross-Organizational Mining
- etc.
Moreover, PM can be used off-line and online. And from a technological point of
view online PM may be challenged.
Even so mature PM techniques and tools are available and it successfully used in
over organizations.
References
[1] Wil M.P. van der Aalst Process Mining: Discovery, Conformance and
Enhancement of Business Processes
ISBN: 978-3-642-19344-6, DOI 10.1007/978-3-642-19345-3, Springer Heidelberg
Dordrecht London New York, 2002
[2] Massive Open Online Course: "Process Mining: Data science in Action". Wil van
der Aalst Eindhoven University of Technology, Department Methematics &
Computer Science
[3] IEEE CIS Task Force on Process Mining. Process Mining Manifesto
[4] Wil van der Aalst :“Process Mining: X-Ray Your Business Processes“, Process
Mining, Communications of the ACM CACM Volume 55 Issue 8 (August 2012)
Pages 76- 83, <http://doi.acm.org/10.1145/2240236.2240 257>.
[5] Anne Rozinat1, Wil van der Aalst Process Mining: The Objectification of Gut
Instinct - Making Business Processes More Transparent Through Data Analysis
[6] Website www.habrahabr.ru/post/244879/, article Introduction in Process Mining
[7] Michael Palmer Data is the New Oil
[8] D.Harel and R.Marelly. Come, Let’s play: Scenari-Based Programming Using
LSCs and Play-Engine. Springer, Berlin, 2003.
[9] Website www.processmining.org
[10] Website www.fluxicon.com
[11]J.C. Bose, R. Mans, and W. van der Aalst. Wanna improve process mining
results? Computational Intelligence and Data Mining (CIDM 2013), doi:
10.1109/CIDM.2013.6597227
[12] Burattin Andrea. Applicability of Process Mining techniques in Business
Environments
[13]Christian Günter. Process Mining in Flexible Environment.
[14] Arjel Bautista, Lalit Wangikar, S.M. Kumail Akbar “Process Mining-Driven
Optimization of a Consumer Loan Approvals Process” CKM Advisors, 711 Third
Avenue, Suite 1806, NY, USA