BI Unit 5
BI Unit 5
Vidyalankar School of
Information Technology
Wadala (E), Mumbai
www.vsit.edu.in
Certificate
This is to certify that the e-book titled “BUSINESS INTELLIGENCE” comprises all elementary
learning tools for a better understating of the relevant concepts. This e-book is
comprehensively compiled as per the predefined eight parameters and guidelines.
DISCLAIMER: The information contained in this e-book is compiled and distributed for
educational purposes only. This e-book has been designed to help learners understand relevant
concepts with a more dynamic interface. The compiler of this e-book and Vidyalankar Institute
of Technology give full and due credit to the authors of the contents, developers and all
websites from wherever information has been sourced. We acknowledge our gratitude towards
the websites YouTube, Wikipedia, and Google search engine. No commercial benefits are
being drawn from this project.
Unit V
• Contents
• Recommended Books
• Text Books And Reference Books
• Business Intelligence: Data Mining and Optimization for Decision Making, Carlo
Vercellis, Wiley, First, 2009
• Decision support and Business Intelligence Systems , Efraim Turban, Ramesh
Sharda, Dursun Delen , Pearson , Ninth, 2011
• Fundamental of Business Intelligence, Grossmann W, Rinderle-Ma, Springer, First,
2015
Prerequisites Linkage
Unit V Sem. I Sem. Sem. III Sem. IV Sem. V Sem. MSc IT
II VI Sem. I
Knowledge Discrete - Data Computer Artificial - Data
Management Mathematics Structures Oriented Intelligenc Science
, Artificial Statistical e (Sem I),
Intelligence Technique Big Data
and Expert Analytics
Systems (Sem II),
Machine
Learning,
Robotic
Process
Automation
(Sem III),
Deep
Learning
(Sem IV)
KNOWLEDGE MANAGEMENT
Knowledge Management first came up in mid-1990 and many organizations have tried to use
it and failed to reap benefits. Knowledge Management strategy aims to enhance its
operations by leveraging internal and external expertise and assets, supporting exchange of
knowledge among individuals and groups
The term intellectual capital often used as synonym for knowledge implies that there is
financial value to knowledge. Explicit knowledge deals with more objectives, rational and
technical knowledge. Tacit knowledge is usually in the domain of subjective, cognitive and
experimental learning. It is highly personal and difficult to formalize. Explicit knowledge
comprises the policies, procedural guides, white papers, reports, designs, products,
strategies, goals, mission and core competencies of an enterprise and its IT infrastructure. It
is knowledge that has been codified in a form that can be distributed to others or transformed
into a process or strategy without interpersonal interaction. For example a description of how
to process a job application will be documented in a firm’s human resources policy manual.
Explicit knowledge has also been called leaky knowledge because of the ease with which it
can leave an individual, a document or an organization due to the fact that it can be readily
and accurately documented. Tacit knowledge is the cumulative store of the experiences,
mental maps, insights, acumen, expertise, knowhow, trade secrets, skill sets, understanding
and learning that an organization has as well as the organizational culture that has
embedded in the past and present experiences of the organizations people, process and
values. Tacit knowledge also referred to as embedded knowledge is usually either localized
within the brain of an individual or embedded in the group interactions within a department or
a branch office. Tacit knowledge typically involves expertise or high skill levels.
A learning organization must have organizational memory and a means to sane and share its
organizational knowledge. Individuals ideally tap into these past procedures and policies for
knowledge when faced with issues or problems. A KMS can capture the new knowledge and
make it available in its enhanced form.
Organizational learning is the development of new knowledge and insights that have the
potential to influence an organizational behavior. Learning skills include openness to new
perspectives, awareness of personal biases, exposure to unfiltered data and a sense of
humility. IT plays a critical role in organizational learning and management must place
emphasis on this area to foster it. Because organizations are becoming more virtual in their
operations they must develop the methods for effective organizational learning.
An organizations ability to learn, develop memory and share knowledge is dependent on its
culture. Over the time organizations learn what works and what does not. As the lessons
become old they become part of organizational culture. New employees learn the culture
from their mentors along with knowhow. Strong culture usually produces good results and
various incomes. Sharing initiatives and motivations are critical for the success of KM.
Following are the reasons due to which people does not want to share knowledge
• General lack of time to share knowledge and time to identify colleagues in need of specific
knowledge
• Apprehension or fear that sharing may reduce people’s job security
• Low awareness and realization of the value and benefit of the knowledge others process
• Dominance in sharing explicit over tacit knowledge such as knowhow and experience that
requires hands-on learning, observation, dialogue and interactive problem solving
• Use of strong hierarchy, position based status and formal power
• Insufficient capture, evaluation, feedback, communication and tolerance of past mistakes
that would enhance individual and organizational learning effects
• Differences in experience level
• Lack of contact time and interaction between knowledge sources and recipients
• Poor verbal or written communication and interpersonal skills
• Age and/or gender differences
• Lack of social network
• Differences in education level
• Ownership of intellectual property due to fear of not receiving recognition and accreditation
from managers and colleagues
• Lack of trust in people because they may misuse the knowledge or take unjust credit for it
• Lack of trust in the accuracy and credibility of knowledge due to source
• Differences in national culture or ethnic backgrounds and values and beliefs associated
with it
Knowledge Management Activities
Due to the changing dynamics and intense competition organizations need to leverage
intellectual resources in order to reduce the loss of intellectual capital due to people leaving
the company as well as to reduce the cost by decreasing the number of times the company
has to solve the same problem repeatedly. Companies use business intelligence to reveal
opportunities and create revenue generating programs to exploit them. Most KM initiatives
aim to make knowledge visible, to develop knowledge intensive culture and to build
knowledge infrastructure.
Knowledge creation is the generation of new insights, ideas and routines. It is interplay
between tacit and explicit knowledge and growing spiral as the knowledge moves among the
individual, group and organizational levels. The four modes of knowledge creation are
socialization, externalization, internalization and combination. The socialization mode refers
to the conversion of tacit knowledge to new tacit knowledge through social interactions and
shared experience among organization members. The combination mode refers to the
creation of new explicit knowledge by merging, categorizing, reclassifying and synthesizing
existing explicit knowledge. The other modes involve interactions and conversions between
tacit and explicit knowledge. Externalization refers to converting tacit knowledge to new
explicit knowledge. Internalization refers to creation of new tacit knowledge from explicit
knowledge.
Knowledge sharing is the willful explication of one person’s ideas, insights, solution,
experiences to another individual via a computer based system or directly. In many
organizations information and knowledge are not considered organizational resources to be
shared but individual competitive weapons to be kept private. Organizational members may
share personal knowledge with trepidation; they perceive that they are of less value if their
knowledge is part of organizational public domain. Research in organizational learning and
knowledge management suggest that some facilitating conditions include trust, interest and
shared knowledge.
Knowledge seeking also refereed as knowledge sourcing is the search for and use of
internal organizational knowledge. Lack of time or lack of reward may hinder the seeking of
knowledge. Individuals sometimes prefer not to reuse the knowledge if they feel that their
own performance review is based on originality or creativity of ideas. Individuals may engage
in knowledge creation, sharing and seeking with or without the use of IT tools
Topic – Knowledge Management
Source - https://www.youtube.com/watch?v=ny78qImWLVY
Approaches to Knowledge Management
There are two approaches to knowledge management
Process Approach – It attempts to codify organizational knowledge through formalized
controls, processes and technologies. Organizations that adopt this approach implement
explicit policies governing how knowledge is to be collected, stored and disseminated
throughout the organization. This approach frequently involves the use of IT, such as
intranets, data warehousing, knowledge repositories, decision support tools and groupware
to enhance the quality and speed of knowledge creation and distribution in organization. The
main criticisms of the process approach are that it fails to capture much of the tacit
knowledge embedded in firms and it forces individuals into fixed patterns of thinking. This
approach is favored by firms that sell relatively standard products that fill common needs.
Best Practices - Best practices are the activities and methods that the most effective
organizations use to operate and manage various functions. Various levels of best practices
are
• A good idea that is not yet proven but makes intuitive sense
• A good practice, an implemented technique, a methodology, a procedure or a process that
has improved business results
• A local best practice, a best approach for all or a large part of the organization based on
analysis of hard data
• An industry best practice, similar to the third level but using hard data from the industry
Most knowledge repositories are developed using several different storing mechanisms
depending on the types and amount of knowledge to be maintained and used. Each has
strengths and weakness when used for different purposes within a KMS. The most important
aspects and difficult issues are making the contribution of knowledge relatively easy for the
contributor and determining a good method for cataloguing the knowledge. Typical
development approaches include developing a large scale Internet based system or
purchasing a formal electronic document management system or a knowledge management
suite. The structure and development of the knowledge repository are a function of the
specific technology used for the KMS
KMS cycle – A functioning KMS follows six steps in cycle as seen in the figure below. The
reason for the cycle is that knowledge is dynamically refined over time. The knowledge in a
good KMS is never finished because the environment over time and the knowledge must be
updated to reflect the changes.
Knowledge discovery in databases (KDD) – It is a process used to search for and extract
useful information from volumes of documents and data. It includes tasks such as knowledge
extraction, data archaeology, data exploration, data pattern processing, data dredging and
information harvesting. All these activities are conducted automatically and allow quick
discovery even by non-programmers. Data and document mining is ideal for eliciting
knowledge from databases, documents, e-mail and so on. Data are often buried deep within
very large databases, data warehouses, text documents or knowledge repositories all of
which may contain information and knowledge gathered over many years. KDD can also be
used to identify the meaning of data or text, using knowledge management tools that scan
documents and email to build up an expertise profile of a firm’s employees. Data mining can
speed up analysis by providing needed knowledge
Web 2.0. – It has evolved from a tool for disseminating information and conducting business
to a platform for facilitating new ways of information sharing, collaboration and
communication in the digital age. A new vocabulary has emerged as mashups, social
networks, media sharing sites, RSS, blogs and wikis have come to characterize the genre of
interactive applications collectively known as Web 2.0. These technologies rea expected to
give KM a strong boost by making it easy and natural for everyone to share knowledge over
the Web. The ultimate value of Web 2.0 is its ability to foster greater responsiveness, better
knowledge capture and sharing and ultimately more effective collective intelligence.
Topic – Knowledge Representation and Reasoning in Artificial Intelligence
Source - https://www.youtube.com/watch?v=9iN3O_oL2ac
The CEO, Officers and Mangers of the Organization – CEO is responsible for
championing a knowledge management effort. He ensures that a competent and capable
CKO is found and that the CKO can obtain all resources needed for the project. CEO must
also gain organization wide support for contributions to and use of the KMS. CEO must also
prepare the organization for the cultural changes that are about to occur. CEO is a primary
change agent of the organisation.
The officers generally must make available to the CKO the resources needed to get the job
done. The Chief Financial Officer (CFO) must ensure that the financal resources are
available. The Chief Operating Officer (COO) must ensure that people beginto embed
knolwledge management practices into their daily work processes. The Chief Informaion
Officer (CIO) is responsible for the IT vision of the oragnization and for the IT architecture.
Managers must also support knowledge management effort and provide access to sources of
knowledge. In many KMS managers are an integral part of the communities of practice. A
Community Of Practice (COP) is a group of people in an organization with acommon
professional interest. All KMS users should each be in atleast one COP. Properly creating
and nurturing COP is one key to KMS success. COP owns the knowledge that it cintributes
because it manages the knowledge on its way into the system and must approve
modifications to it. The COP is responsible for the accuracy and timeliness of the knowledge
it contributes and for identifying its potential use. COP makes organization run smoothly
because they enable knowledge flow
KMS Developers – These are the team menbers who actually develop the system. They
work for the CKO. Some are organizational experts who develop strategies to promote and
manage the organizational culture shift. Others are involved in system software and
hardware, selection, programming etc.
KMS Staff – KMS requires full staff ato catalogue and mange knowledge. This staff is either
located at the firm’s headquarters or dispersed in knowledge centers throughout the
organization. Some KMS staff are functional area experts who are now cataloging and
apprioving knowledge contributions and pushing the knowledge to clients and employees
who can use the knowledge.
ARTIFICIAL INTELLIGENCE AND EXPERT SYSTEMS
A system that integrates knowledge from experts is commonly called Knowledge Based
Decision Support System (KBDSS) or an Intelligent Decision Support System (IDSS). The
foundation for building such systems is the tools and techniques that have been developed in
the area of artificial intelligence and expert systems
Experts – An expert is person who has special knowledge, judgement, experience and skills
to put his or her knowledge in action to provide sound advice and to solve complex problems
in a narrowly defined area. An expert knows which facts are important and also understands
and explains the dependency relationship among those facts. Typically human experts are
capable of doing the following
• Recognizing and formulating a problem
• Solving a problem quickly and correctly
• Explaining a solution
• Learning from experience
• Restructuring knowledge
• Breaking rules if necessary
• Determining relevance and associations
• Declining gracefully
Expertise – Expertise is the expensive task specific knowledge that experts possess. The
level of the expertise determines the performance of a decision. It is often acquired through
training, reading and experience in practice. It includes explicit knowledge such as theories
learned from a textbook or in classroom and implicit knowledge gained from experience. The
following is the list of possible knowledge types
• Theories about the problem domain
• Rules and procedures regarding the general problem domain
• Heuristics about what to do in a given problem solution
• Global strategies for solving these types of problems
• Metaknowledge
• Facts about the problem area
Expertise often includes following characteristics
• Usually associated with a high degree of intelligence but it is not always associated with
the smartest person
• Usually associated with a vast quantity of knowledge
• Based on learning from past success and mistakes
• Based on the knowledge that is well stored, organized and quickly retrievable from an
expert who has excellent recall of patterns from previous experiences.
Features of ES
ES must have following features
Expertise – ES must possess expertise that enables it to make expert level decisions. The
system must exhibit expert performance with adequate robustness
Symbolic reasoning – In ES knowledge must be represented symbolically and the primary
reasoning mechanism must be symbolic. Typical symbolic reasoning mechanisms include
backward chaining and forward chaining
Deep knowledge – It concerns the level of expertise in a knowledge base. The knowledge
base must contain complex knowledge not easily found among nonexperts
Self-knowledge – ES must be able to examine their own reasoning and provide proper
explanations as to why a particular conclusion was reached. ES also need to be able to learn
from their success and failures as well as from other sources.
The development of ES is divided into two generations one which uses if-then rules to
represent and store knowledge and the second generation which uses fuzzy logic, neural
networks or genetic algorithms with rule based inference to achieve a higher level of decision
performance
Source - https://www.youtube.com/watch?v=T3cOU_ljfr8&t=128s
Explanation Subsystem (Justifier) – It can trace the responsibility and explain the ES
behaviour by interactively answering the questions. In most ES the questions are answered
by sowing the rule that required asking a specific question and showing the sequence of the
rules that were used to derive the specific recommendations.
Knowledge Refining System – Human experts have a knowledge refining system that is
they can analyse their own knowledge and its effectiveness, learn from it and improve on it
for future consultations. Such evaluation is necessary in expert systems so that a program
can analyse the reasons for its success or failure which could lead to improvements resulting
in a more accurate knowledge base and more effective reasoning. The critical component of
a knowledge refinement system is the self-learning mechanism that allows it to adjust its
knowledge base and its processing of knowledge based on the evaluation of its recent past
performances.
Knowledge Engineering
The collection of intensive activities encompassing the acquisition of knowledge from human
experts and conversion of this knowledge into a repository are called knowledge engineering.
Knowledge engineering requires cooperation and close communication between the human
expert and the knowledge engineer to successfully codify and explicitly represent the rules
that a human expert uses to solve problems within a specific application domain. A major
goal of knowledge engineering is to help experts articulate how they do and what they do and
to document this knowledge in a reusable form. According to narrow perspective knowledge
engineering deals with the steps necessary to build the expert system and according to the
broad perspective the term describes the entire process of developing and maintaining any
intelligent systems. The following figure shows the process of knowledge engineering
A critical problem in the development of ES is the identification of the experts. This problem
can be solved by using multiple experts in the knowledge elicitation process. Some of the
advantages and shortcomings of multiple experts are given below
Four possible scenarios can be used when working with multiple experts
Individual experts – In this case several experts contribute knowledge individually. Using
multiple experts in this manner relieves the knowledge engineer of the stress associated with
multiple expert teams. However this approach requires that the knowledge engineer have a
means of resolving conflicts and handling multiple lines of reasoning.
Primary and secondary experts – A primary expert may be responsible for validating
information retrieved from other domain experts. Knowledge engineers may initially consult
the primary expert for guidance in domain familiarization, refinement of knowledge acquisition
plans and identification of potential secondary experts
Small groups – Several experts may be consulted together and asked to provide agreed
upon information. Working with small groups of experts allows the knowledge engineer to
observe alternative approaches to the solution of a problem and the key points made in the
discussion among experts
Panels – To meet the goals of ongoing development efforts a program may establish a
council of experts. The members of the council meet together at times for the purpose of
reviewing knowledge base development efforts, content and plans.
Production Rules – These are the most popular form of knowledge representation of an
expert system. It is represented in the form of condition/action pairs like
IF the stop light is red and you have stopped THEN the right turn is okay
Knowledge And Inference Rules – Knowledge rules or declarative rules state all the facts
and relationships about a problem. Inference rules or procedural rules offer advice on how to
solve a problem. The knowledge engineer separates the rules in which knowledge rules go to
knowledge base and inference rules become a part of inference engine. The knowledge rules
might look like
IF the inflation rate declines THEN the price of gold goes down
Inference rules might look like this
IF the data needed are not in the system THEN request them from the user
Inferencing – It is the process of using the rules in the knowledge base along with the known
facts to draw conclusions. It requires some logic embedded in a computer program to access
and manipulate the stored knowledge. This program controls the reasoning process and is
called inference engine or rule interpreter.
Development of Expert Systems
The development of ES includes the following parts
Defining the nature and the scope of the problem – This is the first step of ES where rule
based ES is appropriate when the nature of the problem is qualitative, knowledge is explicit,
and experts are available to solve the problems effectively and provide their knowledge.
Another important factor is to define a feasible scope. The current technology is still very
limited and is capable of solving smaller problems. Therefore the scope of the problem
should be specific and narrow.
Identifying proper experts – The next step is to find proper experts who have the
knowledge and are willing to assist in developing the knowledge base. No ES can be
designed without the strong support of knowledgeable and supportive experts. A proper
expert should have a thorough understanding of the problem-solving knowledge, the role of
the ES and decision support technology and good communication skills.
Acquiring knowledge – Knowledge acquisition is a time consuming and risky process.
Experts may be unwilling to share their knowledge because their knowledge is proprietary
and valuable or certain knowledge is tacit and the expert may not have the skill to dictate the
rules and considerations or they may be too busy to have enough time for communication or
certain knowledge may be confusing or contradictory in nature or the engineer may
misunderstand the expert and inaccurately document knowledge. The result of knowledge
acquisition is a knowledge base that can be represented in different formats like decision
trees or decision tables or if-then-else rules. The knowledge in the knowledge base must be
evaluated for its consistency and applicability
Selecting the building tool – After the knowledge base is built the next step is to choose a
proper tool for implementing the system. There are three different kinds of tools
General purpose development environment – Most programming languages support the
if-then statement, therefore it is possible to use C++ to develop an ES for a particular
problem domain. Because these programming languages do not have built in inference
capabilities using them is very time consuming and costly affair. Prolog and LISP are two
languages for developing intelligent systems. It is easier to use them than C++ but they are
still specifically designed for professional programmers and are not very friendly. For recent
Web based applications Java and computer languages that support Web services are also
useful.
ES shells – This tool is specifically designed for ES development. An ES shell has built-in
inference capabilities and a user interface, but the knowledge base is empty. System
development is therefore a process of feeding the knowledge base with rules elicited from the
expert.
Tailored turn-key solutions – This tool is tailored to a specific domain and can be adapted
to similar application very quickly. This tool contains specific features often required for
developing applications in a particular domain. This tool must adjust or modify the base
system by tailoring the user interface or relatively small portion of the system to meet the
unique needs of an organization.
Choosing an ES tool – Choosing among these tools foe ES development depends on few
criteria’s. Tailored turn-key solutions are the most expensive option and one must consider
the total cost and not just the cost of the tool. Secondly one must determine whether the tool
provides the functionality you need and how easily it allows the development team to make
necessary changes. Thirdly you must consider the tools compatibility with the existing
information infrastructure in the organization. Finally you need to consider the reliability and
vendor support of the tool
Coding the system – After choosing the tool the development team can focus on coding the
knowledge base on the tools syntactic requirements. The major concern at this stage is
whether the coding process is efficiently and properly managed to avoid errors. Skilled
programmers are helpful and important.
Evaluating the system – After the ES system is built it must be evaluated. Evaluation
includes both verification and validation, verification ensures that the resulting knowledge
base contains knowledge exactly the same as that required from the expert. In other words
verification ensures that no error occurred at the coding stage. Validation ensures that the
system can solve the problem correctly. In other words validation checks whether the
knowledge acquired form the expert can indeed solve the problem effectively
• Graded Questions
1. What is Knowledge Management?
2. What are the characteristics of Knowledge Management?
3. Write a short note on Explicit knowledge and Tacit knowledge
4. What are the reasons due to which people does not want to share knowledge?
5. Explain in brief what is knowledge creation?
6. What is knowledge sharing and knowledge seeking?
7. What is process approach in knowledge management
8. What is practice approach in knowledge management?
9. What are the best practices in knowledge management?
10. Write a short note on KMS cycle
11. What are the different AI methods used in KMS
12. Explain in brief Intelligent Agents
13. Write a short note on knowledge discovery in databases
14. How can KMS be integrated with AI?
15. What is the purpose of CKO?
5 It is difficult to estimate the value of sharing one’s knowledge or even who will benefit
most from it means ________________.
a Fragmentation, leakage and b Extraordinary leverage an increasing returns
the need to refresh
c Uncertain value d Value of sharing
6 ______________ deals with more objectives, rational and technical knowledge.
a Explicit knowledge b Expressive knowledge
c Tacit knowledge d All of these
8 __________ is interplay between tacit and explicit knowledge and growing spiral as
the knowledge moves among the individual, group and organizational levels.
a Knowledge Creation b Knowledge Sharing
c Knowledge Seeking d Knowledge Operating
12 ___________ is a process used to search for and extract useful information from
volumes of documents and data.
a Knowledge discovery in b Extensible markup language
databases
c Intelligent Agents d Web 2,0
14 ___________ has evolved from a tool for disseminating information and conducting
business to a platform for facilitating new ways of information sharing, collaboration
and communication in the digital age.
a Knowledge discovery in b Extensible markup language
databases
c Intelligent Agents d Web 2,0
18 In __________ step of KMS cycle knowledge is created as people determine new ways
of doing things or develop knowhow.
a Capture Knowledge b Refine Knowledge
c Create Knowledge d Store Knowledge
19 In _________ step of KMS cycle new knowledge must be identified as valuable and be
represented in a reasonable way.
a Capture Knowledge b Refine Knowledge
c Create Knowledge d Store Knowledge
20 In _______ step of KMS cycle new knowledge must be placed in context so that it is
actionable
a Capture Knowledge b Refine Knowledge
c Create Knowledge d Store Knowledge