Business Intelligence?

Business intelligence, or BI for short, is an umbrella term that refers to competencies, processes, technologies, applications and practices used to support evidence-based decision making in organizations. In the widest sense it can be defined as a collection of approaches for gathering, storing, analyzing and providing access to data that helps users to gain insights and make better fact-based business decisions.

BI used for?
Organizations use Business Intelligence to gain data-driven insights on anything related to business performance. It is used to understand and improve performance and to cut costs and identify new business opportunities, this can include, among many other things: Analyzing customer behaviors, buying patterns and sales trends. Measuring, tracking and predicting sales and financial performance Budgeting and financial planning and forecasting Tracking the performance of marketing campaigns Optimizing processes and operational performance Improving delivery and supply chain effectiveness Web and e-commerce analytics Customer relationship management Risk analysis Strategic value driver analysis

Basics of Business Intelligence

Gathering data is concerned with collecting or accessing data which can then be used to inform decision making. Gathering data can come in many formats and basically refers to the automated measurement and collection of performance data. For example, these can come from transactional systems that keep logs of past transactions, point-of-sale systems, web site software, production systems that measure and track quality, etc. A major challenge of gathering data is making sure that the relevant data is collected in the right way at the right time. If the data quality is not controlled at the data gathering stage then it can harm the entire BI efforts that might follow always remember the old adage - garbage in garbage out Storing Data Storing Data is concerned with making sure the data is filed and stored in appropriate ways to ensure it can be found and used for analysis and reporting. When storing data the same basic principles apply that you would use to store physical goods say books in a library you are trying to find the most logical structure that will allow you to easily find and use the data. The advantages of modern data-bases (often called data warehouses because of the large volumes of data) is that they allow multi-dimensional formats so you can store the same data under different categories also called data marts or datawarehouse access layers. Like in the physical world, good data storage starts with the needs and requirements of the end users and a clear understanding of what they want to use the data for. Analyzing Data The next component of BI is analysing the data. Here we take the data that has been gathered and inspect, transform or model it in order to gain new insights that will support our business decision making. Data analysis comes in many different formats and approaches, both quantitative and qualitative. Analysis techniques includes the use of statistical tools, data mining approaches as well as visual analytics or even analysis of unstructured data such as text or pictures. Providing Access In order to support decision making the decision makers need to have access to the data. Access is needed to perform analysis or to view the results of the analysis. The former is provided by the latest software tools that allow end-users to perform data analysis while the latter is provided through reporting, dashboard and scorecard applications. *******************************************************

What

is

Data

Warehouse?

A data warehouse is a relational database that is designed for query and analysis rather than for transaction processing. It usually contains historical data derived from transaction data, but it can include data from other sources. In addition to a relational database, a data warehouse environment includes an extraction, transportation, transformation, and loading (ETL) solution, an online analytical processing (OLAP) engine, client analysis tools, and other applications that manage the process of gathering data and delivering it to business users. A data warehouse is a subject-oriented, integrated, time-variant and non-volatile collection of data in support of management's decision making process(characteristics of Datawarehouse).

Subject-Oriented: A data warehouse can be used to analyze a particular subject area. For example, "sales" can be a particular subject. Integrated: A data warehouse integrates data from multiple data sources. For example, source A and source B may have different ways of identifying a product, but in a data warehouse, there will be only a single way of identifying a product. Time-Variant: Historical data is kept in a data warehouse. For example, one can retrieve data from 3 months, 6 months, 12 months, or even older data from a data warehouse. This contrasts with a transactions system, where often only the most recent data is kept. For example, a transaction system may hold the most recent address of a customer, where a data warehouse can hold all addresses associated with a customer. Non-volatile: Once data is in the data warehouse, it will not change. So, historical data in a data warehouse should never be altered. *******************************************************

Data Warehouse Architecture

Now we are goanna define each and every terminology in the above picture to facilitate better understanding of the subject. 1. Operational Data Store : is a database designed to integrate data from multiple sources for additional operations on the data. The data is then passed back to operational systems for further operations and to the data warehouse for reporting. 2. ERP : Enterprise resource planning integrates internal and externalmanagement information across an entire organization, embracingfinance/accounting, manufacturing, sales and service, etc. Its purpose is to facilitate the flow of information between all business functions inside the boundaries of the organization and manage the connections to outside stakeholders. 3. CRM : Customer relationship management is a widely-implemented strategy for managing a companys interactions with customers, clients and sales prospects. It involves using technology to organize, automate, and synchronize business processesprincipally sales activities, but also those for marketing, customer service, and technical support. Customer relationship management describes a company-wide business strategy including customer-interface departments as well as other departments. 4. Flat Files In data Ware Housing : Flat Files Doesnt Maintain referential Integrity like RDBMS and are Usually seperated by some delimiters like comma and pipes etcs. Right from Informatica 8.6 unstructured data sources like Ms-word,Email and Pdf can be taken as source. 5. ETL (Extract,Transform, And load) : is a process in database usage and especially in data warehousing that involves: Extracting data from outside sources Transforming it to fit operational needs (which can include quality levels) Loading it into the end target (database or data warehouse) 6. Data Marts: A data mart (DM) is the access layer of the data warehouse (DW) environment that is used to get data out to the users. The DM is a subset of the DW, usually oriented to a specific business line or team. For the Definition of the Data Warehouse Please Refer to Introduction to the Data ware Housing. 7. OLAP : OLAP (Online Analytical Processing) is a methodology to provide end users with access to large amounts of data in an intuitive and rapid manner to assist with deductions based on investigative reasoning. OLAP systems need to: Support the complex analysis requirements of decision-makers, Analyze the data from a number of different perspectives (business dimensions), and Support complex analyses against large input (atomic-level) data sets. 8. OLTP : Online transaction processing, or OLTP, refers to a class of systems that facilitate and manage transaction-oriented applications, typically for data entry and retrieval transaction processing.

9. Data Mining: Is the process of extracting patterns from large data sets by combining methods from statistics and artificial intelligence withdatabase management. Data mining is seen as an increasingly important tool by modern business to transform data into business intelligence giving an informational advantage. *******************************************************

Data

modeling

Data modeling is the process of creating a data model by applying formal data model descriptions using data modeling techniques. In other words Data modeling can be defined as a method used to define and analyze data requirements needed to support the business processes of an organization. Conceptual, logical and physical schemas : Conceptual schema: This consists of entity classes, representing kinds of things of significance in the domain, and relationships assertions about associations between pairs of entity classes. A conceptual schema specifies the kinds of facts or propositions that can be expressed using the model. Logical schema: This consists of descriptions of tables and columns, object oriented classes, and XML tags, among other things.. Physical schema: describes the physical means by which data are stored. This is concerned with partitions, CPUs, table spaces, and the like. Modeling methodologies : Data models represent information areas of interest. While there are many ways to create data models, But only two modeling methodologies standard top-down and bottom-up are used in real time environment : Bottom-up models : are often the result of a reengineering effort. They usually start with existing data structures forms, fields on application screens, or reports. These models are usually physical, applicationspecific, and incomplete from an enterprise perspective. They may not promote data sharing, especially if they are built without reference to other parts of the organization. Top-down logical data models : on the other hand, are created in an abstract way by getting information from people who know the subject area. A system may not implement all the entities in a logical model, but the model serves as a reference point or template. *******************************************************

Dimensional

Modeling

Dimensional modeling (DM) is the name of a set of techniques and concepts used in data warehouse design. Dimensional modeling always uses the concepts of facts (measures), and dimensions (context). Facts : Facts are typically (but not always) numeric values that can be aggregated, and dimensions are groups of hierarchies and descriptors that define the facts. Types of Facts : There are three types of facts: Additive: Additive facts are facts that can be summed up through all of the dimensions in the fact table. Semi-Additive: Semi-additive facts are facts that can be summed up for some of the dimensions in the fact table, but not the others. Non-Additive: Non-additive facts are facts that cannot be summed up for any of the dimensions present in the fact table. Types of Fact Tables There are two types of fact tables: Cumulative: This type of fact table describes what has happened over a period of time. For example, this fact table may describe the total sales by product by store by day. The facts for this type of fact tables are mostly additive facts. The first example presented here is a cumulative fact table. Snapshot: This type of fact table describes the state of things in a particular instance of time, and usually includes more semi-additive and non-additive facts. The second example presented here is a snapshot fact table. Dimension : A dimension is a data element that categorizes each item in a data set into non-overlapping regions. A data warehouse dimension provides the means to "slice and dice" data in a data warehouse. Dimensions provide structured labeling information to otherwise unordered numeric measures. Types of Dimension : Conformed dimension : In data warehousing, a conformed dimension is a dimension that has the same meaning to every fact with which it relates. Conformed dimensions allow facts and measures to be categorized and described in the same way across multiple facts and/or data marts, ensuring consistent reporting across the enterprise. A conformed dimension can exist as a single dimension table that relates to multiple fact tables within the same data warehouse, or as identical dimension tables in separate data marts. Junk dimension : A Junk Dimension is a dimension table consisting of attributes that do not belong in the fact table or in any of the existing dimension tables .The junk dimension should contain a single row representing the blanks as a surrogate key that will be used in the fact table for every row returned with a blank comment field. The designer is faced with the challenge of where to put attributes that do not belong in the other dimensions,Solution is to create a new dimension for each of the remaining attributes, but due to their nature, it could be necessary to create a vast number of new dimensions resulting in a fact table with a very large number of foreign keys.

Degenerate dimension : A dimension key, such as a transaction number, invoice number, ticket number, or bill-of-lading number, that has no attributes and hence does not join to an actual dimension table. Degenerate dimensions are very common when the grain of a fact table represents a single transaction item or line item because the degenerate dimension represents the unique identifier of the parent. Degenerate dimensions often play an integral role in the fact table's primary key. Dimensional modeling structure: The dimensional model is built on a star-like schema, with dimensions surrounding the fact table. To build the schema, the following design model is used: Choose the business process Declare the Grain Identify the dimensions Identify the Fact Benefits of dimensional modeling : Benefits of the dimensional modeling are following: Understandability Query performance Extensibility ***************************************************************

Data

mining

Data mining is the process of extracting patterns from large data sets by combining methods from statistics and artificial intelligence with database management. Process: 1.Pre-processing Before data mining algorithms can be used, a target data set must be assembled. As data mining can only uncover patterns already present in the data, the target dataset must be large enough to contain these patterns while remaining concise enough to be mined in an acceptable timeframe. A common source for data is a data mart or data warehouse. Pre-process is essential to analyze the multivariate datasets before clustering or data mining. The target set is then cleaned. Cleaning removes the observations with noise and missing data. The clean data are reduced into feature vectors, one vector per observation. A feature vector is a summarized version of the raw data observation. The feature(s) selected will depend on what the objective(s) is/are; obviously, selecting the "right" feature(s) is fundamental to successful data mining.

The feature vectors are divided into two sets, the "training set" and the "test set". The training set is used to "train" the data mining algorithm(s), while the test set is used to verify the accuracy of any patterns found. 2.Data mining Data mining commonly involves four classes of tasks:[13] Clustering is the task of discovering groups and structures in the data that are in some way or another "similar", without using known structures in the data. Classification is the task of generalizing known structure to apply to new data. For example, an email program might attempt to classify an email as legitimate or spam. Common algorithms include decision tree learning, nearest neighbor, naive Bayesian classification, neural networks and support vector machines. Regression Attempts to find a function which models the data with the least error. Association rule learning Searches for relationships between variables. For example a supermarket might gather data on customer purchasing habits. Using association rule learning, the supermarket can determine which products are frequently bought together and use this information for marketing purposes. This is sometimes referred to as market basket analysis. 3.Results validation The final step of knowledge discovery from data is to verify the patterns produced by the data mining algorithms occur in the wider data set. Not all patterns found by the data mining algorithms are necessarily valid. It is common for the data mining algorithms to find patterns in the training set which are not present in the general data set, this is called overfitting. To overcome this, the evaluation uses a test set of data which the data mining algorithm was not trained on. The learnt patterns are applied to this test set and the resulting output is compared to the desired output. For example, a data mining algorithm trying to distinguish spam from legitimate emails would be trained on a training setof sample emails. Once trained, the learnt patterns would be applied to the test set of emails which it had not been trained on, the accuracy of these patterns can then be measured from how many emails they correctly classify. A number of statistical methods may be used to evaluate the algorithm such as ROC curves. If the learnt patterns do not meet the desired standards, then it is necessary to reevaluate and change the preprocessing and data mining. If the learnt patterns do meet the desired standards then the final step is to interpret the learnt patterns and turn them into knowledge.

***************************************************************

Difference between Datamarts and Data warehouse

the department level and developed Oriented, Integrated, Time-Variant, with a specific issue or subject in Nonvolatile collection of data in support mind, a data mart is a data of decision making. warehouse with a focused objective. 2.A data mart is used on a business 2.A data warehouse is used on an division/ department level. enterprise level. 3.A Data Mart is a subset of data 3.A Data Warehouse is simply an from a Data Warehouse. Data Marts integrated consolidation of data from a are built for specific user groups. variety of sources that is specially designed to support strategic and tactical decision making. 4.By providing decision makers with only a subset of data from the Data Warehouse, Privacy, Performance and Clarity Objectives can be attained. 4.The main objective of Data Warehouse is to provide an integrated environment and coherent picture of the business at a point in time.

In simple and logical terms,Data Warehouse is filled up with Data Marts, and each DataMart is a particular subject oriented collection of data