Planning a Data

Warehouse
Overview
 Review the essentials of planning for a data
warehouse
 Distinguish between data warehouse projects and
OLTP system projects
 Learn how to adapt the life cycle approach for a
data warehouse project
 Introduce agile development methodology for DW
projects
 Discuss project team organization, roles, and
responsibilities
Factors causing failures

 Improper planning
 Inadequate project management
 Company not ready for a data warehouse
 Insufficient staff training
 Improper team management
 No support from top management
Questions
 Develop criteria for assessing the value
expected from your data warehouse
 Decide
 the type of data warehouse to be built
 where to keep the data warehouse
 where the data is going to come from
 whether you have all the needed data
 who will be using the data warehouse
 how they will use it
 at what times will they use it
Decisions
 Decide
 the type of data warehouse to be built
 where to keep the data warehouse
 where the data is going to come from
 whether you have all the needed data
 who will be using the data warehouse
 how they will use it
 at what times will they use it
 Key Issues

Single
Value and Risk Top-Down or Vendor or
Build or Buy
Expectations Assessment Bottom-Up Best-of-
Breed

Asses the More than Plan and

value to be calculating define
derived from the loss from overall
the proposed the project requirements
data costs Find the
High level of
warehouse, proper
Look at the integration or
balance
Take into pros and products
between in-
account the cons of these best suited
house and
opportunities methods for
vendor
Make a list that will be objectives
software.
of realistic missed if Weight these
benefits and there is NO options and
expectations data document
warehouse them
Driving Force
 Business Requirements, Not Technology
 Understand the requirements
 Focus on
 user’s needs
 Data needed
 How to provide information
 Use a preliminary survey to gather general
requirements before planning
Preliminary Survey
 Mission and functions of each user group
 Computer systems used by the group
 Key performance indicators
 Factors affecting success of the user group
 Who the customers are and how they are classified
 Types of data tracked for the customers, individually and as
groups
 Products manufactured or sold
 Categorization of products and services
 Locations where business is conducted
 Levels at which profits are measured—per customer, per
product, per district
 Levels of cost details and revenue
 Current queries and reports for strategic information
Justification
1. Calculate the current technology costs to produce the applications and
reports supporting strategic decision making. Compare this with the
estimated costs for the data warehouse and find the ratio between the
current costs and proposed costs. See if this ratio is acceptable to
senior management.
2. Calculate the business value of the proposed data warehouse with the
estimated dollar values for profits, dividends, earnings growth, revenue
growth, and market share growth. Review this business value
expressed in dollars against the data warehouse costs and come up
with the justification.
3. Do the full-fledged exercise. Identify all the components that will be
affected by the proposed data warehouse and those that will affect the
data warehouse. Start with the cost items, one by one, including
hardware purchase or lease, vendor software, in-house software,
installation and conversion, ongoing support, and maintenance costs.
Then put a dollar value on each of the tangible and intangible benefits,
including cost reduction, revenue enhancement, and effectiveness in
the business community.
 Challenges for Data Warehousing
Project Management
DATA ACQUISITION DATA STORAGE INFO. DELIVERY

 Large number of  Storage of large data  Several user types

sources
 Many disparate  volumes  Queries stretched to
sources  Rapid growth limits
 Different computing  Need for parallel  Multiple query types
 platforms  processing  Web-enabled
 Outside sources
 Data storage in staging  Multidimensional
 Huge initial load
area analysis
 Ongoing data feeds
 Data replication  Multiple index types  OLAP functionality
considerations  Several index files  Metadata management
 Difficult data  Storage of newer data  Interfaces to DSS
integration
types apps.
 Complex data
transformations  Archival of old data  Feed into Data Mining
 Data cleansing  Compatibility with tools  Multi-vendor tools
 RDBMS & MDDBMS
Cope with differences in Data
Warehousing Projects
 Recognize that a data warehouse project
 has broader scope,
 tends to be more complex, and
 Involves many different technologies.
 Do not hesitate to find and use specialists wherever in-house talent is not
available.
 A data warehouse project has many out-of-the-ordinary tasks.
 Metadata in a data warehouse is so significant that it needs special
treatment throughout the project.
 Pay extra attention to building the metadata framework properly.
 to build and complete the infrastructure.
 to decide on the architecture design.
 for the evaluation and selection of tools.
 for training the users in the query and reporting tools.
 Involve the users in every stage of the project. Data warehousing could
be completely new to both IT and the users in your company. A joint effort
is imperative.
 Allow sufficient time Because of the large number of tasks in a data
warehouse project, parallel development tracks are absolutely necessary.
Be prepared for the challenges of running parallel tracks in the project life
cycle.
 Readiness Assesment Report

Purpose of
Assesment Report

The project Lower the risks of big surprises occurring during

manager performs implementation
assessment Provide a proactive approach to problem resolution
with the assistance
of an outside Reassess corporate commitment
expert.
Review and reidentify project scope and size

A formal readiness Identify critical success factors

assessment report
before the project
Restate user expectations
plan is prepared Ascertain training needs
Advantages of the life cycle
approach
• Accomplishes all the major objectives in the system
development process.
1

• Enforces orderliness and enables a systematic approach to

building computer systems.
2
• Breaks down the project complexity and removes any
ambiguity with regard to the responsibilities of project team
3 members.

• Implies a predictable set of tasks and deliverables.

4
Life Cycle Approach

The approach
A one-size-fits- for a data
The life cycle
all life cycle warehouse
approach
approach will project has to
breaks down the
not work for a include iterative
project
data warehouse tasks going
complexity
project. through cycles
of refinement.
System Development Life
Cycle for data warehousing
Sample Outline of a Project
Plan
 INTRODUCTION
 PURPOSE
 ASSESSMENT OF READINESS
 GOALS & OBJECTIVES
 STAKEHOLDERS
 ASSUMPTIONS
 CRITICAL ISSUES
 SUCCESS FACTORS
 PROJECT TEAM
 PROJECT SCHEDULE
 DEPLOYMENT DETAILS
DEVELOPMENT Phases
Development Phases

•The design phase and construction

phase for these three components of
DW may run in parallel.
•The phases must include tasks
•to define the architecture as composed
of the three components of DW
•and to establish the underlying
infrastructure to support the architecture.
What is Agile Development
 Based on iterative development
 Requirements and solutions evolve through
collaboration between self-organizing cross-
functional teams

Receive
Feedback

Client Tests Code/Design

Deliver Alpha
 Agile Development
striving for simplicity and not being bogged down in complexity, providing
Core and obtaining constant feedback on individual development tasks, fostering
Values free and uninhibited communication, and rewarding courage to learn from
mistakes.

Core encouraging quality, embracing change, changing incrementally,

Principles adopting simplicity, and providing rapid feedback.

creating short releases of application components, performing

Core development tasks jointly , working the 40-hour work week intensively, not
Practices expanding the time for ineffective pursuits, and having user representatives
on site with the project team.

Control variables that can be manipulated for trade-offs to achieve

Variables
results are time, quality, scope, and cost.
Project Team
Caution!
• Complexity overload
• Responsibility Ambiguity

List all the project challenges and specialized skills needed.

• planning,
• defining data requirements,
• defining types of queries,
• data modeling,
• tools selection,
• physical database design,
• source data extraction,
• data validation and quality control,
• setting up the metadata framework,
• ...
Using the list of challenges and skills prepare a list of team roles
needed to support the development work.
• assign individual persons to the team roles with the right abilities,
suitable skills and the proper work experience.
 Organizing the Project Team

Not necessary to assign one or

more persons to each of the
identified roles.
If the data
warehouse effort is Remember that the
Do not fail to
not large and your user representatives
recognize the users
company’s must also be
as part of the team
resources are considered as
and to assign them
meager, try making members of the
to suitable roles.
the same person project team.
wear many hats

Skills, experience, and knowledge

Important properties of team attitude, team spirit,
members : passion for the data warehouse effort,
strong commitment
Classification of Roles in the
Project Team
Data warehousing authors classify the roles or job titles in various
ways. They first come up with broad classifications and then include
individual job titles within these classifications.
 Staffing for
 initial development,
 testing,
 ongoing maintenance,
 data warehouse management
 IT and end-users,
 Subclassifications
 further subclassifications
 Front office roles, back office roles
 Coaches, regular lineup, special teams
 Management, development, support
 Administration, data acquisition, data storage,
information delivery
Job Titles in the Project Team
 Executive Sponsor  Data Acquisition  Data Mart Leader
Developer  Infrastructure
 Project Director
 Data Access Specialist
 Project Manager Developer  Power User
 User Representative  Data Quality Analyst
Manager  Training Leader
 Data Warehouse  Technical Writer
 Data Warehouse Tester
Administrator  Tools Specialist
 Maintenance  Vendor Relations
 Organizational Developer
Change Manager Specialist
 Data Provision  Web Master
 Database Specialist
Administrator  Data Modeler
 Business Analyst
 Metadata Manager  Security Architect
 System
 Business Administrator
Requirements
Analyst  Data Migration
Specialist
 Data Warehouse
Architect  Data Grooming
Specialist
Some Team Roles
 Executive sponsor  Data warehouse
 Project manager administrator
 User liaison manager  Data transformation
 Lead architect specialist
 Infrastructure specialist  Quality assurance analyst
 Business analyst  Testing coordinator
 Data modeler  End-user applications
specialist
 Development programmer
 Lead trainer
Roles and Responsibilities of
a Project Team
Executive Data Warehouse
• Direction, support, arbitration. Administrator
• DBA functions.
Sponsor
Data
• Data extraction, integration,
Project • Assignments, monitoring, Transformation
transformation.
Specialist
Manager control.
Quality Assurance
• Quality control for warehouse data.
User Liaison • Coordination with user Analyst
Manager groups.
Testing
Coordinator
• Program, system, tools testing.
Lead Architect • Architecture design.
End-User
• Confirmation of data
Applications
Specialist meanings/relationships.
Infrastructure • Infrastructure
Specialist design/construction. Development
Programmer
• In-house programs and scripts.

Business
• Requirements definition. • Coordination of User and Team
Analyst Lead Trainer
training.

• Relational and dimensional

Data Modeler
modeling.
 Roles and skills/experience levels required in the P
Executive Sponsor Project Manager User Liaison Manager Lead Architect

• Senior level executive, • People skills, • People skills, • Analytical skills,

• in-depth knowledge of • project management • respected in user • ability to see the big
the business, experience, community, picture,
• enthusiasm and ability to • business and user • organization skills, • expertise in interfaces,
moderate and arbitrate oriented, • team player, • knowledge of data
as necessary. • ability to be practical and • knowledge of systems warehouse concepts.
effective. from user viewpoint.

Infrastructure Specialist Business Analyst Data Modeler

• Specialist in hardware, • Analytical skills, • Expertise in relational

operating systems, • ability to interact with and dimensional
computing platforms, users, modeling with case tools,
• experience as operations • sufficient industry • experience as data
staff. experience as analyst. analyst.
 Roles and skills/experience levels required
in the Project Team
Data Warehouse Administrator Data Transformation Specialist Quality Assurance Analyst
• Expert in physical database design • Knowledge of data structures, • Knowledge of data quality
and implementation, • in-depth knowledge of source techniques,
• Experience as relational DBA, systems, • knowledge of source systems
• MDDBMS experience a plus. • experience as analyst. data,
• experience as analyst.
Testing
Coordinator
• Familiarity
with testing
methods
and
standards,
• use of
testing
tools,
End-User Applications
Specialist
Development Programmer Lead Trainer • knowledge
of some
• In-depth knowledge of • Programming and analysis • Training skills, data
source applications. skills, • experience in IT/User warehouse
• experience as programmer in training, information
selected language and • coordination and delivery
DBMS. organization skills. tools,
• experience
as
programme
r/analyst.
User Participation in DW Development
Project Planning
• Provide goals, objectives, expectations, business information during preliminary survey; grant active
top management support; initiate project as executive sponsor

Requirements Definition
• Actively participate in meetings for defining requirements; identify all source systems; define metrics
for measuring business success, and business dimensions for analysis; define information needed
from data warehouse.

Design
• Review dimensional data model, data extraction and transformation design; provide anticipated
usage for database sizing; review architectural design and metadata; participate in tool selection;
review information delivery design.

Construction
• Actively participate in user acceptance testing; test information delivery tools; validate data extraction
and transformation functions; confirm data quality; test usage of metadata; benchmark query
functions; test OLAP functions; participate in application documentation.

Deployment
• Verify audit trails and confirm initial data load; match deliverables against stated expectations;
arrange and participate in user training; provide final acceptance.

Maintenance
• Provide input for enhancements; test and accept enhancements.
Team Roles for Users
• responsible for supporting the project effort all the way (must be an
Project sponsor executive)

User department • help IT to coordinate meetings and review sessions and ensure active
participation by the user departments
liaison representatives
• provide guidance in the requirements of the users in specific subject areas
Subject area experts and clarify semantic meanings of business terms used in the enterprise

• review the data models prepared by IT; confirm the data elements and
Data review specialists data relationships

Information delivery • examine and test information delivery tools; assist in the tool selection
consultants
User support • act as the first-level, front-line support for the users in their respective
departments
technicians
Project Management
Considerations
 The effort of data warehouse project has
been successful if there is critical effective
project management.
 Project management issues are applied to
build success data warehouse projects :
 project management principles,
 warning signs,
 success factors,
 adopting a practical approach,.
Project Management Considerations:
Guiding Principles.
 Some of the guiding principles that pertain to
 data warehouse projects exclusively :
•Project Manager
•Sponsorship
•Team Roles
•New Paradigm
•User Requirements
•Data Quality
•Training
•Building for Growth
•Realistic
•Project Politics
Expectations
•Dimensional Data Modeling
•External Data
Project Management Considerations:
Adopt a Practical Approach.

 A practical approach is simply a common-

sense approach that has a nice blend of
practical wisdom and hard-core theory.
 While using a practical approach, you are
totally results-oriented, and you are not
driven by technology, you are motivated by
business requirements.
 WARNING SIGN INDICATING ACTION

Stop the capturing of

The Requirements unwanted information.
Suffering from “analysis
Definition phase is well Remove any problems
paralysis.”
past the target date. by meeting with users.
Set firm final target date.

If there is time and

Selected third party budget, get different
Need to write too many
tools running out of tools. Otherwise
in-house programs.
steam. increase programming
staff.

Very delicate issue.

Users not cooperating to Concerns over data Work with executive
provide details of data. ownership. sponsor to resolve the
issue.
WARNING SIGN INDICATING ACTION

ensure that the

selected query tool
is appropriate.
Users not
Users not trained
comfortable with the
adequately.
query tools
provide additional
training.

Revisit all data

transformation and
integration routines.

Continuing
Data transformation
problems with data Ensure that no data
and mapping not
brought over to the is missing.
complete.
staging area.

Include the user

representative in the
verification proces
Indications of Success
Queries and • rapid increase in the number of queries and reports
requested by the users directly from the data
reports warehouse

Query types • queries becoming more sophisticated

Active users • steady increase in the number of users

• users spending more and more time in the data

Usage warehouse looking for solutions

Turnaround • marked decrease in the times required for obtaining

strategic information
times