1.

Managing Business Analytics Projects

- Stakeholders
● Business users v DS/DA
- Business people focus on defining the scope of the problem
- DS/DA focus on how to leverage data to solve the problem
- E.g. Business users should not tell DS/DA they want a predictive model; DS/DA should decide if that’s the best
way to do that
● Aligning Stakeholders
- Nirmal: Tl;dr → Quick alignment strat is just to be able to see from other people’s perspective; Get to the root of their
concern, see from their POV

- Types of Analytics
● Descriptive
- Analytics/metrics that describe what happened
- E.g. GMV per month
- Nirmal: Descriptive is about organizing data so I can isolate/identify some things. Use an example of a library organizing
books → Each book belongs to their respective section. Way of organizing the books is ‘classifying’ the books; You’re
basically assigning a category to each book
● Diagnostic
- Addresses why a certain thing happened
- E.g. 10k GMV per month because item X drove the sales
- Nirmal: Basically the idea that you wanna find patterns in the data. Example, COVID’s rate of spread is
exponential; that's a pattern. The business problem here is you wanna determine how long you want people to
quarantine. A Lot of diagnostic analytics is about guessing/finding the pattern
● Predictive
- Addresses what will happen
- E.g. Based on GMV of past 12 months, we predict Y GMV for next 6 months
● Prescriptive
- Addresses how we can make a pattern/trend happen
- E.g. We can try to push more sales of item X if we want to achieve target of Y GMV next month
- Nirmal: Prescriptive analytics is concerned with decisions. Analogize to a doctor gives you a prescription - it's
about making a decision to address a problem
● Example → Analytics in Airlines

- Data Types x Scope Complexities

● Within a project, if different data types exists, it can increase complexity of the project
- E.g. Big Data Analytics, Streaming Analytics, Unstructured Data Analytics
- Analytics Process: CRISP-DM [CRoss Industry Standard Process for Data Mining]

1. Business Understanding
- Identify/address the following questions:
(i) Who is the end user?
(ii) What is the end benefit/purpose?
(iii) How will the end solution be deployed?
(iv) What are the hypotheses of the business users?
2. Data Understanding
(i) What data do you need?
(ii) What data is available?
(iii) How easy is it to access the data?
(iv) Will the data be available during deployment?
(v) What are the privacy norms to comply with?

Nirmal: General idea is that you need to be v clear for (1) and (2) before you can move on to step (3) onwards.

3. Data Wrangling/Preparation
- This is the most time consuming step
(i) Data Janitoring
(ii) Data Quality
(iii) Data Transformation
4. Data Visualization
(i) Take a look at the overall trends
(ii) Does the business agree w the trends/insights
(iii) Does visualization show any new insights?
5. Modeling
- General idea is that a model is an approximation of reality
- You wanna use easy measurements to approximate/predict difficult concepts
6. Model Evaluation
- Essentially the step to determine if your model is performing well
- Has a business aspect to it → Metrics might indicate your model is accurate, but it might not necessarily be helpful to have
a v accurate model in certain business context
7. Decision Engineering
- Nirmal: Generally in a business, you must try to incorporate costs into it. You make the decision that incurs less
costs
 ● Example → Model on the right side
incurs less cost, tho less accurate. It is
likely that business will pick the less
accurate model due to less incurred
costs
- Example: Debt collection decision
making can be implemented via a
policy
- Nirmal: Idea is that there is a limit
number of work hours debt
collection executives have in a
given day/month, you wanna
prioritize their collection strategy to
maximize the ROI (Assuming you
wanna go for the best ROI)

8. Deployment
- DA/DS builds the model/analytics solution → IT Team deploys the solution into existing workflow/app → Business Team
makes use of the solution within the workflow
- Summary
→ See slide 48 onwards
→ Idea is that you wanna define business problem and understand data before you can move on
2. Analytics Projects Management Methodology/Lifecycle
- Project Management Triangle

● Balancing these things is your job as project manager

● Most of the time you try to compromise/manage the scope

- Definitions
● Project: Set of related tasks that are coordinated to achieve a specific objective in a time limit
● Project Management: Application of knowledge, skills, tools and techniques to project activities to meet project
requirements

- Project, Program, Portfolio Management

- Project Lifecycle
● It is a collection of project phases; Can be categorized into the following:
(1) Predictive/Waterfall
Definition
- Project is fully plan driven
- Scope, time and cost are determined as early as possible
- Each phase is executed in sequence and only once
- Changes to scope are carefully managed

Use Cases:
- When product to be delivered is well understood
- There is substantial industry practice
- Product needs to be developed in full to have value
- Example: Hardware development, infrastructure set-up

(2) Incremental
Definition
 - Some phases of the project are repeated to deliver an increment/release
- Increments may be done sequentially/overlapping
- Each delivered release successfully adds functionality/capability
- Each delivered release is usable (by users), capable of doing useful work and provides a subset of required total
capabilities

Use Cases:
- Phased IT migration projects

(3) Iterative
Definition
- Product is developed through a series of repeated cycles/iteration
- Each iteration enhances/adds functionality to the product
- High-level vision/scope may be developed initially for planned iterations to ensure software adaptability
- Scott: This is key → Each new iteration incorporates feedback & experience of earlier releases; You learn what works and
doesn’t work, then you incorporate this into the next design iteration. I.e. Test and learn

Use Cases
- Business to business software, internal software

(4) Adaptive/Agile
Definition
- Each iteration is short and timeboxed (usually 2-4 weeks) with fixed time and cost
- Respond well to high levels of change and ongoing stakeholder involvement
- Overall scope decomposed into prioritized requirements in the work items

Use Case
- Most mobile/web apps

- Choosing the appropriate Lifecycle

- Success Factors in Business Analytics Projects
- Class Workshop Discussion Point
** Questions **
1. Perform Business Understanding
- What are main problems you can solve w Analytics
(i) What are the models you can build for each problem
(ii) What is the mathematical construct
(iii) What is the metrics
(iv) What is the implementation process/construct
(v) What are the data dependencies required to solve the problem

Dissect it into Offline v Online

* Offline *
> Movie Recommendation System
- Recommend movies
- Existing customer/new customer
- Existing customer -> Model using purchase history
- New customer -> Model using store history
- Think about the customer touchpoint -> Offline will likely be at cashier/POS terminal
- Think about how many recommendations to give to customer (2/3/4) -> A/B testing to find out optimal amount
- For recommended products, think if you wanna give discounts -> Also A/B testing to find out optimal discount

> Inventory Management System

- Can sorta use the movie recommender system as an inventory management system
- Based on customer profiles and what they purchase, you project how much inventory you need2

--> Nirmal: Fundamental difference between online/offline is inventory. You don't store obscure titles/products offline due
to inventory space limitations. Tweak to recsys offline/online is that offline needs to take into account of available
inventory

** Own Brainstorm Points **

> Customer segmentation based on demographics/psychographics
- Maybe can do clustering here
- Simple sales dashboard breakdown by demographic group works well too
- Probably dashboarding/visualization engine/feature needs to be built into CAS
- Data Required: Sales data, customer demographics, etc...

> Inventory management engine

- Dashboard/Rule based alert system works well too
- To incorporate the find closest store w product is quite possibly a rule-based approach as well
- Dashboard for logistics side, some automated UI/interface for customers
- Data Required: Location data of stores, inventory management system needs to be built (Stock taking needs to be
digitized, most likely incorporated into the sales checkout system as well)

> Recommender system to recommend products

- Can also do some clustering/apriori algorithm/item/user similarity scores
- Depending on who is using it - Customer recommender system can be on Website UI, marketing team probably some
built-in UI on CAS. Dashboard is possible too
- Data Required: Purchase/sales records, customer demographics, product reviews etc...

> Campaign analytics [Assuming it is to assess the effectiveness of the marketing campaign]
3. Project Inception

Revise Slides 1-18

- Disciplined Agile for Business Analytics Projects

● Inception helps with proper resourcing & uncovering critical path
● Critical path: Step/process of project that is the bottleneck

- Inception Phase of a BA Project

● Scott: Generally, it helps to have the ‘end in mind’. Think about the deliverables your project should have
● In general, a BA project have 2 forms of deliverables:
(1) Reports & Dashboard → Purpose: Help humans make better decisions
(2) Business Analytics embedded into an existing product → Purpose: Help machines make better decisions

● DA Inception Phase [Usually takes around 4 weeks]

Objectives:
(1) Clarify business problem
(2) Identify viable technical solution
(3) Plan the approach
(4) Set up the work environment and team
(5) Gain stakeholder concurrence that it makes
sense to proceed w chosen strategy
- Inception Phase Process Goals
1. Form Initial Team
- Some questions to consider:

Primary Disciplined Agile Roles

(i) Delivery/Project Lead (SCRUM Master)
- Keeps team focused on achievement of goals
- Removes impediments
(ii) Product Owner
- Owns the product vision, scope & priorities of the
solution
(iii) Architecture Owner (Tech Lead)
- Owns architecture of decision & tech priorities
- Helps mitigate technical risks
(iv) Team Members
- Cross functional team members that deliver the
solution
- E.g. Developers, Data Scientist/Engineer, etc…
(v) Stakeholders
- Customer, project sponsor, DevOps, etc…
Scott: Project sponsor is about the why → Why do we have to do this. Product owner is about the what → What do we have to do to
complete this project
Secondary Disciplined Agile Roles → Usually happens when the project is at a v large scale
(i) Specialist
- Usually a business analyst, program manager, etc…
- Helps augment your primary team member’s skillset
(ii) Domain Expert
- Someone with deep knowledge of the domain
- E.g. Legal/marketing expert
(iii) Technical Expert
- Someone with deep technical knowledge whose help is needed for a short period
- E.g. Security engineer, UX professional
(iv) Independent Testers
- Test/quality professional outside the team who validates the quality of the work
(v) Integrator
- Someone responsible for operation of overall team

Data Science Hierarchy of Needs vs Roles

(1) Collect → Data Engineer
(2) Move/Store → Data Engineer
(3) Explore/Transform → DS/DA
(4) Aggregate/Label → DS/DA
(5) AI, Deep Learning → ML Engineer
Stakeholders as part of Project Team
- Stakeholders: Individuals/Groups/Organizations who may affect/be affected by a decision/activity/outcome of a project
They may be actively involved in project or have interests that may be impacted by completion of project
- Need to manage stakeholders → Can do so via these frameworks

- Strategies to Manage to the Stakeholder

- Use a Stakeholder Register to manage Stakeholders

2. Develop Common Vision
Create a Project Charter
- Define the Business Problem
- What is the proposed solution/high-level scope
- What is the business value, measure of success, project team goals

Example: Project Charter with People’s Association

3. Align w Enterprise Direction
- Two aspects of alignment:
(1) Alignment with teams/groups within the organization
(2) Alignment with business & technical direction

Alignment Consideration for Business Analytics

4. Explore Initial Scope

- Scott: At the start, you don't need to plan so detailed. There’s diminishing returns to it if you plan too much at the
start. Best to to detailed planning in a JIT manner

What needs to go into Initial Scope for Business Analytics Projects

 Scott:
- There's a Development &
Deployment & Maintenance
phase because it's often not a
pure analytics project. It's
usually analytics product built
onto an existing app/product,
some development work
needs to be done to integrate
analytics product into existing
app/product
- Sandbox is used to test,
reduce risk of your analytics
product not working as intended. Test in sandbox first before deployment

Analytics Sandbox

5. Develop Initial Release Plan

Scott: Difference between release plan and roadmap
(1) Release Plan
- It's more granular in terms of time
- Specifies the project deliverables in detail
(2) Roadmap
- Roadmap is more about outcomes you want to drive
- Lay out the key milestones/deliverables you have

Release Plan Considerations

How to put together a Release Plan

Example of an Initial Release Plan

Time and Resource Allocation Plan

6. Secure Funding

- Staged Gate: Minimum funding required up to the end of the Sandbox stage
- Scott: Idea of doing this is to appease stakeholders. You ask for funding up till point X, then rest of the funding is
conditional upon achieving some pre-defined milestones

7. Form Work Environment

8. Identify Initial Technical Strategy

- Create a Business Analytics Architecture Diagram
 9. Identify Risks - To manage key risks

Example

- Completing Project Inception

● Inception ends when:
(1) All stakeholders agree it makes sense to proceed based on scope, schedule, budget, constraints and
other criterias related to business case
(2) Risks has been identified and seem tolerable
(3) There is agreement on using adaptive & agile process for building the solution
(4) Team and environment has been set-up/in process of being set-up
(5) Process and governance strategies have been agreed upon by team and stakeholders
- Summary on Best Practices
4. Managing Business Analytics Projects - Construction Phase
- Agile Methodology Overview
● Focus is on cost and efficiency
● Agile mindset & practices implies delivery cadence
- Meaning periodic release → v1, v2, v3, etc…
● Ability to work with various teams in the organization

- Typical Analytics Project Phases

● Richard: Maintenance of model is important as the performance will decay over time as environment
factors/features change over time

- Types of Business Analytics Projects

● Descriptive Analytics
- Data Extraction, preparation, visualization
● Diagnostic Analytics
- Add causal analysis
● Predictive Analytics
- Add predictive modeling
● Prescriptive Analytics
- Add decision engineering + productionalizing the analytics

- Business Analytics Framework

● CRISP-DM
 ● Richard’s Framework

- Project Delivery Mantra

● Idea is to deliver value early and often
● Value: Defined as what stakeholder/enterprise is willing to pay for a product/service to (1) produce a benefit, (2)
improve a service
● Richard: Delivery value early and often because as time goes on, value provided will drop
- Release Plan

- Objectives of Construction Phase

● Produce a consumable and deployable release
(1) Needs Exploration
- Understand pain points & concerns
- Break down the initial scope according to stakeholder’s priorities and concerns into smaller more manageable components
(i) Ensure delivery w specified features & functions
(ii) Improve accuracy of scope management, cost, time and resource estimation
(iii) Have a baseline for perform measurement & control
(iv) Facilitate clear responsibility of assignments
→ Outcome will be a ‘Product Backlog’/’Work Item Stack’

Product Backlog / Work Item Stack

- Richard: Complete the highest priority one. For higher prioritized tasks, find out more details about it

User Story Considerations

- General Idea: Describes the feature your user needs. A unit of task that your team commits to in a
release/iteration
- Use SMART criteria to develop tasks

Technical Story
- Targets the non-functional features/business-related capabilities to support
- Must be aligned w your enterprise architecture
- Typically involves technical analysis, design, solution architecting work, prototyping

User Story/Technical Story Template

(1) Who is the user we are building for?
(2) What is the goal/objective of what we are building?
(3) What value does it bring to the user?

(2) Solution Exploration

- General Idea: Understand the problem you are trying to solve so that you can frame the most appropriate
solution

- Richard: Work with your stakeholders to understand their requirements, pain points, objectives etc…
- Available solutions may vary: (1) Can be Best of Breed [Best in class for a specific function, need to integrate them
together], (2) All-in-One [One solution fits all needs], (3) Buy/Build, (4) Insource/Outsource, (5) On-premise/In-Cloud
(3) Development Strategy
- COT/Bespoke
- In-house/Outsource/Hybrid
- ATDD: Acceptance Test Driven Development → Create your acceptance test, then develop
- FDD: Feature Driven Development → Understand your feature before development
- TDD: Test Driven Development
- BDD - Behavior Driven Development

(4) Rolling Wave Planning

- Planning must be done at all levels

- Organize your project into a set of releases & interaction
- This allows for better estimate of cost/value and risk identification
- Plan must include a cadence/regular iteration of releases

Example: Rolling Wave Planning

(5) Review & Retrospective

(i) Review your product
- Work with your stakeholders to review your product
- Demo your product to ensure it works & meet stakeholders expectations
- Close off all pending project documentation
- Complete & finalize any reports (if required)
(ii) Retrospective
- Assess the successes & shortcomings of work processes
- Learn from experience gained
- Re-deploy resources
- Close off & archive documentation
(iii) Celebrate your Launch
- Get formal recognition: E.g. Sign-off/commissioning letter

(6) Assurance
- Improve the quality of: (i) System, (ii) Documentation
- Work on risk management
- Leverage enterprise professionals to validate your product
- Understand & agree on your definition of ‘Done’

Definition of Done
Testing

Managing Business Analytics Project Risk

- Richard: In BA projects, data is a big risk. Must consider what types of data you have, then for each type of data,
there’s different ways of handling them
- Possible List of Risks in BA project: Link

- Richard: Severity = Impact x Probability

- Richard: Idea is that you wanna monitor the risk over a period of time. Risk can change over time in severity and
probability. Then you plot a chart to assess the ‘Risk Burn Down’
 (7) Managing Assets
- Assets: Models, data models, code, etc…
- Management in terms of: (1) Documentation, (2) Storage, (3) Version control/Traceability
- Also, how to deploy our solution

Deployment Strategy
- Plan the cadence for deployment (Big Bang approach / Pilot Deployment)
- Setup helpdesk
- Work with DevOps to deploy solution
- Prepare fit-for-us documentation and conduct fit-for-use training

● Address changing stakeholder needs

(1) Requirements Elicitation
(i) Traditional → Use BRUF [Big Requirements Up Front]
- Generally high wastage because we are not good at defining things upfront. There’s also a tendency to add more things than
necessary
- Requirements change process tend to de-motivate people too
(ii) Agile Methodology → Use Wireframes/Personals/User Stories/Storyboards

(2) Manage Work Item Stack

 (3) Work with Stakeholders to Prioritize - Prioritization Techniques
- Assess business value & risk
- Assess operational emergency dependency
- Prioritize items that have an earlier due date
- Prioritize tasks that are simplest to complete
- MH SH CH WH → Must have, should have, could have, won't’ have

● Prove your architecture early

(1) Validate & Verify your Architecture
- Work w other enterprise professions to leverage/re-use existing infra as much as possible
- Allow your solution to evolve the infra to reflect the organization’s strategy
- Data preparation - includes acquisition, ingestion, manipulation, storage
- Avoid technical debt
- Review technical risks

(2) Ensure Data Governance & Security Compliance

- How important is data governance & security
- What is our weakest point
- What should we look out for?
- How do you ensure compliance?

● Tracking and control

(1) Burn Down Chart
- Richard: Y-axis: Story points/action items; X-axis: Sprints → 8 action items per sprint
Tracking Release Items
 (2) Velocity Factor
- It is used to plan the release date
- Allows you to add some buffers for uncertainty and risk
- Measures how fast the team is delivering functionality in an iteration
- Velocity is calculated at the end of the iteration by totalling the points for all fully completed User Stories

Example

Determining Release Date

● Transition
(1) Ensure the solution is consumable
 (2) Deploy the solution

(3) Review & Retrospective -

- See above

● Roll Out
(1) Review Roll Out Plan
(2) Execute Roll Out Plan
(3) Review & Retrospective

● Maintenance
(1) Model Health Check / Improvements
(2) UI / Reports / Data Visualizations Improvements
(3) Bug Fixes / Refactoring