Module in Strategic Business Analysis

Lorenzo B. Cabili, CPA, LlB, MPA, MBA

Faculty, College of Business Administration

7
SOLUTION ASSESSMENT AND
VALIDATION

“When the territory and the map disagree, believe the

territory.”
— Swiss Army Manual

Overview
Welcome to Module 7!

The Solution and Validation Knowledge Area describes tasks performed to

ensure that solutions meet business needs and facilitate successful
solution implementation. These activities are not just limited to
application software, but address a variety of other areas such as business
processes, organizational change, outsourcing agreements, and any other
component of the solution.

Learning Objectives

At the end of this module, you should be able to:

Evaluate solutions
Discuss the importance of organization’s readiness on propose
solution
Illustrate ways in validating solutions
 ASSESS PROPOSED SOLUTION

Care and feeding of a solution has many facets. One important aspect is
the assessment of the organization’s readiness to effectively use the
solution once it is made available. Training, information sharing, and
documentation may be required to overcome any obstacles to changing
the way things are done. During the solution’s operational life, you will
want to measure its performance and quantify the business value it
delivers. Keep in mind that there are many ways to implement a solution
once it has been selected.

The first task in the Solution Assessment and Validation knowledge area
is assessing your proposed solution or solutions relative to the
stakeholder and solution requirements. When you assess one or more
solutions, be sure to evaluate the solution or solutions relative to the
approved stakeholder and solution requirements for your project. If the
requirements are not yet approved, you will not be able to make a final
decision on whether or not your solution meets the requirements,
addresses the business need, and provides value to the business.

Several key inputs are needed to adequately assess a proposed solution.

These key inputs are produced by a number of other business analysis
tasks; they include assumptions and constraints, the prioritized and
approved requirements, and the solution option or option to be assessed.
Let’s look at each of these task inputs in greater detail.

Assumptions and Constraints

Assumptions are factors Requirements (Prioritized

that you believe and but you have not yet
to be true,
confirmed. Assumptions may lead Approved)
you to favor certain solutions during
your solution assessment activities. Constraints are limits or boundary
Prioritized
conditions stakeholder and solution
that impact your solution.requirements allow limit
Constraints might you the
to evaluate
number
solutions relative to the most important requirements
of solution options that are available for you to assess. on your project.
Approved stakeholder and solution requirements should be used during
solution assessment so you can make a decision based on a baselined set
 Solution Option or Options

Each possible solution should be assessed using relevant information

about that solution. You and your organization need to decide exactly
what solution information should be evaluated. If you are comparing
multiple solutions, each solution should be presented with the same
information to facilitate a well – informed decision.

Several techniques are available for assessing a solution or a set of

multiple solution options. Let’s take a look at these two recommended
techniques in greater detail right now.

Acceptance and Evaluation

Acceptance
criteria define a minimal set
Criteria
of requirements that must be met in
order for a solution or a solution component to be
considered acceptable to its key stakeholders.
Evaluation criteria are a set of requirements used
to choose between multiple solutions to a
particular problem. They are typically built to allow
Vendor Assessment scoring
Vendor of the variousallow
assessments solutions
you under consideration.
to assess an external
vendor’s abilityIn order to
to provide allevaluate
or part of your
potential
solution. You look solutions,
at their thisability,
technical set offinancial
requirements
stability, staff skills, and reputation inbytheir
is prioritized and ranked orderwork
of
Remember that this assessment should
space. Externalbevendors
basedareonoften theinvolved
prioritized
in the and
design, construction, implementation,
approved stakeholder and solution requirements for your project. Several and
maintenance
key business analysis stakeholders activities
might be on our projects.
involved in the selection
process when solution options are being assessed.
 ASSESS ORGANIZATIONAL
READINESS

As a business analyst, you are an agent of change; you shepherd new

solutions from conception to completion. The trick is making sure that
folks are ready and willing to use the new solutions to get their job done.
Plus, the organization needs to be willing to maintain and support this new
way of doing business.

Assessing organizational readiness involves communicating the impacts that

a new solution will have on the business. This allows everyone to be
prepared for the upcoming changes versus being surprised by them.
Training is another aspect of organizational readiness. You might need to
train your end users to use the new solution before it is operational.

Several key inputs are needed to assess your organization’s readiness for
a new solution. These key inputs are produced by a number of other
business analysis tasks, and they include the designed solution, the
solution scope, and any concerns your stakeholders might have about the
solution. Let’s look at each of these task inputs in greater detail.

Enterprise Architecture

The enterprise architecture tells you the current state of the

organization. You can assess the organizational structure existing
business processes, existing systems, and information relative to your
new solution.

Solution (Designed)

The designed solution should be used when assessing your

organizational readiness. It is the technical design and definition of the
solution that will be built and deployed. Use this in tandem with the
solution scope.

Solution Scope

The solution scope defines the parts of the business or enterprise

architecture that are impacted by this solution. Use this as your starting
point for assessing organizational readiness for the new solution.

Stakeholder Concerns

Stakeholder concerns are the documented issues and problems that

you discovered while developing your project requirements. You need to
check this list, as it may contain solution implementation issues or
problems that need to be addressed when assessing organizational
readiness for that new solution.
Three key elements are found in a thorough organizational readiness
assessment. Together they make up the bulk of your assessment findings.
They are

 Cultural assessment
 Operational or technical assessment
 Stakeholder impact analysis

Let’s look at each of these three elements in greater detail.

 Cultural Assessment
The cultural part of your assessment focuses on the
willingness of your key stakeholders to change. You need to put
your marketing hat on and sell this solution as part of your business
analysis activities. This involves understanding each key
stakeholder’s willingness to change and getting them on board with
the changes that are coming with the new solution. Your
stakeholders need to understand and buy into the rationale for the
new solution and the benefits it provides.

 Operational or Technical Assessment

This part of your assessment focuses on the capabilities that
the new solution will provide and how the stakeholders will use
those capabilities. You might need to create new policies and
procedures governing use of the solution. Training might be needed
so folks know how to use the solution correctly. Systems support
and maintenance also need to be planned and put in place prior to
solution implementation.

 Stakeholder Impact Analysis

Your assessment must address how the changes from your
new solution will affect your stakeholders. There are a number of
things to consider.

There are several techniques that you may use when assessing
organizational readiness. Be sure to add business rules analysis to your
list of possibilities. Addressing the changes to business policies and
procedures as part of requirements allocation is important.

Force Field Analysis Force field analysis is associated with assessing

organizational change readiness relative to
deploying a new solution. This technique lets you
evaluate the pros and cons of each significant
change associated with the solution. Force field
analysis graphically depicts the positive and
negative forces that either support or oppose a
particular change.
 Data flow diagrams can be very helpful as you
identify activities that will change when the new
solution gets implemented. They also tell you which
Data Flow Diagram
stakeholders perform these activities so you know
who is impacted by the change.

Problem Tracking Problem tracking is the formal vehicle for

identifying and tracking stakeholder issues with the
new solution and its associated changes. This
technique ensures that the issues are addressed and
resolved.

MIND CHALLENGE # 1

How important it is to assess the readiness of organization on

the implementation of new solutions?

On your own point of view, what do you are steps in the journey
for readiness? You may discuss the possible actions you would
do in building readiness.
Define Transition Requirements
Transition requirements define what needs to be done to transition from
the existing solution to your new solution. The capabilities defined in the
transition requirements target making a smooth transition from the old
solution to the new solution. Transitioning to a new solution can be very
challenging. In many projects, it may be necessary to have an
operational system in place during the solution transition activities. In
such instances, be sure to develop comprehensive implementation plans
to ensure the job is done right and nothing is forgotten.

Three key elements must be considered when you are developing the
transition requirements for your project. It is very important that you
understand what is going on with the current, deployed solution. Key
sources to consider when defining your transition requirements include:

 Data
 Ongoing work
 Organizational change

Let’s look at each of these three elements in greater detail.

Data
Be sure to have a look at the data in and used by the existing solution. Some data
might need to be archived, while other data might need to be migrated to the
new solution. Migrated data might require conversion or reformatting in order to
be used by the new solution. The business rules governing data usage should also
be revisited to see if they require revision or replacement.

On-going Work

Stakeholders typically need to continue working with the deployed solution while
the new solution is being implemented. You will need to decide how to handle
this need to get work done and decide what happens when the new solution is
up and running.

Organizational Change

Often, you will find yourself creating a process for managing the changes your
new solution brings to the people who will use it. Job functions might change as a
result of the new solution. New information from the new solution might now be
available to stakeholders, and new skills might be required in order to use the
new solution.
 You can select from five general techniques to develop your transition
requirements. Let’s take a look at them now.

Business Rule Analysis

Business rules analysis allows you to define the business
policies and rules that govern business decisions and
operations in your organization. Business policies are
directives that support business goals. Additional business
rules might be required in order to transition from the
deployed (existing) to the designed (new) solution. You
may find yourself maintaining two sets of business rules
for a time if models
Organization you phase in athe
offer newofsolution
view at the same
your stakeholder
Organization Modeling time you and
groups phase stakeholders
out the old solution.
for use when
assessing organizational readiness and
developing transition requirements.
Data Flow Diagrams
Data flow diagrams help you to identify information
and activities

Process Modeling
Data Modeling
Process
Physical models help you
data models cantobeidentify
used towho does
map what and
information
when
needsthey
fromdothe
it in
oldboth the old
solution toand
the the
newnew solutions.
solution. This will
help you decide what data needs to be archived,
transferred, or revised in order to get the new solution up
Transition requirementsand running.
are elicited, analyzed, managed, and
communicated just like the other requirements for your project. However,
transition requirements are not just any old requirement. They are very
specific to implementing the solution and cannot be defined until after
that solution has been designed. Transition requirements are
no longer needed after the new solution is operational, because they
focus only on what is required to get that solution up and running.

VALIDATE SOLUTION

Solution validation makes certain that your new solution addresses the
business and stakeholder needs that triggered your project in the first
place. To do this, compare your solution and its capabilities against the
project’s business, stakeholder, and solution requirements that were
created earlier in the project life cycle. Any problems or discrepancies you
find during solution validation are recorded, prioritized, and resolved.

When validating your new solution relative to its requirements, you’ll need
to:
 Investigate defective solution outputs.
 Assess any defects and issues.

Let’s look at each of these two activities in greater detail.

Investigate Defective Solution Outputs

Defective solution outputs occur when the outputs are below the
previously defined acceptable level of quality for that output. Defective
outputs require investigation and resolution as part of your solution
validation activities.

Assess Defects and Issues

Any defects that are identified during solution validation need to be
reviewed and assessed. Some defects require immediate resolution, while
others may be mitigated using a workaround or accepted for the time
being. There are several ways to mitigate the impact of a defect using a
workaround, such as:

 Adding quality control checks

 Introducing new manual processes
 Removing support for exceptions and errors

There are three general techniques for you to select from when validating
a solution and addressing any defects or issues that you might find. Let’s
take a look at them now.

It is essential that you define the set of requirements

Acceptance and
Evaluation Criteria that will be used to validate the solution. This set of
Definition requirements must have defined, measurable,
acceptance criteria for you to use in your validation
efforts.
 Problem Tracking

Problem tracking is your formal vehicle for identifying

and tracking identified defects. This technique ensures
that the defects found during solution validation are
addressed and resolved.

Root Cause Analysis Root cause analysis allows you to determine the underlying
reason for a defect. This allows you to correct the actual
cause of the defect versus correcting a symptom of that
defect and not the defect itself.

Once you have selected and applied one or more of these techniques as
part of your solution validation efforts, you are ready to continue with
your other assessment and validation tasks.

Defects, Mitigating Actions, and the Solution Validation

Assessment

Your solution validation efforts produce three distinct and related outputs:

 Identified defects
 Mitigating actions
 Solution validation

assessment Let’s review each

output in greater detail.

Identified Defects
Identified defects are known problems that have been found in a
constructed solution. They should be logged, reviewed, and resolved.

Mitigating Actions
Mitigating actions are steps or process that you will take to reduce
or eliminate the impacts an identified defect has on a stakeholder or
stakeholder group. Think of mitigating actions as workarounds, allowing
you to sidestep the impacts of the defect instead of directly addressing
and correcting its cause
.
Solution Validation Assessment

The solution validation assessment documents whether the constructed

solution meets the business need and the project requirements at an
acceptable level of quality.

You are responsible for performing solution validation on your project.

There are a number of stakeholders who should assist you in doing this
work. Testers play a primary role in solution verification, where they make
certain that the solution behaves as the solution requirements say that it
will. Implementation SMEs will be key players in this task, working with
you to investigate and correct defects. The project manager must be
involved with validating the solution so they can coordinate the resources
required to perform the work. Several other business analysis
stakeholders may be involved with your solution validation efforts,
including the:

 Domain SME
 End user
 Operational support
 Project manager
 Regulator
 Sponsor

EVALUATE SOLUTION PERFORMANCE

Evaluating solution performance begins once your constructed solution is

deployed and in operational use. You will find yourself investigating how
the solution is being used and assessing the positive and negative impacts
it has on the organization and its stakeholders. Some folks evaluate
solution performance in a post - implementation assessment performed
shortly after their project is complete.

Three elements are involved with evaluating solution performance. The

solution being evaluated must be live and in operation. If it isn’t being
used, there won ’ t be much for you to evaluate. The detailed elements of
this task include:

 Understanding the value delivered by the solution

 Validating the solution metrics
 Considering solution replacement or

elimination Let’s look at each of these three

elements in greater detail.

Understanding the Value Delivered by the Solution

You need to collect the metrics that describe solution performance.
Some software applications may provide these metrics to you
automatically. Other types of solutions will require manual collection of
quantitative and qualitative solution metrics for evaluation.
Validating the Solution Metrics
If you have a solution that is over - or under – performing and not
meeting one or more of the business goals and objectives, it may be
that your solution metrics associated with those goals and objectives are fl
awed. The solution metrics should also be validated and defined more
appropriately.

Considering Solution Replacement or Elimination

At some future time, it will become necessary for you to evaluate
your solution as a candidate for replacement or elimination. This can
happen for a number of reasons, including outdated technology or new
business goals and objectives.
There are four issues that might influence your decision to
replace or eliminate a deployed solution.

Ongoing Cost versus Initial Investment

You might discover that the costs to maintain a solution over time
have increased. This needs to be balanced by looking at new solutions
that may have a high initial investment cost but lower maintenance costs
over time. Doing a cost - benefit analysis is always a good idea when
evaluating solutions for
elimination or replacement.

Opportunity Cost
Opportunity costs are the potential value you are giving up by
continuing with an existing solution rather than doing something
different. Opportunity cost equals the benefits from doing something
different. Be sure that you are not giving up a significant future
opportunity by staying with your old solution.

Necessity
Most solutions and solution components have a limited lifespan.
After a certain point in time, it may be impossible to maintain them and
they will need to be replaced or eliminated. Solutions frequently need to
be upgraded or replaced when vendors stop supporting older versions of
their products.

Sunk Cost
Sunk costs are the money and effort you have already spent for
the current solution. Sometimes it is difficult to consider replacing or
eliminating a solution when you look at what you have invested in that
solution. You need to make sure that you look at future investment and
benefits as well.

Techniques to consider

The following techniques are recommended when you are evaluating

solution performance. They are summarized for you here.

Decision Analysis Decision analysis allows you to examine and model the
consequences of different decisions when evaluating the financial impacts
a solution is having on your business. You must make sure that you look
at all solution costs and benefits that are relevant to the organization’s
values, goals, and objectives.
Focus Groups Focus groups provide a facilitated group approach
to understanding the value the solution provides to its stakeholders. Focus
groups can also be used to define or validate the solution performance
metrics that are being used.

Observation Observation allows you to observe the stakeholders using

the solution. This technique may reveal problems or issues that have not
been noticed or reported.

Survey/Questionnaire Surveys and questionnaires provide you with a

vehicle to gather quantitative and qualitative information from large
numbers of stakeholders about how a solution is performing.

MIND CHALLENGE #2

In a recent consulting assignment, Ginger discovered first hand that solution performance
evaluations can yield some surprising results. She was performing a post - implementation
review of a large software systems implementation at a major telecommunications
company.

The new system automated the trouble - ticketing and problem - resolution capabilities
between the telecommunications company and its business partners. Whenever a circuit
problem was reported, the system would initiate and carry on an electronic conversation
between the two companies. The conversation stepped through the problem – resolution
process from start to finish with the computer systems on each end performing the lion’s
share of the work.

Rather than talk with one another and do a lot of manual data entry and status updates, the
service representatives were able to simply check the status of the trouble tickets and make
sure the repairs that needed to be done were actually scheduled and completed. The new
system was viewed by senior management as a system that would greatly enhance
workplace productivity by minimizing the manual transactions and telephone conversations
that took up so much of the service representative’s time.

As part of her solution performance evaluation work, Ginger visited a major service center
to see how well the new system was being used. She expected to see a very quiet and effi-
cient workspace with a lot of activity going on behind the computer screens. What Ginger
expected and what she encountered on her visit were two very different things.

Ginger walked around the service center to see the end users in action. As predicted, once
the trouble ticket was opened in the new system, the computer took over the conversation
with the company responsible for addressing the problem. However, instead of allowing the
electronic conversation to go on unaided, the service reps were on the phones with the
counterparts at the other company. As the computer systems exchanged information and
updated problem status, they were discussing what they saw on the screens and making
sure that all was going as planned. This behavior on the part of the end users was most
unexpected.
Question:

Based on the case problem given above, if you are Ginger, what would be
your post-implementation assessment on the new software system
implemented by the company? Was there any failed or missed issues that
were not taken into account such as job satisfaction and relationship
building? Discuss briefly your point.

Warmest
congratulation
s for a job
well done!
The tasks in Solution Assessment and Validation knowledge area guide
you in effectively assessing, selecting, and validating the solution that
will be implemented in the organization to meet the business
requirements. You will also develop the transition requirements and
perform release planning for your project. Transition requirements define
the solution capabilities required to transition from the existing solution to
the new solution. One interesting fact about these requirements is that
they are no longer needed once the transition to the new solution is
complete.

Effective problem - solving skills are an underlying competency enabling

you to do this solution - focused work. You will be asked to analyze
problems with the designed and constructed solution and recommend
ways to address any defects and issues that you and your stakeholders
find. Applying your structured problem - solving skills results in
implementing a solution that meets or exceeds its quality and acceptance
criteria.