6/12/2012

Using Data
for
Improvement
in Healthcare:
The Essential Toolkit

Sandra K. Murray
[email protected]

Objectives
• Participants will be able to:
- Identify fundamental differences between data when used for
improvement, accountabilityy and research
- Appreciate the value of viewing data graphically and over
time
- Learn when to use and how to interpret data on tools
fundamental to improvement:
• Run chart to identify statistically significant signals of change
• Shewhart Chart (Introduction only)
• Pareto chart
• Histogram (Frequency Plot)
• Scatter Plot
- Select the appropriate tool for the question being asked

Copyright © 2012

1
 6/12/2012

References
Books:
1. The Health Care Data Guide: Learning from Data for Improvement. Lloyd Provost and
Sandra Murray, Jossey-Bass, 2011.
2. Total Quality Tools For Health Care. Productivity-Quality Systems, Inc. Miamisburg
Ohio. ISBN: 1-882683-04-8 Tel. 1-800-777-2255.
p
3. The Improvement Guide. Gerald J. Langley,
g y Kevin M. Nolan, Thomas W. Nolan,
Clifford L. Norman, Lloyd P. Provost, Jossey-Bass, 2009.
Video:
1. Making Sense Out of Control Charts. NAHQ. 1-800-966-9392
Software Used to Produce Charts:
1. ChartRunner. PQ Systems. 1-800-777-3020.
2. QI Charts. API, 1-512-708-0131
3. Minitab,1-814-238-3280
Articles:
1 The
1. Th run chart:
h a simple
i l analytical
l i l tooll for
f learning
l i from
f variation
i i in i healthcare
h lh
processes. Rocco J Perla, Lloyd P Provost and Sandra K Murray. BMJ Qual Saf 2011
20: 46-51.
I

Copyright © 2012

Purpose of Measurement

• Measurement for Improvement

p

• Measurement for Accountability

• Measurement for Research

The Three Faces of Performance Measurement: Improvement, Accountability and
Research. Journal on Quality Improvement, Volume 23, Number 3, March, 1997.

Copyright © 2012

2
 6/12/2012

Data for Improvement, Accountability and Research in Health Care

Aspect Improvement Accountability or Research
Judgment
Aim: Improvement of care Comparison for judgment, New generalizable knowledge
processes, systems and choice, reassurance, spur
outcomes for change
Methods: Test observable No test, evaluate current Test blinded
performance
Bias: Accept consistent bias Measure and adjust to Design to eliminate bias
reduce bias
Sample Size: “Just enough” data, small Obtain 100% of available, “Just in case” data
sequential samples relevant data

Flexibility of Hypothesis flexible, No hypothesis Fixed hypothesis

Hypothesis: changes as learning
takes place
Testing Strategy: Sequential tests No tests One large test

Determining if a Run charts or Shewhart No focus on change Hypothesis, statistical tests (t-
Change is an control charts Shewhart charts for test, F-test, chi square, p-
Improvement: monitoring values)
Confidentiality of Data used only by those Data available for public Research subjects’ identities
the Data: involved with consumption protected
improvement
Frequency of Use: Daily, weekly, monthly Quarterly, annually At end of project
Copyright
Source: The Health Care Data Guide: Learning from Data for Improvement. Developed from Solberg, Leif I., Mosser, © 2012
Gordon and McDonald, Susan. “The
Three Faces of Performance Measurement: Improvement, Accountability and Research.” Journal on Quality Improvement. March 1997, Vol.23, No. 3.

Graphical Display of Data

• Effective visual presentations of data, instead of
tabular displays, provide the most opportunity
from variation

Copyright © 2012

3
 6/12/2012

The Health Care Data Guide: Learning from Data for Improvement. Lloyd Provost and Sandra Murray,
Copyright Jossey-Bass, 2011.
© 2012

Graphical Display of Data

• Effective visual presentations of data, instead of
tabular displays, provide the most opportunity
from variation
• Viewing variation over time enhances learning

Copyright © 2012

4
 6/12/2012

Unit 1

Cycle Time
Results for Units
1, 2 and 3 Unit 2

Unit 3

The run chart: a simple analytical tool for learning from variation in healthcare processes. Copyright © 2012
Rocco J Perla, Lloyd P Provost and Sandra K Murray. BMJ Qual Saf 2011 20: 46-51.

What’s the Question You’d Ask Here?

Run Chart of Measure

100
95 Goal = 90

90
Median = 84
85
%

80
75
70
65
60
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

5
 6/12/2012

Model for Improvement

Pareto Charts What are we trying to
accomplish?
Run or Shewhart Charts How will we know that a
change is an improvement?
Run and Shewhart
What change can we make that
Charts, Pareto charts, will result in improvement?
Frequency Plots, Scatter
Plots

Act Plan

Study Do
Run or Shewhart Charts
AND Qualitative Data Copyright © 2012
The Health Care Data Guide: Learning from Data for Improvement. Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

Repeated Use of the PDSA Cycle

Model for Improvement
What are we trying to Reduce Per-op harm by 30%
accomplish?
How will we know that a •% Pts with Peri-op harm
change is an improvement?
What change can we make that
•Peri-op Harm Rate Changes That
p
will result in improvement? •Unplanned returns OR
Result in
--DVT Prophylaxis
--Beta Blocker Prophy
Improvement
--SSI interventions

Implementation
of Change
Wide-Scale
Use clippers
Instead of A P Tests of
Shaving site S D Change
Follow-
Hunches up Tests
Theories Very Small
Ideas Scale Test Copyright © 2012
The Health Care Data Guide: Learning from Data for Improvement. Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

6
 6/12/2012

Tools for Understanding Variation

• Run Chart: Study variation in data over time; understand
the impact of changes, detect signals of improvement.
• Shewhart Chart: Distinguish between special and
common causes of variation. Is process stable, predictable?
• Pareto Chart: Where should we focus? Focus
improvement on area with greatest potential impact.
• Frequency Plot: Understand distribution of data (e,g,
central location, spread, shape, and patterns).
• Scatter Plot: Analyze potential relationship between two
variables.

Copyright © 2012

Tools to Learn from Variation in Data

Frequency Plot Pareto Chart Scatter Plot

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

7
 6/12/2012

Run Chart
• Graphical display of data plotted in some type of order.
Also has been called a time series or a trend chart.

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

Fundamental Uses of Run Charts

• How much variation do we have?
- Display data to make process performance
visible
• Have our changes yielded improvement?
- Determine whether a change resulted in evidence
of improvement
• Are the gains we made slipping away?
- Determine whether we are holding the gain made
by our improvement

Copyright © 2012

8
 6/12/2012

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

How Do We Tell a Change is an Improvement?

• Run charts speak for

themselves or
themselves…or..

• Analyze with probability-based rules

Copyright © 2012

9
 6/12/2012

6
Unplanned Returns to OR: Pilot Population
(N~200/Mo.)
Pre-Procedural Briefings
5 Prophylactic ABX Timing

Razors to Clippers
4 Bleeding risk assessment, DVT Proph

Beta Blocker use, Normothermia

3
%

0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
Months

Copyright © 2012

The Health Care Data Guide: Learning from Data for Improvement.
Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

10
 6/12/2012

How Do We Tell a Change is an Improvement?

• Run chart may speak for itself

• If run chart does not speak for itself we can
analyze it further using probability-based rules
- Can detect signal of change ( a non-random
pattern in the data)

Copyright © 2012

Looking at Timeliness
Month % Timely Month % Timely
1‐ 2007 32 1‐2008 23
2 23 2 32
3 32 3 36
4 38 4 29
5 35 5 38
6 35 6 42
7 40 7 39
8 21 8 36
9 38 9 50
10 26 10 48
11 22 11 39
12 27 12 44

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

11
 6/12/2012

MEDIAN

MEDIAN:
I a series
In e ie off numbers,
be the median
edi is i
physically the middle number .
It has the same number of points equal to it or
above it as it has equal to it or below it.
MEAN: The average.
average

Copyright © 2012

Why Median Rather Than Mean?

• 8,10,11,14,16,18,20 Mean= 13.8
Median=14

• 8,10,11,14,16,18,95 Mean= 24.5

Median=14

• 1,10,11,14,16,18,20 Mean= 12.8

Median=14

Mean = arithmetic average of data

Median = middle value of ordered data

Copyright © 2012

12
 6/12/2012

50
48
Finding the Median: Reordering the Data
44
42
40
39
• To find the median reorder the
39 numbers from high to low and find
38
38
the number physically in the middle.
38 If you have
h two
t numbersb left
l ft iin th
the
36
middle, add them together and
36
35 divide by two.
35
32
• Excel: place cursor in blank cell and
32 type=MEDIAN(A2:A21) where A2
32
is the first cell you want to include
29
27 andd A21 the
th llast)
t)
26
23
23
22
21

Copyright © 2012
The Health Care Data Guide: Learning from Data for Improvement.
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

13
 6/12/2012

Why Bother..What Do we Do With A Signal?

• Signals can be evidence of improvement

- That changes are adding up to improvement
• Signals can be evidence that things got worse
- Changes caused unexpected degradation of process
or outcome
- Something else entered the process
- resulting in a signal
• Our job when seeing a signal
- Go learn from signal and take appropriate
- action
Copyright © 2012

Rule 1: Shift
• Six or more consecutive POINTS either all above or all below the
median. Skip values on the median and continue counting points.
Values on the median DO NOT make or break a shift.

Rule 1

25
or Characteristic

20

15

10
Measure o

5
Median=11
Median=10
0
1 2 3 4
Median
5 6 7 8
10 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

The Health Care Data Guide: Learning from Data for Improvement. L
loyd Provost and Sandra Murray, Jossey-Bass, 2011. Copyright © 2012

14
 6/12/2012

Rule 1 YES

5 6
2 8
1 3 4 7

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

Rule 2: Trend
•Five points all goingup or all going down. If the value of two or more
successive points is the same count the first one then ignore the identical
points when counting; like values do not make or break a trend.

Rule 2
25
M easurre or C haracteristic

20

15

10

5
Median 11

0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

The Health Care Data Guide: Learning from Data for Improvement.
Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

15
 6/12/2012

Rule 1-YES
Rule 2-NO

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

Rule 3: Runs Microsoft Word

Document

To Determine The Number of Runs Above and Below

the Median:
- A run iis a series
i off points
i t in
i a row on one side
id off
the median. Some points fall right on the median,
which makes it hard to decide which run these points
belong to.
- So, an easy way to determine the number of runs is to
count the number of times the data line crosses the
median and add one.
- Statistically significant change signaled by too few or
too many runs.

Copyright © 2012

16
 6/12/2012

Rule 3: NUMBER OF RUNS Microsoft Wo

Documen
• Steps
- Count the # of data points not falling on the median (in this case 10)
- Count the # of runs (# times data line crosses the median + 1) (in this case
2)
- Go
G to table
bl andd find
fi d out if you have
h too few
f or too many runs

Rule 3
Characeristic

Data line crosses once

25 Too few runs: total 2 runs
20
15 Median 11.4
Measure or C

10
5
0
1 2 3 4 5 6 7 8 9 10

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

Rule 3: # of Runs
Table for Checking for Too Many or Too Few Runs on a Run Chart
Total number of data Lower limit for the number of runs Upper limit for the number of runs
points on the run chart (< than this number of runs is “too few”) (> than this number of runs is “too many”)

that do not fall on the

median
10 3 9
11 3 10
12 3 11
13 4 11
14 4 12
15 5 12
16 5 13
17 5 13
18 6 14
19 6 15
20 6 16
21 7 16
22 7 17
23 7 17
24 8 18
25 8 18
Table is based on about a 5% risk of failing the run test for random patterns of data. Frieda S. Swed and Churchill Eisenhart,
(1943). “Tables for Testing Randomness of Grouping in a Sequence of Alternatives.Copyright
Annals©of2012
Mathematical
Statistics. Vol. XIV, pp.66 and 87, Tables II and III

17
 6/12/2012

Rule 3
• To Determine The Number of Runs Above and Below the Median:
Microsoft Word
- A run is a series of points in a row on one side of the median. Some points fall right on the
Document
median, which makes it hard to decide which run these points belong to.
- So, an easy way to determine the number of runs is to count the number of times the data
line crosses the median and add one.
- Statistically significant change signaled by too few or too many runs.

8 Rule 3
7
6
5 Median 3.66
4
3
2
1 20 data points not on median. 18 crossings +1= 19 Runs= Too many runs

0
J- F M A M J J A S O N D J- F M A M J J A S
03 04

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

Rule 3: # of Runs
Table for Checking for Too Many or Too Few Runs on a Run Chart
Total number of data Lower limit for the number of runs Upper limit for the number of runs
points on the run chart (< than this number of runs is “too few”) (> than this number of runs is “too many”)

that do not fall on the

median
10 3 9
11 3 10
12 3 11
13 4 11
14 4 12
15 5 12
16 5 13
17 5 13
18 6 14
19 6 15
20 6 16
21 7 16
22 7 17
23 7 17
24 8 18
25 8 18
Table is based on about a 5% risk of failing the run test for random patterns of data. Frieda S. Swed and Churchill Eisenhart,
(1943). “Tables for Testing Randomness of Grouping in a Sequence of Alternatives.
Copyright © Annals
2012 of Mathematical
Statistics. Vol. XIV, pp.66 and 87, Tables II and III

18
 6/12/2012

Rule 3: NUMBER OF RUNS Microsoft Wo

Documen
• To Determine The Number of Runs
- A run is a series of points in a row on one side of the median. Some points fall right on the median,
which makes it hard to decide which run these points belong to.
- So, an easy way to determine the number of runs is to count the number of times the data line crosses the
median and add one.
- A signal is evidenced by too few, or too many runs.

• Steps
- Count the # of data points not falling on the median (in this case 10)
- Count the # of runs (# times data line crosses the median + 1) (in this case 2)
- Go to table and find out if you have too few or too many runs ( in this case should have 3-9
runs. Only have 2, so too few runs.)

• What does it mean?

- T
Too few
f runs with
i h data
d goingi ini our desired
d i d direction
di i isi signal
i l
of improvement
- Too few runs if data going in undesirable direction is signal of
degradation

Copyright © 2012

Rule 1-YES
Rule 2-NO

7 + 1 + 8 Runs

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

19
 6/12/2012

Rule 3: # of Runs
Table for Checking for Too Many or Too Few Runs on a Run Chart
Total number of data Lower limit for the number of runs Upper limit for the number of runs
points on the run chart (< than this number of runs is “too few”) (> than this number of runs is “too many”)

that do not fall on the

median
10 3 9
11 3 10
12 3 11
13 4 11
14 4 12
15 5 12
16 5 13
17 5 13
18 6 14
19 6 15
20 6 16
21 7 16
22 7 17
23 7 17
24 8 18
25 8 18
Table is based on about a 5% risk of failing the run test for random patterns of data. Frieda S. Swed and Churchill Eisenhart,
(1943). “Tables for Testing Randomness of Grouping in a Sequence of Alternatives.
Copyright © Annals
2012 of Mathematical
Statistics. Vol. XIV, pp.66 and 87, Tables II and III

Rule 1-YES
Rule 2-NO
Rule 3-NO

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

20
 6/12/2012

RULE 4: Astronomical
For detecting unusually large or small numbers:
• Data that is Blatantly Obvious as a different value
• Everyone studying the chart agrees that it is unusual
• Remember:
– Every data set will have a high and a low - this does not mean the high
or low are astronomical

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

Rule 1-YES
Rule 2-NO
Rule 3-NO
Rule 4-NO

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

21
 6/12/2012

How Do We Tell a Change is an Improvement?

• Run chart may speak for itself

• If run chart does not speak for itself we can
analyze it further using probability-based rules
- Can detect signal of change ( a non-random
pattern in the data)
- Signal could be improvement or
degradation

Copyright © 2012

Let’s Practice
• Please work in
pairs
• Evaluate the
following run charts
to determine :
- Does the chart show
a signal?
- If signal noted -
which of the four
rules did you use to
find it?

Copyright © 2012

22
 6/12/2012

Rules for Indentifying Non-Random Signals of

Change

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

Behavioral Health: Crisis Hours Provided In-Network

Hours 625 556 492 699 435 553 526 675 611 700 727 647 664 695 602 789 710 761 710 723 722 712 743 729
Run chart
1,000

900 Desired Direction

800

700
H ours

Median line = 625

600

500

400

Chg. 2 Chg. 4
300
Chg. 1 Chg. 3
200
D J F M A M J J A S O N D J F M A M J J A S O N

Copyright © 2012

23
 6/12/2012

Percent Ventilator Associated Pneumonia Bundle Compliance

% 71.0 68.2 84.9 89.9 81.0 62.0 92.3 91.2 95.4 94.1 96.0
Run chart
100

95 Desired Direction

90 Median = 89.9

85
P e rc e n t

80

75

70

65

60

55

50
J A S O N D J-12 F M A M

Copyright © 2012

Change 1 Change 2 Change 3

Copyright © 2012

24
 6/12/2012

Why Bother..What Do we Do With A Signal?

• Signals can be evidence of improvement
- That changes are adding up to improvement
• Signals can be evidence that things got worse
- Changes caused unexpected degradation of process or outcome
- Something else entered the process
resulting in a signal
• Action when seeing a signal
- Go learn from signal and take appropriate action
• If testing change and see no signal:
- Changes not strong enough
- Changes really made?
- Testing on such small scale--not impacting
system yet
- Measure not sensitive
Copyright © 2012

Some Keys to Good Graphical Display with Run

Charts
• When do we begin a run chart?
- As soon as we have a data point

Copyright © 2012

25
 6/12/2012

When Do We Start a Run Chart?

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

Proper Use of the Median

• When should we apply a median?
- Will depend on your situation
• If very little data baseline median may be only a few data
points
• If want to apply probability-based rules for analysis of run
chart need 10 data points for median
- If graph shows no signals (shift, trend, runs
astronomical) and median made from 10 or more
d points
data i freeze
f and extend median i into
i the
h
future
• This will result in earliest possible detection of signals

Copyright © 2012

26
 6/12/2012

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

If median not frozen and extended will result in delayed

detection of signals

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

27
 6/12/2012

A signal is detected utilizing both original and extended

median

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

If a signal is detected and sustained a new median may

be created for the new process performance
• When analyzing run chart with two separate medians rules
are must be applied separately to the data surrounding each median

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

28
 6/12/2012

Plotting Rare Events

• Results in too many zeros
• Makes interpretation difficult and chart of little
value
• Useful alternative is to chart time or workload
between undesirable events
- Up is always good for these charts

Copyright © 2012

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

29
 6/12/2012

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

Days Between MRSA

Run chart
40

35
Days Between Cases off M RSA

30

25

20

15

10
Median line = 7
Extra line Chg 2 Impl
5
Chg 3 Impl
Chg 1 Chg 2 Chg 3
Chg 1 Impl
0
11
6
3 / /7
3/ 5
22
4/ 1
4/ 1
4/ 4
26
3
5 / /3
5/ 3
5/ 9
28
6/ 4
6/ 0
6/ 4
6/ 1
30
3
7/ 7
18
7/ 3
25
2
8 / /8
21
9 / /5
1 1
1 0 0 /8
11 31
12 0
1 /5
14 6
2
2 / /1
2/ 8
3/ 8
19
3/

4/

5/

6/

7/
7/

8/
1

1
1

1
1

1
1
2

/2

1 / 2 /2
/1

1
2
3

2
2/

7/

/
3/

Copyright © 2012

30
 6/12/2012

Judgment Vs. Improvement

The Health Care Data Guide: Learning from Data for Improvement.
Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

Nifty Things You Can Do With Run Charts

Copyright © 2012

31
 6/12/2012

Improvement Projects Require a Family of Measures

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

Improvement Projects Require a Family of Measures

• 2-8 measures typically -Each on a graph -All viewed on one page

Fig 3.6: Improvement Evident Using a Set of Run Charts Viewed on One Page

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

32
 6/12/2012

Small Multiples
• Multiple run charts viewed on one page
• All these run charts are about the same measure
but for a different location,
location provider or segment of
the population
• Each has the same scale vertically and
horizontially
• Allows for rapid comparison

Copyright © 2012

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

33
 6/12/2012

May Display More Than One Measure on a

Graph

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

May Use Different Measure for Each Axis

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

34
 6/12/2012

Sometimes We Don’t Have Much Data

• May not be rich in data but that data may still lead to a high degree of belief
in the change(s) tested
• Characterize the change by describing the before and after medians
• Minimizes ppoint-to-point
p variation

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

Stratification or Disaggregation

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

35
 6/12/2012

Cautions with Graphing Raw Data

• Plotting raw data can be misleading if a useful denominator would
lead to another conclusion
• Use of ratio minimizes confusion from changes in denominator
volume
l
• Ratio = numerator for key measure
denominator (for unit of production or volume related to key measures)
Key Measure (Numerator) Possible Denominator Ratio
# ADEs # Doses Dispensed ADE/Dose
OR Costs # Surgeries OR Cost/Surgery
# Peri‐operative
P i ti Adverse
Ad # Admissions
Ad i i POAE/Ad i i
POAE/Admission
Events
Patients LWBS # Patients Registering in ED Patients LWBS/# Patients
Registered
# Falls # Patient days Falls/Patient Day

Copyright © 2012

Number of Falls

12

10

8
# Falls

0
M- A M J J A S O N D J- F M A M J J A S O N
07 08

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

36
 6/12/2012

Run Chart
• A line graph of data plotted over time
• Data is kept in time order
• Can see flow of data
• Helps answer questions:
- What is our baseline variation?
- How much variation do we have?
- How is process changing over time?
- Has our change resulted in an improvement?
- Did I hold the improvement?

Copyright © 2012

Tools to Learn from Variation in Data

Frequency Plot Pareto Chart Scatter Plot

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

37
 6/12/2012

Tools for Understanding Variation

• Run Chart: Study variation in data over time; understand
the impact of changes.
• Shewhart Chart: Is myy pprocess stable;;
predictable? Distinguish between special and
common causes of variation.
• Pareto Chart: Focus improvement on with greatest
potential impact.
• Frequency Plot: Understand distribution of data (e,g,
(e g
central location, spread, shape, and patterns).
• Scatter Plot: Analyze potential relationship between two
variables.
Copyright © 2012

Shewhart Control Charts: What

Am I Looking At and Why Bother!
• What is Shewhart chart?
• Special and common cause variation
• H tto interpret
How i t t one
• Uses of Shewhart charts
• Why bother?
• There are different kinds of Shewhart charts

Copyright © 2012

38
 6/12/2012

Shewhart Chart:
What Is It?
• A tool to differentiate
special from common
cause variation
• Data is usually displayed
over time
• Most often in time order

Shewhart chart will include:

•Center line (usually mean)
•Data points for measure
•Statistically calculated upper and lower 3 sigma limits
(Limits typically created with 20 or more subgroups)

The Health Care Data Guide: Learning from Data for Improvement.
Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

Introduction to Shewhart Chart

• Statistical tool used to distinguish special from
common cause variation

Straight limits indicate

equal subgroup size

The Health Care Data Guide: Learning from Data for Improvement.
Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

39
 6/12/2012

Smaller subgroup = wider limits

Larger subgroup – tighter limits
Varying limits indicate unequal subgroup size

The Health Care Data Guide: Learning from Data for Improvement.
Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

Types of Variation: Common Cause

• The variation is due to the process or system

design
• It is produced by interactions of inherent
variables in the process
• The causes affect everyone working in the process
and all outcomes of the process
• Process having only common cause affecting the
outcome is called stable
- Performance is predictable

The Health Care Data Guide: Learning from Data for Improvement.
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011. Copyright © 2012

40
 6/12/2012

Management Strategy: Common Cause System

STRATEGY TO TAKE:
• Process Study and Redesign!!
- Understand that process performance will not change unless
process design is fundamentally altered
- Identify process variables contributing to common cause
variation
- Determine which aspect of the process to change
- PDSA the process change
ACTIONS TO AVOID:
• Doing nothing at all!
• Tampering
• Trying to attach specific meaning to fluctuations in the data
(i.e. explain the difference between points that are high vs... low)

The Health Care Data Guide: Learning from Data for Improvement.
Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

Types of Variation: Special Cause

• Variation in the process assignable to a

specific cause or causes - not part of
the usual process
• This variation due to specific
circumstances
• Process not stable
- Is not predictable

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

41
 6/12/2012

Management Strategy: Special Cause System

IMPROVEMENT STRATEGY:
• Investigate, learn and standardize the process!!
- Immediately try to understand when Special Cause occurred
- Study what was different when Special Cause occurred
- Identify ways to prevent or use it, if understandable, to
standardize the process
• either standardize back to where the process was
• or standardize in a new better place
ACTIONS TO AVOID:
• Doing nothing at all
• Failing to involve the people who work in the process
in identifying special causes

The Health Care Data Guide: Learning from Data for Improvement.
Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

Distinguishing Special from Common Cause

Variation

The Health Care Data Guide: Learning from Data for Improvement.
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011. Copyright © 2012

42
 6/12/2012

Understanding Variation
• We can make two mistakes
- Mistake 1: thinking an outcome is due to a
special cause when it was really due to
common causes
- Mistake 2: thinking an outcome is due to
common causes when it was really due to a
special cause
• Shewhart
Sh h charts h hhelp
l minimize
i i i these
h two
mistakes

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

Rules or detecting a special cause

Note: A point exactly on a control limit is not considered Note: A point exactly on the centerline does
outside the limit Note: Ties between two consecutive points do not
not cancel or count towards a shift
cancel or add to a trend.
When there is not a lower or upper control limit
Rule 1 does not apply to the side missing limit

When there is not a lower or upper control limit

Rule 4 does not apply to the side missing limit

The Health Care Data Guide: Learning from Data for Improvement.
Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

43
 6/12/2012

Let’s Analyze One Together

• We always apply all 5 rules to each chart
- Anyy one rule “activated” indicates special
p cause
in that area
- Common cause is determined by “ruling out”
special cause (none of 5 rules activated)
• Let’s consider appropriate action based on your
analysis
l i
- Special cause action?
- Common cause action?
Copyright © 2012

Percent Handwashing Compliance

p c hart
100

90 UCL

80
CTL
%

70
LCL

60

50
J 08 F M A M J J A S O N D JAN 09 F M A M J J A

Copyright © 2012

44
 6/12/2012

Let’s Practice
• Please analyze one of these charts
• Apply all 5 rules to each chart
• Circle
i l special
i l cause if you find
fi d it
i
• What action would you take based on your
analysis?
- Special cause action?
- What would you do if it is solely common cause?

Copyright © 2012

Percent Parent Satisfaction in Top Box

p chart
30

25 UCL = 23.95

20

15
%

CTL = 11.73

10

0
07

07

07

07

07

07

07

07

07

08

08

08

08

08

08
7

7
00

00

00
20

20

20

20

20

20

20

20

20

20

20

20

20

20

20
/2

/2

/2
1/

1/

1/

1/

1/

1/

1/

1/

1/

1/

1/

1/

1/

1/

1/
/1

/1

/1
1/

2/

3/

4/

5/

6/

7/

8/

9/

1/

2/

3/

4/

5/

6/
10

11

12

Copyright © 2012

45
 6/12/2012

Average Time to Acknowledge Referrals

Individuals
34
32
30 UCL = 29.05
28
26
24
22
20
H o u rs

18
16
14
12
10 Mean = 9.44
8
6
4
Chg 1 Chg 2 Chg 3 Chg 4
2
0
15

22

11
14

26

13

19

28

10

14

21

30

23

18
25
6

8
3/

3/

4/

5/

5/

6/

7/

7/

8/

8/
3/

3/

4/
4/

4/

5/

5/

5/

6/

6/

6/

6/

7/

7/
7/
Weeks

Copyright © 2012

Why Distinguish Special From Common Cause

Variation?
• When monitoring key processes
- Can tell if they have remained the same,
degraded or improved
• When working specifically to improve:
- Special cause:
• may be evidence of improvement
• or… an unintended consequence such as degradation
in the results
- Common cause:
• indicates that the changes have not resulted in
improvement

Copyright © 2012

46
 6/12/2012

Using a Control Chart

• Learn how much variation exists in process
- If stable are predictable. Can use info in planning,
communicate with staff, patients, family
• Assess stability and determine improvement strategy
(common or special cause strategy)
• Monitor performance and correct as needed
• Find and evaluate causes of variation
• Tell if our changes yielded improvements
• See if improvements
p are “sticking”
g

Copyright © 2012

Are Our LOSs for DRG XXX Stable?

Data4 6 7 5 4 6 4 8 3 6 7 5 6 7 8 7 7 8 6 8 9 6 7 8 6
14
Individuals

12
Stable process
UCL=11.1 Predictable
10
LOS in Days

Mean-6.2
6

2 LCL=1.7

0
19

20

21

22

23

24

25
18
1

16

17
10

11

12

13

14

15

Sequential Cases
Copyright © 2012

47
 6/12/2012

Using a Shewhart Chart

• Learn how much variation exists in process
• Assess stability and determine improvement strategy
(common or special cause strategy)
- When sponsoring improvement effort it’s it s helpful
helpful, if data
readily available, to determine if process has only
common cause or if special cause also present
• Monitor performance and correct as needed
• Find and evaluate causes of variation
• Tell if our changes yielded improvements
• S if iimprovements are “sticking”
See “ i ki ”

Copyright © 2012

Are Our LOSs for DRG XXX Stable?

Data4 6 7 5 4 6 4 8 3 6 7 5 6 7 8 7 7 8 6 8 9 6 7 8 6
14
Individuals

12 Stable not
Stable but perhaps process
good enough
UCL=11.1 Requires process redesign to improve
Predictable
10
LOS in Days

Mean-6.2
6

2 LCL=1.7

0
19

20

21

22

23

24

25
18
16

17
1

6
2

10

11

12

13

14

15

Sequential Cases

Copyright © 2012

48
 6/12/2012

Coding Errors per Groups of 20 Records

c c hart
30
Special Cause variation
What is our action here?
25

20 UCL = 19.37
# Co d in g E rro rs

15

Mean = 9.92
10

LCL = 0.47
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
Sequential Subroups of 20 Records

Copyright © 2012

Using a Shewhart Chart

• Learn how much variation exists in process
• Assess stability and determine improvement strategy
(common or special cause strategy)
• Monitor
M it performance
f and
d correctt as needed
d d
• Find and evaluate causes of variation
• Tell if our changes yielded improvements
• See if improvements are “sticking”

Copyright © 2012

49
 6/12/2012

The Health Care Data Guide: Learning from Data for Improvement.
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011. Copyright © 2012

Using a Shewhart Chart

• Assess stability and determine improvement
strategy (common or special cause strategy)
• Monitor performance and correct as needed
• Find and evaluate causes of variation
• Tell if our changes yielded improvements
• See if improvements are “sticking”

Copyright © 2012

50
 6/12/2012

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

Using a Shewhart Chart

• Learn how much variation exists in process
• Assess stability and determine improvement strategy
(common or special cause strategy)
• Monitor performance and correct as needed
• Find and evaluate causes of variation
• Tell if our changes yielded improvements
- When you intend to improve process you are on the lookout
for special cause indicative of improvement
• See
S if iimprovements are “sticking”
“ i ki ”

Copyright © 2012

51
 6/12/2012

Percent Unplanned Returns to OR P chart

# Surgeries 984 982 996 998 1070 1031 886 964 1128 960 1193 998 1070 895 852 963 956 1001 956 995 987 943 965 980 923 1106

# Pts Return 27 20 25 23 31 17 21 28 24 22 19 24 30 22 15 18 12 22 8 2 9 6 20 6 2 6

p c hart
4.0

UCL = 3.54 Good

3.5

3.0
Percent

2.5
CTL = 2.16
2.0

1.5

10
1.0
LCL = 0.78

Chg 2 & 3 Chg 14

0.5 Goal = 0.5
Chg 7 & 8 Chg 10 & 11
Chg 1 Chg 4 & 5Chg 9 Chg 12 & 13
Implement
0.0
F 04M A M J J A S O N DJ 05F M A M J J A S O N DJ 06F M A M

Copyright © 2012

Using a Shewhart Chart

• Learn how much variation exists in process

• Assess stability and determine improvement
strategy (common or special cause strategy)
• Monitor performance and correct as needed
• Find and evaluate causes of variation
• Tell if our changes yielded improvements
• See if improvements are “sticking”

Copyright © 2012

52
 6/12/2012

The Health Care Data Guide: Learning from Data for Improvement.
Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

The Health Care Data Guide: Learning from Data for Improvement.
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011. Copyright © 2012

53
 6/12/2012

Run Vs. Shewhart Chart

The Health Care Data Guide: Learning from Data for Improvement.
Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

Technique matters!
-Obtain baseline mean/limits from stable period and freeze them
-Minimum baseline 12, preferred 20-30

The Health Care Data Guide: Learning from Data for Improvement.
Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

54
 6/12/2012

Selecting the Appropriate Shewhart Chart

Type of Data
Count or Classification Continuous
(Attribute Data) (Variable Data)

Count Classification Subgroup Unequal or Equal

(Nonconformities) (Nonconforming) Size of 1 Subgroup
Size

Equal Area of Unequal Area of Unequal or Equal

Opportunity Opportunity Subgroup Size
I Chart (X chart) X‐Bar and S
C Chart U Chart P Chart chart
Number of Nonconformities
Percent Individual Measures Average and
Nonconformities Per Unit Nonconforming Standard Deviation
Other types of control charts for attribute data: Other types of control charts for continuous data:
1. NP (for classification data) 7. X‐bar and Range
2. T-chart [time between rare events] 8. Moving average
3. Cumulative sum (CUSUM) 9. Median and range
4. Exponentially weighted moving average (EWMA) 10. Cumulative sum (CUSUM)
5 G chart (number of opportunities between rare events) 11. Exponentially weighted moving average (EWMA)
6. Standardized control chart
12. Standardized control chart
Copyright © 2012
Source: The Health Care Data Guide. Provost and Murray Jossey-Bass, 2011

Tools to Learn from Variation in Data

Frequency Plot Pareto Chart Scatter Plot

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

55
 6/12/2012

Fall Rate per 1000 Resident Days

# Days/1000 3.357 3.012 3.718 2.983 3.108 2.948 2.721 2.690 2.567 2.667 2.824 2.882 3.429 2.829 3.092 2.605 2.610 2.531 2.502 2.615 2.662 2.806 2.591 2.403
# Falls 8 13 11 7 11 18 10 15 9 9 5 16 9 9 8 4 7 12 15 9 12 13 9 15
u chart

7 UCL = 7.16
R a te p e r 1 0 0 0 R e s id e n t D a y s

4 CTL = 3.73

1
LCL = 0.29

0
ar

ar

r
10

ay

g
p

v
Ja ec
11

ay

g
p

v
c
l

t
Ju

Ju
Ap

Oc

Ap

Oc
Ju

No

Ju

No
De
Fe

Au
Se

Fe

Au
Se
M

M
D
M

M
n

n
Ja

Copyright © 2012

Factors Associated with Resident Falls

N=254
104
40.94%
# of Events

52
20.47% 46
18.11%

21
8.27% 17
6.69%
6 4
2.36% 2 2
1.57% 0.79% 0.79%
Getting Out of Bedd

Using Rest Room

Bending Over

M issed Chair
No Glasses

W et Floor
Bum ped

Other
Trip

Copyright © 2012

56
 6/12/2012

Number of Falls by Time of Day

Histogram
30

25

20
# F a lls

15

10

0
0 2 4 6 8 10 12 14 16 18 20 22
Time of Day (24 Hour Clock)

Copyright © 2012

Pareto Chart
• Bar chart with bars in rank order
• Each bar represents a different variable,
factor or problem
• Becomes useful with 30-50 pieces of data
• Looking for 20% of bars representing 80% of
opportunity
• Want to know where to focus our efforts
- Which are the vital few areas we should concentrate on?
- Which variables out of many are occurring most?

Copyright © 2012

57
 6/12/2012

Pareto Chart: What Does One Look Like?

Reasons Cited for Lack of Childhood Immunizations: Group A
Count 1,503
791
52.63%

700
600
# Responses

500
400
264
300 17.56%
134 128
200 98
8.92% 8.52% 45 43
6.52%
100 2.99% 2.86%
0
Childcare Needs
No Transport

Don't Believe
Cost of Imm.

No Time

No Info

Other
The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

Pareto Chart: What Does One Look Like?

Reasons Cited for Lack of Childhood Immunizations: Group A
Count 1,503 Percent
100%
1,400

1,200 80%

1,000
791
60%
52.63%
# Responses

800

600 40%

400 264
17.56% 134 128 20%
98
200 8.92% 8.52% 45 43
6.52%
2 99%
2.99% 2 86%
2.86%
0 0%
Childcare Needs
No Transport

Don't Believe
Cost of Imm.

No Time

No Info

Other

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

58
 6/12/2012

When Is It Used?

• When data can be arranged into

categories
• When the rank of each category is
important
• When we need to focus on the most
i
important
t t problems
bl or causes off
variation

Copyright © 2012

Model for Improvement

What are we trying to
accomplish?
How will we know that a
change is an improvement?
What change can we make that
will result in improvement?

Act Plan

Check Do

Copyright © 2012
The Health Care Data Guide: Learning from Data for Improvement. Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

59
 6/12/2012

Frequency Table: Where Are Needlesticks

Occurring?
Variable Wk1 Wk 2 Wk 3 Wk 4 Total %
3W 17 14 16 12 59 26.34
ED 4 3 4 6 17 7.59
ICU 4 7 2 3 16 7.14
2N 1 3 5 3 12 5.36
3N 7 2 3 5 17 7.59
2W 4 6 4 6 20 8.93
Allergy/Imm 6 2 3 3 14 6.25
2S 10 13 12 13 48 21.43
Lab 3 2 3 4 12 5.36
Other 3 3 3 0 9 4.02
Grand Total 224 100 %

Copyright © 2012

Needlesticks By Location
(n=224)
Count Percent
59
60 26.34%

25%

48
50 21.43%

20%

40
Count

15%
30

20
8.93%
10%
20 17 17
16
7.59% 7.59% 14
7.14%
6 25%
6.25% 12 12
5.36% 5.36% 9
4.02% 5%
10

3W 2S 2W 3N ED ICU Allergy/Imm. Lab 2N Other

Copyright © 2012

60
 6/12/2012

How Is It Interpreted?
• Look for the Pareto effect
• We won
won’tt always find it!
- Is entire chart speaking to us?
- Can we re-stratify?
- Last choice is selecting a column and
tackling it!

Copyright © 2012

Factors Related To Severely Mentally Disabled Adult Recidivism (Readmission)

Count 196 Percent
103
52.55%
100
50%

80 40%
C ount

60 30%

38
40 19.39% 20%

20
10.20%
20 14 10%
7.14% 11
5.61% 8
4.08%
2
1.02%

Non-Comp.w/Meds ETOH/Oth Sub. Instablity Housing Non-Comp other TX. Lack Fam. Supt. Other Psyhosoc. Other

Copyright © 2012

61
 6/12/2012

How Is It Interpreted?
• Look for the Pareto effect
• We won
won’tt always find it!
- Is entire chart speaking to us?
- Can we re-stratify?
- Last choice is selecting a column
and tackling it!

Copyright © 2012

Factors Related to Pediatric Head Injury

Count Percent
20
29.41% 30%
20

16 25%
23.53%

15 14
20.59%
20%
Count

10 15%

6
10%
8.82%

5 4
5.88%

2 2 2 5%
2.94% 2.94% 2.94%
1 1
1.47% 1.47%

Rollerblade Skateboard Bike Motor Veh. Fall Struck Pedestrian Motorcycle Other Fight

Copyright © 2012

62
 6/12/2012

How Is It Interpreted?
• Look for the Pareto effect
• We won
won’tt always find it!
- Is entire chart speaking to us?
- Can we re-stratify?

Copyright © 2012

Location of Resident Falls

Count # Falls

53
20.87%

50
43 42
16.93% 16.54%
40 37
14.57%
# o f F a lls

29
30 11.42%

19
20 7.48%
14
5.51% 12
4.72%
10 5
1.97%

0
Rest Room s

Dining Area

Gardens

Lounge

Library
Room

Other
Halls

Trips

Copyright © 2012

63
 6/12/2012

Factors Associated with Resident Falls

N=254
104
# of E vents 40.94%

52
20.47% 46
18.11%

21
8.27% 17
6.69%
6 4
2.36% 2 2
1.57% 0.79% 0.79%
G etting O ut of Bed

Us ing Res t Room

Bending O ver

M is sed Chair
No G las s es

W et Floor
Bum ped

O ther
Trip

Copyright © 2012

Other Ways To Use Pareto

• Stratification

Copyright © 2012

64
 6/12/2012

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

65
 6/12/2012

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

Pareto Chart
• Bar chart with bars in rank order
• Each bar represents a different variable, factor or
problem
bl
• Looking for 20% of bars representing 80% of
opportunity
• Want to know where to focus our efforts
- Which are the vital few areas we should concentrate on?
- Which variables out of many are occurring most?

Copyright © 2012

66
 6/12/2012

Tools to Learn from Variation in Data

Frequency Plot Pareto Chart Scatter Plot

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

Frequency Plot (Histogram): What Is It?

• A bar chart for one variable only
• Most often used with time, money, throughput or a scaled
measurement ((i.e. dollars,, weight,
g , age,
g , height)
g )
• Used to visualize central location, shape and spread of the
data
• Each bar equal, each distinct
• Becomes useful with 30-50 pieces of data
• Frequency Plot does little good for interpretation if process
not stable
• Doesn’t show stability

Copyright © 2012

67
 6/12/2012

The Tool List

• Frequency Plot:
- How is this one variable
distributed (what is the spread of
LOS, Cost, HA1C, etc. in our
population)?
p p )

Copyright © 2012

What Does a Histogram Look Like?

Count
Age of People with Diabetes Who Have HbA1C> 8

140
131

120

100

80

60
52 51
43
40 40 40 40

29

20 16
7

0-9 10-19 20-29 30-39 40-49 50-59 60-69 70-79 80-89 90-99

Copyright © 2012

68
 6/12/2012

When Is It Used?
1. Have a set of values related to your
question (i.e.
(i e arrival times in ED)
2. Want to see central location, shape,
spread of data to learn about system
- Any patterns that bear looking into?
- Does all of process fit within needs? (Our
standards)

Copyright © 2012

Model for Improvement

What are we trying to
accomplish?
How will we know that a
change is an improvement?
What change can we make that
will result in improvement?

Act Plan

Study Do

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

69
 6/12/2012

Jun 2011 ED Patient Arrival Times (1 Week Weekdays, n=524)

80

70

60

50
# P ts

40

30

20

10

0
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
Time of Day 24 Hour Clock

Copyright © 2012

How Is It Interpreted?
• Evaluate central location
• Evaluate spread
p
• Learn from shape

Copyright © 2012

70
 6/12/2012

g j
30

25 uries 24

20
# of Head Inju

15

10 8 8 7
6
5
5
2 3
1
0

Age in Years
Copyright © 2012

How Is It Interpreted?
• Evaluate central location
• Evaluate spread
p
• Learn from shape

Copyright © 2012

71
 6/12/2012

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

How Long do Patients Wait In Our Clinic?

45

40
his Time Range

35

30
# of Times a Patient Waited in Th

25

20

15

10

Minutes

Copyright © 2012

72
 6/12/2012

How Is It Interpreted?
• Evaluate central location
• Evaluate spread
p
• Learn from shape

Copyright © 2012

Common Frequency Plot Shapes

Symmetrical Bimodal
•normal distribution •two peaks
•data
d t from
f two
t processes
•separate and analyze each

Copyright © 2012

73
 6/12/2012

30 How is Age Distributed Among Patients Who Fell in our Care?

# of Falls in That Age Category

25

20

15

10

Age in Years
Copyright © 2012

Common Frequency Plot Shapes

Uniform Random
•provides little info •provides little info
•check to see if multiple •check to see if multiple
sources variation combined sources variation combined
•if
if so,
so re-stratify
re stratify and graph •iff so, re-stratify
f and graph
•may mean not enough bars •May mean too many bars
•if so, change bar width and •if so, change bar width and
graph graph

Copyright © 2012

74
 6/12/2012

40
What Time Do People Call the Crisis Hotline?

This Timeframe
35

30

25
s Hotline Called In T

20

15

10
# Times

Hours (24 Hour Clock)

Copyright © 2012

30
What Time Do People Call the Crisis Hotline?
This Timeframe

25

20
s Hotline Called In T

15

10
# Times

Hours (24 Hour Clock)

Copyright © 2012

75
 6/12/2012

Shewhart’s Rules
• When average, range or histogram used
to summarize data:
- Summary should not mislead user into
taking any action user would not take
if data were presented in a time series
(graph)
- Averages,
Averages etc
etc.. are useful,
useful but seeing
the sequence and variation in data is
most meaningful

Copyright © 2012

Are They the Same?

Clinic Avg. Annual Sat Capitated Cost

(1-5 Scale) Annual
A 3.9 $980

B 3.9 $940

C 3.9 $945

Copyright © 2012

76
 6/12/2012

Comparison of Averages, Frequency Plots and Run Charts

A v er age Client Satis f ac tion- Clinic A Client Satis f ac tion - Clinic A

# M o n t h s F a llin g in E a c h C a t e g o ry
7 Individuals
5.0

6 4.8
4.6
5
UCL = 4.47
4.4

4 4.2 Mean = 4.17

4.0
3
LCL = 3.88
3.8
2 3.6
3.4
1
3.2
0 3.0

34
3.4 35
3.5 36
3.6 37
3.7 38
3.8 39
3.9 40
4.0 41
4.1 42
4.2 43
4.3 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Aver ag e Sati s fac ti on Sc or e Month

A v er age Client Satis f ac tion- Clinic B Client Satis f ac tion - Clinic B
# M o n t h s F a llin g in E a c h C a t e g o ry

8 Individuals
5.0
7 4.8 UCL = 4.69
4.6
6
4.4
5 Mean = 4.17
4.2
4 4.0

3 3.8
LCL = 3.66
3.6
2
3.4
1 3.2
0 3.0

3.4 3.5 3.6 3.7 3.8 3.9 4.0 4.1 4.2 4.3 4.4 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Aver ag e Sati s fac ti on Sc or e Month

Average Client Satis faction-Clinic C Client Satis faction -Clinic C
# M o n t h s F a llin g in E a c h C a t e g o ry

7 Individuals
5.0

6 4.8 UCL = 4.70

4.6
5
4.4
4.2 Mean = 4.17
4
4.0
3
3.8
LCL = 3.65
2 3.6
3.4
1
3.2

0 3.0

3.4 3.5 3.6 3.7 3.8 3.9 4.0 4.1 4.2 4.3 4.4 1 2 3 4 5 6 7 Copyright
8 9 10 11 ©122012
13 14 15 16 17 18 19 20 21 22 23 24

Aver ag e Sati s fac ti on Sc or e Month

Stratification with Frequency Plot

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

77
 6/12/2012

Stratification with Frequency Plot

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

Another View of Stratification

Average Satisfaction with Clinic (1-5 Scale)
Indiv iduals
5.0

4.8
UCL = 4.68
4.6

4.4

4.2
Average Satisfaction

4.0
Mean = 3.81
3.8

3.6

3.4

3.2

3.0 LCL = 2.94

2.8

2.6

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
Sequential Weeks

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

78
 6/12/2012

Stratification

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

Various Formats

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

79
 6/12/2012

Frequency Plot (Histogram): What Is It?

• A bar chart for one variable

• Used to visualize central location, shape and
spread of the data
• Each bar equal, each distinct
• Most often used with time, money, throughput
or a scaled measurement (i.e. dollars, weight,
age, height,)
- Frequency Plot does little good for interpretation if process not
stable
- Doesn’t show stability or capability in and of itself

Copyright © 2012

Tools to Learn from Variation in Data

Frequency Plot Pareto Chart Scatter Plot

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

80
 6/12/2012

SCATTER PLOT: What Is It?

• Graph to evaluate theory about relationship
between one variable and another
- Test for possible cause and effect
- Does not prove a C & E relationship exists
- A cause and effect relationship will be verified only when the
improvement is tested and results studied using a control
chart
• Each dot on the chart represents a pair of measures
• Becomes useful between 30-50 data points

Copyright © 2012

SCATTER PLOT: What Does It Look Like?

Does Customer Waiting Time Affect

Customer Satisfaction?

High Y N ti C
Negative Correlation
l ti
Satisffaction Ratings
Customer
C

Low Customer Waiting Time X

High

Copyright © 2012

81
 6/12/2012

Model for Improvement

What are we trying to
accomplish?
How will we know that a
change is an improvement?
What change can we make that
will result in improvement?

Act Plan

Study Do

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

Data for Scatter Plot : Does wait time impact satisfaction with clinic?

Min Wait Sat Score Min Wait Sat Score Min Wait Sat Score
49 3.5 42 4 74 2
78 1 51 3.5 72 1.5
3 5 76 3 15 5
55 2.5 46 5 64 3.5
15 4 83 2 17 4
28 3 31 5 91 2
96 1.5 60 2 10 4.5
47 3 85 2.5 5 5
15 3.5 70 1.5 9 4
82 1 5 5 71 1.5
24 4 50 3 7 5
68 3 74 2 55 3
64 1 21 4.5 74 2.5

Copyright © 2012

82
 6/12/2012

Draw Graph
•Independent Variable on X Axis (Horizontal)
•Dependent Variable on Y Axis (Vertical)
•Values higher as go up on graph
•Start scale with actual lowest value in your data set
Wait Time and Satisfaction-Blank
Scattergram
5.0

S a tis fa c tio n ((1 -5 S c a le ) D e p e n d e n t V a ria b le

4.5
(49,3.5)
4.0

3.5

3.0

2.5

2.0

1.5

1.0
10 20 30 40 50 60 70 80 90
Wait time (Min) Independent Variable

Copyright © 2012

Wait Time and Satisfaction

Scattergram

5.0
S a tis fa c ti o n ( 1 -5 S c a le ) D e p e n d e n t V a ria b le

4.5
Negative Correlation
40
4.0

3.5

3.0

2.5

2.0

1.5

1.0

10 20 30 40 50 60 70 80 90
Wait time (Min) Independent Variable

Copyright © 2012

83
 6/12/2012

How Is It Interpreted?
• Look for patterns in the scatter plot
- A narrow band of dots
- A circular pattern
- Peaks or troughs

Copyright © 2012

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

84
 6/12/2012

How Is It Interpreted?
• Outliers
- Points that do not fall into the pattern of the
others
the
- Do not cluster with other points
• Should investigate why appear
• May be a measurement error
• Possible may be a signal of a process change
• Possible may be change in relationship between
the factors

Copyright © 2012

Outliers
Case Load Related to Sick Leave
18

16
ed

14
ays of Sick Leave Use

12

10

4
Da

0
30 35 40 45 50 55 60 65 70

Case Load

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

85
 6/12/2012

All Departments: Does Case Load Impact Sick Leave Use? Department A
16 16

14 14

12 12
D a y s S ic k (Y )

D a y s S ic k (Y )
10 10

8 8

6 6

4 4

2 2

0 0
30 35 40 45 50 55 60 65 30 35 40 45 50 55 60 65
Case Load (X) Case Load (X)

Department B Department C
16 16

14 14

12 12
D a y s S ic k ( Y )

D a y s S ic k ( Y )
10 10

8 8

6 6

4 4

2 2

0 0
30 35 40 45 50 55 60 65 30 35 40 Copyright
45 © 201250 55 60 65
Case Load (X) Case Load (X)

Stratification Using Symbols to Distinguish Each Department

The Health Care Data Guide: Learning from Data for Improvement. Copyright © 2012
Lloyd Provost and Sandra Murray, Jossey-Bass, 2011.

86
 6/12/2012

Acuity vs Cost-Total Acuity vs Cost-Department A

Scattergram Scattergram
8 8

7 7

6 6
$ in T h o u s a n d s

$ in T h o u s a n d s
5 5

4 4

3 3

2 2

1 1

0 0
3.0 3.5 4.0 4.5 5.0 5.5 6.0 6.5 3.0 3.5 4.0 4.5 5.0 5.5 6.0 6.5
Acuity Acuity

Acuity vs Cost-Department B Acuity vs Cost-Department C

Scattergram Scattergram
8 8

7 7

6 6
$ in T h o u s a n d s

$ in T h o u s a n d s
5 5

4 4

3 3

2 2

1 1

0 0
3.0 3.5 4.0 4.5 5.0 5.5 6.0 6.5 3.0 3.5 4.0 4.5 © 2012
Copyright 5.0 5.5 6.0 6.5
Acuity Acuity

What Did We Address?

• The value of displaying data graphically vs. table
of numbers or summary statistics
• The differences between data used for
improvement, accountability and research
• The value of displaying data over time:
- when working to determine impact of changes
beingg tested
- To see if are sustaining gains
• The Model for Improvement

Copyright © 2012

87
 6/12/2012

What Did We Address?

• Run charts: what they are, when used, how
interpreted
- Median vs.
vs mean: median used as center line
- Rules for analysis to detect signals of improvement
or degradation
• Ways to use
- Family of measures for improvement project
- Small multiples
- Stratification with
• Importance of good technique with median
Copyright © 2012

What Did We Address?

• Introduction to Shewhart charts: what they are,
what they are used for, how interpreted
- Common and Special cause variation
- Different approaches to improvement with two types of variation
- What are upper and lower limits and where come from
- 5 rules for analysis to detect special cause
• Importance of good technique with limits
• There are different types of Shewhart charts
• How purpose of run chart differs from purpose of
Shewhart chart
Copyright © 2012

88
 6/12/2012

What Did We Address?

• Pareto charts, histograms and
scatter plots:
- what each looks like
- what each is used for
- how each is interpreted
- stratification using these tools
• Matching each of the 5
fundamental tools to the question
being asked
Copyright © 2012

References
Books:
1. The Health Care Data Guide: Learning from Data for Improvement. Lloyd Provost and
Sandra Murray, Jossey-Bass, 2011.
2. Total Quality Tools For Health Care. Productivity-Quality Systems, Inc. Miamisburg
Ohio. ISBN: 1-882683-04-8 Tel. 1-800-777-2255.
3. The Improvement Guide. Gerald J. Langley, Kevin M. Nolan, Thomas W. Nolan,
Cliff d L.
Clifford L Norman,
N Lloyd
Ll d P.
P Provost,
P t Jossey-Bass,
J B 2009
2009.
Video:
1. Making Sense Out of Control Charts. NAHQ. 1-800-966-9392
Software Used:
1. ChartRunner. PQ Systems. 1-800-777-3020.
2. QI Charts. API, 1-512-708-0131
3. Minitab,1-814-238-3280
Articles:
1. The run chart: a simple analytical tool for learning from variation in healthcare
processes. Rocco J Perla, Lloyd P Provost and Sandra K Murray. BMJ Qual Saf 2011
20: 46-51.
I

Copyright © 2012

89