Basics of Research Part 8 Edward A.

Panacek, MD, MPH

Performing Chart Review Studies

Editors’ note: This article is the eighth in a multipart series designed to improve the knowledge base of read-
ers, particularly novices, in the area of clinical research. A better understanding of these principles should
help in reading and understanding the application of published studies. It should also help those involved in
beginning their own research projects.

By definition, retrospective studies are those in which the gency medicine journals for 4 years and found that approxi-
events of interest have already occurred before the research mately 25% of their research articles used the chart review
project is begun. In other words, the interventions or expo- technique. That article provides a framework for reviewing
sures, the outcomes of interest, and all other relevant obser- manuscripts that use this technique and for planning and per-
vations or data measurements have already occurred before forming chart review studies.
the investigator begins the project. However, there are many Chart review studies have several advantages. First, they
different forms of retrospective studies, and it is unfair to are relatively inexpensive to perform. The researchers simply
lump them all together in one category. Different study commit to chart review time. Second, they can be accom-
designs can be used—for example, case-control studies versus plished quickly and using time convenient to the research
retrospective cohort studies.1 team. Charts are generally available any time. Prospectively
In addition, there is a study design called “nested” studies enrolled subjects are not as easily studied. Third, they do not
in which the investigator retrospectively goes to an existing require special laboratories or equipment. All of these factors
database and extracts the subjects of interest that are relevant lead to greater use of a chart review method.
to a new research project. In other words, the study subjects Collecting data from the records retrospectively is a research
are found “nested” within a larger existing database. These dif- technique fraught with more potential errors than is true for
ferent study designs vary substantially in many ways, includ- prospective studies. Understanding the potential pitfalls in
ing their degree of scientific validity. Therefore, it can be mis- these studies allows the investigator to attempt to address them
leading to simply refer to all of them as retrospective or in the research design phase. Principal to performing chart
archival studies. Regardless of the study design selected, it is review studies is an understanding of how the information gets
important to point out that all retrospective studies should into the medical record. That information goes through an
have an “a priori” research question and prospectively defined imprecise process to get to into the medical chart3 and then
study protocol before enrolling the first subject. goes through another imprecise process to be abstracted from
The most common form of retrospective research is the the medical record and into the research database.4,5 That
chart review study, a form of research that is relatively easy to imprecision can, and often does, lead to measurement error. If
perform. It is also easy to do them poorly, so these studies are severe, those errors can completely invalidate the study results.
frequently criticized. However, there are ways to improve the The path the information takes to get to the research data base
methodology and therefore the validity of these studies. This is summarized in Figure 1. That process comprises as many as
part in this series on basic research principles will address just 10 different steps. There is a potential for error at each step. It
that: how to perform chart review studies properly. is the job of the researcher to attempt to identify and minimize
these errors, to obtain the most accurate information possible.
Archival Data Research Table 1 lists an approach to performing chart review studies to
The term archival research refers to the use of archives or maximize the validity of the results. Each step is discussed.
records that already exist. The medical record review, a spe-
cific type of archival data research, is the most common Limitations of Retrospective Data
method to get started in clinical research. This article
Patient History
describes each of the elements of the medical record review
used for clinical research. In a study of the emergency medi- To best use the information available in the medical
cine literature, Gilbert et al2 reviewed three different emer- record, one must identify the actual events that led to the

206 Air Medical Journal 26:5

 Figure 1. The Flow of Information in Chart Review Studies
1. Actual events of interest (motor vehicle crash, syncope, chest pain)
2. Description of the events (by patient, family, EMS, police, etc.)
3. Receipt of the information (by the health care team)
4. Recording of the medical information (by the health care practitioner)
5. Coding and filing of records (by the medical records or billing units)
6. Obtaining the medical records (by the research team)
7. Interpreting the records (by the research team)
8. Abstracting data from the records (by the research team)
9. Collating, recording the data points (into database programs, by research team)
10. Analyzing, interpreting the results (by the study investigator)

Do not assume that information obtained directly from

patients is always true. For example, limited patient recollec-
tion of events can result in recall bias. The longer the time
Table 1. Archival Data Research: Pros and Cons
period, the less a patient will remember.12-14 Even information
Advantages that we expect everyone should recall, such as children’s birth
1. Less resource-intensive than prospective research weight,15,16 allergies,17 and medication histories,18 are frequent-
designs (faster and less expensive) ly inaccurate. The ability of patients to recall information
2. Can quickly evaluate a number of possible associa- varies with patient characteristics, such as age,16 medical con-
tions, which can be evaluated using more focused ditions,19 and who did the reporting.20,21 All of these can cause
prospective studies errors in the data.
3. Can generally be performed at times of convenience
Disadvantages Documentation Process in the Medical Record
1. Numerous potential sources of bias The medical record has many purposes, but research was
2. Serious questions about both internal and external not one of them when originally generated. Inaccuracies of the
study validity medical record are well known. The method of recording also
3. Always problems with missing data can affect accuracy. For example, the process of dictation and
4. Not able to establish true cause-and-effect transcription has been shown to introduce more inaccuracies
relationships (can identify potential associations) into the medical record, such as in recording childhood
immunizations.22 However, a dictated and transcribed medical
record usually contains more information than handwritten
medical records.23 Other technologies, such as voice-recogni-
final chart. In the simplest case, this means that the patient, tion dictation systems and other keyless entry devices, gener-
or surrogate, told something to someone, often the physician ally have improved the accuracy and completeness of docu-
or nurse, who recorded it in the medical record. Because of mentation.24-28 All charting technologies should be assumed to
the retrospective nature of chart review studies, accurately contain errors until formally evaluated for accuracy. However,
determining how the transfer of information from the patient electronic charts can also provide opportunities to access large
to the record occurred and what information loss/degradation databases for studies.29
resulted is usually impossible. Although not the subject of this article, it should be men-
Everyone is familiar with the children’s game called tioned that registries are often used for archival studies. Disease
“telephone” and how statements can change with retelling. registries (eg, cancer or trauma registries) were generally creat-
A similar process occurs in recording the medical ed for surveillance and epidemiologic purposes. Their develop-
encounter. One study6 showed that the physician asks ment was usually not intended primarily for research or to
enough questions to obtain only 68% of the information replace the medical record. Therefore, if used for purposes
available about the mechanism of injury from trauma other than intended, a potential for bias exists.30-33
patients. Of the information obtained, only 67% of it was Diagnoses placed on a discharge summary sheet or a billing
recorded in the medical record, so less than half of the form might have biases based on the intended use of the data.
available information actually was recorded. Differences Such lists are also used by third-party payers, such as insurance
also existed in the amount of information obtained by level companies and the Centers for Medicare and Medicaid Services.
of training; medical students obtained the most information As a result, biases affect what gets documented in this list.34 For
from the patient but recorded the least, and attending example, The Department of Health and Human Services has
physicians obtained the least information from the patient agreed that hospitals can record additional diagnoses that “affect
but recorded the most.7-11 patient care, requiring clinical evaluation, therapeutic treat-

September-October 2007 207

ment, diagnostic procedures, extended length of hospital stay, sions are prone to bias. Bias also can occur when information
or increase nursing care or monitoring.”35 Other discharge diag- must be coded as “missing,” “negative,” or “unsure.”
nosis problems include indistinct coding, variable thresholds If the abstraction is done by multiple personnel, consisten-
for listing chronic conditions, and reluctance for physicians to cy is an issue. Differences in technique between individuals
record complications.36 Even death certificates, which are con- must be measured and minimized. More potential biases and
trolled by law in all 50 states, have been shown to be inaccurate. errors occur as the information gets transferred from the med-
The reasons for this vary with disease entity and location but the ical research to the research database. In addition to the cod-
situation is present in many nations in the world.37,38 ing and categorical errors that can be made, simple transcrip-
Finally, the process of coding that occurs with most med- tion errors can always occur when entering data.
ical records affects subsequent database creation. This is rou- Every study should have an operations manual. At every step
tinely done by medical records or billing personnel.39-41 The in the process of information flow and medical record review,
coding process is not done for research and can cause prob- the investigator should record what has been done, how, and
lems with identifying study charts. Meeting with the coders why. Do not rely on memory. Subsequent publications should
can help the researcher best identify the desired subjects. describe the study methodology in sufficient detail to allow the
reader to, generally, reproduce the study themselves.
Abstracting the Medical Record
The process of reviewing the medical record and abstract- Strategies to Improve the Validity of
ing the information to be used for research is one of the last Retrospective Studies
steps in the flow of information in chart review studies. Although chart review studies are particularly prone to
However, before any chart is abstracted for a given research bias, not all such studies are poorly done. In fact, many are
project, the investigator must clearly identify the research excellent. No universally accepted criteria for a “well-conduct-
question and case definition (which patients you are going to ed” medical record abstraction process exist. However, there
include and exclude), as well as all other important variables are recommended strategies to enhance the validity, repro-
in the study. Even retrospective studies need inclusion and ducibility, and overall quality of data collected from clinical
exclusion criteria. Once there is a definition of the study records.2,3 These strategies include case selection, variable def-
cases and variables, they should not be changed during the initions, abstraction forms, training, monitoring, blinding,
study. If, after reviewing the initial charts, the definitions testing inter-rater agreement, and meetings. However, before
need to be altered, the study must re-start at the beginning applying these strategies or starting the data collection
again with new chart abstraction. Otherwise, study patients process, there must also be a prospective definition of the
would enter using two different criteria, which could intro- study question and the rules addressing the handling of prob-
duce significant bias. lematic data, even though it is a “retrospective” study. Post hoc
Keep accurate records about the charts that are available (aka “data dredging”) analyses are just as scientifically invalid
and those that are missing. Invariably, charts will be missing. for retrospective research as they are for prospective studies.
If less that 5% of all the charts, it can usually be ignored as a Together, these recommendations constitute the “10
source of bias, especially if the study is large. If 10% are miss- Commandments” for properly performing chart review stud-
ing, the results may only be 90% accurate, and an effort ies (Table 2).
should be made to determine why. This could cause signifi-
cant bias. Computerized logs or census data can assist in Case Selection
determining whether the missing charts have a common Specific inclusion and exclusion criteria must be identified
thread or are missing for a specific purpose. before any chart is selected for abstraction. Setting these crite-
No matter how diligent, there will always be some charts ria identifies the research study population. Keeping accurate
and individual data items that remain missing. If careful eval- counts of the subjects included and excluded is important and
uation reveals that the missing items do not represent a pat- should be described in the manuscript results section. The
tern that would introduce a significant bias, a decision must be case selection criteria should identify who, what, where,
made regarding how to handle the holes in the study database. when, and why a patient is included or excluded.
A number of potential approaches, ranging from averaging
only the available data to entirely dropping that chart or Definition of Variables
group, as appropriate, are possible. Regardless of the process All medical chart review studies should prospectively
used, it should be established in advance wherever possible identify the variables that need to be abstracted from the
and applied consistently throughout the study. Discuss this record before performing the study. Many of these variables
with an experienced researcher or a statistician. (eg, age, sex, death) are objective and straightforward.
There are several important issues relating to abstractors. However, others are subjective and prone to misinterpreta-
First, they must be qualified. Consistency and completeness are tion. For example, what constitutes a “good outcome”? A
the keys to accurately reviewing charts. When possible, the per- clear definition of all study variables is necessary for accu-
son actually doing the abstraction of the medical record should rate and consistent abstraction from the medical record.
not know the purpose of the research; this is called blinding and These definitions should be agreed on by all of the investi-
can be difficult to accomplish. It may involve lying to the gators in advance. The definitions must be taught to the
abstractors. However, without it, subjective abstraction deci- abstractors. A study dictionary, containing all the defini-

208 Air Medical Journal 26:5

 ing. Training medical students, for example, to properly
review a record could take over 5 hours. Novice abstractors
Table 2. The 10 Commandments for Performing Chart should then be given “practice” charts to review as part of the
Review Research initial training. When the abstractor becomes proficient with
the practice charts, they can begin abstracting the “real” charts
1. Prospectively (a priori) define the research question from the study. If the abstractor is still in doubt about a given
before any data collection. data item, they should identify the difficulty, copy the exact
2. Prospectively define the study case selection. statements, and review with the primary investigator. If rele-
• Detailed study inclusion and exclusion criteria vant, a rule or policy should be developed for dealing with
3. Prospectively define study variables and develop such data in the future to ensure consistency.
study policies.
• These may need to be expanded or modified as the Monitoring
study progresses. The principal investigator should monitor the performance
4. Ensure high-quality data abstractors. of the chart abstractors to identify any problems. Problems
• Qualifications and training with incomplete or poor interpretation of the chart, taking
5. Ensure consistent data recording. shortcuts, and misplacing charts are unfortunate but do occur.
• Such as with a detailed data form or computer program The abstractors should be held accountable for the quality of
6. “Blind” the data abstractors to the study purpose their work. When monitoring is performed, it should be
(whenever possible). described in the methods portion of the manuscript, so the
7. “QA” the data collection process through periodic reader can understand the diligence put into the project.
monitoring.
8. “QA” data processing through duplicate entry Blinding
techniques. As described earlier, the chart reviewers should be blinded
9. Be consistent throughout. (not be allowed to know) to the study question and hypothe-
• If there are significant changes in any major study vari- sis, or the research purpose. This is not always possible, but it
ables, start over from the beginning. is worth the effort. Nonblinded review of medical records can
10. Monitor the study progress. be very problematic. Subjective bias can be very hard to con-
• Hold periodic meetings. trol if there is not blinding. If this is not possible, an explana-
tion should be provided in the manuscript and included in the
discussion of study limitations.

tions, should be generated that serves as a reference during Testing Inter-rater Agreement
and after the study period. The subsequent manuscript In studies with more than one chart abstractor involved, it
should also include definitions of the key variables used in is important to determine whether the abstraction is being
the study. performed in a consistent manner. One approach to improve
the abstraction process is to generate an example chart dis-
Abstraction Forms playing the relevant information that can be used as a refer-
The chart review methods and the abstraction process ence by the abstractors. There are ways to test the inter-rater
must be standardized. It is routine to use template abstraction agreement, called reliability, by having both abstractors
forms. This guides data collection and ensures uniform abstract a sample of the same charts. Neither should have any
recording of data. The form should be designed to be easy to prior knowledge of the information obtained from the charts
use and have sufficient space to record all the information. It before abstraction (blinded review). The abstraction results
is helpful to design the form to collect data elements in the are then compared using a statistical measure of agreement,
order that they might be present in the medical record. commonly a kappa statistic or intraclass correlation coeffi-
cient. It would be optimal to study the inter-rater reliability
Training before starting the study, during the study, and at the end of
With the notable exception of medical records personnel, the study, while blinding the process from the abstractors
few of us are trained to review the medical record and abstract themselves. Another approach to maximize reliability is to use
data and information. Having experience with medical docu- a “dual” data entry technique. This involves having every chart
mentation does not guarantee an ability to accurately review abstracted by two reviewers. These results are compared. Any
and interpret the medical record. It is imperative that the chart differences are then adjudicated by a higher authority, usual-
reviewers be qualified to perform the job. They should be ly the study primary investigator. This is a highly scientific
trained by someone who is knowledgeable about the medical approach but can double the amount of work.
record, usually the principal investigator. The training should
cover all of the parts of the record that need to be reviewed, Meetings
identifying medical synonyms and colloquialisms that might The principal investigator needs stay informed during the
be used, and discouraging subjective interpretations of the chart abstraction phase of the project. It is best to remedy
information during abstraction. Training can be time consum- problems early before they result in significant bias. This

September-October 2007 209

requires routine scheduling of meetings with chart abstractors 23. Stueven HA, Tonsfeldt DJ, Hargarten KM, Olson DW. A dictated and transcribed
and study coordinators to resolve disputes and conflicts and medical record can be cost effective. J Am Record Assoc 1991;62:37-40.
24. Patrikas EO. Research review: use of keyless data entry in medical record depart-
to review coding rules. Initially, such meetings should be
ments. Top Health Inform Manage 1993;14:69-76.
often, then less frequent as needed. 25. Linn NA, Rubenstein RM, Bowler AE, Dixon JL. Improving the quality of emergency
department documentation using the voice-activated word processor: interim
Conclusion results. Proc Annu Symp Comput Appl Med Care 1992:772-776.
Medical records are informal collections of observations 26. Holbrook J, Aghababian R. A computerized audit of 15,009 emergency department
records. Ann Emerg Med 1990;19:139-144.
and impressions that contain both subjective and objective
27. Nazareth I, King M, Haines A, Rangel L, Myers S. Accuracy of diagnosis of psychosis
information obtained during the patient care process. They on general practice computer systems. Br Med J 1993;307:32-34.
are not created or designed for research but frequently are 28. Chua RV, Cordell WH, Emsting KL, Bock HC, Nyhuis AW. Accuracy of bar codes vs
used for that secondary purpose. Chart review studies are handwriting for recording trauma resuscitation events. Ann Emerg Med
more prone to bias and other errors than is true for prospec- 1993;22:1545-1550.
29. Payne TH, Goroll AH, Morgan M, Barnett GO. Conducting a matched-pairs historical
tive studies. Adhering to guidelines for proper chart review
cohort study with a computer-based ambulatory medical record system. Comput
technique ensures a more valid and reliable study and Biomed Res 1990;23:455-472.
improves the quality of medical record review research. 30. Ing RT, Baker SP, Eller JB, et al. Injury surveillance systems: strengths, weaknesses,
and issues workshop. Public Health Rep 1985;100:582-586.
References 31. Ribbeck BM, Runge JW, Thomason MH, Baker JW. Injury surveillance: a method for
1. Schwartz RJ, Jacobs LM. Analysis and comparison of research abstracts at MMS, 1987- recording E codes for injured emergency department patients. Ann Emerg Med
1990. Air Med J 1992;11:7-11. 1992;21:37-40.
2. Gilbert EH, Lowenstein SR, Koziol-Mclain J, Barta DC, Steiner J. Chart reviews in emer- 32. Langdorf MI, Strange G, Macneil P. Computerized tracking of emergency medicine
gency medicine research: Where are the methods? Ann Emerg Med 1996;27:305-308. resident clinical experience. Ann Emerg Med 1990;19:764-773.
3. Horwitz RI, Yu ED. Assessing the reliability of epidemiologic data obtained from med-
33. Roos LL, Mustard CA, Nicol JP, McLerran DF, Malenka DJ, Young TK,et al. Registries
ical records. J Chron Dis 1984;37:825-831.
and administrative data: organization and accuracy. Med Care 1993;31:201-212.
4. Bertelsen J. Who should abstract medical records? A study of accuracy and cost.
34. Hsia DC, Krushat WM, Fagan AB, Tebbutt JA, Kusserow RP. Accuracy of diagnostic
Evaluation Health Professions 1981;4:79-92.
coding for Medicare patients under the prospective-payment system. N Engl J Med
5. Leininger L, Harris R, Carey T. An analysis of the quality of medical record reviews in gen-
1988;318:352-355.
eral medicine journals [abstract]. Clin Res 1992;123:560.
35. Coding Clinic for ICD-9-CM. Chicago, IL: American Hospital Association Division of
6. Schwartz RJ, Boisoneau D, Jacobs LM, The quantity of cause-of-injury information
Quality Control Management. 1990;7:13.
documented on the medical record: an appeal of injury prevention. Acad Emerg
36. Jencks SF. Accuracy in recorded diagnoses. JAMA 1992;267:2238-2239.
Med 1995:2:98-103.
37. Moussa MA, Shafie MZ, Khogali MM, el-Sayed AM, Sugathan TN, Cherian G, et al.
7. Agius RM, Lee RJ, Symington IS, Riddle HF, Seaton A. An audit of occupational med-
Reliability of death certificate diagnoses. J Clin Epidemiol 1990;43:1285-1295.
icine consultation records. Occup Med 1994;44:151-157.
38. Lapidus G, Braddock M, Schwartz R, Banco L, Jacobs LM. Accuracy of fatal motorcy-
8. Austin K. The identification of mistakes in road accident records. Part 2: Casualty
cle-injury reporting on death certificates. Accid Anal Prev 1994;26:535-545.
variables. Accid Anal Prev 1995;27:277-282.
39. Bertelsen J. Who should abstract medical records? A study of accuracy and cost.
9. Bairstow BM, Burke V, Beilin IJ, Deutscher C. Inadequate recording of alcohol drink-
Evaluation Health Professions 1981;4:79-92.
ing, tobacco-smoking and discharge diagnosis in medical in-patients: failure to rec-
40. Dawson-Saunders B, Mast TA, Finch WT, Konrad HR, Folse JR. Content knowledge
ognize risks including drug interactions. Med Educ 1993;27:518-523.
and problem-solving skill in reviewing medical charts. Med Educ 1984;18:31-35.
10. Kennedy GT, Stem MP, Crawford MH. Miscoding of hospital discharges as acute
41. Yeoh C, Davies H. Clinical coding: completeness and accuracy when doctors take it
myocardial infarction: implications for surveillance programs aimed at elucidating
on [see comments]. BMJ 1993;306:972.
trends in coronary artery disease. Am J Cardiol 1984;53:l000-1002.
11. Martie TJ, Durant H, Sealy E. Pneumonia: the quality of medical records data. Med
Care 1987;25:20-24.
12. Hale WA, Delaney MJ, Cable T. Accuracy of patient recall and chart documentation Edward A. Panacek, MD, MPH, is professor of emergency medicine
of falls. J Am Board Fam Pract 1993;6:239-242. and clinical toxicology at the UC Davis Medical Center in Sacramento,
13. Casey R, Rieckhoff M, Beebe SA, Pinto-Martin J. Obstetric and perinatal events: the California. He can be reached at [email protected].
accuracy of maternal report. Clin Pediatr 1992;31:200-204.
14. Tilley BC, Barnes AB, Bergstralh E, , Labarthe D, Noller KL, Colton T, et al. A compari-
son of pregnancy history recall and medical records: implications for retrospective 1067-991X/$30.00
studies. Am J Epidemiol 1985;121:269-281. Copyright 2007 Air Medical Journal Associates
15. Pyles MK, Stolz HR, MacFarlane JW. The accuracy of mothers’ reports on birth and doi:10.1067/j.amj.2007.06.007
developmental data. Child Dev 1935;6:165-176.
16. Oates RK, Forrest D. Reliability of mothers’ reports of birth data. Aust Paediatr J
1984;20:185-186.
17. Preston SL, Briceland IL, LesarTS. Accuracy of penicillin allergy reporting. Am J Hosp
Pharm 1994;51:79-84.
18. Beers MH, Munekata M, Storrie M. The accuracy of medication histories in the hos-
pital medical records of elderly persons. J Am Geriatr Soc 1990;38:1183-1187.
19. Linet MS, Harlow SD, Mclaughlin JK, McCaffrey LD. A comparison of interview data
and medical records for previous medical conditions and surgery. J Clin Epidemiol
1989;42:l207-12l3.
20. Tsubono Y, Fukao A, Hisamichi S, Hosokawa T, Sugawara N. Accuracy of self-report
for stomach cancer screening. J Clin Epidemiol 1994;47:988-981.
21. Bondy ML, Strom SS, Colopy MW, Brown BW, Strong Le. Accuracy of family history
of cancer obtained through interviews with relatives of patients with childhood sar-
coma. J Clin Epidemiol 1994;47:89-96.
22. Wilton R, Pennisi AJ. Evaluating the accuracy of transcribed computer-stored immu-
nization data. Pediatrics 1994;94:902-906.

210 Air Medical Journal 26:5