AJSLP

Research Article

Comparative Validity of the American Speech-

Language-Hearing Association’s National
Outcomes Measurement System, Functional Oral
Intake Scale, and G-Codes to Mann Assessment
of Swallowing Ability Scores for Dysphagia
Sarah Dungan,a David Gregorio,a Theodore Abrahams,a Baylee Harrison,a Julia Abrahams,a
Destinee Brocato,a Catherine Davis,a Emily Espana,a Rachel Garcia,a Shanara Smith,a
Bryn Taylor,a Tiphanie Higgins,a Leon Daley,a and Giselle Carnabya

Background: The American Speech-Language-Hearing Mann Assessment of Swallowing Ability scores were
Association (ASHA) advocates using the National Outcomes compared for admission and discharge from subacute
Measurement System (NOMS) scales to assist speech- rehabilitation. Analysis included intraclass correlation for
language pathologists (SLPs) in the mandated assigning reliability, Spearman correlation for comparative validity,
of “functional limitation levels” and G-Code for Medicare and area under the receiver operating characteristic
Part B claims. Unfortunately, not all SLPs have access to curve to compare the validity and discriminatory utility
this tool, and it is unclear how other common outcome of measures in classifying dysphagia.
measurement scales relate to ASHA NOMS or G-Codes. Results: Strong correlations (> .6) were noted between all
To explore the utility of other scales in reporting Centers measures at baseline, particularly FOIS and NOMS coding
for Medicare & Medicaid Services G-Codes, we conducted (r = .919). At discharge, superior performance by the FOIS
a comparative validity study comparing ASHA NOMS (area under the receiver operating characteristic curve =
Dysphagia Scale, Functional Oral Intake Scale (FOIS), and 0.819) was demonstrated. Code missingness was higher
Mann Assessment of Swallowing Ability to G-Codes on a for the NOMS than the other scales.
sample of 105 patients who had stroke. Discussion: All 3 clinical dysphagia tools demonstrate
Method: Nine SLP student researchers trained to criterion acceptable validity in supporting G-Code designation to
on the NOMS and FOIS blindly and independently scored stroke cases. The FOIS demonstrated superior validity and
105 stroke cases with dysphagia de-identified from a utility across time points. The NOMS Dysphagia Scale
past study. Three graduate SLP clinicians independently was significantly affected by data missingness due to the
assigned G-Codes. Data from the student researchers and multiconstruct nature of the tool.

D
ysphagia (difficulty swallowing) is a prevalent (SLPs) are at the forefront of evaluating and treating swal-
problem facing many people in the United lowing problems. As part of that treatment, they utilize a
States. It is estimated that as many as one of number of tools or assessment methods to evaluate and
25 adults in the United States will encounter this difficulty measure the outcome of their interventions. In order to
(Bhattacharyya, 2014). Speech-language pathologists be reimbursed for services provided, SLPs must submit
reporting to the Centers for Medicare & Medicaid Services
(CMS) that includes a “functional limitation level” and
a
G-Code for claims for Medicare Part B.
School of Communication Sciences and Disorders, University of These codes are based upon a 7-point complexity or
Central Florida, Orlando
severity modifier that describes a client’s performance abil-
Correspondence to Sarah Dungan: [email protected]
ities in any activity treated (e.g., memory, swallowing,
Editor-in-Chief: Julie Barkmeier-Kraemer cognition). The CMS uses this information to help track
Editor: Debra Suiter
Received April 10, 2018
Revision received June 1, 2018 Disclosure: Giselle Carnaby is the author of the Mann Assessment of Swallowing
Accepted July 2, 2018 Ability and coauthor of the Functional Oral Intake Scale for which she receives no
https://doi.org/10.1044/2018_AJSLP-18-0072 remuneration.

424 American Journal of Speech-Language Pathology • Vol. 28 • 424–429 • May 2019 • Copyright © 2019 American Speech-Language-Hearing Association
patient achievement of goals over time as a result of thera- proposed for G-Code application (Mann, 2002). This
peutic intervention (Resnik, 2013). clinical assessment measure evaluates the swallowing
To help SLPs meet this mandated reporting require- physiology of a patient at bedside. It also has estab-
ment, the American Speech-Language-Hearing Association lished and published validity and reliability (Mann, 2002).
(ASHA) has advocated the use of the National Outcomes Nevertheless, although it has demonstrated strong corre-
Measurement System (NOMS; Mullen, 2004). ASHA’s lation to the FOIS score in patients, its relationship to
NOMS is a three-part national data collection system. It ASHA NOMS and G-Codes remains unknown (Crary
involves an online data collection and reporting system et al., 2005).
developed by a self-reported professional survey, to demon- The health care reimbursement environment increas-
strate the worth of SLP services provided to clients (Mullen, ingly requires providers to consider quality and patient
2004). Within the NOMS system is a series of functional progress in reporting. Similarly, it calls for outcome mea-
communication measures (FCMs) that are disorder specific sures that demonstrate strong psychometric properties to
and are used to describe the client’s function at the onset be utilized (Rao, 2015). In an attempt to meet these calls
of services and at discharge. These 7-point scales are scored and inform the options for reporting patient outcome, this
by the treating SLP and utilized to reflect the amount of study evaluated the utility of commonly used dysphagia
change in performance attributed to the SLP intervention. scales in applying G-Codes and compared those tools with
The NOMS FCMs are similar to but not the same as, the the ASHA NOMS Dysphagia Scale.
mandated G-Code modifiers. Registration (and subscrip- The specific purpose of this study was to evaluate
tion) in the ASHA NOMS system gives SLPs submitting the utility of commonly used dysphagia scales for G-Code
to the registry access to a built-in G-Code converter to as- identification and to evaluate if these dysphagia scales,
sist with coding (Mullen, 2004). However, there has been a namely, FOIS, MASA, and ASHA NOMS Dysphagia
lot of confusion regarding the relationship between ASHA Scale, could be used to support G-Code determination.
NOMS and the reporting of the CMS G-Codes (Sheffler,
2018). In 2014, ASHA and Rehab Care provided a web-
based 2-hr self-study NOMS training course to certified SLPs Method
or clinical fellows, allowing them to become NOMS regis-
tered users and instructing them in the NOMS coding process. Participants
Training, however, is not available to students. Further, to A sample of 105 de-identified stroke cases with dys-
date, no direct training is available to clinicians to assist in phagia from a previous randomized controlled study of
G-Code determination. Research supporting the psycho- dysphagia poststroke were used for this study. All cases
metric validity and comparability of the two scales has had been evaluated for dysphagia at the time of admission
not been published. Further, if an organization has not and discharge from the rehabilitation facility by the treating
completed a contractual agreement with ASHA to use the SLP using the MASA. In addition, videofluoroscopic swal-
NOMS, the 7-point ASHA FCM’s system for G-Code lowing studies confirmed dysphagia identified by MASA
reporting is unavailable. in each case. For this study, only the daily detailed case
These points have created some confusion for clini- notes from the speech pathology service were provided for
cians seeking assistance in G-Coding, as not all treatment the purposes of coding.
sites or clinics participate in the NOMS registry. Moreover,
not all SLPs (who must submit G-Codes) wish to voluntar-
ily participate or know if their facility is registered with Procedure
the ASHA NOMS system. Conversely, CMS does not re- Training Phase
quire that ASHA NOMS scales be utilized to determine Initially, nine student research assistants were educated
the G-Code applied and does not require that the clinician in stroke dysphagia and trained in FOIS and NOMS ad-
identify the assessment or outcome tool utilized to deter- ministration and coding procedures across seven 1-hr
mine the code (Sheffler, 2018). training sessions conducted with experienced dysphagia
For sites or individuals who do not participate in specialists. Thirteen test stroke cases were used for blinded
the NOMS registry, other outcome measurement tools are coding practice within the training sessions. All test cases
available; one example is the Functional Oral Intake Scale had been extensively evaluated and coded by the specialist
(FOIS) for dysphagia (Crary, Mann, & Groher, 2005). clinicians trained and certified in NOMS coding who had
This 7-point scale documents the functional diet level of a access to all evaluations of the cases. Cases used for stu-
patient, that is, the diet consumed at any point in time. dent training were excluded from the final study sample
This tool is used to monitor progress in functional swal- analysis. Researchers assigned codes from the detailed
lowing gains from therapy over time. It has established daily case notes on each subject. Case notes utilized for
validity and reliability and is free to the public. However, study coding included medical record SOAP (subjective,
its correlation to either ASHA NOMS or G-Codes is un- objective, assessment, and plan) notes documented by
known. Correspondingly, another commonly utilized clini- the attending SLP at the time of the patient’s admission
cal evaluation measure of swallowing performance, Mann and discharge. Reliability of the coding was evaluated suc-
Assessment of Swallowing Ability (MASA), has been cessively. Training concluded when reliability in coding

Dungan et al.: Comparative G-Code Study 425

across raters on the three tools was reached (r > .80). Post- the ROC (AUROC) was compared using the method from
training reliability at the final evaluation point was deter- Hanley and McNeil (1982; IBM, 2017). An ROC curve is
mined to be as follows: intraclass correlation coefficient a measurement of the performance or diagnostic accuracy of
(ICC) = .933 for the FOIS and ICC = .931 for the NOMS. a test (IBM, 2017). The curve displays the true-positive rate
(sensitivity) plotted against the false-positive rate (100-
Coding Phase specificity). Tests that perform well demonstrate a curve
Following conclusion of the training phase, the full that is close to the upper left-hand corner of the graph, indi-
coding procedure was initiated. The study sample included cating a higher accuracy of the test. Tests that do not add
105 dysphagic stroke cases randomly computer allocated diagnostic value fall along the diagonal line (Hanley &
to three groups (n = 35 cases) for blinded rating by the nine McNeil, 1982). Percent missingness (defined as the per-
research assistants. The stroke cases used covered a range centage of times a case exemplar could not be assigned
of commonly seen stroke diagnoses and levels of severity either a G-Code, FOIS code, or NOMS code by the re-
(see Table 1). Evaluators were independently assigned a searcher) was explored to evaluate the utility of applying
group of 35 cases to assess using each tool. This arrangement each method. SPSS (IBM, 2017) and MedCalc software
was then rotated until all evaluators had reviewed and (Schoonjans, 1993) were used to analyze ROC curves
assigned grades to each of the cases, resulting in N = 945 for both admission and discharge and to conduct concur-
independent codes. In an identical training protocol, three rent validity. Due to the level of missingness on the NOMS
graduate student judges were trained in G-Code applica- coding, a Spearman correlation coefficient r was utilized
tion using the same test cases. Subsequently, in concurrently for the correlation analysis (IBM, 2017).
held independent coding and then consensus panels, three
final-semester graduate SLP students assigned G-Codes to
the same cases from the detailed case notes. Reliability among Results
the three judges for the G-Code assignment was ICC = 0.94 Sample Characteristics
(2, 3).
The stroke cases randomly computer selected were,
on average, male, with a mean age of 73 years, and had
Statistical Analysis suffered an ischemic stroke. More than half presented with
ICC was used to measure concordance of raters across a first-ever stroke (74.5%) with significant stroke disability.
coding for the three tools and for the G-Code ratings (Inter- Severe dysphagia was present in 42% of the cases, and no
national Business Machines [IBM], 2017). Comparative dysphagia was identified in 7% of the cases.
validity analysis was conducted using the coded data from
each independent rater. Cross-validation of the three tools
was performed using the MASA as the referent. Receiver
Convergent Validity
operating characteristic (ROC) curves were constructed to Comparison of G-Codes with FOIS and MASA
review each test’s ability to detect and model dysphagia by scoring for both time points is displayed in Table 2.
MASA at each time point (IBM, 2017). The area under
Baseline
Table 1. Sample cases’ baseline characteristics. Convergent validity analysis revealed strong correla-
tions between all measures at baseline to the dysphagia
Sample Characteristics Baseline referent (MASA) with superior performance by G-Codes
alone (r = .701; see Figure 1 and Table 3). A strong signifi-
Age, M (SD) 73 (10.3)
Gender (M:F) 59:46 cant correlation was revealed between both FOIS and
Stroke diagnosis (ischemic:hemorrhagic) 94:11 NOMS assessments. The FOIS and NOMS revealed
Stroke type,a n (%) AUROC coefficients of 0.808 and 0.849, respectively, in-
Total anterior circulation stroke 40 (38) dicating a similar utility in classifying dysphagia. G-Code
Partial anterior circulation syndrome 38 (36)
Lacunar syndrome 17 (16) demonstrated the greatest utility, among the three tools, in
Posterior circulation syndrome 10 (9.5) modeling dysphagia at admission (AUROC = 0.952). Con-
Stroke disability (baseline Rankin), M (SD) 3.8 (1.05) cordant validity of the FOIS to the MASA was calculated
Glasgow Coma Scale, M (SD) 11.96 (2.9) at r = .701. Percent missingness for NOMS coding was
Dysphagia severity (MASA), M (SD) 133 (38.8)
Mild 23 (21%) identified as 10.9% at baseline compared with 0% for the
Moderate 35 (33%) FOIS.
Severe 44 (42%)
Aspiration on VFSS, n (%) 52 (49.5)
Discharge
Note. M = male; F = female; MASA = Mann Assessment of
Swallowing Ability; VFSS = videofluoroscopic swallowing study. A strong and significant correlation was again revealed
a
Bamford classification (Pittock et al., 2003). between both FOIS and NOMS assessments at discharge.
Convergent validity analysis for measures at discharge

426 American Journal of Speech-Language Pathology • Vol. 28 • 424–429 • May 2019

 Table 2. Descriptive comparison of Functional Oral Intake Scale (FOIS) and Mann Assessment of
Swallowing Ability (MASA) scores to G-Code.

Admission G-Code, Admission FOIS score Admission MASA,

N = 105 (median), N = 105 M (SD), N = 105

1 1 101.36 (35.3)
2 1.66 133.8 (12.68)
3 3.33 153.7 (23.8)
4 4 161.4 (12.8)
5 4.5 157.6 (30.1)
6 5.3 172.2 (6.62)
7 6.5 184.4 (6.1)
Discharge G-Code, Discharge FOIS score Discharge MASA,
N = 105 (median), N = 96 M (SD), N = 105
1 1 90.1 (25.7)
2 1.5 134.9 (28.4)
3 3 149.2 (22.7)
4 4.5 155.1 (5.3)
5 5 171.2 (10.7)
6 6 176.9 ( 8.3)
7 6.3 186.2 (6.7)

demonstrated a stronger agreement across measures with Discussion

superior performance by the FOIS (see Figure 2 and Table 4).
The FOIS and NOMS revealed AUROC coefficients of This study has identified moderate to strong intercor-
0.819 and 0.764, respectively, indicating a stronger predic- relations between the FOIS, MASA, NOMS, and G-Codes
tive performance of the FOIS when compared with the applied to a sample set of 105 dysphagic stroke cases at
dysphagia referent MASA. Concordant validity of the two time points in the rehabilitation process. Collectively,
FOIS to the discharge MASA was calculated at r = .846. these data suggest that both the MASA and FOIS, along
Percent missingness for NOMS coding at discharge was with the NOMS, can be utilized to assist with G-Code
25.5%, compared with 8.6% for the FOIS. application. Further, the NOMS and FOIS appear statis-
tically interchangeable and yield comparable utility in
assigning G-Codes. However, the FOIS did perform supe-
riorly in assigning G-Codes at discharge. As such, the ability
Figure 1. Receiver operating characteristic curve comparing Functional of the FOIS to better reflect functional feeding gains over
Oral Intake Scale (FOIS), National Outcomes Measurement System time may support its use over the NOMS when evaluating
(NOMS), and G-Codes for dysphagia at baseline. AUROC = area
under the receiver operating characteristic curve; sig = significance; patient recovery and outcomes.
FOISBL = FOIS at baseline/admission; NOMSBL = NOMS at baseline/ G-Code application alone by the graduate clinicians
admission; Gcode1 = G-Code at baseline/admission. was superior in reflecting baseline dysphagia performance
when compared with the MASA. This finding was curious
to us, and in reviewing the case details, this may have re-
sulted from the larger proportion of severely dysphagic pa-
tients immediately following the stroke. The presence of a
swallow complaint plus tube feeding and/or a nil per oral
on the case note frequently leads to an immediate assign-
ment of G-Code/FMC 1 (e.g., 100% impaired, limited, or

Table 3. Concordant validity at admission.

Measures NOMS FOIS G-Codes

MASA .623* .647* .701*

NOMS .919* .858*
FOIS .845*

Note. NOMS = National Outcomes Measurement System; FOIS =

Functional Oral Intake Scale; MASA = Mann Assessment of
Swallowing Ability.
*Significant at the .0001 level.

Dungan et al.: Comparative G-Code Study 427

Figure 2. Receiver operating characteristic curve comparing and ambiguous nature of the NOMS descriptions. The
Functional Oral Intake Scale (FOIS), National Outcomes Measurement
System (NOMS), and G-Codes for dysphagia at discharge. AUROC =
NOMS contains several divergent constructs at each level;
area under the receiver operating characteristic curve; sig = significance; for example, constructs of alternate/feeding method, safety
FOISdch = FOIS at discharge; NOMSdch = NOMS at discharge; of feeding, nutrition level, supervision level, curing, and
Gcodedch = G-Code at discharge. compensatory behaviors may all be present in a single des-
ignation. This multiconstruct nature is also complicated by
the use of undefined subjective language terms such as
“consistent,” “occasional,” “rarely,” and “maximal.” The
use of subjective delineators embedded within the tool
laces the grading with the potential error due to opinion,
emotion, and variable interpretation, making it fraught
with emotion and bias and thus limiting fact-based mea-
surable decision making.
It appears the NOMS may benefit from more specifi-
cally defined terms to enhance its utility for practice. For
example, the NOMS considers swallowing with consistent
use of a cueing hierarchy (maximum–rarely); however,
neither consistency nor the method of cueing is defined.
At Level 3, the NOMS requires the SLP to calculate 50% of
nutrition and hydration. Further, specific food avoidance
is only addressed at a single level. Moreover, the NOMS
refers to occasional and consistent self-cueing; however,
the manner in which the patient is self-cueing is not speci-
fied. Clearly, the ambiguities in the application of this tool
can result in a high proportion of percent missingness,
reducing the validity of the tool and decreasing its overall
utility.
In contrast, the FOIS tool is a concise, user-friendly
restricted), without requiring further detailed clinical evalu-
assessment that is available to SLPs. It showed evidence of
ation data. Consequently, significantly impaired swallow-
a comparable performance with the NOMS and a stronger
ing status at baseline was, in the majority of cases, more
correlation with G-Codes associated with outcomes. The
easily assigned via direct G-Code designation, without assis-
use of this simpler tool may support G-Code application
tance from dysphagia-specific evaluation tools. It is impor-
and act to lessen the financial burden of SLPs and facilities
tant to note, however, that clinical assessment of dysphagia
in which they practice.
also provides extensive information and direction for inter-
The strengths of this study lie in its rigorous training
vention planning and/or the need for additional assessment,
of the research coding team and the blinded review of
which is not captured in a simple G-Code. We therefore
confirmed dysphagic stroke cases. This process ensured
believe that this finding should not be interpreted to suggest
that the application of the tools in the study was standard-
the G-Code as an alternative to performing a comprehensive
ized and consistent across raters. An additional strength
clinical assessment.
is that all dysphagia evaluations (MASA) had been com-
Another finding of note from our study is the level
pleted by independent clinicians and confirmed by video-
of missingness associated with the NOMS coding system.
fluoroscopic swallowing study during patient admission at
Applying the NOMS resulted in 11%–25% data loss across
both time points. Further, the stroke cases utilized covered
time points. It appears that the NOMS codes were overall
a range of commonly seen stroke diagnoses and levels
more difficult to assign by the evaluators than the other
of stroke and dysphagia severity (see Table 1) to reflect
tools. One possible reason for this is the multiconstruct
the clinical distribution of dysphagia for this population.
A potential weakness of the study involves the use of under-
Table 4. Concordant validity at discharge.
graduate students as evaluators for the FOIS and NOMS
coding. The limited clinical experience of these students
Measures NOMS FOIS G-Codes
may not have fully represented the SLP clinician level of
MASA .832* .846* .707* ability in using the tools. However, all students were edu-
NOMS .950* .645* cated in stroke dysphagia and trained in the application
FOIS .689* of the tools to a level of consensus with clinical experts
Note. NOMS = National Outcomes Measurement System; FOIS =
and strong reliability across ratings. We therefore believe
Functional Oral Intake Scale; MASA = Mann Assessment of the data reflect the true use and accuracy of the tools for
Swallowing Ability. this application.
*Significant at the .0001 level. In summary, all three clinical dysphagia tools reviewed
demonstrated acceptable utility in supporting G-Code

428 American Journal of Speech-Language Pathology • Vol. 28 • 424–429 • May 2019

designation to stroke cases. The application of the FOIS Mann, G. (2002). MASA: The Mann Assessment of Swallowing
was superior to the NOMS for mapping discharge G-codes. Ability [Measurement instrument]. Clifton Park, NY: Cengage
Ambiguities in the language, construction, and application Learning.
Mullen, R. (2004). Evidence for whom? ASHA’s National Outcomes
of the NOMS Dysphagia Scale resulted in a higher propor-
Measurement System. Journal of Communication Disorders,
tion of missingness in coding and reduced the comparative 37(5), 413–417.
validity of the tool in this study. Pittock, S. J., Meldrum, D., Hardiman, O., Thornton, J., Brennan, P.,
& Moroney, J. T. (2003). The Oxfordshire community stroke
project classification: Correlation with imaging, associated
References complications, and prediction of outcome in acute ischemic
Bhattacharyya, N. (2014). The prevalence of dysphagia among stroke. Journal of Stroke and Cerebrovascular Diseases, 12(1),
adults in the United States. Otolaryngology–Head & Neck 1–7.
Surgery, 151(5), 765–769. Rao, P. R. (2015). Outcomes and quality: Key characteristics of a
Crary, M. A., Mann, G. D. C., & Groher, M. E. (2005). Initial successful SLKP value journey. SIG 2 Perspectives on Neuro-
psychometric assessment of a functional oral intake scale for physiology and Neurogenic Speech and Language Disorders,
dysphagia in stroke patients. Archives of Physical Medicine 25(3), 94–106.
and Rehabilitation, 86(8), 1516–1520. Resnik, L. (2013). Medicare mandate for claims-based functional
Hanley, J. A., & McNeil, B. J. (1982). The meaning and use of data collection: An opportunity to advance care, or a regula-
the area under a receiver operating characteristic (ROC) curve. tory burden? Physical Therapy, 93(5), 587–588.
Radiology, 143, 29–36. Schoonjans, F. (1993). MedCalc software (Version 25) [Computer
International Business Machines. (2017). IBM SPSS Statistics for software]. Ostend, Belgium.
Windows (Version 25.0) [Computer software]. Armonk, NY: Sheffler, K. (2018). G-Codes versus ASHA NOMS. Retrieved from
IBM Corp. http://www.swallowstudy.com/asha-noms/

Dungan et al.: Comparative G-Code Study 429

Copyright of American Journal of Speech-Language Pathology is the property of American
Speech-Language-Hearing Association and its content may not be copied or emailed to
multiple sites or posted to a listserv without the copyright holder's express written permission.
However, users may print, download, or email articles for individual use.