Professional Psychology: Research and Practice Copyright 2007 by the American Psychological Association

2007, Vol. 38, No. 5, 474 – 478 0735-7028/07/$12.00 DOI: 10.1037/0735-7028.38.5.474

Challenges to the Assessment of Competence and Competencies

James W. Lichtenberg Sanford M. Portnoy

University of Kansas Massachusetts School of Professional Psychology

Muriel J. Bebeau Irene W. Leigh

University of Minnesota School of Dentistry Gallaudet University

Paul D. Nelson Nancy J. Rubin

American Psychological Association University of Alabama School of Medicine—Tuscaloosa
Campus

I. Leon Smith Nadine J. Kaslow

Professional Examination Service Emory University School of Medicine

Challenges to the assessment of competence and competencies in professional psychology are discussed
in this article. These include difficulties in defining competencies in precise and measurable terms;
reaching agreement within the profession about the key elements of each competence domain; estab-
lishing an armamentarium of tools for assessing all components of competence, including the knowledge
base, skills, and attitudes (and their integration); determining appropriate agreed-upon minimal levels of
competence for individuals at different levels of professional development and when “competence
problems” exist for individuals; assuring the fidelity of competency assessments; and establishing
mechanisms for providing effective evaluative feedback and remediation. But even if these challenges

JAMES W. LICHTENBERG received his PhD in counseling psychology from NANCY J. RUBIN received her PsyD in clinical psychology from the Uni-
the University of Minnesota. He is a professor of counseling psychology versity of Denver. She is an associate professor and the director of
and the associate dean for graduate programs and research at the University psychology clinical services, teaching, and research in the Department of
of Kansas. His areas of professional interest and research include social Psychiatry and Behavioral Medicine, the University of Alabama School of
interaction processes and dynamics, legal and ethical issues in counseling Medicine—Tuscaloosa Campus. Her research interests include medical
and psychotherapy, and clinical training. psychology/behavioral medicine, the physician/patient relationship, profes-
SANFORD M. PORTNOY received his PhD in clinical psychology from the sional issues, and eye movement desensitization and reprocessing
University of Massachusetts. He is on the faculty of the Massachusetts (EMDR).
School of Professional Psychology, where he serves as director of the I. LEON SMITH received his PhD in educational psychology from the State
Center for the Study of Psychology and Divorce, and is a member of University of New York at Buffalo. He is president and CEO of Profes-
Needham Psychotherapy Associates in Needham, Massachusetts, and of sional Examination Service, a 501(c)(3) organization whose mission is to
Portnoy Associates in Newton, Massachusetts. His professional and re- provide assessment and consulting services to professions in support of
search interests include the psychology of divorce and the effects of the
sound certification and licensure initiatives. His areas of interest and
legal divorce process on families, couples therapy, and teaching legal
research include reliability and validity issues impacting assessments used
professionals the skills to relate more effectively to their clients.
in certification and licensure testing, as well as public policy considerations
MURIEL J. BEBEAU received her PhD in educational psychology from
in credentialing.
Arizona State University. She is a professor in the School of Dentistry at
NADINE J. KASLOW received her PhD in clinical psychology from the
the University of Minnesota, faculty associate in the university’s Center for
University of Houston. She is a professor and chief psychologist in the
Bioethics, and director of the Center for the Study of Ethical Development.
Her scholarly work integrates the psychology of morality with ethics and Department of Psychiatry and Behavioral Sciences at Emory University
dentistry to design and validate assessment strategies and teaching methods School of Medicine. Her research and clinical practice focus on
to promote professional ethical development. competency-based education, training, and supervision of interns and post-
IRENE W. LEIGH received her PhD in clinical psychology from New York doctoral fellows; family violence; suicidal behavior across the life span;
University. She is a professor in the clinical psychology doctoral program and family systems medicine.
at Gallaudet University in Washington, DC. Her presentations, research, THE AUTHORS of this article are members of the APA Task Force on the
and publications have focused on deaf people and issues related to identity, Assessment of Competence in Professional Psychology. James W. Licht-
multiculturalism, parenting, attachment, depression, and cochlear implants. enberg took primary responsibility for this article. Nadine J. Kaslow is the
PAUL D. NELSON received his PhD from the University of Chicago. He Task Force Chair.
recently retired as the deputy director of education and director of graduate CORRESPONDENCE CONCERNING THIS ARTICLE should be addressed to James
and postdoctoral education and training for the Education Directorate of W. Lichtenberg, University of Kansas, Department of Psychology and
the American Psychological Association. His focus has been on graduate Research in Education, 1122 West Campus Road, J. R. Pearson Hall, Room
and postgraduate education and training. 214, Lawrence, KS 66045. E-mail: [email protected]

474
 SPECIAL SECTION: CHALLENGES TO THE ASSESSMENT OF COMPETENCE 475

can be met, perhaps the single biggest obstacle would be convincing those who are skeptical of the value
of the culture shift toward the implementation of comprehensive competence assessments across the
professional life span.

Keywords: competence, competency assessment, assessment challenges, education and training

There has been considerable acknowledgement in professional should be measured at various stages of professional development,
psychology, as well as in other fields, that although the assessment it is not clear that there is consensus within the profession regard-
of competence and competencies is essential, it is a process fraught ing the core overarching foundational and functional competencies
with many difficulties (e.g., Peterson, 2004; Roberts, Borden, reflecting these domains. Further, there is even less agreement
Christiansen, & Lopez, 2005). It is not the intent of this article to across professional psychology with regard to the key elements of
argue for a culture shift toward the assessment of competence in each competency domain. Contributing to this lack of agreement
professional psychology; that is done elsewhere in this collection are presumed fundamental differences among the applied special-
of articles. Rather, in this article, we briefly outline the key ties in professional psychology and among the various schools of
challenges to the assessment of competence and competencies practice. For example, clinical psychology, counseling psychol-
should that be the direction of the field. These challenges are not ogy, and school psychology, although overlapping in many ways,
necessarily unique to professional psychology, although some are; are (or claim to be) distinctive in their core conceptualizations of
other professions struggle with the same or similar issues. how they approach professional practice. This issue of distinctive-
Our intent in this article is, then, to summarize the nature and ness is of fundamental concern in terms of each specialty’s con-
breadth of these challenges. We do so, moving from conceptual to tinued recognition by the APA Commission on the Recognition of
more practical challenges in the assessment of competence and Specialties and Proficiencies in Professional Psychology (www.
competencies. Although we present challenges, we do not proffer apa.org/crsppp) as a unique specialty within professional psychol-
solutions. It is our perspective that the scope of these challenges is ogy. For this reason, it is not surprising that there are differences
such that addressing them is likely to involve systemic change and among the practice specialties in their perspectives regarding the
the breadth of the profession— educators, supervisors, employers, specific knowledge, skills, and attitudes associated with each com-
and credentialers—working together cooperatively toward shared petency domain. In addition, it may be noted that professional
solutions. psychologists who espouse different theoretical orientations or
different models of training often have marked disagreements
Defining Competencies regarding what specific knowledge and skills are required for a
person to evidence competence within each competency domain.
Although there seems to be no dearth of proposed areas of
In a related vein, and as evidenced by proposals to change the
competence pertinent to the practice of professional psychology,
timing or placement of the internship experience and the need for
professional psychology nevertheless has had problems defining
a supervised postdoctoral prior to licensure, professional psychol-
its competencies in terms precise enough to be measurable, at least
ogy lacks agreement on a seamless, integrated sequence of training
without reducing them to overly simplified, uni-dimensional tasks.
from academic program to internship, to postdoctoral fellowship,
More generally, the field has until recently evidenced disarray in
its identification of the competencies ubiquitous to professional to licensing for independent practice with agreed-upon bench-
practice. Although a degree of consensus among a number of the marks of competence among the various parties/stakeholder
training councils in professional psychology appears to be forming groups.
around “the cube model” of competencies that emerged from the Although it is appropriate that assessors at various developmen-
recent conference on competencies (Rodolfa et al., 2005), it is tal stages evaluate or assess different things, there currently is no
noted that the competency domains described in the model have coherence within the profession regarding what should be mas-
yet to be formally endorsed or accepted across professional psy- tered at each stage of professional preparation, and it is not clear
chology. Indeed, outside of the training community, there has been who does or should set these benchmarks (e.g., doctoral programs
little acknowledgement of or attention paid to the competency and internships may differentially evaluate when a person is
domains outlined in this model. But even if one were to grant the “ready for internship”). Further, it would seem that there is (or at
competency domains specified by the cube, without clearly delin- least has been) insufficient communication among programs, in-
eated descriptions of the competencies within each competency ternships, postdoctoral experiences, and licensing boards regarding
domain, it would be difficult to develop reliable and valid assess- the establishment of defensible professional competency bench-
ment methods to determine the adequacy of people’s performance marks. Indeed, it was in part concerns about students’ readiness for
within each domain. internship and then for independent practice that led to the Com-
petencies Conference 2002, which was sponsored by the Associ-
ation of Psychology Postdoctoral and Internship Centers and held
Lack of Agreement in the Profession About the Key
in Scottsdale, Arizona, November 7–9, 2002. This conference had
Elements of Each Competence Domain
as its purpose the bringing together of representatives from various
Even if one were to assume a general acceptance of the cube education, training, practice, public interest, research, credential-
model of competencies (Rodolfa et al., 2005), reflecting the broad ing, and regulatory constituency groups to work toward identifying
domains within which competence should be demonstrated and core and specialized competencies in professional psychology, the
476 LICHTENBERG ET AL.

formulation of developmental and integrated models of competen- reliability and validity of measurements of those complex profes-
cies for the training, and the development of strategies for the sional functions that are the true indicators of competence within
evaluation of competencies. The conference was intended to serve professional psychology.
as a catalyst for continued collaborative efforts related to the
identification, training, and evaluation of competencies—fostering Fidelity
links between various constituency groups to better train profes-
sional psychologists and to enhance the competence of profes- Many measures lack a high degree of fidelity. Commonly used
sional psychologists so that professional psychology might better paper and pencil assessments are not “experience near.” These
serve and protect the public. examinations often are administered in isolation, and there are
It is relevant to note, however, that subsequent to that confer- many problems with using single measures for evaluating complex
ence, a developmental model for defining and measuring compe- abilities, in part because these tools often lack depth. Paper and
tence in professional psychology had been drafted by the Assess- pencil examinations often are in multiple-choice format, and
ment of Competency Benchmarks Work Group convened by the multiple-choice examinations often do not do an adequate job of
APA’s Board of Educational Affairs. However, this document, assessing skills, attitudes, or the complex integration of relevant
which was crafted in collaboration with the chairs of various knowledge, skills, and attitudes. Assessment techniques typically
professional psychology training councils (Council of Chairs of used do not involve the observation of the persons being evaluated
Training Councils), does not represent policy of the APA. And in simulated or actual situations, do not include feedback from
pertinent to the challenge addressed in this section, this effort has patients/clients or peers, and are not indicative of clinical out-
not involved parties outside of the training community (e.g., cre- comes.
dentialing organizations) who reasonably have an interest in such Although high fidelity methods would strengthen professional
matters. psychology’s ability to accurately assess competencies, they also
Without developmentally informed, clear, and measurable raise concerns and complications. High fidelity measures are, as a
learning objectives for each competency domain and associated rule, more time intensive, labor intensive, and costly. Further, for
benchmarks for performance at each stage of professional devel- some constructs, the higher the fidelity, the lower the generaliz-
opment, the effective measurement of competencies— high fidel- ability of the assessments from one setting to another setting and
ity measurement that is reliable and valid and relevant to actual from one stage of professional development to the next.
practice—is unlikely to be achieved. Further, the current frequent
lack of coherence and continuity between the assessments used in Assessing Competence Problems
graduate schools to meet graduation requirements and those used
for initial licensure contributes to one-shot testing of clinical Assessing competence means (or at least implies) that subtle to
competence, a situation that credentialing bodies find undesirable blatant incompetence would be found on some occasions, and that
(Chambers, Dugoni, & Paisley, 2004). inconsistent competence across domains would be ascertained in
other instances (Overholser & Fine, 1990). Unfortunately, at
present, we lack an agreed-upon nomenclature to describe individ-
Limitations in Assessment Armamentarium for Assessing
uals whose behavior does not meet minimum standards of profes-
All Components of Competence sional competence (Elman & Forrest, 2006). Further, it remains
Knowledge, Skills, Attitudes, and Their Integration necessary for the profession to develop and implement systems
within the profession that enable adequate and effective responses
Within professional psychology, our abilities to assess across to competence problems. These systems must respect competence
the areas of knowledge, skills, and attitudes are not equal. As a standards, protect the public, and whenever possible, help the
profession, we seem better able to assess knowledge than skills or individual being evaluated. When competence problems are as-
attitudes, more effective at evaluating skills than attitudes, and sessed as being significant or an ongoing evaluation reveals they
generally to have few established methods for assessing critical are not responsive to appropriate remediation, appropriate actions
professional attitudes. It may be that we are not actually that must be taken by the profession to protect the public (Forrest,
effective at measuring knowledge either, but nevertheless most of Elman, Gizara, & Vacha-Haase, 1999; Vacha-Haase, Davenport,
our tools are designed to do so. Although professional psychology & Kerewsky, 2004). In this way, the widespread adoption of the
does have tools for evaluating knowledge and skills (e.g., super- competence model and its assessment would create the need to
visors’ reports, examinations, passage of skills courses), it is develop new methodologies for dealing with outcomes. Although
generally held that these often lack reliability, ecological validity, this would add yet another challenge, it may also be of service to
and fidelity to practice. ethics committees, licensing boards, and the public, and it may add
More importantly, psychology does not currently have methods a new level of clarity to professional standards in psychology.
to readily or reliably assess the integration of knowledge, skills,
and attitudes in the performance of professional functions that Implementation
comprise competence (e.g., professional judgment, scientific-
mindedness, relationship skills, team work, internalized ethical As is often the case with system or cultural change, a discipline-
orientation, reflective practice/self-awareness, openness to learn- wide shift toward a competency model is not easily accomplished.
ing, and commitment to professional growth). Yet it is this inte- One challenge is to convince those who are skeptical of the value
gration that reflects the construct of competence (see Kaslow et al., of the culture shift toward more comprehensive assessments across
2007). In addition, there is little available data regarding the the professional life span. Skeptics often question whether such
 SPECIAL SECTION: CHALLENGES TO THE ASSESSMENT OF COMPETENCE 477

assessment(s) would lead to better training for competencies and preparation (training program alternatives) may be valued by the
higher levels of competence overall, whether the assessment of profession and asserted by training institutions. Thus, there may be
competencies would reduce incompetence in the field, and whether a conflict between what training programs value and teach and
the measurement of competencies is a better way to assess pro- what credentialing bodies believe to be relevant.
fessional performance in the long run than what is currently being
done. In this regard, we would note that each of these issues was Evaluative Rigor and the Institutional Stance
raised in one manner or the other by reviewers of our initial draft
of this article. Implementation of competence assessment is more difficult in
These questions are not unreasonable, and each is a part of the an era of grade inflation and inflation with regard to letters of
more general question of the cost– benefit analysis of competency recommendation (Miller & Van Rybroek, 1988; Robiner, Fuhr-
assessment. Specifically, is it worth it to the field (specifically, the man, & Risvedt, 1993; Robiner, Saltzman, Hoberman, Semrud-
academic training programs, internships, postdoctoral programs, Clikeman, & Schirvar, 1998). Passage of courses, course grades,
credentialing organizations, etc.) to undertake and bear the cost of and recommendation letters are not as viable or valid as methods
such assessments? Without prior information on current level(s) of of assessment as they may have been historically. This again points
competence and the prevalence/base rate of professional incom- to the need for assessments of competencies that are scrutinized
petence, it may be hard to justify undertaking what may be seen as and selected on the basis of how well the results provide mean-
an extraordinary effort. ingful data. It also highlights the importance of training raters to
Assessment, by itself, often creates resistance within a system maintain the proper level of objectivity for valid and useful as-
(e.g., doctoral program, internship, postdoctoral program, licens- sessments.
ing board). This resistance may be exacerbated when the compe- The attitudes of the institutions toward assessing competence
tency elements to be assessed are prescribed or imposed from are critical. The adoption of nondefensive postures, avoiding com-
outside of that system. This resistance or reactance can be expected petitive postures within and between institutions, and a commit-
to be especially strong when one considers that training programs ment to using the results of student and program assessment to
(and notably those that are accredited by the APA) are given the strengthen the institution would provide the backdrop for the
latitude to formulate their own (sometimes distinctive) training development of even better assessment strategies and techniques.
models, goals, and objectives. In addition, these programs vary in
terms of training philosophy (e.g., clinical scientist, scientist- Dual Roles for Educators and Trainers
practitioner, and practitioner-scholar), intended training outcomes Academic and clinical faculty members typically serve in both
(e.g., the preparation of academics/scholars/researchers vs. prac- formative and summative evaluation capacities for their students
tice professionals), theoretical orientation within professional psy- or trainees, and many tasks require them to serve in both capacities
chology, and settings in which individuals are being prepared to simultaneously. For example, writing letters of recommendation
work. requires walking a line between aiding and honestly evaluating the
There are also implementation challenges associated with the student. Faculty members’ and supervisors’ roles are to facilitate
assessment of competence beyond initial licensure. Although it is the junior colleague’s professional growth and acquisition of com-
an expectation/requirement of the profession’s current ethics code petence. At various junctures (e.g., upon completion of courses, in
that psychologists seek to enhance their knowledge and abilities conducting qualifying or comprehensive examinations, in deter-
and maintain or increase their competence throughout their ca- mining satisfactory internship or postdoctoral fellowship perfor-
reers, there are no formal means for assessing this beyond initial mance), faculty members and supervisors become evaluators of the
licensure. Even where midcareer assessment of competence does same individuals whose development they have been fostering.
occur, there is often a lack of connection between performance on Those responsible for education and training, and to some extent
current competency assessments and how the individual performs credentialing, may have dual roles with regard to the provision of
in the daily pursuit of his or her professional activities. This is most both formative and summative assessment. Although such dual
likely to be the case when such strategies as a portfolio approach roles are part and parcel of the educational process, a move toward
or other peer-review methods are not used. Many will balk at the competency-based assessment requires educators and trainers to be
notion of being required to demonstrate competence postlicensure aware of the tensions and boundary challenges inherent in these
in any systematic fashion. Additionally, currently there are not different and sometimes disparate roles at different developmental
sufficient gates at which one can or is required to demonstrate phases. The duality of roles reflects the affirming and validating
specialization or expertise beyond the “licensable level” of com- functions on the one hand, and the gatekeeping functions on the
petence. Although voluntary gates do exist for those electing other. It also highlights the tension between the value of relative
advanced certification/specialization (e.g., American Board of Pro- subjectivity in the teaching and supervisory role related to forma-
fessional Psychology), unlike medicine, such certification or spe- tive evaluation versus the need for greater levels of objectivity
cialization is not required and most often not pursued within with regard to summative evaluations. These role conflicts need to
professional psychology (Finch, Simon & Nezu, 2006)—with es- be acknowledged and managed for the ethical assessment of com-
timates of 90%–95% of licensed practitioners in psychology not petence (Roberts et al., 2005).
pursuing advanced certification.
There are varying needs of different stakeholder groups. For
Provision of Feedback
example, licensing boards may seek consistency in the preparation
of individuals for professional practice and the expediency of the Many professionals find it challenging to give effective evalu-
processing of applications. Conversely, a diversity of academic ative feedback. Although positive feedback is easier to give than
478 LICHTENBERG ET AL.

negative feedback for most professionals, it is not uncommon for credentialers) working together collaboratively toward shared so-
educators, trainers, and credentialers to provide limited positive lutions.
feedback. This may reflect the prevailing view that “no news is
good news” and a lack of priority placed on the role that positive References
feedback can play in both the learning and assessment process.
Chambers, D. W., Dugoni, A. A., & Paisley, I. (2004). The case against
Professionals are even less likely to offer negative feedback in a
one-shot testing for initial dental licensure. Journal of California Dental
constructive fashion. Many shy away from confrontation, are
Association, 32, 243–246, 248 –252.
reluctant to shame or hurt the person being evaluated, and have Elman, N. S., & Forrest, L. (2006). From “trainee impairment” to “insuf-
difficulty finding a way to frame their critique in a fashion that ficient professional competence”: Seeking terminology that facilitates
would be useful to their student or peer. In addition, many fear effective action. Pittsburgh, PA: University of Pittsburgh.
institutional or legal consequences if they provide negative input. Finch, A. J., Simon, N. P., & Nezu, C. M. (2006). The future of clinical
For instance, a faculty member may fear that negative feedback psychology: Board certification. Clinical Psychology: Science and Prac-
regarding a student may rebound as negative student assessments tice, 13, 254 –257.
of that faculty member. Often times, a reluctance to give feedback Forrest, L., Elman, N., Gizara, S., & Vacha-Haase, T. (1999). Trainee
regarding problems is an effort to avoid the demoralization that impairment: Identifying, remediating, and terminating impaired trainees
in psychology. The Counseling Psychologist, 27, 627– 686.
one fears would result in response to such a confrontation. People
Kaslow, N. J., Rubin, N. J., Bebeau, M., Leigh, I. W., Lichtenberg, J. W.,
often are concerned that the more subjective or subtle (rather than
Nelson, P., et al. (2007). Guiding principles and recommendations for
objective or blatant) the behavior they are addressing in their the assessment of competence. Professional Psychology: Research and
feedback, the more likely it is to be challenged or dismissed. Practice, 38.
Miller, R. K., & Van Rybroek, G. J. (1988). Internship letters of recom-
Conclusions mendation: Where are the other 90%? Professional Psychology: Re-
search and Practice, 19, 115–117.
The adoption of a competency-based model throughout psychol- Overholser, J. C., & Fine, M. A. (1990). Defining the boundaries of
ogy, from training to mid- or late-career practice, represents both professional competence: Managing subtle cases of clinical incompe-
culture and paradigm shifts for the profession. If one examines past tence. Professional Psychology: Research and Practice, 21, 462– 469.
efforts to categorize skill sets or to define the parameters of Peterson, R. L. (2004). Evaluation and the cultures of professional psy-
chology education programs. Professional Psychology: Research and
practice (e.g., licensing and scope of practice laws, agreement on
Practice, 35, 420 – 426.
specialties), it is evident that new initiatives of such magnitude
Roberts, M. C., Borden, K. A., Christiansen, M., & Lopez, S. (2005).
typically require time, effort, and compelling evidence of necessity Toward a culture of competence: Assessment of competence in the
before they become generally accepted. And so it is for profes- education and careers of professional psychologists. Professional Psy-
sional psychology. For each of the reasons discussed in this article, chology: Research and Practice, 36, 355–361.
psychology’s shift to a competency model would be a challenging Robiner, W., Fuhrman, M., & Risvedt, S. (1993). Evaluation difficulties in
endeavor, but one that is required in view of the need to reinforce supervising psychology interns. The Clinical Psychologist, 46, 3–13.
credibility in competence for the practice of psychology. Achiev- Robiner, W. N., Saltzman, S. R., Hoberman, H. M., Semrud-Clikeman, M.,
ing consensus within the profession and across its diversity of & Schirvar, J. A. (1998). Psychology supervisors’ bias in evaluations
specialties, orientations, and models on the necessary competen- and letters of recommendation. Clinical Supervisor, 16, 49 –72.
Rodolfa, E. R., Bent, R. J., Eisman, E., Nelson, P. D., Rehm, L., & Ritchie,
cies for professional practice is a critical first step. Establishing the
P. (2005). A cube model for competency development: Implications for
mechanisms and systems for competency assessment, and evalu-
psychology educators and regulators. Professional Psychology: Re-
ation and building the commitment across the profession to carry search and Practice, 36, 347–354.
it out, are important additional steps. The history of such change Vacha-Haase, T., Davenport, D. S., & Kerewsky, S. D. (2004). Problematic
indicates that relevant constituencies eventually agree and accept students: Gatekeeping practices of academic professional psychology
further refinements in the field. The adoption of a competency programs. Professional Psychology: Research and Practice, 35, 115–
model for the practice of psychology is likely to follow a similar 122.
path. The challenges delineated herein, and others as yet unfore-
seen, are best seen not as roadblocks, but as steps along that path. Received November 3, 2006
The scope of these challenges would certainly involve systemic Revision received June 7, 2007
change and affected groups (educators, supervisors, employers and Accepted June 13, 2007 䡲