Children and Youth Services Review 57 (2015) 171–178

Contents lists available at ScienceDirect

Children and Youth Services Review

journal homepage: www.elsevier.com/locate/childyouth

The Treatment Outcome Package (TOP): A multi-dimensional level of

care matrix for child welfare
David R. Kraus, Elizabeth E. Baxter ⁎, Pamela C. Alexander, Jordan H. Bentley
Outcome Referrals Inc., 1 Speen Street, Framingham, MA 01701, United States
Institute for Child Outcomes, 1 Speen Street, Framingham, MA 01701, United States

a r t i c l e i n f o a b s t r a c t

Article history: Child welfare level of care (LOC) tools have generally either failed to assess the multiple needs of children or failed
Received 16 June 2015 to use this information to determine the most appropriate and least restrictive placement. LOC recommendations
Received in revised form 12 August 2015 are usually presented on a uni-dimensional continuum, often masking divergent, multi-dimensional issues. We
Accepted 12 August 2015
recommend a multi-dimensional LOC approach. Existing LOC measures are reviewed, a conceptual framework of
Available online 17 August 2015
four important dimensions of need (consisting of supervisory, psychological, medical and academic needs) is de-
Keywords:
scribed, and the use of the Treatment Outcome Package (TOP; Kraus, Boswell, Wright, Castonguay, & Pincus,
Level of care 2010) as a multi-dimensional LOC tool is illustrated. Other advantages of the TOP include its sensitivity to change,
Child welfare its use with multiple informants (including self-report) and its potential for identifying providers who are most
Treatment Outcome Package (TOP) appropriate for addressing a child's particular needs.
© 2015 Elsevier Ltd. All rights reserved.

1. Introduction LOC tool development should strive for a complete appraisal of a

child's needs and strengths. Strengths can help identify treatment strat-
The purpose of level of care (LOC) tools is to guide placement of chil- egies and techniques that help improve functioning. However, strengths
dren and adolescents in the child welfare system into the least restric- are not nearly as important in making specific level-of-care decisions,
tive setting appropriate for the child's needs. Consistent with the U.S. which are made primarily based on needs. In other words, children
Adoption and Safe Families Act (ASFA, 1997), of primary importance are not placed into higher or lower levels of care based on their
are child safety and family support, including helping parents provide strengths and any number of strengths has little or no impact on what
a safe, permanent home for their children, as well as child and family is required to deal with any single caregiving need. For example, an
well-being, including helping the family to ensure that the child's honors student who is depressed and suicidal should not be placed in
basic needs and opportunities for growth are met (Winterfeld & Feild, a lower level of care because of her exceptional school functioning.
2003). Nonetheless, the program in which she is placed should ensure that
However, determining the appropriate placement of a child is a she is provided maximum opportunity to continue to thrive, in order
complex process. For example, agencies understandably try to avoid to prevent future depression and to provide her with pleasurable and
separating siblings, even when those siblings present with different rewarding experiences that will facilitate the treatment of her
needs. Moreover, an intake assessment may fail to capture the real depression.
needs of an adolescent who is reticent to disclose his/her true feelings LOC tools are not immune to the risk of under-placement or over-
to an unfamiliar caseworker completing a child protective services placement of a child. For example, on the one hand, an LOC tool that
(CPS) investigation. While many children are known to the child wel- does not thoroughly assess all of a child's needs may inadvertently
fare system prior to needing placement, due to having open, in-home miss the basis for specific placement needs (such as a teen without
cases, some children come to the system under emergency circum- overt behavior problems who subsequently self-reports psychotic
stances, with the system having very little knowledge of the child's symptoms). On the other hand, a bias of over-placement may be attrib-
needs. In these emergent situations, the caseworker may have little in- utable to an assessment that occurs at intake when a child is usually in
formation at his/her disposal to accurately complete an LOC tool. By crisis due to the prospect of removal. Moreover, states often have rela-
contrast, other informants, such as a grandparent or aunt/uncle, may tively low levels of funding for ancillary community-based services for
have the most reliable information. children who remain in the home or with relatives, so caseworkers
may complete an LOC tool in such a way so as to justify the use of resi-
⁎ Corresponding author at: Outcome Referrals, 1 Speen Street, Framingham, MA 01701,
dential care because of a lack of resources that may be required to
United States. support a less restrictive placement. A single-rater LOC tool will be sus-
E-mail address: [email protected] (E.E. Baxter). ceptible to this bias. While an LOC tool cannot in itself drive state

http://dx.doi.org/10.1016/j.childyouth.2015.08.006
0190-7409/© 2015 Elsevier Ltd. All rights reserved.
172 D.R. Kraus et al. / Children and Youth Services Review 57 (2015) 171–178

funding priorities, its lack of transparency can distort those priorities, 2.1. CALOCUS
contributing to harmful or inappropriate placement decisions.
Indeed, research points to the dangers of inappropriate placements, The Child and Adolescent Level of Care Utilization System
such as disruptions and placement failures (Lin, 2012), and the negative (CALOCUS; Sowers, Pumariega, Huffine, & Fallon, 2003), also known as
effects of deviant peer influences in more restrictive settings (Baker, the Child and Adolescent Service Intensity Instrument (CASII) is an as-
Ashare, & Charvat, 2009; Kepper, van den Eijnden, Monshouwer, & sessment tool with the purpose of aiding both clinicians and resource
Vollebergh, 2014). These placement failures are due in part to the inad- management teams in LOC decision making for youth 6–18 years of
equate assessment of a child's multitude of needs that results in nega- age (Fallon et al., 2006). The CALOCUS has demonstrated inter-rater re-
tive outcomes such as placement instability. Instability is most notable liability with intra-class correlation coefficients ranging from 0.57 to
among children and adolescents with specific needs such as physical 0.95 for the subscales and 0.89 to 0.93 for the composite score. The con-
disabilities (Chmelka, Trout, Mason, & Wright, 2011; Crettenden, vergent aspect of concurrent validity was established by comparing the
Wright, & Beilby, 2014; Hill, 2012), behavioral problems and mental ill- CALOCUS to the Child and Adolescent Functional Assessment Scale
ness (Stewart, Baiden, Theall-Honey, & den Dunnen, 2014), and specific (CAFAS; Hodges, 2000), with a Pearson correlation coefficient of 0.62
academic needs (Hagaman, Trout, DeSalvo, Gehringer, & Epstein, 2010). (Fallon et al., 2006). The discriminant dimensions of convergent validity
Children's and adolescents' needs become even more complicated have not been reported nor has the construct and predictive validity of
when psychological problems interact with medical needs (Nelson, the CALOCUS been published.
Smith, Hurley, Epstein, Thompson, & Tonniges, 2013; Nelson, Smith, The CALOCUS contains eight dimensions: Risk of harm, functional
Pick, Epstein, Thompson, & Tonniges, 2013). Reciprocally, placement in- status, comorbidity, environmental stress, environmental support,
stability itself leads to more behavioral issues (Chamberlain et al., 2006), resiliency and treatment history, child/adolescent acceptance and en-
creating a negative spiral that harms the child. gagement, and parents/primary caretaker acceptance and engagement
While levels of care vary across child welfare jurisdictions, the defi- (Fallon et al., 2006). Scoring should be completed by, or with supervi-
nitions that are typically used are often a version of the following levels, sion from, a highly trained clinician. The eight dimension scores are as-
or placement settings: sociated with an LOC recommendation in which four variables are
considered: care environment, clinical services, support services, crisis
Level 1: Foster home/kin with basic supports. stabilization and prevention services.
Level 2: Foster home/kin with extra supports. Hawaii has implemented CALOCUS as an LOC tool with positive re-
Level 3: Therapeutic foster home, or foster home/kin with therapeutic sults and observed strengths (Pires & Grimes, 2006). However, while
supports. the CALOCUS is an excellent tool for measuring level of mental health
Level 4: Residential, or therapeutic foster home, or foster home/kin services needed, its level recommendations are solely focused on inten-
with intensive therapeutic supports. sity of mental health treatment and fail to address the needs of children
Level 5: Hospital or JCHO-accredited facility (100% Medicaid). above and beyond mental health service need. The requirement of a
highly trained clinician further limits child welfare application. Other
For the purposes of this paper, we will be defining “placement” as dimensions, such as medical needs, are assessed with the CALOCUS
any living environment that was determined for the child by the child comorbidity dimension only if they are co-occurring with another disor-
welfare system. der of a different classification (psychiatric, developmental or substance
Although many variations of LOC tools are used, they are character- use).
ized by important limitations, including strict rater qualification re-
quirements and insufficient attention to the medical and academic 2.2. CAFAS
needs of children. Furthermore, even when multiple dimensions of
needs are assessed, converting them to a uni-dimensional LOC recom- The Child and Adolescent Functional Assessment Scales (CAFAS;
mendation can lead to inappropriate assignments, such as sending a Hodges, 2000) measure functional impairment in children and adoles-
child with no medical or psychiatric issues but who is developmentally cents ages 5 to 19 with psychiatric, behavioral, emotional, or substance
or cognitively delayed and needs constant adult supervision (but not use problems. The CAFAS consists of eight youth subscales: School/
mental health treatment) to a higher level of care when a single-child, work, home, community, behavior toward self and others, moods/
multiple-parent foster home may suffice. emotions, self-harmful behavior, substance use, and thinking. Addition-
Therefore, it is the purpose of this paper to introduce a multi- ally, there are two caregiver subscales: Material needs and family/social
dimensional LOC tool that is empirically-based, easy to use and can be support. The CAFAS is designed to be administered by practitioners
modified to meet the specific needs of different jurisdictions. Existing based on information from clinical evaluation. Inter-rater reliability of
LOC tools will be reviewed, a conceptual framework of four important the CAFAS was assessed for agency workers, graduate students and
dimensions of need will be described, and the use of the Treatment Out- naïve undergraduate students with intra-class coefficients for CAFAS
come Package (TOP; Kraus, Boswell, Wright, Castonguay, & Pincus, total scores ranging from 0.84 to 0.89 (Hodges & Wong, 1996). CAFAS
2010) as a multi-dimensional LOC tool will be illustrated. The goal is has also demonstrated partial concurrent validity based on correlations
to encourage the field to adopt a more nuanced LOC understanding with subscales of the Child Behavior Checklist and the Burden of Care
that better meets each individual child's needs. Questionnaire (Hodges & Wong, 1996).
The CAFAS has been adopted by several states for the purpose of LOC
2. Existing LOC measures decision-making, including North Carolina, Michigan, and Hawaii. Since
scoring the CAFAS does not directly result in an LOC recommendation,
Protocols for LOC decision-making vary by state, and even by juris- states such as North Carolina have converted CAFAS total scores into a
diction, with some jurisdictions using internally created measures of corresponding level of dysfunction and an LOC recommendation
LOC assessment and others using standardized LOC tools. Among this (Pires & Grimes, 2006). The use of a total score rather than specific di-
group of standardized LOC tools is a range of the level of establishment mension scores is limiting in that it combines all areas of need into
in the field and psychometric validation (Pires & Grimes, 2006). A report one score of dysfunction, and therefore does not take into account the
on the LOC decision-making protocol in a sampling of states and juris- possibility that different dimensions of need require different types of
dictions revealed that of the states that use a standardized LOC tool, supervision and professional help. Like the CALOCUS, the CAFAS lacks
three are arguably the most widely used (Pires & Grimes, 2006). They adequate (comprehensive) assessment of medical needs. Also, like the
will be discussed in detail below. CALOCUS, the CAFAS requires a trained clinician to administer and
 D.R. Kraus et al. / Children and Youth Services Review 57 (2015) 171–178 173

score the items, limiting child welfare application. This restriction pre- informants, without the requirement of specific qualifications or train-
vents other members of the child's network (e.g., parents, caregivers, ing, and that will include a multidimensional output or LOC recommen-
and caseworkers) from having an active role in the decision making dation that takes into account four distinct dimensions of need:
process. psychological, medical, supervisory, and academic.

2.3. CANS 3. Dimensions of need

The Child and Adolescent Needs and Strengths (CANS; Lyons, 2008) The decision to remove a child from the home is based on a variety of
is a tool used to aid in treatment planning and level of care determina- factors — most notably, safety concerns, physical abuse, sexual abuse,
tion for a child population. The CANS consists of six subscales: problem emotional abuse and neglect. However, in addition to guaranteeing a
presentation, risk behaviors, functioning, care intensity and organiza- child's safety in out-of-home care, child welfare agencies have a respon-
tion, caregiver capacity, and strengths of the child (Anderson, Lyons, sibility to assure the child's social and emotional well-being (ACYF,
Giles, Price, & Estle, 2003). Retrospective chart review was used to as- 2013), and to avoid retraumatizing the child through additional in-
sess inter-rater reliability between researchers and caseworkers with stances of neglect. Thus, a LOC measure must assess a child's needs,
an intra-class correlation coefficient of 0.81 for the total CANS score tackle the effects of prior maltreatment within the least restrictive envi-
(Anderson et al., 2003). CANS users must be trained and certified in ad- ronment, and avert any additional neglect of the child's needs while the
ministration and scoring with a minimum reliability of 0.70. child is in custody. The scope of services offered must be wide enough to
The CANS is owned by the Praed Foundation. Praed reports that the handle the multifaceted needs of children. As noted by Winterfeld and
tool was “developed from communication theory rather than psycho- Feild (2003), behavioral health needs and services can typically be rep-
metric theory” (www.praedfoundation.org). As such, there is little va- resented on a continuum from low level to high level. In contrast, child
lidity data available. The CANS was developed from the Childhood welfare needs and services are much more complex in which the need
Severity of Psychiatric Illness (CSPI) which has very poor construct va- for high intensity services in one area does not necessarily require the
lidity (Leon, Lyons, & Uziel-Miller, 2000). Data regarding concurrent va- provision of high intensity services in another area. Therefore, a com-
lidity is also limited. For example, the CANS depression and anxiety prehensive and thorough analysis of those divergent needs is essential
scale has a correlation coefficient of 0.18 with the CAFAS Moods/ in order to determine how best to respond. The following is a descrip-
Emotions scale and the CANS family functioning scale has a correlation tion of four essential needs of children and youth who are served by
coefficient of 0.26 with the home performance scale of the CAFAS child welfare agencies that should be assessed by LOC tools.
(Dilley, Weiner, Lyons, & Martinovich, 2003).
CANS scores do not reflect a child's current level of functioning. For 3.1. Supervisory needs
example, “a hyperactive child on stimulants is still rated a ‘2’ as long
as you have to work to control symptoms with medications.” (www. Supervisory neglect is the most prevalent type of child neglect
praedfoundation.org). Consequently, there is little guidance to raters (Mennen, Kim, Sang, & Trickett, 2010), accounting for the largest per-
as to how one should estimate the level of pathology or need a child centage of child fatalities (Damashek, Drass, & Bonner, 2014;
would evidence under the hypothetical scenario of no medical or psy- Putnam-Hornstein, Cleves, Licht, & Needell, 2013; Welch & Bonner,
chiatric help. As a result, the CANS does not discriminate between chil- 2013). It also leads to internalizing and externalizing behavioral prob-
dren at lower levels of care from those placed in the higher residential lems as well as an increased risk for substance abuse (Clark, Thatcher,
levels of care. For example, at the initiation of care, children assigned & Maisto, 2005; Knutson, DeGarmo, Koeppl, & Reid, 2005; Na et al.,
to foster placement vs. residential care had average CANS scores of 2014). However, supervisory neglect can also be elicited in part by psy-
1.644 and 1.725 respectively, a clinically insignificant difference chological dysregulation in children and early adolescent substance
(Lyons, Woltman, Martinovich, & Hancock, 2009). abuse (Clark, Kirisci, Mezzich, & Chung, 2008), suggesting that a child's
Although the CANS does not require that a certified clinician admin- needs for supervision is a product of not only the child's age but also the
ister the assessment, the lack of specificity of items may be difficult for a child's behavior and psychological needs. As noted by Scott, Higgins, and
rater without mental health training to understand. For example, the Franklin (2012), “there is no universally accepted, all-encompassing
CANS depression item refers to “clear evidence of disabling level of de- definition of the concept of ‘neglect,’ and consideration of supervisory
pression that makes it virtually impossible for the child to function in neglect suggests that there is no ‘one size fits all’ definition.” Therefore,
any life domain” (Lyons, 2008). Potentially serious floor and ceiling ef- while frequently the basis for removal, the child's evolving need for su-
fects on CANS exist due to the construction of items and a scale ranging pervision should be an ongoing focus of LOC assessment throughout a
from 0 to 3. General population, clinical, and child welfare norms are child's involvement in the child welfare system.
also unavailable, limiting the interpretation of results.
Existing LOC tools vary widely in both structure and practice; how- 3.2. Psychological needs
ever, a common theme throughout is the lack of a multidimensional
LOC recommendation. While each of the LOC tools described is com- The short-term and long-term effects of maltreatment are indisput-
prised of multiple subscales at the input level of assessment, none pro- able as seen in substantially elevated rates of mental health problems in
duces a multidimensional output in order to account for all aspects of a maltreated children and adolescents (Heneghan et al., 2013; Keller,
child's needs. Furthermore, existing LOC tools are limited by the per- Salazar, & Courtney, 2010; McMillen et al., 2005; Taussig, Harpin, &
spective of a single rater, one who often must be a qualified and/or Maguire, 2014). These rates frequently differ by type of placement, sug-
trained professional. It is essential to consider the perspectives of multi- gesting both a selection effect due to pre-existing differences and a dif-
ple informants, particularly those closest to the child (i.e., parent, foster ferential effect of foster care when rates of onset differ after entry into
parent, teacher, therapist, etc.) and the child him/herself in order to gain care (Keller et al., 2010). Furthermore, except for a diagnosis of PTSD
a well-rounded understanding of the child's needs and prevent raters which generally predates entry into foster care, the onset of other
from gaming the system to achieve the result they desire. By allowing types of disorders tend to follow entry into care, suggesting that pro-
assessments from professionally trained individuals as well as un- viders have a potentially large impact in preventing, addressing, or
trained members of the child's family, it is easier to administer frequent even exacerbating these emerging mental health concerns (Keller
follow-up assessments and gain a full picture of the child's functioning et al., 2010). Therefore, a more thorough analysis of the specific psycho-
in all relevant environments (e.g., home and school). Therefore, we logical risk factors characterizing individual children as well as an em-
propose an LOC tool that will consider the perspectives of multiple pirical basis for matching children with specific providers could help
174 D.R. Kraus et al. / Children and Youth Services Review 57 (2015) 171–178

ward off or mitigate many psychological problems characterizing youth Excellent fit statistics on diverse samples document the TOP's excep-
in the child welfare system. tional construct validity with 11–13 established well-being domains
that cover behavioral and mental health symptoms, functioning, and
3.3. Medical needs quality of life (Kraus, Seligman, & Jordan, 2005; Kraus et al., 2010). De-
pending on the age versions these well-being domains are assessed
Children with medical needs tend to be labeled in child welfare ad- with 48–58 simple and easy to understand questions — all of which
ministrative records as “disabled” whether due to emotional distur- ask about behaviors exhibited in the last two weeks on the same Likert
bance, intellectual and developmental disabilities, learning disabilities, scale from “None of the time” to “All of the time”. In child welfare, every-
a physical disability, or another non-behavioral medical need one involved in the child's care (including the child him/herself) is en-
(Lightfoot, Hill, & LaLiberte, 2011). Therefore, state systems may not ad- couraged to answer the questions to which they readily know the
equately differentiate medical needs from psychological needs or spe- answer. Questionnaires are processed in real time with reports returned
cial education needs. However, disabilities overall appear to be almost immediately, allowing for the comparison of each rater's re-
associated with longer time in out-of-home placement and higher sponses to the general (normal) population and to the child over time.
rates of placement instability (Hill, 2012). Also, as mentioned previous- In addition, 17 medical symptoms and needs are assessed as are
ly, medical needs may interact with psychological needs to complicate items related to life stress, academic performance, and items assessing
placement decisions. A diabetic child who needs close medical monitor- risks and strengths with regard to supervisory needs. The TOP has dif-
ing to adjust insulin levels (but who is otherwise behaviorally compliant ferent versions for children, adolescents, and adults (Kraus et al.,
and cooperative) will obviously need a different placement than a dia- 2005). It typically requires 15–20 min for the initial assessment to be
betic child who is also impulsive and violent. Therefore, it is imperative completed with no special training, and less time for subsequent
that a systematic assessment of a child's medical needs be conducted in assessments.
part to facilitate placement with foster parents or treatment facilities The TOP has been used for over 20 years in behavioral health agen-
that are more able and willing to nurture and support children with spe- cies across the United States (Kraus, Wolfe, & Castonguay, 2006). With
cial needs (Orme, Cherry, & Cox, 2013). the TOP, clinicians and clients are able to track client outcomes and de-
termine what particular domains of functioning the client is struggling
3.4. Academic needs with. More recently, the TOP has been used in child welfare jurisdictions
in Ohio, Colorado, North Carolina, and Delaware to measure the well-
As noted above, research tends to group “educationally relevant” being of kids in care (both in-home and out of home), to encourage in-
disabilities such as communication disorders or developmental delays volvement of multiple stakeholders, and to facilitate case planning. The
with health or orthopedic disabilities (Sullivan & Knutson, 2000). TOP can be used to track well-being for all kids in care, including those
Whether or not academic need is labeled as a disability, academic or remaining in-home. However, the TOP as an LOC tool will provide a
learning problems of school-age children are more prevalent in out- placement recommendation only for out-of-home children (i.e. once
of-home placement settings (Jonson-Reid, Drake, Kim, Porterfield, & the decision has been made to remove the child from his/her home).
Han, 2004; Scarborough & McCrae, 2010). The same can be said for
the academic needs of maltreated children overall, with chronic mal- 4.1. Psychometric properties
treatment explaining a significant portion of the variance in children's
math and reading scores, perceptual reasoning, overall academic perfor- The TOP has excellent test–retest reliability, with intra-class correla-
mance, and absenteeism (Coohey, Renner, Hua, Zhang, & Whitney, tion coefficients for all subscales (with the exception of mania) ranging
2011; Leiter, 2007; Manly, Lynch, Oshri, Herzog, & Wortel, 2013; Mills from 0.87 to 0.94 (Kraus et al., 2005). Confirmatory and exploratory
et al., 2011). The long-term effects of these academic needs, when factor analyses on the TOP demonstrated that it has excellent construct
unmet, are substantial, contributing to delinquency (Mallett, 2014) validity (Kraus et al., 2005; Kraus et al., 2010). Concurrent validation
and lower levels of employment and earnings as adults (Currie & with both convergent and discriminant results has also been substanti-
Widom, 2010). On the other hand, child welfare involvement can help ated with the Beck Depression Inventory (BDI), Brief Symptom
ameliorate the effects of maltreatment on school attendance, grades Inventory (BSI), Minnesota Multiphasic Personality Inventory—2
and cognitive engagement (Font & Maguire-Jack, 2013; Larson, Zuel, & (MMPI-2), the Basis 32, the SF-36, CAFAS, BASC, Deveraux, and CBCL
Swanson, 2011; Leiter, 2007) which is improved with correct place- (Kraus & Seligman, 2006; TOP Manual). The TOP has also demonstrated
ment. Despite this importance, an assessment of a child's academic excellent predictive validity in predicting the need for treatment (Kraus
needs is often missing from the decision-making process regarding et al., 2005), future behavioral health hospitalizations (Youn, Kraus, &
his/her placement and ancillary support services. Indeed, a significant Castonguay, 2012), therapist effectiveness (Kraus, Castonguay,
effect of the academic success of children in out-of-home placements Boswell, Nordberg, & Hayes, 2011), and response to treatment
is attributable to the placement itself, including therapeutic foster care (Nordberg, Castonguay, Fisher, Boswell, & Kraus, 2014).
(Cheung, Lwin, & Jenkins, 2012; Lewis-Morrarty, Dozier, Bernard, The TOP is highly sensitive to change, with Cohen's d effect sizes
Terracciano, & Moore, 2012). Therefore, LOC decisions have a demon- ranging from 0.27 to 0.91 after an average of 7 therapy sessions with
strable impact on this very important need. an adult mental health population (Kraus et al., 2011). Put another
Given that the needs of children and adolescents within the child way, the TOP is able to detect reliable change on more than 96% of indi-
welfare system are complex, a single-dimensional output from an LOC viduals measured (Kraus et al., 2005). This sensitivity to change is par-
tool is insufficient. Instead, it is necessary to use an LOC tool that as- ticularly relevant in a child welfare population when, for example, a
sesses a large number of domains and that captures the potentially ex- child in residential treatment improves to the point of no longer need-
tensive span of functioning within these domains. The Treatment ing such a restrictive level of care. Although this child may still score
Outcome Package, or TOP, represents such a tool. outside of the normal range on problem domains, even his/her small
but significant improvement may warrant consideration of a lower
4. The TOP as an LOC tool level of care and less restrictive setting. The inability of an LOC tool to
measure small changes in a child's level of functioning, particularly at
The Treatment Outcome Package (TOP; Kraus et al., 2010) is a be- the extreme ends of measurement, precludes a recommendation of
havioral health assessment of 13 different domains of well-being for the most appropriate level of care. For example, published CANS data
children (ages 3–18), and 11 domains for adolescents (ages 11–21), in- (Sieracki, Leon, Miller, & Lyons, 2008) suggests that a child would
cluding behaviors frequently associated with a history of maltreatment. need to change more than 2 scale points (a “3” changed to “1”) in
 D.R. Kraus et al. / Children and Youth Services Review 57 (2015) 171–178 175

order to confirm reliable change outside of the tool's measurement 4.2.2. Real-time feedback and emergency alerts
error (Jacobson & Truax, 1991). Requiring this level of change could TOP assessments are automatically scored and reports are generated
mean the child remains in a highly restrictive placement longer than within 15 min of submission. Results are presented in easy-to-read
necessary. graphs and are sent directly to the caseworker's email. If a child scores
The TOP is able to identify the needs of each child with item-specific in the severe range on a given domain, whether through self-report or
level of detail. For example, supervisory needs are identified by an ele- the report of any rater, the caseworker is immediately alerted so that
vated score on items such as “felt that you were going to act on violent he/she is able to respond to the potential crisis as quickly as possible.
thoughts” or “run away”. Academic needs are triggered by elevated
scores on items relating to school functioning (e.g., “been slow at com- 4.2.3. Matching children with specific providers
pleting homework”, and/or “had trouble paying attention in class”). In addition to placing children in the appropriate level of care, chil-
Medical needs are identified with a checklist of 17 medical symptoms dren should be matched with a provider whose strengths are aligned
with responses ranging from “no medical issues in this category” to “af- with the child's needs. Simulations using effectiveness-based matching
fecting one's health” to “on medication” to “been hospitalized in the suggest that positive outcomes could increase from 20% with poor
past year” for the medical issue. In addition, the TOP assesses overall matching to 80% with good matching (Wampold & Imel, 2015). The
health and medical utilization (e.g., “how many times was a physician TOP has been used to determine the problem-specific effectiveness of
seen for medical reasons in the past 2 months?”). With item level spec- therapists by assessing the degree of improvement in their previous cli-
ificity, the TOP is able to pinpoint the specific needs of the child in order ents in the behavioral health population (Kraus et al., 2011). Like client–
to make an individualized level of care recommendation. therapist matching, children and youth with specific needs would ideal-
Validity of TOP domain scores within child welfare samples is indi- ly be matched with the best available provider who has a successful
cated by the significant association of symptom severity on the TOP track-record of helping children with similar needs. Foster parents or
with placement type. Namely, children and adolescents in residential clinicians who are most effective in dealing with a child's internalizing
treatment settings exhibited more severe problems on the majority of problems, for example, may not be equally effective in addressing exter-
TOP domains as compared to those in relative care and foster care nalizing problems (Orme et al., 2013). The TOP offers the potential to
(Kraus, 2015). In as yet unpublished data from a sample of children develop measures of effectiveness, such as ability to handle internaliz-
and adolescents in a Colorado child welfare jurisdiction, independent ing vs. externalizing problems, as the basis for matching children and
reports of trauma experiences were significantly associated with elevat- youth in the child welfare system with appropriate providers.
ed scores on TOP domains such as sleep problems, depression,
suicidality, violence and worrisome sexual behavior. In recently ana- 5. The use of a multifaceted LOC model
lyzed data from the Ohio child welfare jurisdiction, TOP scores were
able to predict placement disruptions in children (N = 380) above Table 1 illustrates a four-dimensional LOC recommendation system
and beyond demographic and placement characteristics. where each child is rated on a 1–6-point scale within each dimension.
An assessment tool like the TOP can be used to generate this output
4.2. Other potential advantages of the TOP and/or it can be supplemented by a caseworker's investigation and pro-
fessional ratings. The model makes no assumption that the level of need
4.2.1. The use of multiple raters in one area will necessarily be correlated with level of need in another.
The TOP incorporates the perspectives of multiple informants, in- These LOC recommendations are less a reflection of problem severity
cluding the child or adolescent him/herself. There are inherent advan- and more a reflection of the amount of professional and caregiver in-
tages of eliciting the views of all those who have had significant volvement. In essence, “it is not how severe the issue is. It is how
interactions with the child during the past month. Each informant much the issue affects the caregiving needs of the child” (T. Feild, per-
brings a unique perspective and usually knows something important sonal communication, August 25, 2014). Relating this concept to TOP
about the child's life. For example, a foster parent will likely be the domain scores, some assessed needs such as a child's significantly ele-
most reliable informant about the child's sleep functioning, while a vated score on domains of incontinence, assertiveness or separation
teacher is likely to know more about behavioral and functional issues anxiety might only require extra supports within a foster home/kin
at school. Including the child's (and especially the adolescent's) voice al- LOC. On the other hand, elevated violence and suicide scores may re-
lows for the earliest possible disclosure of emerging problems such as quire more intensive supervision from a therapeutic foster home, or res-
suicidal thoughts, depression and other severe psychological issues. idential treatment facility.
Collecting diverse perspectives through multiple informants can In many cases, the LOC recommendation must take into account the
help avoid placement disruptions, facilitate the emerging autonomy of number and severity of problem areas as these may exacerbate the
youth transitioning into adulthood, improve children's safety and problem and make it harder for adults to care for a child's needs. For ex-
promote feelings of well-being among/within children who are being ample, the same psychosis domain score may lead to different LOC rec-
evaluated (Rauktis, Kerman, & Phillips, 2013; Strijker, van Oijen, & ommendations especially if they co-exist with other elevated domain
Knot-Dickscheit, 2011; Vis, Strandbu, Holtan, & Thomas, 2011). Includ- scores. For one child, the LOC output might recommend a therapeutic
ing parents in the assessment process increases their engagement and foster home, or foster home/kin placement with therapeutic supports.
positive contacts with their child (Crea, Wildfire, & Usher, 2009; Hojer, In contrast, another child with the same elevated psychosis score
2011), and may increase the chances of positive reunification. Including might require a residential treatment placement with intensive thera-
the adolescent's perspective helps to foster empowerment and provides peutic supports because of other elevated domains such as conduct
the teen with an outlet to voice his/her concerns. problems and violence.
In the absence of much-needed research as to whose perspective is Age and a child's developmental state should also be taken into con-
most predictive of placement success, the report of high-risk behavior sideration. For all of these psychological needs as well as the superviso-
by any respondent on the LOC tool could signal the need for a more de- ry, medical and academic needs of the child, algorithms have been
tailed follow-up assessment. For example, TOP data from one child wel- developed to suggest the most appropriate and least restrictive setting
fare jurisdiction revealed that caseworkers were unaware of suicidal for a child as well as the ancillary services required in order to meet
ideation in 50% of the adolescents who self-reported that they were the child's needs for safety and the family's needs for support.
feeling suicidal (Kraus, 2015). Thus, the use of multiple informants In order to make the LOC output of Table 1 actionable, each LOC
with a LOC tool would permit the most comprehensive assessment of provider would need to be rated for their intake criteria and service
a child's needs and reduce the risk of single-rater bias. array. Table 2 provides a hypothetical example of how this multi-
176 D.R. Kraus et al. / Children and Youth Services Review 57 (2015) 171–178

Table 1
Four-dimensional level of care recommendation.

Level Supervision Psychological Medical School

The level of adult supervision necessary to The level of licensed psychological Level of MD or RN Learning limitations or
keep the child (or others) safe treatment or consultation required treatment or requirements
to manage or treat emotions or monitoring required
behavior

1 Fully independent: able to check in with adults as needed None known None known None/age appropriate needs
2 Normal: age appropriate level required As needed As needed Special education teacher or consultation
required
3 Intermittent: occasionally child needs more supervision Monthly Monthly Special education classroom required
than others his/her own age
4 Hourly: frequent monitoring required Weekly Weekly Special education school required
5 Constant daytime supervision required Daily Daily
6 Overnight and daytime supervision required Hourly Hourly

dimensional LOC system might be used in a small jurisdiction (as men- other impulsive behavior. However, when given proper supervision,
tioned previously, the TOP will only be used as an LOC tool with out-of- this child does engage in school, learns appropriately and appears to
home children). Each provider would be evaluated for its capacity and thrive in a more structured environment. Using the levels and corre-
intake criteria on each of the four dimensions (e.g., using the dimen- sponding service capability in Table 2, it is clear that there is no ideal
sional scores from Table 1). For most providers the output would yield placement for this child, and she will likely need to be placed outside
a range with the lowest number in the range representing the threshold the jurisdiction to obtain the least restrictive environment. The least re-
intake criteria for the provider, and the highest score in the range strictive environment that can handle all of this child's needs is a Ther-
representing the most disturbed or involved child that the provider apeutic Foster-Care placement where parents are trained to handle
could manage on that dimension. more dysregulated children. If possible, this jurisdiction should look to
For example, the two psychiatric residential providers in Table 2 engage foster families that can commit to rather constant daytime su-
have a mission of providing services to children who need around- pervision when the child is not in school.
the-clock supervision and behavioral healthcare. Therefore, their scores As yet another example, without at least one medically specialized
go up to the maximum (6) on these dimensions. At the same time, these foster care setting, this small jurisdiction may be ill-prepared to place
residential providers will not accept children who do not require this a child with significant medical problems (e.g., an insulin-dependent
high-level of care which is reflected by the floor of their range. However, child whose blood sugar levels are not stable, and who frequently runs
these providers are not equipped to handle serious medical problems away). Such children may be otherwise alert, engaged, happy and coop-
and this limitation is recorded in the limited range of the medical di- erative. The jurisdiction may have tried to address the medical needs of
mension. Residential Provider A, who only has access to an MD sporad- similar children by engaging a few nurses to become foster parents, but
ically, can only care for children who need monthly medical supervision, the foster parents cannot provide the level of care necessary for this spe-
while Residential Provider B has a pediatrician who comes to the pro- cific child's increased vulnerability at night. By contrast, one of the psy-
gram weekly to handle basic medical needs. For this small jurisdiction, chiatric hospitals (Hospital B) in the jurisdiction is able to provide this
a child who requires more serious medical attention will need to be level of medical need but is primarily equipped to address needs of
placed in a hospital or a nursing-assisted foster-care placement. As medically involved children (like those with serious seizure or other
such, the result of these rating efforts can illustrate potential gaps in ser- neurological issues) who also have serious behavioral dysregulation.
vice on these continua of care. Given sufficient demand, the jurisdiction This dilemma clearly requires a creative solution in order to avoid the
might consider contracting with one of the residential programs to hire twin dangers of either under-placement (with regard to the child's
a staff pediatrician or engage daily pediatric and nursing support. medical needs) or over-placement (with regard to the child's psycho-
As another example, the assessment of a specific child who is evalu- logical needs). One possibility would be to employ a home health aide
ated with a multi-dimensional LOC tool may reveal moderate–severe to stay with the child during the evening at a fraction of the cost of hos-
behavioral and supervisory needs (Supervision = 5; Psychological = pitalization. Thus, whatever the resources of the jurisdiction, this LOC
3) but no special medical requirements, and minimal educational tool would highlight the multiple needs of the child, giving the jurisdic-
needs (Medical = 1; School = 2). This is a child who has received min- tion an informed opportunity to customize the least restrictive and most
imal supervision from her biological family, who cannot be left unat- appropriate placement.
tended for more than an hour without consequences, and who needs In the ideal model, the LOC evaluation system would be computer-
weekly therapy or consultation to help her control her anger and ized and a list of best-matched placements would be available to the

Table 2
Level of care resource and service capabilities for a hypothetical child welfare jurisdiction by dimension of need.

Level of care Supervision Psychological Medical School

The level of adult supervision The level of licensed psychological Level of MD or RN Learning
necessary to keep the child treatment or consultation required treatment or monitoring limitations or
(or others) safe to manage or treat emotions or required requirements
behavior

Traditional foster care 1–3 1–4 1–3 1–3

Experienced, medically specialized or foster care 1–4 1–4 4–5 1–3
Treatment foster care 1–4 1–4 1–3 1–3
Psychiatric residential treatment A 3–6 5–6 1–3 1–2
Psychiatric residential treatment (accredited) B 3–6 5–6 1–4 1–3
Psychiatric in-patient Hospital A 6 6 1–5 1–4
Psychiatric in-patient Hospital B 6 6 4–6 1–4
 D.R. Kraus et al. / Children and Youth Services Review 57 (2015) 171–178 177

Fig. 1. Sample TOP level of care output.

caseworker or placement team without requiring a manual search assess just how much children's lives can be improved with this type
through lists of providers. As such, the LOC assessment would provide of LOC system.
a multi-dimensional analysis of a specific child's needs such as that
shown in Fig. 1. The LOC database would designate a list of providers Acknowledgments
that are currently available to treat this child's needs. In time, and
with more data, this same system could also be supplemented by the This research was funded in part by the Annie E. Casey Foundation
outcome track-record of each provider when treating similar types of (Grant No. 212.0369) and the Duke Endowment (Grant No. 1935-SP).
children (Kraus et al., 2011). These providers would then be rank or- We thank them for their support but acknowledge that the findings
dered based on their actual outcomes, thereby assuring the optimal and conclusions presented in this report are those of the authors
match between a specific child, available providers and proximity to alone, and do not necessarily reflect the opinions of these foundations.
family.
References
6. Conclusion
Administration for Children and Families (2013). Integrating safety, permanency, and
well-being for children and families in child welfare. Retrieved from http://www.
In conclusion, a multi-dimensional LOC recommendation system acf.hhs.gov/sites/default/files/cb/acyf_fy2012_projects_summary.pdf
could address the limitations frequently reported by child welfare juris- Adoption and Safe Families Act of 1997, Pub. L. 105–89.
dictions. The TOP is ideally suited to assess the multiple needs of chil- Anderson, R.L., Lyons, J.S., Giles, D.M., Price, J.A., & Estle, G. (2003). Reliability of the Child
and Adolescent Needs and Strengths—Mental Health (CANS-MH) scale. Journal of
dren and adolescents, to elicit the perspectives of multiple informants, Child and Family Studies, 12, 279–289.
and to use this information to meet children's complex needs in the Baker, A.J.L., Ashare, C., & Charvat, B.J. (2009). Substance use and dependency disorders in
most appropriate and least restrictive manner. It also has the potential adolescent girls in group living programs: Prevalence and associations with milieu
factors. Residential Treatment for Children & Youth, 26, 42–57.
to be used to match a child with the most effective provider(s) available.
Chamberlain, P., Price, J.M., Reid, J.B., Landsverk, J., Fisher, P.A., & Stoolmiller, M. (2006).
This review should be considered in light of several limitations. First, Who disrupts from placement in foster and kinship care? Child Abuse and Neglect,
the policies and practices discussed are limited to child welfare systems 30, 409–424.
in the United States and do not necessarily mirror the practices of child Cheung, C., Lwin, K., & Jenkins, J.M. (2012). Helping youth in care succeed: Influence of
caregiver involvement on academic achievement. Children and Youth Services
welfare agencies internationally. Furthermore, we acknowledge that Review, 34, 1092–1100.
even within the United States, policies and practices will vary by state Chmelka, M.B., Trout, A.L., Mason, W.A., & Wright, T. (2011). Children with and without
and county, and therefore the information presented here should be disabilities in residential care: Risk at program entry, departure and six-month
follow-up. Emotional & Behavioural Difficulties, 16, 383–399.
considered a general summary of LOC decision-making practices that Clark, D.B., Kirisci, L., Mezzich, A., & Chung, T. (2008). Parental supervision and alcohol use
are commonly adhered to. However, even given this variability, we be- in adolescence: Developmentally specific interactions. Journal of Developmental and
lieve the TOP can provide the customization and flexibility to fit the Behavioral Pediatrics, 29, 285–292.
Clark, D.B., Thatcher, D.W., & Maisto, S.A. (2005). Supervisory neglect and adolescent alco-
needs and service capabilities of any child welfare jurisdiction. hol use disorders: Effects on AUD onset and treatment outcome. Addictive Behaviors,
In the next article in this progression, we will present the expert 30, 1737–1750.
model and the algorithms derived from TOP scores and recommend Coohey, C., Renner, L.M., Hua, L., Zhang, Y.J., & Whitney, S.D. (2011). Academic achieve-
ment despite child maltreatment: A longitudinal study. Child Abuse & Neglect, 35,
the type of LOC supports that a specific child will need for each of the 688–699.
four LOC dimensions. We are currently collecting data from the multiple Crea, T.M., Wildfire, J.B., & Usher, C.L. (2009). The association of team composition and
child welfare jurisdictions that are using the TOP in states such as Ohio, meeting characteristics with foster care placement recommendations. Journal of
Social Service Research, 35, 297–310.
North Carolina, Delaware and Colorado. These data will be used to fine-
Crettenden, A., Wright, A., & Beilby, E. (2014). Supporting families: Outcomes of place-
tune these expert algorithms with actual child welfare specific data to ment in voluntary out-of-home care for children and young people with disabilities
create a learning system that generates recommendations based on out- and their families. Children and Youth Services Review, 39, 57–64.
come data. We then plan to conduct a randomized controlled trial that Currie, J., & Widom, C.S. (2010). Long-term consequences of child abuse and neglect on
adult economic well-being. Child Maltreatment, 15, 111–120.
compares the use of these LOC recommendations in a new jurisdiction Damashek, A., Drass, S., & Bonner, B.L. (2014). Child maltreatment fatalities related to
with a sister jurisdiction making placements as usual to accurately inadequate caregiver supervision. Journal of Interpersonal Violence, 29, 1987–2001.
178 D.R. Kraus et al. / Children and Youth Services Review 57 (2015) 171–178

Dilley, J.B., Weiner, D.A., Lyons, J.S., & Martinovich, Z. (2003, August). The validity of the Mallett, C.A. (2014). Youthful offending and delinquency: The comorbid impact of mal-
Child and Adolescent Needs and Strengths assessment. Poster presented at the annual treatment, mental health problems, and learning disabilities. Child & Adolescent
meeting of the American Psychological Association, Toronto, ONT, Canada. Social Work Journal, 31, 369–392.
Fallon, T., Pumariega, A.J., Sowers, W., Klaehn, R., Huffine, C., Vaughan, T., et al. (2006). A Manly, J.T., Lynch, M., Oshri, A., Herzog, M., & Wortel, S.N. (2013). The impact of neglect
level of care instrument for children's systems of care: Construction, reliability, and on initial adaptation to school. Child Maltreatment, 18, 155–170.
validity. Journal of Child and Family Studies, 15, 143–155. McMillen, J.C., Zima, B.T., Scott, L.D., Auslander, W.F., Munson, M.R., Ollie, M.T., et al.
Font, S., & Maguire-Jack, K. (2013). Academic engagement and performance: Estimating (2005). Prevalence of psychiatric disorders among older youths in the foster care sys-
the impact of out-of-home care for maltreated children. Children and Youth Services tem. Journal of the American Academy of Child & Adolescent Psychiatry, 44, 88–95.
Review, 35, 856–864. Mennen, F.E., Kim, K., Sang, J., & Trickett, P.K. (2010). Child neglect: Definition and iden-
Hagaman, J.L., Trout, A.L., DeSalvo, C., Gehringer, R., & Epstein, M.H. (2010). The academic tification of youth's experiences in official reports of maltreatment. Child Abuse &
and functional academic skills of youth who are at risk for language impairment in Neglect, 34, 647–658.
residential care. Language, Speech, and Hearing Services in Schools, 41, 14–22. Mills, R., Alati, R., O'Callaghan, M., Najman, J.M., Williams, G.M., Bor, W., et al. (2011). Child
Heneghan, A., Stein, R. K., Hurlburt, M. S., Zhang, J., Rolls-Reutz, J., Fisher, E., & Horwitz, S. abuse and neglect and cognitive function at 14 years of age: Findings from a birth co-
M. (2013). Mental health problems in teens investigated by U.S. child welfare agen- hort. Pediatrics, 127, 4–10.
cies. Journal Of Adolescent Health, 52, 634–640. http://dx.doi.org/10.1016/j. Na, K. S., Lee, S.I., Hong, H.J., Oh, M. J., Bahn, G.H., Ha, K., et al. (2014). The influence of un-
jadohealth.2012.10.269. supervised time on elementary school children at high risk for inattention and prob-
Hill, K. (2012). Permanency and placement planning for older youth with disabilities in lem behaviors. Child Abuse & Neglect, 38, 1120–1127.
out-of-home placement. Children and Youth Services Review, 34, 1418–1424. Nelson, T.D., Smith, T.R., Hurley, K.D., Epstein, M.H., Thompson, R.W., & Tonniges, T.F.
Hodges, K. (2000). Child and Adolescent Functional Assessment Scale. Ypsilanti, MI: Eastern (2013a). Association between psychopathology and physical health problems
Michigan University. among youth in residential treatment. Journal of Emotional and Behavioral Disorders,
Hodges, K., & Wong, M.M. (1996). Psychometric characteristics of a multidimensional 21, 150–160.
measure to assess impairment: The Child and Adolescent Functional Assessment Nelson, T.D., Smith, T.R., Pick, R., Epstein, M.H., Thompson, T.W., & Tonniges, T.F. (2013b).
Scale. Journal of Child and Family Studies, 5(4), 445–467. Psychopathology as a predictor of medical service utilization for youth in residential
Hojer, I. (2011). Parents with children in foster care—How do they perceive their contact treatment. Journal of Behavioral Health Services & Research, 40, 36–45.
with social workers? Practice: Social work in action, 23, 111–123. Nordberg, S.S., Castonguay, L.G., Fisher, A.T., Boswell, J.F., & Kraus, D. (2014). Validating
Jacobson, N.S., & Truax, P. (1991). Clinical significance: A statistical approach to defining the rapid responder construct within a practice research network. Journal of Clinical
meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychology, 70, 886–903.
Psychology, 59(1), 12–19. Orme, J., Cherry, D.J., & Cox, M.E. (2013). Measuring willingness to foster children with
Jonson-Reid, M., Drake, B., Kim, J., Porterfield, S., & Han, L. (2004). A prospective analysis disabilities and special medical conditions. Social Work Research, 37, 169–178.
of the relationship between reported child maltreatment and special education eligi- Pires, S.A., & Grimes, K.E. (2006). Health care reform tracking project (HCRTP): Promising
bility among poor children. Child Maltreatment, 9, 382–394. approaches for behavioral health services to children and adolescents and their families
Keller, T.E., Salazar, A.M., & Courtney, M.E. (2010). Prevalence and timing of diagnosable in managed care systems. 8: Clinical decision making approaches for child/adolescent
mental health, alcohol, and substance use problems among older adolescents in the behavioral health care in public sector managed care systems. Tampa, FL: University
child welfare system. Child and Youth Services Review, 32, 626–634. of South Florida, Louis de la Parte Florida Mental Health Institute (FMHI), Research
Kepper, A., van den Eijnden, R., Monshouwer, K., & Vollebergh, W. (2014). Understanding and Training Center for Children's Mental Health.
the elevated risk of substance use by adolescents in special education and residential Putnam-Hornstein, E., Cleves, M.A., Licht, R., & Needell, B. (2013). Risk of fatal injury in
youth care: The role of individual, family and peer factors. European Child & Adoles- young children following abuse allegations: Evidence from a prospective,
cent Psychiatry, 23, 461–472. population-based study. American Journal of Public Health, 103, e39–e44.
Knutson, J.F., DeGarmo, D., Koeppl, G., & Reid, J.B. (2005). Care neglect, supervisory ne- Rauktis, M.E., Kerman, B., & Phillips, C.M. (2013). Transitioning into adulthood: Promoting
glect, and harsh parenting in the development of children's aggression: A replication youth engagement, empowerment, and interdependence through teaming practices.
and extension. Child Maltreatment, 10, 92–107. In H. Cahalane (Ed.), Contemporary issues in child welfare practice. Contemporary social
Kraus, D.R. (2015, March). Well-being research: A game change for child welfare. Presen- work practice (pp. 101–125). New York: Springer Science + Business Media.
tation at The Measure of Leadership: Innovative Child Welfare Practices Conference. New Scarborough, A.A., & McCrae, J.S. (2010). School-age special education outcomes of infants
Orleans, Louisiana. and toddlers investigated for maltreatment. Children and Youth Services Review, 32,
Kraus, D.R., Boswell, J.F., Wright, A.G.C., Castonguay, L.G., & Pincus, A.L. (2010). Factor 80–88.
structure of the Treatment Outcome Package for Children. Journal of Clinical Scott, D., Higgins, D., & Franklin, R. (2012). The role of supervisory neglect in childhood in-
Psychology, 66, 627–640. jury. Australian Government, Australian Institute of Family StudiesChild Family
Kraus, D.R., Castonguay, L., Boswell, J.F., Nordberg, S.S., & Hayes, J.A. (2011). Therapist ef- Community Australia Information Exchange, CFCA Paper No. 8. (pp. 1–10), 1–10.
fectiveness: Implications for accountability and patient care. Psychotherapy Research, Sieracki, J.H., Leon, S.C., Miller, S.A., & Lyons, J.S. (2008). Individual and provider effects on
21, 267–276. mental health outcomes in child welfare: A three level growth curve approach.
Kraus, D.R., & Seligman, D.A. (2006). TOP user's manual. Ashland, MA: Behavioral Health Children and Youth Services Review, 30, 800–808.
Laboratories. Sowers, W., Pumariega, A., Huffine, C., & Fallon, T. (2003). Level-of-care decision making
Kraus, D.R., Seligman, D.A., & Jordan, J.R. (2005). Validation of a behavioral health treat- in behavioral health services: The LOCUS and the CALOCUS. Psychiatric Services, 54,
ment outcome and assessment tool designed for naturalistic settings: The Treatment 1461–1463.
Outcome Package. Journal of Clinical Psychology, 61, 285–314. Stewart, S.L., Baiden, P., Theall-Honey, L., & den Dunnen, W. (2014). Deliberate self-harm
Kraus, D., Wolfe, A., & Castonguay, L.G. (2006). The outcome assistant: A kinder philoso- among children in tertiary care residential treatment: Prevalence and correlates.
phy to the management of outcome. Psychotherapy Bulletin, 41, 23–31. Child & Youth Care Forum, 43, 63–81.
Larson, A., Zuel, T., & Swanson, M. (2011). Are attendance gains sustained? A follow-up on Strijker, J., van Oijen, S., & Knot-Dickscheit, J. (2011). Assessment of problem behaviour by
the educational and child welfare outcomes of students with child welfare involve- foster parents and their foster children. Child & Family Social Work, 16, 93–100.
ment for educational neglect. Children & Schools, 33, 25–35. Sullivan, P.M., & Knutson, J.F. (2000). Maltreatment and disabilities: A population-based
Leiter, J. (2007). School performance trajectories after the advent of reported maltreat- epidemiological study. Child Abuse and Neglect, 10, 1257–1273.
ment. Children and Youth Services Review, 29, 363–382. Taussig, H.N., Harpin, S.B., & Maguire, S.A. (2014). Suicidality among preadolescent
Leon, S.C., Lyons, J.S., & Uziel-Miller, N.D. (2000). Variations in the clinical presentations of maltreated children in foster care. Child Maltreatment, 19, 17–26.
children and adolescents at eight psychiatric hospitals. Psychiatric Services, 51, Vis, S.A., Strandbu, A., Holtan, A., & Thomas, N. (2011). Participation and health—A re-
786–790. search review of child participation in planning and decision-making. Child &
Lewis-Morrarty, E., Dozier, M., Bernard, K., Terracciano, S.M., & Moore, S.V. (2012). Cogni- Family Social Work, 16, 325–335.
tive flexibility and theory of mind outcomes among foster children: Preschool follow- Wampold, B.E., & Imel, Z.E. (2015). The great psychotherapy debate Research evidence for
up results of a randomized clinical trial. Journal of Adolescent Health, 51, S17–S22. what works in psychotherapy (2nd ed.). New York: Routledge.
Lightfoot, E., Hill, K., & LaLiberte, T. (2011). Prevalence of children with disabilities in the Welch, G.L., & Bonner, B.L. (2013). Fatal child neglect: Characteristics, causation, and strat-
child welfare system and out of home placement: An examination of administrative egies for prevention. Child Abuse & Neglect, 37, 745–752.
records. Children and Youth Services Review, 33, 2069–2075. Winterfeld, A., & Feild, T. (2003, April). Tough problems, tough choices: Casey's guidelines
Lin, C. -H. (2012). Children who run away from foster care: Who are the children and for needs-based service planning in child welfare. Paper presented at the 14th National
what are the risk factors? Children and Youth Services Review, 34, 807–813. Conference on Child Abuse and Neglect, St. Louis, MO.
Lyons, J. (2008). Child and Adolescent Needs and Strengths (CANS) comprehensive multisys- Youn, S.J., Kraus, D.R., & Castonguay, L.G. (2012). The treatment outcome package: Facil-
tem assessment manual, 1–28. itating practice and clinically relevant research. Psychotherapy, 49(2), 115–122.
Lyons, J.S., Woltman, H., Martinovich, Z., & Hancock, B. (2009). An outcomes perspective
of the role of residential treatment in the system of care. Residential Treatment for
Children and Youth, 26, 71–91.