bs_bs_banner

Journal of Intellectual Disability Research doi: 10.1111/jir.12426

1

Brief Report

Concurrence of the strengths and difficulties questionnaire

and developmental behaviour checklist among children
with an intellectual disability
L. J. Rice,1 E. Emerson,2 K. M. Gray,4,5 P. Howlin,1,3 B. J. Tonge,4,5 G. L. Warner3
& S. L. Einfeld1
1 Brain and Mind Centre, University of Sydney, Camperdown, New South Wales, Australia
2 Centre for Disability Research and Policy, University of Sydney, Lidcombe, New South Wales, Australia
3 Institute of Psychiatry, Psychology and Neuroscience, King’s College, London, UK
4 Centre for Developmental Psychiatry and Psychology, Department of Psychiatry, School of Clinical Sciences at Monash Health,
Monash University, Victoria, Australia
5 Centre for Educational Development, Appraisal and Research, University of Warwick, Coventry, UK

Abstract SDQ and the DBC-P. We evaluated the concurrent

validity of the SDQ and DBC-P total scores, and
Background The Strengths and Difficulties
the agreement between the DBC-P cut-off and the
Questionnaire (SDQ) is widely used to measure
SDQ cut-offs for ‘borderline’ and ‘abnormal’
emotional and behavioural problems in typically
behaviour.
developing young people, although there is some
Results The SDQ total difficulties score correlated
evidence that it may also be suitable for children
well with the DBC-P total behaviour problem score.
with intellectual disability (ID). The Developmental
Agreement between the SDQ borderline cut-off and
Behaviour Checklist – Parent version (DBC-P) is a
the DBC-P cut-off for abnormality was high (83%),
measure of emotional and behavioural problems that
but was lower for the SDQ abnormal cut-off (75%).
was specifically designed for children and adoles-
Positive agreement between the DBC-P and the
cents with an ID. The DBC-P cut-off has high
SDQ borderline cut-off was also high, with the SDQ
agreement with clinical diagnosis. The aim of this
borderline cut-off identifying 86% of those who met
study was to estimate the relationship between
the DBC-P criterion. Negative agreement was
DBC-P and SDQ scores in a sample of children
weaker, with the SDQ borderline cut-off identifying
with ID.
only 79% of the participants who did not meet the
Method Parents of 83 young people with ID aged
DBC-P cut-off.
4–17 years completed the parent versions of the
Conclusion The SDQ borderline cut-off has some
validity as a measure of overall levels of behavioural
Correspondence: Professor Stewart Einfeld, Brain and Mind Centre, and emotional problems in young people with ID,
University of Sydney, 94 Mallet Street, Camperdown, New South
and may be useful in epidemiological studies that
Wales 2050, Australia. +612 9351 0885 (e-mail: stewart.
[email protected]) include participants with and without ID. However,

© 2017 MENCAP and International Association of the Scientific Study of Intellectual and Developmental Disabilities and
John Wiley & Sons Ltd
 Journal of Intellectual Disability Research
2
L. J. Rice et al. • Concurrence of the SDQ and DBC

where it is important to focus on behavioural profiles The Developmental Behaviour Checklist primary
in children with ID, a specialised ID instrument with carer version (DBC-P; Einfeld & Tonge, 2002;
established psychometric properties, such as the Einfeld et al., 2006; Tonge & Einfeld, 2003) was
DBC-P, may provide more reliable and valid specifically developed for young people with ID aged
information. 4–18 years, and is designed to measure a broad range
of emotional and behavioural problems. The DBC-P
Keywords concurrent validity, developmental
has five factor analytically derived sub-scales
behaviour checklist, intellectual disability, strength
producing a behavioural profile. The total scale score
and difficulties questionnaire
includes a clinical cut-off score, derived from a
Receiver Operating Characteristics (ROC) analysis,
The present investigation arose from the concurrence to differentiate between individuals with and without
of two studies, the Longitudinal Study of Australian clinically significant emotional and behavioural
Children (LSAC), a population study (Gray & Smart, problems. It can also be used as a repeated measure to
2009; Bayer et al., 2011) and the Stepping Stones monitor intervention effectiveness. The DBC-P cut-
Triple P (SSTP) Project, a public health community off has high agreement with clinical diagnosis (Einfeld
intervention for parents of children with & Tonge, 2002).
developmental disabilities aged 2–12 years (Tellegen The aim of this study was to estimate the
& Sanders, 2013). To assess behavioural disturbance, relationship between DBC-P and SDQ scores in the
the LSAC used the Strengths and Difficulties context of children with ID, (1) as a continuous
Questionnaire (SDQ), a measure designed for measure and (2) as a binary classifier registering
typically developing children; the SSTP project uses behaviour indicative of psychopathology. We
the Developmental Behaviour Checklist – Parent examined the concurrent validity of scores on the
version (DBC-P), a measure specifically for children DBC-P and SDQ as well as the positive, negative and
with disabilities. To assess the representativeness of overall agreement between DBC-P and SDQ cut-offs.
children receiving SSTP, it was necessary to compare High correlations between scores on the two
their level of behaviour problems with that of the measures as well as high agreement between the cut-
Australian LSAC intellectual disability (ID) offs would support the use of the SDQ in research
population. As a preliminary to this, our aim in the with children with ID.
present study was to estimate the relationship between
the SDQ and the DBC-P in children with ID.
The SDG (Goodman, 1997) is a widely used Method
measure of psychological well-being in children and
adolescents aged 4–17 years. It is cited in over 6000
publications and translated into almost a hundred
Participants
languages. It has high concurrent validity with other Eighty-three parents of children with ID (55 males, 29
measures of behavioural problems in typically females; mean age 10.8 years, SD 3.0, range 4–17)
developing children, including the Child Behaviour participated. Thirty-four participants (40%) were
Checklist (Goodman & Scott, 1999; Muris et al., recruited through two mental health service clinics in
2003) and the Rutter Parent and Teacher Australia (clinic group; 23 males, 11 females; mean
Questionnaire (Goodman, 1997). Although designed age 9.4 years, SD 2.8, range 5–15 years); the remain-
for typically developing children, there is some der (50; 60%) were recruited from a large-scale survey
evidence that the SDQ may also be suitable for of children with Down syndrome in England and
young people with ID. In both typically developing Wales (Down syndrome group; 32 males, 18 females;
and ID groups, inter-rater agreements between mean age 11.8 years, SD 2.8, range 8–17 years). In
parent, teacher and child reports are similar (Beck total, 50 (60%) participants had Down Syndrome; 28
et al., 2004; Emerson, 2005; Kaptein et al., 2008). (33%) had a diagnosis of Autism Spectrum Disorder;
The pattern of child self-reported SDQ scores is also 5 (6%) had other disorders (Prader-Willi syndrome,
consistent with ICD-10 psychiatric diagnoses in Costello syndrome, Tetrasomy 18p, global develop-
children with ID (Emerson, 2003, 2005). mental delay, cerebral palsy).

© 2017 MENCAP and International Association of the Scientific Study of Intellectual and Developmental Disabilities and
John Wiley & Sons Ltd
 Journal of Intellectual Disability Research
3
L. J. Rice et al. • Concurrence of the SDQ and DBC

Measures and negative agreement (Cicchetti & Feinstein,

1990). Overall agreement = total proportion of
The SDQ comprises 25 items, each rated on a 3-point
agreements, positive and negative; positive
scale (‘not true’, ‘somewhat true’ and ‘certainly true’).
agreement = index of agreement for positive
There are five sub-scales: Conduct Problems,
observations only; negative agreement = index of
Emotional Problems, Hyperactivity/Inattention, Peer
agreement for negative observations only
Relationships and Prosocial Behaviour. The first four
(Mackinnon, 2000). McNemar’s test for 2 × 2
sub-scales (20 items) are summed to generate the
contingency tables was used to determine whether
total difficulties score. Scores ≤ 13 are classified as
there was a statistical difference in the probabilities
normal; scores 14–16 are rated ‘borderline’; scores
between the two tests. Each of these analyses was
≥ 17 indicate abnormality.
conducted separately for the two cohorts (clinic group
The DBC-P (Einfeld & Tonge, 2002) has 96 items
and Down syndrome group) and then for the two
rated on a 3-point scale (‘not true as far as you know’,
groups combined.
‘somewhat or sometimes true’ and ‘very true or often
true’). There are five sub-scales (Disruptive
Behaviour, Self-absorbed, Communication Results
Disturbance, Anxiety and Social Relating Difficulties)
Table 1 summarises mean scores for each of the scales
and a total behaviour problem score provides an
and sub-scales for all participants. As might be
overall measure of behavioural/emotional
expected, total and sub-scale scores for the clinic
disturbance; (Einfeld & Tonge, 2002). The clinical
sample were generally higher than in the Down
cut-off for the DBC-P is 46. This was determined
syndrome group.
using the Likelihood Ratio method by examining the
Correlations between total scores on the SDQ and
quantitative power of the DBC-P to distinguish
DBC-P were high and significant (Table 2). Overall
psychiatric ‘cases’ and ‘non-cases’ determined by
agreement between proportions scoring above cut-off
clinical interviews conducted by psychiatrists with
on the two measures was high for the comparison
expertise in children with developmental disabilities.
between the DBC-P and SDQ borderline cut-off
Sensitivity and specificity of the DBC-P clinical cut-
(0.83) but lower for the SDQ abnormal cut-off (0.75).
off score were 83% and 85%, respectively; area under
Positive agreement between the measures was also
the ROC curve was 92% (Einfeld & Tonge, 2002).
higher for the DBC-P vs. SDQ borderline cut-off
The DBC-P has high concurrent validity with the
(86%) than for the abnormal SDQ cut-off (76%).
Maladaptive Behaviour Scale (Lambert &
Negative agreement was weaker, with 79% agreement
Windmiller, 1981) and the Problem Behaviour
for the SDQ borderline vs. DBC-P cut-off
section of the Scales of Independent Behaviours
comparison and 73% for the SDQ abnormal vs.
(Bruininks et al., 1988). Internal consistency, criterion
DBC-P comparison (Table 2). When the groups were
validity, test–retest reliability and inter-rater reliability
analysed separately, in the clinic sample, the
are high (Einfeld & Tonge, 2002).
borderline SDQ cut-off had good positive agreement
In the Australian sample, the DBC and SDQ were
(93%) but low negative agreement (60%) with the
completed at the same time in the clinic. In the UK
DBC-P cut-off. In the Down syndrome group, overall
sample, SDQs were completed by post; DBC’s were
agreement between the measures was lower, but
completed some weeks later in interviews with
negative agreement was slightly better than the
parents.
positive agreement (Table 1).
Proportions of participants scoring above and
Statistical analysis
below the DBC-P and SDQ cut-offs are presented in
Partial correlations were performed to explore the Table 3. Overall, the SDQ abnormal cut-off classified
relationship between total scores on the DBC-P and fewer children as meeting criteria for emotional and
SDQ, controlling for age, gender and syndrome behavioural problems than the DBC-P (46% vs. 60%)
(Down syndrome, Autism, Other). To test level of although the difference was only significant for the
agreement between DBC-P and SDQ cut-offs Down syndrome group where only 26% scored above
(borderline and abnormal), we used overall, positive the SDQ abnormal cut-off (χ2(1) = 5.40, p = .002).

© 2017 MENCAP and International Association of the Scientific Study of Intellectual and Developmental Disabilities and
John Wiley & Sons Ltd
 Journal of Intellectual Disability Research
4
L. J. Rice et al. • Concurrence of the SDQ and DBC

Table 1 DBC-P and SDQ total raw score and sub-scale raw scores

Down syndrome group Mean (SD) Clinic group Mean (SD) Combined Mean (SD)
(N = 50) (N = 34) (N = 84)

DBC-P total score 46.9 (26.8) 69.8 (3.3) 56.1 (27.7)

DBC-P sub-scales
Disruptive Behaviour 14.0 (8.6) 24.4 (10.45) 18.2 (10.7)
Self-absorbed 15.3 (11.9) 19.4 (9.0) 17.0 (10.9)
Communication 6.9 (4.8) 9.5 (4.5) 7.9 (4.8)
Disturbance
Anxiety 4.2 (2.9) 7.9 (4.0) 5.7 (3.7)
Social Relating Difficulties 3.8 (3.7) 6.7 (3.3) 5.0 (3.9)
SDQ total score 14.1 (5.9) 21.6 (5.6) 17.13 (6.8)
SDQ sub-scales
Conduct Problems 2.4 (1.8) 3.9 (2.2) 3.0 (2.1)
Emotional Problems 2.0 (1.6) 5.2 (2.8) 3.3 (2.7)
Hyperactivity/Inattention 5.7 (2.5) 8.0 (1.8) 6.6 (2.5)
Peer Relationships 4.0 (2.6) 4.2 (2.3) 4.1 (2.5)
Prosocial Behaviour 5.3 (3.0) 5.2 (2.5) 5.2 (2.8)

DBC-P, Developmental Behaviour Checklist – Parent version; SDQ, Strengths and Difficulties Questionnaire.

Table 2 Agreement between SDQ and DBC-P cut-off scores

large scale population studies because of the relative
ease and speed of completion. However, evidence of
reliability and validity when used with cohorts of non-
Analysis Down syndrome Clinic Total
typically developing children is limited. The aim of
the present study was to estimate the relationship
Partial correlation 0.66* 0.77* 0.74*
between the SDQ and an alternative measure of
Overall agreement
SDQ-Borderline 0.80 0.88 0.83 emotional and behaviour problems, the DBC-P,
SDQ-Abnormal 0.70 0.82 0.75 which is specifically designed for individuals with ID
Positive agreement and has been shown to have high agreement with
SDQ-Borderline 0.77 0.93 0.86 clinical diagnosis. Data were obtained from two
SDQ-Abnormal 0.67 0.89 0.76
convenience samples: a group of children with Down
Negative agreement
SDQ-Borderline 0.82 0.60 0.79 syndrome involved in a large survey study and a group
SDQ-Abnormal 0.77 0.57 0.73 recruited through mental health clinics for children
with developmental disabilities, most of whom also
*P < 0.001. had a diagnosis of Autism Spectrum Disorder. As
Note. Developmental Behaviour Checklist – Parent version (DBC-P) expected, the clinic sample scored higher than the
cut-off = 46. Strengths and Difficulties Questionnaire (SDQ) ab-
Down syndrome group on almost all sub-scales. The
normal cut-off = 17; SDQ borderline cut-off = 14.
SDQ abnormal cut-off classified fewer children as
meeting criteria for emotional and behavioural
When comparisons were based on the SDQ
problems than the DBC-P (46% vs. 60%), but when
borderline cut-off, proportions scoring above the cut-
comparisons were based on the SDQ borderline cut-
offs were similar (DBC-P 60%, SDQ 62%).
off, proportions scoring above cut-off were similar
(SDQ 62%, DBC-P 60%). The SDQ abnormal cut-
off also had lower positive and negative agreement
Discussion
with the DBC-P than the SDQ borderline cut-off,
The SDQ (Goodman, 1997) is one of the most widely suggesting that the borderline cut-off may be a better
used measures of behavioural and emotional indicator of emotional and behavioural problems in
problems in young people. It is particularly useful in children with intellectual disabilities.

© 2017 MENCAP and International Association of the Scientific Study of Intellectual and Developmental Disabilities and
John Wiley & Sons Ltd
 Journal of Intellectual Disability Research
5
L. J. Rice et al. • Concurrence of the SDQ and DBC

Table 3 Proportions of participants scoring above and below the DBC-P and SDQ cut-offs

DBC-P SDQ borderline SDQ abnormal

Down syndrome
Below cut-off 28 (56%) 28 (56%) 37 (74%)
Above cut-off 22 (44%) 22 (44%) 13 (26%)
2 2
McNemar’s Test Χ (1) = 0.00, P = 1 Χ (1) = 5.40, P = 0.002
Clinic sample
Below cut-off 6 (18%) 4 (12%) 8 (24%)
Above cut-off 28 (82%) 30 (88%) 26 (76%)
2 2
McNemar’s Test Χ (1) = 1, P = 0.32 Χ (1) = 0.67, P = 0.41
Combined
Below cut-off 34 (40%) 32 (38%) 45 (54%)
Above cut-off 50 (60%) 52 (62%) 39 (46%)
2 2
McNemar’s Test Χ (1) = 0.29, P = 0.59 Χ (1) = 5.76, P = 0.02

The study has several limitations. Data are based and specificity for detecting differences between
on convenience samples involving mainly children individuals with and without clinically significant
with Down syndrome and autism, and although all emotional and behavioural problems unknown. Other
were classified as having ID, details of intellectual important psychometric properties for children with
functioning were not available. The samples were also ID, for example, sensitivity to change and inter-rater
recruited from very different sources, with the clinic agreement, are also missing. Thus, where it is
group, as expected having higher levels of behaviour important to focus on behavioural profiles in children
problems than the survey group. However, this with ID, as in the SSTP Project (Tellegen & Sanders,
provided a wide range of levels of 2013), the DBC-P may be preferable. The DBC-P is
emotional/behavioural problems, and the association also likely to be more informative in clinical settings
between the DBC-P and the SDQ borderline score when these features are particularly important
was confirmed in both cohorts. In the Down (Chandler et al., 2016).
syndrome group, there was also a delay between
parents’ completing the SDQ and the DBC-P, which
Conflict of Interest
may have reduced the agreement between scores on
the two measures. Finally, there was no direct Professor Einfeld and Emeritus Professor Tonge are
assessment of the children’ s behavioural and mental the authors of the Developmental Behaviour
state and the lack of comparison with clinical Checklist. They receive royalties for the German
psychiatric diagnoses for the SDQ, and the language version but derive no personal or financial
concurrent direct psychiatric diagnosis and DBC-P benefit from the sale of any of other versions. No
data are derived from a previous study (Einfeld & other authors reported any financial disclosures.
Tonge, 1995).
Despite these caveats, our findings suggest that in
epidemiological studies such as the LSAC population
References
study (Gray & Smart, 2009; Bayer et al., 2011), which
include both typically developing and atypical Bayer J. K., Ukoumunne O. C., Lucas N., Wake M., Scalzo
participants, the SDQ borderline cut-off has validity K. & Nicholson J. M. (2011) Risk factors for childhood
as a measure of overall levels of behavioural and mental health symptoms: national longitudinal study of
Australian children. Pediatrics 128, e1–e15.
emotional problems in children with ID. It is thus of
Beck A., Daley D., Hastings R. P. & Stevenson J. (2004)
potential value when comparisons are needed with the Mothers’ expressed emotion towards children with and
general population. However, the SDQ lacks ROC without intellectual disabilities. Journal of Intellectual
data for children with ID, which makes its sensitivity Disability Research 48, 628–38.

© 2017 MENCAP and International Association of the Scientific Study of Intellectual and Developmental Disabilities and
John Wiley & Sons Ltd
 Journal of Intellectual Disability Research
6
L. J. Rice et al. • Concurrence of the SDQ and DBC

Bruininks R. H., Hill B. K. & Morreau L. E. (1988) Goodman R. & Scott S. (1999) Comparing the Strengths
Prevalence and implications of maladaptive behaviors and and Difficulties Questionnaire and the Child Behavior
dual diagnosis in residential and other service programs. Checklist: is small beautiful? Journal of Abnormal Child
Springer-Verlag, New York. Psychology 27, 17–24.
Chandler S., Howlin P., Simonoff E., O’sullivan T., Tseng Gray M. & Smart D. (2009) Growing Up in Australia: The
E., Kennedy J. et al. (2016) Emotional and behavioural Longitudinal Study of Australian Children: A valuable
problems in young children with autism spectrum disorder. new data source for economists. Australian Economic
Developmental Medicine and Child Neurology 58, 202–8. Review 423, 367–76.
Cicchetti D. V. & Feinstein A. R. (1990) High agreement Kaptein S., Jansen D. E. M. C., Vogels A. G. C. &
but low kappa: I. The problems of two paradoxes. Journal Reijneveld S. A. (2008) Mental health problems in
of Clinical Epidemiology 43, 543–9. children with intellectual disability: use of the Strengths
Einfeld S. L., Piccinin A. M., Mackinnon A., Hofer S. M., and Difficulties Questionnaire. Journal of Intellectual
Taffe J., Gray K. M. et al. (2006) Psychopathology in Disability Research 52, 125–31.
young people with intellectual disability. The Journal of the Lambert N. M. & Windmiller M. (1981) AAMD Adaptive
American Medical Association 296, 1981–9. Behavior Scale, School edn. McGraw Hill, Montley, CA.
Einfeld S. L. & Tonge B. J. (1995) The Developmental Mackinnon A. (2000) A spreadsheet for the calculation of
Behavior Checklist: The development and validation of an comprehensive statistics for the assessment of diagnostic
instrument to assess behavioral and emotional disturbance tests and inter-rater agreement. Computers in Biology and
in children and adolescents with mental retardation. Medicine 30, 127–34.
Journal of Autism and Developmental Disorders 25, 81–104. Muris P., Meesters C. & van den Berg F. (2003) The
Einfeld S. L. & Tonge B. J. (2002) Manual for the Strengths and Difficulties Questionnaire (SDQ) -further
Developmental Behaviour Checklist: Primary carer version evidence for its reliability and validity in a community
(DBC-P) & teacher version (DBC-T). University of New sample of Dutch children and adolescents. European Child
South Wales, Sydney and Monash University, Melbourne. and Adolescent Psychiatry 12, 1–8.
Emerson E. (2003) Prevalence of psychiatric disorders in Tellegen C. L. & Sanders M. R. (2013) Stepping Stones
children and adolescents with and without intellectual Triple P-Positive Parenting Program for children with
disability. Journal of Intellectual Disability Research 47, 51–8. disability: A systematic review and meta-analysis. Research
Emerson E. (2005) Use of the Strengths and Difficulties in Developmental Disabilities 34, 1556–71.
Questionnaire to assess the mental health needs of children Tonge B. J. & Einfeld S. L. (2003) Psychopathology and
and adolescents with intellectual disabilities. Journal of intellectual disability: The Australian child to adult
Intellectual and Developmental Disability 30, 14–23. longitudinal study. International Review of Research in
Goodman R. (1997) The Strengths and Difficulties Mental Retardation 26, 61–91.
Questionnaire: a research note. Journal of Child Psychology
and Psychiatry 38, 581–6. Accepted 6 September 2017

© 2017 MENCAP and International Association of the Scientific Study of Intellectual and Developmental Disabilities and
John Wiley & Sons Ltd