Data Loading...
A Systematic Review of Meta-Analyses of Psychosocial ... Flipbook PDF
A Systematic Review of Meta-Analyses of Psychosocial Treatment for Attention-Deficit/Hyperactivity Disorder Gregory A. Fa
113 Views
93 Downloads
FLIP PDF 1.16MB
Clin Child Fam Psychol Rev (2015) 18:77–97 DOI 10.1007/s10567-015-0178-6
A Systematic Review of Meta-Analyses of Psychosocial Treatment for Attention-Deficit/Hyperactivity Disorder Gregory A. Fabiano • Nicole K. Schatz • Ariel M. Aloe • Anil Chacko • Andrea Chronis-Tuscano
Published online: 18 February 2015 Ó Springer Science+Business Media New York 2015
Abstract The present report synthesizes outcomes across meta-analyses of psychosocial (i.e., non-pharmacological) treatments for ADHD. A total of 12 meta-analyses were identified that met search criteria. The meta-analyses were notable in that there was surprisingly little overlap in studies included across them (range of overlap was 2–46 %). Further, there was considerable diversity across the meta-analyses in terms of the inclusion/exclusion criteria, types of psychosocial treatments reviewed, methodological characteristics, and magnitude of reported effect sizes, making it difficult to aggregate findings across metaanalyses or to investigate moderators of outcome. Effect sizes varied across the outcomes assessed, with meta-analyses reporting positive and significant effect sizes for measures of some areas of child impairment (e.g., social impairment) and small and more variable effect sizes for distal and/or untargeted outcomes (e.g., academic achievement). Results are reviewed in light of the larger literature on psychosocial interventions for ADHD, and specific recommendations for future meta-analyses of psychosocial treatments for ADHD are offered.
G. A. Fabiano (&) N. K. Schatz University at Buffalo, SUNY, 334 Diefendorf Hall, Buffalo, NY 14214, USA e-mail: [email protected] A. M. Aloe University at Northern Iowa, Cedar Falls, IA, USA A. Chacko New York University, New York, NY, USA A. Chronis-Tuscano University of Maryland, College Park, MD, USA
Keywords Attention-deficit/hyperactivity disorder Meta-analysis Psychosocial treatment
A Review of Meta-Analyses of Psychosocial Treatment for Attention-Deficit/Hyperactivity Disorder: Systematic Synthesis and Interpretation Attention-deficit/hyperactivity disorder (ADHD) is a chronic disorder, with onset in early childhood, characterized by developmentally inappropriate levels of inattention, overactivity, and impulsivity that result in impaired functioning across important domains of daily life (American Psychiatric Association 2013). ADHD is notable in that it is a prevalent disorder, with an average of one to two children in every classroom in the USA estimated as having behaviors consistent with ADHD (Fabiano et al. 2013; Visser et al. 2010). The challenges associated with ADHD result in considerable social, occupational, and academic problems for youth and their families (Fabiano et al. 2006; Kent et al. 2011), as well as economic consequences (Robb et al. 2011; Pelham et al. 2007). The personal and societal costs of ADHD have resulted in efforts to identify and disseminate effective treatments. At the present time, two broad treatment modalities are commonly employed—stimulant medication (Conners 2002) and psychosocial interventions (defined broadly). Across the field, the short-term efficacy of stimulant medication is agreed upon based on a sizable evidence base (Conners 2002; Faraone et al. 2006). Indeed, professional guidelines recommend medication as a first-line intervention based on this research (American Academy of Child and Adolescent Psychiatry (AACAP) 2007; American Academy of Pediatrics (AAP) 2011). Endorsements of psychosocial interventions for ADHD are less clear, and
123
78
this is reflected in professional guideline recommendations. For example, the AAP guidelines classify the strength of evidence for stimulant medications as stronger for elementary- and adolescent-aged children with ADHD, relative to psychosocial treatments. Likewise, the AACAP guidelines (2007) state: ‘‘It seems well established that pharmacological intervention for ADHD is more effective than a behavioral treatment alone’’ (pp. 903). In contrast, criterion-based reviews of the psychosocial treatment literature support the efficacy of these interventions for ADHD (Evans et al. 2013; Pelham and Fabiano 2008; Pelham et al. 1998). Some meta-analytic reviews also present clear and strong support for psychosocial intervention (e.g., DuPaul and Eckert 1997; Fabiano et al. 2009). However, others present results that are equivocal (e.g., Sonuga-Barke et al. 2013; Zwi et al. 2011). Unfortunately, this variability in reported support for psychosocial treatments provides challenges for practitioners and families attempting to choose viable treatment approaches. A plausible explanation for these inconsistent findings may be that there is variability in research questions, inclusion criteria, and study methodology across meta-analyses. More specifically, there are multiple aspects of meta-analytic research design that may vary, and these variations may affect the conclusions reached regarding the efficacy of psychosocial treatment for ADHD. Sources of potential variability in meta-analytic research design include: (1) type(s) of psychosocial interventions for ADHD included in meta-analyses; (2) specific constructs and measures used as indicators of treatment response; (3) inclusion criteria (e.g., related to publication date; related to treatment, study design, or sample characteristics) used to identify individual studies; and (4) the methods used in the meta-analysis, most notably the calculation and analysis of effect sizes. Each of these issues will be addressed briefly, in turn. Type of Treatments Included in Meta-Analyses In contrast to stimulants where the treatment is relatively homogenous, the category of psychosocial interventions represents a heterogenous group of approaches. The most commonly studied treatments are behavioral interventions that include training parents and teachers to manipulate environmental antecedents and consequences to promote appropriate child behavior and improve parenting. Criterion-based reviews strongly support the efficacy of these interventions (Abramowitz and O’Leary 1991; Evans et al. 2013; Pelham and Fabiano 2008; Pelham et al. 1998). Psychosocial treatments may also include interventions to train youth in adaptive functioning skills (e.g., organizational skills, social skills; Evans et al. 2013). Other psychosocial interventions such as cognitive therapy (Abikoff
123
Clin Child Fam Psychol Rev (2015) 18:77–97
1991) or individual neurocognitive training (Chacko et al. 2013; Rapport et al. 2013) have not evinced comparable levels of empirical support (AACAP 2007) but are also included in the broad category of psychosocial treatment in some meta-analytic work (Hodgson et al. 2012). Further, one-to-one counseling, play or other types of therapies, and social skills training are included in this category, in spite of limited evidence of efficacy (Hoagwood et al. 2000). To the extent that a meta-analysis combines studies of these interventions with less empirical support with behavioral approaches with greater empirical support into an overall effect of psychosocial treatment, the effect of the more effective intervention will be diluted by the inclusion of the less efficacious intervention. Furthermore, meta-analyses that combine multiple types of psychosocial interventions into a single effect will likely yield different results than would meta-analyses that do not collapse across different types of interventions in this manner. Measurement of Outcome The approach used for measurement of treatment outcome is also an important parameter to consider in the review of meta-analyses of ADHD treatment. Unlike studies of medication treatments for ADHD where parent and teacher ratings of ADHD symptoms are primarily used as measures of outcome (Conners 2002), psychosocial treatment studies utilize a broader array of outcome measures (e.g., parent and teacher ratings, observations of child behavior and parenting behavior, academic outcomes; DuPaul et al. 2012; Fabiano et al. 2009). This presents a unique challenge for the synthesis of findings across studies and may contribute to variability in conclusions drawn across metaanalyses. Some meta-analyses may present separate effects for each type of outcome measure, some meta-analyses may group effect sizes into overarching categories, and others collapse across dependent measures to create a single effect size to represent each study. Whereas some meta-analyses may utilize traditional symptom-based outcomes, others may include outcomes that would not typically be designated as primary outcome measures in studies of ADHD treatment (e.g., internalizing symptoms; Zwi et al. 2011). Also contributing to the variability in outcome measurement, some meta-analyses emphasize the use of blinded measures of treatment effect (e.g., SonugaBarke et al. 2013). These all may be viable approaches, but specific measures should be viewed within the lens of the overarching conceptual model guiding the study. For instance, observations of parenting are proximal outcomes in studies of behavioral parent training, but they are distal or peripheral in a study of behavioral treatment implemented by teachers within a classroom. Thus, to the extent that meta-analyses include different outcome measures, and
Clin Child Fam Psychol Rev (2015) 18:77–97
these measures are either proximal or distal, findings may vary. Inclusion Criteria for the Meta-Analysis The ADHD psychosocial treatment literature emerged in the 1960s and continues to grow at the present time. This literature encompasses a variety of interventions, measures, and research designs. Variations on inclusion and exclusion criteria within a meta-analysis for the ADHD psychosocial treatment literature may result in heterogeneity in studies included across meta-analyses, and this might result in limitations in conclusions that can be generalized to the entire population of ADHD intervention studies (Cooper and Hedges 1994). For instance, a potential source of variability across meta-analyses concerns the design used in individual treatment outcome studies. The ADHD psychosocial treatment literature is comprised of a diverse group of study designs ranging from single-subject designs to randomized trials, and meta-analyses may have varying inclusion criteria related to design type (e.g., including all design types, including only randomized controlled trials, etc.). The degree to which studies from the larger literature are included within any given meta-analysis needs to be investigated to determine whether meta-analyses by different groups serve as replications or as independent metaanalyses. Methods used to Calculate Effect Sizes There are multiple approaches that investigators can use to quantify psychosocial treatment effects through the calculation of effect sizes. In addition, the preceding discussion impacts these calculations as meta-analysts must grapple with diverse study measures, informants, research designs, and approaches for generating estimates of effect size, a problem also present in the stimulant medication literature (Faraone et al. 2006). Within the psychosocial treatment literature, problems with the design and implementation of primary studies included within a meta-analysis weaken the validity of the findings. This is known in meta-analysis as risk of bias (Higgins and Green 2011), and every primary study included in a meta-analysis is exposed to bias regardless of the specific design. Meta-analyses that include only randomized controlled trials generally tend to have a lower risk of selection bias. However, there are other sources of bias even within these trials such as attrition or more rigorous inclusion criteria, which limits generalizability. Further, until recently, there were few viable solutions for comparing outcomes from randomized trials to crossover and single-subject design studies (Hedges et al. 2012; Shadish et al. 2013). Thus, individual meta-analyses have historically dealt with these issues in a variety of ways,
79
which may have implications for the conclusions reached by each. Understanding the differences in approaches to the calculation of effect sizes is important as these differences may facilitate or preclude direct comparisons. Given the inconsistency in findings across meta-analyses of psychosocial treatments for ADHD, there is a need to conduct a systematic review of these meta-analyses. Further, the reasons for differences across meta-analyses should be explored. In the present review, the results of published meta-analyses for ADHD psychosocial treatments were systematically reviewed in an effort to document the efficacy of such treatments for ADHD. It was specifically hypothesized that discrepancies across studies could be explained by methodological differences related to study design, the outcomes assessed, and inclusion criteria. Potential explanations for divergent findings are also explored. Finally, alignment between the results of the ADHD meta-analyses and the systematic reviews of the literature aimed at identifying evidence-based interventions (Evans et al. 2013; Pelham and Fabiano 2008; Pelham et al. 1998) were also assessed.
Methods In conducting this analysis, recommendations made in standard texts on research synthesis were used to guide procedures (Cooper and Hedges 1994; Cooper et al. 2009; Hunter and Schmidt 2004). Although the original purpose was to synthesize meta-analytic data regarding the efficacy of behavioral interventions for the treatment for ADHD, a review of existing meta-analyses revealed there was not a consistent differentiation between types of psychosocial interventions (e.g., behavioral parent training, cognitive training) to permit aggregation. Therefore, we included in our systematic review all meta-analyses of psychosocial interventions for children and adolescents with ADHD in an effort to be as inclusive as possible. Literature Review Studies included in this review were identified using five main techniques. First, literature searches using PsycInfo and PUBMED were conducted. Search criteria entered into the database included: attention-deficit/hyperactivity disorder, meta-analysis, treatment, and intervention. Based on the results of the computerized search, articles were identified that met the inclusion criteria described below. Each identified article’s reference section was then systematically analyzed, and additional meta-analyses were added to the review in this way. Unpublished meta-analyses were excluded from the review because one goal of the systematic review was to focus on publications likely to
123
80
influence public policy or professional recommendations/ practice parameters. The literature search was terminated in June 2013. Inclusion Criteria A meta-analysis was included in the initial collection based on the following criteria: (1) the meta-analysis reported data from studies of psychosocial interventions for children (i.e., under age 18) with ADHD; (2) the meta-analysis inclusion criteria required that the majority of participants in the individual studies were diagnosed with ADHD or significantly well described to suggest the characteristic behaviors of ADHD (e.g., ‘‘hyperactive,’’ ‘‘inattentive’’). Studies that focused on treatment for children with externalizing behavior problems alone (e.g., ODD, CD, aggressive behavior) were not included; (3) the meta-analysis must provide at least one effect size that summarizes the results of individual treatment outcome studies included; and (4) at least one effect size reported by the meta-analysis must reflect the unique treatment effect of psychosocial interventions for ADHD. That is, meta-analyses that only provided data for combined treatment effects of medication and psychosocial interventions were excluded. The initial search was conducted by the first two authors, which identified 233 total independent reports. After reviewing abstracts, 17 were identified as requiring closer review. The full texts of these 17 papers were independently reviewed by the first two authors to determine eligibility. Disagreements were resolved by consensus. Of these 17 meta-analyses, five were excluded as they did not meet the inclusion criteria. These studies were not included for the following reasons: inclusion of a heterogeneous sample of participants that were not clearly identified as having ADHD (Baer and Nietzel 1991); inclusion of only combined pharmacological and psychosocial treatments (Majewicz-Hefley and Carlson 2007); no report of meta-analytic results (Jadad et al. 1999a; Schachar et al. 2002); or combination of pharmacological and psychosocial treatments in a manner where they could not be disentangled (Shaw et al. 2012). In total, 12 meta-analyses on ADHD treatment were identified for inclusion in the review. Study Characteristics Each meta-analysis collected for the review was coded on a number of domains. These domains included: study design, subject characteristics, inclusion criteria, description of the treatments, outcomes assessed, approach to calculating effect sizes, and results. Coders completed a standardized form for each study, and coders met to discuss coding and reach consensus regarding any discrepancies.
123
Clin Child Fam Psychol Rev (2015) 18:77–97
The text and reference sections of each meta-analysis were systematically examined to determine the level of overlap among meta-analyses with regard to the sample of individual treatment outcome studies included in each analysis. A table was created that included every individual treatment outcome study included in any one of the metaanalyses. The reference sections of each meta-analysis were carefully cross-referenced to identify which individual treatment outcome studies were included in each meta-analysis. Each individual psychosocial treatment outcome study was then identified. Studies were then crossreferenced across meta-analyses. The complete list of studies is available from the study authors. Qualitative Review of Information Included in each MetaAnalysis In this systematic review, the extent to which information was included within each meta-analysis consistent with the Meta-Analysis Reporting Standards (MARS; APA Publications and Communications Board Working Group on Journal Article Reporting Standards 2008) was rated. A checklist was constructed that included each item within the standards and raters checked whether the information for the item was present or absent within the meta-analysis. Reliability of Coding For the characteristics of study designs included in Table 1, two coders independently coded 17 % of the included meta-analysis, and percent agreement was 82 %. For the coding of the MARS criteria, the first two authors independently reviewed each of the meta-analyses, and any discrepancies were discussed and consensus was obtained through a review of the primary source text. Then, the percent of information that was included for each of the MARS categories (Method, Search Procedures, Coding Procedures, Statistics Methods, and Results Reporting) was calculated along with an overall score. It is important to note that in this systematic review, these ratings are indicators of whether information needed to judge the quality of the meta-analysis was included; they are not a direct, overall judgement of the meta-analytic quality as many contextual variables influence quality beyond the presence or absence of information.
Results Overview Twelve meta-analyses were identified through the systematic search to include in this review. See Table 1 for a
Clin Child Fam Psychol Rev (2015) 18:77–97
81
Table 1 Summary of meta-analyses included in the systematic review Study
Years included
k of studies
Inclusion criteria
Exclusion criteria
Design
Treatment
Results
Charach et al. (2013)
1980– November 2011
14
Diagnosed with ADHD, ODD, CD
Studies of interventions related to diet, biofeedback or massage
Majority RCT
Behavioral parent training
Disruptive behavior (d = .75); parenting skills (d = .55); ADHD symptoms (d = .77)
Corcoran and Dattalo (2006)
1970–2003
16
ADHD diagnosis; younger than 18
18 or older
Comparison group included (randomization not required)
Parents were part of the intervention
Teacher (d = .75); parent (d = .43); child (d = .11); child academic performance (d = 8.20); child family functioning (d = .67); child internalizing (d = .64); child ADHD (d = .39); Child externalizing (d = .36); child social competence (d = .07); child selfcontrol (d = -1.67)
DuPaul and Eckert (1997)
1974–1995
8 betweensubject; 17 withinsubject; 38 singlesubject
Kindergarten through 12th grade; diagnosed with ADHD
Intervention not schoolbased
Between-group, within-subject, single-subject
School-based treatment for ADHD
Between-group: academic (d = .31), behavioral (d = .45); within-subject: academic (d = .31), behavioral (d = .64), clinic-based (d = .37); singlesubject academic (d = .82), behavioral (d = 1.16)
DuPaul et al. (2012)
1996–2010
7 betweensubject; 28 withinsubject; 65 singlesubject
Kindergarten through 12th grade; diagnosed with ADHD
Published prior to 1996
Between-group, within-subject, single-subject
School-based treatment for ADHD
Between-group: academic (d = .43), behavioral (d = .18); Within-subject: academic (d = .42), behavioral (d = .72); single-subject academic (d = 3.48), behavioral (d = 2.20)
Fabiano et al. (2009)
1968–2008
Between-group, pre–post, within-subject, single-subject
Behavioral intervention
Between-group (d = .74); Pre–post (d = .70); withinsubject (d = 2.64); single-subject (d = 3.78)
1994–2009
Child had ADHD or ADHD consistent behaviors; under 18 years of age Youth under 18 diagnosed with DSMIV ADHD
Older than 18 years old
Hodgson et al. (2012)
20 betweensubject; 30 pre–post design; 23 withinsubject; 101 singlesubject 14
Comorbidities including ODD, CD, LD
Not reported
Nonpharmacological
Organized by modality and outcome
123
82
Clin Child Fam Psychol Rev (2015) 18:77–97
Table 1 continued Study
Years included
k of studies
Inclusion criteria
Exclusion criteria
Design
Treatment
Results
Klassen et al. (1999)
Studies After 1981
2
Youth with ADHD under 18; Assessment with parent and/or teacher ratings
Not reported
Random assignment to groups
Psychological/ behavioral
Teacher (d = .40); Parent (d = .49)
Lee et al. (2012)
1970–2011
40
Youth with ADHD; Parent training Intervention; Comparison group
Not reported
Included a comparison group
Parent training
(r = .34)
Purdie et al. (2002)
1990–1998
74
Primary diagnosis of ADHD
Not reported
All designs aggregated together
Nonpharmacological
School-based/education (d = .39); behavioral training (d = .50); parent training (d = .31)
SonugaBarke et al. (2013)
Up to April 3, 2012
15
Children 3–18 diagnosed with ADHD or clinically elevated symptom ratings; RCT—comparison condition could include sham/placebo, attention/active control, treatment as usual, or waitlist
Studies of rare disorders (e.g., genetic disorders)
Randomized controlled trial
Behavioral intervention
Most proximal assessment (d = .40); probably blinded assessment (d = .02)
Van de Oord et al. (2008)
January 1, 1985– September 30, 2006
24
Children 6–12 diagnosed with ADHD
Not reported
Intervention trial
Behavioral or cognitivebehavioral
Parent ADHD symptoms (d = .87), ODD ratings (d = .66), social behavior (d = .54); teacher ADHD symptoms (d = .75), ODD ratings (d = .43), social behavior (d = .71); academic performance (d = .19)
Zwi et al. (2011)
Before September 2010
5
Child 5–18 diagnosed with ADHD
Any study that also included a childfocused intervention
Randomized controlled trial
Parent training
Externalizing behavior (d = -.32); internalizing behavior (d = -.48)
In the table, d indicates the effect size estimate from the meta-analysis. Positive effect sizes represent improvement from psychosocial treatment. The reader is cautioned that these d effect sizes cannot be directly compared as they represent different calculation methods across meta-analyses (see Table 5) ADHD attention-deficit/hyperactivity disorder, ODD oppositional defiant disorder, CD conduct disorder, LD learning disability, RCT randomized controlled trial, d represents the effect size calculated by the meta-analysts, see formulas in Table 5
complete list (note Lee et al. 2012 is not included in the figures as these data could not be made comparable to the other reports because the specific number of studies contributing to each effect size (i.e., k1) reported by Lee and colleagues could not be determined). Interventions included within meta-analyses were varied with some focusing solely 1
In this review, the denotation k refers to the total number of studies included within a meta-analysis or contributing to a particular effect size and N refers to the number of subjects in a single study.
123
on particular settings (e.g., school; DuPaul and Eckert 1997, 2012) and others focusing solely on a particular type of intervention (e.g., parent training without any associated child intervention; Zwi et al. 2011). To provide supporting information for a discussion of the aggregate results, each meta-analysis is briefly summarized in Table 1. Aggregate results across meta-analyses are reported. First, aggregate results from meta-analyses are grouped by study designs: (1) between-group, (2) pre–post design, (3) single-subject, (4) within-subject, and (5) mixed designs.
Clin Child Fam Psychol Rev (2015) 18:77–97
83
Fig. 1 Forest plot of effect sizes reported from the identified metaanalyses. The study label is on the left side of the figure, the effect size estimate is illustrated in the figure, and on the right side, k studies
that contributed to the effect size, the effect size, and confidence interval are reported
These descriptive results include the effect size generated within the meta-analyses (for some meta-analyses, multiple effect sizes were generated) in a single graphic (Fig. 1). Then, aggregate results are presented by outcome measure (Figs. 2, 3, 4, and 5). Note that the included meta-analyses reported the standardized mean difference (d) in different ways (see expanded discussion below), making direct comparisons of effect sizes across meta-analyses imprudent. Synthesis across meta-analyses related to the types of interventions included, measurement outcomes, inclusion criteria, and method for calculating effect sizes are then
reviewed. Finally, the results of this systematic review are compared with the results of criterion-based reviews (Evans et al. 2013; Pelham and Fabiano 2008; Pelham et al. 1998). Overlap of Studies Included within Meta-Analyses Table 2 illustrates the overlap of studies included in each meta-analysis with the entire population of studies across all the meta-analyses in this review. It is readily apparent that there is little overlap in the individual treatment studies included across the meta-analyses included in this review
123
84
Clin Child Fam Psychol Rev (2015) 18:77–97
Fig. 2 Illustration of effect size estimates based on total number of studies used to generate the effect size estimate within the metaanalyses for between-group design studies. The study label is on the
left side of the figure, the effect size estimate is illustrated in the figure, and on the right side, k studies that contributed to the effect size, the effect size, and confidence interval are reported
(k = 12), with the percentage of overlapping studies ranging from 2 % (Zwi et al. 2011) to 46 % (Fabiano et al. 2009). No meta-analysis included even half of the total population of ADHD psychosocial intervention studies, defined as the total number of studies included across all 12 meta-analyses. Table 3 illustrates the overlap of included studies for each meta-analysis with the other meta-analyses included in this report. This lack of overlap is important to consider, as it indicates that these meta-analyses are largely reporting on independent samples of studies rather than replicating one another.
Aggregate Results Across Meta-Analyses
123
Figure 1 illustrates a forest plot for the effect sizes reported within each meta-analysis. It is important to note that some studies included only a single effect size, whereas others included multiple effect sizes. As can be seen from the plot, there is considerable variability across the different studies in the magnitude of effect size (e.g., range = -1.53 to .67 for between-group designs), number of studies included within the meta-analysis (k ranges from 1 to 44 across study designs), and the confidence interval for each effect
Clin Child Fam Psychol Rev (2015) 18:77–97
85
Fig. 3 Effect sizes for specific outcomes in within-group design studies. The study label is on the left side of the figure, the effect size estimate is illustrated in the figure, and on the right side, k studies that contributed to the effect size, the effect size, and confidence interval are reported
size. Further, there is diversity in the research designs included within the meta-analyses (see Fig. 1). Integration of Results Across Meta-Analyses Type of Treatments Included in the Meta-Analysis Table 1 lists the descriptive term used by the authors for the psychosocial treatment included in each of the metaanalyses reviewed. Behavioral parent training and a general psychosocial treatment (e.g., ‘‘non-pharmacological;’’ ‘‘behavioral or cognitive-behavioral’’) were the most common psychosocial treatments within meta-analyses (33 %). Some of the meta-analyses focusing on a general psychosocial treatment further categorized the treatments within the results section (e.g., Hodgson et al. 2012; Purdie et al. 2002). ‘‘Behavioral intervention’’ was reviewed by 17 % of the meta-analyses. School-based interventions were reviewed by 17 % of the meta-analyses (note that these two are DuPaul and Eckert 1997; DuPaul et al. 2012, which were non-overlapping meta-analyses). It is readily apparent that there was great diversity within the psychosocial interventions reviewed within each meta-
analysis, with some having a more narrow focus compared to others with a broader inclusion of interventions. This diversity is even apparent within the five metaanalyses that reported effect sizes for behavioral parent training (Charach et al. 2013; Corcoran and Dattalo 2006; Purdie et al. 2002; Lee et al. 2012; Zwi et al. 2011). A review of Table 3 indicates no overlap between studies in the Charach et al., Purdie et al., or Corcoran and Dattallo meta-analyses with the Zwi et al. (2011) meta-analysis, even though the time periods reviewed overlapped. Overlap was also modest between Lee et al. and Charach et al. (four studies in common), Charach et al. and Corcoran and Dattallo/Purdie et al. (one study in common), and Zwi et al. and Purdie et al./Lee et al. (one study in common). Lee et al. and Purdie et al. had an overlap with Corcoran and Dattallo of 10 and two studies included, respectively. Thus, for behavioral parent training, these meta-analyses generated divergent results from largely independent samples of the literature. Thus, even the meta-analyses that reported results for a clearly defined ADHD intervention—parent training—were not comparable due to differences in studies included under this category of treatment.
123
86
Clin Child Fam Psychol Rev (2015) 18:77–97
Fig. 4 Effect sizes for specific outcomes within single-subject design studies. The study label is on the left side of the figure, the effect size estimate is illustrated in the figure, and on the right side, k studies that
contributed to the effect size, the effect size, and confidence interval are reported
Measurement of Outcome
Some patterns are apparent when outcome measures are reviewed. First, there are some measures that do not appear to illustrate an effect of psychosocial treatment including child ratings of ADHD symptoms (d = .11; Corcoran and Dattalo 2006) and ‘‘probably blinded assessments’’ (d = .02; Sonuga-Barke et al. 2013). Other outcomes appear to evince small to moderate effects of psychosocial treatment, including academic measures (effect sizes range from .19 to .43 in between-group studies; DuPaul et al. 2012; DuPaul and Eckert 1997; Van de Oord et al. 2008). Outcomes specific to the areas of concern for ADHD are generally moderate to large including disruptive behavior (d = .75; Charach et al. 2013) and teacher behavioral ratings (d = .40 to .75; Corcoran and Dattalo 2006; Klassen et al. 1999; Van de Oord et al. 2008), though, see Zwi et al. (2011) for an alternative outcome (d = -.32). Overall effect sizes collapsed across categories generally suggest a greater range of effect for between-group design studies (d = .18 to .74; DuPaul et al. 2012; DuPaul and Eckert 1997; Fabiano et al. 2009; Sonuga-Barke et al. 2013).
Effect sizes for specific outcomes within meta-analyses are displayed in Fig. 2 (between-group), Fig. 3 (within-subject), Fig. 4 (single-subject), and Fig. 5 (multiple designs) (note there is no pre–post figure as there is only a single effect size within this design category in Fig. 1). Variability across outcomes is readily apparent, with many categories of outcome measures yielding significant and well as nonsignificant estimates of effect size. Interestingly, social outcomes, a common area of impairment for children with ADHD, had three, positive, significant effects ranging from .50 to .71 (yet, see Corcoran and Dattalo 2006 for a more modest estimate of the more precise outcome of social competence, d = .07). Outcomes such as IQ, academic achievement, and cognitive functioning, which are peripheral to the targets of psychosocial intervention for ADHD, did not appear to be consistently associated with positive outcomes. Figures 4 and 5 illustrate a more consistent pattern of positive, significant effect size estimates across the meta-analyses, indicating more robust effects of psychosocial treatments within these designs.
123
Clin Child Fam Psychol Rev (2015) 18:77–97
87
Fig. 5 Effect sizes for specific outcomes within meta-analyses that included multiple designs. The study label is on the left side of the figure, the effect size estimate is illustrated in the figure, and on the
right side, k studies that contributed to the effect size, the effect size, and confidence interval are reported
Inclusion Criteria
including k = 6–10 studies, the mean effect size increases to .27 (SD = .47). For k = 11 or more studies included within the meta-analysis, the effect size increases to .60 (SD = .21). Thus, the more effect sizes from betweengroup design studies included within a meta-analysis, the greater the estimated magnitude effect of treatment, and the smaller the variability for the estimated mean effect size. This pattern of results is observed in Fig. 1 wherein metaanalytic results are ordered by the k of studies included. The first meta-analytic results in each category are typically positive and statistically significant (i.e., those with the greatest k of studies included), and the metaanalyses listed at the end of each category (i.e., those with the fewest k of studies included, often only one) are typically negative and statistically nonsignificant. Of note, it might be appropriate to ignore the meta-analytic effects from k = 1 studies, as it is not clear a single study can be considered a meta-analytic effect. In the present synthesis, is it not possible to disentangle the effect of k studies included from inclusion criteria given the large variability across meta-analyses and relatively small sample size.
A review of Table 1 indicates that the k of studies included in a meta-analyses ranged from 2 (Klassen et al. 1999) to 174 (Fabiano et al. 2009). Some reasons for this dispersion are clear—some studies only included randomized trials (Klassen et al. 1999; Sonuga-Barke et al. 2013; Zwi et al. 2011), whereas others were inclusive of many different designs (DuPaul and Eckert 1997; DuPaul et al. 2012; Fabiano et al. 2009; Purdie et al. 2002). Other parameters also impacted inclusion such as limiting studies to those between a particular date range (DuPaul et al. 2012; Purdie et al. 2002) or ruling out any study that included children with comorbidities (Hodgson et al. 2012). The k of studies included also appeared to be related to the effect size generated. For instance, larger effect sizes appear to be concentrated within meta-analyses that included a greater number of studies. Descriptively, for metaanalyses reporting between-group effect sizes from k = 1–5 studies, the mean effect size for psychosocial treatments is -.07 (SD = .55). For meta-analyses
123
88
Meta-Analysis Reporting Quality Table 4 lists the rating for each of the meta-analyses overall, and within specific categories with respect to the inclusion of information recommended in the MARS criteria. There was considerable variability in consistency with MARS criteria ranging from a high score of 68 % of criteria reported (Zwi et al. 2011) to a low score of 34 % of criteria reported (Purdie et al. 2002). Most meta-analyses reported procedures and inclusion/exclusion criteria consistent with MARS recommendations, which is indicated via the generally high scores on the methods. Statistical methods were the most common aspect of the MARS criteria to be underreported in the meta-analyses reviewed. A consequence of this is that effect sizes cannot confidently be combined across meta-analyses as equivalencies in effect size outcomes are unclear. Notably, none of the metaanalyses reviewed included all of the information suggested in the MARS criteria. Meta-Analytic Methods Table 5 presents the statistical approach used by each meta-analytic review. For each meta-analysis, the formula and/or description of the approach used to calculate the effect size is reported. Further, any procedures used to account for dependency within studies (e.g., the approach to aggregating multiple effect sizes from study Ni in metaanalysis ki), heterogeneity among effect sizes across individual studies, and publication bias are also presented. As can be seen from Table 5, multiple approaches were used to calculate effect size estimates, which cautions against making direct comparisons across meta-analyses due to differences in metrics. Consistency with Narrative Reviews Recent narrative reviews using operationalized criteria for judging the effectiveness of psychosocial treatments for ADHD have determined that behavioral parent training, school-based contingency management, and training-/peerfocused interventions are evidence based (Evans et al. 2013; Pelham and Fabiano 2008; Pelham et al. 1998). To investigate the degree to which these narrative reviews align with the meta-analytic literature, findings for each of these three treatments were investigated.
Clin Child Fam Psychol Rev (2015) 18:77–97
being randomized trials, and analyzed both proximal (i.e., parenting skills) and distal (i.e., ADHD symptoms and disruptive behavior) outcomes. Corcoran and Dattallo included 14 studies that included a comparison group. Lee included 40 studies, but this meta-analysis was less specific in the categorization of outcomes. It is not clear how many studies Purdie et al. included to calculate the effect size. Zwi et al. included many fewer studies due to restrictive inclusion criteria (i.e., exclusion of any study that included child-focused intervention), and they reported on a small number of outcomes including those peripheral (i.e., internalizing behavior) to ADHD. For these meta-analyses, Charach et al. reported effect sizes ranging from .55 (parenting skills) to .77 (ADHD symptoms), Corcoran and Dattallo reported effect sizes ranging from .11 (child ratings) to .75 (teaching ratings), Lee reported a moderate effect size (r = .34), Purdie reported an effect size of .31, and Zwi et al. reported a negative effect size for externalizing behavior (d = -.32) and internalizing behavior (d = -.48). These results are discrepant, perhaps due to differences in the approaches within each meta-analysis. Regardless, four of the five meta-analytic findings were consistent with the conclusions of the narrative reviews that reported behavioral parent training was an effective intervention for children with ADHD. School-based Contingency Management Three meta-analyses reported results specifically for school-based contingency management strategies (DuPaul and Eckert 1997; DuPaul et al. 2012; Purdie et al. 2002). DuPaul and Eckert reported between-group effect sizes of .45 for behavioral outcomes and .31 for academic outcomes in between-group studies with larger effects reported for crossover and single-subject design studies. DuPaul et al. (2012) reported between-group effect sizes of .18 for behavioral outcomes and .43 for academic outcomes in between-group studies with larger effects again reported for crossover and single-subject design studies. Purdie et al. reported an effect size of .39 for school-based/ educational interventions. There were no studies that overlapped between either of these meta-analyses, indicating the three meta-analyses represent independent portions of the research literature. The results of these three meta-analyses support the use of school-based contingency management as an intervention for ADHD, consistent with systematic review conclusions.
Behavioral Parent Training Training-/Peer-Focused Interventions Five meta-analyses explicitly investigated the effect of parent training for ADHD (Charach et al. 2013; Corcoran and Dattalo 2006; Lee et al. 2012; Purdie et al. 2002; Zwi et al. 2011). Charach included 14 studies, the majority
123
No meta-analyses specifically calculated effect sizes for the training- and peer-focused interventions determined to be evidence based through narrative review.
Clin Child Fam Psychol Rev (2015) 18:77–97
Discussion The present report was initiated to aggregate information across existing published meta-analyses of psychosocial treatments for ADHD. Integrative research reviews are defined as ‘‘Research syntheses attempt to integrate empirical research for the purpose of creating generalizations’’ (pp. 5; Cooper and Hedges 1994). In the metaanalytic literature for ADHD psychosocial treatments, it does not appear that this effort to create generalizable conclusions has been realized. Across meta-analyses, parent training interventions for ADHD work strongly (Charach et al. 2013), moderately (Lee et al. 2012), or are inferior to a control group (Zwi et al. 2011). School-based behavioral treatments for ADHD assessed in betweengroup studies result in larger effects on behavioral outcomes relative to academic outcomes in one meta-analysis (DuPaul and Eckert 1997), but a different pattern was found in a second meta-analysis of research studies published since the first meta-analysis (DuPaul et al. 2012). Divergent study designs also yield divergent effect size magnitudes with crossover design and single-subject design studies yielding much larger estimates of effect relative to between-group designs (see DuPaul and Eckert 1997; DuPaul et al. 2012; Fabiano et al. 2009). Even within meta-analyses, behavioral interventions yield a moderate effect or no effect, depending on the outcome (Sonuga-Barke et al. 2013). In spite of the differing conclusions across meta-analyses, when these meta-analyses were aggregated together some general conclusions can be made. First, the metaanalyses were largely independent reviews of portions of the literature rather than replications, given that there was very little overlap in studies across the meta-analyses (see Tables 2, 3). Second, the approach to meta-analysis of the ADHD psychosocial treatment literature was quite disparate across review groups with diverse designs, inclusion/exclusion criteria, differences in scope, and varying approaches to the calculation of both individual and overall effect size estimates as well as the combining of effect sizes across various outcome measures. Overall, the evidence from the research literature when all meta-analyses and study designs were considered suggests that psychosocial treatments for ADHD are efficacious (i.e., pre– post, within-subject, and single-subject design effect sizes are significant; 82 % of between-group effect sizes generated by a k of 5 or greater were positive and statistically significant). This finding is consistent with narrative and selective reviews of the literature that have strongly endorsed the efficacy of psychosocial treatments for youth with ADHD (Evans et al. 2013; Pelham and Fabiano 2008; Pelham et al. 1998). However, it was also surprising that there was so much diversity across meta-analyses in
89 Table 2 Proportion of all individual studies with a psychosocial intervention included in any meta-analysis that are included in each meta-analysis First author
Year
Total
Papers included in meta-analysis
N
n
(%)
Fabianoa
2009
238
109
46
DuPaul
1997
109
49
45
DuPaula
2012
252
55
22
Lee
2012
252
39
15
Purdie
2002
162
23
14
Sonuga-Barke
2013
252
19
8
Corcoran
2006
209
15
7
Van der Oord
2008
234
14
6
Charach
2013
252
15
6
Klassen
1999
129
5
4
Hodgson
2012
252
9
4
Zwi
2011
249
5
2
Papers included in the total column represent all studies of a psychosocial intervention included in the meta-analyses of any of the papers included in this review. Individual studies were excluded from the total n count if the year of publication was the same year of the meta-analysis or later a Fabiano et al. (2009) included in their meta-analyses three papers for which publication dates were unavailable, and DuPaul et al. (2012) included one paper for which the publication date was unavailable. Because an exact publication date was not available for these papers, they were excluded from the ns for this analysis
conclusions generated. Each of these major results will be reviewed in turn. Apples and Apples or Apples and Oranges?—The Impact of Treatment Approach and Inclusion Criteria A major finding in this review is that there is little to no overlap among the studies included in ADHD meta-analyses of psychosocial treatment. As can be observed from Tables 2 and 3, only a minority of studies overlap across meta-analyses. This leaves a need within the field to conduct a meta-analytic review of the entire population of ADHD treatment studies identified, in order to obtain a robust estimate of the effect of different psychosocial treatments on ADHD-related outcomes. Otherwise, the situation is much like the old fable of the visually impaired men touching different parts of an elephant (Saxe 1892). In a review of meta-analyses of stimulant effects, Conners (2002) also noted a lack of overlap across systematic reviews. However, unlike the present research synthesis, the findings for stimulant medication effects were comparable across meta-analyses. Part of the reason for this is that the specific treatment employed across all studies included in the Conners’ (2002) meta-analyses was identical—stimulant medication. In contrast, a review of the studies
123
90 Table 3 Overlap between pairs of meta-analyses with regard to the individual studies of psychosocial interventions included in each meta-analysis
A dash indicates no overlap between meta-analyses with regard to empirical studies included
Clin Child Fam Psychol Rev (2015) 18:77–97
Number
Study
2
3
4
5
6
7
8
9
10
1
Charach et al. (2013)
2
Sonuga-Barke et al. (2013)
4
3
Hodgson et al. (2012)
–
4
DuPaul et al. (2012)
–
1
1
5
Lee et al. (2012)
4
5
2
6
Zwi et al. (2011)
–
1
–
–
1
7
Fabiano et al. (2009)
3
6
3
10
19
1
8
Van de Oord et al. (2008)
–
3
3
1
7
–
10
9
Corcoran and Dattalo (2006)
1
3
1
–
10
–
10
7
10
Purdie et al. (2002)
1
3
1
–
4
–
7
5
4
11
Klassen et al. (1999)
–
4
–
–
1
–
–
6
2
9
12
DuPaul and Eckert (1997)
–
–
–
–
–
–
17
3
2
3
included across the meta-analyses in the present report yielded numerous types of psychosocial treatments (e.g., social skills training, cognitive therapy, contingency management, parent management training, biofeedback), and often these interventions were combined together in single effect size estimates. Notably, the same approach to combining disparate psychosocial treatments is often used in practice parameters for ADHD where different medication types are clearly distinguished, yet psychoeducation, therapy, counseling, and behavioral treatment are often combined into a single category resulting in at times confusing or unclear recommendations (e.g., AACAP 2007). The idea that these meta-analyses were independent replications does not apply here—the studies and the treatments included within them were too diverse. Comparable combinations would be viewed as absurd in systematic reviews of medication for ADHD. For example, atypical antipsychotics, antidepressants, and anxiolytics would not be combined with stimulant medications in a single effect size estimate to illustrate the efficacy of medication for ADHD treatment. A similar standard should thus be utilized with psychosocial treatments. That is, meta-analyses should avoid combining ‘‘psychosocial’’ interventions and instead report effect size estimates for distinct treatment approaches/modalities separately. The authors of these meta-analyses took multiple approaches to addressing differences in design characteristics, ranging from including all designs (e.g., DuPaul et al. 2012) to excluding all but randomized group designs (e.g., Zwi et al. 2011). These differences were also apparent in the calculation of effect size estimates in meta-analytic reports with some groups reporting effect sizes for each design separately (e.g., DuPaul and Eckert 1997; Fabiano et al. 2009), whereas others collapsed designs together (Purdie et al. 2002). Going forward, meta-analytic research
123
1
11
12
4 1
1
questions within this literature should be sensitive to the need to be inclusive across this diverse array of study designs, yet prudent in how these designs are combined. Further, there were differences across meta-analyses with respect to the inclusion/exclusion criteria used in study search and retrieval. This resulted in at times large differences across studies. For instance, Zwi et al. (2011) had some of the most stringent inclusion criteria, which yielded only five studies of parent training and a negative effect size estimate. This can be contrasted with other reviews that had many more parent training studies represented (e.g., Charach et al. 2013), and reported a positive effect for parent training for youth with ADHD. The Impact of Measurement on Results Measurement issues may also contribute to the differences in findings across meta-analyses. The meta-analyses reviewed in this paper included a wide variety of treatment outcome measures (e.g., ADHD symptoms, externalizing behavior, parenting, and cognitive ability). Conners (2002) reported that stimulant medication effect sizes varied based on the outcome measure used. In the Conners’ (2002) review, it was consistently reported that stimulant medication effects were greater for ratings of ADHD symptoms relative to observational measures conducted in the child’s classroom (see Kavale 1982 for an example of this pattern of results). Psychosocial interventions, specifically behavior therapy, target social and behavioral outcomes beyond symptom ratings, and this is an area where considerable treatment effects have been obtained (Fabiano et al. 2009). Interestingly, in the present review, social outcomes yielded positive and significant effect sizes in meta-analyses for psychosocial interventions, indicating, as expected, that psychosocial interventions typically provide a
Clin Child Fam Psychol Rev (2015) 18:77–97
91
Table 4 Inclusion of information required for meta-analysis quality rating using APA Publications and Communications Board Working Group on Journal Article Reporting Standards (2008) for meta-analysis Meta-analysis
Meta-analysis methods (%)
Search procedures (%)
Coding procedures (%)
Statistical methods (%)
Reporting of results (%)
Overall quality score (%)
Zwi
80
79
83
38
75
68
DuPaul et al. (2012)
80
79
67
54
58
66
Corcoran
80
57
67
69
67
66
SonugaBarke
80
71
67
46
67
64
Charach
40
64
83
46
75
62
DuPaul and Eckert (1997)
80
57
50
62
67
62
Fabiano
80
57
67
54
50
58
Van der Oord
80
43
67
62
42
54
Klassen
80
57
33
46
50
52
Lee Hodgson
80 80
43 50
50 0
15 23
58 58
44 42
Purdie
40
43
0
23
50
34
Information used to calculate compliance with the inclusion of information consistent with quality ratings was obtained from a review of the meta-analysis text and supplementary materials
benefit for this target of treatment. Next steps in the field include a meta-analytic approach that compares different treatment modalities, and the impact on varied outcomes within conceptually relevant domains for the treatment administered (see Sonuga-Barke et al. 2013 for an example of an approach that could be adapted for this specific research question). These analyses should also include a consideration of how setting and informant may influence outcomes. For instance, using a teacher rating as an outcome measure for a behavioral parent training intervention would be inappropriate given that parenting interventions are unlikely to generalize to other settings. Thus, these meta-analytic approaches should separate the report of effect sizes into settings where treatment was employed and settings where treatment was not employed, but may generalize. Another component that varied among the meta-analyses reviewed was the method of calculating effect size estimates. Some meta-analyses had numerous effect sizes generated across outcomes (Hodgson et al. 2012), whereas others combined outcomes to yield a single effect size from each individual study (Fabiano et al. 2009). In addition to these differences in aggregation, Table 5 outlines the differences in meta-analytic approach to calculating effect sizes, which also likely had an influence on outcome. Further, the variability in methodological rigor across meta-analyses may have influenced the results obtained as well (see Table 5). Differences in approaches to calculating effect sizes across meta-analysis are compounded by differences in study designs included. Recently developed, innovative methods for calculating effect sizes in a
consistent manner across study designs will enhance future meta-analysis (Hedges et al. 2012; Shadish et al. 2013). Implications of these Results Currently, there are at times vague and inconsistent recommendations within practice guidelines with respect to psychosocial treatments for ADHD (Vallerand et al. 2014) and wide variation in treatment practices within and across nations (Hinshaw and Schleffer 2014; Hinshaw et al. 2011). It is possible that psychosocial interventions may receive lower rates of endorsement relative to medication in part due to the lack of consistency in systematic reviews and meta-analysis within the psychosocial treatment literature. For example, behavioral parent training has been recommended as an evidence-based treatment for ADHD with clear research support by some systematic reviews (Evans et al. 2013; Pelham and Fabiano 2008; Pelham et al. 1998). Yet, a contemporary meta-analysis concluded that with respect to the efficacy of behavioral parent training interventions, ‘‘the evidence we found was limited in terms of the size of the trials and in their quality, and therefore we do not think it can be used as the basis for guidelines of treatment for ADHD in clinics or schools’’ (p. 8; Zwi et al. 2011). This disconnect may have resulted from rigorous exclusionary criteria within the Zwi et al.’s meta-analysis resulting in only five studies included, and a 2 % overlap with the other meta-analyses included in the present report. Thus, although Zwi et al. (2011) included the most rigorous and clear reporting of methodological characteristics of their meta-analysis (Table 4), a consequence of the rigor
123
92
Clin Child Fam Psychol Rev (2015) 18:77–97
Table 5 Description of methods used to conduct meta-analysis in each of the meta-analyses reviewed Charach et al. (2013) Effect size
Standardized mean difference Þðxctrlbaseline xctrloutcome Þg SMD ¼ fðxT baseline xT outcome SDchange qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi SDchange ¼ SD2baseline þ SD2final ð2 Corr SDbaseline SDfinal Þa
SMD calculated using RevMan (RevMan 5.1; Nordic Cochrane Center, Copenhagen, Denmark) using DerSimonian and Laird random-effect model with inverse variance method. A correlation factor of .3 was assumed for the relationship between baseline and outcome values Dependency
Not reported
Heterogeneity
v2 test
Publication bias
Not reported
Corcoran and Dattalo (2006) Effect size
Cohen’s d 0
xCtrlOutcome Þ Cohen sd ¼ ðxTOutcome SDPooledwithingroup
Effect sizes from individual studies were pooled using Borenstein and Rothstein (2001) software with a correction for small sample size bias (Hedges and Olkin 1985). Effect sizes based on both random-effect and fixed-effect models are reported Dependency
Separate effect sizes were calculated for each of several outcome measures within a study, and then, effect sizes were averaged within studies
Heterogeneity
v2 test
Publication bias
Reported fail-safe N
DuPaul and Eckert (1997) Effect size
Between-subjects designs Unweighted effect sizes (Glass 1977; Glass et al. 1981; Smith et al. 1980; Wolf 1986) were calculated. A weighted effect size was calculated according to the weighted least squares general linear model approach (Hedges and Olkin 1985) Within-subjects designs Effect size weighted by sample size (Becker 1988; Becker 1990) was computed for treatment and control group (Control ES = 0 when no control group). Weighted deltas were calculated based on weighted effect sizes Single-subject designs Busk and Serlin (1992) approach 3 to calculate a unweighted effect sizes which were then weighted by number of baseline and treatment data points
Dependency
Effect sizes were reported for three classes of outcome measures: academic measure, behavioral measures, and clinic-based tests. When a single study included multiple measures within a single class, these effects were averaged
Heterogeneity
v2 test
Publication bias
File drawer analyses (Hunter and Schmidt 1990, p.513)
DuPaul et al. (2012) Effect size
Overall effect sizes based on fixed-effect model (Hedges and Olkin 1985; Lipsey and Wilson 2001) Between-subjects designs Unweighted effect size (Glass 1977; Glass et al. 1981) weighted using least squares GLM (Hedges and Olkin 1985) Within-subjects designs Unweighted and weighted effect sizes (Becker 1988; Becker 1990) Weighted deltas Single-subject designs Unweighted effect size (Approach 3, Busk and Serlin 1992)
Dependency
Effect sizes were reported for three classes of outcome measures: academic measure, behavioral measures, and clinic-based tests. When a single study included multiple measures within a single class, these effects were averaged
Heterogeneity
v2 test
Publication bias
Fail-safe N (Orwin 1983)
Fabiano et al. (2009) Effect size
Between-subjects designs Cohen’s d xCtrlOutcome Þ Cohen sd ¼ ðxTxOutcome SDPooledOutcome 0
ES based on a fixed-effect model reported. Due to a significant Q statistic indicating heterogeneity, ES based on random-effect model also reported
123
Clin Child Fam Psychol Rev (2015) 18:77–97
93
Table 5 continued Pre–post designs ðxpretest xposttest Þ SDpretest Effect sizes were weighted by inverse variance. The Q statistic was not significant, and a fixed-effect model was used Within-subject designs unweighted effect size ðxpretest xposttest Þ SDpretest Single-subject designs Unweighted effect size (Busk and Serlin 1992; White et al. 1989; Stage and Quiroz 1997) ðxbaseline xtreatment Þ
SDbaseline Dependency
Effect sizes were calculated for each dependent measure included in the identified studies and averaged across measures to derive a single effect size for each study
Heterogeneity
v2 test
Publication bias
Fail-safe N (Orwin 1983)
Hodgson et al. (2012) Effect size
Cohen,s d ES weighted by the inverse of the variance (Lipsey and Wilson 2001). For pre–post designs, only posttest data were used to calculate ES
Dependency
When two or more studies shared the same sample with the same treatment and outcome measures, one study was selected for inclusion and the others were excluded. Procedures for handling multiple effect sizes from a single study are not reported
Heterogeneity
Not reported
Publication Bias
Fail-safe N
Klassen et al. (1999) Effect size
ES calculated using RevMan 3.0 with a random-effect model according to DerSimonian and Laird method. Either weighted mean difference or standardized mean difference was calculated depending on whether more than one outcome measures contributed to the effect size
Dependency
An algorithm was developed to select 1 teacher and 1 parent behavior rating scale to use as the outcome measure for each individual study. When a single measure produced multiple scores, either a single score was chosen or the multiple scores were combined
Heterogeneity
v2 test
Publication bias
Not reported
Lee et al. (2012) Effect size
r (Friedman 1968) ES r was calculated for individual studies. It is unclear how individual rs were combined
Dependency
Not reported
Heterogeneity
Heterogeneity of r evaluated using formulas from Rosenthal (1991); v2 test
Publication bias
Not reported
Purdie et al. (2002) Effect size
Effect sizes calculated according to Becker (1988)
Dependency
Weighted least squares approach used to address dependency of effect sizes within individual studies
Heterogeneity Publication Bias
v2 test Not reported
ES corrected for bias due to small sample size (Hedges and Olkin 1985)
Sonuga-Barke et al. (2013) Effect size
Standardized mean difference Þðxctrlbaseline xctrloutcome Þg SMD ¼ fðxTbaseline xToutcome SD pooled
123
94
Clin Child Fam Psychol Rev (2015) 18:77–97
Table 5 continued SMDs weighted by inverse variance and combined using a random-effect model Dependency
For each study, single effect sizes were chosen for each analysis from those reported based on criteria outlined in the manuscript
Heterogeneity
v2 test
Publication Bias
Not reported. The authors deemed assessment of publication bias inappropriate due to small sample of included studies
Van de Oord et al. (2008) Effect Size
Cohen’s d 0
xPostTreatment Þ Cohen sd ¼ ðxPreTreatment SDPooled
ES calculated using Borenstein and Rothstein (1999) software and weighted by sample size. ES were combined using a random-effect model Dependency
Outcome measures were categorized according to four domains (e.g., ADHD symptoms, and ODD symptoms). When a single study reported multiple outcome measures within a single domain, ‘‘the most widely used measure’’ was selected. When a single measure yielded multiple subscales, these were aggregated into a single effect size
Heterogeneity
v2 test
Publication Bias
Fail-safe N (Orwin 1983)
Zwi et al. (2011) Effect size
Standardized mean difference SMD calculated using RevMan 5.0. ES corrected for small sample bias using inverse variance method. ES calculated using both fixed- and random-effect models. Reported ES appear to be based on random-effect model
Dependency
The authors had planned to average the effects of multiple measures of the same outcome within a study; however, in the one instance this occurred, averaging was deemed inappropriate and separate effects were reported
Heterogeneity
v2 test
Publication bias
Funnel plots and other analyses to assess bias were planned but not conducted due to small sample of included studies
Unless otherwise noted, mathematical equations presented in this table were created based on information provided in text or supplemental materials for each meta-analysis. Efforts were made to provide mathematical equations for the effect size employed by each meta-analysis; however, not all meta-analyses provided sufficient information to do this a
Equation displayed in Charach et al. (2013)
may have been a reduction in the scope, clinical meaningfulness, and therefore generalizability of the review. Limitations This review has limitations. First, although the initial intention was to conduct a meta-analytic review of the metaanalyses identified, this approach was untenable due to the considerable methodological differences across the articles identified. Differences in the calculation of effect size estimates, approach to synthesizing data across outcomes, study inclusion/exclusion criteria, and definition of ‘‘psychosocial treatment’’ included within reviews precluded aggregation across meta-analyses. Thus, this manuscript relies largely on descriptive reports of the larger literature. Second, this report focused on meta-analytic reviews of the literature of ADHD treatments available up to June 2013. It does not include other reviews that applied criteria for
123
study efficacy other than effect sizes (Evans et al. 2013; Pelham and Fabiano 2008; Pelham et al. 1998). Further, this report does not include comparative research on different forms of psychosocial treatments as this was outside the scope of the meta-analysis aims. Further, future metaanalytic studies are required for synthesizing comparative outcomes for different psychosocial treatments as well as comparisons between psychosocial and pharmacological/combined approaches (Jadad et al. 1999b). Recommendations for Future Meta-Analyses of ADHD Treatments The present review synthesizes results across meta-analyses and yields an overall finding that supports psychosocial treatments for youth with ADHD with the majority of null or counter findings coming from small k effect size estimates. However, it is unclear whether the current effect size
Clin Child Fam Psychol Rev (2015) 18:77–97
estimates within the meta-analyses reviewed represent a rigorous report of the true effect of psychosocial treatments given that different psychosocial treatments and study designs were combined together in some cases (e.g., Purdie et al. 2002) and disentangled in others (e.g., Fabiano et al. 2009), the samples of studies included often did not represent the entire population of research reports (see Tables 2, 3), and the methodological quality information included within the meta-analyses varied (Table 4). This review is instructive in providing recommendations for future meta-analyses of psychosocial treatments for ADHD. First, there is a fundamental need to include the entire literature of interest in a meta-analysis. The ADHD psychosocial treatment literature is underrepresented in this regard. Some of the poor overlap across meta-analyses can be explained by the various modalities of psychosocial interventions for ADHD (e.g., behavioral parent training, cognitive therapy) as well as the multiple research designs within the literature (e.g., randomized controlled trial; crossover designs; single-subject design studies). This is further compounded by the historical lack of viable solutions for calculating effect sizes across these diverse designs. Inclusion of studies within meta-analyses was further limited by cutoffs regarding publication date. An advantage of meta-analysis is that it provides an overall accounting of a literature, including only a portion of the literature could result in biased or inaccurate results. It is strongly recommended that future meta-analyses aimed at investigating the efficacy of particular treatments include the entire population of studies in their results. For instance, updates of meta-analyses could include the studies from the prior analysis in the overall results, with date of publication being used as a moderator. Second, meta-analytic reviews should not combine diverse psychosocial intervention results into one aggregate analysis. Researchers should also be clear in their description of treatment approaches in titles, abstracts, and methods sections to permit meta-analytic reviews of psychosocial treatments for ADHD to be more precise. An additional recommendation within the ADHD literature is increased uniformity in outcome assessments. There are a number of well-validated observational tools for ADHD researchers to utilize in treatment studies (Pelham et al. 2005). Emphases on methodologically rigorous, blinded outcomes (e.g., Sonuga-Barke et al. 2013) must be appropriately balanced with sensible approaches to outcome assessment that acknowledge any psychosocial or pharmacological treatments for ADHD are unlikely to generalize to untreated settings (i.e., some blinded assessments within Sonuga-Barke et al. 2013 were collected from raters within settings where treatment did not occur). Further, meaningful change within functional areas that are conceptually unrelated to the
95
treatment approach should not be weighted in the same manner as change within targeted domains within metaanalyses. Innovations within the field of clinical psychology are needed to create and implement methods to promote the objectivity and practical sensibility within meta-analyses that include diverse outcome assessments. Past this, consistency across meta-analytic work could be improved by categorizing outcomes as related to the symptoms of ADHD, comorbid symptoms, and functional impairment. It is recommended that outcome measures unrelated or peripheral to ADHD core impairments (e.g., IQ; internalizing symptoms) or invalid for assessing treatment outcome (i.e., child report of ADHD symptoms; Pelham et al. 2005) be removed from meta-analytic reviews as these can undermine significant results of treatment in targeted areas. Future meta-analytic reviews must work to be inclusive of the large literature available while accounting for variability in treatment modality, research design, and appropriate outcome measures. Additionally, it is imperative that future meta-analyses employ appropriate methods for calculating effect sizes for individual studies and for combining individual effect sizes to create an overall effect size even in designs that are non-traditional within meta-analysis (e.g., Hedges et al. 2012; Shadish et al. 2013). Methods for calculating effect sizes and overall effect sizes should be clearly documented and should include formulas to facilitate replication. This will likely require the collaboration between clinical researchers, methodologists, and statisticians.
Conclusion Across these meta-analyses of psychosocial treatments for ADHD, there is diversity in results, studies included, and approaches. Precise conclusions on the efficacy of psychosocial interventions, for particular targeted groups, on specific outcomes, are unable to be generated from the current meta-analytic literature. The lack of overlap among articles included in published meta-analyses is likely a strong contributor to some disagreements in results, with meta-analyses typically reporting on an independent sample of psychosocial treatment studies. Thus, using any one of the meta-analyses reviewed herein to make policy decisions or determine the efficacy of psychosocial treatments for ADHD appears unwise at this time. Although a major task, there is a strong need within the field for a comprehensive meta-analysis across all studies in the psychosocial treatment literature, reporting separate effect sizes for different psychosocial treatment approaches, so that the field can continue to move toward more evidenceinformed practice in the treatment for ADHD.
123
96 Acknowledgments During preparation of portions of this manuscript, Dr. Fabiano and Dr. Schatz were supported by funding from the Eunice K. Shriver National Institute for Child Health and Human Development Number R01HD58588.
References Abikoff, H. B. (1991). Cognitive training in ADHD children: Less to it than meets the eye. Journal of Learning Disabilities, 24, 205–209. Abramowitz, A. J., & O’Leary, S. G. (1991). Behavioral interventions for the classroom: Implications for students with ADHD. School Psychology Review, 20, 219–233. American Academy of Child and Adolescent Psychiatry. (2007). Practice parameter for the assessment and treatment of children and adolescents with attention-deficit/hyperactivity disorder. Journal of the American Academy of Child and Adolescent Psychiatry, 46, 894–921. American Academy of Pediatrics, Subcommittee on AttentionDeficit/Hyperactivity Disorder, & Steering Committee on Quality Improvement and Management. (2011). ADHD: Clinical practice guideline for the diagnosis, evaluation, and treatment of attention-deficit/hyperactivity disorder in children and adolescents. Pediatrics. doi:10.1542/peds.2011-2654. American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed.). Washington, DC: American Psychiatric Association. APA Publications and Communications Board Working Group on Journal Article Reporting Standards. (2008). Reporting standards for research in psychology: Why do we need them? What might they be? American Psychologist, 63, 839–851. Baer, R. A., & Nietzel, M. T. (1991). Cognitive behavioral treatment of impulsivity in children: A meta-analytic review of the outcome literature. Journal of Clinical Child Psychology, 20, 400–412. Becker, B. J. (1988). Synthesizing standardized mean change measures. British Journal of Mathematical and Statistical Psychology, 41, 257–278. Becker, B. J. (1990). Coaching for the Scholastic Aptitude Test: Further synthesis and appraisal. Review of Educational Research, 60, 373–417. Borenstein, M., & Rothstein, H. (1999). Comprehensive Metaanalysis: A computer program for research synthesis. Englewood, NJ: Biostat Inc. Borenstein, M., & Rothstein, H. (2001). Comprehensive metaanalysis. Englewood, NJ: Biostat Inc. Busk, P. L., & Serlin, R. C. (1992). Meta-analysis for single-case research. In T. R. Kratochwill & J. R. Levin (Eds.), Single-case research design and analysis (pp. 187–212). Hillside, NJ: Lawrence Erlbaum. Chacko, A., Bedard, A. C., Marks, D. J., Feirsen, N., Uderman, J. Z., Chimiklis, A., et al. (2013). A randomized clinical trial of Cogmed Working Memory Training in school-aged children with ADHD: A replication in a diverse sample using a control condition. Journal of Child Psychology and Psychiatry. doi:10. 1111/jcpp.12146. Charach, A., Carson, P., Fox, S., Ali, M. U., Beckett, J., & Lim, C. G. (2013). Interventions for preschool children at high risk for ADHD: A comparative effectiveness review. Pediatrics, 131, 1–21. Conners, C. K. (2002). Forty years of methylphenidate treatment in attention-deficit/hyperactivity disorder. Journal of Attention Disorders, 6(1), S17–S30.
123
Clin Child Fam Psychol Rev (2015) 18:77–97 Cooper, H., & Hedges, L. V. (1994). The handbook of research synthesis. New York, NY: Russell Sage Foundation. Cooper, H., Hedges, L. V., & Valentine, J. C. (2009). The handbook of research synthesis and meta-analysis. New York, NY: Russell Sage Foundation. Corcoran, J., & Dattalo, P. (2006). Parent involvement in treatment for ADHD: A meta-analysis of the published studies. Research on Social Work Practice, 16, 561–570. DuPaul, G. J., & Eckert, T. L. (1997). The effects of school-based interventions for attention deficit hyperactivity disorder: A metaanalysis. School Psychology Review, 26, 5–27. DuPaul, G. J., Eckert, T. L., & Vilardo, B. (2012). The effects of school-based interventions for attention deficit hyperactivity disorder: A meta-analysis 1996–2010. School Psychology Review, 41, 387–412. Evans, S. W., Owens, J. S., & Bunford, N. (2013). Evidence-based psychosocial treatments for children and adolescents with attention-deficit/hyperactivity disorder. Journal of Clinical Child and Adolescent Psychology. doi:10.1080/15374416.850700. Fabiano, G. A., Pelham, W. E., Coles, E. K., Gnagy, E. M., Chronis, A. M., & O’Connor, B. C. (2009). A meta-analysis of behavioral treatments for attention-deficit/hyperactivity disorder. Clinical Psychology Review, 29, 129–140. Fabiano, G. A., Pelham, W. E., Majumdar, A., Evans, S. W., Manos, M., Caserta, D., et al. (2013). Elementary and middle school teacher perceptions of attention-deficit/hyperactivity disorder incidence. Child & Youth Care Forum, 42, 87–99. Fabiano, G. A., Pelham, W. E., Waschbusch, D., Gnagy, E. M., Lahey, B. B., Chronis, A. M., et al. (2006). A practical impairment measure: Psychometric properties of the impairment rating scale in samples of children with attention-deficit/hyperactivity disorder and two school-based samples. Journal of Clinical Child and Adolescent Psychology, 35, 369–385. Faraone, S. V., Biederman, J., Spencer, T. J., & Aleardi, M. (2006). Comparing the efficacy of medications for ADHD using metaanalysis. Medscape General Medicine, 8, 4. Glass, G. V. (1977). Integrating findings: The meta-analysis of research. Review of Research in Education, 5, 351–379. Glass, G. V., McGaw, B., & Smith, M. L. (1981). Meta-analysis in Social Research. Beverly Hills, CA: Sage Publications. Hedges, L. V., & Olkin, I. (1985). Statistical methods for metaanalysis. Orlando, FL: Academic Press. Hedges, L. V., Pustejovsky, J. E., & Shadish, W. R. (2012). A standardized mean difference effect size for single case designs. Research Synthesis Methods, 3, 224–239. Hinshaw, S. P., & Schleffer, R. M. (2014). The ADHD explosion: Myths, medication, money, and today’s push for performance. New York, NY: Oxford University Press. Hinshaw, S. P., Schleffer, R. M., Fulton, B. D., Aase, H., Banaschewski, T., Cheng, W., & Weiss, M. D. (2011). International variation in treatment procedures for ADHD: Social context and recent trends. Psychiatric Services, 62, 1–5. Hoagwood, K., Kelleher, K. J., Feil, M., & Comer, D. M. (2000). Treatment services for children with ADHD: A national perspective. Journal of the American Academy of Child and Adolescent Psychiatry, 39, 198–206. Hodgson, K., Hutchinson, A. D., & Denson, L. (2012). Nonpharmacological treatments for ADHD: A meta-analytic review. Journal of Attention Disorders. doi:10.1177/1087054712444732. Hunter, J. E., & Schmidt F. L. (1990). Methods of meta-analysis: Correcting error and bias in research findings. Newbury Park, CA: Sage. Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings (2nd ed.). Thousand Oaks, CA: Sage Publications.
Clin Child Fam Psychol Rev (2015) 18:77–97 Jadad, A. R., Booker, L., Gauld, M., Kakuma, R., Boyle, M., Cunningham, C. E., & Schachar, R. (1999a). The treatment of attention-deficit hyperactivity disorder: An annotated bibliography and critical appraisal of published systematic reviews and meta-analyses. Canadian Journal of Psychiatry, 44, 1025–1035. Jadad, A. R., Boyle, M., Cunningham, C. E., Kim, M., Schachar, R., Willan, A. R., et al. (1999b). Evidence report/technology assessment number 11: Treatment of attention-deficit/hyperactivity disorder. Rockville, MD: Agency for Healthcare Research and Quality. Kavale, K. (1982). The efficacy of stimulant drug treatment for hyperactivity: A meta-analysis. Journal of Learning Disabilities, 15, 280–289. Kent, K. M., Pelham, W. E., Molina, B. S., Sibley, M. H., Waschbusch, D. A., Yu, J., et al. (2011). The academic experience of male high school students with ADHD. Journal of Abnormal Child Psychology, 39, 451–462. Klassen, A., Miller, A., Raina, P., Lee, S. K., & Olsen, L. (1999). Attention-deficit hyperactivity disorder in children and youth: A quantitative systematic review of the efficacy of different management strategies. Canadian Journal of Psychiatry, 44, 1007–1016. Lee, P., Niew, W., Yang, H., Chen, V. C., & Lin, K. (2012). A metaanalysis of behavioral parent training for children with attentiondeficit/hyperactivity disorder. Research in Developmental Disabilities, 33, 2040–2049. Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Thousand Oaks, CA: Sage. Majewicz-Hefley, A., & Carlson, J. S. (2007). A meta-analysis of combined treatments for children diagnosed with ADHD. Journal of Attention Disorders, 10, 239–250. Orwin, R. G. (1983). A fail-safe-N for effect size in meta-analysis. Journal of Educational Statistics, 8, 157–159. Pelham, W. E., & Fabiano, G. A. (2008). Evidence-based psychosocial treatment for ADHD: An update. Journal of Clinical Child and Adolescent Psychology, 37, 184–214. Pelham, W. E., Fabiano, G. A., & Massetti, G. M. (2005). Evidencebased assessment of attention-deficit/hyperactivity disorder in children and adolescents. Journal of Clinical Child and Adolescent Psychology, 34, 449–476. Pelham, W. E., Foster, E. M., & Robb, J. A. (2007). The economic impact of ADHD in children and adolescents. Ambulatory Pediatrics, 7, 121–131. Pelham, W. E., Wheeler, T., & Chronis, A. (1998). Empirically supported psychosocial treatments for attention deficit hyperactivity disorder. Journal of Clinical Child Psychology, 27, 190–205. Purdie, N., Hattie, J., & Carroll, A. (2002). A review of the research on interventions for attention deficit hyperactivity disorder: What works best? Review of Educational Research, 72, 61–99. Rapport, M. D., Orban, S. A., Kofler, M. J., & Friedman, L. M. (2013). Do programs designed to train working memory, other executive functions, and attention benefit children with ADHD? A meta-analytic review of cognitive, academic, and behavioral outcomes. Clinical Psychology Review, 33, 1237–1252.
97 Robb, J. A., Sibley, M. H., Pelham, W. E., Foster, E. M., Molina, B. S. G., Gnagy, E. M., & Kuriyan, A. B. (2011). The estimated annual cost of ADHD to the US education system. School Mental Health, 3, 169–177. Rosenthal, R. (1991). Meta-analytic procedures for social research. Newbury Park, CA: Sage Publications, Inc. Saxe, J. G. (1892). Saxe’s complete poetical works. New York, NY: Houghton, Mifflin, and Company. Schachar, R., Jadad, A. R., Gauld, M., Boyle, M., Booker, L., Snider, A., & Cunningham, C. E. (2002). Attention-deficit hyperactivity disorder: Critical appraisal of extended treatment studies. Canadian Journal of Psychiatry, 47, 337–348. Shadish, W. R., Hedges, L. V., & Pustejovsky, J. E. (2013). Analysis and meta-analysis of single-case designs with a standardized mean difference statistic: A primer and applications. Journal of School Psychology, 52, 123–147. Shaw, M., Hodgkins, P., Caci, H., Young, S., Kahle, J., Woods, A.G., & Arnold, L. E. (2012). A systematic review and analysis of long-term outcomes in attention deficit hyperactivity disorder: Effects of treatment and non-treatment. BMC Medicine. Retrieved http://www.biomedcentral.com/1741-7015/10/99. Smith, M. L., Glass, G. V., & Miller, T. I. (1980). The benefits of psychotherapy. Baltimore, MD: Johns Hopkins University Press. Stage, S. A., & Quiroz, D. R. (1997). A meta-analysis of interventions to decrease disruptive classroom behavior in public education settings. School Psychology Review, 26, 333–368. Sonuga-Barke, E. J. S., Brandeis, D., Cortese, S., Daley, D., Ferrin, M., Holtmann, M., et al. (2013). Nonpharmacological interventions for ADHD: Systematic review and meta-analyses of randomized controlled trials of dietary and psychological treatments. American Journal of Psychiatry, 170, 275–289. Vallerand, I. A., Kalenchuk, A. L., & McLennan, J. D. (2014). Behavioural treatment recommendations in clinical practice guidelines for attention-deficit/hyperactivity disorder: A scoping review. Child and Adolescent Mental Health. doi:10.1111/camh. 12062. Van de Oord, S., Prins, P. J. M., Oosterlaan, J., & Emmelkamp, P. M. G. (2008). Efficacy of methylphenidate, psychosocial treatments and their combination in school-aged children with ADHD: A meta-analysis. Clinical Psychology Review, 28, 783–800. Visser, S. N., Bitsko, R. H., Danielson, M. L., Perou, R., & Blumberg, S. J. (2010). Increasing prevalence of parent-reported attentiondeficit/hyperactivity disorder among children—United States, 2003 and 2007. Morbidity and Mortality Weekly Report, 59, 1439–1443. White, D. M., Rusch, F. R., Kazdin, A. E., & Hartmann, D. P. (1989). Applications of meta-analysis in individual subject research. Behavioral Assessment, 11, 281–296. Wolf, F. M. (1986). Meta-analysis: Quantitative methods for research synthesis. Newbury Park, CA: Sage. Zwi, M., Jones, H., Thorgaard, C., York, A., & Dennis, J. A. (2011). Parent training interventions for attention-deficit/hyperactivity disorder (ADHD) in children aged 5–18 years. Cochrane Database of Systematic Reviews 2011, 12: CD003018.
123