HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS		QUICK SEARCH:   [advanced] Author: Keyword(s): Year:  Vol:  Page:

This Article Abstract Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal My Folders Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Fey, M. E. Search for Related Content PubMed PubMed Citation Articles by Fey, M. E.

Commentary on "Making Evidence-Based Decisions About Child Language Intervention in Schools" by Gillam and Gillam

Contact author: Marc E. Fey, KUMC, Department of hearing and Speech, 3901 Rainbow Boulevard, Kansas City, KS 66160-7605. E-mail: mfey{at}kumc.edu

	ABSTRACT

TOP ABSTRACT THE COSTS OF EBP SYSTEMS FOR RATING EXTERNAL... CONCLUSION: INTEGRATING EXTERNAL... REFERENCES

 

PURPOSE: This paper is a personal reaction to Gillam and Gillam's treatise on evidence-based decision making in schools. This evaluation focuses principally on the costs and benefits of clinicians' search for external evidence, potential problems associated with efforts to grade that evidence, and the integration of this evidence with "internal" evidence related to specific children and their families, clinician preferences and experience, and the school setting.

METHOD: Descriptive methods (that are not evidence-based) are used to evaluate critically Gillam and Gillam's proposals on the implementation of evidence-based practice (EBP) in schools.

CONCLUSION: Early efforts to implement the plan for EBP illustrated by Gillam and Gillam are likely to be difficult, forcing clinicians to be flexible and patient. Nevertheless, if the plan is followed, it will result ultimately in improved speech-language pathology services in the schools and improved clinical research to support those services.

KEY WORDS: evidence-based practice, efficiency, effectiveness, speech-language intervention

Evidence-based practice (EBP) has made its way into speech-language pathology as it has for most other health- and education-related fields. On the surface, it is difficult to see a downside to this development. EBP does not limit speech-language pathologists (SLPs) to the use of practices that have been demonstrated to be efficacious and effective, as is sometimes feared. On the other hand, it does require clinicians to (a) critically self-examine their own practices; (b) consider alternatives that may have stronger evidentiary bases; (c) justify their uses of approaches that have weaker support from research investigations than other available approaches; and (d) integrate available evidence from published literature with existing evidence concerning client and family preferences and needs as well as clinician experience, expertise, and theoretical perspective. Again, at least in principle, achievement of these outcomes would seem to represent a large step forward for our profession (Fey & Justice, in press; Johnson, 2006).

In practice, however, there are many obstacles to the implementation of EBP principles, especially over the short term. These problems are all tractable, but they are real and must be addressed head on. A number of these realities concerning the implementation of EBP are dealt with directly in the article by Gillam and Gillam. The authors describe basic EBP terms and principles and present the now relatively familiar set of steps to be followed by EBP clinicians. They also provide an example of how the procedures might actually be followed by clinicians who attempt to implement EBP for dealing with children with speech and language disorders. I find far more to agree with than to disagree with in their excellent article, so in this brief commentary, I will focus on some troublesome issues that their paper brought to mind for me. These issues include the costs to clinicians of attempting to do evidence-based speech-language pathology for school-aged children at this time and some problems or misunderstandings potentially associated with systems for rating levels of external evidence. I conclude with a comment on Gillam and Gillam's system of integrating internal and external evidence.

	THE COSTS OF EBP

TOP ABSTRACT THE COSTS OF EBP SYSTEMS FOR RATING EXTERNAL... CONCLUSION: INTEGRATING EXTERNAL... REFERENCES

 
Far and away, most of what has been written about EBP for children with language disorders has emphasized the importance of methods for searching and critically evaluating the literature for external evidence that should be brought to bear when addressing clinical questions. It is difficult to over-emphasize EBP steps related to searching the literature and analyzing the validity, importance, and precision of the research findings identified in that search (Dollaghan, 2004; Johnson, 2006). On the other hand, it is easy to underestimate the difficulties that busy clinicians are likely to have in searching and critically evaluating the extant research base relevant to a clinical question. Gillam and Gillam claim that "one strength of this procedure is that it does not require an undue amount of time" (p. 305). I would submit first that what constitutes "an undue amount of time" is relative and open to interpretation, and second, that especially in their early attempts to employ the procedures proposed by Gillam and Gillam and others, clinicians will find that the process does indeed take a good, and perhaps in the minds of some, undue, bit of time. Certainly, it will take longer than the 30 min per week that Sackett and colleagues (2000) suggest is a likely amount of time for medical practitioners to contribute to searching for external evidence as part of their EBP.

There are several reasons why searches will take longer than 30 min per week. First, searches limited to finding systematic reviews and randomized controlled trials (RCTs) often will yield limited if any information because there are relatively few such studies available for client populations with communication disorders, especially for those of school age. Second, relevant research articles that are available and may even report high-quality studies often are not written in a way that makes accessible the needed information, such as the extent to which testers and coders were blind to treatment conditions, the outcome measures used, and the size and clinical significance of observed effects. Third, searches that allow for weaker forms of evidence often will yield large numbers of studies that must be culled through carefully despite the fact that many are poorly controlled and poorly reported for purposes of EBP. Finally, the grading of research reports is not entirely objective, making it difficult for individual clinicians to feel certain that their evaluations of research reports are accurate and reliable. This is especially true when different studies report conflicting outcomes.

Over time, the number and magnitude of such problems is almost certain to diminish. For example, the number of relevant, high-quality studies that provide high levels of evidence will increase, as will systematic reviews of these studies. These improvements are inevitable as the need and pressure for such studies increases and external support for them from the National Institutes of Health, Department of Education, and other major sources of funding becomes increasingly available. In addition, reports of clinical studies now must meet uniform standards of presentation, making it easier for clinicians to glean the information needed to determine whether reading of the complete manuscript would be a wise time investment. For example, reports of RCTs that have been submitted to American Speech-Language-Hearing Association (ASHA) scholarly journals now must conform to the presentation standards of Consolidated Standards for Reporting Trials (CONSORT) (Moher, Schulz, & Altman, 2001). In addition, reports of nonrandomized clinical evaluations must follow the Transparency of Reporting Evaluations of Nonrandomized Designs (TREND) (Des Jarlais, Lyles, Crepaz, & Group, 2004), and studies of diagnostic accuracy must meet the Standards for Reporting of Diagnostic Accuracy (STARD) (Bossuyt et al., 2003). Even abstracts for journal articles are affected by EBP considerations. All ASHA journals now require a structured abstract that provides crucial EBP information, such as the research question, the research design, the observed effects, and effect sizes. Finally, as has happened in other areas within and outside of our profession, groups are likely to get together to develop evidence-based guidelines. These panels also have the opportunity to discuss crucial details of studies that are not always clear to the individual clinician or researcher evaluating a research report. Were the participants properly randomized? Were the transcribers and coders truly blind to participant group assignments and time of testing? Were the effects sufficiently precise and clinically significant? In their useful example dealing with intervention for morphosyntactic form, there were several instances in which my ratings of my own studies were not as generous as those given by Gillam and Gillam. The point is not really who is right in these decisions; it is that the decisions oftentimes are difficult to make even when the relevant information is available. The formation of groups charged with creating evidence-based guidelines would dramatically reduce the burden of searching and evaluating the literature on individual clinicians.

For now, however, clinicians who are committed to EBP will have to work long and hard to ferret out the information they need, basing their efforts on their faith that over the long haul they will learn to become better, more informed clinicians who are prepared to make better decisions for their clients and their families. I liken this situation to the cost of trying out a new computer software package. If the software is powerful and ultimately useful, it is likely that learning to use it will take a lot of time. The benefits of using it pay off only gradually over time, with the net returns being greatest once the user becomes facile with the nuances of the program. Clinicians attempting to adopt EBP are likely to have a similar experience, even if they are veteran data hounds and accomplished analysts of research reports.

In sum, from my perspective, it is better to lead clinicians to anticipate some significant growing pains in getting started than to intimate that the road to EBP will be relatively easy and cost-free. One way of limiting these pains, of course, is for clinicians to not go it alone. As Johnson (2006) and others have suggested, clinicians can develop working groups to share the workload involved in getting evidence-based answers to their most important clinical questions.

None of this is lost on Gillam and Gillam, of course, and I do not mean to imply that they have misled their readers into thinking that EBP is easy. In fact, much of their article exists to illustrate ways that clinicians can go about the business of EBP when the conditions for that practice are far less than ideal. Their article represents, at least in part, their effort to demonstrate what clinicians can do during the difficult time that the profession prepares itself to better support EBP practitioners. The case example that they provide also illustrates many of the complexities that clinicians will have to face when they attempt EBP and the costs involved in overcoming them.

	SYSTEMS FOR RATING EXTERNAL EVIDENCE

TOP ABSTRACT THE COSTS OF EBP SYSTEMS FOR RATING EXTERNAL... CONCLUSION: INTEGRATING EXTERNAL... REFERENCES

 
In their plan to rate levels of external evidence, Gillam and Gillam use an adaptation of the system of the Oxford Centre for Evidence Based Medicine, which is presented in column 1 of their Table 4. Although I endorse the use of this system (Fey & Justice, in press), there are three major points that I believe should be raised when considering it. First, not everyone agrees that evidence-based decisions for children with language disorders should be based on the results of RCTs (Johnston, 2005). This is especially the case when we do not have a critical mass of such studies.

Second, the levels presented in Gillam and Gillam's Table 4 are determined first by the design of the study and second by study quality. It is crucial to keep these two constructs distinct. For example, higher levels of evidence are first assigned to designs that include more and better controls for maturation and the effects of nontreatment variables, with true experiments or RCTs at the top and expert opinion at the bottom. The level of evidence provided by a study of any design, however, can be downgraded if the investigators have not taken certain steps to ensure that (a) the intervention was carried out as described; (b) the pre- and post-experimental measures were valid and reliable; and (c) the measures were collected by individuals who were unaware of the research questions, interventions, or time (i.e., pre- or post-) at which the data (e.g., language samples) were collected. Low-quality studies generate results that are not as credible as high-quality studies. The important point here is that we can speak of a high- or low-quality case study and a high- or low-quality RCT; quality and study design are orthogonal constructs.

If we do not consciously distinguish between research designs and study quality, it is easy to regard studies that provide lower levels of evidence as necessarily lower in quality than studies at higher levels and generally unsuitable for publication. Liza Finestack and I (Fey & Finestack, in press) have argued that this is a dangerous trap to fall into, one that could have negative ramifications for our efforts to develop both a solid body of evidence to support our clinical practices and a corps of young clinical researchers needed to contribute to that body of evidence. The reason for this is straightforward: Studies that produce higher levels of evidence generally require more participants, more controls, more time, and more resources to carry out. Junior-level researchers who begin their careers with such studies are almost certain to fail. Importantly, the motivation for higher level studies and the justification for sponsoring them financially generally come from studies that have already produced encouraging results using less costly, lower level research designs. Bright young researchers with innovative ideas must begin with these lower level designs and programmatically move upward as their ideas evolve and receive increasing levels of support. If they are to develop their research programs in this way, their work and its publishability must be based on the quality of their studies and the development of their clinical ideas rather than on the research designs they implement in their studies. As these researchers grow, so the body of evidence on which EBP clinicians can base their decisions will grow.

A third and closely related issue is that the method for determining levels of evidence proposed by Gillam and Gillam makes no reference to the distinction between efficacy and effectiveness. Efficacy studies examine possible cause–effect relationships between intervention variables and outcomes under ideal, laboratory conditions. Effectiveness studies logically serve as follow-ups to efficacy studies to determine whether the same or similar effects are observed under more typical clinical situations (Fey & Finestack, in press; Pring, 2004; Robey & Schultz, 1998). The extent to which clinicians can generalize the results of an efficacy study to their own clinical conditions will vary from one situation to the next, but clinicians should be aware that it may be difficult to replicate in their own clinical setting the effects of interventions shown to be efficacious in a laboratory context. Furthermore, there are many examples of RCTs in our field that demonstrate under laboratory conditions the efficacy of a procedure or strategy that would make up only one small part of a clinician's comprehensive service to a child. These "early efficacy" (Fey & Finestack, in press) trials may tell us which parts of our intervention approaches may be regarded as potentially "active ingredients." They may provide limited or no insights, however, into the performance of such efficacious procedures when they are implemented under more typical clinical conditions as part of much broader clinical approaches. Until effectiveness studies are completed, clinicians might reasonably question whether at least some clinical decisions are best based on efficacy trials alone, even if those trials are high-quality RCTs (Pring, 2004).

	CONCLUSION: INTEGRATING EXTERNAL AND INTERNAL EVIDENCE

TOP ABSTRACT THE COSTS OF EBP SYSTEMS FOR RATING EXTERNAL... CONCLUSION: INTEGRATING EXTERNAL... REFERENCES

 
Gillam and Gillam state quite correctly that "evidence-based practice does not require clinicians to use external research evidence as the only basis for clinical decisions" (p. 308). In fact, because EBP requires clinicians to integrate external evidence with information on client and family values and preferences as well as on clinician experience and expertise (Sackett et al., 2000), I would argue that clinicians who fail to consider client and clinician factors and attend only to research-based evidence are not doing EBP. Having said this, it is remarkable how little focus has been placed on the process of evaluating internal evidence and merging it with evidence from research. To their credit, Gillam and Gillam present a detailed plan for doing so, and they include consideration of internal evidence as a separate step in the EBP process they advocate. I find their general approach and specific proposals in this area to be the most innovative and thought-provoking aspect of their paper. Rather than simply talking about the integration of research-based and client- and clinician-based evidence, as is the rule, Gillam and Gillam are explicit in proposing levels of internal evidence that function as analogues to the levels proposed for external evidence. They then add these level assignments to a matrix in their Table 5 to illustrate how evidence from one domain might be pitted against that from another in forming a clinical decision.

It is fairly easy to quibble with their selection of level assignments for each different type of evidence. For example, they treat student–parent engagement as Level 4, below even family financial considerations. At least some interventions, however, such as our parent-administered focused stimulation approach to grammar facilitation (Cleave & Fey, 1997; Fey, Cleave, Long, & Hughes, 1993), simply are not possible without considerable commitment on the part of the parents. This suggests that level of parent cooperation should be considered at a much higher level than Level 4. Other clinicians will no doubt have even stronger disagreements with other aspects of the ordered system that Gillam and Gillam have proposed. To dwell on these disagreements in this venue, however, would miss the point. Gillam and Gillam recognize that their solution is not final, and they state explicitly that the levels may change in some inherent ways depending on a clinician's place of practice and the types of clients served. They merely regard their proposal as a point of departure. My hope is that, like the rest of their paper, it serves as the basis for discussion and even argument among clinicians and researchers about the best ways to go about the implementation and proliferation of EBP in speech-language pathology.

Received June 5, 2006
Accepted June 9, 2006

	REFERENCES

TOP ABSTRACT THE COSTS OF EBP SYSTEMS FOR RATING EXTERNAL... CONCLUSION: INTEGRATING EXTERNAL... REFERENCES

 

Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., & Irwig, L. M., et al (2003). The STARD statement for reporting studies of diagnostic accuracy: Explanation and elaboration. Clinical Chemistry, 49, (1) 7–18.[Abstract/Free Full Text]

Cleave, P. L., & Fey, M. E. (1997). Two approaches to the facilitation of grammar in children with language impairments: Rationale and description. American Journal of Speech-Language Pathology, 6, (1) 22–32.

Des Jarlais, D., Lyles, C., Crepaz, N., & Group, T. T. (2004). Improving the reporting quality of nonrandomized evaluations of behavioral and public health interventions: The TREND statement. American Journal of Public Health, 94, 361–366.[Abstract/Free Full Text]

Dollaghan, C. A. (2004). Evidence-based practice in communication disorders: What do we know, and when do we know it? Journal of Communication Disorders, 37, (5) 391–400.[CrossRef][Medline]

Fey, M. E., Cleave, P. L., Long, S. H., & Hughes, D. L. (1993). Two approaches to the facilitation of grammar in children with language impairment: An experimental evaluation. Journal of Speech and Hearing Research, 36, (1) 141–157.[Medline]

Fey, M. E., & Finestack, L. H. (in press). Research and development in children's language intervention: A 5-phase model. In R. G. Schwartz (Ed.), & Handbook of child language disorders.

Fey, M. E., & Justice, L. M. (in press). Evidence-based decision making in communication intervention Baltimore: Brookes.

Gillam, S. L., & Gillam, R. B. (2006). Making evidence-based decisions about child language intervention in schools. Language, Speech, and Hearing Services in Schools, 37, 304–315.[Abstract/Free Full Text]

Johnson, C. J. (2006). Getting started in evidence-based practice for childhood speech-language disorders. American Journal of Speech-Language Pathology, 15, 20–35.

Johnston, J. R. (2005). Re: Law, Garrett, and Nye (2004a). "The Efficacy of Treatment for Children With Developmental Speech and Language Delay/Disorder: A Meta-Analysis.". Journal of Speech, Language, and Hearing Research, 48, 1114–1117.[Free Full Text]

The CONSORT statement: Revised recommendations for improving the quality of reports of parallel-group randomized trials. Journal of the American Medical Association, (2001). 285, (15) 1987–1991.[Abstract/Free Full Text]

Moher, D., Schulz, K. F., Altman, D.Pring, T. (2004). Ask a silly question: Two decades of troublesome trials. International Journal of Language, & Communication Disorders, 39, (3) 285–302.

Robey, R. R., & Schultz, M. C. (1998). A model for conducting clinical-outcome research: An adaptation of the standard protocol for use in aphasiology. Aphasiology, 12, (9) 787–810.

Sackett, D. L., Straus, S. E., Richardson, W. S., Rosenberg, W., & Haynes, R. (2000). Evidence-based medicine: How to practice and teach EBM Edinburgh: Churchill Livingstone.

This article has been cited by other articles:

				N. J. Lass and M. Pannbacker The Application of Evidence-Based Practice to Nonspeech Oral Motor Treatments Lang Speech Hear Serv Sch, July 1, 2008; 39(3): 408 - 421. [Abstract] [Full Text] [PDF]

				T. Brackenbury, E. Burroughs, and L. E. Hewitt A Qualitative Examination of Current Guidelines for Evidence-Based Practice in Child Language Intervention Lang Speech Hear Serv Sch, January 1, 2008; 39(1): 78 - 88. [Abstract] [Full Text] [PDF]

				F. M. Cirrin and R. B. Gillam Language Intervention Practices for School-Age Children With Spoken Language Disorders: A Systematic Review Lang Speech Hear Serv Sch, January 1, 2008; 39(1): S110 - S137. [Abstract] [Full Text] [PDF]

This Article Abstract Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal My Folders Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Fey, M. E. Search for Related Content PubMed PubMed Citation Articles by Fey, M. E.