| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
Clinical Forum |
Contact author: Nan Bernstein Ratner, Department of Hearing and Speech Sciences, 0100 Lefrak Hall, The University of Maryland, College Park, College Park, MD 20742. E-mail: nratner{at}hesp.umd.edu
 |
ABSTRACT |
|---|
 TOP ABSTRACT HOW DOES ONE DEFINE...HOW DOES ONE EVALUATE...WHAT IS THE ROLE...HOW DOES ONE DISTINGUISH...HOW DOES A FIELD...WHAT ARE THE NEXT...CONCLUSIONREFERENCES |
|---|
PURPOSE: The purpose of this article is to consider some of the ramifications that arise when a discipline newly endorses evidence-based practice (EBP) as a primary guiding principle. Although EBP may appear straightforward, events experienced by peer disciplines that have preceded us in the implementation of EBP raise questions about defining acceptable forms of evidence for treatment effectiveness and efficacy, the potential roles of nonspecific or common factors, therapist quality in achieving therapy outcomes, and eventual applications of EBP that may overly confine which treatments are considered acceptable and reimbursable.
METHOD: Through narrative review of the literature, the article examines valuable as well as controversial features of EBP in addition to obstacles that may impede the transition of evidence (research findings) to clinical practice.
CONCLUSION: EBP is a valuable construct in ensuring quality of care. However, bridging between research evidence and clinical practice may require us to confront potentially difficult issues and establish thoughtful dialogue about best practices in fostering EBP itself.
KEY WORDS: evidence-based practice (EBP), common factors, outcomes
Recent changes to the requirements for the American Speech-Language-Hearing Association (ASHA) Certificate of Clinical Competence (CCC) have added a mandate that clinicians engage in evidence-based practice (EBP, Standard III-F). Specifically,
the applicant must demonstrate knowledge of processes used in research and the integration of research principles into evidence-based clinical practice... (and) must demonstrate comprehension of the principles of basic and applied research and research design. In addition the applicant should know how to access sources of research information and have experience relating research to clinical practice. (ASHA, 2005, p. 6)
As originally detailed by Sackett and colleagues, the process of EBP has a number of component behaviors and stages (Sackett, Rosenberg, Gray, Haynes, & Richardson, 1996; Straus & Sackett, 1998). Minimally, it includes the following assumptions: First, clinical skills grow with the application of currently available data, not simply personal, educational, and clinical experience. The practice of speech-language pathology is dynamic, and training received in graduate school, as well as clinical experience garnered over time, may not provide clinicians with knowledge of the most efficacious approaches to assessment and treatment of the cases they will confront in everyday practice over the span of their careers. In my case, a recent informal survey of the published literature using the database vendor EBSCO to identify various topics within the discipline of speech-language pathology (e.g., stuttering, aphasia, specific language impairment, dysphagia, etc.) that were published since my initial certification (1977) yielded roughly 20,000 potentially relevant publications that have appeared since I began my career as a speech-language pathologist (SLP). Clinicians cannot rely on what they learned in professional school to carry them through a lifetime of professional practice. Hess (2004) noted that on average, research evidence relevant to EBP doubles in less than 10 years. In order for practice to be effective, information must be updated constantly.
Second, the expert clinician should consistently seek new information to improve therapeutic effectiveness. To this end, clinicians should be data seekers, data integrators, and critical evaluators of the application of new knowledge to clinical cases. Thus, even if something appears to work, new information may assist the therapeutic process to work better. We cannot afford, and our clients cannot afford, for therapy to be less efficient or effective than it might be given the state of research available to us.
In this article, I will consider some of the relevant questions that emerge when a concept such as EBP transitions from goal to implementation. In doing so, I will discuss how one obtains and evaluates evidence relevant to practice. In an ideal world, treatments emerge, are validated through clinical trials, are disseminated to avid consumers of clinical updates, and are applied seamlessly to those whose profiles fit the trial population characteristics. In the "real" world of practicing clinicians, however, frequently employed treatments may actually have little firm validation data, may or may not reach the practicing clinician (who may or may not appear to endorse and use them), and may have unclear relevance to the actual profile of the client one is seeing.
In addition, as the obvious value of EBP expands from its origins in medicine to a host of allied health fields and education, it is not clear that its precepts can be lifted in their entirety. For example, as discussed further in this article, practitioner skill, interpersonal talents, problem-solving abilities, and so forth are difficult features of intervention to quantify but may be necessary to understand if we are to fully appreciate whether it is therapies, therapists, or both that do the work in achieving our best outcomes.
Thus, the purpose of this article is not to define EBP, but rather to ask some very general questions about how one takes EBP from the language of the standards to practice. Among the questions it poses are: How does one define evidence relevant to clinical practice? How does one evaluate the strength and applicability of clinically relevant evidence? What is the role of theory in guiding our interpretation of the evidence? How does one distinguish between the contributions of "practices" and "practitioners?" How does a field ensure that clinically relevant evidence reaches its constituency and is endorsed by professionals? Finally, we ask, What are the next steps required to make EBP transition optimally from its roots in medicine to speech-language pathology?
|
HOW DOES ONE DEFINE EVIDENCE RELEVANT TO CLINICAL PRACTICE? |
|---|
|
TOPABSTRACTHOW DOES ONE DEFINE...HOW DOES ONE EVALUATE...WHAT IS THE ROLE...HOW DOES ONE DISTINGUISH...HOW DOES A FIELD...WHAT ARE THE NEXT...CONCLUSIONREFERENCES |
|---|
In the absence of large-scale meta-analyses, clinicians may be aided by reports of the effectiveness or comparative effectiveness of specific therapies (e.g., recent comparison of the outcomes of clinic- vs. parent-administered intensive behavior interventions for autism [Sallows & Graupner, 2005], a recent randomized clinical trial [RCT] for the treatment of early stuttering using the Lidcombe method [Jones et al., 2005]), nonstatistical summaries of effective interventions [e.g., Justice & Pullen's 2003 recent review of three evidence-based approaches for promoting early literacy skills in children] and other contributions in this LSHSS forum). It is not the focus of this article to summarize the body of literature that supports EBP. A substantial degree of research literature backs many of the everyday practices of SLPs, although our field cannot currently boast as large a set of analyses as one might find in most areas of medical intervention.
However, this lack of an enormous current research base can be remedied somewhat as we progress in our evidence-gathering initiatives. For example, the notion that we need to search and read outside our own literature is particularly important given the early stages of data accumulation in our field. Many of the therapy practices in speech-language pathology have close parallels in other disciplines that have already documented their value; for example, there is a large body of work in psychology and other disciplines that confirms the utility of cognitive behavioral therapies in treating disorders that have affective and cognitive components, such as chronic stuttering (see Bernstein Ratner, 2005, for review). Additionally, as noted below, some of the most compelling direct evidence that what we do as SLPs "works" does not appear in our own discipline's journals (see recent examples for aphasia therapy by Jette, Warren, & Wirtalla, 2005, and Meinzer, Djundja, Barthel, Elbert, & Rockstroh, 2005).
How Much Evidence Is Required to Practice EBP?
When relevant, peer-reviewed information is located, we must also ask how much information is enough information to change current clinical practice. There are some issues that arise in this regard. First, it is always important for research findings to receive empirical support via replication by nonstakeholders—a problem that has been identified and compounded by a publication bias against replication studies (Thomas & Howell, 2001). The emergence of therapies with what can be a large body of findings produced by single research teams is problematic for more reasons than simple ability to validate and generalize findings. For example, there are numerous outcome reports on the effectiveness of the Lidcombe program (Jones et al., 2005) in treating early stuttering; however, virtually all have had at least one program developer as a co-author. Investigator allegiance to a particular intervention has been shown to be an incredibly potent factor in outcomes: Shoham and Rohrbaugh (1999) reviewed data that suggest a correlation as high as .85 between ratings of differential outcomes and therapist allegiance to a particular therapeutic approach. Thus, multicenter replication of therapy approaches is critical when asking if a therapy is effective.
The opposite of too little data is the problem that sometimes more data appear to be more confusing when one attempts to select optimal intervention approaches. In fields with much more active research programs examining major health issues, it is not uncommon for numerous studies on a topic to produce inconsistent findings (e.g., current nutritional advisements or guidelines for optimal weight, hormone replacement therapy, and any number of common and important health questions). Even the Cochrane Reports often conclude that data on particular narrowly specified treatments are quite equivocal, despite large numbers of carefully conducted studies on large numbers of patients. Cohen, Stavri, and Hersh (2004) concurred, noting that the results of many published RCTs in medicine are frequently in disagreement with each other.
We should not despair, however. Even in our limited literature, some meta-analyses are suggesting the effectiveness of speech-language pathology in treating phonological disorders and expressive vocabulary delay in children (Law, 2004), and whereas some meta-analyses of medical interventions disagree in their interpretations of existing evidence, others provide strong rationales for professional decision making. For example, Cohen et al. (2004) felt that the EBP initiative has tremendous potential to make important changes in the field of preventive medicine.
|
HOW DOES ONE EVALUATE THE STRENGTH AND APPLICABILITY OF CLINICALLY RELEVANT EVIDENCE? |
|---|
|
TOPABSTRACTHOW DOES ONE DEFINE...HOW DOES ONE EVALUATE...WHAT IS THE ROLE...HOW DOES ONE DISTINGUISH...HOW DOES A FIELD...WHAT ARE THE NEXT...CONCLUSIONREFERENCES |
|---|
EMDR, as some analysts of EBP in psychology have observed, has accumulated some reasonable evidence of effectiveness. A relatively simple procedure, it has rapidly spread over the past decade to the application of such diverse problems as post-traumatic stress disorder, phobias, sexual maladjustment, and even dermatological disorders. However, as many (e.g., summaries in Follette & Beitz, 2003; Lohr, DeMaio & McGlynn, 2003; Rosen & Davison, 2003) have noted, EMDR's mechanisms of action are not only unclear, but those proposed by its originator have been found to be irrelevant to outcomes. Thus, some scientists have offered it as perhaps one of the most compelling modern examples of a pseudoscientific treatment (Herbert et al., 2000), one that may or may not work, and whose outcomes are clearly mediated by common or extraneous factors independent of the treatment components. Arguably, what is working in EMDR is the older, less currently researched concept of systematic desensitization (a notion familiar to those who treat stuttering), leading one researcher to note unhappily as he observed the raging popularity of the treatment that "what is effective in EMDR is not new, and what is new is not effective" (McNally, 1999, p. 619).
|
WHAT IS THE ROLE OF THEORY IN EVALUATING EVIDENCE? |
|---|
|
TOPABSTRACTHOW DOES ONE DEFINE...HOW DOES ONE EVALUATE...WHAT IS THE ROLE...HOW DOES ONE DISTINGUISH...HOW DOES A FIELD...WHAT ARE THE NEXT...CONCLUSIONREFERENCES |
|---|
Evidence also needs to be evaluated within a context that asks, What type of information makes sense to me as a practicing clinician? In many fields, there is concern that therapy practices must be grounded in theory that adequately explicates the underlying mechanisms of change (Lipsey & Cordray, 2000; Rosen & Davison, 2003; see a discussion that relates this concept to fluency therapy in Bernstein Ratner, 2005). Unfortunately, whether one examines the literature of clinical psychology, family counseling, education, or other related disciplines that are moving toward assimilating EBP principles that originated in medicine, there is a uniform complaint that "research into why and how therapy works is virtually non-existent" (Larner, 2004, p. 21).
Ideally, our field needs therapy outcome results that cohere with theory. Why might some approaches to fluency treatment, aphasia, articulation, and child language learning be expected to work well? One might choose an intervention based on research literature that pinpoints a specific mechanism that produces the problematic behavior and an intervention that then ameliorates the underlying problem. "Science involves more than demonstrating experimental control over some phenomenon. An essential task of science is understanding the phenomenon under study" (Hughes, 2000, p. 304); you need to know the mechanisms in order to feel confident that you can take the therapy into professional practice, problem shoot, and branch as necessary. Treweek (2005) trenchantly argued that complex medical interventions, which may be seen as somewhat analogous to the types of interventions that SLPs implement, should be particularly guided by theory, although to date, few are, even though there are numerous potential theories to guide interventions. Michie (2005) made similar arguments for the place of theory in understanding and interpreting behavior change, including that which is necessary to translate evidence into practitioner adherence and bridge analysis of the respective contributions of therapies, therapists, and clients.
Thus, the notion of "mechanism of action" has emerged as a major concern in evaluating the eventual value of therapy efficacy findings as fields enter the EBP fray, as noted elsewhere in this article. In addition to EBP's roots in medicine and pharmacology, there are renewed calls for explication of the means by which a therapy is presumed to accomplish its goals in psychology (Rosen & Davison, 2003) and in education, where the National Academy of Sciences has made it a primary component of research design (Odom et al., 2005). Without mechanism of action, it also appears more difficult to achieve practitioner "buy in" of new therapy approaches—a problem that has been identified in medicine. In particular, "life experience" may lead some physicians to resist newer contributions to EBP when it does not mesh with their particular cases or experience (Pope, 2003). A few physicians have lamented that "discussions on evidence-based medicine tend to concentrate on research methodology and reduce clinical practice to the technical implementation of research findings," minimizing their years of clinical experience in knowing what appears to work and what does not (van Weel & Knottnerus, 1999, p. 917). They suggest that equal prominence be given to the evidence we see in daily record keeping if we do it correctly. In particular, it appears difficult to change longstanding practice patterns in our fellow professions, and it is not yet clear in most cases that we should, if it has yielded satisfactory outcomes: "When knowledge is applied to the individual patient, the logic of affirmation often over-rules the scientific logic of refutation. ... [This] tacit knowing held and applied by proficient practitioners... represents a valuable form of clinical knowledge...which should be investigated, shared and contested" (Malterud, 2001, p. 399). Thus, as we explore the dissemination of best practice guidelines, we should be open to a two-way discussion of how such recommendations mesh with clinicians' experiences "in the trenches."
|
HOW DOES ONE DISTINGUISH BETWEEN THE CONTRIBUTIONS OF "PRACTICES" AND "PRACTITIONERS?" |
|---|
|
TOPABSTRACTHOW DOES ONE DEFINE...HOW DOES ONE EVALUATE...WHAT IS THE ROLE...HOW DOES ONE DISTINGUISH...HOW DOES A FIELD...WHAT ARE THE NEXT...CONCLUSIONREFERENCES |
|---|
One reason might be that few documented evidence-based treatments fit the full profile of client complaints; sometimes, the clinician sees the mismatch or incompleteness, and unfortunately, sometimes only the client does. For example, in the past month, a stuttering listserv that is prowled by both clinicians and clients provided numerous complaints by consumers that their therapists have tended to concentrate on fluency counts to the exclusion of other client needs in the treatment of stuttering. The statements brought to mind a comment I saw from a physician who was less than overwhelmed by the current emphasis on EBP: "The task of the physician is two-fold—to understand the patient and understand the disease" (Malterud, 2001, p. 397). Unfortunately, although we have clinical methods for accomplishing the second, we are much further behind in understanding the first. Arguably, understanding the patient might require some unified theory of the universe, but we should not confuse knowing about conditions with knowing our clients and patients.
When We Get Good Outcomes, Is It the Therapies or the Therapists?
For many fields, but not yet our own, there is growing evidence that distinguishes among therapies and therapists in terms of their respective impacts on desirable therapeutic outcomes. In clinical psychology, in particular, the issue of "nonspecific effects" or "common factors" (Lambert, 2005) has been deeply argued and researched. For example, in psychotherapy, studies tend to show that many different treatments work equally well, suggesting the importance of nonspecific treatment effects (Deegear & Lawson, 2003). These effects include the client's and the clinician's expectations for improvement, therapist confidence, problem-solving abilities, ability to establish a therapeutic relationship that fosters risk taking, and efforts at skill mastery, among other things (Lambert, 2005).
The medical model on which ESTs are based says, "Seek a therapist who uses techniques with demonstrated ability to alleviate your condition," whereas the contextual (common factors) [emphasis mine] model advises, "Seek an interpersonally competent therapist who uses a treatment approach that you find compatible with your worldview." (Messer, 2004, p. 582)
In fact, in some psychological therapies, the outcomes that appear to be attributable to the therapist's specific participation in evaluation of therapies is "an order of magnitude greater than the proportion [of variation ... in outcomes] due to [specified] treatments" (Wampold, Lichtenberg, & Waehler, 2005, p. 36). In other words, in some meta-analyses of therapy outcomes, therapists seem to matter more than therapies in achieving outcomes. The same authors flatly warn that, "it is grossly misleading to identify efficacious interventions and ignore provider differences" (p. 36), which may differ dramatically in eventual therapeutic effectiveness apart from choice of specific intervention techniques. Whether or not common factors render therapies less important than therapists is much a matter for continued investigation.
Although clinical psychology has identified a large problem in discriminating common factors from active therapeutic factors specific to particular therapy approaches, social work has found relatively few common factors and has confirmed that some interventions do appear to work better than others (Reid, Kenaley, & Colvin, 2004). Where speech-language pathology will fall on this continuum remains to be seen. However, it would seem intuitive that in our field, it would be a rare therapist who adheres strictly to a small set of well-specified treatment approaches; I cannot recall fielding too many calls over the years that asked me to recommend a practitioner of a treatment rather than a "good therapist."
Another problem in linking outcomes specifically to treatments: Fitting the treatment to the client. Thinking about therapies, therapists, and clients as though they are freely exchangeable and recombinable elements may not be wise. In a recent study (Beutler, Moleiro, Malik, & Harwood, 2003), fit of client factors to a therapy approach was more predictive of positive outcome for the treatment of depression and addiction than the specific intervention or nonspecific factors. This implies that some therapies will work better for some clients and vice versa, but we rarely explore this possibility in terms of anything other than presenting symptoms. Ellis Weismer (1991) made a similar call many years ago that we need to explore which therapies or approaches for child language remediation work best for particular children, rather than seeking out therapies that were presumed to work equally well for all cases we see.
Types of evidence: What to do vs. what NOT to do. The usual heat in publishing treatment outcome studies is to advance a therapy approach, document it, and advocate it. We usually do not get as much information about what does not work, which is what is needed to paradigm shift. Similarly, getting the ineffective approaches out of the system seems perversely difficult. As an example, I received a recent e-mail from an SLP asking me whether she should use auditory integration (AI) therapy with children who stutter, despite two troubling facts: that there is nothing about the nature of the disorder or the therapy that suggests that it would logically apply to stuttering in the first place, and that ASHA distributed a position paper in 1994 warning that there is no evidence to support the effectiveness of AI, recently re-affirming its stance in 2004. As we prepare to disseminate practice guidelines, we should be mindful that we need to convince therapists to replace existing therapeutic approaches with newer ones, but also to consider that, not only may they be using approaches that have not yet been fully validated, but they may actually be doing something that has already been found to be ineffective—an ugly but true problem. My personal suspicion is that such approaches stay alive in the more typical contexts of clinical problem solving, such as consulting old texts or acquaintances for therapeutic ideas rather than the peer-reviewed literature—a trend we have identified in recent research (Nail-Chiwetalu & Bernstein Ratner, 2006, in press; Zipoli & Kennedy, 2005).
How Should SLPs Rank the Quality of Evidence? The Example of RCTs
Imitation may be a very sincere form of flattery, but a number of the disciplines that are now joining the EBP movement are discovering that one cannot import all of its tenets regarding the ranking of evidence (and even the medical field is tempering some of its perspectives on its nature, interpretation, and application). For example, medical EBP's emphasis on RCTs as the "gold standard" for validating treatments may be misguided for some medical problems and many nonmedical interventions. Large aspects of our therapies do not lend themselves to the principles of RCTs (e.g., one cannot easily administer placebos, nor can one blind clients and therapists; it is difficult to see how one might wash out or remove the effects of cognitive training, as a few examples; see additional criticisms regarding the appropriateness of RCTs for speech-language pathology by Pring, 2004.)
Further, the strengths of RCTs (high internal validity for efficacy findings) can be offset by problems in the generalizability of results (Gonzales, Ringeisen, & Chambers, 2002). In fact, the characteristics of clients who are usually involved in RCTs are thought to be somewhat distinct from those of the larger body of clients having the condition under study. Thus, medicine has more recently realized that it must combine complementary methods of research.
Although many physicians have historically regarded qualitative methods of inquiry into disease management as "soft" (Malterud, 2001), there have been recent calls to combine quantitative and qualitative methods in ensuring that large-scale research findings can effectively apply to individual case profiles. Likewise, in addressing the obvious heterogeneity of clients, physicians are urged to ask themselves: "Could my patient have been randomized in this trial?" (Summerskill, 2005). It is increasingly recognized that the better that family practitioners feel they know their clients, the less likely they are to apply external evidence to guide management. Recent studies show that a number of physicians, even when they are aware of published evidence regarding particular treatments, disagreed with the conclusions of systematic literature review; "factors other than the evidence obviously affects group judgments" (Burgers & van Everdingen, 2004, p. 392) In short, professionals often find it hard to identify their particular clients in the published reports.
In adopting the RCT as a standard, medicine has had a more rational movement from basic science to practice (e.g., drug testing, from mechanisms of action to animal models to clinical trials) that aids the interpretation of therapy trials and fine-tuning of treatments. Thus, although we have some excellent tutorials that demonstrate how RCTs can be viewed as a gold standard for both medical and SLP practice (e.g., Robey, 1998), we need not only understand their possible limitations in studying some of our most common treatment approaches, but should also consider that the "gold standard" studies in intervention research are those that not only demonstrate efficacy, but also demonstrate that "the postulated change mechanisms...do indeed carry the weight of improvement on treatment outcomes" (Hughes, 2000, p. 307). In other words, all disciplines need to be able to identify the "active ingredients" of a treatment. This has been a concern for some in our field when evaluating the relationships between therapy components and outcomes in programs that treat child language disorder or early stuttering (see Bernstein Ratner & Guitar, 2006, for discussion).
What Is Our Evolutionary Stage in the Application of EBP and Where Do We Need To Go?
Compared with medicine, we are a newer, less well-funded initiative. We start from a very low level of documentation, meaning few therapies are robustly validated and most are uninvestigated using systematic, large-scale methodology. Additionally, as a field, we treat many conditions across a wide age range and can be seen as analogous to general practice in medicine, where RCTs have shown the least progress in identifying best practices (Pring, 2004). It has been noted that more than half, and perhaps up to 85%, of all mainstream medical treatments have never been validated by clinical trials (Steinberg & Luce, 2005). For those medical treatments that can claim some sort of evidence base, it has been estimated that only 4% have a base that can be considered robust (Steinberg & Luce, 2005)—a fact that might not be comforting for my peers who might be reading this while sitting in a doctor's waiting room.
There are also subtle biases at play in disseminating evidence through peer-reviewed journals. As Maher, Sherrington, Elkins, Herbert, and Moseley (2004) noted, at least two of these biases can yield an evidence base that may not be representative of the true success rates of various therapies. The first is a publication bias toward positive, rather than negative, results. Reports of therapies that work are more likely to be published than are reports of therapies that do not work. In most fields, there also appears to be a practitioner bias toward reading reports that have been disseminated to the Internet in full text rather than abstract only (see also Nail-Chiwetalu & Bernstein Ratner, in press). This influence on practitioner awareness of evidence has been given the picturesque acronym of the FUTON (full text on the net) bias.
Moreover, even validated treatments often do not "cure": In published studies across medical, psychological, and allied health fields, 50% to 80% of "successful" cases retain some of their original symptoms, and Treweek (2005) estimated that complex interventions achieve only an average of 10%–20% change in behavior. Moreover, empirically supported treatments (ESTs) tend to impact only 50%–70% of patients who participate. We should not view this as an indictment of ESTs, but rather as a reminder that many of our clients will need some other form of attention paid to whatever is left over after an EST has been applied (Paul, 2004), or they will not respond to a treatment the way the majority of participants did.
Moreover, we need to be aware that the lay public often does not understand this (and perhaps our own fellow professionals are confused by this notion as well). Our standard for effectiveness of a treatment cannot, in most cases, be complete restoration of normal function because we do not yet know if this is a truly reasonable expectation for the vast majority of individuals having a particular diagnosis, such as stuttering or specific language impairment (SLI). In particular, it is hard to imagine what kind of child language testing identifies the totality of the child's strength and weaknesses, and in fact, "teaching to the test" is not a recommended strategy (Lund & Duchan, 1993). For such children, remediation will always be to some extent opportunistic, taking advantage of being able to witness appropriate or inappropriate language use in context and then responding to it with suitable strategies that maximize the moment of learning. Whether or not we ever bring such children to a level of function that is fully equivalent to that of children who were not previously diagnosed with SLI is somewhat in doubt (Johnson, Beitchman, & Young, 1999; Law, 2004; Law et al., 2003; Rescorla, 2005). This makes what we do somewhat distinct from some problem spaces in medical or other allied health fields.
Ethical Concerns in Gathering Evidence
Although we may not have a deep EBP base, we are not such a new and naïve field that we can explore some factors in therapy outcomes ethically—for instance, can we ever justify comparison groups that receive no or "bogus" treatment (thus controlling for common effects)? Will a child's genetic history suggest risk factors for intervention for some disorders known to have such high remission rates (e.g., language delay, stuttering) that we currently cannot decide if any treatments for them exceed the benefits of the normal developmental process? Other allied health professions (e.g., occupational therapy) are in a similar predicament, having sparse amounts of efficacy data (Steultjens, Dekker, Bouter, Leemrijse, & van den Ende, 2005) yet difficulty in randomizing patients to no-treatment conditions.
We may also face more difficult EBP questions than does medicine. Berliner (2002) suggested that research in education, one of our peer professions, represents "hardest-to-do science" rather than hard science. We cannot just ask if something works, as in a drug therapy or surgical procedure; we need to ask for whom it works, in what setting, and how. As such, we may need to keep administering current therapies until we can validate them. What are our other options?
What Might be Appropriate Directions To Take Next?
Law (2004) noted that EBP is "a journey not a state" (p. 218). There are many potential directions to take in the near future; the following appear reasonable options to me: I believe that we should emphasize collaboration among sites to validate/invalidate existing therapies that are in common use (e.g., determining the nature of current practice strategies and then assessing their efficacy, as opposed to the heat and light that often accompany newer, emerging therapy proposals). This is important because, compared to medicine, we have relatively few academic programs that routinely generate peer-reviewed research studies, we have relatively few hospital or center-based research groups, and the number of clients required to achieve adequate statistical power in evaluating therapy outcomes can be quite enormous (Harrison, Onslow, & Menzies, 2004).
I believe that we need to keep an open mind about therapies that are conventional yet not yet fully documented (no evidence that something works YET is not the same as evidence that it does not work) (see Pring, 2004, for similar comments).
The fact that some therapies have been shown to be efficacious in RCTs does not mean that others are invalid. Meta-analyses...have found very few differences among...therapies...that have a firm theoretical structure, have been practiced extensively over time, and have a research foundation. (Messer, 2004, p. 582)Larner (2004) put it more bluntly:
It is scientific to be open and not foreclose on what works in therapy. There is a difference between grounding clinical practice in empirical research and rejecting any therapeutic approach or idea that has not first been demonstrated to be RCT effective. One is evidence-based, the other is evidence-obsessed [italics added]. (p. 28)
I believe that we need to be less territorial about "best practices" as though there may be just one per problem area. In particular, we may need to be less combative about contrasting ESTs and "less scientific" approaches to validating therapy. As Messer (2004) noted,
The issue need not be framed so starkly as science versus humanism or the use of evidence versus clinicians' subjectivism and clinical judgment. ESTs provide one kind of relevant data on which clinicians can rely, but they are not and cannot be the sole source of evidence-based practice. (p. 586)In particular, we need to be sensitive to the fact that by definition, most evidence that is obtained in support of EBP applies to populations rather than individuals. As such, we need to extend the questions beyond the therapy to client characteristics, intervention setting, and other service delivery variables (Gonzales et al., 2002). In my survey of other fields transitioning to EBP principles, academically based clinical educators also emphasized the need for and value of training in and use of single-subject research designs.
From a practical standpoint, the vast majority of applied psychologists will never conduct a large n study once they complete their training. However, these same psychologists will see hundreds of clients over the life of their careers, often without once collecting data because of little or no training in single-case designs. Teaching trainees how to efficiently evaluate change processes on a case-by-case basis would not only be in keeping with the spirit of the scientist–practitioner, but could provide potential databases. (Deegear & Lawson, 2004, p. 276)The observation appears extremely germane to our profession. The utility of single-subject research design is virtually absent from most discussions of EBP in most fields, although there has been a recent call for its inclusion (Cohen, Stavri, & Hersh, 2004).
Providing Visible Role Models for the Application of EBP
The Evidence-Based Intervention Work Group of school psychologists (2005) noted that getting an effective intervention broadly applied requires a number of factors, including congruence of the intervention with the individual's pre-existing knowledge, beliefs, and experience. However, the group additionally targeted the instant influence of peer role models (social proof) as well as successful trigger experiences (i.e., single demonstrative proofs that an approach works successfully). Their discussion of the "stages of concern" that accompany any innovative practices makes it clear that simply making information available is but the lowest rung of a fairly complex ladder that requires clear explication of how to implement a new intervention and how to contrast its benefits and potential trade-offs with what is currently done.
It is important to note that ASHA and the Special Interest Divisions are taking an active leadership role in highlighting EBPs, documenting their evidence bases, explicating their mechanisms, tangibly demonstrating their application through multimedia instructional modules, and overtly contrasting them with additional evidence-based options as well as options known not to be efficacious. Among the current activities that should help to refine and expand the best aspects of EBP for practicing clinicians are the following:
Activities such as these should greatly enable thoughtful application of EBP within our discipline.
|
HOW DOES A FIELD ENSURE THAT CLINICALLY RELEVANT EVIDENCE REACHES ITS CONSTITUENCY AND IS ENDORSED BY PROFESSIONALS? |
|---|
|
TOPABSTRACTHOW DOES ONE DEFINE...HOW DOES ONE EVALUATE...WHAT IS THE ROLE...HOW DOES ONE DISTINGUISH...HOW DOES A FIELD...WHAT ARE THE NEXT...CONCLUSIONREFERENCES |
|---|
First, in some fields, there really is too much evidence being generated for practitioners to keep up with, let alone access and interpret (Burke, Judelson, Schneider, DeVito, & Latta, 2002; Davis, Ciurea, Flanagan, & Perrier, 2004; Ely et al., 2002; Saint et al., 2000). A second problem is what we might call the mindset of the average health practitioner. A number of surveys of health practitioners indicate that, when faced with a clinical question, the vast majority of professionals do not consult the current peer-reviewed literature that is germane to the topic. Rather, they are more likely to rely on the opinions of professional colleagues, refer to old texts, or consult generalized Web sites (Cullen, 2002; McAlister, Finlay, Graham, Karr, & Laupacis, 1999; Rappolt & Tassone, 2002).
Surveys of randomly selected practicing SLPs show a very similar profile (Nail-Chiwetalu & Bernstein Ratner, 2006, in press; Zipoli & Kennedy, 2005), as does a recently completed study that was conducted by the ASHA National Office (Mullen, 2005). We need to more strongly establish a culture that positions the peer-reviewed literature as the first bastion of clinical question seekers.
In encouraging professionals to consult the literature, we need to adopt the combined focus of ensuring that practitioners know what has been shown to be ineffective (in my example of sensory integration therapy, ASHA position papers do not appear to have had a pervasive impact on some SLPs' practice patterns) through more active dissemination of position papers, while encouraging a culture that emphasizes the need to search out and use the evidence being produced daily that is relevant to clinical practice decisions. This is not the default in most fields, and in fact, some recent research on how to bridge research findings into clinical practice has revealed that even targeted dissemination of printed materials urging the adoption of practices does not greatly impact clinical behavior in health professionals (Grimshaw, Eccles, & Tetroe, 2004). What does seem to work is structured educational outreach in which the peer-reviewed evidence is paired with face-to-face instruction in the rationale and method for adoption of alternatives to current practice. In this regard, it is not clear that the new CCC continuing education unit requirement will accomplish this end if clinicians prefer, as they seem to, non-peer-reviewed local workshops to reading the primary research literature or understanding its value.
Finally, there are real and substantive barriers to the use of evidence by the vast majority of our professional colleagues. Those who are not currently affiliated with well-funded university libraries will find that accessing the most inclusive of academic databases to locate research citations, and then purchase of the full-text versions of these articles, can be somewhat costly, requiring database or per-print subscriptions. Although the recent initiative by ASHA to place all ASHA journals online in full text, accessible to all members, is extremely admirable, one must recognize that not all of the relevant clinical data will be found in ASHA journals. A recent and relevant example is the recent study by Meinzer et al. (2005) in Stroke regarding the efficacy of an intensive speech-language therapy for stroke patients in diminishing the impact of aphasic symptoms.1
After reviewing effective ways to change clinician behavior, Grimshaw et al. (2004) suggested that approaches to altering existing treatment approaches toward newer EBPs may need to be quite multifactorial. Approaches should provide targeted outreach that emphasizes the research bases for recommendations, provide written materials, and clearly demonstrate the practice that is targeted for dissemination. Follow-up should also include audits of the facility or individual to inquire whether or not practices have changed, reminding providers of the intervention recommendations, and work with local opinion leaders.
Their meta-analysis, however, of the degree to which such intensive programs may change clinical behavior was somewhat sobering: Although many studies of professional behavior change strategies have shown positive results, the mean effective change from prior behavior was only about 10%. Thus, as we adopt EBP principles, we should already be considering what steps may need to be taken to ensure broad adoption of those we consider to be most important in treating our clientele.
Some researchers have argued that systematic reviews may enable practitioners to more quickly identify and use the evidence as it emerges in particular fields. As noted earlier, there have been few systematic reviews conducted to date in speech-language pathology (see Law et al., 2003). Additionally, those that have been conducted have been criticized on a number of grounds that were touched on in different places in this article: They tend to be atheoretical (Pring, 2004; Treweek, 2005), reflect publication bias (Mykhalovskiy & Weir, 2004), and may collapse together studies differing enough in methodology and quality to obscure or misrepresent outcomes (Law, 2004).
|
WHAT ARE THE NEXT STEPS REQUIRED TO MAKE EBP TRANSITION OPTIMALLY FROM ITS ROOTS IN MEDICINE TO SPEECH-LANGUAGE PATHOLOGY? |
|---|
|
TOPABSTRACTHOW DOES ONE DEFINE...HOW DOES ONE EVALUATE...WHAT IS THE ROLE...HOW DOES ONE DISTINGUISH...HOW DOES A FIELD...WHAT ARE THE NEXT...CONCLUSIONREFERENCES |
|---|
Wampold et al. (2005) suggested some principles that may be used to guide fields seeking to begin the systematic practice of endorsing empirically supported interventions. They are:
Systematically tackle various increasing levels of specificity when documenting outcomes. A discipline may first wish to endorse broad levels at which interventions can be conceptualized to be effective. An example might be family counseling or aphasia rehabilitation of all types. A narrower level of specificity might be group versus individual therapy or counseling, or family-based therapies (see summary in Hoagwood, 2005). For our field, arguably, knowing whether or not children who are treated in groups do as well as those who are treated individually might have much wider ramifications for the large proportion of our client base who are treated in school settings than will the validation of any specific treatment protocol.
Still narrower might be more specified approaches to therapy (e.g., contrasting different articulation therapies with children having articulation disorders of unknown etiology, or fluency shaping vs. stuttering modification vs. operant programs for those who stutter). The narrowest level of specificity would document that a particular therapy is most effective for a particular client profile (e.g., the potential discovery that some early interventions for stuttering may be more or less efficacious given the child's concomitant skills in language and/or articulation or family history of persistent stuttering). Wampold et al. (2005) suggested that fields should begin with the first set of questions and fully document those before arguing the merits of narrower levels of treatment efficacy evidence.
Related to this is the second of their proposed principles—that we should not limit our sights to interventions defined by diagnosis. If we look, we may find that certain client attitudes, beliefs, or cultural values suggest the relative merits of competing general approaches to the therapeutic relationship and general approach to intervention selected.
A third principle endorses systematic and statistically rigorous meta-analysis of a complete body of literature on a particular intervention, a noncontroversial concept, but one for which we have relatively little recent data for most aspects of our discipline's activities (Law, 2004; Pring, 2004), although work is currently underway in many circles to accumulate this type of knowledge.
A fourth tenet suggests that we never confuse absolute efficacy (treatment vs. doing nothing) with relative efficacy. As noted earlier, finding real differences between competing approaches is difficult once many common factors are eliminated. At that point, the small margin of efficacy of a given approach may be obliterated by administering it regardless of client circumstances or preferences. The authors use the example of behaviorally oriented programs offered to individuals who prefer a more socioemotional approach to treatment.
A fifth principle cautions that we must verify the active ingredients in a therapy approach through careful analysis of potential mediating factors, manipulation of suspected active ingredients, and other steps we have mentioned when discussing the value of "dismantling" therapies to understand them more fully.
A sixth principle suggests that outcomes be broadly, rather than narrowly, defined, and should include functional outcomes, client perspectives on the treatment approach, as well as cost-benefit analysis of the therapy approach.
A seventh and final principle suggests that because local and individual application of even validated treatment approaches may yield different outcomes than carefully controlled studies, "science should inform practice [italics added], rather than ... the alternative, [that] science should dictate practice (Wampold et al., 2005, p. 35).
The possibility that interpretations of the science might dictate practice already appears to be a reality in some states and health intervention areas, as Tanenbaum (2005) noted, using examples from the District of Columbia, mentioned previously, and other jurisdictions. She suggests that we need to keep alert to the notion that EBP inherently involves solving some controversies, while maintaining awareness of a potentially troublesome outcome. The first controversy is definition of evidence. It is but a short leap from narrow definitions of evidence to short lists of validated treatments. We will need to assess the relative values of "treatments" as opposed to more loosely defined sets of therapeutic behaviors that characterize adept clinicians and certain types of therapy approaches.
The second controversy will involve determination of how research evidence is best applied. How should it be disseminated? How does probabilistic research impact the treatment of individual clients? How much discretion will be endorsed in the application of intervention guidelines? This discretion may begin as a debate among peer professionals, but is likely, in today's climate, to gravitate quickly to managed care. Our field needs to be cognizant of that. In particular, Tanenbaum (2005) noted that highly manualized treatments do have the eventual potential to completely change how therapy is delivered. "In one well-developed [mental health provider] scenario, ... practice adheres increasingly to research-based manuals, and most treatments are performed by clinicians with less training." (p. 168). In contrast, those who are perceived to be more highly trained or more adept at problem solving will design the treatments, conduct research on their effectiveness, manage the quality of service provision, and serve as the second line of defense for those who do not respond to "manualized" care. For a discipline that has recently considered the "pros" and "cons" of credentialing therapy aides, these could be unsettling notions. For a field that is currently considering the potential value of clinical doctorates for SLPs in addition to audiologists (Lubinski, 2003), such scenarios may be frankly discordant with how our profession views its disciplinary growth.
|
CONCLUSION |
|---|
|
TOPABSTRACTHOW DOES ONE DEFINE...HOW DOES ONE EVALUATE...WHAT IS THE ROLE...HOW DOES ONE DISTINGUISH...HOW DOES A FIELD...WHAT ARE THE NEXT...CONCLUSIONREFERENCES |
|---|
As we conduct such studies, we need to be prepared for a number of outcomes. The first one seems to be anticipated by many—that some therapies will turn out to be better than others. But this may not, in fact, be the case to a widespread degree. In other fields, therapist competence, being able to problem solve and tailor treatment for individual cases, has been shown to outweigh application of specific therapies in predicting favorable outcomes. Only time will tell which is true of our discipline, or parts of it.
Evidence is only helpful to professionals and their clients if health service providers seek it out, understand it, and apply it. Our fellow professionals in other disciplines that first developed EBP or are now trying to integrate it have already confronted this potentially frustrating break in the chain that goes from research to practice. We need to inculcate the value of reading the updated professional, peer-reviewed literature and help equip today's students to understand that this is the first step in EBP: When you do not know the best way to treat your client, look it up, and look it up in the right place. If you thought you knew it last year, check and see if it is still true. A potentially difficult problem is that speech-language pathology, together with some other professions, such as education and psychology, has been to some extent swept up in a sea change that originated in a peer, but distinctly different, profession. Some of the tenets of EBP would appear to require thoughtful adaptation if we are to make maximum use of this important concept in moving forward the practice of speech-language pathology.
|
Footnotes |
|---|
Received August 17, 2005
Accepted March 9, 2006
|
REFERENCES |
|---|
|
TOPABSTRACTHOW DOES ONE DEFINE...HOW DOES ONE EVALUATE...WHAT IS THE ROLE...HOW DOES ONE DISTINGUISH...HOW DOES A FIELD...WHAT ARE THE NEXT...CONCLUSIONREFERENCES |
|---|
American Speech-Language-Hearing Association. (2005). Background information and standards and implementation for the certificate of clinical competence in speech-language pathology. Available at http://www.asha.org/members/deskrefjournals/deskref/default.
Apel, K., & Self, T. (Sept. 9, 2003). The marriage of research and clinical services. The ASHA Leader, 6–8.
Berliner, D. (2002). Educational research: The hardest science of all. Educational Researchers, 31, 18–20.
Bernstein Ratner, N. (2005). Evidence-based practice in stuttering: Some questions to consider. Journal of Fluency Disorders, 30, 163–188.[CrossRef][Medline]
Bernstein Ratner, N., & Guitar, B. (2006). Treatment of very early stuttering and parent-administered therapy: The state of the art. In N. Bernstein Ratner, & J. Tetnowski (Eds.), Current issues in stuttering research and practice(pp. 99–124) Mahwah, NJ: Erlbaum.
Beutler, L., Moleiro, C., Malik, M., & Harwood, M. (2003). A new twist on empirically supported treatments. International Journal of Clinical and Health Psychology, 3, 423–437.
Burgers, J., & van Everdingen, J. (2004). Beyond the evidence in clinical guidelines. The Lancet, 364, 392–393.
Burke, D., Judelson, A., Schneider, J., DeVito, M., & Latta, D. (2002). Reading habits of practicing physiatrists. American Journal of Physical and Medical Rehabilitation, 81, 779–787.
Cohen, A., Stavri, P., & Hersh, W. (2004). A categorization and analysis of the criticisms of evidence-based medicine. International Journal of Medical Informatics, 73, 35–43.[CrossRef][Medline]
Cullen, R. (2002). In search of evidence: Family practitioners' use of the Internet for clinical information. Journal of the Medical Library Association, 90, 370–379.[Medline]
Davis, D., Ciurea, I., Flanagan, T., & Perrier, L. (2004). Solving the information overload problem: A letter from Canada. Medical Journal of Australia, 180, Suppl. 6, S68–S71.[Medline]
Deane, K., Whurr, R., Playford, E., Ben-Schlomo, Y., & Clarke, C. (2005). Speech and language therapy for dysarthria in Parkinsons disease: A comparison of techniques. The Cochrane Library, 4.
Deegear, J., & Lawson, D. (2003). The utility of empirically supported treatments. Professional Psychology: Research and Practice, 34, 271–277.[CrossRef]
Dollaghan, C. (April 13, 2004). Evidence-based practice: Myths and realities. The ASHA Leader, 4–5, 12.
Ellis Weismer, S. (1991). Child language intervention: Research issues on the horizon. In J. Miller (Ed.), & Research on child language disorders(pp. 233–241) Austin, TX: Pro-Ed.
Ely, J., Osheroff, J., Ebell, M., Chambliss, M., Vinson, D., Stevermer, J., & Pifer, E. (2002). Obstacles to answering doctors' questions about patient care with evidence: A qualitative study. British Medical Journal, 23, 710–713.
Evidence-Based Intervention Work Group. (2005). Theories of change and adoption of innovations: The evolving evidence-based intervention and practice movement in school psychology. Psychology in the Schools, 42, 475–494.
Follette, W., & Beitz, K. (2003). Adding a more rigorous scientific agenda to the empirically supported treatment movement. Behavior Modification, 27, 369–387.[Abstract]
Gonzales, J., Ringeisen, H., & Chambers, D. (2002). The tangled and thorny path of science to practice: Tensions in interpreting and applying "evidence". Clinical Psychology: Science and Practice, 9, 204–209.
Grimshaw, J., Eccles, M., & Tetroe, J. (2004). Implementing clinical guidelines: Current evidence and future implications. The Journal of Continuing Education in the Health Professions, 24, S31–S37.[CrossRef][Medline]
Harrison, E., Onslow, M., & Menzies, R. (2004). Dismantling the Lidcombe program of early stuttering intervention. International Journal of Language & Communication Disorders, 39, 257–268.[Medline]
Herbert, J., Lilienfeld, S., Lohr, J., Montgomery, R., O'Donahue, W., & Rosen, G. (2000). Science and pseudoscience in the development of eye movement desensitization and reprocessing: Implications for clinical psychology. Clinical Psychology Review, 20, 945–971.[CrossRef][Medline]
Hess, D. (2004). What is evidence-based medicine and why should I care? Respiratory Care, 49, 730–741.[Medline]
Hoagwood, K. (2005). Family-based services in children's mental health: A research review and synthesis. Journal of Child Psychology and Psychiatry, 46, 690–713.[CrossRef][Medline]
Hughes, J. (2000). The essential role of theory in the science of treating children: Beyond empirically supported treatments. Journal of School Psychology, 38, 301–330.
Jette, D., Warren, R., & Wirtalla, C. (2005). The relation between therapy intensity and outcomes of rehabilitation in skilled nursing facilities. Archives of Physical Medicine and Rehabilitation, 86, 373–379.[CrossRef][Medline]
Johnson, C., Beitchman, J., & Young, A. (1999). Fourteen-year follow-up of children with and without speech/language impairments: Speech/language stability and outcomes. Journal of Speech, Language, and Hearing Research, 42, 744–760.
Johnston, J. (2005). Re: Law, Garrett and Nye (2004a). "The efficacy of treatment for children with developmental speech and language delay/disorder: A meta-analysis". Journal of Speech-Language and Hearing Research, 48, 1114–1120.
Jones, M., Onslow, M., Packman, A., Williams, S., Ormand, T., Schwarz, I., & Gebski, V. (2005). Randomised controlled trial of the Lidcombe programme of early stuttering intervention. British Medical Journal, 331, (7518) 659–661.
Justice, L. M., & Fey, M. E. (Sept. 21, 2004). Evidence-based practice in schools: Integrating craft and theory with science and data. The ASHA Leader, 4–5, 30–32.
Justice, L., & Pullen, P. (2003). Promising interventions for promoting emergent literacy skills: Three evidence-based approaches. Topics in Early Childhood Special Education, 23, 99–113.
Lambert, M. (2005). Early response in psychotherapy: Further evidence for the importance of common factors rather than "placebo effects". Journal of Clinical Psychology, 61, 855–869.[CrossRef][Medline]
Larner, G. (2004). Family therapy and the politics of evidence. Journal of Family Therapy, 26, 17–39.[CrossRef]
Law, J. (2004). The implications of different approaches to evaluating intervention: Evidence from the study of language delay/disorder. Folia phoniatrica et logopaedica, 56, 199–219.[CrossRef][Medline]
Law, J., Garrett, Z., & Nye, C. (2003). Speech and language therapy interventions for children with primary speech and language delay or disorder. Cochrane Database Systemic Reviews, (3):CD004110.
Lipsey, M., & Cordray, D. (2000). Evaluation methods for social intervention. Annual Review of Psychology, 51, 345–375.[CrossRef][Medline]
Lohr, J., DeMaio, C., & McGlynn, F. (2003). Specific and nonspecific factors in the experimental analysis of behavioral treatment efficacy. Behavior Modification, 27, 322–368.[Abstract]
Lubinski, R. (2003). Revisiting the professional doctorate in medical speech-language pathology. Journal of Medical Speech-Language Pathology, 11, (4) 14–18.
Lund, N., & Duchan, J. (1993). Assessing children's language in naturalistic contexts(3rd ed.) Englewood Cliffs, NJ: Prentice Hall.
Maher, C., Sherrington, C., Elkins, M., Herbert, R., & Moseley, A. (2004). Challenges for evidence-based physical therapy: Accessing and interpreting high-quality evidence on therapy. Physical Therapy, 84, 644–654.
Malterud, K. (2001). The art and science of clinical knowledge: Evidence beyond measures and numbers. The Lancet, 358, 397–400.
McAlister, F., Finlay, A., Graham, I., Karr, G., & Laupacis, A. (1999). Evidence-based medicine and the practicing clinician. Journal of General Internal Medicine, 14, 236–242.[CrossRef][Medline]
McNally, R. (1999). On eye movements and animal magnetism: A reply to Greenwald's defense of EMDR. Journal of Anxiety Disorders, 13, 617–620.[CrossRef]
Stroke Meinzer, M., Djundja, D., Barthel, G., Elbert, T., & Rockstroh, B. (2005). Long-term stability of improved language functions in chronic aphasia after constraint-induced aphasia therapy [Epub ahead of print]..
Messer, S. (2004). Evidence-based practice: Beyond empirically supported treatments. Professional Psychology: Research and Practice, 35, 580–588.[CrossRef]
Michie, S. (2005). Changing behavior: Theoretical development needs protocol adherence. Health Psychology, 24, 439.[CrossRef][Medline]
Mullen, R. (2005). Survey tests members' understanding of evidence-based practice. The ASHA Leader, 10, (15) 4–14.
Mykhalovskiy, E., & Weir, L. (2004). The problem of evidence-based medicine: Directions for social science. Social Science and Medicine, 59, 1059–1069.[CrossRef]
Nail-Chiwetalu, B., & Bernstein Ratner, N. (2006). Information literacy for speech-language pathologists: A key to evidence-based practice. Language, Speech, and Hearing Services in Schools, 37, 157–167.
Nail-Chiwetalu, B., & Bernstein Ratner, N. (in press). An assessment of the information-seeking abilities and needs of practicing speech-language
