Evidence from Cochrane Systematic Reviews in Anesthesiology is Insufficient to Support or Reject the Studied Interventions and Highlights the Need of Further Research

Systematic reviews are one of the best methods for mapping, gathering and producing scientific evidence, because they assess several data from different studies simultaneously, based on reproducible and consistent methodology. According to the number and type of included studies in systematic reviews, meta-analyses can be plotted, representing the combined statistical data from the trials that have met the inclusion criteria.

The Cochrane Collaboration has the largest database of systematic reviews and it aims to produce and report systematic reviews with the highest standard of quality [1]. In addition to its methodological accuracy [2], the inferences from its conclusions are translated into implications for clinical practice and research. There are 53 review groups (Cochrane Review Groups) in several medical fields, thus constituting the database. Established in 2000, the Cochrane Anaesthesia Review Group, currently named Cochrane Anaesthesia, Critical and Emergency (ACE) Care Group, publishes systematic reviews in the areas of anesthesiology, critical care, perioperative  medicine and emergency medicine, and it has the largest number of the reviews in these areas of all the existing databases and journals [3].

In addition to these several advantages, systematic reviews are often criticized due to the production of data with no statistical significance, the inconsistency of evidence and the absence of solid recommendations for interventions in clinical practice. Only a small proportion of these systematic reviews have been capable of solidly recommending or discouraging specific interventions with no need for further studies, according to their authors’ conclusions [4]. Thus, there is an urgent need to perform primary studies, i.e., randomized controlled trials, in larger numbers and with better quality [5]. In anesthesiology, there is also the necessity of comprehending the results of systematic reviews so that other primary studies and systematic reviews can be performed in different subfields of knowledge. The absence of such comprehensiveness is responsible for the excess of studies in some fields and the lack of studies in others. Moreover, many issues have not been adequately explored by randomized controlled trials, which very often have high levels of heterogeneity, making it difficult to plot them in meta-analyses and to obtain clearer and more precise conclusions.

Previous publications that mapped all the groups of Cochrane database showed that their systematic reviews were invariably inconclusive, concerning the applicability of their results in clinical practice [4,5]. For this reason, a question arises regarding the uncertainties of systematic reviews in anesthesiology, relative to the applicability of their results in clinical practice and research.

In the present study, we proposed the mapping of systematic reviews published in the Cochrane Anaesthesia, Critical and Emergency Care Group and, according to their author’s conclusions, classify the evidence as sufficient or not to support or reject the interventions studied. The recommendation or not for further research, based on each author’s conclusion and the general characteristics of the reviews relative to the included studies and meta-analyses were also described. So far, this is the first study made under such criteria in the anesthesia field.

All systematic reviews of the Cochrane Anaesthesia, Critical and Emergency Care Group published until February 28, 2014 were included. All of the information was extracted from the reviews according to their authors’ conclusions. The information concerned the existence or not of benefits from a given studied intervention, as well as the need for further research, according to the evidence to recommend or not the intervention for clinical practice.

This methodology was similar to that of previously published studies that mapped and qualified the evidence from systematic reviews [4,5]. Only complete systematic reviews of randomized controlled trials were included. Protocols and systematic reviews that did not directly involve randomized controlled trials were excluded. Systematic reviews that recommended or did not recommend further research based on cost-benefit analyses were not considered, unless their primary endpoint was the cost of a treatment or intervention.

We used an excel spreadsheet where we indicated the name of the study and one of the three possible categories: “evidence supporting the interventions”; “evidence against the interventions” and “insufficient evidence to support or reject the interventions”. The studies were then classified in two subgroups in each category: “recommendation for further studies” and “no recommendation for further studies”.

Two investigators working together (RED and RSSJ) identified and extracted all the complete systematic reviews of randomized controlled trials to be included in the study from the Cochrane Anaesthesia, Critical and Emergency Care Group via Cochrane Library database and printed two copies of the full text for further independent analysis. Two investigators independently analyzed and classified each review (RSSJ, PNJ). In cases of disagreement or doubt, a third investigator (RED) would be consulted for consensus.

The classification of the systematic reviews is based on the authors’ conclusions according to their overall analysis of the results, regardless of what they considered, i.e., one primary outcome or several secondary outcomes. Then, the systematic reviews were allocated into six possible categories.

When the authors concluded that the evidence supported the intervention and recommended further studies, we defined the category #1, using the word “likely”:

(1) Evidence likely supporting the interventions and the authors recommending further research.

When the authors concluded that the evidence supported the intervention with no recommendation for further research, we defined the category #2:

(2) Evidence supporting the interventions and the authors not recommending further research.

When the authors concluded that the evidence favored control group and recommended further research, we defined category #3 using the word “likely”:

(3) Evidence likely against the interventions and the authors recommending further research.

When the evidence favored control group and the authors did not recommend further research, we defined category #4:

(4) Evidence against the interventions and the authors not recommending further research.

Insufficient evidence to draw a conclusion and the authors recommending further research was category #5:

(5) Insufficient evidence to support or to reject the interventions and the authors recommending further research b And when there was insufficient evidence to draw a conclusion and the authors not recommending further research, we defined category #6:

(6) Insufficient evidence to support or to reject the interventions and the authors not recommending further research.

Evidence supporting the intervention was defined as effectiveness (the extent to which a specific intervention, when used under ordinary circumstances, does what it is intended to do) or efficacy (the extent to which an intervention produces a beneficial result under ideal conditions) as compared to the control group, whether placebo, standard care or another intervention [6]. On the other hand, evidence against the intervention was defined when effectiveness or efficacy was seen on the control group instead of the intervention group.

The numbers of clinical trials and meta-analyses in each systematic review were also analyzed. Meta-analyses plotted with just one study were not computed.

The implications for clinical practice and research are presented as absolute numbers, percentages and 95% confidence intervals (CI) of all of the evaluated systematic reviews of the Cochrane Anaesthesia, Critical and Emergency Care Group. Meta-analyses and the included studies in each review are expressed as medians, minimum numbers, first and third quartiles, maximum numbers and modes.

One hundred eighteen systematic reviews were identified. Two reviews were excluded from the study due to duplication and one due to being a review summary. After the exclusions, 115 reviews were analyzed (Figure 1). The median number of randomized controlled trials per systematic review was 9.5, for a total of 2258 studies in all of the reviews. The median number of meta-analyses per review was six, for a total of 1072 meta-analyses in all of the reviews (Table 1).

Figure 1: Flow diagram of the study.

Table 1: Statistical data of the systematic reviews.

There was 100% inter-rate agreement between investigators on the classification of the systematic reviews. The commonest event observed in this study, occurring in 57.4% of the reviews, was insufficient evidence to support or reject the interventions with the authors recommending more studies. From the total, in only five reviews (4.3%), the authors did not recommend more studies, considering the actual evidence sufficient to recommend or discourage the intervention. Therefore, 95.7% of the reviews recommended the implementation of additional research, independent of the results obtained from a given intervention, i.e., benefit, harm or absence of evidence to support or discourage the intervention. Data on implications for clinical practice and research are expressed in Table 2.

Table 2: Outcomes of the systematic reviews related to clinical practice and research.

This study demonstrated a lack of solid evidence produced by the majority of the systematic reviews of the Cochrane Anaesthesia, Critical and Emergency Care Group. According to their authors, 110 out of 115 reviews were not considered sufficiently powerful to answer questions regarding clinical practice convincingly.

Evidence-based medicine represents the bond between good clinical research and clinical practice. Therefore, it should offer doctors and health policy-makers the best available evidence, which should be adequate, consistent and straightforward [7]. High-quality methodological research, with internal and external validation, is an essential condition for achieving these goals.

At the top level of the evidence hierarchy are systematic reviews, which are secondary studies built from primary studies. They are considered the best method for documenting, mapping and producing scientific evidence [2], while they play a major role in supporting fields in need of more research. The best primary evidence consists of randomized controlled trials, but depending on the condition to be studied, cohort studies could represent the best evidence. A systematic review of randomized controlled trials can conclude whether a tested intervention is effective, ineffective or harmful or whether there is insufficient evidence for any conclusion. Unfortunately, the inconsistency of results and absence of sufficient evidence to answer a clinical question are the most frequent outcomes found, which has been a subject of criticism and controversy [3].

The mapping of systematic reviews of the Cochrane Anaesthesia, Critical and Emergency Care Group resulted, in general, in absence of evidence to recommend or not a given intervention. Furthermore, their authors suggested the production of more research, similar to what was recently observed when all of the databases were mapped [5]. The same situation was shown in 2007 [4] and there have not been changes so far, demonstrating the need to focus on primary studies with better quality. The recommendation to conduct further randomized controlled trials occurred in 96.9% [4] and 95.4% [5] of the cases, respectively.

The lack of evidence is a consequence of several factors. In general, it is due to the poor methodological quality of the primary studies, specifically no clear randomization [8], single-center research with small sample sizes, no concealment of patients’ distribution into groups, lack of blinding of results for outcome evaluators [9], bias due to pharmaceutical company financing [10] and different types of heterogeneity.

Publication bias can also erroneously affect the clinical care of patients and future research. De Oliveira et al. [11] analyzed 1163 papers in anesthesiology, demonstrating that a positive result, or a result favorable to an intervention, was an independent predictive factor for publication in higher-impact journals. Moreover, a study with negative or unfavorable results took considerably more time to be accepted for publication [12]. In the present study, there were a higher number of systematic reviews with positive or favorable results in comparison to the negative outcomes, similarly to the fact previously described. A recently published meta-epidemiological analysis [13] that included 93 meta-analyses from 735 randomized controlled trials, published in journals from different specialties, showed significant variation in the sample sizes of the studies in their meta-analyses, with the smaller studies having a higher probability of overestimation of the positive effects of the tested treatments. It is known that research with smaller sample sizes is more susceptible to publication bias [14], whereas results from clinical studies with larger sample sizes tend to be more representative, valuable and publishable.

In our investigation, although it was not an objective, we observed important methodological heterogeneity among the various studies included in the reviews, creating more biases and placing the quality and value of the results at risk. Moreover, the majority of the systematic reviews had no meta-analyses (mode=0) and there was great diversity in the numbers of included studies (0 to 737). One of the most important findings was the lack of randomized controlled trials in systematic reviews (mode=4), which curiously were supposed to be systematic reviews from randomized controlled trials. Due to the heterogeneity of their primary studies, many systematic reviews do not present statistical data or meta-analyses. In the present investigation we observed a high number of systematic reviews with no meta-analyses (mode=0). This fact is probably related to both, the absence or the low number of randomized controlled trials included in the systematic reviews. Heterogeneity of included studies may have contributed to this observation.

One of the major principles of evidence-based medicine is the appraisal and classification of evidence. However, such criteria have been questioned by some investigators, who have suggested remodeling the concepts of power and quality in studies, as well as the resultant evidence [15]. They have suggested that additional dimensions not traditionally considered for classifying evidence should be incorporated, assigning weight to each dimension when determining the overall quality of the evidence. From the results we obtained, we believe that such changes have become necessary, not only in anesthesiology but also in all fields.

There were some limitations of the present investigation. The obtained results were based on the conclusions of the authors of each systematic review. We did not interpret the reviews, nor did we question the authors’ interpretations. Possible biases were not analyzed. It is possible that scrutiny of each review would have raised different conclusions from those presented by their authors. Even so, we believe that little would have changed in the results, since most of systematic reviews showed no evidence to support or to reject the intervention. We did not include other databases either, which might have contained different information. Nonetheless, the Cochrane Collaboration has an exclusive profile, gathering most of the published systematic reviews and following a pattern for their conclusions and outlining the efficacy and/or effectiveness of a treatment, according to the primary outcome and the control group. It also suggests the need, or not, for more studies to support or refute a given intervention, thereby enabling responses to clinical questions.

We conclude that a significant number of the systematic reviews of the Cochrane Anaesthesia, Critical and Emergency Care Group lack solid and consistent evidence for decision-making in clinical practice. On the other hand, around 50% of the systematic reviews offer suggestions that could influence clinical practice and this number may be considered reasonable. Nonetheless, the uncertainties observed in the reviews’ conclusions and the lack of included studies and meta-analyses reaffirm the need for more high-quality randomized controlled trials, which could improve the conclusiveness of the systematic reviews. Knowledge translations studies are necessary to verify the real impact of these systematic reviews on clinical practice.

The majority of the systematic reviews of the Cochrane Anaesthesia, Critical and Emergency Care Group resulted in no evidence or insufficient evidence to strongly recommend or discourage specific interventions for clinical practice and their authors did recommend further randomized controlled trials to provide clear evidence in future updates of systematic reviews.

Non-financial competing interests.

Regarding conflict of interests, we inform that one of the authors (RED) is a member of the Cochrane Diagnostic Test Accuracy Working Group and the Cochrane Prognosis Methods Group and the manuscript reflects our opinion, not Cochrane Collaboration’s opinion.