Validity of a Cochrane Systematic Review and meta-analysis for determining the safety of vitamin E

Abstract

The public safety of α-tocopherol has been called in question by several meta-analyses which have raised concern among regulatory authorities. The objective of this study was to evaluate the Cochrane Database Systematic Review 2012 (CD007176) which concludes that α-tocopherol forms of vitamin E have a statistically significant effect on mortality, by assessing the trials and datasets used and determining their effect upon the primary outcome.

The Alpha-Tocopherol, Beta-Carotene Cancer Prevention Study (ATBC), a factorial design study of Finnish smokers was a pivotal paper in the Cochrane Review owing to the high mortality rate observed which resulted in a substantial weighting (42.6%) in the meta-analysis. The Cochrane meta-analysis used a 3 cell analytical method comparing all vitamin E cells (vitamin E alone plus vitamin E + β-carotene) to the placebo only cell. This had the unfortunate effect of incorrectly inflating the mortality risk attributed to vitamin E by not balancing the contribution to mortality of the β-carotene intervention. Re-analysis of the ATBC trial using data derived from the more generally accepted ‘inside the table’ (2 cell – vitamin E versus placebo) or ‘at the margins’ (4 cell – all vitamin E versus all non-vitamin E) analytical methods demonstrates a statistically non-significant result.

The data from the ATBC study has been given in 5 datasets (the trial alone and four extended post-trial follow-up time periods). Using the 3 cell analysis method only the 6 and 8-year (used in the meta-analysis) follow-up periods were statistically significant. Using the 2 or 4 cell method the outcome remains non-significant over all time periods.

The impartiality of excluding trials with zero mortality is also examined and questioned.

This paper raises concerns overall as to the appropriateness of datasets chosen, the validity of methods and generalisability of results when using meta-analysis as a tool to determine safety. Issues raised in this paper are not unique to the Cochrane study in question. Until we have new tools, there may be a need to rely on conventional narrative systematic literature synthesis in the assessment of safety or contain our results to specific sub-populations where more conclusive results can be determined.

Background

Over the last decade there have been several meta-analyses examining the effect of vitamin E on all-cause mortality [1,2,3,4] which have drawn the conclusion of a positive association between vitamin E supplement use and mortality. It is important to recognise that vitamin E in the context of these meta-analyses refers only to alpha-tocopherol which is only one of the eight isoforms of the vitamin (alpha, beta, gamma and delta tocopherols; and the alpha, beta, gamma, and delta tocotrienols). To remain consistent to the terminology used in the Cochrane review which is the focus of this paper we continue to use the term ‘vitamin E’ in the acknowledgement that our use pertains only to the synthetic and natural forms of alpha-tocopherol used in the various trials considered. While there has been some criticism of these meta-analyses [5, 6] they have not only received considerable media attention but also frequent citation in scientific papers; the papers by Bjelakovic et al. [1,2,3] have been cited over seventeen hundred times (Scopus 31st December 2016).

We set out to evaluate the validity of the 2012 systematic review on vitamin E published by the Cochrane Collaboration [3] by examining the included trials and the data used and the conclusions drawn. This study was chosen because the earlier 2008 meta-analysis on vitamin E published by the Cochrane Collaboration [2] was the main publication cited by the Australian drug regulator (Therapeutic Goods Administration) whom had requested industry consultation on its intention to change the regulatory status of vitamin E (2008). In addition, this meta-analysis provides the most transparent methods and comprehensive datasets due to the Cochrane Collaboration’s guidelines for publication.

Methods

Data source

The 2012 Cochrane meta-analysis [3] was retrieved from the Cochrane website and data from the table ‘Analysis 01.11. Comparison 01 Antioxidants versus placebo/no intervention, Outcome 11 Mortality in vitamin E trials with a low or high risk of bias’ (page 250) were used. For ease of description this data table will be abbreviated as A11. Trials were assigned to either low or high risk of bias by Bjelakovic et al. from an assessment of the methodology reported for each trial in the following six domains: allocation sequence generation, allocation concealment, blinding, complete outcome data reporting, selective outcome reporting, and other apparent biases. Data analysis in the results section of our paper pertains to the low risk of bias subgroup only and not the total analysis (which includes both high and low bias sub-groups) which replicates the primary findings in the Bjelakovic et al. paper.

ATBC study

Examination of A11 showed that one paper, the ATBC 2003Low (Alpha-Tocopherol, Beta-Carotene Cancer Prevention Study, dataset from 2003 publication, low bias study) was given a weighting of 42.6% in the analysis. The next closest study (HPS 2002) has a weighting of 12.7%. Given the important contribution of the ATBC study to the meta-analysis outcome pertinent publications concerning the ATBC trial were retrieved [7,8,9].

Meta-analyses

Meta-analysis calculations were carried out using the Review Manager (RevMan) software (Version 5.2. Copenhagen: The Nordic Cochrane Centre, The Cochrane Collaboration, 2012). In any analysis the risk ratio and 95% confidence intervals were calculated using the Mantel-Haenszel method for dichotomous data in a random effects model to replicate the methods used in the Bjelakovic meta-analysis [3].

Data analysis

To commence, the data from A11 was copied into RevMan and checked against the original table for accuracy.

To determine the contribution of the ATBC 2003 study a sensitivity analysis was undertaken by removing this study from the A11 low risk of bias subgroup and re-running the analysis.

To determine the sensitivity of the meta-analysis to trial data that explicitly would show a high level of safety, a large fictitious trial was added to A11 in the low risk of bias sub-group (with ATBC 2003 study included) and the analysis re-run. We postulated a dummy trial with a vitamin E dosage of 500 IU daily over a period of 10 years in a cohort with a low rate of underlying mortality (1%) in which no difference in mortality was observed between the vitamin E and the placebo groups. The reason chosen for these dummy variables was to simulate a trial that would indicate independently that vitamin E at a commonly available commercial dose (500 IU) taken daily over an extensive period (10 years) in large population was demonstrably safe. The cohort size chosen was equal to the total numbers in the vitamin E (n = 97,523) and placebo (n = 73,721) arms given in the low bias section of A11.

The ATBC study was conducted as a factorial study and data was available for five time periods - the study period, follow up at 3, 6, 8, and 18 years [7, 8]. The 8 year period was used in the meta-analysis published in 2012; the 18 year post-trial data was published in 2014 [9]. A new series of meta-analyses were run substituting the dataset from each of these time periods using both ‘inside the table’ and ‘at the margins’ data into the original A11 dataset. Simple factorial trials have four arms Active X and Active Y (A); placebo X and active Y (B); active X and placebo Y (C); placebo X and placebo Y (D) and the analysis is classically presented in a 2 × 2 table. In assessing the effect of a single substance the data can be analysed in two ways, 1) “inside the table” analysis looks at the effect of one substance alone versus placebo (A compared to D or B compared to D); and 2) “at the margins” analysis looks at the total use of specific substance versus the total non-use of that substance (A + B compared to C + D or A + C compared to B + D) [10]. See Table 1.

Table 1 “At the margins” (4 cell) and “inside the cell” (2 cell) factorial design analysis

Results

The result of the replicated A11 analysis is given in Table 2 and this became the starting basis for all further meta-analyses. These results are an exact replica of the Cochrane meta-analysis [3] and demonstrate a risk ratio of 1.03 (95%CI 1.00 to 1.05; p˂0.04) for vitamin E supplementation in the low risk of bias subgroup. The associated forest plot can be found in Additional file 1: Fig. S1.

Table 2 Sensitivity analyses of Risk Ratio (RR) low risk of bias data subset by a) removing ATBC study, and b) adding fictitious dataset to original low risk of bias trials

The sensitivity analysis with the ABTC study removed from A11 shows the risk ratio in the low risk of bias subgroup changing from statistically significant to becoming statistically non-significant with a risk ratio 1.01 (95%CI 0.98–1.05; p = 0.44). See Table 2. The associated forest plot can be found in Additional file 1: Fig. S2.

Further sensitivity analysis was done with the addition to A11 of a large fictitious low risk trial which doubles the overall population sample size in the low risk of bias subgroup. The fictitious trial gets a weighting of 6.1% but the overall risk ratio of mortality in the low risk of bias subgroup was unchanged – 1.03 (95%CI 1.00–1.05, p = 0.04). See Table 2. The associated forest plot can be found in Additional file 1: Fig. S3.

In the original A11 analysis the ATBC study uses the 8 year post-trial follow-up dataset. The meta-analysis was rerun in the low risk of bias subgroup using 5 time periods for the ATBC study (the study period itself and 4 post-trial follow up periods of 3,6, 8 and 18 years) utilising both ‘inside the table’ (2 cell) and ‘at the margins’ (4 cell) analysis. None of these analyses were of statistical significance. The data used in these analyses and the resultant risk ratios are given in Table 3. All the forest plots for this series are given in Additional file 1: Figs. S4-S13.

Table 3 Risk ratio (RR) according to ATBC study period analysed using three methods - ‘inside the table’ (2 cell), ‘at the margins’ (4 cell) and Bjelakovic (3 cell) - utilising the Bjelakovic low risk of bias dataset (A11)

In constructing the ‘inside the table’ and ‘at the margins’ datasets it became apparent that the data used for the ATBC study in A11 was neither ‘inside the table’ (2 cell) nor ‘at the margins’ (4 cell) but was a 3 cell analysis (vitamin E alone and vitamin E plus beta-carotene versus the placebo only alone arm). Analysis was therefore repeated for the other ATBC study periods using this 3 cell method in the low risk of bias subgroup. A statistically significant risk ratio occurred only for the 6 year (1985 to 1999) post-trial period in addition to the 8 year (1985 to 2001) post-trial period used in the Cochrane meta-analysis [3]. The datasets used in these analyses and the resultant risk ratios are given in Table 3. All the forest plots for this series are given in Additional file 1: Figs. S14-S17.

Discussion

Examination of the 2012 Cochrane review by Bjelakovic et al. raises several major issues which have broader implications for all similar meta-analyses; these are sensitivity, internal validity and external validity.

A cursory examination of A11 shows the overwhelming impact of the ATBC study on the meta-analysis outcome due to its high weighting (42.6%). The removal of the ATBC study from A11 renders the low bias subgroup analysis statistically non-significant. It could be argued that this is because of a diminution in study power; however, even the addition of a fictitious low-event trial doubling the baseline sample size had little impact on the overall risk ratio. The reason for this is that the weighting of a clinical trial’s contribution to a meta-analysis is based on the inverse of the variance observed and variance is determined by the number of events e.g. deaths. This means that studies with low numbers of events, i.e. low mortality rates, have little or no impact on the analysis as our fictitious study demonstrated. Yet these studies logically contribute to an understanding about the safety of the preparation under review. We hypothesised that a study the size of our fictitious study should provide evidence of safety at the dosage used for the specified duration in the study population. In the Cochrane meta-analysis [3] it was stated that 481 antioxidant trials (approximately 42,000 persons) were found that had zero mortality in both the experimental and control groups. These trials were excluded from the meta-analyses on the grounds that exploratory analyses by adding an imagined trial with 1 death and 21,000 participants in each intervention group had no impact on the outcome (this would be a mortality rate of 0.0047%). It was concluded that “the influence of zero events trials on the our final result was not noticeable” [3]. However, exclusion of low mortality data that implicitly illustrate safety is a major failing that risks substantial bias and undermines the conclusions of the current meta-analysis. Current meta-analysis models are designed to assess efficacy where zero outcome in a clinical trial generally equals a poor response, while in the assessment of safety a zero outcome is the preferred response. One commentator noted about the issue of weighting that “when dealing with efficacy, that is not a problem, as everyone has an outcome, and the question is what the level or the direction of the effect is, but it is [a problem] when you’re looking at safety: zero outcomes – no heart attacks for instance – is important information” [11]. While studies that have high underlying mortality rates may individually make important contributions to our understanding of the effect of an agent on mortality in a specific clinical situation at a specific dose for a specific duration; their inclusion into mathematical models along with studies that have no or low mortality gives them a high weighting potentially distorting their relative contribution to external validity. Unfortunately, current meta-analysis methods do not allow for studies with low (or no) mortality to be incorporated in a meaningful way.

The choice of datasets from any included studies is obviously important for internal validity. The ATBC study was a 2 × 2 factorial clinical trial of approximately 30,000 Finnish male smokers at high risk of cancer assigned to one of four arms: a low dose of synthetic vitamin E (50 mg), a moderate dose of synthetic β-carotene (20 mg), a combined arm of vitamin E and β-carotene or a placebo [7]. The most appropriate type of analysis would be to compare the vitamin E arm alone to the placebo alone arm i.e. ‘inside the table’ (2 cell) or by comparing all vitamin E to all non-vitamin E i.e. ‘at the margins’ (4 cells). The 3 cell method used in the Cochrane meta-analysis (1) in A11 compared both vitamin E arms (alone and with β-carotene) to placebo only. This analysis is confounded by the mortality associated with β-carotene and cannot be used to determine mortality associated with vitamin E alone. Utilising data from the 2 cell ‘in-table’ or 4 cell ‘at the margins’ is the only correct method for determining mortality associated with alpha-tocopherol and doing so in this case renders the risk ratio statistically non-significant.

Another important question relating to data selection is the use of post-trial follow-up data. The use of extended post-trial data can have a significant impact on the weighting of a study to the overall meta-analysis because the longer the study the more deaths and the greater the weighting, even if there were no changes in the risk ratio between the active and placebo group. The weighting for the ATBC study using the trial period only (1985–1993: ‘at the margins’) was 21.5% while use of the ATBC study 18 year post-trial data (1985–2011: ‘at the margins’) increased the weighting to 81.7% (see Table 3). Of note is that in A11, 28 of the 46 included studies in the low bias group and 13 of the 18 included studies in the high bias group have a weighting of less than 0.1%.

This example of disproportionate weighting raises the important aspect of evaluating the external validity of a meta-analysis as one would with a clinical trial. The question of external validity should be a critical concern and is central to the intent of any meta-analysis. An investigator needs to determine if the available datasets have the clinical and methodological homogeneity required for a pooled estimate. In this case the vitamin E studies used in the Cochrane 2012 meta-analysis have considerable clinical and study design heterogeneity. There are pooled studies on: 1) healthy individuals and individuals with chronic diseases (e.g. cancer, Alzheimer’s disease, Parkinson’s disease, renal disease); 2) different gender mixes (males only, females only, mixed); 3) smokers only, non-smokers or mixed (where there is likely to be differences in vitamin E pharmacokinetics [12, 13]; 4) different forms of vitamin E (natural and synthetic – known to have different bioavailability and pharmacokinetics [14]; 5) different doses from as little as 33 IU daily to 5000 IU daily; 6) different intervention time periods ranging from 6 weeks to 5 years; and 7) the use of vitamin E alone or vitamin E together with different concomitant nutrients and/or pharmaceutical drugs. The 2012 Bjelakovic meta-analysis concluded, incorrectly in our opinion, that there was an increased risk of mortality with vitamin E supplementation – i.e. 1.03 (95%CI 1.00–1.05, p = 0.04). A simple question would be, to whom would this increase in risk apply to? With beta-carotene, data from the ATBC [8] and CARET [15] trials made it clear that male smokers and persons exposed to asbestos were at higher risk. In the case of vitamin E with the existing meta-analysis it would be difficult to define a specific at-risk group given the considerable heterogeneity of the pooled studies and the disproportionate weighting of a trial of male smokers. Unfortunately, the conclusions reached of increased mortality for vitamin E supplementation in the Cochrane 2012 meta-analysis lack any of the precision required in toxicology and leave open the interpretation that any form of vitamin E, taken at any dose, for any duration, by anybody, at any age is potentially fatal.

Bailar notes that a rigorous technical review of a meta-analysis requires the reviewer to identify, re-abstract, and interpret a fair sample of the original papers [16]. He considers very few editors and reviewers will do this and suggests that this may be one reason why there are so many poor quality meta-analyses in the literature. We suggest that journals require authors of factorial trials to publish results of all treatment arms separately so that both ‘inside the table’ and ‘at the margins’ analysis be performed if deemed necessary. In addition, it is not readily evident from reading the methodological sections of some papers that they are part of a broader factorial study as some factorial trials are very complex in their design. In some published meta-analyses it is unclear what datasets are actually used because the number of events and totals are not specified on the forest plots. Perhaps there needs to be an adjustment to the PRISMA guidelines for meta-analysis, not just on the inclusion and exclusion of the trials, but inclusion and exclusion of the trial datasets per se used in the meta-analysis and mandatory reporting of the actual event and non-event datasets in forest plots or tables.

Conclusions

We believe there are several methodological issues in the meta-analysis of vitamin E safety in the 2012 Cochrane Review that negate the original findings. These include the technique used to analyse data in a key factorial paper, disproportionate weighting in the same trial with use of extended post-trial data, the inability of meta-analysis to incorporate low or no mortality datasets and overall poor external validity given the disparate nature of the trials included. We would strongly argue that new models need to be developed that appropriately weight studies with low to no mortality outcomes as failure to account for these datasets obscures potentially important data. Until then we may need to rely on conventional narrative systematic literature synthesis in the assessment of safety or contain our results to specific sub-populations where more conclusive results can be determined.

Currently it is not always possible for readers to query the selection of a study clinical trial dataset used within a meta-analysis and we call on changes to be made to the PRISMA guidelines to ensure appropriate data transparency in any meta-analysis.

Vitamin E supplementation for prevention of mortality in adults

• Goran Bjelakovic,
• Dimitrinka Nikolova &
• Christian Gluud 

1. The Cochrane Hepato-Biliary Group, Copenhagen Trial Unit, Centre for Clinical Intervention Research, Department 7812, Rigshospitalet, Copenhagen University Hospital, Copenhagen, Denmark

   Goran Bjelakovic, Dimitrinka Nikolova & Christian Gluud

2. Department of Internal Medicine - Gastroenterology and Hepatology, Medical Faculty, University of Nis, Nis, Serbia

   Goran Bjelakovic

We read with interest the article by Oliver and Myers questioning the validity of the results of vitamin E on all-cause mortality in our systematic review [2, 17]. The authors reanalysed some of the meta-analyses in our systematic review, using several scenarios which led them to questioning our findings of increased mortality in people supplemented with vitamin E [2, 17].

For all analyses in our review, we extracted data using the same methods [2]. In the trials with parallel group design with more than two intervention groups and additional therapy, we compared the groups with only antioxidant intervention versus placebo or no intervention [2]. For trials with factorial design, we compared all groups that received antioxidant supplements versus placebo or no intervention group [2]. Similarly, in factorial trials that tested vitamin E, we compared all groups supplemented with vitamin E versus placebo or no intervention group. Among the 46 trials with a low risk of bias included in our Analysis 1.11, ‘Mortality in vitamin E trials with low risk of bias’ there are 8 factorial trials (6 trials with 2 by 2 factorial design, and 2 trials with 2 by 2 by 2 factorial design). Some of these factorial trials tested vitamin E singly or combined with beta-carotene (found to have a potential detrimental effect on mortality), while the remaining factorial trials tested vitamin E singly or combined with selenium (found to have a potential beneficial effect on mortality), or vitamin E singly or combined with vitamin C (found to have a neutral effect on mortality) [2] In some of the included parallel group trials, vitamin E was tested combined with multivitamins or multiminerals. The most commonly used dietary supplements in adults are multivitamins or multiminerals [18]. Therefore, there is a possibility of synergistic or antagonistic interactions between tested antioxidants. We believe we used the best available approach to analyse beneficial and harmful effects of vitamin E. Furthermore, we assessed whether the different doses of beta-carotene, vitamin A, and vitamin E affect mortality in primary and secondary prevention randomised clinical trials at low risk of bias [19]. We found that beta-carotene and vitamin E in doses higher than the RDA seem to significantly increase mortality [19].

Our results are in accordance with the results of Miller and colleagues who used ‘at margins’ or ‘2-cell’ analysis, who also found increased mortality in people supplemented with high doses of vitamin E [4].

In future updates of our review, we shall consider also to employ the ‘at margins’ methodology suggested by Oliver and Myers. At the same time, we notice that in none of the scenario analyses conducted by Oliver and Myers, vitamin E showed any significant benefit [17]. Moreover, as our own analysis 1.16 show antioxidants without potential influence from confounders in factorial trials versus placebo or no intervention causes a significant increase in all-cause mortality of 10% (relative risk 1.10; 95% CI 1.05 to 1.15; P = 0.0002; I² = 0%).

As we mentioned in our Cochrane review, we performed our review according to the recommendations of the Cochrane Handbook for Systematic Reviews of Interventions [20]. Accordingly, we assessed outcome data at the end of the trial follow-up period for each included trial, making it uniform for the whole review. Based on our meta-regression analyses stratified by the intervention regimen, the duration of follow-up was not a predictor of the estimated intervention effect [2].

We have extensively discussed the issue of different forms of vitamin E used for supplementation. We could not include trials with tocotrienols, gamma tocopherol, or any other form of vitamin E because such randomised clinical trials were not published at that time. The majority of the trials conducted tested alpha-tocopherol.

During recent years, several major randomised clinical trials on vitamins and supplements have produced disappointing results. One recent example is the Selenium and Vitamin E Cancer Prevention Trial (SELECT), originally scheduled to end in 2011 [21]. SELECT was terminated in October, 2008 over a disproportionally high incidence of prostate cancer in trial participants supplemented with vitamin E [21]. The chance that vitamin E may provide benefit seems low [22,23,24].

We have dealt with trials with zero mortality extensively in our review. Trials without deaths were excluded. First, they provide very little information. Second, most of these trials were not really ‘preventive trials’, but rather explanatory trials assessing the impact of antioxidant supplements on potential surrogate outcomes. Third, we assessed the influence of trials with zero events in the treatment or control group by re-calculating the random-effects meta-analyses with 0.5, 0.05, and 0.005 continuity corrections. Here our results were robust. Fourth, our results were also robust to exploratory analyses adding an imagined trial with 20 000 participants and one death in each intervention group. Accordingly, the increased mortality does not seem to be an artifact created by exclusion of trials with zero events in both trial groups.

We are aware that pharmaceutical companies are influenced by our findings. Antioxidants together with other food supplements have turned into multi-billion $ markets without much evidence supporting their use. ‘Vitamins, Inc.’ was the most elaborate, longest lasting, and most harmful of all the international cartels ever discovered [25]. Pharmaceutical companies have been cheating us into buying vitamin supplements at too high prices [25]. If harmful effects of antioxidant supplements stand to the test of science and time, then we may end up realising that we have also been fooled into believing ‘the more antioxidant supplements, the merrier’.