Open Access
Open Peer Review

This article has Open Peer Review reports available.

How does Open Peer Review work?

An evaluation of the completeness of safety reporting in reports of complementary and alternative medicine trials

  • Lucy-Ann Turner1Email author,
  • Kavita Singh1,
  • Chantelle Garritty1,
  • Alexander Tsertsvadze1,
  • Eric Manheimer2,
  • L Susan Wieland2,
  • James Galipeau1 and
  • David Moher1
BMC Complementary and Alternative MedicineThe official journal of the International Society for Complementary Medicine Research (ISCMR)201111:67

https://doi.org/10.1186/1472-6882-11-67

Received: 5 May 2011

Accepted: 22 August 2011

Published: 22 August 2011

Abstract

Background

Adequate reporting of safety in publications of randomized controlled trials (RCTs) is a pre-requisite for accurate and comprehensive profile evaluation of conventional as well as complementary and alternative medicine (CAM) treatments. Clear and concise information on the definition, frequency, and severity of adverse events (AEs) is necessary for assessing the benefit-harm ratio of any intervention. The objectives of this study are to assess the quality of safety reporting in CAM RCTs; to explore the influence of different trial characteristics on the quality of safety reporting.

Methods

Survey of safety reporting in RCTs published in 2009 across 15 widely used CAM interventions identified from the Cochrane Collaboration's CAM Field specialized register of trials. Primary outcome measures, the adequacy of reporting of AEs; was defined and categorized according to the CONSORT for harms extension; the percentage of words devoted to the reporting of safety in the entire report and in the results section.

Results

Two-hundred and five trials were included in the review. Of these, 15% (31/205) reported that no harms were observed during the trial period. Of the remaining 174 trials reporting any safety information, only 21% (36/174) had adequate safety reporting.

For all trials, the median percentage of words devoted to the reporting of safety in the results section was 2.6. Moreover, 69% (n = 141) of all trials devoted a lesser or equal percentage of words to safety compared to author affiliations. Of the predictor variables used in regression analysis, multicenter trials had more words devoted to safety in the results section than single centre trials (P = 0.045).

Conclusions

An evaluation of safety reporting in the reports of CAM RCTs across 15 different CAM interventions demonstrated that the reporting of harms was largely inadequate. The quality of reporting safety information in primary reports of CAM randomized trials requires improvement.

Background

Adequate reporting of safety in primary publications of randomized controlled trials (RCTs) is a pre-requisite for accurate and comprehensive profile evaluation of conventional as well as complementary and alternative medicine (CAM) interventions. Clear and concise information on the frequency and severity of Adverse Events (AEs) is necessary in assessing the benefit-harm ratio when administering any intervention. In 2007, it was estimated that 38.3% of American adults use some form of CAM [1]. With such high usage rates, it is essential that the evidence is of the highest quality and that safety information is reported in sufficient detail.

In 1998 and 2001, Ioannidis and Lau published a study evaluating the reporting of safety data in 7 medical areas [2, 3]. This evaluation concluded that the reporting of safety data was largely inadequate. Moreover, the absolute space given to author affiliations was often more than that given to the reporting of safety. Ioannidis called for the replication of such an evaluation across many fields.

While there have been recent advances in conventional medicine to standardize the collection, analysis and reporting of efficacy data in clinical trials [4], and more recently, the reporting of safety [5], there is little evidence regarding adequacy of reporting within CAM trials. Referencing Ioannidis' objectives and methods, this study evaluates safety information across 15 CAM interventions. Our aim was to assess the adequacy of safety reporting in CAM trials, to gain a better understanding of the predictors of adequate reporting, and to extend and assess adherence of CAM RCTs to the CONSORT recommendations for harms extension [6].

Methods

Trial Database

We searched the Cochrane Complementary Medicine Field (CAM Field) Specialized Register of trials [7] and obtained citations to all CAM RCTs published in 2009 pertaining to 15 CAM intervention categories. The inclusion of study reports was not restricted by study sample size or language of publication. All non-full text reports were excluded along with reports which could be categorized by more than one intervention; had AEs as a primary outcome; or for which full text articles were not locally available.

Selecting Interventions

The top 5 most commonly used CAM therapies according to the National Health Interview Survey 2007 were the following: 1. Natural products (17.7%); 2. deep breathing (12.7%); 3. meditation (9.4%); 4. Chiropractic and osteopathic (8.6%); and 5. massage (8.3%)[8]. Because natural products are by far the most commonly used CAM therapy, we focused on that CAM topic area, however only included the top 10 most commonly used natural products in our sample [9]. There is wide acceptance of minimal, if any, AEs associated with deep breathing and meditation; hence these topics were excluded from our study. In addition, we decided to include acupuncture and Chinese herbal medicine CAM topic areas despite not being among the top 10 most commonly used CAM therapies, as both topics are of popular interest, widely used globally [10, 11] and there is much available literature [12].

As such, 15 CAM interventions classified according to categories developed by the Cochrane CAM field, were chosen based on their respective impact, accessibility and frequency of use [13]. Interventions include: acupuncture, massage, chiropractry, traditional Chinese herbal medicine and other traditional medicine, fish oil/omega 3, glucosamine, flaxseed, ginseng, combination herbal products, ginkgo biloba, garlic supplements, coenzyme Q-10, echinacea and chondroitin. Acupuncture was defined as a needling intervention, breaking the skin [14], as such Moxibustion, TENS (transcutaneous electrical nerve stimulation), laser acupuncture and acupressure trials were not included.

Study selection and data extraction

Retrieved trials were screened and data extracted using online review software, DistillerSR. Title and abstract screening was conducted in duplicate by two of three review authors and subsequent full text articles were retrieved and screened independently by two of five reviewers, any disagreements were discussed and remaining conflicts were resolved by a third independent author.

Standardized data extraction forms were created to ensure all reported data was collected from the trials meeting inclusion criteria. All data to be collected was determined a priori. Data extraction was completed independently and verified by two authors by taking a 10% random sample of trials. A discrepancy of 10% in the number of words extracted was defined to be acceptable, however only two trials exceeded this cut off. Only 6.5% of all extracted data was modified. There were no amendments to outcomes which were considered to threaten the reliability of the data and as a result, no further explicit verification by study was conducted.

Evaluation and Analysis

Descriptive Characteristics

Descriptive characteristics of included trials were based on variables which could be potentially used to predict qualitative and quantitative measures of reporting safety data. Founded upon the variables considered by Ioannidis with additions and modifications subject to the context of this evaluation, descriptive characteristics included: Total sample size; whether or not a trial was reported as double-blind; whether or not the trial reported significant results for efficacy; type of funding, industry versus non-industry; longest duration to follow up subject to CAM trials (a 6 month divide was deemed appropriate). We recorded information to assess if more words or more adequate reporting was given to safety information in paediatric trials, therapeutic trials (versus preventative) studies which were in combination with conventional medicine (i.e. Acupuncture and Chemotherapy) and studies which were conducted at multiple centres. We also collected 2009 impact factor information for publishing journals of included studies; this information was collected from the journal websites.

Adequacy of Safety Reporting: Qualitative and Quantitative Measures

Qualitative and quantitative components of AEs reporting offer complementary information. The measures developed by Ioannidis in 2001 were used as guidance in the selection of qualitative and quantitative measures for this evaluation. We measured the following two qualitative components: 1. the reporting of dropouts due to AEs and whether the total number of dropouts were reported; and 2. Whether severity of the described AEs were reported, such reporting was classified as adequate, partially adequate, inadequate or no harms reported (Figure 1).
Figure 1

Defining qualitative parameters for adequacy of reporting.

In order to determine the relative emphasis of safety reporting in these CAM reports we assessed quantitative measures, namely, the percentage of words reporting safety in the body of the text and the percentage of words reporting safety in the results section. The full body of the text was defined as all words in the trial report excluding the title, abstract, acknowledgements, appendices and affiliations. Sentences including any words reporting harms were not broken. Like Ioannidis, we compared the relative reporting of safety in the body of the text with the text given to author affiliations. We also extracted the number of words devoted to the reporting of safety and the total number of words in the results section to compute the percentage of words devoted to safety in the results section.

CONSORT for Harms

In 1996, a group of international experts published the CONSORT Statement [15]. This reporting guideline was updated in 2001 and more recently in 2010 [16, 17]. Several extensions to facilitate reporting of other trial designs, such as cluster trials, and other types of data including harms, have been developed. We applied the CONSORT for harms extension collecting data on each of the first seven recommendations described as part of this evaluation (Figure 2).
Figure 2

CONSORT for Harms Data Recommendations.

Data Analysis

All descriptive characteristics are reported by CAM area as the frequencies and percentages of trials with the exception of sample size and 2009 journal impact factor for which median and inter-quartile ranges are reported. Descriptive characteristics were used as predictor variables in both least-squares multiple regressions and binary logistic regressions. The qualitative and quantitative measures of percentage of words devoted to the reporting of safety in the results section, and adequacy of reporting were respective dependent variables. Multiple least-squares analyses for each predictor were run adjusting for CAM area by dummy variables. Logistic regressions of clinical AEs (no harms reported and adequate reporting versus partially adequate and inadequate) were also run. Similarly, all analysis was adjusted for CAM therapy using dummy variables. All analysis was conducted using Minitab® Version 16.1.1.

Results

Search results and included trials

A total of 487 trials were identified. Of these, 166 trials were excluded as full text articles were not available; 29 trials were excluded as they did not fall under any of the 15 pre-defined CAM intervention categories; 87 more trials were excluded for not meeting our eligibility criteria (e.g., protocols, non-randomized trials, secondary publications, primary outcome AEs, incalculable word count, non-human study). Two of the fifteen CAM areas, echinacea and chondrotin, did not yield any trials. The remaining 205 trials, representing 13 different CAM areas, were included in the review (Figure 3).
Figure 3

Study Flow Diagram.

Descriptive Characteristics of Included Trials

Descriptive characteristics of included trials are presented in Table 1. Of the 205 included trial reports, 48% (n = 99) were described as double-blind. The majority of trials (74%, n = 152) evaluated a therapeutic CAM intervention versus a preventative intervention, and 23% (n = 47) of interventions were a combination of CAM and non-CAM treatments. Only five of the 13 CAM categories (acupuncture, Chinese herbal medicine, co-enzyme Q-10, combination herbal products and fish oil) included a paediatric population.
Table 1

Characteristics of included CAM RCTs

Intervention

Sample Size Median (Q1-Q3), Total

Double-blind No. (%)

Significant results for efficacy No. (%)

Industry funded No. (%)

Longest duration to follow up ≥ 6 m, No. (%)

Paediatric population No. (%)

Therapeutic studies No. (%)

Combined Studies No. (%)

Multicenter trials No. (%)

Journal IF > 5, No. (%)

Acupuncture, n = 75

60 (32-102) 8,717

16 (21)

58 (77)

5 (7)

9 (12)

4 (5)

71 (95)

23 (31)

11 (15)

9 (12)

Chinese herbal Medicine, n = 46

71 (48-98) 6,170

20 (43)

36 (78)

3 (6.5)

6 (13)

2 (4)

32 (70)

12 (26)

11 (24)

1 (2)

Chiropractic, n = 6

32 (28-47) 402

1 (17)

4 (67)

1 (17)

1 (17)

-

5 (83)

2 (33)

1 (17)

-

Coenzyme Q-10, n = 8

56 (34-166) 789

8 (100)

6 (75)

2 (25)

4 (50)

2 (25)

5 (63)

-

1 (13)

2 (25)

Combination herbal products, n = 11

85 (52-97) 882

9 (82)

6 (55)

6 (55)

2 (18)

1 (9)

9 (82)

1 (9)

3 (27)

-

Fish oil/Omega 3, n = 29

64 (33-105) 2,671

25 (86)

14 (48)

7 (24)

10 (34)

4 (14)

14 (48)

6 (21)

3 (10)

2 (7)

Flax Seed, n = 6

48 (38-68) 321

4 (67)

4 (67)

1 (17)

1 (17)

-

2 (33)

-

-

-

Garlic supplements, n = 2

542 (303-780) 1,083

2 (100)

2 (100)

1 (50)

2 (100)

-

-

1 (50)

-

-

Gingko Biloba, n = 4

80 (56-835) 3,243

2 (50)

2 (50)

1 (25)

1 (25)

-

3 (75)

-

1 (25)

1 (25)

Ginseng, n = 6

62 (30-85) 394

5 (83)

2 (33)

3 (50)

1 (17)

-

2 (33)

1 (17)

-

2 (33)

Glucosamine, n = 3

22 (19-122) 260

1 (33)

2 (67)

-

2 (67)

-

3 (100)

-

1 (33)

-

Massage, n = 1

- 35

1 (100)

1 (100)

-

-

-

-

-

-

-

Traditional medicine, n = 8

66 (48-108) 643

5 (63)

7 (88)

2 (25)

-

-

6 (75)

1 (13)

3 (38)

-

TOTAL, (n = 205)

63 (36-100) 25,610

99 (48)

144 (70)

33 (16)

39 (19)

13 (6)

152 (74)

47 (23)

35 (17)

17 (8)

'-' Denotes zero trials

'NR' Not reported

Seventy six (37%) trials did not report a source of funding. Thirty three (16%) trials were funded by industry, 47 (23%) by government, 49 (24%) by academic institutions. Eighty three (40%) trials did not report the longest duration of follow up. Only 35 (n = 17%) of trials were multi-centered. The median 2009 journal impact factor for all publications was 1.86 (range 0.42 to 9.81). One study, published in JAMA, was not included in the calculation due to a comparatively large 2009 impact factor.

Adequacy of safety reporting

Qualitative measures

Fifteen percent (31/205) of trials explicitly reported that no AEs had been experienced during the trial. More than half of all trials (56%, 114/205) had inadequate reporting of safety data. Of these, 69% (79/114) had no words dedicated to the reporting of harms anywhere in the trial report. In 18% (36/205) of all trials the reporting of safety was deemed to be adequate and in 12% (24/205) of the trials, the safety reporting was considered as partially adequate. Although treatment discontinuation was reported in 79% (162/205) of trials, the number reporting details on dropouts due to AEs was less at 60% (124/205) (Table 2).
Table 2

Safety Reporting: Qualitative measures

 

Clinical Adverse Events, No. (%)

Discontinuations

Intervention

Adequate

Partially adequate

Inadequate

No harms reported

Total reported

Due to adverse events

Reporting adverse events when reporting discontinuations

Acupuncture, n = 74

8 (11)

12 (16)

46 (61)

9 (12)

57 (76)

45 (60)

45/57

Chinese herbal

Medicine, n = 46

10 (22)

5 (11)

23 (50)

8 (17)

32 (70)

31 (67)

31/32

Chiropractic, n = 6

2 (33)

1 (17)

2 (33)

1 (17)

5 (83)

5 (83)

5/5

Coenzyme Q-10, n = 8

1 (13)

-

4 (50)

3 (38)

7 (88)

4 (50)

4/7

Combination herbal products, n = 11

3 (27)

-

6 (55)

2 (18)

9 (82)

8 (73)

8/9

Fish oil/Omega 3, n = 29

3 (10)

3 (10)

19 (66)

4 (14)

27 (93)

15 (52)

15/27

Flax Seed, n = 6

2 (33)

-

4 (67)

-

4 (67)

3 (50)

3/4

Garlic supplements, n = 2

-

-

2 (100)

-

2 (100)

-

0/2

Gingko Biloba, n = 4

2 (50)

-

-

2 (50)

4 (100)

3 (75)

3/4

Ginseng, n = 6

2 (33)

-

4 (67)

-

6 (100)

5 (83)

5/6

Glucosamine, n = 3

-

-

2 (67)

1 (33)

2 (67)

1 (33)

1/2

Massage, n = 1

-

-

1 (100)

-

-

-

-

Traditional medicine, n = 8

3 (38)

3 (38)

1 (13)

1 (13)

7 (88)

4 (50)

4/7

Total (n = 205)

36 (18)

24 (12)

114 (56)

31 (15)

162 (79)

124 (60)

124/162

Quantitative measures

The median percentage of words for reporting safety in the full body of the text was 0.94 (Q1-Q3 0.00-3.26). The median percentage of words in the results section devoted to the reporting of safety was 2.55 (Q1-Q3 0.00-11.07). Less than 30% (n = 60) of trials devoted an equal or greater percentage of words for the reporting of safety than to author affiliations (Table 3). In 141 (69%) trials more words were given to the reporting of author affiliation information than safety information, in 4 (2%) trials the percentage of words were considered to be the same (within 0.1%), and 60 (29%) trials gave more space to the reporting of harms relative to the reporting of author information.
Table 3

Safety of Reporting: Quantitative measures

 

Percentage of words for safety reporting

Relative emphasis on safety reporting, No. (%)

Other sources of safety data, No. (%)

Intervention

Body of the text, median (Q1-Q3)

Results section, median (Q1-Q3)

Safety < affiliations

Safety = affiliations

Safety > affiliations

= 1 Table for safety data

= 1 Figure for safety data

Additional resources

Acupuncture, n = 75

0.7 (0.0-2.0)

1.2 (0.0-9.0)

56 (75)

1 (1)

18 (24)

7 (9)

3 (4)

2 (3)

Chinese herbal Medicine, n = 46

1.5 (0.0-5.1)

7.1 (0.0-15.7)

27 (59)

1 (2)

18 (39)

6 (13)

-

-

Chiropractic, n = 6

2.0 (0.1-3.7)

1.8 (0.0-15.3)

3 (50)

-

3 (50)

1 (17)

1 (17)

-

Coenzyme Q-10, n = 8

1.2 (0.0-3.1)

1.9 (0.0-5.1)

6 (75)

-

2 (25)

1 (13)

1 (13)

-

Combination herbal products, n = 11

2.7 (1.0-4.5)

6.6 (0.5-13.3)

5 (45)

1 (9)

5 (45)

2 (18)

2 (18)

1 (9)

Fish oil/Omega 3, n = 29

0.0 (0.0-1.8)

0.0 (0.0-6.9)

21 (72)

1 (3)

7 (24)

1 (3)

3 (10)

1 (3)

Flax Seed, n = 6

0.0 (0.0-1.4)

0.0 (0.0-6.0)

5 (83)

-

1 (17)

-

-

-

Garlic supplements, n = 2

0.0 (0.0-0.0)

0.0 (0.0-0.0)

2 (100)

-

-

-

-

2 (100)

Gingko Biloba, n = 4

1.6 (1.5-1.9)

9.0 (7.4-11.7)

4 (100)

-

-

-

-

1 (25)

Ginseng, n = 6

0.5 (0.0-2.1)

2.0 (0.0-0.6)

5 (83)

-

1 (17)

1 (17)

-

-

Glucosamine, n = 3

0.0 (0.0-2.4)

0.0 (0.0-3.5)

2 (67)

-

1 (33)

-

-

-

Massage, n = 1

-

-

1 (100)

-

-

-

-

-

Traditional medicine, n = 8

3.6 (1.1-8.4)

9.1 (3.3-21.6)

4 (50)

-

4 (50)

2 (25)

-

-

TOTAL (n = 205)

0.9 (0.0-3.3)

2.6 (0.0-11.1)

141 (69)

4 (2)

60 (29)

21 (10)

10 (5)

7 (34)

For all trials the percentage of words for reporting author affiliations was statistically greater than that for reporting safety data (Wilcoxon test, p < 0.001). Two of the thirteen CAM areas, Acupuncture and Fish Oil/Omega 3 (P < 0.001 and P < 0.01 respectively) also showed significant differences in the number of words devoted to safety and affiliations, with more words given to the reporting of author affiliations. Only 10% of trials reported safety data in at least one table, and 5% of all trials reported safety data in a figure. Seven (3%) trials referred to additional external resources (online or archived) with additional safety data.

CONSORT for Harms

In general, we found a low compliance with seven CONSORT for harms recommendations. Of these recommendations, 'participant flow' yielded the highest frequency for reporting in 30% (n = 61) of trials. The lowest compliance (< 5% of all trials) was observed with recommendation 2, the reporting harms information in the introduction (Table 4).
Table 4

Reporting of Safety in accordance with CONSORT for Harms Recommendations

 

CONSORT for Harms Recommendation, No. (%)

Intervention

1

2

3

4

5

6

7

Acupuncture, n = 74

13 (17)

2 (3)

5 (7)

9 (12)

6 (8)

29 (39)

16 (21)

Chinese herbal Medicine, n = 46

16 (35)

3 (7)

7 (15)

11 (24)

5 (11)

9 (20)

10 (22)

Chiropractic, n = 6

2 (33)

1 (17)

-

-

1 (17)

2 (33)

2 (33)

Coenzyme Q-10, n = 8

1 (13)

-

1 (13)

1 (13)

-

3 (38)

1 (13)

Combination herbal products, n = 11

5 (45)

1 (9)

-

2 (18)

-

4 (36)

2 (18)

Fish oil/Omega 3, n = 29

1 (3)

1 (3)

1 (3)

4 (14)

-

8 (28)

2 (7)

Gingko Biloba, n = 4

-

-

-

1 (25)

-

1 (50)

-

Ginseng, n = 6

1 (17)

-

-

1 (17)

-

2 (33)

2 (33)

Glucosamine, n = 3

1 (33)

-

-

1(33)

-

-

-

Traditional medicine, n = 8

4 (50)

1 (13)

-

3 (38)

-

1 (13)

2 (25)

TOTAL (n = 205)

43 (21)

9 (4)

13 (6)

34 (17)

12 (6)

61 (30)

37 (18)

Please Note: None of the trials of flax seed, garlic supplements or massage adhered to any of the CONSORT recommendations

Regression Analysis

In univariate analysis, the percentage of words devoted to the reporting of safety in the results section increased significantly for multicenter trials (p = 0.045), (Table 5). In multivariable analysis, multicenter trials were retained as a statistically significant predictor for increasing the percentage of words in the results section of a trial irrespective of CAM area. It is of interest that Coenzyme Q-10 (p = 0.017), Fish oil/Omega 3 (p = 0.018) and Flax Seed (p = 0.043) all saw significant reductions in the percentage of words in the results section reporting safety. Logistic regressions did not yield any significant increases in odds of adequate reporting of safety based on predefined predictor variables. Although non-significant, industry funded trials odds of adequate reporting of harms were 2.49 times (95% CI, 0.98 to 6.32) that of non-industry funded trials; and multicenter trials had an odds of 1.89 (95% CI, 0.98 to 1.43) times those of single centre trials of adequately reporting safety information.
Table 5

Regression Analysis

 

Increase in percentage of words devoted to the reporting of safety in the results sectionŦ

Adequacy of reporting

Predictors

Least-square regressions (Adjusted¥) Effect Size (95% CI)

Logistic regressions (Adjusted) OR (95% CI)

Sample Size (per 10 unit increase)

0.02 (-0.04-0.07)

1.00 (1.00-1.00)

Double Blind

2.86 (-1.04-6.76)

1.05 (0.47-2.31)

Significant results for efficacy

-1.93 (-5.36-1.49)

1.18 (0.03-1.40)

Industry Funded

0.27 (-4.04-4.58)

2.49 (0.98-6.32)

Longest duration to follow-up, ≥ 6 m

-0.07 (-4.04-3.91)

0.99 (0.42-2.33)

Paediatric population

-4.50 (-11.07-2.07)

0.56 (0.11-2.90)

Therapeutic studies

3.21 (-0.84-7.26)

1.81 (0.73-4.48)

Combined studies

-0.32 (-4.03-3.39)

0.79 (0.34-1.81)

Multicentre

4.02 (0.12-7.92)*

1.89 (0.84-4.27)

2009 Journal Impact Factor (per 10 unit increase)

1.05 (-6.01-8.18)

1.19 (0.98-1.43)

*Statistically significant, p = 0.045, ¥All regressions adjusted for CAM intervention, ŦMultiple least-squares regressions for each predictor detailed. Adjusted forced entry (p < 0.1) and stepwise elimination and entry models retain multicenter (p = 0.03) and therapeutic studies (0.02).

Discussion

In general, findings of this evaluation indicate that the safety reporting across trials of CAM interventions is inadequate and often ignored altogether. Largely, the percentage of words devoted to the reporting of safety was equal to or less than that devoted to author affiliations. Adequate reporting of AEs was only observed in 18% of trials, with a further 9% of trials reporting that no harms were observed. Thirty-nine percent of trials gave no mention whatsoever to potential AEs. The regression models showed that the percentage of words devoted to the reporting of safety was significantly greater for the reports of multicenter trials compared to the reports of single centered trials; but otherwise none of the considered predictor variables were significant in modelling the percentage of words in the results section or the adequacy of reporting.

Consistent with the somewhat acceptable reporting of dropouts due to AEs, the assessment of adherence to the CONSORT for harms extension demonstrates that participant flow diagrams are the most successfully reported recommendation; however, this is under 30% of trials. Notably, perhaps, only just over 20% of trials discuss AEs in the abstract (Recommendation 1). Similar to the adequacy of reporting of safety outcome, the compliance of trials to recommendations of the CONSORT extension is weak.

These main findings are somewhat consistent with Ioannidis' evaluation of seven medical areas from a decade ago. The percentage of trials adequately reporting safety information in CAM trials was lower (18% of the total, or 21% of those trials not explicitly reporting no harms, versus 39%). Our finding that a larger percentage of words were devoted to author affiliations compared to AEs mirrors Ioannidis' finding that more publication area is devoted to author affiliations than to AEs. The overall conclusions of both evaluations are consistent in that the percentage of trials reporting harms and the adequacy by which they are reported is largely inadequate irrespective of defined predictors.

Inadequate reporting of harms also has consequences for systematic reviewers. The synthesis of harms from individual trials will be seriously compromised by inadequate reporting. The net effect is to negate the altruistic volunteerism of trial participants; there are important ethical and moral reasons to improve the quality of reporting of all research [18].

We would like to note some limitations to our evaluation. This is a sample across both physical interventions and natural products, therefore we may be assessing interventions for which AEs rarely occur [19, 20] (or rarely in comparison to drug interventions), or by which symptoms of overdose or usage are very minor and would not require medical attention. As such authors may not have considered reporting safety information. This by no means justifies the lack of adequacy of reporting of safety for these trials. If it is deemed by the trialist that no such AEs could occur independently or by interaction with other medications [21, 22], this should be sufficiently described in the trial report.

Due to limited resources, only locally available trials were included. This excluded a large number of trials reported in languages other than English. However, based on the similarity of included results of the reporting of adequacy across all CAM areas, it is unlikely that the exclusion of these trials would greatly affect our results. We have drawn comparison between proportion of words given to author affiliation and safety of reporting. Our results suggested that 69% of trials devoted more words to author affiliations than reporting of safety, and 29% devoted more words to safety than affiliation. This discrepancy may be somewhat confounded by a number of external influences such as editorial policy, number of investigators involved with a study and for the trials which report no harms. We feel however, that although the difference may not be as large as reported, this does not negate the finding that more trials devoted more space to author affiliations than to the reporting of safety.

We would also like to note that not all trials were included in the regression analysis (n = 172). Trials with missing data on duration to follow up and trials published in new journals without 2009 impact factors were omitted. The fit of all regression models was poor. This would suggest that either there are other variables associated with the quality of reporting of safety data which we did not consider, or these characteristics can by no means predict the quality of reporting, and that ultimately, the reporting of AEs is independently subject to the study author.

It could be hypothesised that there would be an association between journal endorsement of the CONSORT statement and trial adherence to the CONSORT for harms extension. Hopewell et al. report in 2008 the adherence to the CONSORT for harms statement is mentioned in only 3 of 165 high impact medical journals [23]. It is likely that this lack of endorsement is consistent within our sample; we did not investigate this association in this evaluation as the proportion of adhering journals would be sparse.

This study provides strong evidence, including many trials across many well established CAM interventions, that the reporting of safety data is inadequate. While there have been ongoing strides in conventional medicine to standardizing the collection, analysis and reporting of efficacy data in clinical trials [4], and more recently, the reporting of safety [5]; the Cochrane CAM Field has found no other evidence regarding the adequacy of such reporting within CAM trials. Hence, this study is the likely the first to provide such information.

Given the increasing use of CAM, it is necessary to improve such reporting to enable accurate and objective marketing and use of such treatments or products. Although large robust trials have a lot of information to report, it is not acceptable for the reporting of safety to be overlooked. Sufficient guidance must be provided to authors to ensure that the safety data is comprehensive and accessible to readers even when there have been no observed AEs.

The CONSORT for harms extension is a valuable resource for improving the quality of reporting of harms in conventional medicine trials. However, it is clear that there is a need for the use of such guidance to become standard procedure for authors and editors when presenting findings for all trials. Few journals explicitly endorse CONSORT for harms or other reporting guidelines. Parenthetically, there is growing evidence that use of reporting guidelines is associated with improved quality of reporting [2426]. Plint and colleagues reported a systematic review of eight trials evaluating the use of the CONSORT checklist and found that its use was associated with improved quality of reporting. A recent update of this review [27] now includes 49 studies examining almost 9000 trials provides strong stronger support for the association between use of the CONSORT checklist and improved reporting.

Conclusions

An evaluation of safety reporting in the reports of CAM RCTs across 15 different CAM interventions demonstrated that the reporting of harms was largely inadequate. The quality of reporting safety information in primary reports of CAM randomized trials requires improvement. We hope that these data will impress journal editors who, in turn will now endorse reporting guidelines as an important way to improve the quality of reporting harms.

Declarations

Acknowledgements

This publication was made possible by grant number R24 AT001293-04 from the National Center for Complementary and Alternative Medicine (NCCAM) at the National Institutes of Health. Its contents are solely the responsibility of the authors and do not necessarily represent the official views of NCCAM. No conflicts of interest have been declared by authors or other involved individuals. Primarily we would like to thank Dr. Ioannidis who kindly gave consent for the part methodological replication of his work. We would like to thank Ms. Sophia Tsouros and Ms. Mistrel Pratt who made significant contributions to the project with screening and extracting data. We would like to thank Raymond Daniel for developing, running the search and acquiring articles and we would also like to thank Mr. Steve Doucette for his statistical support.

Authors’ Affiliations

(1)
Ottawa Methods Centre, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa Hospital
(2)
Centre for Integrative Medicine, University of Maryland School of Medicine

References

  1. NIH News: According to a New Government Survey, 38 Percent of Adults and 12 Percent of Children Use Complementary and Alternative Medicine. Accessed March 17, 2011, [http://nccam.nih.gov/news/2008/121008.htm]
  2. Ioannidis JPA, Contopoulos-Ioannidis DG: Reporting of safety data from randomized trials. Lancet. 1998, 352: 1752-1753. 10.1016/S0140-6736(05)79825-1.View ArticlePubMedGoogle Scholar
  3. Ioannidis JPA, Lau J: Completeness of Safety Reporting in Randomized Trials. An Evaluation of 7 Medical Areas. JAMA. 2001, 285 (4): 437-443. 10.1001/jama.285.4.437.View ArticlePubMedGoogle Scholar
  4. The Asilomar Working Group on Recommendations for Reporting of Clinical Trials in the Biomedical Literature. Checklist of information for inclusion in reports of clinical trials. Ann Intern Med. 1996, 124: 741-743.Google Scholar
  5. CONSORT. Transparent reporting of Trials: Harms. Accessed March 16, 2011, [http://www.consort-statement.org/extensions/data/]
  6. Ioannidis JPA, Evans SJ, Gotzsche PC, O'Neill RT, Altman DG, Schulz K, Moher D: Better reporting of harms in randomized trials: an extension of the CONSORT statement. Ann Intern Med. 2004, 141 (10): 781-788.View ArticlePubMedGoogle Scholar
  7. Cochrane CAM Field. Trials Register. Accessed March 17, 2011, [http://www.compmed.umm.edu/cochrane_about.asp]
  8. The 10 most common CAM therapies among adults 2007. Accessed March 17, 2011, [http://nccam.nih.gov/news/camstats/2007/72_dpi_CHARTS/chart4.htm]
  9. The use of Complementary and Alternative Medicine in the United States. Use of Natural Products. Accessed March 17, 2011, [http://nccam.nih.gov/news/camstats/2007/camsurvey_fs1.htm#top]
  10. Chung CK, Wyman JF, Halcon LL: Use of complementary and alternative therapies in community-dwelling older adults. J Altern Complement Med. 2007, 13 (9): 997-1066. 10.1089/acm.2007.0527.View ArticleGoogle Scholar
  11. Chen F, Chen T, Kung Y, Chen YC, Chou LF, Chen FJ, Hwang SJ: Use frequency of traditional Chinese medicine in Taiwan. BMC Health Serv Res. 2007, 7: 26-10.1186/1472-6963-7-26.View ArticlePubMedPubMed CentralGoogle Scholar
  12. Barnes PM, Bloom B: Complementary and Alternative Medicine Use among Adults and Children: United States. 2007, Accessed March 17, 2011, [http://198.246.98.21/nchs/data/nhsr/nhsr012.pdf]Google Scholar
  13. Cochrane CAM Field. Glossary of CAM Terms. Accessed March 17, 2011, [http://www.compmed.umm.edu/cochrane_glossary.asp]
  14. Langevin H, Wayne PM, MacPherson H, Schnyer R, Milley RM, Napadow V, Lao L, Park J, Harris RE, Cohen M, Sherman KJ, Haramati A, Hammerschlag R: Paradoxes in acupuncture research: strategies for moving forward. Evid Based Complement Alternat Med. 2011, 180805-Google Scholar
  15. Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I, Pitkin R, Rennie D, Schulz KF, Simel D, Stroup DF: Improving the quality of reporting of randomized controlled trials: the CONSORT statement. JAMA. 1996, 276: 637-639. 10.1001/jama.276.8.637.View ArticlePubMedGoogle Scholar
  16. Moher D, Schulz KF, Altman DG, the CONSORT Group: The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet. 2001, 357: 1191-4. 10.1016/S0140-6736(00)04337-3.View ArticlePubMedGoogle Scholar
  17. Schulz KF, Altman DG, Moher D, the CONSORT Group: CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials. BMC Medicine. 2010, 8: 18-10.1186/1741-7015-8-18.View ArticlePubMedPubMed CentralGoogle Scholar
  18. Moher D: Reporting Research results: a moral obligation for all researchers. Can J Anaesth. 2007, 54: 331-335. 10.1007/BF03022653.View ArticlePubMedGoogle Scholar
  19. MacPherson H, Thomas K, Walters S, Fitter M: A prospective survey of adverse events and treatment reactions following 34,000 consultations with professional acupuncturists. Acupunct Med. 2001, 19: 93-102. 10.1136/aim.19.2.93.View ArticlePubMedGoogle Scholar
  20. Melchart D, Weidenhammer W, Streng A, Reitmayr S, Troph DO, Hoppe A, Ernst E, Linde K: Prospective Investigation of Adverse Effects of Acupuncture in 97 733 Patients. Arch Intern Med. 2004, 164: 104-105. 10.1001/archinte.164.1.104.View ArticlePubMedGoogle Scholar
  21. University of Maryland Medical Center: Medical Alternative Medicine Index. Accessed March 17, 2011, [http://www.umm.edu/altmed/]
  22. MedlinePlus Herbs and Supplements. Accessed March 17, 2011, [http://www.nlm.nih.gov/medlineplus/druginfo/herb_All.html]
  23. Hopewell S, Altman DG, Moher D, Schulz KF: Endorsement of the CONSORT Statement by high impact factor medical journals: a survey of journal editors and journal 'Instructions to Authors'. Trials. 2008, 9: 20-10.1186/1745-6215-9-20.View ArticlePubMedPubMed CentralGoogle Scholar
  24. Prady SL, MacPherson H: Assessing the Utility of the Standards for Reporting Trials of Acupuncture (STRICTA): A Survey of Authors. The Journal of Alternative and Complementary Medicine. 2007, 13 (9): 939-943. 10.1089/acm.2007.7186.View ArticlePubMedGoogle Scholar
  25. Hopewell S, Dutton S, Yu L, Chan AW, Altman DG: The quality of reporting of randomised trials in 2000 and 2006: comparative study of articles indexed in PubMed. BMJ. 2010, 340: c723-10.1136/bmj.c723.View ArticlePubMedPubMed CentralGoogle Scholar
  26. Plint AC, Moher D, Morrison A, Schulz K, Altman DG, Hill C, Gaboury I: Does the CONSORT checklist improve the quality of reports of 17andomized controlled trials? A systematic review. MJA. 2006, 185 (5): 263-267.PubMedGoogle Scholar
  27. Moher D, Plint AC, Altman DG, Schulz KF, Kober T, Galloway EK, Weeks L, Dias S: Consolidated standards of reporting trials (CONSORT) and the quality of reporting of randomized controlled trials (Protocol). Cochrane Database of Systematic Reviews. 2010, 3Google Scholar
  28. Pre-publication history

    1. The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6882/11/67/prepub

Copyright

© Turner et al; licensee BioMed Central Ltd. 2011

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Advertisement