This article has Open Peer Review reports available.
Evaluating complementary and alternative medicine interventions: in search of appropriate patient-centered outcome measures
© Verhoef et al; licensee BioMed Central Ltd. 2006
Received: 27 July 2006
Accepted: 21 November 2006
Published: 21 November 2006
Central to the development of a sound evidence base for Complementary and Alternative Medicine (CAM) interventions is the need for valid, reliable and relevant outcome measures to assess whether the interventions work. We assessed the specific needs for a database that would cover a wide range of outcomes measures for CAM research and considered a framework for such a database.
The study was a survey of CAM researchers, practitioners and students. An online questionnaire was emailed to the members of the Canadian Interdisciplinary Network for CAM Research (IN-CAM) and the CAM Education and Research Network of Alberta (CAMera). The majority of survey questions were open-ended and asked about outcome measures currently used, outcome measures' assessment criteria, sources of information, perceived barriers to finding outcome measures and outcome domains of importance. Descriptive quantitative analysis and qualitative content analysis were used.
One hundred and sixty-four completed surveys were received. Of these, 62 respondents reported using outcome measures in their CAM research and identified 92 different specific outcomes. The most important barriers were the fact that, for many health concepts, outcome measures do not yet exist, as well as issues related to accessibility of instruments. Important outcome domains identified included physical, psychological, social, spiritual, quality of life and holistic measures. Participants also mentioned the importance of individualized measures that assess unique patient-centered outcomes for each research participant, and measures to assess the context of healing and the process of healing.
We have developed a preliminary framework that includes all components of health-related outcomes. The framework provides a foundation for a larger, comprehensive collection of CAM outcomes. It fits very well in a whole systems perspective, which requires an expanded set of outcome measures, such as individualized and holistic measures, with attention to issues of process and context.
Over the past 15 – 20 years, Complementary and Alternative Medicine (CAM) use has increased dramatically. Not surprisingly, questions of whether CAM interventions are safe, efficacious and effective have become key research issues. A review of articles indexed in MEDLINE between 1966 and 1996 suggests that both the number and proportion of reports of CAM clinical trials is increasing at a significant rate . More recently, there is increasing attention to Integrative Medicine (IM), which combines mainstream medical therapies and CAM therapies for which there is some high-quality scientific evidence of safety and effectiveness .
Central to the development of a sound evidence base for CAM interventions is the need for valid, reliable and relevant outcome measures to assess whether the interventions work, in what conditions and for what population groups. The number of available patient-centered outcome measures from a variety of health and social science disciplines is enormous. Several books have been published on health outcome measures (e.g., [3–9]) and, more recently, several websites have been developed to provide brief overviews of outcome measures (e.g., Measurement Excellence and Training Resource Information Center (METRIC); Patient-Reported Outcome and Quality of Life Instruments Database (PROQOLID)). Despite these developments, there is growing recognition by CAM practitioners and researchers that the current array of outcome measures is not sufficient for use in CAM research and practice, as they do not cover the full spectrum of observed treatment effects.
Many CAM interventions (e.g., traditional Chinese medicine, chiropractic, naturopathy, massage therapy) may be conceptualized as whole systems of care [10–14] which consist of multiple components that provide an internally consistent, individualized approach to treatment. Components of care include, for example, the intervention itself as well as the process and context of healing, which are intertwined in a whole system based on a unique and consistent philosophical foundation. An acknowledgement of the whole systems' nature of CAM interventions (and of IM) requires an expanded set of outcome measures, of which survival, conventional assessments of biomedical outcomes and quality of life are only part. CAM interventions and IM are often aimed at affecting more than one aspect of a patient's life, and instead focus on maximizing the individual patient's capacity to achieve mental and physical balance and to, globally, restore his/her own health [11, 13]. Thus, both individualized and global outcome measures appear necessary. It is becoming increasingly clear that commonly used outcome measures fall short in addressing these aspects. This perspective has led to a search for a broader range of outcome measures which may be less well-known, and for which psychometric quality is not always clear. To help raise awareness of the range of relevant and available outcome measures, we set out to develop an online database of outcome measures relevant to CAM/IM interventions which would be widely available. Here we present the results of a survey aimed at identifying the specific needs for a database that would cover a wide range of outcomes measures for CAM research. The objectives of the study were to identify: 1) where CAM researchers look for outcome measures; 2) which specific outcome measures are in use for CAM research; 3) assessment criteria used when selecting outcome measures; 4) perceived barriers to finding outcome measures for CAM research; and 5) outcome domains that are important for CAM research.
The study design was a survey of CAM researchers, practitioners and students. The need for such a survey and the identification of relevant questions was pre-tested in a workshop on CAM outcome measures offered at the second Annual Symposium of the Canadian Interdisciplinary Network for Complementary and Alternative Medicine Research (IN-CAM) in November 2005. Thirty participants attended the workshop and as part of the workshop completed a brief paper and pencil questionnaire. The responses were used to refine the final questionnaire; for example, small changes were made to make the questionnaire more "practitioner-friendly" (some practitioner participants felt the questionnaire was not applicable to them as they did not conduct academically oriented research) and to facilitate online completion of the questionnaire. The final questionnaire took approximately 5–10 minutes to complete.
In February 2006, a link to an online questionnaire was emailed to 1,100 IN-CAM members and 165 members of the CAM Education and Research Network of Alberta (CAMera). Membership in these networks is not a formal paid membership but consists of signing up in a membership database in return for which 'members' receive the networks' newsletters and newsflashes (emails). IN-CAM and CAMera members include CAM researchers, practitioners, educators, policy makers and students. One separate email describing the objectives and process of this survey was sent to all members of each network within the span of one week. There is some overlap in membership between the networks, so it is possible that some individuals received the notification more than once.
Sample Questions Asked on Electronic Questionnaire
Many outcome domains have been identified as important for CAM research that fall into the broad categories of biological, physical (e.g., function, pain), mental (e.g., depression, anxiety), social (e.g., role function, social support), emotional (e.g., hope) and spiritual outcomes domains.
Are there any OTHER outcomes of CAM interventions you feel are important and should be included in CAM research (please describe)?
What CAM outcome measures would you like to use but have not been able to find (please describe)?
Common assessment criteria that are used when selecting outcome assessment tools include validity, reliability, responsiveness, relevance, burden and cost.
Are there any OTHER assessment criteria that you use to assess measurement tools (please describe)?
What do you perceive to be barriers to finding outcome measures for CAM research (please describe)?
Analysis of responses to the open-ended questions was conducted using qualitative content analysis: responses to each open-ended question were first read as a complete list and then common responses were sorted into a set of data-derived categories. The goal of the analysis was, for each question, to summarize the range of responses and to suggest the importance of some categories over others based on frequency of mention.
Data from a question soliciting respondents' perceptions of important CAM outcome domains (Table 1, Question 1) was used to develop a preliminary framework of outcome domains important for CAM research. The framework emerging from the data was very basic and relationships between the domains were not specified. Subsequently, this framework was refined through a consensus building meeting attended by all authors and through consultation with experts in the field of CAM and outcome assessment.
Characteristics of Survey Respondents
Researcher or Research Staff
Actively involved in CAM research*:
Currently use outcome measures in research:
Resources for outcome measures
Respondents indicated that they most commonly refer to colleagues as a resource for outcome measures (34.4%), followed by libraries (27.4%), and the Internet (22.3%). PubMed was identified most frequently as an Internet site to help identify outcome measures. Google, the Cumulative Index to Nursing and Allied Health Literature and the National Center for Complementary and Alternative Medicine (United States) website were also mentioned, but less frequently.
Use of outcome measures
The 62 respondents who reported using outcome measures in their CAM research identified 92 different specific outcomes, of which 63 were identified by name. The remaining identified outcome measures were biological measures such as heart rate, blood pressure and range of motion, or general concepts, such as satisfaction, control and attitudes but with no specific tool name mentioned. The most commonly identified tools were: Short Form 36 Health Status Survey (SF-36: n = 9), Profile of Mood States (POMS: n = 7), Arizona Integrative Outcomes Scale (AIOS: n = 5), pain visual analogue scales (pain VAS: n = 4) and Measure Your Medical Outcome Profile (MYMOP: n = 4). Some of these outcome measures are also frequently used in conventional medical research (e.g., SF-36, pain VAS); however, others were specifically developed for and validated in CAM patient populations (e.g., AIOS, MYMOP).
Assessment criteria for outcome measures
Many assessment criteria were identified in addition to the provided list, including: accessibility (e.g., ease of locating, copyright issues), ease of use and scoring, extent of use, readability, overall look and word choice.
Barriers to finding outcome measures
Respondents identified many barriers that they face when trying to find outcome measures for their CAM research. The most commonly identified barrier was that appropriate and relevant outcome measures do not exist (n = 16). Respondents stressed that outcomes in CAM research deviate somewhat from those in conventional medical research and that measurement tools have not yet been developed to address the unique process, context and outcomes of CAM interventions. Closely related to this barrier was the belief that many of the concepts important to CAM research have not yet been explicitly defined (n = 8), for example personal transformation  or unstuckness . Other identified barriers included: a lack of psychometric data on existing tools (n = 11); lack of a common resource of outcome measures (n = 7); not knowing where to look (n = 5); poor documentation regarding outcome measures used in published research, making comparative studies difficult (n = 4); copyright issues (n = 2); and prohibitive cost (n = 2).
Important domains of outcome measures
Participants responded in many different ways to the question regarding important outcome domains for CAM research. Many identified that it is important to look at different perspectives of outcome measures such as the patient, provider, family and community perspective. Others identified aspects of interventions other than outcomes, such as process, context and meaning. An additional group provided examples of outcome measures that they might have found difficult to group in domains, such as serenity, integration, body image, ability to relax, awareness and holism.
A framework of outcome domains important in CAM research
Preliminary Framework of Outcome Domains Important for CAM Research
Holistic wellness/well-being (encompassing each of physical, psychological, social, spiritual)
Attitudes and Beliefs
Health care utilization
Function/Activities of Daily Living
Role function in daily life and work
Locus of control
Openness to experience
Patient perceived self-efficacy
Patient perceptions of care
Patient perceptions of risk
Patient preference for control
Sense of Coherence
Quality of Life (assessing at least two of physical, psychological, social, spiritual)
e.g. Measure Your Medical Outcome Profile, Measure Your Concerns and Well-being, Goal Attainment Scaling
Context of healing
Negotiations between patients and practitioners
Patient-centeredness of healing
Practitioner attitudes towards integration
Process of healing
Engagement with intervention experience
Table 3 identifies each main category heading and lists sub-concepts to describe specific outcomes in that domain. For example, awareness, balance and enablement are fundamental concepts within the spiritual domain. Awareness, however, may also be conceptualized as a psychological outcome and for this reason is listed under both the spiritual and psychological domains. Although the nine categories are not mutually exclusive, our aim was for as much exclusivity as possible. Many relevant concepts, by their nature, address more than one clinically relevant outcome domain and therefore exclusivity appears impossible.
Our preliminary framework provides a foundation for the development of a large, comprehensive collection of CAM outcomes. The framework fits very well in a whole systems perspective, which requires an expanded set of outcome measures. Several authors have previously conceptualized the implications of a whole systems approach to the range of potential patient-related outcomes [14, 20, 21], although it was not their intent to develop a framework of CAM outcomes. All identified the importance of the process and context of an intervention, in particular the patient-provider relationship. Each also addressed the impact of the philosophy and practice of health and healing on the therapeutic relationship, for example assisting patients in self-healing and providing individualizing treatments. Our framework builds on this work by highlighting how all relevant outcome domains, including issues of process and context, are linked and encompassed by a consistent philosophical approach to health and healing. Although measures do not exist for every outcome domain in the framework, the framework can be used to identify "gaps" and to direct future outcome measure development.
Our proposed framework may also be used when making decisions regarding which outcome measures to use to ensure all appropriate domains are considered. The framework and the data from this survey will be used to develop an online database of outcome measures relevant to CAM research. The database will help address some of the identified barriers to finding outcome measures – for example, the lack of a common resource. Further, appropriate marketing and hosting the database on IN-CAM's web site  may address the barrier of not knowing where to look.
Undoubtedly, the question will be posed whether there is a need to make such a clear distinction between conventional and CAM outcomes and whether we are not, in fact, adding to the polarization between conventional medicine and CAM. We do not believe this is the case, and would encourage a similarly broad perspective in conventional medical research. It is easy to see how conventional interventions can also be multidimensional, placing value on the patient-provider relationship and acknowledging non-specific aspects. For example, the assessment of treatment interventions in cancer care often falls short in addressing the complexity of cancer treatment outcomes. Whether or not CAM is included as part of treatment, cancer treatment consists of many component parts that are difficult to tease apart, such as support from friends and family, the degree to which information needs are met and a strong patient-provider relationship. Each component plays an essential role in patient healing and when considered as a whole system of care, the whole is much greater than the sum of the component parts. All effectiveness research, whether CAM or conventional, should be designed to capture all factors relevant to the intervention and all relevant outcomes. Other common examples of conventional interventions that may be regarded as whole systems are palliative care, addiction treatment and chronic pain management.
Our outcomes are particularly relevant to the concept of Integrative Medicine, which ideally covers the entire spectrum of 'good medicine' [23, 24]. Currently, many CAM and integrative medicine practitioners do not use outcome measures to assess their interventions and, further, many CAM/IM researchers limit themselves to biomedical and quality of life outcomes. Therefore, it is important that comprehensive information is available to encourage the use of psychometrically sound and relevant outcome measures in CAM/IM research and practice. We hope that the availability of an outcomes database will assist CAM/IM researchers in refining and clarifying their research questions, which too often are broad and unspecific. The framework highlights the fact that there are many potentially relevant outcomes of CAM/IM interventions. Ideally, research should encompass all relevant domains; but, the decision of which outcome measures to use must balance the choice between an all-encompassing set of outcome measures of high psychometric quality and patient burden. Further work on identifying a core group of outcomes for CAM research is clearly needed.
Some limitations to our study must be acknowledged, specifically with respect to our sampling method and response rate. First, we conducted an online survey of a motivated group of individuals interested in CAM research. Each member of our sampling frame has access to the Internet and had taken the time to sign up as a member of IN-CAM, CAMera or both. This group is therefore likely not representative of all individuals in Canada with an interest in CAM research. Further, our response rate was relatively low when standard survey methodology is considered; however it is the highest response rate received for an online survey conducted by either of the research networks. Since membership is free of charge and there are no requirements for network membership, it is to be expected that our survey was relevant to only some of the 'members'. Our respondents are clearly, then, a highly motivated group of individuals interested in CAM outcomes and may not be representative of CAM researchers and practitioners as a whole. We believe, however, that 164 responses, plus the length of many written responses, suggest the relevance of our project to those involved or interested in CAM research. The magnitude of the response also suggests that the area of CAM outcomes is in need of development and clarification.
A publicly available resource for accessing outcome measures that are relevant for CAM research is needed. We propose a framework for considering outcomes for clinical research into CAM from a whole systems perspective. We invite comments from readers regarding our framework and the database development. If you would like to be included on a list to remain updated about the progress of the database development, send the corresponding author an email.
This study resulted from a recommendation of the Advisory Board of the Canadian Interdisciplinary Network for Complementary and Alternative Medicine Research (IN-CAM), following a priority-setting meeting on 20th May 2005. We acknowledge the support of IN-CAM, Heather Boon and the other members of the Advisory Board, and wish to specifically acknowledge the substantial assistance of Linda Ferguson and Patrick Gignac. We are grateful for insightful comments from Cheryl Ritenbaugh, Iris Bell, and Ian McDowell on the proposed framework.
This research was made possible by a grant from the Lotte and John Hecht Memorial Foundation with a contribution from IN-CAM.
- National Center for Complementary and Alternative Medicine: What is Complementary and Alternative Medicine. [http://nccam.nih.gov/health/whatiscam/]
- Barnes J, Abbott NC, Harkness EF, Ernst E: Articles on complementary medicine in the mainstream medical literature: an investigation of MEDLINE, 1966 through 1996. Arch Intern Med. 1999, 159: 1721-5. 10.1001/archinte.159.15.1721.View ArticlePubMedGoogle Scholar
- McDowell I: Measuring health: A guide to rating scales and questionnaires. 2006, New York: Oxford University Press, 3View ArticleGoogle Scholar
- Bowling A: Measuring Disease: a Rreview of Disease-Specific Quality of Life Measurement Scales. 2001, Buckingham: Open University Press, 2Google Scholar
- Bowling A: Measuring Health: a Review of Quality of Life Measurement Scales. 2005, Buckingham: Open University Press, 3Google Scholar
- Bowling A: Research Methods in Health: Investigating Health and Health Services. 2002, Milton Keynes: Open University Press, 2Google Scholar
- Frank-Stromberg M, Olsen SJ: Instruments for clinical health care research. 2002, Boston: Jones and Bartlett, 2Google Scholar
- Spilker B: Quality of Life and Pharmacoeconomics in Clinical Trials. 1996, Philidelphia: Lippincott-Raven, 2Google Scholar
- Lipscomb J, Gotay C, Snyder C: Outcomes Assessment in Cancer: Measures, Methods and Applications. 2005, Cambridge: Cambridge University PressGoogle Scholar
- Bell IR, Caspi O, Schwartz GER, Grant KL, Gaudet TW, Rychener D, Maizes V, Weil A: Integrative medicine and systemic outcomes research. Arch Intern Med. 2002, 162: 133-140. 10.1001/archinte.162.2.133.View ArticlePubMedGoogle Scholar
- Ritenbaugh C, Verhoef M, Fleishman S, Boon H, Leis A: Whole systems research: a discipline for studying complementary and alternative medicine. Altern Ther Health Med. 2003, 9: 32-6.PubMedGoogle Scholar
- Verhoef M, Lewith G, Ritenbaugh C, Boon H, Fleishman S, Leis A: Complementary and alternative medicine whole systems research: beyond identification of inadequacies of the RCT. Complement Ther Med. 2005, 13: 206-12. 10.1016/j.ctim.2005.05.001.View ArticlePubMedGoogle Scholar
- Bell I, Cunningham V, Caspi O, Meek P, Ferro L: Development and validation of a new global well-being outcomes rating scale for integrative medicine research. BMC Complement Altern Med. 2004, 4: 1-10.1186/1472-6882-4-1.View ArticlePubMedPubMed CentralGoogle Scholar
- Long A: Outcome measurement in complementary and alternative medicine: unpicking the effects. J Altern Complement Med. 2002, 8: 777-86. 10.1089/10755530260511793.View ArticlePubMedGoogle Scholar
- Perrin EB: SAC Instrument Review Process. Medical Outcomes Trust Bulletin. 1995, 3: 1-4.Google Scholar
- Mulkins AL, Verhoef M: Supporting the transformative process: experiences of cancer patients receiving integrative care. Integr Cancer Ther. 2004, 3: 230-7. 10.1177/1534735404268054.View ArticlePubMedGoogle Scholar
- Bell IR, Koithan M, Gorman M, Baldwin CM: Homeopathic practitioner views of changes inpatients undergoing constitutional treatment for chronic disease. J Altern Complement Med. 2003, 9: 39-50. 10.1089/107555303321222937.View ArticlePubMedGoogle Scholar
- Paterson C, Britten N: In pursuit of patient-centred outcomes: a qualitative evaluation of the 'Measure Yourself Medical Outcome Profile. J Altern Complement Med. 2000, 5: 27-36.Google Scholar
- Kiresuk TJ, Lund SH, Larsen NE: Measurement of goal attainment in clinical and health care programs. Drug Intell Clin Pharm. 1982, 16: 145-53.PubMedGoogle Scholar
- Miller WL, Crabtree BF, Duffy MB, Epstein RM, Stange KC: Research guidelines for assessing the impact of the healing relationship in clinical medicine. Altern Ther Health Med. 2003, 9: 80A-95A.Google Scholar
- Paterson C, Britten N: Acupuncture as a Complex Intervention: A Holistic Model. J Altern Complement Med. 2004, 10: 791-801.View ArticlePubMedGoogle Scholar
- Interdisciplinary Network for Complementary and Alternative Medicine Research. [http://www.incamresearch.ca]
- Lee R, Kligler B, Shiflett S: Chapter 1: Integrative Medicine: Basic principles. Integrative Medicine. Principles for Practice. Edited by: Kligler B, Lee RD. 2004, New York: McGraw-HillGoogle Scholar
- Snyderman R, Weil AT: Integrative medicine: bringing medicine back to its roots. Arch Intern Med. 2002, 162: 395-7. 10.1001/archinte.162.4.395.View ArticlePubMedGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6882/6/38/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.