Acupuncture and chiropractic care for chronic pain in an integrated health plan: a mixed methods study
© DeBar et al; licensee BioMed Central Ltd. 2011
Received: 31 October 2011
Accepted: 25 November 2011
Published: 25 November 2011
Substantial recent research examines the efficacy of many types of complementary and alternative (CAM) therapies. However, outcomes associated with the "real-world" use of CAM has been largely overlooked, despite calls for CAM therapies to be studied in the manner in which they are practiced. Americans seek CAM treatments far more often for chronic musculoskeletal pain (CMP) than for any other condition. Among CAM treatments for CMP, acupuncture and chiropractic (A/C) care are among those with the highest acceptance by physician groups and the best evidence to support their use. Further, recent alarming increases in delivery of opioid treatment and surgical interventions for chronic pain--despite their high costs, potential adverse effects, and modest efficacy--suggests the need to evaluate real world outcomes associated with promising non-pharmacological/non-surgical CAM treatments for CMP, which are often well accepted by patients and increasingly used in the community.
This multi-phase, mixed methods study will: (1) conduct a retrospective study using information from electronic medical records (EMRs) of a large HMO to identify unique clusters of patients with CMP (e.g., those with differing demographics, histories of pain condition, use of allopathic and CAM health services, and comorbidity profiles) that may be associated with different propensities for A/C utilization and/or differential outcomes associated with such care; (2) use qualitative interviews to explore allopathic providers' recommendations for A/C and patients' decisions to pursue and retain CAM care; and (3) prospectively evaluate health services/costs and broader clinical and functional outcomes associated with the receipt of A/C relative to carefully matched comparison participants receiving traditional CMP services. Sensitivity analyses will compare methods relying solely on EMR-derived data versus analyses supplementing EMR data with conventionally collected patient and clinician data.
Successful completion of these aggregate aims will provide an evaluation of outcomes associated with the real-world use of A/C services. The trio of retrospective, qualitative, and prospective study will also provide a clearer understanding of the decision-making processes behind the use of A/C for CMP and a transportable methodology that can be applied to other health care settings, CAM treatments, and clinical populations.
We describe here our study designed to understand clinically meaningful outcomes (from both patient and provider perspectives) of acupuncture and chiropractic (A/C) care as delivered in routine practice settings for the treatment of chronic musculoskeletal pain (CMP). The centerpiece of the study is a prospective cohort study. However, before we undertake this phase of work, we will conduct an analysis of electronic medical record (EMR) data and qualitative data collection to provide the foundation for identifying a more meaningful comparison of outcomes between those receiving and not receiving A/C care. Our goal is to test an exportable methodological approach that can be used to understand critical outcomes associated with the receipt of an array of treatment services in everyday practice settings for a realistically diverse set of patients.
Chronic pain is a highly prevalent condition, often resulting in large decrements in health-related quality of life (QOL) and functional status, with substantial associated medical care costs, disability, and productivity loss [1–3]. CMP in particular is both prevalent and costly [4–7], affecting 60-80% of American adults at some point during their lives; CMP symptoms are among the top five reasons that patients visit clinics and emergency departments [6, 8]. Recent alarming increases in delivery of opioid treatment and surgical interventions for chronic pain--despite their high costs, potential adverse effects, and modest efficacy [9–12]--suggest the need to evaluate outcomes associated with promising non-pharmacological/non-surgical approaches for CMP management and treatment, including complementary and alterative medicine (CAM). Americans seek CAM treatments far more often for CMP than for any other condition .
Substantial recent research has examined the biological basis and efficacy of many types of CAM therapies but, despite calls for effectiveness-oriented research, "real-world" use of CAM remains understudied [14, 15]. Use of CAM for CMP appears to be increasing. A national survey  found that 38% of U.S. adults used some form of CAM, most commonly for relief of back and neck pain, joint pain and stiffness, arthritis, and other musculoskeletal conditions. Among CAM treatments for CMP, acupuncture and chiropractic care are considered the most highly accepted by physician groups [16, 17] with the best evidence to support their use [18–21]. Nearly 90% of states mandate insurance reimbursement for chiropractic care and approximately 25% do so for acupuncture . Further, a survey of acupuncturists and chiropractors in Massachusetts, Arizona, and Washington found that back pain was the most common reason given by patients for seeking treatments; overall, 40-76% of patients included CMP among their reasons for seeking such treatment .
Patients also report high levels of satisfaction with acupuncture  and chiropractic care . A Consumer Reports survey found that while more than half of the respondents reported being highly satisfied with care from acupuncturists (53%) and chiropractors (59%) for back pain, only 44% reported similar satisfaction with care from specialist physicians and 34% with care from primary care physicians . Despite some positive findings among observational studies and randomized controlled trials (RCTs) regarding the impact of acupuncture [18, 26, 27] and chiropractic care [18, 28–30] on CMP, highly controlled trials have suggested that expectation and non-specific effects may be substantial contributors to observed treatment effects [31–33]. These findings highlight the importance of examining patient expectations and treatment decision-making factors when evaluating such outcomes. Given both the popularity of A/C for CMP treatment and outcome findings, an important next step is to examine the use of these CAM therapies for CMP as they are delivered by providers in routine practice settings.
This project will use data collected from a prepaid group practice model health maintenance organization (HMO) that offers A/C coverage for CMP treatment; in this setting, enabling factors of access (insurance coverage, co-pays, lower out-of-pocket costs) will likely mitigate the influence of economic factors [47, 49, 50]. Consequently, we will be able to explore more fully those non-financial predisposing and need factors that affect A/C decision making. This framework also helps guide us toward the most important domains for measurement.
Finally, there is increased demand for innovative study designs using data from routine practice settings to compare strengths and weaknesses of various medical interventions. Ideal settings are health care systems that use EMRs, provide insurance coverage, and document the use of provider-delivered CAM services, such as A/C care. Innovative research designs in these settings can provide information for health care providers and patients about which CAM and conventional treatments are likely to provide the best clinical, functional, and quality-of-life outcomes for everyday users in routine practice settings [51–53]. These designs will include a more diverse range of study subjects than would RCTs and allow longer follow-up, facilitating identification of groups that may uniquely benefit or encounter complications. These settings permit study designs that can examine the full array of treatments and associated outcomes that patients may encounter.
Despite these advantages, making causal inferences in observational studies is more challenging than in RCTs because of confounding by indication. That is, patients (and/or doctors) choose treatments using information that may not be evident to researchers. Patients who receive a given treatment in everyday practice may be dissimilar from their counterparts who do not, in which case treatment outcomes may be at least partially related to unmeasured pre-treatment differences rather than the treatment received. For pre-treatment differences where we have valid information, recent analytic techniques offer the promise of identifying patients with similar probability of receiving a particular treatment [54–58]. In the event that one of these patients receives the treatment and the other does not, clinical outcomes may be validly compared for such individuals. Using such approaches can complement what can be achieved with RCTs relatively quickly and efficiently. Such observational studies can highlight important domains for subsequent confirmatory RCTs or point to patterns of utilization and outcomes that are not predicted by existing RCTs.
The aims of the present study are two-fold. The first goal is to examine who (i.e., CMP patients with what characteristics and history of clinical care) will have improved outcomes from A/C care, and to identify the specific characteristics of such care (e.g., duration, comprehensiveness of employed A/C modalities). The second goal is to test an exportable methodological approach that can be used to examine clinically meaningful outcomes for patients: (1) in different settings, (2) using different types of CAM or conventional medical services, with (3) different health care conditions and characteristics.
Overall Study Eligibility Criteria
≥ 18 years of age
Meets the following definition of musculoskeletal pain as determined review of the individual's electronic medical record (EMR) chart:
≥ 3 outpatient (emergency department, ambulatory visit, email and telephone encounters are acceptable) encounters spanning at least 180 days but no more than 18 months:
nonspecific chronic pain (nCP)
or chronic musculoskeletal pain (CMP). CMP includes arthritis, temporomandibular joint disorder, tension headaches, carpal tunnel syndrome, fibromyalgia and myofascial pain, pain in limb, pain in joint, back/neck pain, vertebral fractures, vertebral somatic dysfunction, spondylolysis, vertebral dislocation, spinal strains and sprains, spinal deformities
Three qualifying diagnoses can be achieved in any of the following ways:
Three CMP diagnoses
First diagnosis CMP and two subsequent diagnoses of nCP
First diagnosis CMP with one additional CMP diagnosis and one nCP
Diagnoses associated with a physical therapy visit will NOT be considered in identifying prospective participants
Only one CMP/nCP diagnosis can be counted in any given 7-day window to minimize the possibility that qualifying diagnoses reflect multiple visits/services for the same acute pain condition.
Study participant exclusion criteria include:
Medical history of malignant cancer other than non-melanoma skin cancer
Medical history of cognitive impairment (dementia, developmental delay, or psychosis)
Medical history of hospice or end-of-life palliative care
This study is being conducted at Kaiser Permanente Northwest (KPNW), a large HMO in the Pacific Northwest with several features that create a uniquely favorable environment for this research. First, KPNW has an extensive history of CAM integration. Due to Washington State's 1998 law  mandating insurance coverage for a broad array of licensed providers (including chiropractors and acupuncturists), KPNW began to offer CAM coverage for all health plan members. Administration of this coverage is provided by a managed complementary care network that maintains electronic data from HCFA-1500 forms (including information on diagnoses, procedures, and dates of service) and these data are available to the project. Second, any health plan member can be referred by a KPNW clinician for A/C care for qualifying conditions, including acupuncture for CMP and chiropractic care for acute exacerbations of CMP conditions. Although some patients pay out-of-pocket for A/C care, our out-of-plan CAM use survey will help us discern this prevalence. Third, in 1995, KPNW implemented a comprehensive EMR system (HealthConnect) that has since captured A/C services and documented all outpatient provider encounters, diagnoses, referrals, laboratory and imaging studies, and prescriptions. The HMO provides most services in its own hospitals and clinics, yet data are compiled and available on reimbursable external services received by health plan patients. Together these provide an all-inclusive picture of members' covered health care utilization.
Study participants are health plan members aged 18 or older with evidence of CMP. To identify such patients from the EMR, we use ICD-9 codes consistent with CMP-related diagnoses, as outlined in Table 1. Musculoskeletal pain is thought to contribute to tension headaches, carpal tunnel syndrome, and temporomandibular disorders (TMD). Accordingly, and because A/C care is sometimes sought for the treatment of such conditions, we include participants with these diagnoses. To ensure chronicity of the pain disorder, visits must indicate such diagnoses or receipt of commensurate health services spanning at least 180 days. Exclusion criteria include history of cancer or cancer-related pain, hospice or other end-of-life palliative care, cognitive impairment (dementia, developmental delay, and psychosis) that would make it unlikely that the patient would receive A/C and/or could participate in study assessments. In later sections, we describe additional inclusion and exclusion criteria specific to the qualitative (Phase 2) and prospective cohort (Phase 3) portions of the study.
KPNW Health Plan Members with CMP in 2010
No CAM referral in 2010
Chiropractic and/or acupuncture referral in 2010*
Mean (SD) or % (N)
Mean (SD) or % (N)
Total members 18 and older with CMP
Type of Chronic Pain
Back and Neck Pain
Joint Pain (including Osteoarthritis)
Fibromyalgia and other Myofascial Pain
Two or more types of pain from CMP definition
Two or more types of pain from more general
chronic pain definition
% White (1)
% Hispanic (2)
Phase 1: Construction of CMP sample and A/C Services Using EMR Data and an Out-of-Plan CAM Use Survey
The first phase of the project involves constructing a sample of HMO members with CMP over a five-year period (1/01/2006-12/31/2010). One difference between EMR-based and RCT-based research is the importance of the occurrence of medical events over time, in the absence of any research intervention. Each patient in the sample will have a virtual enrollment date, at which they first satisfied the CMP-eligibility criteria. All EMR data prior to that date correspond roughly to "baseline" data in an RCT, the primary difference being that the amount and duration of EMR-based events will almost certainly vary widely from patient to patient. As one moves forward from the virtual enrollment date, a given patient can be considered to be someone who uses A/C during that time period if they initiate A/C visits, or not if they rely solely on conventional care. Consequently, whether a patient is considered as treated (with A/C) or "control" (without A/C) depends on the time period over which outcome events are accumulated. This feature substantially complicates the analysis, above and beyond the absence of a formal intervention. Those who receive A/C treatment or are identified as "control" subjects can then be compared on clinical care outcomes and costs derived from EMR data.
Phase 1 includes a focus on analytic innovations that capitalize on our very large data set (122, 896 health plan members with EMR-identified CMP from 2006 to 2010) to describe pain-related services. We will also explore and compare outcomes found through multiple methods for identifying suitable comparable groups of patients (those who did and did not receive A/C services) and best controlling for potential confounders between the groups. This approach is consistent with recommendations for observational studies to employ multiple analytic strategies to ensure consistency of findings . Reviewed below are our methods for organizing our data and creating descriptions of trajectories of pain care services (event-stream methods) followed by the three principal analytic approaches to be employed to match participants receiving A/C to suitable controls - selection models, propensity score analysis, and the analysis of matched comparison groups. These analytic methods will be oriented toward reduction of indication bias.
Management of longitudinal patient records is difficult because patients receive services in diverse ways. Unlike clinical trials in which all measures are collected in the same time frame, clinical records include patients who visit the health plan at different times for different services, each displaying a unique, idiographic "trajectory of care." For CMP patients, for example, this might include use of patient education, physical therapy, and opiate medications over a particular period of time. The event-stream method allows a richer description of patient care over time, more consistent with how patients receive services in routine care. The method defines an efficient data interface between the clinical/administrative databases and the analysis datasets [61, 62]. Event-stream methods can be used to construct the types of variables that can then be used in a conventional analysis. This approach simplifies both the data extraction and statistical analysis and allows a richer array of variables for identifying matches or clusters of patients.
To compare CMP patients who are users and non-users of A/C care on their health care utilization, we will need to adjust for features that might be associated with both the use of CAM (A/C care) and health care utilization overall. One of the challenges the project will address will be to determine the potential confounding factors from the EMR data alone. The chief threat to the validity of clinical observational studies is indication bias, in which factors that predict clinical outcomes are associated with treatment choices. This problem is not unique to biomedical research, and in fact the foundational development of selection models was carried out in economics . Therefore, Heckman's selection models will be used to produce analyses based on joint estimation in a regression model (for the outcome) and a logistic regression model (for the treatment selection), which permits these two processes to be linked. Of all the approaches to be used in this project, this is the one that most closely mirrors statistical analysis in a conventional clinical trial.
Propensity score analysis
Phase 1: Variables available from EMR for Clustering/Matching Patients
Types/duration of pain condition(s)
Comorbid conditions (e.g., depression, obesity)
Demographics (age, gender, race, marital status, employment status)
Receipt of pain-related disability compensation
Overall healthcare service use
Prescription medications received (e.g., short and long-acting opiates, sedatives, hypnotics, and antidepressants)
Pain-related procedures received (injections, imaging, and surgeries)
Pain-related visits (primary and specialty care including physical therapy)
ER/urgent care visits
Prior use of health insurance reimbursed CAM
Characteristics of primary care provider (e.g., degree [MD, DO, NP] frequency of CAM referrals, gender)
Analysis of Matched Comparison Groups (MCG)
Phase 1: EMR-derived Clinical Outcomes
Overall use of health care services
Medication use (opiates, sedatives, hypnotics, and antidepressants)
Pain-related procedures (injections, imaging, surgery)
Pain-related visits (primary and specialty care)
Emergency room/Urgent care visits
Evidence of sleep problems
Adverse events attributed to conventional, acupuncture, or chiropractic care treatment for chronic musculoskeletal pain
Mood disorder diagnoses (depression, anxiety)
In identifying MCGs, we will use all CMP patient EMR data collected over a 5-year span (2006-2010). Patients must have a long enough duration of consecutive health plan membership to establish CMP eligibility and evaluate outcomes over the following year. To ensure maximum comparability, eligible patients must have clinical data available over a period when the A/C referral patterns were highly comparable. For each analytic approach, for each outcome measure, an A/C effect (comparing A/C and non-A/C patients) will be computed for each informative MCG. Table 4 shows the EMR-derived clinical outcomes that we plan to examine. We will employ conventional between-group estimates of the A/C effect measures. We anticipate, however, that there may be substantial heterogeneity of A/C effects across MCGs, so that it will be important to characterize where A/C has a beneficial effect, no effect, or an adverse effect. We will use both graphical methods as well as clustering of MCGs. We will also use explanatory linear models, usually ordinary regression, but potentially variants, depending on the specific type of outcome.
Multiple levels of analysis
In clinical trials, it is appropriate to pre-specify the main analyses; however, in clinical non-intervention research, the reverse is the case. As emphasized in Guo and Fraser , because each analytic approach has its own strengths and weaknesses, it is essential to carry out a multiplicity of different analyses and assess their potentially conflicting results in the context of broad knowledge about clinical processes and patient behaviors. This virtually guarantees that reports of research results from non-intervention studies will be more lengthy and complex than their clinical trial counterparts.
Sample size considerations
Unlike an RCT, at this stage of EMR-based research we do not have simple, prespecified hypotheses to test. Furthermore, in the light of the desirability of carrying out multiple types of analyses with different endpoints and analysis variants, it is not feasible to make standard sample size computations. Instead, study justification rests on the very large pool of eligible patients (122, 896 health plan members with EMR-identified CMP from 2006 to 2010) and analysis relies on precise estimation of potential effects, rather than on statistical decision-making. Moreover, part of the purpose of an EMR-based study is to portray the variability of response patterns at the individual patient level (as opposed to an average result for the entire sample) to the extent that this is observable. Here again, statistical decision-making does not seem to be an optimal approach.
Survey of A/C use by CMP patients
Our first-step Phase 1 analyses are confined to variables available in the EMR, which we will use to develop a broadly replicable and efficient approach to evaluating A/C impact. However, patients may be using A/C and other CAM therapies in ways not visible within the EMR. Similarly, there may be patient characteristics contributing to the decision to use A/C therapies that are also not readily visible in the EMR. To address these issues, we will identify patients whose EMR records suggest the presence of CMP within the past two years; we will then invite them to complete a survey identifying current and prior out-of-plan A/C use and use of related treatments/practices. A copy of the survey is available from the first author. We will then replicate the Phase 1 procedures (described above) with the enhanced data set including only those individuals who responded to the survey. This will allow us to evaluate the robustness of our findings by comparing the MCG characteristics identified using only EMR data with those identified in this augmented data set. Additional insights and characteristics predictive of out-of-plan use will help guide sampling of CMP participants for the focus groups in Phase 2, as well as the MCG design for matching participants in Phase 3. Data on out-of-plan use will be most accurate for current use patterns, which are most important for Phases 2 and 3. There will still be "noise" in this data for two reasons: first, not all CMP patients will respond to the survey; second, retrospective self-reporting may not allow us to determine with certainty whether self-care practices or out-of-plan CAM use preceded receipt of covered A/C care. Nonetheless, reporting such practices indicates a patient's general proclivity to seek CAM services or engage in self-care practices to address health concerns. Information on this tendency may enhance our interpretation of patient decision-making and outcomes.
Phase 2: Describing the CAM Decision-Making Process Using Qualitative Methods
The second phase of the project will use qualitative research methods to achieve three goals: first, to identify characteristics of A/C received by CMP patients through A/C provider interviews; second, to explore patients' and allopathic providers' decision-making processes in choosing to use/recommend A/C; and third, to identify important outcomes and factors associated with patient and provider satisfaction. Each of these components will inform the construction of instruments for Phase 3.
For this study phase, we will use qualitative methods because they can elicit the participants' perspectives in defining the range and variability of beliefs, behaviors, and experiences, all within the context of the natural language people use to discuss these issues. Qualitative methods will also allow us to discover themes not anticipated by study investigators. Standardized approaches [68–70] will guide research activities. Our data collection methods will include focus group interviews [71–74] and in-depth interviews [68, 75, 76], analyzed using a content analysis approach .
Through a descriptive case-study approach, we will investigate the full range of details of A/C use, including processes by which allopathic providers decide to make A/C referrals, how CMP patients decide to use such services, and how A/C providers decide the components of care to deliver. The primary questions that guide this case-study research are: (1) What are the barriers and facilitators to seeking/referring A/C for CMP patients? (2) What are the nature and extent of the services received from acupuncturists and chiropractors (i.e., services broader than needling for acupuncture and spinal manipulation for chiropractic care)? and (3) What are the clinical, functional, and quality-of-life outcomes patients and providers associate with receipt of A/C, and which processes or outcomes are most closely associated with patient and/or provider satisfaction?
Phase 2 qualitative data will be collected in two waves: we will use information from the first wave to refine the questions asked during the second wave interviews and focus groups and, ultimately, to collect data on potential confounders (i.e., patient factors related to decisions to use A/C services not reflected in EMR data) for use in Phase 4 analyses.
Focus groups with patient participants
The following types of patients will be represented among those we interview: (1) patients who were clinician-referred and received A/C services; (2) patients who received A/C without a clinician referral; and (3) comparable CMP patients who have not received A/C (including those referred who did not follow through). We will use data collected through Phase 1 EMR analyses and out-of-plan surveys to identify these types of patients for sampling. We anticipate holding multiple focus groups for each category of participants. Those who received acupuncture and chiropractic care will be recruited for separate focus groups.
Focus groups are an efficient mechanism for investigating how people conceptualize, experience, and talk about issues, for examining a range or consensus of experiences, and for collecting qualitative data from many individuals in a short time. Focus groups will be audio-recorded and consist of 6-12 participants, an optimal size range for discussing behaviors and experiences . Each patient focus group will address the three main topics described above. As part of that process, we will explore two additional issues related to the decision-making process as shown in the model in Figure 1: factors that prompt individuals to seek and, once initiated, continue or discontinue A/C care, and reasons for deciding to seek care with or without primary care provider concurrence. The first wave focus group results will be used to refine the draft Phase 3 questionnaires. Second wave focus groups of prior A/C users will be asked to complete the draft questionnaires prior to focus group attendance and be prepared to advise researchers on domains or questions relevant to their use of A/C services not covered in the questionnaire, as well as to comment on their understanding of the questions we designed. We will not alter questions that are part of validated instruments, but we will use the feedback to refine our overall assessment battery and to adjust items that we have created based on first-wave focus groups.
Health care provider participants
In in-depth interviews, researchers will introduce a series of prepared, open-ended questions designed to elicit factual information on treatment (A/C providers) and referral (allopathic providers) practices, as well as providers' knowledge, beliefs, attitudes, and experiences relating to A/C for CMP conditions, including what they would perceive to be clinically meaningful improvement. An interview guide, ongoing QA meetings, and transcript reviews will assure uniformity of approach and coverage in the interviews. We will conduct in-depth interviews with acupuncturists and chiropractors who see a high volume of CMP patients from the health plan to learn about their treatment styles and decision-making processes for CMP patients as well as their experiences with KPNW members who are physician-referred or self-referred. We will also interview health plan providers, sampling from among those with high numbers of A/C referrals, those with some A/C referral history, and those who never or almost never refer CMP patients for acupuncture or chiropractic services.
Qualitative data analysis
Audio recordings of all focus groups and interviews will be transcribed verbatim using standardized transcription protocols. Using content analysis techniques, we will analyze transcripts and field notes [68, 77, 78] in two stages. First, we will use topical indexing to identify text pertaining to each interview question and prompt, followed by development of a more detailed coding scheme to capture content, themes, and sentiments. Data management and reduction will be supported by use of ATLAS text analysis software (Sage-Scolari, Thousand Oaks, CA). We will compare participant responses within and across categories to identify beliefs, attitudes, behaviors, and experiences important to understanding decisions to seek and continue A/C care and outcomes associated with A/C care for CMP. We will analyze response frequency and content and the vocabulary describing concepts and experiences. This will allow identification of key issues by exploring areas of consensus and contradiction within and across focus group participants and provider respondents [68, 79]. Potential bias will be minimized and data credibility enhanced through comprehensive training of interviewers and coders, and multiple members of the research team will complete each type of task. Coder reliability will be determined through check-coding of 1 out of every 6 interviews and focus groups. Because ATLAS can interface with statistical programs, participant characteristics and responses to quantitative questions (e.g., questionnaire draft) can be included in the database and will enable us to retrieve text specific to respondent subgroups.
Phase 3: Prospective Cohort Study to Evaluate Outcomes Associated with the Use of A/C Services
Set up a weekly surveillance tracking system to identify CMP patients referred for A/C care to allow timely recruitment of these (index) patients and two carefully matched CMP control patients.
Coordinate telephone or online assessments for index and comparison patients upon recruitment (baseline; prior to A/C care) and at 1-, 3-, 6-, and 12-month follow-ups.
Compare outcomes (clinical and psychosocial/QOL/satisfaction outcomes and health care utilization) and explore health care costs between those receiving A/C and comparison patients not receiving such services, examining relationships between patient characteristics and decisions to use A/C.
Conduct comparative and sensitivity analyses between results using methods that rely solely on EMR-derived data (Phase 1) with those supplemented by patient and provider data (Phase 3).
Phase 3 participants and recruitment
Index participants (i.e., those with an incident A/C referral, n = 200 acupuncture and 200 chiropractic) and matched comparison participants in this prospective portion of the study must meet the overall CMP eligibility criteria listed in Table 1 and be able to read and respond to assessment questionnaires. Both index and comparison patients must have had no A/C treatment over the previous six months to ensure that we are prospectively following a new episode of A/C care. We purposefully minimized exclusion criteria to ensure that study findings are as broadly representative of the CMP population as possible.
Health plan members will be recruited over a two-year period through a weekly EMR review to identify CMP patients referred for A/C. Those referred for A/C will be mailed an invitation to the study, including a brochure and letter describing the study, and indicating that a staff member will call soon to tell them more about the study. There will also be a study website where prospective participants can log on to learn more about the study and eligibility or opt out if they so choose. Eligible participants will be guided through the consent process and baseline assessments by an interviewer or online, according to their preferences.
Identification of comparison participants
To recruit comparison cases for each participant in our cohort of A/C users, we will identify CMP-patient matches using the clustering/matching approach determined from Phase 1 and refined based on Phase 2 findings. As with Phase 1, participants must have a primary care visit within one month of the index patients' A/C referral so that we know that an opportunity for A/C referral existed. When an index patient enrolls online, the matching process is triggered: those matches will be located, mailed the same informational materials, and invited to enroll online. We will recruit two or more comparison participants for every index participant. A greater number of comparison participants allows a buffer of comparison participants in the event that some report A/C use during the assessment window.
Outcome measures and study instruments
Phase 3 Assessment Schedule
Chronic Musculoskeletal Pain Patient Measures:
Pain and related disability (patient-reported, primary outcomes)
Pain Severity (BPI-SF subscale)
Pain Interference (BPI-SF subscale)
Pain Bothersomeness (single item)
Secondary Outcomes (survey instruments)
Overall well-being (Arizona Integrative Outcomes Scale)
Patient global impression of change
Quality of sleep (ISI)
Work- and activity-related impairment (NHIS questions)
Quality of Life/health utility index (SF-12)
Secondary outcomes (health care utilization collected through EMR and administrative records)
Overall use of healthcare services and associated costs
Use of medications (short and long-acting opiates, sedatives, hypnotics, antidepressants)
Pain-related procedures (injections and imaging)
Pain-related visits (primary and specialty care)
ER/urgent care visits
Adverse events associated with conventional and CAM CMP treatment3
Use of health insurance reimbursed CAM
Patient characteristics and potential moderators (including predisposing, enabling, and need factors associated with A/C use decision from Figure 1)
Received/seeking pain-related disability compensation
Patient expectations/CAM attitudes
Other characteristics of pain condition (type(s)/duration)
Previous use of acupuncture/chiropractic services (adapted from NHIS survey)
Utilization of non-plan CAM health services & products
Healthcare Provider (Allopaths/Acupuncturists/Chiropractors) Measures:
Allopathic provider characteristics and CAM beliefs
Acupuncturist/chiropractic service provided & general practices
Primary outcomes: Pain and related disability
Two subscales--the 4-item pain severity and the 7-item pain interference subscales--from the short form of the Brief Pain Inventory (BPI-SF)[84, 85] will be used to assess pain and related disability. The BPI has sound psychometrics and has been widely adopted for clinical pain assessment, epidemiological studies, and studies of treatment efficacy. We will also include a measure of how bothered participants are by their pain. This instrument uses a 0 to 10 scale of "symptom bothersomeness, " where 0 represents "not at all bothersome" and 10 is "extremely bothersome." This question has been frequently used in studies of back pain [18, 19, 86] and shown to have adequate construct validity .
Secondary outcomes (patient-reported)
Global assessment of change. The Patient Global Impression of Change scale (PGIC ) will be used to evaluate participants' overall evaluation of the impact of their A/C treatment.
Work-related impairment. Three questions about work/disability status and missed work due to CMP will be adapted from the National Health Information Survey (NHIS) .
Depression severity. Using the Personal Health Questionnaire (PHQ-8) [90, 91], we will measure depression severity. The PHQ-8 is established as a valid screener and severity measure for depressive disorders in large clinical studies [90, 92–94]. Baseline depression will also be examined as a possible moderator of outcome.
Anxiety. The 2-item Generalized Anxiety Disorder Scale (GAD-2) [95, 96] will be used to screen for anxiety disorders. The GAD-2 has been used to detect anxiety disorders in primary care and has well established psychometric properties. Baseline GAD-2 scores will also be examined as another possible moderator of outcome.
Quality of life/health utility. The SF Health Survey (SF-12v2)  will be used to measure quality of life. The SF12 is a widely used and well-validated generic measure of functioning that is a recommended IMMPACT measure for examining functioning (this will also be used as a health utility measure) .
Patient satisfaction. Finally, participant satisfaction with services provided by the health plan (allopathic providers) (0 = worst health care possible, 10 = best health care possible) will be assessed as well as satisfaction with A/C provider when pertinent . We will also use the practitioner skill subscale from a self-report outcome measure designed for assessing CAM treatments [99, 100].
Secondary outcomes (health care utilization through EMR and administrative records)
These EMR-based outcome measures were listed in Table 4 as they also form the basis for our Phase 1 outcome analyses.
Patient characteristics and potential moderators include factors that may influence decisions to use CAM (see Figure 1) or that have been found to moderate pain-related outcomes [102–104]. These factors might then be used as additional baseline matching variables or covariates to maximize comparability between index- and matched comparison-participants included in the analyses. Demographics include age, gender, race/ethnicity, education, marital status, employment status, income, and income source. Administrative records will identify participants seeking or receiving worker's compensation or other disability payments for their CMP. This will be augmented by a single-item question about disability status of participants on the questionnaire. We will assess general preventive behavior [105, 106] by collecting data on self-management behaviors including physical activity practices. In addition, we will assess participants' general expectations of CMP improvement, improvements with A/C treatment [27, 31], and how helpful a variety of CAM services and practices (provider- and self-directed) may be . We will measure other characteristics of the participant's pain condition, including age of onset, perceived etiology of CMP, and type(s) of current CMP. Finally, we will collect data on use of other CAM services/products, including any CAM treatment visits not referred through the health plan (e.g., chiropractic and acupuncture, massage therapists, naturopaths) as well as CAM-related self-care practices (e.g., yoga, meditation, tai chi) and products (e.g., dietary supplements).
Health care provider measures
Allopathic provider characteristics and factors associated with A/C referrals including questions describing general provider characteristics (i.e., gender, specialty type, professional degree, years in practice, and training/expertise in CMP) and A/C referral attitudes, as well as an EMR-based variable reflecting the inclination for a given provider to refer for A/C services (referral rate per CMP patients on panel).
A/C provider characteristics and treatment patterns include questions adapted from the A/C-visit data forms developed by Sherman and colleagues  and refined during our Phase 2 interviews with the CAM providers. We also have access to data collected by the health plan on CAM-referred services patients receive (e.g., number of sessions; assigned diagnostic, procedure, and treatment [CPT] codes.
Phase 3 Data Analysis
For the patient-reported outcomes in Phase 3, we will examine differences in baseline measures between participants with and without data at the 1-, 3-, 6-, and 12-month follow-ups. Variables related to missingness will be included in the main analyses as covariates to reduce bias in the estimates [109, 110].
We will use multilevel modeling to examine differences across time in primary (pain intensity/interference/bothersomeness) and secondary patient-reported outcomes between A/C users and matched comparison participants. Two parameters (linear and quadratic slope) will characterize change across time, with linear slope capturing initial rate of change and quadratic slope reflecting the degree to which the change slowed (or increased) over time. Thus, the quadratic model for time (baseline and 1-, 3-, 6-, and 12-month follow-ups) captures nonlinear change across time. Unlike repeated measures analysis of variance, multilevel modeling does not require data at all time points from each subject, so data from all subjects will be included in analyses. Depending upon the distribution, normal, logistic, or Poisson models will be used. Importantly, these analyses can be adjusted using additional information collected from participants during the baseline assessment that we may identify in Phase 2 as influencing A/C decision-making. This will better control for factors not available during comparison patient identification when only EMR data will be available for matching/clustering.
To examine total health care utilization across 12 months, the analysis issues laid out for Phase 1 continue to apply. Using the MCG as the unit of analysis, the conventional statistical methods from Phase 1 continuous measures will be assessed by one-sample location tests and discrete outcomes will be assessed by binomial methods or by ordinal multinomial methods in the case of multiple outcomes. Like Phase 1, MCG characteristics in the Phase 3 sample will be related to A/C effects using the models described above. Implementing the same analytic strategy here as in Phase 1 (i.e., using EMR data alone), we will be able to test directly for the contribution provided by the additional prospective information on patient-and-provider characteristics (e.g., pain severity, self-care behaviors, provider attitudes to A/C referral). We will do so by including patient-supplied variables at the patient level in the analytic model and then introducing random-effects terms for the MCGs to account for within-MCG correlation.
Health plan costs are one of our secondary outcomes, as we are interested in how total health plan costs differed between A/C users and nonusers. Costs will be estimated by applying internal unit costs developed and tested in previous studies [111, 112] to the HMO patient-level utilization measures, with the final cost variables acting as proxies for HMO resource cost. We are interested in the effect of patient-level factors--in particular, patients' use (or not) of A/C services on health care costs. Admittedly, the challenges of analyzing typical cost data are well known and significant; these include 1) having a large proportion of non-users (i.e., zero costs); 2) the fact that non-zero costs are usually right-skewed; and 3) heteroscedasticity. Assuming these challenges exist in our study cost data, we will tackle them by exploring transformation models and generalized linear models (GLM). Transformation models convert skewed cost data using a transformation (e.g., log-normal) to a better-behaved distribution (i.e., one more normal and symmetric, homoscedastic, less skewed, promoting more efficient estimation). A complication of transformations is possible re-transformation bias in switching from the transformed scale to the original dollar-based scale. GLM avoids re-transformation bias but can be highly imprecise if the residual pattern is misspecified. We will explore both model forms to determine the most accurate and descriptive approach for our study data. Cost analyses will be performed in STATA version 11.
Phase 3 Sample Size
Sample and Effect Sizes for Related and Current Study
Cherkin, 2009 
Cherkin, 2009 
Liang, 2011 
chronic neck pain
Franca, 2008 
acu v PT
Molsberger, 2010 
acu v UC
Hondras, 2009 
LBP > 55 yrs
chiro v UC
Rx/self care v UC
0.56 (BPI-S) 0.59 (BPI-I)
acu v UC
BPI-S & BPI-I
chiro v UC
BPI-S & BPI-I
Written consent will be obtained from each participant in Phases 2 and 3 and from those completing the out-of-plan A/C survey in Phase 1. For the data-only portion of Phase 1, health plan members' contracts with the HMO provide consent for use of their data in HMO-sponsored research studies. This study protocol was approved by the institutional review boards (IRB) of Kaiser Permanente Northwest, University of Arizona, and Oregon Health and Science University. This study protocol is registered with http://www.clinicaltrials.gov (NCT01345409).
We expect that successful completion of this study will provide an evaluation of the outcomes--from the perspectives of patients, providers, and health services utilization--associated with the real-world use of A/C services. By comparing the results of the EMR-only analyses with the more comprehensive results available in Phase 3, we will be able to identify the strengths and weaknesses of EMR-alone analyses and potentially identify the most critical additional data needed. We further anticipate that the trio of EMR-based analyses, qualitative assessment, and prospective cohort studies will provide a clearer understanding of the decision-making processes behind the use of A/C for CMP; these will also provide a transportable methodology that can be applied to other health care settings, CAM treatments, and clinical populations.
With regard to advancing our understanding of patients' use of CAM in everyday practice settings, much CAM research to date has focused narrowly on "early adopters" (those likely to use CAM services regardless of barriers ). Yet within our cohort, cost and access barriers are substantially reduced, allowing us to study other factors (see Figure 1) influencing A/C care receipt and outcomes and to develop a comprehensive characterization of A/C use patterns and outcomes in patients with increasing access to such CAM services. An important goal of this study is to clarify how CMP patients use A/C compared to conventional care (to augment or replace it), when and why patients disclose A/C use to allopathic providers, and how such practices may vary by patient characteristics. Our mixed methods approach will help us develop a better understanding of the process behind the decision to initiate (patients), recommend (allopathic providers), and continue (patients and providers) A/C care for the treatment of CMP. Finally, studying this in everyday treatment settings allows us to consider the range of services received in A/C care (e.g., needling or spinal manipulation vs. whole systems-based care) and variation by characteristics of the CMP patient and/or A/C provider.
In addition to advancing our understanding of real world A/C practices and their impact on CMP, this study affords us the opportunity to build upon the increasingly available EMR systems to conduct research that may be more broadly adopted to better understand the real world interplay of conventional medicine and A/C services. Using EMRs exclusively may limit information on important variables (covariates) on which to match or cluster like patients. However, by augmenting data sets through direct reports by providers and patients regarding important decision-making factors suggested by qualitative findings, we may substantially reduce potential confounding in comparing those who do and do not receive A/C treatment. Thus, we can identify whether widely available data in EMR and administrative systems can be used to adequately forecast patients' outcomes and evaluate the extent to which supplementation from additional patient- and provider-collected information is a necessary adjunct to EMR-based data analysis.
Finally, our MCG-focused methodology gives us a unique opportunity to conduct patient-centered research; that is, to identify like groups based upon actual individual patient care characteristics rather than hypothetical group averages. A typical RCT may not include enough participants to form homogeneous clusters for separate evaluation, yet an EMR database of a large clinical population like the CMP sample described here does include sufficient numbers of patients with similar characteristics and disease severity to form clinically meaningful clusters as the basis of the analyses. Accordingly, this research may allow us to address the clinical question as follows: for whom does supplementary CAM care result in clinically significant improvements in pain severity and overall functioning? Our planned MCG methodology will allow us to identify the characteristics of CMP patients for whom A/C may have the greatest impact on satisfaction, functioning, clinical, and QOL outcomes; it will also clarify whether the timing of services relative to conventional care and the course of the CMP affects such outcomes.
List of abbreviations
complementary and alternative medicine
chronic musculoskeletal pain
health maintenance organization
Kaiser Permanente Northwest
electronic medical record
International Statistical Classification of Diseases - 9th edition codes
current procedural terminology codes
matched comparison groups
general linear models
randomized clinical trial
doctor of osteopathic medicine
Initiative on Methods, Measurement, and Pain Assessment in Clinical Trials
Brief Pain Inventory-Short Form
Brief Pain Inventory Severity Subscale
Brief Pain Inventory Interference Subscale
Patient Global Impression of Change
National Health Information Survey
Personal Health Questionnaire-8 item version
Generalized Anxiety Disorder-2 item version
Roland Morris Disability Questionnaire
Northwick Park Neck Pain Questionnaire
Acknowledgements and funding
The authors gratefully acknowledge funding support for this study from the National Institutes of Health, National Center for Complementary and Alternative Medicine Grant (R01 AT005896). In addition at the time this article was prepared, co-author, John Dickerson, was funded in part from a training grant from the Agency for Healthcare Research and Quality (T32HS013853). We thank Dana Foley for her helpful comments on previous versions of this article. Richard Hammerschlag and Mitch Haas provided important feedback on study design. In addition, Chris Catlin, Stephanie Hertert, Gail Morgan, Matt Hornbrook, Maureen O'Keefe-Rosetti, Lin Neighbors, and Lisa Fox have all contributed invaluable support for the early phases of the study.
- Institute of Medicine: Relieving Pain in America: A Blueprint for Transforming Prevention, Care, Education and Research. 2011, Washington DC, The National Academies Press
- Jensen MP, Chodroff MJ, Dworkin RH: The impact of neuropathic pain on health-related quality of life: review and implications. Neurology. 2007, 68: 1178-1182. 10.1212/01.wnl.0000259085.61898.9e.PubMed
- Verhaak PF, Kerssens JJ, Dekker J, Sorbi MJ, Bensing JM: Prevalence of chronic benign pain disorder among adults: a review of the literature. Pain. 1998, 77: 231-239. 10.1016/S0304-3959(98)00117-1.PubMed
- Goldenberg DL, Burckhardt C, Crofford L: Management of fibromyalgia syndrome. JAMA. 2004, 292: 2388-2395. 10.1001/jama.292.19.2388.PubMed
- Gerdle B, Bjork J, Coster L, Henriksson K, Henriksson C, Bengtsson A: Prevalence of widespread pain and associations with work status: a population study. BMC Musculoskelet Disord. 2008, 9: 102-10.1186/1471-2474-9-102.PubMed CentralPubMed
- Manek NJ, MacGregor AJ: Epidemiology of back disorders: prevalence, risk factors, and prognosis. Curr Opin Rheumatol. 2005, 17: 134-140.PubMed
- Maetzel A, Li L: The economic burden of low back pain: a review of studies published between 1996 and 2001. Best Pract Res Clin Rheumatol. 2002, 16: 23-30. 10.1053/berh.2001.0204.PubMed
- Kaiser Permanente Northwest Clinical Practice Guidelines: Low Back Problems and Lumbar Radiculopathy. An Evidence-Based Clinical Practice Guideline. 2004
- Deyo RA, Mirza SK, Turner JA, Martin BI: Overtreating chronic back pain: time to back off?. J Am Board Fam Med. 2009, 22: 62-68. 10.3122/jabfm.2009.01.080102.PubMed CentralPubMed
- Deyo RA, Mirza SK, Martin BI, Kreuter W, Goodman DC, Jarvik JG: Trends, major medical complications, and charges associated with surgery for lumbar spinal stenosis in older adults. JAMA. 2010, 303: 1259-1265. 10.1001/jama.2010.338.PubMed CentralPubMed
- Kuehn BM: Opioid prescriptions soar: increase in legitimate use as well as abuse. JAMA. 2007, 297: 249-251. 10.1001/jama.297.3.249.PubMed
- Kalso E, Edwards JE, Moore RA, McQuay HJ: Opioids in chronic non-cancer pain: systematic review of efficacy and safety. Pain. 2004, 112: 372-380. 10.1016/j.pain.2004.09.019.PubMed
- Barnes PM, Bloom B, Nahin RL: Complementary and alternative medicine use among adults and children: United States, 2007. Natl Health Stat Report. 2008, 1-23.
- Institute of Medicine: Complementary and Alternative Medicine in the United States. 2005, Washington, D.C., The National Academies Press
- National Center for Complementary and Alternative Medicine: Exploring the Science of Complementary and Alternative Medicine: Third Strategic Plan 2011-2015. 2011
- Levine SM, Weber-Levine ML, Mayberry RM: Complementary and alternative medical practices: training, experience, and attitudes of a primary care medical school faculty. J Am Board Fam Pract. 2003, 16: 318-326. 10.3122/jabfm.16.4.318.PubMed
- Berman BM, Singh BB, Hartnoll SM, Singh BK, Reilly D: Primary care physicians and complementary-alternative medicine: training, attitudes, and practice patterns. J Am Board Fam Pract. 1998, 11: 272-281.PubMed
- Cherkin DC, Deyo RA, Battie M, Street J, Barlow W: A comparison of physical therapy, chiropractic manipulation, and provision of an educational booklet for the treatment of patients with low back pain. N Engl J Med. 1998, 339: 1021-1029. 10.1056/NEJM199810083391502.PubMed
- Cherkin DC, Sherman KJ, Avins AL, Erro JH, Ichikawa L, Barlow WE, Delaney K, Hawkes R, Hamilton L, Pressman A, Khalsa PS, Deyo RA: A randomized trial comparing acupuncture, simulated acupuncture, and usual care for chronic low back pain. Arch Intern Med. 2009, 169: 858-866. 10.1001/archinternmed.2009.65.PubMed CentralPubMed
- Hurwitz EL, Morgenstern H, Harber P, Kominski GF, Yu F, Adams AH: A randomized trial of chiropractic manipulation and mobilization for patients with neck pain: clinical outcomes from the UCLA neck-pain study. Am J Public Health. 2002, 92: 1634-1641. 10.2105/AJPH.92.10.1634.PubMed CentralPubMed
- Sherman KJ, Cherkin DC, Ichikawa L, Avins AL, Barlow WE, Khalsa PS, Deyo RA: Characteristics of patients with chronic back pain who benefit from acupuncture. BMC Musculoskelet Disord. 2009, 10: 114-10.1186/1471-2474-10-114.PubMed CentralPubMed
- Bunce V, Wieske JP: Health Insurance Mandates in the States. 2009, Alexandria, VA, Council for Affordable Health Insurance
- Cherkin DC, Deyo RA, Sherman KJ, Hart LG, Street JH, Hrbek A, Davis RB, Cramer E, Milliman B, Booker J, Mootz R, Barassi J, Kahn JR, Kaptchuk , Eisenberg DM: Characteristics of visits to licensed acupuncturists, chiropractors, massage therapists, and naturopathic physicians. J Am Board Fam Pract. 2002, 15: 463-472.PubMed
- Cherkin DC, Eisenberg D, Sherman KJ, Barlow W, Kaptchuk TJ, Street J, Deyo RA: Randomized trial comparing traditional Chinese medical acupuncture, therapeutic massage, and self-care education for chronic low back pain. Arch Intern Med. 2001, 161: 1081-1088. 10.1001/archinte.161.8.1081.PubMed
- Consumer Reports. Relief for aching backs: Hands-on therapies were top-rated by 14, 000 consumers. Consumer Reports May 2009, 12-13. 2009
- Sobel DS: MSJAMA: mind matters, money matters: the cost-effectiveness of mind/body medicine. JAMA. 2000, 284: 1705-10.1001/jama.284.13.1705.PubMed
- Linde K, Witt CM, Streng A, Weidenhammer W, Wagenpfeil S, Brinkhaus B, Willich SN, Melchart D: The impact of patient expectations on outcomes in four randomized controlled trials of acupuncture in patients with chronic pain. Pain. 2007, 128: 264-271. 10.1016/j.pain.2006.12.006.PubMed
- Chou R, Huffman LH: Nonpharmacologic therapies for acute and chronic low back pain: a review of the evidence for an American Pain Society/American College of Physicians clinical practice guideline. Ann Intern Med. 2007, 147: 492-504.PubMed
- Hurwitz EL, Morgenstern H, Kominski GF, Yu F, Chiang LM: A randomized trial of chiropractic and medical care for patients with low back pain: eighteen-month follow-up outcomes from the UCLA low back pain study. Spine (Phila Pa 1976). 2006, 31: 611-621. 10.1097/01.brs.0000202559.41193.b2.
- Haas M, Goldberg B, Aickin M, Ganger B, Attwood M: A practice-based study of patients with acute and chronic low back pain attending primary care and chiropractic physicians: two-week to 48-month follow-up. J Manipulative Physiol Ther. 2004, 27: 160-169. 10.1016/j.jmpt.2003.12.020.PubMed
- Myers SS, Phillips RS, Davis RB, Cherkin DC, Legedza A, Kaptchuk TJ, Hrbek A, Buring JE, Post D, Connelly MT, Eisenberg DM: Patient expectations as predictors of outcome in patients with acute low back pain. J Gen Intern Med. 2008, 23: 148-153. 10.1007/s11606-007-0460-5.PubMed CentralPubMed
- Ernst E: Manual therapies for pain control: chiropractic and massage. Clin J Pain. 2004, 20: 8-12. 10.1097/00002508-200401000-00003.PubMed
- Mior S: Manipulation and mobilization in the treatment of chronic pain. Clin J Pain. 2001, 17: S70-S76. 10.1097/00002508-200112001-00015.PubMed
- Bell IR, Caspi O, Schwartz GE, Grant KL, Gaudet TW, Rychener D, Maizes V, Weil A: Integrative medicine and systemic outcomes research: issues in the emergence of a new model for primary health care. Arch Intern Med. 2002, 162: 133-140. 10.1001/archinte.162.2.133.PubMed
- Nahin RL, Pontzer CH, Chesney MA: Racing toward the integration of complementary and alternative medicine: a marathon or a sprint?. Health Aff (Millwood). 2005, 24: 991-993. 10.1377/hlthaff.24.4.991.
- Kaptchuk TJ, Miller FG: Viewpoint: what is the best and most ethical model for the relationship between mainstream and alternative medicine: opposition, integration, or pluralism?. Academic Medicine. 2005, 80: 286-290. 10.1097/00001888-200503000-00015.PubMed
- Leckridge B: The future of complementary and alternative medicine--models of integration. Journal of Alternative & Complementary Medicine. 2004, 10: 413-416. 10.1089/107555304323062446.
- Caspi O, Sechrest L, Pitluk HC, Marshall CL, Bell IR, Nichter M: On the definition of complementary, alternative, and integrative medicine: societal mega-stereotypes vs. the patients' perspectives. Alternative Therapies in Health & Medicine. 2003, 9: 58-62.
- McCaffrey AM, Pugh GF, O'Connor BB: Understanding patient preference for integrative medical care: results from patient focus groups. J Gen Intern Med. 2007, 22: 1500-1505. 10.1007/s11606-007-0302-5.PubMed CentralPubMed
- Ben-Arye E, Frenkel M, Klein A, Scharf M: Attitudes toward integration of complementary and alternative medicine in primary care: perspectives of patients, physicians and complementary practitioners. Patient Educ Couns. 2008, 70: 395-402. 10.1016/j.pec.2007.11.019.PubMed
- Andersen RM: Revisiting the behavioral model and access to medical care: does it matter?. J Health Soc Behav. 1995, 36: 1-10. 10.2307/2137284.PubMed
- Andersen RM: National health surveys and the behavioral model of health services use. Med Care. 2008, 46: 647-653. 10.1097/MLR.0b013e31817a835d.PubMed
- Fouladbakhsh JM, Stommel M: Using the Behavioral Model for Complementary and alternative Medicine: The CAM Healthcare Model. Journal of Complementary and Integrative Medicine. 2007, 4: 1-19.
- Calvin A, Becker H, Biering P, Grobe S: Measuring patient opinion of pain management. J Pain Symptom Manage. 1999, 18: 17-26. 10.1016/S0885-3924(99)00025-1.PubMed
- Lazarus H, Neumann CJ: Assessing undertreatment of pain: The patients' perspective. Journal of Pharmaceutical Care in Pain & Symptom Control. 2001, 9: 5-34.
- Berki SE, Ashcraft ML: On the analysis of ambulatory utilization: an investigation of the roles of need, access and price as predictors of illness and preventive visits. Med Care. 1979, 17: 1163-1181. 10.1097/00005650-197912000-00001.PubMed
- Muller C: Review of twenty years of research on medical care utilization. Health Serv Res. 1986, 21: 129-144.PubMed CentralPubMed
- Verbrugge LM, Patrick DL: Seven chronic conditions: their impact on US adults' activity levels and use of medical services. Am J Public Health. 1995, 85: 173-182. 10.2105/AJPH.85.2.173.PubMed CentralPubMed
- Freeborn DK, Pope CR: Promise & performance in managed care: The prepaid group practice model. 1994, Baltimore: The Johns Hopkins University Press
- Lawrence RS, Jonas S: Ambulatory care. Health Care Delivery in the United States. Edited by: Kovner AR. 1990, New York, New York: SPringer Publishing Company, Fourth
- Institute of Medicine: Initial National Priorities for Comparative Effectiveness Research. 2009, Washington, DC, National Academies Press
- Mushlin AI, Ghomrawi H: Health care reform and the need for comparative-effectiveness research. N Engl J Med. 2010, 362: e6-10.1056/NEJMp0912651.PubMed
- Weinstein MC, Skinner JA: Comparative effectiveness and health care spending--implications for reform. N Engl J Med. 2010, 362: 460-465. 10.1056/NEJMsb0911104.PubMed CentralPubMed
- Frey BJ, Dueck D: Clustering by passing messages between data points. Science. 2007, 315: 972-976. 10.1126/science.1136800.PubMed
- Rubin DB: The design versus the analysis of observational studies for causal effects: parallels with the design of randomized trials. Stat Med. 2007, 26: 20-36. 10.1002/sim.2739.PubMed
- Griswold ME, Localio AR, Mulrow C: Propensity score adjustment with multilevel data: setting your sites on decreasing selection bias. Ann Intern Med. 2010, 152: 393-395.PubMed
- Brookhart MA, Schneeweiss S: Preference-based instrumental variable methods for the estimation of treatment effects: assessing validity and interpreting results. Int J Biostat. 2007, 3: 14-PubMed Central
- Pizer SD: An intuitive review of methods for observational studies of comparative effectiveness. Health Serv Outcomes Res Method. 2009, 9: 54-68. 10.1007/s10742-009-0045-3.
- Elder C: Integrating CAM Into Practice: The KP Northwest Story. The Permanente Journal. 2002, 6: 57-59.
- Guo SY, Fraser MW: Propensity Score Analysis: Statistical Methods and Applications. 2010, Los Angeles CA: Sage Publications
- Vogt TM, Feldstein AC, Aickin M, Hu WR, Uchida AR: Electronic medical records and prevention quality: the prevention index. Am J Prev Med. 2007, 33: 291-296. 10.1016/j.amepre.2007.05.011.PubMed
- Vogt TM, Aickin M, Ahmed F, Schmidt M: The Prevention Index: using technology to improve quality assessment. Health Serv Res. 2004, 39: 511-530. 10.1111/j.1475-6773.2004.00242.x.PubMed CentralPubMed
- Heckman J: The common structure of statistical models of truncation, sample selection, and limited independent variables and a simple estimator for such models. The Annals of Economic and Social Measurement. 1976, 5: 475-492.
- Rosenbaum PR, Rubin DB: The central role of the propensity score in observational studies for causal effects. Biometrika. 1983, 70: 41-55. 10.1093/biomet/70.1.41.
- Rubin DB, Thomas N: Matching using estimated propensity scores: relating theory to practice. Biometrics. 1996, 52: 249-264. 10.2307/2533160.PubMed
- Rubin DB: Estimating causal effects from large data sets using propensity scores. Ann Intern Med. 1997, 127: 757-763.PubMed
- Leone M, Sumedha , Weigt M: Clustering by soft-constraint affinity propagation: applications to gene-expression data. Bioinformatics. 2007, 23: 2708-2715. 10.1093/bioinformatics/btm414.PubMed
- Bernard HR: Research Methods in Anthropology: Qualitative and Quantitative Approaches. 1994, Thousand Oaks, CA: Sage, 2
- Denzin NK, Lincoln YS: Handbook of qualitative research. 1994, Thousand Oaks, CA: Sage
- Sofaer S: Qualitative methods: what are they and why use them?. Health Serv Res. 1999, 34: 1101-1118.PubMed CentralPubMed
- Hughes D, DuMont K: Using focus groups to facilitate culturally anchored research. Am J Community Psychol. 1993, 21: 775-806. 10.1007/BF00942247.
- Krueger RA: Focus Groups: A practical guide for applied research. 1998, Newbury Park, CA: Sage
- Morgan DL: Practical strategies for combining qualitative and quantitative methods: applications to health research. Qual Health Res. 1998, 8: 362-376. 10.1177/104973239800800307.PubMed
- Stewart DW, Samdasani PN: Focus groups: theory and practice. 1990, Newbury Park, CA: Sage
- Briggs CL: Learning how to ask: A sociolinguistic appraisal of the role of the interview in social science research. 1986, Cambridge: Cambridge University Press
- Britten N: Qualitative interviews in medical research. British Medical Journal. 1995, 311: 251-253. 10.1136/bmj.311.6999.251.PubMed CentralPubMed
- Crabtree BF, Miller WL: A template approach to text analysis: Developing and using codebooks. Doing Qualitative Research. Edited by: Crabtree BF, Miller WL. 1993, Newbury Park, California: Sage Publications
- Morgan D: Successful Focus Groups: Advancing the State of the Art. 1993, Newbury Park, California: Sage Publications
- Spradley JP: Participant Observation. 1980, New York: Holt, Rinehart & Winston
- Witt C, Brinkhaus B, Jena S, Linde K, Streng A, Wagenpfeil S, Hemmelsberger J, Walther HU, Melchart D, Willich SN: Acupuncture in patients with osteoarthritis of the knee: a randomised trial. Lancet. 2005, 366: 136-143. 10.1016/S0140-6736(05)66871-7.PubMed
- Vickers AJ, Rees RW, Zollman CE, McCarney R, Smith CM, Ellis N, Fisher P, Van Haselen R: Acupuncture for chronic headache in primary care: large, pragmatic, randomised trial. BMJ. 2004, 328: 744-10.1136/bmj.38029.421863.EB.PubMed CentralPubMed
- Dworkin RH, Turk DC, Farrar JT, Haythornthwaite JA, Jensen MP, Katz NP, Kerns RD, Stucki G, Allen RR, Bellamy N, Carr DB, Chandler J, Cowan P, Dionne R, Galer BS, Hertz , Jadad AR, Krramer LD, Manning DC, Martin S, McCormick CG, McDermott MP, McGrath P, Quessy S, Rappaport BA, Robbins W, Robinson JP, Rothman M, Royal MA, Simon L, Stauffer JW, Stein W, Tollett J, Wernicke J, Witter J, IMMPACT: Core outcome measures for chronic pain clinical trials: IMMPACT recommendations. Pain. 2005, 113: 9-19. 10.1016/j.pain.2004.09.012.PubMed
- Turk DC, Dworkin RH, Allen RR, Bellamy N, Brandenburg N, Carr DB, Cleeland C, Dionne R, Farrar JT, Galer BS, Hewitt DJ, Jadad A, Katz NP, Kramer LD, Manning DC, McCormick CG, McDermott MP, McGrath P, Quessy S, Rappaport BA, Robinson JP, Royal MA, Simon L, Stauffer JW, Stein W, Tollett J, Witter J: Core outcome domains for chronic pain clinical trials: IMMPACT recommendations. Pain. 2003, 106: 337-345. 10.1016/j.pain.2003.08.001.PubMed
- Cleeland CS, Ryan KM: Pain assessment: global use of the Brief Pain Inventory. Ann Acad Med Singapore. 1994, 23: 129-138.PubMed
- Keller S, Bann CM, Dodd SL, Schein J, Mendoza TR, Cleeland CS: Validity of the brief pain inventory for use in documenting the outcomes of patients with noncancer pain. Clin J Pain. 2004, 20: 309-318. 10.1097/00002508-200409000-00005.PubMed
- Patrick DL, Deyo RA, Atlas SJ, Singer DE, Chapin A, Keller RB: Assessing health-related quality of life in patients with sciatica. Spine (Phila Pa 1976). 1995, 20: 1899-1908. 10.1097/00007632-199509000-00011.
- Dunn KM, Croft PR: Classification of low back pain in primary care: using "bothersomeness" to identify the most severe cases. Spine (Phila Pa 1976). 2005, 30: 1887-1892. 10.1097/01.brs.0000173900.46863.02.
- Guy W: Clinical Global Improvement Scale. Assessment Manual of Psychopharmacology. 1976, 76:
- National Center for Health Statistics: National Health Information Survey. 2010
- Kroenke K, Strine TW, Spitzer RL, Williams JB, Berry JT, Mokdad AH: The PHQ-8 as a measure of current depression in the general population. J Affect Disord. 2008
- Spitzer RL, Kroenke K, Williams JB: Validation and utility of a self-report version of PRIME-MD: the PHQ primary care study. Primary Care Evaluation of Mental Disorders. Patient Health Questionnaire. JAMA. 1999, 282: 1737-1744. 10.1001/jama.282.18.1737.PubMed
- Lowe B, Kroenke K, Herzog W, Grafe K: Measuring depression outcome with a brief self-report instrument: sensitivity to change of the Patient Health Questionnaire (PHQ-9). J Affect Disord. 2004, 81: 61-66. 10.1016/S0165-0327(03)00198-8.PubMed
- Lowe B, Unutzer J, Callahan CM, Perkins AJ, Kroenke K: Monitoring depression treatment outcomes with the patient health questionnaire-9. Med Care. 2004, 42: 1194-1201. 10.1097/00005650-200412000-00006.PubMed
- Kroenke K, Spitzer RL, Williams JB: The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med. 2001, 16: 606-613. 10.1046/j.1525-1497.2001.016009606.x.PubMed CentralPubMed
- Kroenke K, Spitzer RL, Williams JB, Monahan PO, Lowe B: Anxiety disorders in primary care: prevalence, impairment, comorbidity, and detection. Ann Intern Med. 2007, 146: 317-325.PubMed
- Kroenke K, Spitzer RL, Williams JB, Lowe B: An ultra-brief screening scale for anxiety and depression: the PHQ-4. Psychosomatics. 2009, 50: 613-621. 10.1176/appi.psy.50.6.613.PubMed
- Ware JA, Kosinski M, Keller SD: A 12-item short-form health survey: Construction of scales and preliminary tests of reliability and validity. Med Care. 1996, 34: 220-233. 10.1097/00005650-199603000-00003.PubMed
- AHRQ: CAHPS 2.0 Survey and Reporting Kit. 2002, Silver Springs, MD, Publications Clearinghouse
- Eton DT, Koffler K, Cella D, Eisenstein A, Astin JA, Pelletier KR, Riley D: Developing a self-report outcome measure for complementary and alternative medicine. Explore (NY). 2005, 1: 177-185. 10.1016/j.explore.2005.02.007.
- Eton DT, Temple LM, Koffler K: Pilot validation of a self-report outcome measure of complementary and alternative medicine. Explore (NY). 2007, 3: 592-599. 10.1016/j.explore.2007.08.004.
- Giordano J, Boatwright D, Stapleton S, Huff L: Blending the boundaries: steps toward an integration of complementary and alternative medicine into mainstream practice. Journal of Alternative & Complementary Medicine. 2002, 8: 897-906. 10.1089/10755530260511892.
- Atlas SJ, Chang Y, Keller RB, Singer DE, Wu YA, Deyo RA: The impact of disability compensation on long-term treatment outcomes of patients with sciatica due to a lumbar disc herniation. Spine. 2006, 31: 3061-3069. 10.1097/01.brs.0000250325.87083.8d.PubMed
- Rainville J, Sobel JB, Hartigan C, Wright A: The effect of compensation involvement on the reporting of pain and disability by patients referred for rehabilitation of chronic low back pain. Spine. 1997, 22: 2016-2024. 10.1097/00007632-199709010-00016.PubMed
- Rohling ML, Binder LM, Langhinrichsen-Rohling J: Money matters: A meta-analytic review of the association between financial compensation and the experience and treatment of chronic pain. Health Psychol. 1995, 14: 537-547.PubMed
- Hibbard JH, Stockard J, Mahoney ER, Tusler M: Development of the Patient Activation Measure (PAM): conceptualizing and measuring activation in patients and consumers. Health Serv Res. 2004, 39: 1005-1026. 10.1111/j.1475-6773.2004.00269.x.PubMed CentralPubMed
- Hibbard JH, Mahoney ER, Stockard J, Tusler M: Development and testing of a short form of the patient activation measure. Health Serv Res. 2005, 40: 1918-1930. 10.1111/j.1475-6773.2005.00438.x.PubMed CentralPubMed
- DeBar LL, Vuckovic N, Schneider J, Ritenbaugh C: Use of complementary and alternative medicine for temporomandibular disorders. J Orofac Pain. 2003, 17: 224-236.PubMed
- Sherman KJ, Cherkin DC, Deyo RA, Erro JH, Hrbek A, Davis RB, Eisenberg DM: The diagnosis and treatment of chronic back pain by acupuncturists, chiropractors, and massage therapists. Clin J Pain. 2006, 22: 227-234. 10.1097/01.ajp.0000169668.62900.ca.PubMed
- Gibbons RD, Hedeker D, Elkin I, Waternaux C, Kraemer HC, Greenhouse JB, Shea MT, Imber SD, Sotsky SM, Watkins JT: Some conceptual and statistical issues in analysis of longitudinal psychiatric data. Application to the NIMH treatment of Depression Collaborative Research Program dataset. Arch Gen Psychiatry. 1993, 50: 739-750. 10.1001/archpsyc.1993.01820210073009.PubMed
- Heckmann JJ: Sample selection as a specification error. Econometrica. 1979, 47: 153-161. 10.2307/1912352.
- Lynch FL, Hornbrook MC, Clarke GN, Perrin N, Polen M, O'Connor E, Dickerson J: Cost-effectiveness of an intervention to prevent depression in at risk teens. Arch Gen Psychiatry. 2005, 62: 1241-1248. 10.1001/archpsyc.62.11.1241.PubMed
- Hornbrook MC, Goodman MJ: Chronic disease, functional health status, and demographics: a multi-dimensional approach to risk adjustment. Health Serv Res. 1996, 31: 283-307.PubMed CentralPubMed
- Krebs EE, Bair MJ, Damush TM, Tu W, Wu J, Kroenke K: Comparative responsiveness of pain outcome measures among primary care patients with musculoskeletal pain. Med Care. 2010, 48: 1007-1014. 10.1097/MLR.0b013e3181eaf835.PubMed
- Kroenke K, Bair MJ, Damush TM, Wu J, Hoke S, Sutherland J, Tu W: Optimized antidepressant therapy and pain self-management in primary care patients with depression and musculoskeletal pain: a randomized controlled trial. JAMA. 2009, 301: 2099-2110. 10.1001/jama.2009.723.PubMed CentralPubMed
- Tan G, Jensen MP, Thornby JI, Shanti BF: Validation of the Brief Pain Inventory for chronic nonmalignant pain. J Pain. 2004, 5: 133-137. 10.1016/j.jpain.2003.12.005.PubMed
- Dworkin RH, Turk DC, Wyrwich KW, Beaton D, Cleeland CS, Farrar JT, Haythornthwaite JA, Jensen MP, Kerns RD, Ader DN, Brandenburg N, Burke LB, Cella D, Chandler J, Cowan P, Dimitrova R, Dionne R, Hertz S, Jadad AR, Katz NP, Kehlet H, Kramer LD, Manning DC, McCormick C, McDermott MP, McQuay HJ, Patel S, Porter L, Quessy S, Rappaport BA, Rauschkolb C, Reviciki DA, Rothman M, Schmader KE, Stacy BR, Stauffer JW, von Stein T, White RE, Witter J, Zavisic S: Interpreting the clinical importance of treatment outcomes in chronic pain clinical trials: IMMPACT recommendations. J Pain. 2008, 9: 105-121. 10.1016/j.jpain.2007.09.005.PubMed
- Caspi O, Koithan M, Criddle MW: Alternative medicine or "alternative" patients: a qualitative study of patient-oriented decision-making processes with respect to complementary and alternative medicine. Med Decis Making. 2004, 24: 64-79. 10.1177/0272989X03261567.PubMed
- Liang Z, Zhu X, Yang X, Fu W, Lu A: Assessment of a traditional acupuncture therapy for chronic neck pain: a pilot randomised controlled study. Complement Ther Med. 2011, 19 (Suppl 1): S26-S32.PubMed
- Franca DL, Senna-Fernandes V, Cortez CM, Jackson MN, Bernardo-Filho M, Guimaraes MA: Tension neck syndrome treated by acupuncture combined with physiotherapy: a comparative clinical trial (pilot study). Complement Ther Med. 2008, 16: 268-277. 10.1016/j.ctim.2008.02.006.PubMed
- Molsberger AF, Schneider T, Gotthardt H, Drabik A: German Randomized Acupuncture Trial for chronic shoulder pain (GRASP) - a pragmatic, controlled, patient-blinded, multi-centre trial in an outpatient care environment. Pain. 2010, 151: 146-154. 10.1016/j.pain.2010.06.036.PubMed
- Hondras MA, Long CR, Cao Y, Rowell RM, Meeker WC: A randomized controlled trial comparing 2 types of spinal manipulation and minimal conservative medical care for adults 55 years and older with subacute or chronic low back pain. J Manipulative Physiol Ther. 2009, 32: 330-343. 10.1016/j.jmpt.2009.04.012.PubMed
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6882/11/118/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.