Development of the adult and child complementary medicine questionnaires fielded on the National Health Interview Survey

The 2002, 2007, and 2012 complementary medicine questionnaires fielded on the National Health Interview Survey provide the most comprehensive data on complementary medicine available for the United States. They filled the void for large-scale, nationally representative, publicly available datasets on the out-of-pocket costs, prevalence, and reasons for use of complementary medicine in the U.S. Despite their wide use, this is the first article describing the multi-faceted and largely qualitative processes undertaken to develop the surveys. We hope this in-depth description enables policy makers and researchers to better judge the content validity and utility of the questionnaires and their resultant publications.

Analyses of the 2002 and 2007 surveys have resulted in more than 100 publications in peer-reviewed journals [2]. It is expected that the 2012 supplement will have the same significant impact on research and policy-making communities as did the 2002 and 2007 versions, which provided the most comprehensive nationally representative data on complementary medicine use available for the United States. Although many research priorities and specific content identified during the development of the 2002 questionnaire remained as part of the 2007 and 2012 surveys, each questionnaire contained unique topics. For example, the 2012 survey included a series of items related to using complementary therapies for wellness-related reasons. NCCAM decided to include this topic during development of its third strategic plan [3], which placed an emphasis on use of complementary therapies to promote health, wellness and well-being.
This article describes the multi-faceted and largely qualitative approach taken to develop the three questionnaires, particularly detailing the results of expert input and testing of cognitive salience and clarity of questionnaire items. This article will guide researchers and policy makers who plan to use data generated from the surveys, as well as inform quantitative assessments of questionnaire validity and of the research value provided by the resulting datasets.

Overall development process
The development of the three NHIS complementary medicine questionnaires was not performed within the framework of a planned research study. Instead, questionnaire design and development generally followed steps documented in the literature [4][5][6][7][8] and in an NCHS report on questionnaire design [9]. The NCHS protocol included four phases: 1) planning, 2) exploratory studies, 3) developmental methods, and 4) testing methods. Planning is particularly important for government surveys which generally have tight timeframes, require multiple clearances, and may involve staff from multiple organizations. For each questionnaire, development began 18-24 months prior to data collection in collaboration with survey methodologists from NCHS. The second phase, exploratory studies, is useful for identifying the response process and response problems unique to a given set of questions. This step applies to specific aspects of complementary medicine modalities such as the relationship between self-identity and use of nonvitamin, nonmineral dietary supplements (NVNMDS) [10] discussed later in this article. The third phase, developmental methods, involves iterative rounds of questionnaire evaluation through participant interviews as the main approach for discovering problems with questionnaires. Although the number of respondents need not be large (20-50 respondents), ideally they should differ widely in demographic characteristics to maximize generalizability. The final phase described in the NCHS report, testing methods, refers to large-scale field testing. Although budget constraints did not allow for large-scale field pre-testing, input from field staff was solicited prior to fielding and resultant modifications were made.
Development of the three NHIS questionnaires applied the definition of complementary medicine established by NCCAM: "a group of diverse medical and health care systems, practices, and products not generally considered part of conventional medicine" [1]. Although the specific therapies included varied slightly across survey years, the following were included in all three questionnaires: acupuncture, Ayurveda, biofeedback, chelation therapy, chiropractic care, energy healing therapy, hypnosis, massage, naturopathy, NVNMDS, homeopathic treatment, diet-based therapies, yoga, tai chi, qi gong, and meditation and other relaxation techniques. The 2002 questionnaire collected complementary medicine use for adults only, whereas the 2007 and 2012 included children and adults. All three surveys were approved by the NCHS Research Ethics Review Board.
Specific steps detailed in this article include: (a) literature reviews, (b) feedback from scientific and practitioner communities, (c) expert input and workshops, and (d) participant interviews (cognitive testing). Additionally, for the 2012 child questionnaire, quantitative analysis of data from the 2007 child complementary medicine questionnaire (frequencies analysis of prevalence of therapies by age, race/ethnicity, disease status, and use for health conditions) and focus groups with health professionals and the lay public were conducted. Although these steps took place sequentially within the overall development process (literature review, then expert panels, then draft initial questions, then cognitive testing, then refinement), the synthesis of these components to yield the final content was an iterative process. For example, after the expert panels, during the drafting of initial questionnaire items, we often discussed a particular series of questions with the expert panel member who focused the corresponding discussion during the meeting. Likewise, while refining the questionnaire based on results of cognitive testing, we often referred back to notes from workshops, consulted with expert panel members, or sought additional research articles to decide on the most appropriate solution. Specific steps used in the development of each questionnaire varied (Table 1) and are described in discrete terms below.

Literature reviews
The goal of each literature review was to identify previous surveys on complementary medicine use in order to help determine which therapies to include in the NHIS questionnaire. These surveys were reviewed by the expert panels when making recommendations to NCCAM and NCHS staff on what complementary therapies should be included in the NHIS. Literature reviews for all three surveys employed the search paradigm utilized by Wootton and Sparber [11]. Development of the 2002 questionnaire began by reviewing smaller-scale national and regional surveys identified by Wooton and Sparber through January 5, 2001. Development for the subsequent surveys examined U.S. national or regional surveys on complementary medicine that were published after January 5, 2001. Only U.S. surveys were considered during the literature review since the terminology, definitions and regulations vary considerably from country to country. For instance, although in the U.S. dietary supplements are regulated as food-based in the Dietary Supplement Health and Education Act [12], in other countries, such as Germany, they are often regulated as drugs [13] or, as in Japan, covered by national health insurance [14].

2002
The systematic review by Wooton and Sparber [11] identified seven population-based surveys published in peer-reviewed journals through January 5, 2001. Of these, five reported data from national population-based surveys on complementary medicine use in the U.S. [15][16][17][18][19] and two reported data from state-level population-based surveys -Florida [20] and South Carolina [21]. The report by Druss and Rosenheck [19] was based on data from the 1996 Medical Expenditures Panel Survey supported by the U.S. Agency for Healthcare Research and Quality. These seven surveys served as a starting point by the expert panel for identifying content to be added to the 2002 NHIS. Item wording from these surveys was often changed to be consistent with the NHIS format.

2007
In preparation for the 2007 NHIS complementary medicine survey an updated search of the scientific literature published up to January 2006 was conducted, and five additional national population-based surveys in adults [22][23][24][25][26] and one state-level -Michigan [27] were identified. Two of these were initial reports from the 1999 and 2002 NHIS surveys [22,24]. When reviewing the four non-NHIS surveys, particular attention was paid to complementary approaches not included in the NHIS surveys, as well as to reasons individuals said they used complementary therapies. Among the complementary therapies included on these surveys but not on the 2002 NHIS were traditional healers and movement therapies, both of which were added to the 2007 survey.

2012
From published reports through January 2011, the search update for the 2012 NHIS identified three new national-population based surveys [28][29][30], three state-level population-based surveys -California [31], Hawaii [32], and Minnesota [33] -and one survey of active duty military personnel in the U.S. Navy and marine corps [34]. These were reviewed for complementary approaches not contained in the 2007 NHIS, as well as wellness-related reasons individuals used these therapies. No previously unidentified complementary therapies were found in these surveys.
In addition, for the 2012 questionnaire, literature searches were employed specifically to inform questionnaire design related to reasons and motivations for using complementary therapies, such as for wellness or well-being, as well as questions asking about insurance coverage [35][36][37][38][39][40]. PubMed and Google Scholar were searched, and additional references were suggested by outside experts attending a workshop on wellness (Section Wellness workshop) and members of a think tank (Section Research topic think tank) to identify research priorities for the 2012 questionnaire. Wellness-related reasons identified in the literature for why individuals use complementary therapies include physical, spiritual, and emotional well-being, nutrition and lifestyle, and to avoid pharmaceuticals [35][36][37][38][39][40].
To identify additional dietary supplements for inclusion in the 2012 NHIS, dietary supplement sales data were reviewed [41]. Among the many products added to the NHIS survey were Acai, bee pollen and other bee products, digestive enzymes and methylsulfonylmethane (MSM). In total, 119 nonvitamin, nonmineral dietary supplements were queried in the 2012 NHIS versus 44 in the 2007 NHIS.   Table 3 shows a complete listing of expertise on each panel. The second panel did not convene as a group, but expert opinion was solicited in writing from each individual member related to the person's area of expertise and additional modifications were made based on this input. The first expert panel suggested specific therapies to include in the 2002 NHIS. The expert panel also identified four areas of research priority: 1) Prevalence of complementary medicine use; 2) purpose of complementary medicine use (whether for a specific condition, general wellness, or both); 3) disclosure of complementary medicine use to conventional providers; and 4) inventory of natural supplements and prescription medications (not fielded due to logistical reasons). The highest priority was to identify the specific conditions or diseases individuals report using complementary therapies to treat, which is distinct from assessing the prevalence of complementary medicine use among individuals who experience one or more health conditions. Analysis of this later construct is possible by linking the NHIS complementary medicine questionnaires to other NHIS core questionnaire data files. Additional conditions and diseases hypothesized to correlate with complementary medicine use were added to those already in the NHIS core questionnaire. The second panel identified ways to improve the cultural sensitivity of the questionnaire. For example, the use of the word "traditional" to refer to Western medical practices was replaced with "conventional" based on the recommendation from a cultural expert concerned that "traditional" has different meanings to individuals with regard to various culturally related practices.
Based on review of the literature and input from the panels of outside experts, a questionnaire was constructed, edited and refined by NCHS and NCCAM staff. Underlying this process was a set of criteria for questionnaire development identified from the literature [42][43][44] (Table 4). Once the questionnaire was constructed, the expert panels were given an opportunity to provide additional feedback. The draft questionnaire was also circulated to experts throughout the NIH for comment.  2012

Wellness workshop
As evidenced in NCCAM's strategic plans, the Center's priorities have evolved during the past decade from a focus on treatment of disease to symptom management and promotion of optimal health [1,3]. In August 2009, NCCAM staff sponsored a one-day workshop on how to best capture the concept of wellness and optimal health via survey items. Twenty-seven individuals participated in the workshop including behavioral scientists from universities and non-profit organizations, as well as representatives from NCCAM and other parts of NIH, CDC, and the Food and Drug Administration (Table 3). Because it was the sole internationally accepted definition at the time, NCCAM utilized the World Health Organization's definition of wellness to generate discussion: "wellness is the optimal state of health of individuals and groups. There are two focal concerns: the realization of the fullest potential of an individual physically, psychologically, socially, spiritually, and economically, and the fulfillment of one's role expectations in the family, community, place of worship, workplace and other settings" [45]. In preparation for the workshop, the planning committee identified criteria to help guide discussion [46][47][48][49][50][51][52][53][54][55][56].
Workshop attendees discussed several topics including how best to define wellness and differences between wellness and well-being. Discussion at the workshop focused on psychological, physical, spiritual, and social measures of wellness, as well as the relationship between wellness and use of complementary therapies. Workshop Interviewer protocols are guides to assist in the conduct of qualitative interviews and generally contain verbal probing and follow-up questions to aid in providing some standardization across interviews [57]. The interviewer protocol used for the testing of the wellness concept contained questions to elicit reasons or motivations for using complementary therapies without presupposing a "treatment" reason and respondents were asked about changes noted in "mind or body" when using the therapy (see section Concepts of wellness and well-being).

Research topic think tank
In September 2010 NCCAM staff convened a panel of experts to review the previous two NHIS complementary medicine questionnaires and identify items to retain as well as key research areas not previously covered. The panel was comprised of 24 individuals with expertise in health services research, minority health, economics, and psychometrics, as well as complementary therapies (Table 3). Based on feedback from the research topic think tank, an extensive set of questions was included in the draft questionnaire about insurance coverage and out-ofpocket expenses. This draft set of questions captured detailed information about deductible amounts, number of visits and other dollar amounts covered by insurance, and whether the respondent's complementary medicine use was reduced due to limited insurance coverage. This section was ultimately pared down due to complexity and time restraints (see Section Draft questionnaire).
Members of the think tank also recommended that deep breathing be removed as an independent intervention in the survey given the wide variability in how participants interpret the question as identified during cognitive interviews. Instead, queries on the use of deep breathing exercises were imbedded as follow-up questions for participants using yoga, tai chi, qi gong, meditative techniques, hypnosis and biofeedback. Finally, the think tank recommended that the survey provide some specificity on the types of meditation being used. Thus, instead of asking a global question as to whether a participant used meditation, the 2012 survey asked about three broad classes of meditation identified in an Agency for Healthcare Research and Quality systematic review: mantra, mindfulness and spiritual [58].
NCCAM staff members reviewed successive drafts for face validity and item phrasing, and an NCCAM staff member trained in cognitive interviewing techniques administered the draft questionnaire to nine NCCAM staff members with various cultural and academic backgrounds to better refine the questionnaire before it underwent cognitive testing with volunteers from the general public.

Cognitive interviewing
Cognitive interviewing, a technique to learn respondents' thought processes as they answer survey questions, has become standard practice for survey development [59]. The aim is to uncover how respondents interpret the intent and meaning of survey questions, and whether or not these match those of the researcher. The interview generally involves one interviewer and one respondent, usually takes between 30 to 90 minutes, and respondents are often compensated for their time. The interviewer will read the draft questions to the respondent and then use probing techniques to determine how the respondent is interpreting the questions and what the respondent is thinking while answering. The interviews were videotaped and their verbal content entered into Q-notes, a software system created by NCHS's Questionnaire Design Research Laboratory (QDRL) staff to organize and help with the analysis of cognitive interview data. Interpretations of key terms and concepts in each question were examined and compared among participants in order to identify inconsistent interpretations. Ideally, interviewing continues until "saturation" is reached, that is, no new data are emerging and all the concepts are well defined and explained [60]. However, because the NHIS must begin data collection in January each year, real-world deadlines did not allow for saturation for all sections of the questionnaire; rather, cognitive interviewing focused mainly on new topic areas added in each survey year, or existing sections that were substantially expanded (e.g. NVNMDS in 2007). After receiving IRB approval, QDRL staff used a variety of recruitment methods to obtain volunteers for cognitive testing including advertisements in the Washington Post and alternative medicine magazines, flyers placed at various co-ops and yoga studios, and the offices of various providers of complementary therapies. Informed consent was obtained from all participants prior to the start of the interview. For each survey year, a total of 25-48 in-depth semi-structured or open-ended interviews were conducted with volunteers from the general public with a range of socio-demographic characteristics (Table 5). During these interviews, questionnaire concepts were tested for cognitive salience and clarity. Possible actions taken as a result of cognitive testing included accepting the original question, accepting the original question with minor edits, accepting the original question with major edits, dropping the original question, and writing a new question ( Table 6). The questionnaire development process was fluid and iterative so that changes were made and the questionnaire re-tested throughout the development process. Table 7 provides additional examples of problems found during cognitive review of all three survey questionnaires and the The total for the 2012 race/ethnicity category is greater than the number of participants for 2012 because the data were collected such to allow overlap between Hispanic and other categories.  Accept original question Cognitive testing confirmed that respondents were primarily thinking of "Western," "mainstream" medical doctors when they responded to this question, as intended.
Original question: Did you do breathing exercises as part of [certain therapies]?
Accept original question with minor edits Testing revealed that respondents understood "breathing exercises" differently so that some thought it included merely taking a deep breath whereas others thought it was a more formal technique. Therefore, a definition was included in the text of the question.
Final question: Did you do breathing exercises as part of [certain therapies]? Breathing exercises may involve actively controlling the way air is drawn in, or the rate or depth of breathing.
Original question: During the past 12 months, did you see practitioner for/use [therapy] for any of these reasons…"

Accept original question with major edits
This question was divided into two questions and re-worded so that only respondents using medical treatments were asked whether experiences with medical treatments influenced their use of complementary therapies, and the type of medical treatment is explicit. Cognitive testing of the original question revealed that respondents found the reference to "medical treatments" unclear. For respondents who were not receiving or using any medical treatments, these questions appeared not to be applicable. In fact, a number of respondents specifically commented that these questions were not relevant to their circumstances.
…Medical treatments were not helping you?
…Medical treatments were too expensive?

…[modality] combined with medical treatments would help you?
Revised and restructured into 2 new questions: Did you receive any of the following medical treatments for [condition using complementary therapy for]…" Prescription medications?
Over-the-counter medications? Surgery?
Physical therapy?
Mental health counseling?
[next question asked for respondents who said yes to any items in question above]?
DURING THE PAST 12 MONTHS, did you see a practitioner for/use [modality] for any of these reasons… These medical treatments do not work for the health problem you want to treat or prevent?
[next question asked for respondents using prescription medications and/or over the counter medications] Prescription medications/Over the counter medications cause side effects?
Do you currently see a practitioner for [therapy] more, less or about the same as you did one year ago?
Question dropped Question was problematic for respondents who see a practitioner only sporadically or on an "as needed" basis. For instance, one respondent who had used homeopathic treatment in the past 12 months said she only uses it when she has a flare up and therefore found it difficult to compare her current use to her use 1 year ago. Similarly, a respondent who did not practice yoga with any regularity had difficultly answering.
Original question: During the past 12 months, did you see a practitioner for/use [therapy] because it is how you were raised?
New question written Although the meaning is similar, this item was re-worded to better capture the way some respondents are influenced to use complementary therapies by cultural factors. When hearing the original question, several respondents hesitated and seemed to find the original wording awkward, evidenced by one respondent who said, "That sounds very odd." The phrasing "how you were raised" suggested a more deliberate action than merely being exposed to complementary therapies at an early age. "It was part of your upbringing" better captured unintentional or cultural exposure which was the intent of the item.
Final item: During the past 12 months, did you see a practitioner for/use [therapy] because it was part of your upbringing?
General wording of "disease or health problem" found in many questions

Modified
Changing wording to "health problems, symptoms, or conditions" is more comprehensive and inclusive resulting modification made in order to reduce response error.

2002
Twenty-five in-depth semi-structured interviews were conducted; except for chelation therapy, collectively, interviewees had used every complementary medicine identified by literature review and outside experts for inclusion in the questionnaire. Two main findings quickly emerged: 1) modalities were practiced differently with respect to seeing a practitioner or using on one's own; and 2) many of the initial modalities tested were not discrete and overlapped with other modalities tested. To mitigate the first problem, NCHS staff, with input from the expert panel, identified 10 therapies that are mainly practitioner-based. For these therapies, the scope was narrowed to ask "Have you ever seen a provider for…" rather than "Have you ever used…" This better matched the way respondents described using complementary therapies during interviews. To reduce overlap among therapies, "Chinese medicine" was dropped because it was found to encompass many of the more specific therapies such as acupuncture and NVNMDS. In subsequent versions of the questionnaire, "home remedies" was tested as a self-care therapy and was deleted because it was found to mean different things to different respondents, as well as overlap with NVNMDS, vitamins, and homeopathy.
Ultimately, a wide range of modifications and restructuring took place based on the results of cognitive testing including: 1) practitioner-based modalities were separated from self-use modalities; 2) similar modalities were consolidated (e.g. yoga, tai chi, and qi gong were combined) or deleted (e.g. Chinese medicine); 3) questions that lacked meaningful responses or consistent interpretations were deleted (e.g. age of first use, "bad reactions," and "home remedies.") [61].

2007
Thirty-two cognitive interviews were conducted on seven different versions of the 2007 draft questionnaire. Results from testing reaffirmed two issues that surfaced during cognitive testing of the 2002 questionnaire. The first relates to home remedies, which was re-tested for the 2007 questionnaire as a practitioner-based therapy, with the goal of capturing visits to complementary medicine practitioners in minority and ethnic populations. However, testing revealed that respondents often do not consider home remedies to be practitioner-based, even when the question text includes the phrase "Have you ever seen a practitioner for…" Although respondents spoke of both home remedies and folk medicine as practices passed down through the family, as found during cognitive interviewing for the 2002 questionnaire, the two terms meant different things to different participants and They discouraged use of it in the past?
You were worried they would discourage it?
You were concerned about a negative reaction?
You didn't think they needed to know?
They didn't ask?
You don't think they know as much about it as you do?
They didn't give you enough time to tell them?
were therefore deleted. Second, heavy users of complementary therapies had difficulty recalling how old they were when they first used a particular therapy even when provided age categories. Because testing showed that estimates would be less reliable for long-time users of complementary therapies compared to recent users, the question was deleted [62]. In addition to these findings, cognitive testing also revealed that the question on "deep breathing exercises" produced numerous false positive responses. Many respondents had very little knowledge of this technique and essentially guessed at its meaning. One respondent said yes to having used deep breathing because "this is when you let out a sigh of being home to get your second wind to cook dinner." [62]. To try to mitigate this problem, a definition was added to the term, but it was ultimately dropped from the 2012 survey based on Think Tank recommendations (Section Research topic think tank). Particular attention was also given to three sections that were new and/or expanded from the 2002 questionnaire (see below).

Child section
For the 2007 NHIS, NCCAM expanded the questionnaire to include a limited set of items on complementary medicine use by children, a topic that had not been previously studied on a national level. The NHIS child section is answered by proxy, usually the child's parent. As expected, testing revealed that parents do not always know the answers to items on motivations and behaviors related to their children, such as whether a child prays for his or her health. Questions on motivations and prayer were, therefore, not asked in the child section.

Mineral and NVNMDS sections
Based on interest and funding by the NIH Office of Dietary Supplements, the 2007 questionnaire included expanded sections on vitamin and mineral supplements Have you ever used high dose or megavitamin therapy for your own health or treatment?
Many respondents who took vitamin supplements and/or a daily multi-vitamin responded "yes" even though they were not taking megavitamins.

Misinterpretation
Question divided into two parts so respondents who took any kind of vitamins had a chance to say "yes." Second question screened out the non high-dose users.
Have you EVER seen a provider or practitioner for movement therapies?
This question posed some definitional problems. Respondents had different definitions of movement therapy, including yoga and, more commonly, physical therapy. It was nearly impossible for persons who were heavy users of complementary therapies for a long time to remember the age at which they started using various complementary therapies.

Failure to recall
Question dropped Have you EVER used natural herbs or other non-vitamin supplements for your own health?
For some people, using herbal, non-vitamin supplements carries meaning beyond simply using an herb for a specific purpose. As one respondent put it, using alternative therapies is a "way of life." As a result, people who do not see themselves in this light define their supplement use differently from those they view as "users" of supplements. Several respondents who had no complementary therapy identity wanted to be sure they weren't labeled as "users." And this concern affected the way they interpreted and answered the questions. In the end, false negatives were elicited from respondents with this perspective; that is, the question is not capturing all people who use herbal supplements. Furthermore, the data suggest herbal users may not be missed at random, representing a pattern that would lead to biased estimates [62].
Answer based on self-concept rather than actual behavior This question asked respondents about something they have no way of knowing for sure ("I don't know"; "I don't think so"; "I'm guessing no"; "not that I'm aware of"). Those with small children can make a better assessment because prayers are sometimes said out loud together, but those with older children usually have to guess [62].

Information unknown
Question dropped (2007) and NVNMDS. These sections collected detailed followup information for up to two specific vitamins or minerals or NVNMDS supplements and an expanded list of reasons for using these. Although the sections on vitamins and minerals and NVNMDS were identical, cognitive testing revealed problems unique to the NVNMDS items [10,62]. First, it became apparent that respondents lacked a consistent, agreed-on definition of an herbal supplement. Respondents held wide-ranging and varying definitions of NVNMDS and efforts to craft a definition that met researchers' criteria and was understandable to respondents fell short. Ultimately, the best approach was to simply ask respondents about taking specific supplements listed on a flashcard. This made the survey response task concrete and less reliant on shared definitions, although, the tradeoff was the loss of information about supplements not listed on the card. Second, some respondents viewed themselves as having or not having a "complementary medicine identity" and this perception affected the way they interpreted and answered questions, such that response error occurred even when the term "natural herbs" was understood as intended. To mitigate this, the description of "a typical herb user's" lifestyle was removed from the lead-in statement [10,62].

Perceptions of complementary therapies
A section on perceptions of complementary therapies was explored for the 2007 questionnaire. The questions were to capture respondents' perceptions about the scientific evidence, safety, effectiveness, and holistic aspects of complementary therapies, as well as items concerning locus of control and how decisions are made to use or not use complementary therapies. The initial plan was to ask a long series of items about each individual complementary therapy, but time constraints made this unfeasible. QDRL staff then tested a version asking about complementary therapies as a whole, and determined that respondents' views vary by specific therapies such that they could not arrive at a single answer. Ultimately the section was drastically reduced and embedded within each therapy rather than as a stand-alone section.

Concepts of wellness and well-being
In fall 2010, staff from NCHS's cognitive lab performed inductive interviews with 24 volunteers to learn more about individuals who use complementary therapies to promote their wellness, general health and well-being. The interviewer protocol contained questions to elicit reasons or motivations for using complementary therapies without presupposing it was to treat a specific condition. In particular -and in keeping with the published literature [35,36,[38][39][40] -respondents were asked about changes noted in "mind or body" when using complementary therapies. Interviewing focused on how respondents who identified "wellness" and "well-being" as their main reason for using complementary therapies understood and conceptualized these terms. Because the intent was to learn more about a general concept or concepts rather than test draft questionnaire items, a more open-ended approach was utilized. When analyzing and interpreting the interviews, researchers aimed for "thick description," that is, to provide a deeper, more nuanced understanding of the ways that individuals interpret events and phenomena in their social worlds. New participants were recruited and interviewed until no new conceptual insights were identified (saturation). Testing revealed that although respondents reported benefits to their physical health, they primarily associated complementary medicine use with emotional and psychological benefits. The most frequently mentioned benefits were the positive effect on one's mood, stress level, and overall mental and emotional health. In addition to these emotional and psychological benefits, users also described many different physical benefits, most often mentioned in conjunction with their use of acupuncture, massage, or chiropractic care, and in particular, to help treat and eliminate back pain. The primary components of respondents' conceptualizations of well-being included being centered, finding balance, and not simply being disease free, but optimally healthy [63]. Based on these qualitative research findings and related literature, a series of items for the 2012 questionnaire was drafted by NCCAM and NCHS staff to capture wellnessrelated reasons for using complementary therapies. Examples include "because it is natural," and "because it focuses on the whole person, mind, body, and spirit." Additionally, some respondents mentioned having benefited from wellness-related outcomes as a result of using complementary medicine. Hence, the questionnaire includes several items asking if using complementary medicine led to outcomes such as "improve your overall health and make you feel better," "motivate you to eat healthier," and "help to reduce your stress level or to relax" (Table 8).

Draft questionnaire
Twenty-four cognitive interviews were conducted on several different versions of the 2012 draft questionnaire in an iterative process during which revisions were made and retested. Testing focused particularly on two sections: 1) A large set of questions about insurance and payment; and 2) the expanded list of reasons for using the therapy, especially for general wellness and wellbeing. Cognitive testing and focus group feedback (see below) determined that the questions on insurance coverage of, and payment for, complementary therapies were too complex to adequately capture valid information in the limited survey time available. They were replaced with a simple question asking if health insurance covered any of the costs; if yes, To cleanse/detoxify your body Deleted Respondents offered a wide variety of interpretations and meaning varied tremendously from person to person. Some considered it to be referring to "a cleanse" in the very literal sense, such as a colon cleanse. One respondent provided the example of the cayenne pepper and maple syrup diet. Others thought it referred to the period after an addict stops using drugs or alcohol and attempts to purge the body of these substances. For still others this was interpreted to be more symbolic, as in the case of a "mental or emotional cleans"' as one respondent put it [63].
Because the practitioner spends more time with you than medical doctors Deleted Several respondents explained that they pay for the time with the practitioner. As one respondent put it after hearing this question, "Well, yeah, [because] I paid for 20 minutes." Another respondent similarly explained that you "pay for the time" with the practitioner and so "of course you get more time with them" (he was thinking specifically about massage practitioners) [63].
Because the practitioner treats the whole person and not just one part because it focuses on the whole person and not just one part Combined/Modified to "it focuses on the whole person, mind, body, and spirit" The question did not work well when focused on "the practitioner," especially for massage. Several noted that they actually want their masseuse to focus on only one part of, for example, their backs, and not their entire bodies. Also, the phrase "mind, body, and spirit" resonated with respondents' experiences with complementary therapies [63].
Because I can participate in decisions about my health with my practitioner

Deleted
The question had varying interpretations and varied widely based on the type of practitioner being seen. Respondents using "hands-on" treatments such as massage said they can tell their practitioners where to focus and which parts of their bodies need more attention. Others thought it meant "sitting down with my doctor, presenting the symptoms and then working together to come up with a solution. [63]" To stay healthy Deleted A number of respondents commented that this either seemed "too broad" or 'too obvious." As one respondent put it, "You go to a doctor to stay healthy. It's too obvious." Another asked, "What's healthy? That's too general." [63] Thinking about seeing a practitioner for [modality], please tell me if any of these statements are true for you. Using [modality] has… Modified to accommodate proxy reporting and include items salient to children Question stem and individual items modified to ask parent proxy whether he/she "thinks" these items applied to his/her child, and items thought to be unknown by parent were dropped. Also, the items on alcohol and smoking were not asked for children. did it cover all, some, or none, and how much the respondent paid in total or per visit for the particular therapy [64].

CAMHI conducted focus groups and quantitative analyses
To provide additional input on the 2012 questionnaire, investigators from the Child and Adolescent Health Measurement Initiative (CAMHI) based at the Oregon Health & Sciences University performed quantitative analysis of data from the 2007 child complementary medicine questionnaire including frequencies analysis of the prevalence of each therapy by age group (0-5, 6-11, 12-17), race/ethnicity (Hispanic, white non-Hispanic, black non-Hispanic, other non-Hispanic), disease status, and whether the child used any complementary medicine for reported health conditions [65]. Also explored was how these prevalence rates varied depending on which definition of complementary medicine was used (e.g., whether vitamins and minerals were included in the definition). These analyses led to the identification of sample size deficiencies, appropriateness of given therapies among different age groups, and construct validity of items related to complementary medicine use [65]. Resulting modifications to the 2012 survey included: 1) restricting the child survey to children aged 4 and older; 2) for both the child and adult questionnaires, follow-up questions were not asked for individual traditional healers, movement therapies, and special diets. Rather, individual complementary approaches within each category were grouped to increase cell sizes; and 3) follow-up questions on the reasons children or adults used Ayurveda and chelation therapy were not asked due to small cell sizes on earlier surveys.
In addition, CAHMI organized and conducted key informant interviews and focus group discussions with 21 volunteers from the Family Voices Network [66], as well as from the CAHMI network of expert advisors, pediatric provider partners and family leaders partnering on an NIH/NCCAM grant related to pediatric use of complementary therapies. Informed consent was obtained from all interview and focus group participants. Informant interview and focus group recommendations included: (1) further refinement of the cost and insurance questions, (2) inclusion of child-specific health and well-being reasons for using complementary medicine, (3) the addition of questions about complementary medicine informationseeking, (4) communication with other health care providers about complementary medicine use, and (5) the addition of an item on satisfaction of complementary medicine use for a given reason for use [65]. Although CAMHI-conducted focus groups and quantitative analyses focused on the child component, the provided input contributed to refinement of both the 2012 child and adult questionnaires.

Discussion/conclusions
In this paper, we detail the multifaceted, largely qualitative, iterative process based on literature reviews, expert panel input, and cognitive interviewing methods used to develop and refine the NHIS complementary medicine questionnaires. For the 2012 questionnaire, we also used inductive cognitive review to conceptualize the concept of "wellbeing" and focus groups with family leaders and subject matter experts to further refine the child questionnaire. By soliciting feedback from subject matter experts, family leaders, and the lay public, we improved the likelihood that the questionnaire items will be understood and interpreted as intended. We describe the systematic evaluation of questionnaire items, the decision-making processes, and rationale for how items were retained, modified, or deleted. By providing this in-depth description of how these influential national surveys were developed, we hope to enable policy makers and researchers to better judge the content validity and utility of the questionnaires and the resultant findings. We also illustrate how the questionnaires have adapted over time to new scientific information and societal shifts so that modifications can be better understood by data users.
There are several limitations to the approach we used to develop the NHIS complementary medicine questionnaires. First, due to time constraints, in many cases, we were not able to continue cognitive testing until we reached saturation and may have missed some problematic issues with the questionnaires that could potentially reduce validity or generalizability. Second, although great care was taken to select individuals for cognitive testing who differed widely in demographic characteristics, the volunteers may not have been representative of the U.S. population in all important respects, which could limit generalizability of the testing results. Third, we did not include lay people on the expert panels convened to determine content areas for the questionnaires and thus could have missed salient topics of particular interest to users of complementary medicine. However, lay public were included in the CAHMI-organized focus groups that identified additional survey topics not prioritized by the think tank experts. Fourth, there is no universally accepted definition of what constitutes a complementary medicine approach. As a result, there is no definitive list of which therapies should be included in the questionnaire. For instance, whether prayer for health reasons should be considered within the realm of complementary therapies has received considerable discussion [67,68]. For these surveys, we have been as inclusive as possible given time constraints, thus allowing greatest flexibility to data users. Fifth, the NHIS has limitations common to all cross-sectional studies collecting participant-reported data: it is not possible to make inferences about cause and effect, and data are not independently verified. This latter limitation may be particularly important for the section on out-of-pocket expenditures in which respondents likely provide estimates versus exact amounts. Finally, in the 2007 survey and for some respondents in the 2012 survey, although the cost per visit or purchase was collected as continuous data within the range of $0-$500 per expenditure, the number of visits or purchases per person was collected as categorical data (based on recommendations from cognitive testing). This mixing of continuous and categorical data makes the computation of actual expenditures on complementary therapies very complex [69]. For the 2012 survey, we hope to provide better guidance and technical documents explaining how to analyze the expenditure data.
The three NHIS complementary medicine questionnaires were developed to assess the U.S. public's use of complementary health approaches and factors thought to be associated with this use. It is our expectation that quantitative analyses on the validity and reliability of the questionnaires will supplement the qualitative development approaches describe here, and will inform future survey design and provide guidance to users of existing survey data. Through changes over time, the NHIS complementary medicine surveys have attempted to meet the challenge of providing information to the NCCAM/NIH, national academies, researchers, clinicians, and policy makers. The NHIS complementary medicine datasets remain the primary national source of data on complementary medicine use in the United States.