Table 7 Examples of response error in cognitively tested questions for 2002, 2007, and 2012 questionnaires

Have you ever used high dose or megavitamin therapy for your own health or treatment?

Many respondents who took vitamin supplements and/or a daily multi-vitamin responded “yes” even though they were not taking megavitamins.

Misinterpretation

Question divided into two parts so respondents who took any kind of vitamins had a chance to say “yes.” Second question screened out the non high-dose users.

(2002)

Have you EVER seen a provider or practitioner for movement therapies?

This question posed some definitional problems. Respondents had different definitions of movement therapy, including yoga and, more commonly, physical therapy.

Definition unknown

Specific movement techniques (Feldenkreis, Alexander Technique, Pilates, Trager Psychophysical Intergration) were asked about individually

(2007)

How old where you when you first saw a practitioner for [therapy]?

It was nearly impossible for persons who were heavy users of complementary therapies for a long time to remember the age at which they started using various complementary therapies.

Failure to recall

Question dropped

(2007)

Have you EVER used natural herbs or other non-vitamin supplements for your own health?

For some people, using herbal, non-vitamin supplements carries meaning beyond simply using an herb for a specific purpose. As one respondent put it, using alternative therapies is a “way of life.” As a result, people who do not see themselves in this light define their supplement use differently from those they view as “users” of supplements. Several respondents who had no complementary therapy identity wanted to be sure they weren’t labeled as “users.” And this concern affected the way they interpreted and answered the questions. In the end, false negatives were elicited from respondents with this perspective; that is, the question is not capturing all people who use herbal supplements. Furthermore, the data suggest herbal users may not be missed at random, representing a pattern that would lead to biased estimates [62].

Answer based on self-concept rather than actual behavior

Question limited to specific pills, capsules, tablets, or liquids labeled as a dietary supplement, listed on a flashcard shown to the respondent

(2007)

DURING THE PAST 12 MONTHS, did [child] pray specifically for the purpose of his/her OWN health?

This question asked respondents about something they have no way of knowing for sure (“I don’t know”; “I don’t think so”; “I’m guessing no”; “not that I’m aware of”). Those with small children can make a better assessment because prayers are sometimes said out loud together, but those with older children usually have to guess [62].

Information unknown

Question dropped

(2007)