Subjective Outcomes: Questionnaires and Data Forms

One of the first aspects to consider is the mode of administration. There are different modes of administration: self-administered questionnaires, surveys administered by a caregiver or a family member, and researcher-administered forms – each with its various benefits and disadvantages (Peat, 2011). For instance, self-administered questionnaires are time and cost-effective, and as such, they are extremely beneficial in large samples. However, self-administered forms are prone to unclarity and low response rates. Surveys administered by a family member or a caregiver can be used in pediatric populations or for adults who cannot respond for themselves. However, they can refer only to observed symptoms, such as vomiting – simply because some outcomes, such as morning stiffness, can be known only to the patients. On the other hand, surveys administered by a researcher can help experts collect more complex data, which is paramount in rare diseases. Unfortunately, such tools are more expensive and prone to interpretation bias.

Another important factor is the procedure itself: surveys can be done in person, via the phone and post, or web-based. Recently, health technology has been established as an effective tool, and online surveys have slowly replaced paper forms. No matter what type of administration experts choose, it’s extremely important for experimenters to be consistent throughout the whole study, which will increase the internal validity and reliability of the new test.

Creating the Right Questions

Creating new research questions is perhaps one of the most exciting and challenging parts of research. As explained above, a literature review is crucial to help experts gain a better understanding of the topic of interest and the existing measurements. Knowing if there are other similar and valid surveys is vital, and in fact, it can save precious time (Peat, 2011).

All questions should be easy to understand and relevant. Therefore, the main characteristics of each sample (including size) should be considered. In fact, when it comes to participants, the sampling procedures (randomized, etc.) also need to be established. On top of that, researchers must decide if the questionnaire will be confidential for follow-up purposes or completely anonymous.

The content, the wording, the order, and the length itself are other vital aspects researchers need to focus on. Note that sometimes the same question can be asked twice but in a different way to double-check responses and social desirability.

We should mention that, usually, research questions tackle two types of medical information: qualitative (which is used to generate hypotheses) and quantitative (which is needed to test hypotheses) (“Questionnaire Design”). For qualitative data, exploratory and open questions might be better. Although they are more difficult to code and analyze, open questions widen the scope of research and help experts generate new ideas (Peat, 2011). On the other hand, standardized close-ended items are needed for collecting quantitative information. They also come with lots of challenges, such as attracting random responses. Still, they help researchers collect data quickly, via fixed and pre-coded replies (Peat, 2011).

In any case, experts should always try to reduce unambiguity. If one could measure exposures, confounders, outcomes, and demographics at the same time, this would be the ultimate testing tool. Therefore, focus groups can be extremely valuable to collect new ideas, census forms to help researchers generate questions, and peer review to establish internal validity.

Note that when sensitive information, such as ethnicity or income, is needed, surveys become more complicated. Do not forget that such questions may reduce the response rate, so they can be excluded or added at the end of the survey.  In fact, using wording similar to the national census, for instance, is a good method to make participants feel more comfortable (Peat, 2011).

The Power of Wording

As explained above, questions should be relevant, simple, valid, and responsive to change. Therefore, the wording is a paramount aspect of the development of a new survey.

• Positive wording is more attractive and generates better response rates. Therefore, experts should avoid medical jargon and terms.
• The construction of the items should be simple to understand and easy to answer, code, and analyze. Experts must ask only one question at a time. In case it’s needed, capitalization of words can be applied to clarify the question and help subjects focus better.
• Pilot studies are a must. They can help experts deal with unambiguity. During a pilot study, experts can ask participants to rephrase questions they don’t understand, and consequently, achieve some new and better wording.
• If there’s one correct answer, the response options should be clear. If there are multiple-response categories, these different groups should have the same meaning for everyone. For instance, timing responses, such as ‘seldom’ or ‘rare,’ may mean different things to different people.
• ‘Don’t know’ options should be avoided. People are more willing to choose ‘Don’t know,’ especially when the meaning of the question is unclear. What’s more, this effect can lead to low validity and generalizability of the tool.
• Dealing with missing data is also vital and can help experts increase generalizability. In addition, researchers should decide if missing answers can be used as negative responses in the analysis.
• Last but not the least, when it comes to questionnaires in different languages, second opinion and translation are a must.

Presentation Matters

Apart from the wording of the questions, the layout is also vital. Note that electronic surveys engage participants more, especially younger subjects. Tools should be clear, short, and attractive (Peat, 2011). They need to have:

• Large font
• Enough space
• Mild colors
• Numbered questions
• Titles of the questionnaires
• Instructions on how to answer

Note that tick boxes are more attractive than other response options. Circling, percentages or frequency of behaviors can be confusing. In fact, time responses may be inconsistent and may mean different things to different people, as explained above. A suggestion is, instead of ‘usually,’ to ask for a precise indication of frequency (‘1-6 times a year, etc.).

Again, it’s crucial to decide if missing answers mean negative responses. For instance, a parent can skip a statement like ‘Is affectionate’ about their child. Well, we have to ask ourselves: have they missed the item unintentionally, or does that mean that the child is not affectionate?

When answers are given via a time scale, there are some tricky details as well. For instance, it’s a fact that subjects avoid the ends of the scale. So, instead of having a 5-point scale with ‘never’ and ‘always’ at the two ends, extend the scale and put ‘almost never’ and ‘almost always’ in between. On top of that, short scales may not detect subtle changes, which decreases responsiveness. Note that although scales require larger samples, they are statically more powerful than ‘yes’ or ‘no’ answers, or other fixed responses, such as ‘true’ or ‘false’ and ‘agree’ or disagree’ (“A Step-by-Step Guide to Developing Effective Questionnaires and Survey Procedures for Program Evaluation & Research”).

Last but not the least, confidential or anonymous, researchers should always add a ‘Thank you!’ section at the end of the survey.

Coding & Safety

When it comes to new questionnaires, coding should be considered. Coding is a method that benefits data collection and analysis.

Self-coding data collection forms, for instance, can spare researchers crucial time.

Usually, in paper forms, and expert transfers the codes to a spreadsheet (“How to analyze questionnaire responses”). Thus, to guarantee safety, observer data should also be entered, and hard copies of the original forms should be kept to maintain quality control.

Pilot Study Wanted

When the initial draft of the new questions is ready, a pilot study should be conducted, for both questionnaires and data collection forms. Note that a number of pilot studies might be required (Peat, 2011). Some of the main steps are to administer the test in the same way it will be administered and to a similar sample to the target population. Experts should check if all answers are completed, measure time, and ask for feedback. As a result, they should reword the items, exclude unclear questions, and shorten the scale. If needed, another pilot testing can be done.

Ensure Validity & Repeatability

Of course, validity and repeatability are crucial aspects. When it comes to internal validity, peer review may help experts ensures good face validity. Factor analysis and logistic regression, on the other hand, can help experts analyze which items contribute to the topic and to what extent. Cronbach’s alpha can also be used to assess the degree of correlation between the new items. Note that this coefficient varies between zero and one, but when it’s too high (close to one), results indicate that some items are too identical and should be excluded (Peat, 2011). For instance, ‘vomiting’ and ‘feeling sick’ may be too similar and overlapping.

To sum up, choosing the outcomes is crucial and sometimes a new questionnaire or a data collection form may be required. In the end, it’s important to establish a harmonic relationship between clinical experience, measures, statistics, and theory – with the sole purpose to understand a disease and improve people’s well-being