Another important aspect of the practical side of your ESM study is to create meaningful and engaging questions. Note that questions for ESM studies should not be time-consuming. As participants will be asked to report behavior, thoughts, and other issues several times per day (for one or more weeks), questions should not become a burden (emotionally or cognitively).
When deciding on the number of questions needed, researchers should focus on how motivated their participants are, how long the study is going to take, and how much effort the testing itself requires. Note that researchers have accepted a burden cap of 20 minutes per day (Conner, 2015). Formulate the questions correctly – ESM studies should not ask people for their experiences in general, but for current events. Usually, researchers can convert a traditional study into an ESM measure to ensure good validity and reliability.
The actual text of the questions should be short, especially if small screens of mobile phones or personal digital assistants (PDAs) are used as preferred reporting methods.
2.3.1. Is It Reliable?
Well done on the hard job of creating a questionnaire! However, hold on for a second. It’s time to ask yourself: Are the ESM scales used in the study reliable? Note that some researchers claim that reliability matters only in the service of validity (Bergkvist & Rossiter, 2009). Still, experts are concerned the most about the following problem: Do ESM items measure between-person variations only, instead of within-person factors (Shrout & Lane, 2011)?
The first step to check if the created questionnaire shows good reliability is to report Cronbach’s alpha (especially when multi-item scales are present). When we create questions, we should remember that at least three items for each ESM construct are needed (Shrout and Lane, 2011). Single items work fine as well, for example, a 7- to 10-point scale or a 0-100 slider scale, which in addition is a method that can be used to increase variance (Van Hooff, Geurts, Kompier, and Taris, 2007). However, complicated constructs that change over a longer time span should be assessed with multi-item scales.
After that, it’s time to focus on the most important aspect of each research – participants. We have to agree that building up on knowledge for the pure purpose of knowledge is intriguing. However, the human factor is the most important part of a research. By applying the existing knowledge to everyday life, scientists can improve people’s well-being and interactions. And that’s what matters the most.
Researchers might face various difficulties when experts choose participants. Factors, such as sample size and demographics, are very important. Usually, ESM studies test the same participants and their daily lives, with variables that fluctuate over the short term. You might consider even recruiting and motivating your research participants (Conner & Lehman, 2011). Having intrinsically and extrinsically motivated participants is one of the factors that will influence your results positively. Rewarding your participants might be extremely helpful.
Note that when it comes to participants and ESM studies, researchers worry if constant self-reporting can affect respondents’ perceptions. However, we can look at this effect as one of the main advantages of ESM testing. Simply because frequent self-monitoring can be used as a therapeutic intervention, creating reactivity (Barta, Tennen, & Litt, 2011).
2.5. Pilot Testing
When all research plans are finalized, it’s time to give your research endeavors a go. Pilot testing is more than beneficial. Choose a group of people similar to the intended participants of your ESM study. However, note that human beings are unique and what works fine for one sample might go wrong for another.
If you think that pilot testing is just a waste of time, you should think twice. Did you know that Schneider and Waite (2005) did three pilot-tests of their national study of families? Pilot testing can give you valuable insights on question-wording, length and other crucial aspects of your ESM questionnaire. Not only that, though. The post-collection procedures and analysis of data, such as descriptive statistics, can show researchers some problematic parameters of the questionnaire, such as coding, that was not obvious during administration.
Most of all, pilot testing is a great way to connect to all participants. The size of the pilot sample depends on the purposes of the full-scale study. In some cases, 10-20 pilot subjects are enough, in others, smaller samples are also acceptable. Although ESM studies don’t take too much time, real testing might interrupt some important daily activities, which as a result may affect the intrinsic motivation of the subjects. Thus, listening to people’s feedback and analyzing their notes taken during the study is important as it can help researchers improve their ESM study design. For instance, always ask if: response options were meaningful; signals were clear; study situations were difficult to understand; testing procedure was frustrating;, etc. From giving instructions to signaling methods, pilot-tests contain a lot of useful information and may give some humane touch to the ESM design.
Also remember that when you thrive for perfection, you should embrace errors. Focus on missing or noncompliant reports and adopt procedures to battle similar problems that may arise in the future. Ask participants if items were too irrelevant or too confusing. Make sure that the procedure used in the study wasn’t too difficult to administer, or check if signals were unclear. At the same time, create a more realistic testing environment: encourage pilot subjects to give false data or to miss a response from time to time. All these problematic situations are things that could happen even to the best ESM study designs.
Do not forget that research assistants also have to undergo some training so they can relate to their participants and understand the testing procedure better. In other words, even the “pilots” of the study have to go through the pilot testing procedure.
Also, when you choose a tech device for a reporting method, have a spare in case something fails or breaks. The signaling equipment and the procedures for dealing with data are also a subject of errors. Mind that plans don’t always go smoothly and learn from results!