The Experience Sampling Method (ESM) is one of the most reliable procedures for studying daily lives and people’s experiences, activities, and feelings throughout the day. With the great idea of Kurt Lewin (1935) to create a topology of daily activities and the ambitious techniques created by Roger Barker and Herbert Wright, ESM studies have become incredibly popular today.
Each research method and findings benefit science, for sure. From longitudinal studies to archival research, adding more to the existing knowledge is always beneficial. However, ESM has numerous advantages when compared to other techniques in the field of biology, psychology, and medicine. For instance, while laboratory experiments limit research with the artificiality of some procedures, ESM does not take subjects out of their natural environment. In fact, that’s why ESM studies are called “ecological” – because all research and observation take place within natural settings, in “real-time” and “real-life” situations (Schiffman, Stone, and Hufford, 2008). On the other hand, while observational studies cannot eliminate the influence of the observer completely, ESM relies on self-reports and people’s active participation. In other words, ESM studies give researchers the chance to implement controlled and natural settings at the same time.ESM studies are applicable to a wide range of scientific topics, such as panic attacks, medical interventions, pain management, reflecting on feelings, etc., they can be used within organizational settings as well. As mentioned earlier, ESM studies are among the most valuable research assets to science.
You’re probably already familiar with ESM studies and their advantages, and what you actually need is some practical help with designing an ESM study. When doing that, a few basic steps should be followed:
Aim for an Aim: Having a clear goal is the first step that can take your experiment towards success. In other words, having an identified aim can give a powerful kick to your research journey. In the midst of scientific ideas, experimental curiosities and philosophical mysteries revolving around human nature, having research questions that can actually be tested is crucial. Being realistic and specific is the only way to get real results with some statistical strength.
Do Your Research: Having an aim will open the door to research. Conducting in-depth research is crucial. From online sources to journals, our world is full of information. Do not set any limits. Literature research might be time-consuming but expand on your study by focusing on references and related topics. Background research can help scientists generate more ideas and investigate new aspects of the topic. That said, brainstorming ideas within academia and other communities might also contribute to the field of research.
2.2.1. Research Hypothesis:
After the hard work, it’s time for a more precise design. You’re the designer, so the beauty of your research lies in your hands. Formulate a hypothesis – a hypothesis that’s in fact testable. In order to do that, systematize all variables that might have an impact on the study. Decide if you want to study behaviors, such as drinking or eating, experiences, such as mood or cognition, or physiological functions, such as breathing or hormonal levels. Having a clear operational definition is important because – by having a clearer idea of what’s being studied – you have more control over the process.
2.2.2. Define Your Variables:
As you know, the relation between the variables of interest will be tested in your hypothesis. Hypotheses usually focus on various dynamic within-person interactions. The best way to achieve a meaningful result is to study variables that change over a short period of time (Conner & Lehman, 2012). Do not forget that having a null hypothesis is also important. Last but not least, remember that even by rejecting your hypothesis, you contribute to science.
2.2.3. Statistical Procedures:
Having a clear idea of how to analyze the collected data is also very important. Simply because sometimes we might have data, which we cannot use, or too many figures we do not know how to analyze. Collecting, recording, interpreting, and presenting data is vital, and research should be factual and based on statistical procedures.
Note that ESM can be a great quantitative method. Simply because all techniques provide large numbers of data. A method that includes multiple questions, multiple contexts, and multiple times every single day is definitely a research technique that is a dream.
On the other hand, ESM studies can be used as a wonderful qualitative method because each participant self-reports their feelings, behaviors, and thoughts in their everyday life.
Do not forget to include charts and tables for the visual representation of your data.
2.2.4. ESM Design:
Then, it’s time for another major aspect of your ESM study: choose an ESM design. If having a hypothesis and data is needed in each experiment, ESM designs are a unique characteristic applicable only to ESM studies. We’ll discuss this section and the different types of ESM design later.
2.2.5. Reporting Methods:
Just like ESM designs, the different reporting methods used in ESM studies are also unique. When one starts designing a study, they are often overwhelmed by ideas and expected results and applications. However, the practical side of your study should also be considered and planned in detail. In addition, you have to decide if your study will benefit from an active or passive reporting method. An example of an active reporting method is daily diaries, while a passive method is neuroendocrine sampling, for example. Choosing the right reporting method is crucial. Note that Smartphone Apps engage participants the most, and in today’s society, even older participants have no problems using technology. This section will also be discussed at a later stage.
Another important aspect of the practical side of your ESM study is to create meaningful and engaging questions. Note that questions for ESM studies should not be time-consuming. As participants will be asked to report behavior, thoughts, and other issues several times per day (for one or more weeks), questions should not become a burden (emotionally or cognitively).
When deciding on the number of questions needed, researchers should focus on how motivated their participants are, how long the study is going to take, and how much effort the testing itself requires. Note that researchers have accepted a burden cap of 20 minutes per day (Conner, 2015). Formulate the questions correctly – ESM studies should not ask people for their experiences in general, but for current events. Usually, researchers can convert a traditional study into an ESM measure to ensure good validity and reliability.
The actual text of the questions should be short, especially if small screens of mobile phones or personal digital assistants (PDAs) are used as preferred reporting methods.
2.3.1. Is It Reliable?
Well done on the hard job of creating a questionnaire! However, hold on for a second. It’s time to ask yourself: Are the ESM scales used in the study reliable? Note that some researchers claim that reliability matters only in the service of validity (Bergkvist & Rossiter, 2009). Still, experts are concerned the most about the following problem: Do ESM items measure between-person variations only, instead of within-person factors (Shrout & Lane, 2011)?
The first step to check if the created questionnaire shows good reliability is to report Cronbach’s alpha (especially when multi-item scales are present). When we create questions, we should remember that at least three items for each ESM construct are needed (Shrout and Lane, 2011). Single items work fine as well, for example, a 7- to 10-point scale or a 0-100 slider scale, which in addition is a method that can be used to increase variance (Van Hooff, Geurts, Kompier, and Taris, 2007). However, complicated constructs that change over a longer time span should be assessed with multi-item scales.
After that, it’s time to focus on the most important aspect of each research – participants. We have to agree that building upon knowledge for the pure purpose of knowledge is intriguing. However, the human factor is the most important part of the research. By applying the existing knowledge to everyday life, scientists can improve people’s well-being and interactions. And that’s what matters the most.
Researchers might face various difficulties when experts choose participants. Factors, such as sample size and demographics, are very important. Usually, ESM studies test the same participants and their daily lives, with variables that fluctuate over the short term. You might consider even recruiting and motivating your research participants (Conner & Lehman, 2011). Having intrinsically and extrinsically motivated participants is one of the factors that will influence your results positively. Rewarding your participants might be extremely helpful.
Note that when it comes to participants and ESM studies, researchers worry if constant self-reporting can affect respondents’ perceptions. However, we can look at this effect as one of the main advantages of ESM testing. Simply because frequent self-monitoring can be used as a therapeutic intervention, creating reactivity (Barta, Tennen, & Litt, 2011).
When all research plans are finalized, it’s time to give your research endeavors a go. Pilot testing is more than beneficial. Choose a group of people similar to the intended participants of your ESM study. However, note that human beings are unique and what works fine for one sample might go wrong for another.
If you think that pilot testing is just a waste of time, you should think twice. Did you know that Schneider and Waite (2005) did three pilot tests of their national study of families? Pilot testing can give you valuable insights on question-wording, length, and other crucial aspects of your ESM questionnaire. Not only that, though. The post-collection procedures and analysis of data, such as descriptive statistics, can show researchers some problematic parameters of the questionnaire, such as coding, that were not obvious during administration.
Most of all, pilot testing is a great way to connect to all participants. The size of the pilot sample depends on the purposes of the full-scale study. In some cases, 10-20 pilot subjects are enough, in others, smaller samples are also acceptable. Although ESM studies don’t take too much time, real testing might interrupt some important daily activities, which as a result may affect the intrinsic motivation of the subjects. Thus, listening to people’s feedback and analyzing their notes taken during the study is important as it can help researchers improve their ESM study design. For instance, always ask if: response options were meaningful; signals were clear; study situations were difficult to understand; the testing procedure was frustrating;, etc. From giving instructions to signaling methods, pilot tests contain a lot of useful information and may give some humane touch to the ESM design.
Also remember that when you thrive for perfection, you should embrace errors. Focus on missing or non-compliant reports and adopt procedures to battle similar problems that may arise in the future. Ask participants if items were too irrelevant or too confusing. Make sure that the procedure used in the study wasn’t too difficult to administer, or check if signals were unclear. At the same time, create a more realistic testing environment: encourage pilot subjects to give false data or to miss a response from time to time. All these problematic situations are things that could happen even to the best ESM study designs.
Do not forget that research assistants also have to undergo some training so they can relate to their participants and understand the testing procedure better. In other words, even the “pilots” of the study have to go through the pilot testing procedure.
Also, when you choose a tech device for a reporting method, have a spare in case something fails or breaks. The signaling equipment and the procedures for dealing with data are also a subject of errors. Mind that plans don’t always go smoothly and learn from results!
So you have your hypothesis, constructs to measure and samples of interest. As mentioned above, it’s time for you to find the right ESM design. There are two major types: event-based and interval-based designs (Shiffman et al. 2007; Wheeler and Reis, 1991). ESM designs are a crucial part in designing an ESM study.
Before you choose a protocol, try to familiarize yourself with the different types of ESM design in order to choose the most appropriate protocol for your study. Event-based studies focus on various episodes in people’s lives and are effective for the measurement of events, such as panic attacks (Taylor et al., 1990), intake of medicines (Jonasson et al., 1999), cravings (Shiffman et al., 1997), etc. On the other hand, time-based studies test people’s experiences over various events and are effective for the measurement of pain, blood pressure (Kamarck et al., 1998), etc. A lot of medical researchers use ESM designs, finding these studies effective for the measurement of various physiological changes. Of course, there are combined methods that can be used effectively in some complicated research topics. In fact, ESM designs are always sophisticated. We can expand on the topic by dividing the types of design into interval-contingent reporting, signal-contingent reporting, event-contingent reporting, continuous reporting, and combinations of these approaches (Bolger, Davis, & Rafaeli, 2003).
Signal-contingent studies involve the use of a signal after which the participants respond to questions. Signal-contingent ESM studies request participants’ reports at varying times each day. Signals can be given by a beeper or a notification on the phone. Not only the device but the intervals should be considered: fixed times, regular repeating times, random times, or semi-random times. Signal-contingent designs are great for measuring constantly changing variables, such as depressive mood or fatigue. However, studies should not last for more than 1-2 weeks as they might become too burdensome for the participants.
As signal-based studies depend on many factors, such as the frequency of signaling, the duration of the signals, and of course, the length of the study itself, we should go into detail and explain a little bit more about the signal-contingent method. First of all, researchers should consider the main aspect of each ESM study: their participants and response rate. Note that in studies where participants were signaled 1-2 a day, the response rate was around 95%. In comparison, studies with signals more than eight times a day showed that participants tend to respond less, with a response rate ranging between 60% and 80% (Conner Christensen et al., 2003). The duration of the ESM study is also crucial for the design and the planning of all signaling procedures. If your testing forms require more than 3 minutes, do not expect people to fill them out more than six times a day as you’ll easily reach the burden cap in ESM research.
Also, consider all plans for any statistical analysis you may compute. Snijders and Bosker (1999) suggest that one can compute a multilevel power analysis to predict the number of cases necessary to measure within-subjects variance. Do not forget that the statistical power of the ESM study is a major factor to consider.
Interval-contingent reporting requires participants to respond at certain times every day. In other words, interval-contingent studies require reports at the same time each day. This design is perfect for routine activities (e.g., how much water one drinks), or things that are easy to remember. As they don’t require too much time or effort, these studies usually last about 1-2 weeks. However, some researchers worry that such studies are focused on a particular context. Let’s say that you test participants only at 10 A.M. when they are at work. Subject A shows higher levels of depression, while subject B seems more relaxed. So here researchers might face a problem: Can we implement these findings into everyday life and family life, for example? Is subject A always more depressed? We can’t be sure, so instead of making fake generalizations, we should consider combined methods that might be helpful.
Note that daily diaries are a type of interval-contingent study, with answering questions once a day, mainly at the end of the day. This is a great option for asking reflective questions. Studies using daily diaries can last longer as they are not so demanding, they last even about 1-4 weeks. We’ll present a daily diary study at the end of the guide.
Event-contingent ESM studies require participants to respond to questions related to a certain event (for instance, smoking) (Moskowitz & Sadikaj, 2011). They are applied to social interactions, anger management, incidents of conflict, and stress. This ESM design is the most reliable option for special events, which are not frequent. Thus, these studies can last longer – from 1 week to several months. As events are unpredictable, signaling schedules are not needed. However, some researchers find it beneficial to signal participants just to remind them to respond (Cote & Moskowitz, 1998).
Continuous reporting involves monitoring research variables all the time. Although this method can be employed for studying some factors over short periods of time, it is generally used for physiological variables, such as heart rate or temperature. This design can require a lot from participants or can generate large amounts of data (especially when it comes to passive reporting, such as heart rate). Thus, studies that rely on this design usually last not more than a week.
As mentioned above, another alternative is to combine some of these schedules. For instance, one might combine event-contingent reports with a time-based schedule (Shiffman, 2007).
To sum up, let’s present the pros and cons of the three main scheduling methods: event contingent, signal contingent, and interval contingent, as systematized by Fisher and March (2012).
Interval-contingent schedules can be conducted with simple technology. Paper-pencil or computer, this type of research allows the use of various reporting methods. Also, as participants know which times of the day they are expected to respond, they might find this method less frustrating when compared to other ESM studies. However, as mentioned earlier, as interval-contingent studies focus only on certain times, they might miss some crucial information about people’s feelings, behaviors, and experiences that happen at other times. A suggestion for researchers is to use a signal just as a reminder throughout the day. In addition, to avoid memory decay, have participants report morning experiences reported at noon, for example.
Signal-contingent reports capture momentary changes and experiences without memory decay and even ten signals per day might be used in signal-based ESM studies. However, the unpredictable side of the signaling methods might become too burdensome to participants, and also the presence of some kind of signaling technology can be ineffective and even unappropriated (for the workplace, for example). A suggestion for researchers: design the given signals by basing them on the time it takes for each variable to fluctuate.
Event-contingent reports focus on certain events and also eliminate memory errors. However, researchers have no control over the reporting methods. Let’s say that a participant decided to light a cigarette without reporting the event. How can you know about it? Well, a tip is to train participants to recognize which events should be reported and motivate them. As mentioned above, remuneration should be considered.
Sometimes researchers forget about another important aspect: collecting data. Choosing an appropriate reporting method is crucial for each ESM study.
There are various methods that can be used.
Paper-pencil Methods: The paper-pencil method is one of the oldest and simplest methods that combine not only traditions but cost-cutting procedures. However, they are time-consuming as all data should be entered by hand. What’s more, “front fill” or “backfill” effects might also occur and contaminate the data (Stone et al., 2002).
Personal Digital Assistant (PDAs): With the advance of technology, research methods have improved. We won’t exaggerate if we say that PDAs revolutionized research. Having digital data has made collecting results easier, and in addition, backfilling can be eliminated as reports are time-stamped. However, today PDAs are kind of outdated technology, which on top of that requires the participants to carry a device that they wouldn’t usually have with them.
SMS: Messaging is a great reporting method as participants can use their own device (mobile phone, to be more precise), which doesn’t add any additional costs to the study. However, like any other text message, messaging forces limited schedules and limited length.
Emails: Emails, on the other hand, are great for daily dairies. What’s more, they are easy to use, and even older participants have no problems. However, if you decide on emails, remember that these approaches require a good Internet connection. On top of that, emails and diaries can’t detect momentary changes.
Physiological Sensors: If one wants to examine momentary changes, physiological sensors are the best methods. Sensors can detect momentary physiological changes and provide high levels of accuracy. However, they are expensive and require high levels of commitment.
Smartphone Apps: Last but not least, Smartphone apps have become the most preferred reporting method, which is easy, engaging, and entertaining. Okay, apps can’t detect physiological changes, but they are highly flexible and scalable. But, we have got you covered through device integrations. Data reveals that participants show high levels of compliance every time they have been prompted (Collins et al., 1998, Trull et al., 2008).
Emoticons have become an inseparable part of virtual communication. Although people often associate emoticons with online chats or teen forums, emoticons can be a scientific tool. Some studies even implement the use of emoticons to present people’s feelings. In fact, assessing emotions when using technology is difficult, so a new and challenging approach is to use emoticons on mobile devices. One of the main advantages is that emoticons can save space and create the positive user experience.
As mentioned earlier, ESM studies are among the most reliable approaches to study participants’ everyday lives and experiences.
ESM studies are used in a wide range of settings; from studying individual differences to assessing various treatment programs, ESM studies have a lot of applications.
Individual Differences: First of all, ESM studies explore people’s feelings, behaviors, and routines. On top of that, it’s been proven that ESM studies are highly reliable and valid methods. For instance, eliminating recall bias makes ESM really effective, something that can be achieved by using a digital reporting method.
Natural History: It’s not only that, ESM designs can elaborate natural history, with independent variable (IV) time and dependent variable (DV) subject’s intrinsic variation, for instance. That said, ESM designs can be used to study withdrawal symptoms in ex-smokers (McCarthy et al., 2006).
Temporal Sequences: ESM studies can help researchers explore various temporal sequences. In other words, ESM studies also show the connection between dynamic processes and situational cases focusing on the sequence of events. For instance, ESM has been used a lot to study smoking cessation and relapse (Shiffman at al., 1997).
Contextual Associations: One of the many applications is that ESM experiments study the connection between events that are occurring simultaneously, not only their sequence. An interesting application is ESM in studying schizophrenia and the relation between emotions and stressful events (Myin-Germeys et al., 2001). Other symptoms of psychopathology, bipolar disorders, in particular, are also linked to stress (Stiglmayr et al., 2005), and can be studied in depth with ESM studies.
Treatment: ESM studies are very popular within clinical settings. ESM studies can be used for ongoing assessment during treatment of illnesses, such as depression (Kramer et al., 2014). Also, ESM studies are extremely effective when studying bipolar disorders and the dyadic relationships of illnesses and events (Ebner-Priemer et al., 2009).
Intervention: As mentioned earlier, ESM is a revolutionary approach that can be used not only in treatment but intervention programs as well. Some of the applications that ESM studies have can be applied to panic disorders (Newman et al., 1997), addictions (Riley at al., 2002), eating disorders (Norton et al., 2003), and many other health problems.
Organizational Settings: Although many researchers focus on the clinical applications of ESM studies, you can freely use an ESM design within organizational settings (Shrout & Lane, 2011). It’s been proven that ESM studies provide valuable information about people’s productivity, loyalty, and motivation.
So now when we know the main applications of ESM studies let’s have a closer look at some studies that have shown the scientific power of ESM designs.
As mentioned above, ESM studies have some miraculous applications within clinical settings. For instance, ESM designs can be used in the treatment of addictions. Shiffman is one of the influential researchers that have implemented ESM in their practice. In particular, Shiffman (2009) shows how ESM studies can facilitate the recovery of many addicted individuals. Usually, substance abuse is captured by event-based designs (having a drink, for example). By examining cravings and triggers, researchers can understand better why people use drugs. However, when studying addictions based on ESM designs and self-reports, examiners need to consider compliance and accuracy. Will all people report cravings and relapses and are reports given by an intoxicated person valid? That’s said, all data collected during research is meaningful as all addictive behaviors are complex events.
Another important application of ESM is the use of ESM studies in relation to mental illnesses, psychosis in particular. Although ESM studies are very effective in this context, Oorschot (2009) tackles the problem of the accuracy of reports. Just like the reports given by people struggling with drugs, can we trust a report given by someone who might be experiencing a psychotic breakdown? Still, several research teams have provided important insights on the relationship between psychosis, stress, drug abuse, and other deficits in coping with everyday life.
Last but not least, we should focus on the physiological part of ESM methods. ESM studies have given researchers some amazing ideas about the core of many panic disorders and phobias. Alpers (2009) claims, for instance, that monitoring some physiological states is crucial as many anxiety disorders involve bodily processes. Soma and psyche in one!
Last but not least, let’s focus on a classical ideographic study (Allport 1962) as these studies have influenced today’s ESM designs: Lorraine’s Week as presented by Larson and Csikszentmihalyi.
Lorraine (a pseudonym) was a high school senior. She had a psychological trauma during the week of her ESM reports. Lorraine couldn’t realize her plans to go to Spain, and what’s more, she couldn’t join her friends on a trip. Before that, according to her self-reports, Lorraine was very positive – talking to people, watching TV, and going out. After the bad news, her mood changed dramatically and for a short period of time, between 12.15 P.M. and 1.30 P.M., she went from happy to extremely sad. After those traumatic events, her mood stayed low, and she even reported that the pager used for signaling was a nuisance. In fact, she was happy only when she was with someone else. In the interviews after the study, she reported, ‘‘When I was with people, it was better.”
This study took a week, but still, it gave a clear picture of this girl’s everyday life and her experiences. The findings from her daily diary might seem too individual, but that’s not 100% true. In fact, they show a lot about people’s interactions. On top of that, some findings can be generalized and applied to other teenagers – listening to music and going out is something that most teens do.
In the end, by being statistically powerful, each ESM method has a lot of practical strength, and the results can be generalized.
To sum up, one of the biggest advantages of all ESM studies is that many researchers can achieve their goals within various clinical and organizational settings. Having the chance to work across various contexts is beneficial. On top of that, ESM studies have high ecological validity (Csikszentmihalyi et al. 2103; Shiffman et al. 2008). By designing a study that relies on technology, Smartphone apps, in particular, scientists have the chance to approach more people (Intille, 2007). ESM studies collect not only data – they can be used for sharing vital information, such as videos and geographical location.
The only requirement is to keep ESM studies short as they can become too time-consuming. Also, they should not always focus on negative feelings as that negativity might trigger emotional pain or relapse (in the case of addictions). Follow-up interviews can help researchers eliminate this problem. Last but not least, a major concern of many researchers is acceptance and compliance. Thus, training and reward are important.
Interestingly enough, some findings show that participants are more open to ESM studies, especially when the testing methods are based on technology such as smartphone apps, when compared to clinicians and researchers. Thus, having a motivated and trained team is crucial for each ESM study and research in general. In the end, researchers and participants work together and form a single happy team.
Following simple steps can help researchers organize their thoughts and work. Doing research, reporting design, recruiting participants, conducting pilot-tests, focusing on data collection and implementation of findings and results, these are all fundamental steps for the successful design of your ESM study. ESM is a powerful methodology, which on top of that is becoming highly accessible, thanks to the use of Smartphones and apps.
And of course, do not forget about the specific ingredients of each scientific recipe: a drop of creativity and a lot of eagerness to learn from mistakes.
Have a project in mind?
Monday – Friday
9 AM – 5 PM EST
DISCLAIMER: ConductScience and affiliate products are NOT designed for human consumption, testing, or clinical utilization. They are designed for pre-clinical utilization only. Customers purchasing apparatus for the purposes of scientific research or veterinary care affirm adherence to applicable regulatory bodies for the country in which their research or care is conducted.
