Introduction

The Likert scale and conventional pretest-posttest selfreports are two popular methods utilized by researchers to measure respondents’ experiences Renis Likert introduced the response scale in 1932, and it is still currently the most popular response format (Hodge & Gillespie, 2003). Likert scales make the respondent specify his or her experience by selecting one graduated category from the many category options, with Likert scales usually containing four to eleven categories (Flynn, 2004).

The categories are anchored by verbal descriptors (e.g., agree, disagree). In a similar vein, the conventional pretest-posttest self-report measurement design is by far one of the most popular methods to collect and analyze change data (Sprangers, 1989). After an individual participated in a treatment or experience, it is important to estimate the “true change” which occurred in the individual (Cronbach & Furby, 1970). However, both the Likert scale and the conventional pretest-posttest self-report have limitations.

The Likert scale’s artificial categories are not sufficient to capture a continuous phenomenon (Joyce, Zutshi, Hrubes, & Mason, 1975). The conventional pretest-posttest self-report is limited in its assessment of change, and does not capture the full change story of the individual (Aiken and West, 1990). Two ways to improve upon the Likert scale and conventional pretestposttest self-report measurement designs are through visual analog scales (VAS) and retrospective pretestposttest self-report designs. Hayes and Patterson (1921) were the first to use and describe the VAS (Couper, Tourangeau, Conrad, Singer, 2006). The VAS is a horizontal continuous scale with verbal anchors on the extremes of a continuous line. Respondents mark a point on the continuum that best represents how they feel. The distance between the marked point and the origin of the line is measured to quantify the magnitude of the response. In Figure 1, the respondent indicated feeling about 32% of the worst pain possible.

Reasons to use the VAS as opposed to the Likert scale.

The VAS allows for finer distinctions than the Likert scale, providing a greater amount of information to the researcher (Aitken, 1969; Rausch &Zehetleitner, 2014). With computer aided technology, respondents can freely specify the exact position of their response which can subsequently be quantified up to the pixel With Likert scales, respondents are limited to only certain categories on the continuum (Flynn, 2004). The increased sensitivity has been reported by researchers. When Joyce, Zutshi, Hrubes, and Mason (1975) measured pain on a four-point Likert scale and a VAS, the researchers concluded the VAS displayed more sensitivity. Neely and Borg (1995) reached the same conclusion when measuring perception of color change using both the Likert and VAS response formats. Compared to the Likert scale, the visual analog scale allows for true interval-level data while the Likert scale only allows for ordinal-level data.

The categories of Likert scales cannot be assumed to be equally spaced. For example, the distance on the continuum between ‘somewhat agree’ to ‘strongly agree’ might not be the same as the distance between ‘somewhat disagree’ and ‘undecided’ (Hodge & Gillespie, 2003; Svensson, 2001).The lack of precision of measurement means the data from Likert scales are at the ordinal-level, so it is not technically appropriate to sum or average such responses, even though researchers often do so ( Contrasted against the Likert scale, the visual analog scale provides a smoothed continuum for respondents (e.g., a score of 100 is twice the value of a 50 and is at the interval level). Also, some respondents find the VAS more intuitive (Little & McPhail, 1973; Aitken, 1969). For example, Zeally and Aitken (1969) concluded the VAS was the simplest method for their hospital patients.

Other researchers also reported that children preferred the VAS (Abu-Saad, Kroonen, & Halfens, 1990; Shields, Palermo, Powers, Grewe, & Smith, 2003).

The biggest limitation of the VAS are related to the reliability of the scoring protocol (Bijur, Silver, & Gallagher, 2001). However, due to the recent advancements in technological capabilities, these limitations are negated by modern survey software.Historically, the VAS has not been widely used due to the practical limitations related to scoring (Couper, Tourangeau, Conrad, & Singer, 2006; Kersten et al., 2012). Researchers had to first measure the distance from the respondent’s mark to the origin of the line with a ruler. Next, researchers needed to transcribe the distance for the data-entry; not only was this task timeconsuming but the process was prone to error (Haefeli, & Elfering, 2006). Today, software can record the location of the click of the mouse automatically with no extra effort by the researcher.

Despite the aforementioned advantages of the VAS, researchers have not always reported positive results. Even though it is more precise, some researchers who have compared the VAS and the Likert scale reported that respondents could not distinguish more than seven to eleven different categories while using VAS (e.g., Thomeé et al., 1995; Munchi 2011).Also, some respondents (e.g., children and the elderly) prefer the Likert scale. (Couper, Tourangeau, Conrad, & Singer, 2006; Joyce, Zutshi, Hrubes,, & Mason, 1975; Paediatr, 2004). The elderly described the difficulty in using VAS as requiring them to convert their responses using mathematical logic. Additionally, elderly participants reported that the VAS required more time to respond compared against the Likert scale (Joyce, Zutshi, Hrubes, & Mason, 1975). Although these criticisms have been identified in the literature, the documented benefits of the VAS scaling outweigh the potential problems. Furthermore, additional researcher training on the use of the VAS protocol can eliminate the potential scaling problems.

Reasons to use the retrospective pretest-posttest as opposed to conventional self-report methods

The second suggestion to improve the psychometric properties of traditional response formats is through retrospective pre-test designs. Conventional pretestposttestself-report measurements are frequently used to evaluate a treatment, intervention, or change in experience. Participants are given a questionnaire prior to receiving treatment, and are presented with the same questionnaire at the conclusion of the treatment.

The conventional pretest-posttest self-reports are contextually helpful in examining implicit attitudes or sentiments that individuals might not feel comfortable or able to articulate (Cohen, 2014). Yet, despite the frequent usage of conventional pretest posttest selfreport measures, conventional self-report designs have methodological limitations.

The most discussed and problematic limitation associated with conventional pretest-posttest selfreport designs is response-shift bias. Response-shift bias can be defined as an instance when an individual changed his or her perception or understanding of their initial functioning in response to a treatment (Howard, 1980). However, after participating in a program, participants have gathered more knowledge about the construct and are therefore more suited to give an accurate report. Yet, the response on the posttest will be answered from a different frame of reference compared against the pretest reference frame (Drennan & Hyde, 2008; Howard, Dailey, & Gulanick, 1979; Howard, 1980). The internal axes of reference for the pretest and the posttest differed within the individual when they completed the selfreport measures. Therefore, it understandable why comparisons between the two self-reports are not appropriate (Bray, Maxwell, & Howard, 1984).

The concept of examining response shift bias as a latent variable meta-construct of three interrelated processes has been proposed by researchers (Schwartz and Schwartz, 1999; Sprangers, Carey, & Reed, 2004). The processes of the response shift bias meta-construct are categorized as recalibration, reprioritization, and reconceptualization Recalibration refers to an individual’s changes his or her internal standards; an indication of this response shift effect occurring would be when the individual changes his or her meaning of the target construct.

When an individual has a change in his or her values or priorities of the target construct, this change suggested a reprioritization response shift has occurred. An individual’s change in his or her definition of the target construct. To test the degree of the response shift bias, particularly the three specific interrelated constructs of response shift bias, a latent variable modeling approaches such as factor analysis, confirmatory factor analysis, and longitudinal structural equation modeling have been proposed to measure response shift bias and its effect on the target construct of interest. For further discussion of these methods, please refer to Oort, Nieuwkerk, and Sprangers (2001), Schwartz et al. (2004), and Oort (2005).

To overcome some of the limitations imposed by conventional pretest-posttest self-report measures, the retrospective pretest-posttest design should be utilized, The retrospective pretest-posttest is specifically designed to control for response shift bias by accurately depicting change in an individual after a treatment or intervention (Howard, Dailey & Gulanick, 1979). To illustrate an instance of this design, image a group of individuals has participated in a certain type of treatment, however, they do not fill out a survey prior to the start of the program. Once the program has concluded, participants completed a posttest survey and a retrospective pretest.

The retrospective pretest allowed for the individuals to consciously reflect back to their state prior to the start of the program, and determine whether they have undergone some change in knowledge or attitude (Cohen, 2014; Davis, 2002; Howard, 1980). The posttest and retrospective pretest are taken at the same point in time and are responded to using the same internal frame of reference for both self-reports. With the posttest and retrospective pretest being answered from the same point of reference, comparisons of change can be determined from self-reports. As Hoogstraten (1985) and others have noted, the key benefit of the retrospective pretest-posttest designs over conventional pretest-posttest self-reports is the elimination of the validity threat due to response-shift bias. Additionally, individuals tend not to overestimate or under-report their emotions and knowledge of their behaviors on retrospective self-report measures. Breetvelt and Van Dam (1991) conducted a study on a sample of cancer patients and demonstrated that there is an underreporting with measures of emotional behavior or attitudes. Because of this underreporting bias, the researchers advocated using the retrospective self-report design to control for response-shift bias. Retrospective pretest designs are convenient to assess change of knowledge or attitude in an individual. Lastly, retrospective self-reports are extremely flexible since questions can be formulated to actually reflect the program content as it evolves over time in the program (Pratt, McGuigan & Katzev, 2000).

For further discussion of the retrospective methods, and application refer to Howard et al. (1979), as well as Howard (1980), Nakonezny, Rodgers, and Nussbaum (2003), and Nakonezny and Rodgers (2005).

We have discussed why retrospective pretest self-report measures are an effective method of capturing change in response scales, especially after an individual has participated in a program or treatment. Yet, retrospective designs do have a few limitations that need to be mentioned. Drennan and Hyde (2008) noted how retrospective pretest self-report measures could suffer from social desirability and impression management. Demand characteristics and memory related problems have also been shown to influence the recall process in retrospective pretest selfreports (Pratt et al., 2000). Howard, Millham, Slaten and O’Donnell (1981) investigated social desirability and demonstrated that retrospective pretest-posttest selfreports actually diminish the effects of social desirability in participant responses.

Despite the noted methodological limitations of selfreports, the retrospective pretest is a valuable strategy to control for response-shift bias and underestimation or overestimation of program effects. Schwartz et al. (2004) validated the retrospective pretest to be able to detect recall bias and recalibration bias. They compared the retrospective pretest results against covariance analytic approaches to see how the three different aspects of response shift bias interrelate. The results of the study were not redundant; they found that individuals changed their internal standards over time. Additionally, this finding is compounded by the authors’ suggestion of measuring response shift bias using the retrospective pretest-posttest coupled with latent variable modeling methods.

Likert scales and conventional pretest-posttest self-report measures are certainly the dominant methods used in measuring subjective experiences in contemporary social and behavioral sciences. Yet, these techniques are not sufficient in capturing the true experiences of an individual. To capture and measure change after a proposed treatment, researchers should consider incorporating both VAS and retrospective pretest-posttest self-reports in their research designs.

Research such as this has shown that retrospective pretest designs can overcome the limitations of conventional pretest-posttest self-reports, particularly the threat of response-shift bias. However, retrospective pretest self-reports are not welcomed in all research circles, with points of contention being related to philosophical objections (Howard, 1980). On the contrary, we suggest that researchers should consider implementing both VAS and retrospective pretest-posttest measurements in future research. We also encourage researchers to think innovatively in development measurement systems that are highly sensitive to changes in the level of a construct and to changes in the construct over time.

References

Publication Dates

History