Qualitative Measures
Qualitative research methods produce information, which is not arrived at by any means of quantification, such as statistical procedures. Examples of qualitative methods are content analysis (e.g., of written text or thinking aloud protocols), interviews, and ethnographic approaches. Qualitative methods are most often used in exploratory research aimed at generating hypotheses.
Qualitative research typically results in very rich, detailed information, which helps to gain a deeper understanding of the user experience. In this sense, qualitative methods can be a valuable addition to quantitative measures. Another advantage is that qualitative methods do not shape and constrain participants’ responses in the way that questionnaires do, but allow them to choose their own answers, using their own language and terminology (IJsselsteijn, 2004).
The wealth of information which qualitative methods usually produce is not only an advantage but also a drawback, because data-analysis tends to take a lot of time. Also, the central role of the researcher’s interpretation in the data-analysis forms a threat to reliability. A final disadvantage is that it is often difficult to generalize findings.
Sensitivity, reliability, and validity are not discussed in this part of the compendium. Qualitative measures are primarily applied for explorative purposes; therefore these aspects cannot be determined or are not relevant.
Description
Concept: Physical presence or Social presence.
Retaux (2003) defined the feeling of “being in” a game as “performing, concentrating, being implicated, being touched and changing of scenery”. This definition was based on conducted interviews and online definition groups with users of virtual game environments. The “autoconfrontation method” is proposed as a continuous, subjective measure of presence. The user is shown a video of the experience and is asked to explain his/her behaviour and feelings and rate feeling of presence on a 9-point scale. Based on this report, the experimenter constructs a presence chronogram.
Research
The technique was tried out in an experiment investigating the relation between experience task type and presence. Participants (n=15, within-subjects design) performed a total of 12 training games and 2 test games, distributed over 2 sessions. Afterwards, users viewed the videotape of the test game (lasting 5 minutes) and were asked to rate and verbalize their feelings of presence. Based on this report, the experimenter constructed a presence chronogram. No difference was found between presence scores in the 2 test games (this was expected because of difference in experience). A significant difference was found between presence scores during 2 different activities: searching for equipment and fighting an enemy.
Primary Source
Description
Concept:Social presence.
Rourke, Anderson, Garrison, & Archer (1999) identified 12 indicators of social presence in 3 categories based on previous research, literature, and analysis of transcripts:
- Interactive responses: Continuing a thread, Quoting from other’s messages, Referring explicitly to each other’s messages, Asking questions, Complimenting/expressing appreciation, Expressing agreement.
- Affective responses: Expression of emotions, Use of humor, Self-disclosure.
- Cohesive responses: Vocatives, Referring to group using inclusive pronouns (e.g. we, us), Phatics/salutations.
Transcripts of (online) communication are coded on the basis of a template. “Social presence density” is calculated by summing the raw number of instances then dividing them by the total number of words. The template contains the 12 indicators of social presence and is given in the paper, along with examples.
Research
The method was tried out on transcripts from two graduate courses (n=31, number of messages =134). Average interrater reliability was .91 - .95 (although much lower for latent indicators such as humor). Social presence density was higher in one transcript, which confirmed intuitive impressions.
Primary Source
Description
Concept:Physical presence.
Ethnographic research focuses on the exploration of sociocultural phenomena. In general, its aims are descriptive rather than analytic. It is a suitable tool for hypothesis generation. It does not consist of one method, but a combination of methods is used to study naturally occurring behaviourr. Ethnographic methodology can be applied in presence research. Methods that have been used are open-ended questionnaires, unstructured interviews, and observation. The latter two can be used to gather data during the experience.
Research
McGreevy (1992) used ethnographic methods to study the sense of presence of planetary geologists (n=2, within-subjects design) while exploring a desert location. They performed similar tasks both with and without a head-mounted video camera/display/recorder replacing natural vision. They were observed and interviewed (unstructured) during this experience, while performing “typical tasks”. Resulting data were narrative descriptions and explanations, and information about gestures and performed actions. When wearing the HMD, the geologists commented on the poor resolution, the loss of context as a result of narrow field of view, poorer sense of depth perception, and several other factors complicating their tasks.
Primary Source
- McGreevy, M. W. (1992). The presence of field geologists in Mars-like terrain. Presence: Teleoperators and Virtual Environments, 1, 375 - 403.
Description
Concept: Physical presence or Social presence.
Gaggioli, Bassi, and Delle Fave (2003) propose to study the impact of VR on daily life and subjective experience from a theoretical perspective that stresses the active role of individuals in interacting with their natural and cultural environment. Cognitive, motivational, and affective components are seen as relevant to this experience.
The experience sampling method (ESM) constitutes repeated on-line assessment of the external situation and personal states of consciousness, as real events and situations occur. Participants carry a beeper, and fill out a form on receiving a signal. Form contain open-ended questions addressing place, activities, social context, thoughts, goals, etc., and 0-12 Likert scales regarding the 3 components of quality of experience: affect, activation, and cognitive efficiency. An Experience Fluctuation Model has been developed to analyze the results.
Research
ESM has not yet been used to study presence, but is suggested for investigating the multidimensional structure of presence, use of VR in daily life, and comparison across different media. An experiment is planned in which ESM results will be compared to ITC-SOPI.
Primary Source
Description
Concept:Physical presence or Social presence.
Focus group research involves organized discussion in small groups (4-8 individuals), in order to gain information about their opinions and experiences. Audio recordings are generally transcribed and coded manually, possibly with the aid of a software program. Focus groups are useful when “there is a gap between professionals and their target audiences and when investigating complex behaviours and motivations”. In Freeman and Avons (2000), a discussion guideline containing questions and a standard set of prompts are reported.
Research
A focus group study was conducted to elicit non-expert descriptions of 3DTV. Participants (n=25, divided over 4 groups, within-subjects design) were shown the same stimulus materials on both a normal and a stereoscopic TV. All groups reported “a sensation of being there” whilst watching 3DTV, without prompting, but the term presence was not used. Several comments are reported that link this sensation with physical responses (postural responses, wish to interact). There was consensus among groups that factors realism, naturalness and interest/involvement made them feel more present. There is some qualitative support for the multidimensional view of presence.
Primary Source
Description
Concept:Physical presence or Social presence.
Participants are asked to give a retrospective, written description of an experience. These essays are then analyzed.
Research
In an exploratory pilot study into sense of place, Turner et al. (2003) asked participants (n=18) to provide a 150-350 word written description of a familiar place, “as if telling a friend about the experience”. Descriptions were analyzed against the concept of place identity as described in literature, and practical dimensions relevant to the particular project. Striking differences were observed in the overall richness of description. The visual modality was predominant in the descriptions. All three dimensions of place identity (physical setting, activities, meanings and affect) described in literature were found in the descriptions.
In a second study, (n=31), participants experienced either a virtual version of a botanical garden (27 participants) or the real environment (4 participants). They were asked to describe the place to an interviewer, as if telling it to a friend.
Murray et al. (2000) have used this approach in a study investigating the effect of hearing loss on sense of presence. Participants (n=15) carried out everyday activities on the university campus while wearing earplugs. Afterwards, they were asked to provide a written account of the experience. The accounts contained many, varied descriptions of participant’s feelings, which indicate a loss of sense of presence.
Primary Sources
- Turner, S., Turner, P., Caroll, F., O'Neill, S., Benyon, D., McCall, R., et al. (2003). Re-creating the Botanics: Towards a sense of place in virtual environments. Paper presented at the 3rd UK Environmental Psychology Conference, Aberdeen, 23-25 June 2003. [pdf, 420 KB]
- Murray, C., Arnold, P., & Thornton, B. (2000). Presence accompanying induced hearing loss: Implications for immersive virtual environments. Presence: Teleoperators and Virtual Environments, 9, 137-148.
Description
Concept:Physical presence.
Interaction analysis is a qualitative method for the systematic analysis of action, based on discourse analysis. Videotapes of participants’ interaction with technology are recorded. Fragments which contain occurrences of the studied phenomenon are extracted and coded. Single actions are analyzed, paying attention to order and context.
Research
Examples of various research projects are discussed in the paper. In total, 20 videotaped sessions of users immersed (HMD) in a virtual library involving an agent were analyzed. The following questions were considered: what actions do participants envisage, what do virtual objects look like for participants, what norms regulate the organization of a VE, which resources are imported in the VE, how rich is the environment in terms of projected activities?
Primary Source
Description
Concept:Physical presence or Social presence.
There are many different ways of interviewing; the semi-structured or unstructured approach appears to be most suitable for exploring presence, because it allows participants to express their personal experience, in their own words.
Research
This method was used by Murray et al. to investigate the influence of hearing loss on presence. Participants (n=6) wore earplugs while performing everyday activities such as shopping. Afterwards, a semi-structured interview was conducted. Interviews were transcribed, and significant themes were identified. Participants reported a heightened awareness of self, a feeling of remoteness of being removed from the activities around them, heightened environmental awareness, a decreased feeling of social presence. They reported no doubts of actually being present in the environment.
Primary Source
- Murray, C., Arnold, P., & Thornton, B. (2000). Presence accompanying induced hearing loss: Implications for immersive virtual environments. Presence: Teleoperators and Virtual Environments, 9, 137-148.
Description
Concept:Physical presence.
This methodology is based on the view that a sense of place is an important aspect of presence. The approach is partly based on Relph’s (1976) model of place, which defines three components of “place identity”: physical setting, activities afforded by the place, and meanings attributed to the place. One of the goals of the presence probe is to be able to compare (benchmark) real and virtual environments. It has been inspired by Gaver’s (1999) cultural probes approach.
The presence probe consists of several sections:
- “Visitors book” in which participants shortly describe their experience of the visited place.
- Participants are asked to sketch a map of the visited place
- 3 sets of 3 semantic differential items, combining Relph’s conditions of place (physical features, activities afforded, affect engendered) with Osgood’s axes of semantic differentials (good-bad, strong-weak, active-passive).
- Participants are asked to pick a photograph that best exemplifies their experience, and to write down 6 words which best describe their experience of the place.
Research
The presence probe was tried out in a pilot study on three locations in Prague. Per location, 30-40 visitors of the locations were asked to complete the presence probe. Results suggest participant understood the probe very well. Rich, diverse responses were gathered, varying in level of detail. Examples of responses are given in the paper.
Primary Source
- McCall, R., O'Neill, S., Carroll, F., & Benyon, D. (2004). The Presence Probe. Paper presented at the Workshop on Designing and Evaluating Virtual Reality Systems, University of Notthingham, UK.
Description
Concept:Physical presence.
The repertory grid technique (Kelly, 1955, in Steed, and McDonnell, 2003) is a method for exploring personal constructs. It was originally developed to explore constructs about interpersonal relationships in a clinical context (psychotherapy). It has since been applied in many other contexts. The underlying idea is that people represent their experiences by placing alternative constructions upon them. Steed, and McDonnell (2003) have applied the approach in presence measurement.
A participant is presented with different experiences (e.g., VE’s). Afterwards, personal constructs are elicited by comparing different combinations of three elements (experiences). Participants are asked to compare the items of a group and to discuss differences and similarities. Any distinction that is relevant to the participant becomes a construct. A grid is drawn up in which the elements form columns and the identified constructs form rows. Descriptions of the poles of each construct label both ends of a row. Participants then assign a rating to each construct for each experience. Elements and constructs can be clustered, revealing a pattern of personal meaning. Conversation with the participant is central to the technique.
Research
Two exploratory pilot trials were conducted using repertory grid analysis. In the first trial participants (n=3, within-subjects design) experienced 6 different environments: 3 experiences were immersive (HMD), 3 were not (desktop). Content and design also varied. Participants generated 7-8 constructs; there was similarity between the constructs generated by different participants. In the second trial, participants (n=5, within-subjects design) again experienced 6 different environments; however, this time all experiences were immersive and the environments were more similar to each other. The SUS questionnaire was completed for each environment. Although the results of both techniques (RGA and SUS) could not be compared, the constructs generated by the participants could be used to generate hypothesis about the causes of presence. The authors suggest that RGA could be used as a tool for constructing or refining questionnaires.
Primary Source
Description
Concept:Physical presence.
The thinking aloud method requires participants to verbalize their thought processes, perceptions or feelings during an experience. These think-aloud protocols can be analyzed afterwards. The method is well-established in psychological research on problem solving, and is also used in usability testing.
Research
Turner, McGregor, Turner, and Carroll (2003) used the thinking aloud method in a study investigating presence and sense of place in a soundscape. A soundscape of a specific environment (a computer centre) was created. Participants (n=40, between-subjects design) were assigned to one of four conditions: 1. physically present in the real environment, 2. physically present in the real environment and thinking aloud, 3. Blindfolded and exposed to the soundscape, and 4. blindfolded and exposed to the soundscape and thinking aloud. Participants in all conditions completed the SUS. The think-aloud protocols yielded rich and interesting quantitative date. However, scores on the SUS item relating to really being in the computer centre indicated that the speaking aloud task interfered with the feeling of being there.
In a different study (Turner et al., 2003), participants (n=31, between-subjects design) experienced either a virtual version of a botanical garden (27 participants) or the real environment (4 participants). They were asked to provide a running commentary as they explored the real/virtual environment. There were considerable differences in the richness and level of detail of the verbal reports. Participants in the virtual world commented on the absence of objects or sensations typical to the environment, and their restricted ability to perform actions. The real world stimulated much more reference to memory than the virtual world.
Primary Sources
- Turner, P., McGregor, I., Turner, S., & Carroll, F. (2003). Evaluating soundscapes as a means of creating a sense of place. In E. Brazil, & B. Shinn-Cunningham (Eds), Proceedings of the 2003 International Conference on Auditory Display, 148-151. [pdf, 792 KB]
- Turner, S., Turner, P., Caroll, F., O'Neill, S., Benyon, D., McCall, R., et al. (2003). Re-creating the Botanics: towards a sense of place in virtual environments. Paper presented at the 3rd UK Environmental Psychology Conference, Aberdeen, 23-25 June 2003. [pdf, 420 KB]
|