What is the immersive news experience really all about?

Introduction

Immersive Virtual Reality (VR) for the portrayal of current affairs stories was first introduced by de la Peña et al. (2010) where participants experienced from a first person perspective a situation based on the interrogation of a Guantánamo Bay prisoner. The scenario was delivered using a head-tracked wide field of view head-mounted display (HMD), with real-time motion capture of the movements of the participant that were mapped to a life-sized virtual body that apparently substituted the person’s real body seen from their first person perspective (1PP). In other words when participants would look down toward their actual body while in the VR, they would see the virtual body instead. The first novel aspect of this was that although participants were seated comfortably in a chair, they experienced the virtual reality as if they were standing in a stress position, alone in an empty cell, but where they could hear an interrogation in the cell next door. Hence their virtual body posture did not match their real posture, but the participants in the case study experienced discomfort, and anxiety due to the harsh interview techniques that they could hear¹. The second novel aspect was that the piece used parts of the transcript of the actual interrogation of a prisoner that was heard as if from the adjacent cell. The approach was referred to as “immersive journalism” a term, which although widely used, is not quite appropriate—since is not the journalism that is “immersive” but the delivery of its results. The basic premise of “immersive journalism” is that people can read news stories, or watch them on TV, but personal and direct experience can never be achieved that way—whereas VR has the possibility to provide this.

In particular VR can deliver four related illusions that together go beyond just 3D surrounding content. The first two are components of what is usually called presence (Held and Durlach, 1992; Sheridan, 1992; Sanchez-Vives and Slater, 2005). We refer to “Place Illusion” (PI) as the illusion of being in the place depicted by the VR, irrespective of what might be depicted as happening there. This is a perceptual illusion, based on the extent to which the system delivers natural sensorimotor contingencies—that is to be able to use our whole bodies for perception in the normal way (Slater, 2009). We use this term rather than just “presence” since the latter has been confounded with many other attributes of an experience, such as the degree of interest, engagement, “flow” and so on. While all of these other attributes are important, we distinguish them from PI, which refers solely to the illusion of being in the place. The “Plausibility Illusion” (Psi) is the illusion that the events depicted in the VR are really occurring (Slater, 2009). This can be facilitated by (i) the virtual world responding to participant actions (ii) contingent events that refer personally to the participant (iii) the extent to which the portrayal of the virtual world and its events conforms to expectations where this is applicable (Bergstrom et al., 2017) and maintains internal consistency (Skarbez et al., 2017). Body ownership is the third illusion, referring to the extent that a virtual self-representation within the VR is illusorily experienced as the participant’s own body (Slater et al., 2010b; Blanke et al., 2015). The fourth illusion, closely related to body ownership is agency, that actions of the virtual body are attributed to the self (Haggard and Chambon, 2012), which may be veridical agency (where the virtual body moves synchronously and in correlation with real body movements) or illusory agency, where actions of the virtual body are self-attributed even when its actions were not those of the participant (Wegner et al., 2004; Banakou and Slater, 2014, 2017). In the case of each illusion, of course participants know that they are illusions, and yet nevertheless their responses at many levels are as if these were real.

The research we describe in this paper started from an already existing VR news scenario known as “We Wait.”² This depicts a situation on a shore in Turkey amongst a group of refugees, waiting for a boat to arrive to take them to Europe. Our goal was to explore whether PI, Psi and body ownership and subsequent follow-up of information about the refugee crisis were impacted by two factors related to the design of the virtual environment. The two factors were “Responsiveness” and “Embodiment.” The two levels of Responsiveness were “None” and “Look at.” In the first the virtual characters did not respond or look at the participant. In the second virtual refugees occasionally looked at the participant when the participant looked toward them. The Embodiment factor consisted of “No Body” or “Body.” In the first if the participant looked down toward themselves they would be invisible, and in the second they would see a virtual body substituting their own. The body was animated only with respect to head position, no hand-tracking was done, but nonetheless it could be seen as substituting the participant’s own body from first person perspective. We used questionnaires to assess the level of PI, Psi and body ownership. In addition, after the experience participants were given a link to a web page where they could follow-up to find out more information about the refugee crisis. We were interested in whether and how many times, and how long after the experience the participants would look at this web page, and whether this was influenced by the factors (Responsiveness and Embodiment).

Our hypothesis was that the responsiveness of the virtual characters towards the participant, and the participant having a virtual body would lead to greater reported levels of PI, Psi and body ownership. Moreover, we were interested to discover how these factors influenced whether participants engaged in follow-up indicated by accessing the web page.

Background

Immersive Journalism

Virtual Reality representation of news and current events typically provides a passive experience where the participant simply observes an unfolding scenario/story, albeit in a 360 degree surrounding world. It is usually video based, sometimes model based, and with stereo vision. Although there has been a long tradition of using VR for narrative, for example, (Pausch et al., 1996), since the work by (de la Peña et al., 2010) there has been a growing interest in and examples of immersive journalism. Nonny de la Peña’s group developed “Hunger in Los Angeles” about events on a food line in Los Angeles which was exhibited at the Sundance Film Festival in 2012. The 2014 World Economic Forum included her “Project Syria” about a bomb explosion in a Syrian town. “One Dark Night” depicted the shooting of teenager Travyon Martin, and “Kiya” about murder in the context of domestic violence. All of these were based on real events, and combined graphics rendering of the scenarios with recorded sounds and other information from the depicted events.

Early in 2014, interest in using 360 video for HMD viewing to create immersive journalism began to grow. For example, Louis Jebb and Edward Miller produced a 360 film of a protest in Hong Kong in 2014 (Hong Kong Unrest). In 2015 the UN sponsored “Clouds over Sidra,” a VR documentary film about a child refugee in the Syrian war, which was created by Gabo Arora and Chris Milk.

Chris Milk’s description of VR as “the ultimate empathy machine” in a Ted Talk in April 2015³ further fuelled interest in VR’s potential for news. Empathy was one of the promises of VR explored by the Tow Report Virtual Reality Journalism: “a core question is whether virtual reality can provide similar feelings of empathy and compassion to real-life experiences” (Aronson-Rath et al., 2016).

Since then, many news organizations and broadcasters including The New York Times, The Guardian and the BBC have developed a number of other 360 degree VR news stories or documentaries. Overall the topic has generated a great deal of interest (Doyle et al., 2016; Slater and Sanchez-Vives, 2016; Watson, 2016).

In “We Wait,” the BBC and Aardman Interactive, experimented with using computer graphics-based VR to put the viewer in a place that few foreign reporters could risk—on a boat with a group of refugees crossing the Mediterranean or waiting on a beach for the boat to appear. The purpose was to create the feeling that the participant was part of the unfolding events. Though scripted, it was based on BBC news coverage of the migrant crisis in 2015. “We Wait” was premiered at the Sheffield Documentary Festival in 2016⁴. It won the Broadcast Digital Award for VR in 2017⁵.

Photorealism was an objective that was unattainable, so the creators chose a low polygon style of animation, with strong art direction to evoke the scenes depicted. The characters in the story were designed to have expressive eyes to strongly convey human emotion in response to gaze interaction (see Figure 1). Motion capture was used to ensure that character movement was as close to realistic as possible. When viewed on a flat screen the characters look like simplistic cartoons. However, in various demonstrations and public exhibitions, these avatars generated an empathic response. This was anecdotal observation though, and part of the purpose of the study was indeed to examine the effectiveness of the scenario.

Learn more: “We Wait”—The Impact of Character Responsiveness and Self Embodiment on Presence and Interest in an Immersive News Experience