BBC 360° Video Evaluation for MSc Dissertation
Selected in CHI '17 Late-Breaking Work
Category for cutting edge or emerging work that has not been fully realized or developed
Techniques Used
SPSS (Quantitative Analysis) Controlled Lab Experiment
Thematic Analysis (Qualitative Insight) Wireframing
This independent project was done in collaboration with BBC for my MSc in Human Computer Interaction dissertation. The BBC 360° video team wanted to gain quantified insights on their video content and their UX design. They had been running informal qualitative studies, and wanted to further evaluate their editorial practices in more depth.
What is 360° Video?
360° videos allow the user to control their viewing direction much like a panorama. There are three available platforms for viewing 360° videos that pose different user needs and requirements:
1) Magic window (MW) on mobile phone
Viewers physically move the smartphone or tap their fingers to pan 360° around the environment
2) Desktop video
Viewers use their mouse or overlaid arrow keys to navigate 360° around the environment
3) Head-mounted display
Viewers wear a Google Cardboard or Samsung Gear VR etc. using a mobile phone, turning the experience into VR
Client Needs
While 360° video and VR technologies have been catalysts for new forms of immersive journalism, there are no established editorial guidelines and viewing instructions are often unsubstantiated.
I met with various BBC 360° video producers to gain a better understanding of how they want to develop their content in this emerging field. Their main focus is to determine how this technology can complement the kind of journalism, narrative, and educational content they produce. Specific challenges include narrative continuity, directing user attention, and sustaining user interest past the initial 'wow' moment.
Based on these conversations, I identified areas for a quantitative study.
Is Cardboard the Best Immersive Experience?
News outlets like the BBC and NYT were extolling Cardboard viewing as the “best immersive experience.”
However, there had been no prior research that supported Cardboard as the most immersive.
Do Headphones Increase Immersion?
News outlets were also prompting the viewer to wear headphones for Cardboard and MW.
Although there is evidence that supports headphones increase immersion and attention for solely audio, there had been no prior research that supported headphones as increasing immersion for 360° videos.
Does Empathy Increase with Immersion?
VR leaders such as Chris Milk of Within were calling VR the ultimate empathy machine, when this hadn't been supported by any research.
Institutions such as Stanford’s Virtual Human Interaction Lab are exploring VR’s potential empathy impact using Oculus Rift. But the majority of audiences don't have access to such expensive devices.
Gaps in Research
This highlighted the need to test empathy's impact with more accessible 360° platforms that use mobile phones: Cardboard and MW. Mobile phones are also becoming the #1 device for news; Reuters reported that in the UK, smartphone news-use has risen to a record 46%, with many publishers reporting the majority of traffic from mobile devices.
Study Design
The following research questions (RQ) were created to fill gaps in existing research:
I designed and conducted a quantitative, controlled lab study measuring Immersion scores and State Empathy scores. Qualitative data was additionally collected through Demographic Survey and Semi-Structured exit interviews to give the statistical results richer context.
Participants
A total of 40 participants were recruited via opportunity sampling: 20 females, 19 males, and 1 non-binary individual. The majority of participants were 25-34 years old.
Design
This was a 2 x 2 between-subjects study. The two independent variables were the viewing platforms, Google Cardboard and MW. Dependent variables were the Immersion scores and State Empathy scores.
The study design was strengthen by making it a:
1) Between-subjects design to avoid order-effects
2) Matched pair design to control possible confounding variables (i.e. even amount of females and males per condition)
There were 10 participants per viewing condition:
Apparatus and Materials
Devices
• iPhone 6 • Sony MDR-ZX300 Headphones • Google Cardboard
Questionnaires:
• State Empathy Questionnaire • Film Immersion Experience Questionnaire (IEQ)
Video
Participants watched Fire Rescue, a 5:50 minute BBC produced video, about firefighters rescuing six children from a house fire on Christmas night. The video experiments with first POV and fade out cuts.
Pilot Study
The pilot study highlighted overlooked details and areas of participant confusion. Addressed issues and iterations include:
Issue Iteration
Participant had difficulty using Cardboard as a spectacle wearer
Matched pair design was utilized to ensure spectacle wearers were only in the MW condition
Participant was confused whether she could choose multiple options on the Demographic Questionnaire
Instructions to “Choose as many as you like” were added to applicable questions on the Questionnaire
The combined Film IEQ and State Empathy Questionnaire was not segmented, and caused the participant to hesitate in confusion
Questionnaires were segmented into Part 1 and Part 2 for clarity
Results
Quantitative Analysis
Film IEQ scores and State Empathy scores were calculated and imported into SPSS. A Two-Way Between-groups Analysis of Variance (ANOVA) was conducted to determine whether conditions had significant difference and to answer the RQs.
Yes, Cardboard’s Immersion range and mean (M = 150.35) was higher than Magic Window’s Immersion range and mean (M = 140.55), with two outliers in the Cardboard condition.
Order of conditions with the highest to lowest immersion scores:
1
M=155.00
2
M=147.60
3
M=145.70
4
M=133.50
However, there was no significant difference for mean immersion score between viewing devices, nor between with/without headphones.
There was only significant difference for the interaction effect between headphones and device (p = 0.039). While headphones increased immersion for Cardboard viewing, it decreased immersion for MW. An ANOVA was run again removing without the two outliers’ data, and there was still a significant interaction difference.
Insight
Headphones may be a hindrance for MW viewing.
MW's video screen is still juxtaposed with physical reality. The combination of the physical reality around participants and the predominant preference of not wearing headphone could be the reason why headphones decreased immersion for MW.
For Cardboard, hearing environmental sounds while your field of vision is consumed by the virtual world is most likely disconcerting. Since the viewer's field of vision is already consumed by the virtual world, it makes sense to have contained audio.
MW had the higher State Empathy scores, although there was no significant difference for State Empathy score in any of the independent variables. Similar to Immersion scores, the addition of headphones increased State Empathy scores for Cardboard viewing, but decreased State Empathy scores for MW viewing; however, this interaction effect was not significant for Empathy.
Order of conditions with the highest to lowest State Empathy scores:
1
M=2.79
2
M=2.69
3
M=2.67
4
M=2.38
State Empathy Scores on MW may have been higher because it's more akin to a typical video, in which participants can focus more on the story. The exit interviews revealed that with Cardboard, participants were preoccupied with the novel viewing platform and less focused on the story.
A Pearson correlation was computed to assess the relationship between immersion and empathy scores. There was a moderate positive correlation between the two variables (p < 0.001).
Insight
Increases in immersion are correlated with increases in empathy! This supports existing work out there, and encourages 360° video makers to continue striving for immersion.
Qualitative Thematic Analysis
Four main themes were identified from the 40 semi-structured exit interviews:
1) Straining to Focus
A. Dizziness (60% of participants)
B. Fear of Missing Out (FOMO) (35% of participants)
2) 360 Video as Information Communicator
A. Today I Learned (60% of participants)
B. Medium Distracting from the Content (33% of participants)
3) Vicarious Fear (10% of participants)
4) User Preferred Content
"And I’m kind of terrified about the fire, and the background music and audio. I’m in a safe environment, but I feel like I was in that situation.”
- P15
“I don’t think I heard much of the information because I kept on scrolling around to look at the different scenes. … [I] have a stronger impression about the environment than the story"
-P13
“It was cool, I was just getting dizzy looking around anywhere. At times I didn’t know where to look.”
- P20
"I felt disappointed! That I might have missed something, in terms of visual information."
- P31
Insight
Specific genres may be better for 360° video format, such as explorable environments.
Design Recommendations
I created the following wireframes based off user feedback. I presented a formal, executive summary of my work, results, and design recommendations to the BBC team.
Navigational Cues
Navigational cues could assist with FOMO.
Inspiration can be taken from hypervideos, which have navigational overlays and hotspot indicators informing the user of points of interest (Chambel et al.).
Some participants re-watched Fire Rescue and exclaimed that they didn't notice the firefighters sliding down the pole.
This wireframe shows an example of how a view area & orientation pie with hotspot availability on the pie and the video can direct attention towards the right, where there's more action.
Mini map on the bottom of the video could address FOMO, as the whole scene is laid out.
A mini map cursor can control the movement, in addition to the mouse and arrow keys.
P31 actually suggested, “What if you had a small map on the bottom to show where to look? Like in a video game, I think that would help.”
Shifting from Viewer to Visitor
Taking note from Jessica Brillhart of VR at Google, we should think about the user as visiting the virtual environment. We can borrow from AR to achieve this.
BBC Taster experimented with a "stare to activate" interface. This could be used for information points and choosing to proceed to the next scene from a firefighter's POV.
Implementation
BBC released 360° video for 2016 Rio Olympics that featured a navigational sphere on the bottom right.
360° videos of the very popular Planet Earth II series reflects what participants wanted to watch, stunning nature videos.
