This section examines previous work executed and leading to the space of 360° video particularly.
Wider Than Wide Screen
It started with the Panorama, which was invented (and coined) by a painter named Robert Barker. The Panorama has a long and varied history which need not be explored. It motivated advancements of panoramic film cameras (such as the Al-Vista, The Eastman Kodak No. 1 Panoram, or the KMZ Horizon cameras. Most panoramic film cameras used a swinging lens or spinning body method, neither of which suited themselves to be easily adapted for motion film when such technology came along. To capture extremely wide scenes, filmmakers either used ultra-wide angle lenses or multiple film cameras which they stitched together. The most famous, or at least the most grandiose, example of this method was perhaps the Cinerama, a sort of predecessor to IMAX-level aspirations of picture size and scale. The first Cinerama film released to tell a fictional story was Oscar-winning "The Wonderful World of the Brothers Grimm", shot in 1962. ("How The West Was Won") was filmed first, but released later.
After these two films were released, the 3-camera technique for Cinerama was abandoned in favor of single lens one. This is for a variety of reasons, from logistical to cost-cutting. One notable reason of the forfeiture of this method is that, because of the geometry of the cameras and how the three images were projected, actors on may not appear like they were looking at each other. For How The West Was Won, the actors had to cheat their eye line inwards to the camera, and pretend they were looking at the fellow actors. While adapting these films for non-curved screens, this perspective distortion is all the more a challenge - one that filmmakers are still faced with today, filming with such wide-angle lenses.
Storytelling and VR
But what about Narrative?
First, the work of Austin Baur, whose undergraduate honors thesis is titled "Exploring Cinematic VR: An Analysis of the Tools, Processes, and Storytelling Techniques of Virtual Reality Filmmaking." His work is a highly pragmatic examination of 360 video, and includes very insightful interviews and examinations with creators, dissecting many of the decisions that creators have during the process. This work repeats many of his conclusions and goes into more detail than I do in other areas. I highly recommend reviewing it in tandem with my work here.
A Historical Perspective
Storytelling is a relatively new endeavor for 360 cameras, but not for virtual reality. A common thread found in articles from the 90's discussing the future of VR is how it was positioned hand-in-hand with interactive fiction. VR has always had an intrinsic link to interactivity, and the idea to use such systems to tell passive stories would have been hard to imagine without the existence of competent 360 video cameras to film with. Computer driven animation was still in its infancy - Pixar only released it's first short, The Adventures of André and Wally B in 1984.
"Early sensory display experiences included the Sensorama. The Sensorama was the brainchild of cinematographer and inventor Morton Heilig. Demonstrated in 1956, Sensorama was a scripted multimodal experience in which a participant was seated in front of a display screen equipped with a variety of sensory stimulators. These stimulator displays included sound, wind, smell, and vibration. The noninteractive scenario was driving a motorcycle through an environment with the appropriate stimulators triggered at the appropriate time. For example, riding near a bus exposed the rider to a whiff of exhaust.
The Sensorama system, however, was lacking a major component of the modern virtual reality system: response based on user’s actions."
It is worth noting that the Sensorama was the first interactive multimodal (multi-sensory) passive entertainment system, it's films were not narrative but offered "adventures in surrogate travel" (Craig, p. 238).
Disney Imagineering created "a high fidelity virtual reality (VR) attraction where guests fly a magic carpet through a virtual world based on the animated film "Aladdin. Unlike most work in on VR, which has focused on hardware and systems software, we assumed high fidelity and focused on using VR as a new medium to tell stories." (Pausch et al. 1996)
Pausch et al. wrote a summary paper about what they learned creating this endeavor (tested on 45,000 guests at EPCOT). Notably:
"Our underlying premise is that VR is a new medium, as film, radio, and television once were. As motion pictures matured, directors and audiences developed a lexicon including close ups, cross cuts, flash backs, etc. Over time a common language, or lexicon, will evolve for VR; this project is our first step towards that goal."
One of the interesting challenges that the imagineering team reflected on was how guests wouldn't know to turn their heads, letting characters walk right out of the frame.
Of course, the creators of Aladdin Adventure viewed cutting between scenes as an impossibility:
"Because we let the guest control the viewpoint we must build characters and scenes that look good from all vantage points. By establishing entrances to scenes we control the initial view of each scene, a technique used in well-designed theme parks. The inability to cut from scene to scene or view to view is very frustrating for content authors. We have experimented with having characters that are attached to the guest’s head, and appear to be hanging off the front of the HMD. This allows us to interject a brief “scene” including that character."
They relied on storytelling techniques achieved from theme-park rides, notably an excellent background in manufacturing suspension of disbelief.
Modern 360 Filmmakers
Recently, as VR-capable devices reach a consumer market, and 360 videos are finding mass appeal (with support and attention from platforms such as Facebook and YouTube), there has been a new surge in storytelling development with 360 video, one largely inspired not only by video games or interactive efforts, but also by journalists and independent creators.
Creators that are doing things right.
There is a lot of work I admire and hope to not only learn from, but expand on top of.
Within the recent surge of interest in VR, 360 video, and distributed immersive entertainment, journalists have expressed a lot of interest in the medium. Journalists have really seen the potential of many elements of 360 video, and have capitalized on it.
Some of the most impressive journalistic work has been produced by WITHIN (Formerly VRSE), Walking New York and Clouds over Sidra are great examples of 360 filmmaking. WITHIN uses cross fades or dips to black to cut shots together, and a consistent audio narration to provide context for each scene. They cut between voice over and in-person narration. The in-person scenes capitalize on the user's ability to look around the environment the narrator speaks from, learning about the narrator and who they are from their context; their 'home'. Sidra narrating from her bare-walled room is much more powerful than if she had been in a Cyclorama or studio. In these films, there are no direct cuts or quick takes, but the audio narration moves the story along quickly and concisely.
WITHIN, with their documentaries, has scenes with members looking at the camera. Their curiosity for the strange hardware is translated as curiosity about the viewer, giving a sense of role and presence without turning the 'camera' into a voiceless character. In almost every shot (the exception was a sunset), a person is the point of interest. It's people - their movements, eyelines, actions, faces, and voices - that WITHIN uses to capture and control the viewer attention, although they also frequently forfeit control - overwhelming the viewer with fascinating scenes full of detail and movement - or surrounding the camera with people (say, running in a shrinking ring towards the device). These scenes usually end on a dip to black followed by a return to the narrator.
WITHIN has proven itself as one of the predominant, revolutionary, and capable names in 360 and Virtual Reality filmmaking, and their work is likely to have a huge influence on the narrative language of 360 films.
There are countless independent filmmakers rigging together GoPro cameras and publishing videos onto YouTube. Few seem to be thinking heavily on how narratives may be constructed. One notable and influential filmmaker is the prolific Casey Neistat.
Casey Neistat often films holding a 360 camera out in front of him, and talking while facing the camera. He uses tripods and such, but is also almost always in his scenes. This gives an anchor point for him to edit together scenes with. By more-or-less consistently putting himself in the same location (user head position), the user has a visual connection between multiple cuts. This cohesion between cuts is otherwise done by keeping the camera immobile, and lining up environments or environmental features - techniques Jessica Brillhart speaks about. It is interesting to see that more than the environmental or architectural cues utilized to provide a visual connection between shots.
Lastly, Casey Neistat understands the importance of where the camera is placed, and how to show something that is always visually and/or emotionally stimulating to some degree.
Jessica Brillhart works at Google as a resident "principal filmmaker". Google is developing a hardware and software platform for 360 video creation called JUMP, and Jessica has been tasked not only for creating films with Google's camera system, but has taken up writing and reflecting on her experiences. Google is trying, through Jessica Brillhart's efforts, to nurture the field of 360 filmmaking. It's reading her writing where I first thought "Okay, this person really gets it". She knows the importance of editing and storytelling, and won't resign herself just to long takes or slow fades. Go Habs Go! is a solid example of the work she is doing. Like with WITHIN's work, a voice-over narrator provides context for scenes and transitions.
Jessica is the only 360 filmmaker out there intensely exploring cutting between scenes, from the perspective of an editor. Following users eyelines, points-of-interest, and exploring when and how to give the user total control - and how to make them think they have more control than they really do. Jessica's work is probably the most influential to me and other 360 creators right now, both in content and in form - she is clearly speaking directly to other filmmakers about the pure craft of 360 video.
Michael Naimark describes himself as an 'VR OG type', and stresses the importance of studies: "I’m a big fan of studies (think Muybridge) and frankly am bewildered how little the VR community has understood the value and leverage of them." (Naimark, 2016).
Michael is thinking about the construction of VR from a craftsman perspective, and has a world of experience operating with these cameras and systems. His work so far contains an excellent breakdown of various methods and possibilities that the platform has to offer.
EleVR is a research team exploring a wide variety of new media works, largely focused on VR and 360 video. They are exploring Cinematography and Editing in 360 video, and putting together experiments to explore various possibilities - such as some very interesting experiments on Multi Camera spherical video.
EleVR keeps their experiments ("pieces") focused and small. They have not explored filming full narrative works as much as created a huge number of very small scenes that explore very particular aspects.
Creators that are not quite there.
Rooster Teeth makes sketch comedy and other videos online, and Samsung sponsored a 360 video that they made, called "Kidnapped". This video operates on many of the 'first assumptions' that have emerged on 360 video:
- Don't cut without a conceit.
- Don't cut if you don't absolutely have to.
- The camera is a character in the scene.
None of these assumptions are all that wrong. Some excellent videos are being made with these premises - but cuts and the lack of a vocal first-person positioned character (that the viewer embodies) are all being explained away with ever-absurd narrative conceits, and there is little room for narrative exploration.
Seeker Daily has been putting together reports in 360 video, such as "Inside The Arctic Doomsday Seed Vault For The First Time". Their video is professionally filmed and edited, but feels off-place. It's essentially a radio story - all narration, with little to look at. Aside from the beautiful landscape shots we got before 'arriving' in the seed bank, anything interesting in the 360 videos were random and inconsistent (and factually useless) graphic overlays, or normal video inserted into the scene, made to look like a projector was putting it on the wall (bad texturing and fake old-school projector sound and all). There also was no movement - they could have been 360 photospheres, for all I knew. This piece had no inherent reason to be in 360, and the producers clearly struggled with how to fill it with visual content.
This is a radio piece full of B-roll, and without any motivation to be in 360 or for the user to discover anything when looking around, it falls flat. This video is a good example of many 360 videos of it's kind - creators struggling to figure out not only how to edit 360 video, but just what to film with it.
WETA Digital is a VFX company, and they recently made "Elliot's Flyover". This film is marketed as an "experience" - our first tip-off that there is little content or meaning to be found outside of the sheer sensation of getting to look around in VR. It's a flyover of some rather interesting mountains in New Zealand, with the interesting part (below you) hidden by a fairly well composited dragon, which you are riding. A tie-in to an up-coming high budget Disney movie, support from Nokia (and their new camera), and in partnership with Air New Zealand produced a phenomenally uninteresting, un-cut 36 seconds of aerial footage. It's unmoving (despite the score), and certainly not unique in it's format: drone/helicopter footage, no cuts, dramatic music, no directed engagement.
There are many more "poor" videos out there, and the three I picked above were practically chosen at random, exemplifying many of the common 360 video tropes I have seen recently, which I hope to see less and less of as this media develops. Youtube's 360 videos channel, full of popular, trending, and 'successful' videos, has the following categories: Trending, Immersive with Spacial Audio, VR Videos (stereoscopic), A Musical Adventure (music videos and concerts), Total Horror (jump scares and flickering lights), Brave Souls (action/sports videos), In The Amusement Park (rides), Step Into The Games (video game footage rendered in 360). There is no genre of storytelling present.
I've read Walter Murch's essays on film editing (In The Blink Of An Eye) countless times over the years, and it's his thoughts that - at core - got me thinking the first thoughts that led to this thesis. Understanding how a cut works in the mind of a viewer will be critical to achieving similar effects in an interactive, immersive, and unfamiliar territory.
It was after re-reading his essay I first noticed some of the issues with 360 video, and how the cut's took place. None of them, it seemed, lined up with when I, a viewer, wished to look somewhere else. I could turn my head, but not move. 360 video - I thought - should be able to cut at that moment I turn my head, and allow me to get to my insert or scene change in a natural way that aligns with my minds transition between spaces (i.e.: in the blink of an eye). These were the first musings that brought to looking intensely at 360 video, and the first experiments I will embark on will be utilizing his theories.
Works In Interactive Storytelling
360 videos are not only immersive, but they are interactive - the viewer has a level of agency, choosing where to look. Consciously or subconsciously, the viewer has input (turning head) that is responded to by the media system (video pans/tilts).
This presents a whole new set of challenges and works to consider. Any creator who has to consider not only how their presentations are being received, but how the audience may respond and change to them - or travel through their stories in multiple cohesive ways - is a work to be considered. Notably, video games have been faced with this challenge forever. Narrative-driven video games such as The Walkind Dead games by Telltale or Life Is Strange by Dontnod are working to deliver a story that works not only as it adjusts to big character decisions, but small ones - which order the viewer may encounter NPC dialogs, what sort of content have they missed or might not understand, and so on.
Mike Cavazza's work in passive interactive fiction are key examples of the type of thinking - to add an per formative element of media systems, let them be capable of 'reading' the audience and adapting, like human performers do. Cavazza's work on PINTER, (Passive INTERaction) is not necessarily successful, but it's the sort of thinking I am getting behind in my research. Interaction that the viewer is aware of leads to many issues and traps that can be avoided if the viewer's task is understanding and enjoyment, and not being tested, judged, or asked arbitrary plot questions of. It is disappointing to see this area between passive and interactive entertainment as un-researched as it is.
The Man With A Movie Camera
This classic film is a staple of film history. The film has had variety of scores composed for it's silent film stock over the years. I recommend the version recently scored by the Electric Orchestra. The Man With A Movie Camera's director, Dziga Vertov, issued a statement about the film:
This new experimentation work by Kino-Eye is directed towards the creation of an authentically international absolute language of cinema – ABSOLUTE KINOGRAPHY – on the basis of its complete separation from the language of theatre and literature.
Emphasis mine. Dziga Vertov, 1929.
Vertov explored many different editing and montage techniques, and his film was as much creative exploration as it was a statement on the direction films and cinematic editing should head.
Manual For Urban Projection
Ali Momeni showed me his guide for urban projection. This - a guide for artists, detailing not only the niche technical methods and tools, but artistic and conceptual motivations relating to urban intervention. I find this work influential not in it's content but in it's message. In an interview Momeni described the work as "easily photocopiable". It was written for a specific audience of creators, and that directed message I find the most fascinating. My goals are not to speak to a general public or the world of film theorists and film writers, but to speak to creators and artists trying to work in the medium of 360 video.