The Core Challenge
The fundamental problem of 360 video editing is that we do not know where the audience is looking. Thus, the framing, the direction of action, the visual weight of the frame, the eye-trace, and more are unknown to the editor. There is never complete certainty.
There are strategies for dealing with this - many of them well explained by the work of Jessica Brillhart before me. Also, see the section on points of interest. Another, the most common, approach is to let the audience be the editor. Shoot with minimal cuts and frame/block the scenes in such a way that the audience is not confused, they can look around to follow the action; as opposed to cuts.
This section is not about strategies for pulling off an edit in spite of the medium. This section is about how to approach editing in such a way that we work with the medium of spherical video. In other words, this is how to think about editing 360 video.
How To Think About Editing 360 Video
Why do filmmakers edit? There are a lot of reasons. A theatrical production exists in fairly continuous stream. Scenes are long and transitions are scarce. 'Edits' happen thanks to curtains, lighting, or stagecraft. These edits did one of two main things: Jump ahead in the timeline of the world/story or jump across to different characters or locations in the timeline. Without getting into esoteric or experimental works,1 they were jumps to an "elsewhere".
There is nothing about 360 video that gets rid of our ability to do this type of editing! Dip to black, crossfade, or - yes - just cut to a new scene. Estimate the viewers gaze and align architecture/vanishing points, characters, points of interest, or whatever. Add a few extra moments of lead-in time after the cut. Cut before the character's entrance in order to - like theater - give a moment for the audience to get their bearings.2 Edit like a stage curtain. See the section on editing primitives for specific strategies for this edit.
Yet, we aren't limited to these tools.
Film copied these tools for much of the early history of film, cutting infrequently, and with clear and obvious plot-motivated purpose. 360 video should at least start here. Slowly editing began to fall into it's own craft, utilizing the camera to more creative ends - things like close-ups, match-on-actions, and more required editing within a scene. Not to mention montage theorists. How this creative editing works is where 360 will diverge from film.
But spherical video editing doesn't need to erase film history and start over. The effort of understanding what this will look like with spherical video is closer to a translation. Some tools we may have to forgo - film staple: the shot-reverse-shot; for example. Others we may have to distort, and some will barely be affected at all.
I think translation is a poor word choice, as 360 video does not need to exist in conversation with film, but can exist as it's own unique medium, with unique considerations that conceivably could be derived from scratch. I believe this approach is epistemically possible. Such an effort would be nonsensical but I must conceptually stand the medium apart for consideration outside of film as a medium. They share many medium characteristics - video - and thus share many tools.
A good working definition of what it means for a medium to exist "as a medium" is that the expectations that an audience have with it are not, in the audience's mind, in conversation with other mediums. People don't pick up poetry books and expect novels, and we would consider these to be separate mediums. If an audience can pick up a 360 video and expect, anticipate, a story being told through the tools of 360 video; then 360 video is its own medium. This work must assume that this has already happened in order to conceive and illustrate what the tools the audience may expect are. In other words, in order to competently write on these tools, we must assert the rest of the aforementioned 'is a medium' definition is true, so that the writing can allow for this possibility to be coherent. We have to shape this puzzle piece such that it will fit the puzzle we want to fit, and this puzzle is where 360 video is a "medium of its own".
The goal is not to figure out how to make films with 360 cameras. The goal is how to tell stories with them.
This digression is important because it frames the following discussion of editing, and I must be clear that 360 video editing not only may not and in all likelihood will not look like film editing, but need not either. Not having the ability to do a shot-reverse-shot, for example, is a signifier of a limitation only if one is trying to make regular films [in 360]. The lens about which one views spherical video must shift away from looking through film (or video games, or VR, or whathaveyou) in order for this examination to be successful.
Let's talk about the fundamental tool that is the cut. Switching from one video stream to another in an instant or near-instant. Why cut?
- Jump forward in time
- Change locations
- Switch concurrent linear stories, action lines, or character focus
- Show a space without panning/zooming/moving camera. Potentially accomplish "impossible" blocking.
- Cut to what a character is reacting to or looking at
- Relieve tension
- Set the pacing of the story
- Clarify or emphasize a point/cue
- Create meaning through intercutting multiple [brief] shots
- Establish the environment/go into the established environment
This is not an exhaustive list. Nor is it terribly taxonomically coherent, as 'relieve tension' is either a catch-all or an ontological parent, for example, but stay with me - my point is elsewhere.
Motivate the Cut
Ensuring the viewer understands the reason to cut is far more important in 360 video than in film for two major reasons: viewers are unfamiliar with 360 video and the more immersive an experience is, the more jarring a cut can be.
There have not been enough videos using similar methods seen by a wide audience, to establish any firm consensus on the "language of 360 video editing".
*"And the question arose immediately as to what might, indeed, be done with this new panoramic system in the way of developing a dramatic story on the screen. All of the scenes were in long shots, with little or no use at all of "montage"—the cinematic means of expression and emphasis through cutting and juxtaposition of shots."*
Since the days of Cinerama, this trouble of translating has been the question for innovative takes on film as a medium. I have, during conversations at the Weird Reality VR Conference, heard it mentioned as the "big problem" of 360 video. Yet, I disagree. First, I believe it is a misnomer to say this language does not exist. The medium is here, and audiences are experiencing them. Audiences are reacting and behaving in 360 video (as opposed to vomiting and ripping off their HMD), so there is some kind of "language".
Further, The "lack" of a language doesn't stop us - it frees creator's up. A medium's language is the sum of the expectations the audience has that allows the medium to take certain liberties in its execution. We can set these expectations ourselves, not merely work with the set of those that the audience walks in with.
A single film can establish it's own language in the course of the film. For example, cutting in similar ways throughout a piece can teach the audience what to expect and how to behave.
A cut in 360 video can be as jarring as a sudden teleportation for the viewer. The goal of filmmakers must be to take advantage of the immersiveness of the medium in ways that help the experience and the story. At the same time, the filmmakers must back away from immersiveness that hurts the story. I'll discuss how to achieve this apparent paradox later, it comes down to firmly and consistently establishing the role of the audience.
Establishing The Language
The task falls on the creators to establish the language of 360 storytelling. Not for the sake of the advancement of this medium as a whole, but for the sake of your story. Consider your film it's own genre, just for a moment. What are the rules? What conventions does this storytelling follow? How can we demonstrate (not tell) these rules to the audience?
Our goal is for, during the cut, the audience not to be taken "out of the story". In other words, we want to edit "invisibly". In more grounded terms, we want the audience's thoughts, and the questions they ask themselves, to be about the story, not about the storytelling. The audience shouldn't stop and think "where am I?", it would be better to have them ask "Why am I here?" or "Why is the character here?". Again, these questions are not necessarily active questions floating in the forethought of the audience's mind. They are the small curiosities and tensions that allow storytelling to happen, and we must - as storytellers - be aware of the machinations at hand.
We must take responsibility of editing a film in such a way as to teach the audience - during the course of our film - how to watch the film. This means consistency, patterns, and clear codified signals. We can do whatever we want, as creators, but 360 video is not presently at a point where one can simply tell you what to do and how to edit.
In fact, you have my permission to ignore all of the tools and tips that I give in the "tools" section of this work. That section is my effort to groundwork for creators, to create a sensible language that allows for effective storytelling. I do this through learning from what is out there currently, and experimenting myself.
Yet, you still have my permission to ignore all of it, if it means you tell your story better. While I believe it is unlikely for that section to end up being dead wrong, it is very likely that you may be producing a 360 video for a known audience, an audience that you can make assumptions about, and this will change how you approach storytelling, and how you approach editing.
There is always some counter-example, but we stay focused in order not to get tied down with exceptions or complicated cases. We have a clear initial goal: We need to be able to tell stories. We need the fundamental tools that let stories in this medium happen. Our goal could be continuity and "readability" of a story. Our secondary or follow-up goal is to let the editing impart meaning of its own on the story experience. ↩
Why else would curtains open and then characters walk on from off-stage? To establish an exit point and greater world, sure, but I would argue that this is at least in part done to give the audience a chance to take in the scenery and setting, before changing focus to the characters. ↩