There’s always been an evolution in the art of filmmaking. But, from small content creators to large film studios, the advancements in virtual production have enabled everyone to create immersive and high-quality films in a variety of ways. How did this happen? Who is affected by it? Even if its worth it, how much would it cost? Let’s talk about some of these aspects when it comes to virtual production in filmmaking.
The Rise of Virtual Production
One would think that the recent pandemic caused a rise in virtual productions but, really, this technology has been available in one form or another. COVID wasn’t really the impetus for studios, but rather an advancement in technologies. Although, COVID certainly played a part in other areas.
James Cameron’s use of advancing technology started with “Avatar” in 2009. Since then, it’s grown tenfold and the world will soon see what’s currently capable when the sequel is released later this year. The first film was created using blue screens and actors in motion capture suits. Production then keyed in the actors, live, to fill the virtual world. This enabled Cameron to look at his monitor and see everything in real time.
Since then, there’s been a big push in accessibility and faster turnaround times. Virtual artists can build a world in Unreal Engine beforehand and then manipulate it while on set. This enables the artists to manipulate them any way they want, keeping the same tone and style. This can save a ton of time and money because things are more controllable.
What Can Virtual Production Consist Of?
Assume we’re talking about a real virtual set — whether that’s a small independent production, art exhibit, or major studio film — there are usually three things needed.
- Virtual and VFX Artists – They help to create and alter the world and ultimately save that world for future scenes, movies, or edits. Most of the artists I know all work in Unreal Engine, which is open source.
- A Green Screen or LED wall – Some smaller productions may even use just one or two 4k, full array LED TV’s for something as simple as, let’s say, a driving scene where the camera is tight on an actor.
- A powerful computer (or multiple computers) – These should be capable of running a tracking system that feeds the images of the world to the LED wall and read the camera information inside the computer, such as what lens is being used, focus depth, etc. The tracking information sets all of that and places the camera inside the world as far as lens height, focus, depth, and more. This way, the world is able to read how it is perceived in that particular setup.
Actors, instead of reacting to a tape mark placed on a blue or green screen, can actually react to a world they can physically see. Even with the Star Wars spinoff “The Mandalorian” they reused sets plenty of times with minor alterations and projected them onto the LED walls. The filmmaker can see it live while filming, along with the actors, helping both of them with imagination and a sense of realism that green screens just can’t provide.
What Parts of the Production Life Cycle Can Be Made Virtual?
Over the last few years — whether because of the pandemic or advancements in technology — there’s been a few areas of the production pipeline that have gone virtual in some way.
- Auditions have all but gone virtual at this point. The days of going into a casting director’s office at a specified time are almost non-existent. Of course, this is a bit market dependent. Even for myself, it’s easily been four years since I’ve seen the inside of one. Everything is now taped at home on my phone or with my DSLR and sent in to casting directors.
- Callbacks for auditions may still be done in person. Although, a majority of them are over Zoom or Eco Cast, live in front of the casting director and executives.
- PreVis has certainly seen its share with the rise of apps such as PreVis Pro (for storyboarding and shot listing) and Cinetracer (which allows cinematographers and directors to create scenes and lighting in Unreal Engine.)
- Cloud Collaboration has seen a massive increase over the past year or two from the likes of Frame.io and Davinci Resolve. Department heads and collaborators can be almost anywhere in the world, live editing, making sure they’re getting the coverage on set.
- Voice-over and ADR can be done from home now. Technology has become so cheap and attainable that most artists have their own booths (or closets) set up at home and can record live with an engineer over Source Connect. Whenever I’ve had to go into an ADR session, I’ll be in a booth in New York with post-production producers in Los Angeles on a video feed, hearing and seeing me in real time, giving me notes on the spot.
That’s only a few to get started. As COVID restrictions started to lift, people were anxious to get back to work but wanted to be safe. Many sets still limit the number of people allowed on set, but spend extra time in pre-production. When it comes time to film, there is usually only the utmost important crew on set with an agency representative or producer(s) on Zoom with a direct link to the camera feed. They could give notes in real time without having to physically be there.
The pandemic made people start to think more creatively on how to get things done, while staying away from each other. Sometimes cinematographers even work virtually via Zoom.
How Have These Parts Been Made Virtual?
Most of the things mentioned above — and ones not — have been made more possible and popular simply because of the advancement in technology. Pretty much everyone has access to a phone, tablet, laptop and an internet connection. As it becomes even more accessible and cheaper, it will more than likely replace many other areas of production.
For virtual productions that use green and blue screens, LED walls will almost certainly replace them in the future. The technology has already gotten cheaper over the past few years. This will enable filmmakers to do more in smaller studios as well.
Opinions from Filmmaking Professionals
I recently spoke to two industry professional about the emerging virtual production sector and how it is currently being used. They’re from different parts of the industry but use the same technology.
I sat down with Dominick Sivilli, a New York-based cinematographer who recently started his own small virtual production company.
“The accessibility to be collaborative from anywhere is hands down the biggest feat.”
“The biggest thing I’ve seen is the idea that clients and directors can virtually Zoom in easily and affordably,” Sivilli said. “When production picked back up after the pandemic, we would spend twice as long in pre-production, resulting in fewer people on a more efficient set.
Sivilli described how producers and other teams would tie directly into the monitors over Zoom and see what the on-set crew was seeing. Although, he wasn’t fully convinced that this workflow makes the job easier.
“It also created an issue where you’ll have clients on-set, plus Zoom, making things a bit chaotic,” he said.
For color corrections sessions, it’s a different story. Sivilli can zoom into a session in another country and still be able to get a pretty accurate rendition.
“The accessibility to be collaborative from anywhere is hands down the biggest feat,” he said.
While I was only able to get her for a few brief moments over email, Michelle Penland Dodson, Chief Creative Officer and one of the founding members of Choreografx, was able to give me some insight into her thoughts and work. She mainly works on film production, music videos, and 3D projection for art exhibits and concerts.
She mentioned that the biggest pro of virtual production is being able to see everything in-camera in real-time without having to imagine it for post-production. Also, background elements can be quickly adjusted, such as adding or removing lights, or even hiding objects.
“Being able to make visual choices on the fly with the director, DP, and gaffer gives a huge amount of creative freedom,” Dodson said via email.
Benefits of Virtual Productions
Of course, during the pandemic, there were benefits to making things more virtual. It enabled us to get back to work in a safe environment, saving some costs, and limiting team size to only mandatory positions. Although, costs were added back in with testing and post-production editing. But while many of those restrictions have been lifted or lessened recently, there are other benefits as well.
The biggest is probably the ability for these technologies to enable filmmakers to boost their imagination. Unreal Engine (currently 5) is an open source program that’s incredibly user-friendly. This means that people can download it for free, get in right away and start building. With this program, production teams can collaborate and add things to the virtual world.
You can also use an LED wall as a backdrop outside a window of your scene, such as a skyline. It’s ultimately the same thing as a painted backdrop but with much more realism.
Cons of Virtual Productions
With the good comes the bad — at least from some people’s perspectives. For right now, things are still fairly new and expensive. Not a lot of people have experience or access to these types of programs except top-tier studios or companies with heavy financial backing. There’s also a huge learning curve to some of these things, especially LED or XR walls. Countless hours of testing and practice must be done in a space of suitable size before you ever press record.
Think about virtual productions as if you were going to shoot on a green screen. If you’re not able to shoot it in a reasonable way, then consider just shooting on-location. VR walls offer some restrictions, such as one-shots. You can only look in certain directions as most walls don’t cover a 360°. If your camera looks up too far, you’re going to see the sound stage whereas, on location, you wouldn’t.
The believability is another factor. When using green screens or VR walls, some sets — like an apartment — can look hokey unless you mix in some practical elements in the foreground that help add a sense of realism. An alien world you created, however, would be different because there is no real comparison.
As far as collaboration access, if it’s easier for more people to join a session online, it can cause a bit of chaos and differences in direction on set.
Final Thoughts
How do we proceed from here? One has to wonder — or at least I do — if these new technologies will hinder creativity and human interaction. The fun of making movies lies in the process.
While I do think it’s an inevitable part of the future, I don’t think the virtual aspects of production are going to totally replace on-location filmmaking. But, as the technology becomes cheaper and more accessible, it has already started to replace many aspects of the production pipeline. How far will it go? Only time and technology will tell.
