# Virtual cinematography

> Mediated Wiki article. Canonical URL: https://mediated.wiki/source/Virtual_cinematography
> Markdown URL: https://mediated.wiki/source/Virtual_cinematography.md
> Source: https://en.wikipedia.org/wiki/Virtual_cinematography
> Source revision: 1330880993
> License: Creative Commons Attribution-ShareAlike 4.0 International (https://creativecommons.org/licenses/by-sa/4.0/)

**Virtual cinematography** is the set of [cinematographic](/source/Cinematography) techniques performed in a [computer graphics](/source/Computer_graphics) environment. It includes a wide variety of subjects like [photographing](/source/Photograph) real objects, often with [stereo](/source/Stereo_camera) or [multi-camera setup](/source/Multi-camera_setup), for the purpose of recreating them as three-dimensional objects and [algorithms](/source/Algorithm) for the automated creation of real and [simulated](/source/Computer_simulation) [camera angles](/source/Camera_angle). Virtual cinematography can be used to shoot scenes from otherwise impossible camera angles, create the photography of animated films, and manipulate the appearance of computer-generated effects.

## History

### Early stages

An early example of a film integrating a virtual environment is the 1998 film, [*What Dreams May Come*](/source/What_Dreams_May_Come_(film)), starring [Robin Williams](/source/Robin_Williams). The film's visual effects team utilized actual building blueprints to create scale wireframe models, which were then used to generate the virtual world.[1] The film went on to garner numerous nominations and awards including the [Academy Award for Best Visual Effects](/source/Academy_Award_for_Best_Visual_Effects) and the [Art Directors Guild Award](/source/ADG_Excellence_in_Production_Design_Award) for Excellence in Production Design.[2] The term "virtual cinematography" emerged in 1999 when special effects artist [John Gaeta](/source/John_Gaeta) and his team wanted to name the new cinematic technologies they had created.[3]

### Modern virtual cinematography

[The Matrix trilogy](/source/The_Matrix_(franchise)) (*[The Matrix](/source/The_Matrix)*, *[The Matrix Reloaded](/source/The_Matrix_Reloaded)*, and *[The Matrix Revolutions](/source/The_Matrix_Revolutions)*) used early Virtual Cinematography techniques to develop virtual "filming" of realistic computer-generated imagery. The result of John Gaeta and his crew at ESC Entertainment's work was the creation of photo-realistic CGI versions of the performers, sets, and actions. Their work was based on [Paul Debevec](/source/Paul_Debevec) et al.'s findings on the acquisition and subsequent simulation of the reflectance field over the human face acquired using the simplest of light stages in 2000.[4] Famous scenes that would have been impossible or exceedingly time-consuming to produce within the context of traditional cinematography include the burly brawl in *The Matrix Reloaded* (2003) where [Neo](/source/Neo_(The_Matrix)) fights up-to-100 [Agent Smiths](/source/Agent_Smith) and the beginning of the final showdown in *The Matrix Revolutions* (2003), where Agent Smith's cheekbone gets punched in by Neo[5] leaving the digital look-alike unharmed.

For *The Matrix* trilogy, the filmmakers relied heavily on virtual cinematography to attract audiences. [Bill Pope](/source/Bill_Pope), the [Director of Photography](/source/Cinematographer), used this tool in a much more subtle manner. Nonetheless, these scenes still managed to reach a high level of realism and made it difficult for the audience to notice that they were actually watching a shot created entirely by visual effects artists using 3-D computer graphics tools.[6]

In *[Spider-Man 2](/source/Spider-Man_2)* (2004), the filmmakers manipulated the cameras to make the audience feel as if they were swinging together with [Spider-Man](/source/Spider-Man) through New York City. Using [motion-capture](/source/Motion-capture) camera radar, the cameraman moves simultaneously with the displayed animation.[7] This makes the audience experience Spider-Man's perspective and heightens the sense of reality. In *[Avengers: Infinity War](/source/Avengers:_Infinity_War)* (2018), the Titan sequence scenes were created using virtual cinematography. To make the scene more realistic, the producers decided to shoot the entire scene again with a different camera so that it would travel according to the movement of the Titan.[8] The filmmakers produced what is known as a synthetic lens flare, making the flare very akin to the originally produced footage. When the classic animated film [*The* *Lion King*](/source/The_Lion_King_(2019_film)) was remade in 2019, the producers used virtual cinematography to make a realistic animation. In the final battle scene between [Scar](/source/Scar_(The_Lion_King)) and [Simba](/source/Simba), the cameraman again moves the camera according to the movements of the characters.[9] The goal of this technology is to immerse the audience in the scene further.

## Methods

### Virtual cinematography in post-production

In [post-production](/source/Post-production), advanced technologies are used to modify, re-direct, and enhance scenes captured on set. [Stereo](/source/Stereo_camera) or [multi-camera setups](/source/Multiple-camera_setup) photograph real objects in such a way that they can be recreated as 3-D objects and algorithms. [Motion capture](/source/Motion_capture) equipment such as tracking dots and [helmet cameras](/source/Helmet_camera) can be used on set to facilitate the retroactive data collection in post-production.[10]

[Machine vision](/source/Machine_vision) technology called [photogrammetry](/source/Photogrammetry) uses [3-D scanners](/source/3D_scanning) to capture 3-D geometry. For example, the Arius 3D scanner used for the Matrix sequels was able to acquire details like fine wrinkles and skin pores as small as 100 μm.[4]

Filmmakers have also experimented with [multi-camera](/source/Multiple-camera_setup) rigs to capture motion data without any on-set motion capture equipment. For example, a [markerless motion-capture](/source/Motion_capture#Markerless) and multi-camera setup photogrammetric capture technique called [optical flow](/source/Optical_flow) was used to make digital look-alikes for the Matrix movies.[4]

More recently, [Martin Scorsese](/source/Martin_Scorsese)'s crime film *[The Irishman](/source/The_Irishman)* utilized an entirely new [facial capture](/source/Facial_motion_capture) system developed by [Industrial Light & Magic](/source/Industrial_Light_%26_Magic) (ILM) that used a special rig consisting of two [digital cameras](/source/Digital_camera) positioned on both sides of the main camera to capture motion data in real time with the main performances. In post-production, this data was used to digitally render computer-generated versions of the actors.[11][12]

[Virtual camera](/source/Virtual_camera_system) rigs give cinematographers the ability to manipulate a virtual camera within a 3-D world and photograph the [computer-generated](/source/Computer-generated_imagery) 3-D models. Once the virtual content has been assembled into a scene within a [3-D engine](/source/Game_engine), the images can be creatively written, relighted, and re-photographed from other angles as if the action was happening for the first time. The virtual "filming" of this realistic CGI also allows for physically impossible camera movements such as the [bullet-time](/source/Bullet_time) scenes in *[The Matrix](/source/The_Matrix)*.[4]

Virtual cinematography can also be used to build complete [virtual worlds](/source/Virtual_world) from scratch. More advanced [motion controllers](/source/Motion_controller) and tablet interfaces have made such visualization techniques possible within the budget constraints of smaller film productions.[13]

### On-set effects

The widespread adoption of visual effects spawned a desire to produce these effects directly on-set in ways that did not detract from the actors' performances.[14] Effects artists began to implement virtual cinematographic techniques on-set, making computer-generated elements of a given shot visible to the actors and cinematographers responsible for capturing it.[13]

Techniques such as [real-time rendering](/source/Real-time_computer_graphics), which allow an effect to be created before a scene is filmed rather than inserting it digitally afterward, utilize previously unrelated technologies, including video game engines, projectors, and advanced cameras to fuse conventional cinematography with its virtual counterpart.[15][16][17]

The first real-time motion picture effect was developed by [Industrial Light & Magic](/source/Industrial_Light_%26_Magic) (ILM) in collaboration with [Epic Games](/source/Epic_Games), utilizing the [Unreal Engine](/source/Unreal_Engine) to display the classic *[Star Wars](/source/Star_Wars)* "light-speed" effect for the 2018 film *[Solo: A Star Wars Story](/source/Solo:_A_Star_Wars_Story)*.[15][18] The technology used for the film, dubbed "Stagecraft" by its creators, was subsequently used by ILM for various Star Wars projects as well as its parent company [Disney](/source/The_Walt_Disney_Company)'s 2019 photorealistic animated [remake of *The Lion King*](/source/The_Lion_King_(2019_film)).[19][20]

Rather than scanning and representing an existing image with virtual cinematographic techniques, real-time effects require minimal extra work in post-production. Shots including on-set virtual cinematography do not require any of the advanced post-production methods; the effects can be achieved using traditional CGI animation.

## Software

- [Autodesk Maya](/source/Autodesk_Maya) is a 3-D computer graphics software that runs on [Windows](/source/Windows), [OS X](/source/OS_X) and [Linux](/source/Linux).
- [Autodesk 3ds Max](/source/Autodesk_3ds_Max) is a professional 3-D computer graphics program for making 3-D animations, models, games and images for Windows only.
- [Blender (software)](/source/Blender_(software)) is a [free](/source/Copyleft) and [open-source](/source/Open-source_license) [3-D computer graphics software](/source/3-D_computer_graphics_software) product used for creating animated films, visual effects, art, 3-D printed models, interactive 3-D applications and video games, intended for [DIY](/source/DIY) virtual cinematographers.
- Pointstream Software by Arius3D is a professional dense motion-capture and [optical flow](/source/Optical_flow) system using a [pixel](/source/Pixel) and its [movement](/source/Euclidean_vector) as the unit of tracking usually over a [multi-camera setup](/source/Multi-camera_setup).

## See also

- [Entertainment technology](/source/Entertainment_technology)
- [Extended reality](/source/Extended_reality)
- [History of computer animation](/source/History_of_computer_animation)
- [On-set virtual production](/source/On-set_virtual_production)
- [Timeline of computer animation in film and television](/source/Timeline_of_computer_animation_in_film_and_television)
- [Timeline of CGI in movies](/source/Timeline_of_CGI_in_movies)
- [Uncanny valley](/source/Uncanny_valley)
- [Virtual actor](/source/Virtual_actor)
- [Virtual camera system](/source/Virtual_camera_system)

## References

1. Silberman, Steve (2003-05-01). ["MATRIX2"](https://www.wired.com/2003/05/matrix2/). *Wired*. [ISSN 1059-1028](https://www.worldcat.org/issn/1059-1028). Retrieved 2020-04-02.

1. ["What Dreams May Come"](http://www.imdb.com/title/tt0120889/awards), IMDb, retrieved 2020-04-02

1. Feeny, Catherine (Mar 9, 2004). ["'The Matrix' Revealed: An Interview with John Gaeta"](http://www.uemedia.net/CPC/vfxpro/article_7062.shtml). *VFXPro*. Creative Planet Communities, United Entertainment Media. [Archived](https://web.archive.org/web/20040318114836/http://www.uemedia.net/CPC/vfxpro/article_7062.shtml) 2004-03-18 at the Wayback Machine. Retrieved 2020-04-02.

1. Paul Debevec; Tim Hawkins; Chris Tchou; Haarm-Pieter Diuker; Westley Sarokin; Mark Sagar (2000). "Acquiring the reflectance field of a human face". *Proceedings of the 27th annual conference on Computer graphics and interactive techniques - SIGGRAPH '00*. pp. 145–156. [doi:10.1145/344779.344855](https://doi.org/10.1145/344779.344855). ISBN 1-58113-208-5. [S2CID 2860203](https://api.semanticscholar.org/CorpusID:2860203)

1. George Borshukov, Presented at Imagina'04. ["Making of *The Superpunch*"](https://docs.wixstatic.com/ugd/02edaa_83a116ead7f84ff2aeb8c012f703cd67.pdf) (PDF).

1. ["David Fincher – Invisible Details"](https://www.youtube.com/watch?v=QChWIFi8fOY), May 26, 2017, [archived](https://ghostarchive.org/varchive/youtube/20211221/QChWIFi8fOY) 2021-12-21 at the Wayback Machine, retrieved 2020-04-02

1. ["The Amazing Spider-Man 2 – Virtual Cinematography"](https://www.youtube.com/watch?v=YILJNoxKmVM), April 23, 2015, [archived](https://ghostarchive.org/varchive/youtube/20211221/YILJNoxKmVM) 2021-12-21 at the Wayback Machine, retrieved 2020-04-02

1. ["Avengers: Infinity War VFX | Breakdown – Cinematography | Weta Digital"](https://www.youtube.com/watch?v=YviKYEqy6JI), March 28, 2019, retrieved 2020-04-02

1. ["*The Lion King* – virtual cinematography and VFX"](https://www.youtube.com/watch?v=iw96ZN6e9vw), February 2020, [archived](https://ghostarchive.org/varchive/youtube/20211221/iw96ZN6e9vw) 2021-12-21 at the Wayback Machine, retrieved 2020-04-02

1. Breznican, Anthony (9 December 2019). ["The Irishman, Avengers: Endgame, and the De-aging Technology That Could Change Acting Forever"](https://www.vanityfair.com/hollywood/2019/12/the-de-aging-technology-that-could-change-acting-forever). *Vanity Fair*. Retrieved 2020-04-03.

1. ["Robert De Niro said no green screen. No face dots. How 'The Irishman's' de-aging changes Hollywood"](https://www.latimes.com/entertainment-arts/movies/story/2020-01-02/irishman-de-aging-of-hollywood-how-they-did-it). *Los Angeles Times*. 2020-01-02. Retrieved 2020-04-03.

1. Desowitz, Bill (2019-12-06). ["'The Irishman': How Industrial Light & Magic's Innovative De-Aging VFX Rescued Martin Scorsese's Mob Epic"](https://www.indiewire.com/2019/12/the-irishman-ilm-vfx-de-aging-1202194908/). *IndieWire*. Retrieved 2020-04-03.

1. ["Virtual Cinematography: Beyond Big Studio Production"](https://idea.library.drexel.edu/islandora/object/idea:4229). *idea.library.drexel.edu*. Retrieved 2020-04-02.

1. ["Sir Ian McKellen: Filming *The Hobbit* made me think I should quit acting"](https://www.radiotimes.com/news/2013-11-14/sir-ian-mckellen-filming-the-hobbit-made-me-think-i-should-quit-acting/). *[Radio Times](/source/Radio_Times)*. Retrieved 2020-04-02.

1. Roettgers, Janko (2019-05-15). ["How Video-Game Engines Help Create Visual Effects on Movie Sets in Real Time"](https://variety.com/2019/biz/features/video-game-engines-visual-effects-real-time-1203214992/). *Variety*. Retrieved 2020-04-02.

1. Leonard Barolli; Fatos Xhafa; Makoto Ikeda (eds.) (2016). *CISIS 2016 : 2016 10th International Conference on Complex, Intelligent, and Software Intensive Systems : proceedings : Fukuoka Institute of Technology (FIT), Fukuoka, Japan, 6–8 July 2016*. Los Alamitos, California: IEEE Computer Society. ISBN 9781509009879. [OCLC 972631841](https://www.worldcat.org/oclc/972631841)

1. Choi, Wanho; Lee, Taehyung; Kang, Wonchul (2019). ["Beyond the Screen"](http://dl.acm.org/citation.cfm?doid=3355088.3365140). *SIGGRAPH Asia 2019 Technical Briefs*. Brisbane, Queensland, Australia: ACM Press. pp. 65–66. [doi:10.1145/3355088.3365140](https://doi.org/10.1145/3355088.3365140). ISBN 978-1-4503-6945-9. [S2CID 184931978](https://api.semanticscholar.org/CorpusID:184931978)

1. Morin, David (February 14, 2019). ["Unreal Engine powers ILM's VR virtual production toolset on "Solo: A Star Wars Story""](https://www.unrealengine.com/en-US/spotlights/unreal-engine-powers-ilm-s-vr-virtual-production-toolset-on-solo-a-star-wars-story). *www.unrealengine.com*. Retrieved 2023-04-02.

1. ["How Lucasfilm's New "Stagecraft" Tech Brought 'The Mandalorian' to Life and May Change the Future of TV"](https://www.slashfilm.com/the-mandalorian-stagecraft/). */Film*. 2019-11-20. Retrieved 2020-04-02.

1. ["'The Lion King's' VR helped make a hit. It could also change movie making"](https://www.latimes.com/entertainment-arts/business/story/2019-07-26/disneys-lion-king-remake-is-a-hit-its-virtual-reality-technology-could-change-how-movies-are-made). *Los Angeles Times*. 2019-07-26. Retrieved 2020-04-02.

## Further reading

- Debevec, Paul (2006). ["Virtual Cinematography: Relighting through Computation"](http://www.debevec.org/Publications/DEBEVEC-IEEE-Computer-200608-high.pdf). *Computer*. **39** (8): 57–65. IEEE. [Bibcode:2006Compr..39h..57D](https://ui.adsabs.harvard.edu/abs/2006Compr..39h..57D). [doi:10.1109/MC.2006.285](https://doi.org/10.1109/MC.2006.285). [ISSN 0018-9162](https://www.worldcat.org/issn/0018-9162). [S2CID 11904037](https://api.semanticscholar.org/CorpusID:11904037). [Archived](https://web.archive.org/web/20070209202337/http://www.debevec.org/Publications/DEBEVEC-IEEE-Computer-200608-high.pdf) 2007-02-09 at the Wayback Machine. Retrieved 2006-11-06.
- Jhala, Arnav (2005). ["Introduction to Virtual Cinematography"](https://web.archive.org/web/20070102004752/http://www.cs.unc.edu/~lastra/comp870/intro_vc,%20Arnav%20Jhala.pdf). North Carolina State University. Archived from [the original](http://www.cs.unc.edu/~lastra/comp870/intro_vc,%20Arnav%20Jhala.pdf) on January 2, 2007. Retrieved 2006-11-06.
- Newman, Bruce (2003). ["*Matrix* sequel: Virtual Cinematography"](https://web.archive.org/web/20060427050830/http://www.temple.edu/ispr/examples/ex03_05_13b.html). *[San Jose Mercury News](/source/San_Jose_Mercury_News)*. Archived from [the original](http://www.temple.edu/ispr/examples/ex03_05_13b.html) on April 27, 2006. Retrieved 2006-11-07.

---
Adapted from the Wikipedia article [Virtual cinematography](https://en.wikipedia.org/wiki/Virtual_cinematography) by Wikipedia contributors ([contributor history](https://en.wikipedia.org/wiki/Virtual_cinematography?action=history)). Available under [Creative Commons Attribution-ShareAlike 4.0 International](https://creativecommons.org/licenses/by-sa/4.0/). Changes may have been made.