Virtual reality software accurately alters film scenes

Virtual reality software developed at the School of Computer Science has helped Hollywood automate post-production alterations in real-world film scenes. Warner Bros, 20th Century Fox and Paramount Pictures have all used the software in box office hits such as Watchmen and Battle Los Angeles, which grossed over $385 million combined.

Virtual reality banner image

The technique, known as 'camera tracking', is central to almost all modern film production. It allows directors to merge real life footage with computer-generated effects, creating scenes that have been altered in realistic and accurate ways.

The first software to combine automatic tracking with interactive construction, yielding accurate structured geometric models was developed by researchers at the School of Computer Science.

The software package, called ICARUS, first caught the eye of the research community in 2002 when we released a free, binary-only version of the software for non-commercial use, which accrued 10,000 downloads in its first three months.

The Pixel Farm

In 2003, The Pixel Farm purchased a licence for ICARUS covering its exclusive use in the film and video post-production industries.

Dr Simon Gibson, one of the original designers of ICARUS from the School of Computer Science, became Research Director at The Pixel Farm. His former colleague, Jon Cook, joined him at the same time as a software engineer. Both researchers still work for the company.

The Pixel Farm rebranded and further developed ICARUS as PFTrack, and has since become a leading global provider of camera tracking software within a highly competitive industry. It employs 10 staff and has a turnover of more than £1 million annually.

PFTrack

PFTrack enjoyed early use in the production of the award-winning Cold Mountain (2003) and the Marvel film, Fantastic Four (2005). It has since been used for many high grossing films including: Watchmen, the Harry Potter series, Battle Los Angeles, The Boat that Rocked and The Hunt for Gollum.

Several TV series and adverts have used PFTrack, including: Smallville, Doctor Who, Grandpa in my Pocket, The Wrong Door, and a TV commercial for Kia Motors.

Use of PFTrack is taught in the Digital Effects MSc course at the UK's National Film and Television School.

The major contributions of the research are:

Turnover infographic

Turnover

The annual turnover of The Pixel Farm is more than £1 million.
Downloads infographic

Downloads

10,000 free copies of the software were downloaded in the first three months.

Background

Simon Gibson profile image

Dr S Gibson

Researchers at the School of Computer Science developed ICARUS based on their research and algorithms to build geometric models of real-world scenes directly from digital images.

The University of Oxford developed much of the underlying theory and also one of the first automatic tracking algorithms. At the same time, the University of Leuven developed its own tracking and reconstruction system. But both of these prior systems generated polygon meshes. A structured geometric model, means geometry similar to the output from a CAD modeller that can then be used by other software. Other systems tend to generate unstructured 'polygon soup'.

The University of California at Berkeley developed the first system for structured models, but this relied heavily on user interaction for camera calibration and therefore lacked accuracy. The ICARUS tracker and reconstruction methods, which were new, represented a significant improvement over these other systems, and its adoption enabled The PixelFarm to become a market leader in this field.

The new approach used a hierarchical, two-pass tracking algorithm for camera calibration.

The algorithm integrates three innovations:

  • Estimates of inter-frame motion to guide the feature tracker and reduce the incorrect tracking of features.
  • Projective reconstruction using sub-sequences of video data and hierarchical algorithms to merge sub-sequences into a single, reliable reconstruction.
  • Use of random sampling algorithms for 'self-calibration' to give accurate geometric projections.
▲ Up to the top