Film & Circus performance
Volumetric capture for creating virtual performances through collaboration between NICA and FGA students
This project will pilot the use of cutting edge Volumetric Capture technology at Swinburne’s new Volumetric Capture Studio in Prahran. We will create a prototype virtual circus performance by creating 3-dimensional spatial recordings of circus performances by National Institute of Circus Arts (NICA) students in collaboration with students from Film, Games and Animation (FGA).
These recordings can be distributed as Virtual Reality (VR) or Augmented Reality (AR) artifacts extending the audience for the student circus artists and digital content developers.
Resources
Two resources have been created as part of the wider project.
1. Utilising Adobe Mixamo to Rig 3D Characters – This project helps students to learn a 3D character rigging pipeline to generate skills applicable to a range of augmented and virtual reality platforms. Students will extend their knowledge in processing 3D characters using this project Instruction resource. It introduces character rigging in the context of character generation for AR and VR to assist in supporting advanced content creation learning outcomes. This resource could be used successfully in extended reality content development, game development and three-dimensional filmmaking. The project instructions are available to Swinburne staff via Commons.
2. Colour-correcting and removing spill from volumetric video using Adobe After Effects – This project asks students to process a volumetric video to understand this aspect of the post-processing pipeline. Students use Adobe After Effects to colour-correct and remove spill from volumetric video captured In a green background environment. The project instructions are available to Swinburne staff via Commons and also available on the Adobe Education Exchange.
Problem
In the newly created volumetric capture unit (FTV30024 Visual Effects Production), students learn the pipeline for creating volumetric video assets, full solid 3D performers suitable for a range of mediums such as Augmented Reality (AR) and Virtual Reality (VR). The students learn to capture short performances by an actor, usually one of the student team. As an alternative, we were interested to test whether professional performances by circus artists from Swinburne’s National Institute of Circus Arts (NICA) would be good subject matter for the volumetric captures. This would entail some collaboration between organisations, similar to what would happen in industry. This would also give the circus artists access to three-dimensional recordings of their performances that could be used for learning, marketing and social sharing.
We weren’t able to change the curriculum of two units to fully accommodate a collaboration by multiple student cohorts. Instead, we piloted the use of circus performance as learning subject matter for the volumetric capture students and also developed student capacity to create AR and VR apps which would be ideal avenues to incorporate the volumetric capture assets. The AR / VR capabilities were explored in FTV30006 Production for Digital Platforms.
These steps in utilising circus performance as subject matter, developing student’s volumetric capture skills and developing student’s AR and VR application creation skills are the main pieces of the puzzle to enable future collaboration in a full pipeline for the creation of AR and VR applications that highlight Circus performance skills using volumetric capture.
Project Overview
This project piloted the use of the cutting edge Volumetric Capture technology at Swinburne’s new Volumetric Capture Studio in Prahran for capturing circus performance to be used in the learning of the volumetric capture pipeline. A balancing act was chosen as it fit the confines of the volumetric capture space (approximately 3 metre diameter capture volume) and didn’t require any props or external apparatus. This performance was captured at the beginning of semester in order for the recordings to be available for the students to practice post-processing of the takes during semester.
This circus performance became the central content for students learning the volumetric capture pipeline. It gave the FTV30024 students a well-defined performance where the costume, speed and positioning of the performance were known to work well. Students worked through the post production pipeline using this recording prior to embarking on their own recordings.
Students in FTV30006 were learning the intricacies of creating AR and VR applications suitable for mobile device distribution. The pipeline was investigated to see whether volumetric captured assets could be incorporated into the pipeline. This was determined to be the case paving the way for future incorporation of volumetric assets into the AR and VR application students build.
Being able to see the associations between skills learned in different units and how they can be combined to create more powerful learning outcomes was one goal of this project. Even though the pipelines in the different units are different, this project shows how the learnings are complimentary and can be used to create more ambitious projects in the future student careers.
Emergent Outcomes (outcomes of the project)
This project piloted the workflow and pipeline necessary for creating high-quality volumetric recordings of circus art performances suited to digital distribution for VR, AR and web-based delivery. The use of volumetric captures has many potential outcomes:
- Creates a great opportunity for increased audience access to appreciate the high-level skills of the circus arts students
- Documents the learnings of NICA and FGA students in professional-level collaboration
- Creates a training resource for sharing and feedback
- Identifies key learnings required for potential integration into curriculum
The challenges for digital screen studies students were many in coming to terms with spatial 3-dimensional film-making, which will be of great benefit to the students as content generation moves beyond traditional 2-dimensional screens and production pipelines.
One of the first cohorts of students undertaking FTV30024 – Visual Effects Production was introduced to the tools and processes required to undertake a full volumetric capture pipeline. Performers and circus skill performances had to be matched to the physical dimensions of the volumetric capture studio which has a limited capture volume. Part of the pilot focused on best practice for a workable pipeline students can successfully engage with. We needed to see the ramifications of costumes, apparatus, speed and other aspects of the performance. Prime candidates for circus skills suited to volumetric capture are the range of balancing acts mastered by students.
In FTV30024 Visual Effects Production, students created a prototype virtual circus performance by post-processing 3-dimensional spatial recordings of circus performances by a National Institute of Circus Arts (NICA) circus performer. The volumetric assets were recorded as videos, as creating AR and VR applications from the assets is beyond the scope of the single unit. By concurrently having students learn AR and VR app development, the skills base is there for students to combine these skillsets to create fully capable AR and VR experiences using volumetric capture assets. This could for example be part of a major capstone project where the range of skills can be combined and applied.
In FTV30006 Production for Digital Platforms, students developed skills in creating AR and VR applications. While incorporating volumetric capture performances into the AR and VR applications was currently beyond the scope of this unit, it points the way for future integration.
Digital Literacy outcomes (student/staff engagement with Digital Literacies pillars)
This project had major Technological and Critical Literacy outcomes. Students were tasked with learning a state-of-the-art volumetric capture pipeline and producing a digital artifact suitable for Extended Reality (XR) applications such as Virtual Reality (VR) and Augmented Reality (AR).
Students undertaking FTV30024 Visual Effects Production in the Bachelor of Screen Production utilised recordings of a circus performer from Swinburne’s National Institute of Circus Arts (NICA) in order to develop their skills in volumetric capture post-production.
One component of the volumetric capture post-production process was green screen removal, colour correction and green spill correction, in order to achieve the best possible footage for further processing the volumetric videos. Adobe After Effects and in particular its Keylight plugin was used for advanced colour correction and spill suppression. Students in FTV30024 were able to use these advanced tools to achieve professional results.
In the unit FTV30006, the skills required for creating AR and VR applications were introduced. These skills are potentially closely aligned with the production of volumetric assets and something we will develop in the future. In this unit, students used a more common method for embedding characters, using 3D avatars and animating these avatars with recorded motion capture data. One key aspect of this pipeline was rigging their created 3D characters so that the motion capture can animate the skeleton of the characters. Students used Adobe Mixamo (www.mixamo.com) to easily auto-rig their 3D characters ready for animating in their AR or VR projects.
Key Learnings (personal shifts in teaching, approaches etc)
There were a lot of moving parts in trying to design collaboration across schools (NICA and FGA), across campuses (Hawthorn and two Prahran campuses, EMD Studio and NICA) and very different cohorts of students. Approaching the different aspects as asynchronous tasks across units proved the best way to link the different possibilities.
The circus performance captured volumetrically was a great subject for learning the volumetric capture pipeline. Most student groups had well defined scenarios for their volumetric capture projects so mandating they work with circus performance wasn’t appropriate for this cohort. However, in the future it would be a strong option as the circus performances proved ideal subjects for the medium.
The combination of elements was a good example of what a more ambitious pipeline would look like and something we can provide to students before undertaking capstone projects should they want to create a work I the AR or VR medium.
Full team collaboration between NICA and FGA cohorts would require a lot more resources and planning. This breaking down of tasks into smaller components across units was more manageable and still pointed the way to asynchronously developing full scale applications.
Impact
Giving students access to a high quality volumetric capture performance early in the semester enabled the students to surmount the steep learning curve of the volumetric capture pipeline more quickly that in previous cohorts. The nature of the circus performance meant there were few recording artifacts and post processing was relatively easy. This is reflected in the generally high technical quality of the student’s final works.
This project has also provided the basis for a larger grant application to look at volumetric capture as a learning tool for circus students and tutors.