The past five years have seen groundbreaking production of theatre and opera that have integrated augmented reality (AR) and virtual reality (VR) – or, as the industry term refers to all types of immersive technology, XR – to develop new kinds of live and virtual experiences. The Welsh National Opera, in 2017, combined ‘The Magic Flute’ and ‘Madam Butterfly’ for ‘Magic Butterfly,’ an experience that used Google Daydream VR to demystify opera for audiences, in a new venue. Commonwealth Shakespeare Company produced ‘Hamlet 360: Thy Father’s Spirit’ in the past year, working with Google’s AR/VR Lens team to create five seven-minute acts in VR.

Yet, some of the boldest work in XR for theatre is not surprisingly being produced in New York. An event produced by Chris Pfaff, and sponsored by the VR AR Association New York Chapter and Verizon’s 5G Lab, on Wednesday, November 6, 2019 featured some of the leading practitioners of XR in the theatre world who discussed their work and and their platform and technology choices, as well as how to shape narratives for interactive audiences in real-time experiences. Demonstrations of their work, as well as a guest appearance from award-winning XR production firm Felix & Paul Studios, were held prior to and after the main event, which was held in Verizon’s 5G Lab at the Alley powered by Verizon.

The entire event can be viewed at: https://www.youtube.com/watch?v=wQQhDrp0vtQ&t=31s

Brandon Powers, producer/choreographer of ‘Frankenstein AI,’ a piece that premiered at the Sundance Film Festival in 2018, and producer/choregrapher of ‘Queerskins,’ discussed how he worked with the Intel Studios volumetric studio to use his choreographic and immersive theater expertise to move ‘visitors’ through the virtual space. ‘Frankenstein AI,’ which expressed our AI inhabiting a human body to communicate with the audience, and ‘Queerskins’ demonstrate ways for theatre producers to create new work with volumetric capture. In ‘Frankenstein AI,’ the human dancer performed choreography, which was created using a new choreographic notation that Brandon invented. The choreography was manipulated by an AI in real-time based on how the audience responded to the AI’s prompted questions, prompting the dancer to alter her performance for a unique live experience. Brandon is also developing a new episode in the VR experience ‘Queerskins,’ which involves an interactive choreographed duet utilizing volumetric capture which will give a unique experience for each user.

Kiira Benzing, from Double Eye Studios, the renowned producer of ‘Loveseat,’ a VR theatre piece that was featured at the 2019 Venice Biennale, in which two lonely, ordinary people are drawn into a reality show competition to win the love of a Perfect Partner (who looks an awful lot like an empty chair). Part-story, part-interaction between virtual and real worlds; the performers intersect realities in this comedy. Performed simultaneously to virtual and real world audiences. Kiira also discussed her work on ‘Loveseat’ and her more recent work on ‘Runnin,’ which was shot at Intel Studios in Los Angeles. ‘Runnin’ is currently available through the Samsung Gear app through Oculus Quest. Her work in designing virtual worlds has led her to ask ‘what is a stage?’ and ‘what is an audience?’ when audience members are represented by virtual beings and avatars.

Maciej Wisniewski, 99 Center Opera founder, and producer and composer of ‘Edelweiss VR, Act 1,’ or EVR1, the first original VR opera experience, discussed and demonstrated his work in creating a music-driven narrative that enables viewers to unlock hidden aspects of the story. Maciej’s work includes blockchain for encrypted messages that will be retrieved by the players of the EVR1 game, to unlock content. The story of EVR1 takes viewers through an interactive VR story in five acts, based on a true World War II experience of his great-aunt escaping a prison in Poland and becoming a Resistance fighter. Maciej’s work involves breathing light and texture into scenes based on music.

David Rodriguez, founder of Zanni, the first Audience Experience Design Studio, demonstrated the firm’s XR tools, Ovees™ and Zanni™ AXD, for both immersive audiences and theatre producers. David’s work with the Metropolitan Opera led him to consider how opera, and theatre, could evolve through VR and AR tools for both audience participation and narrative extensions, as well as how to enable immersive workflows for theatre/opera producers. Ovees is an XR take on stereoscopic opera glasses – and Zanni AXD – a mixed reality design tool that creates space to view stage production elements, reducing design process time and enhancing stage experience planning for design teams. Zanni is a new company, and the event marked the public debut of Zanni.

The panel discussion delved into how traditional theatre producers, directors, lighting designers, choreographers, and actors themselves are moving to expand the medium through immersive technologies. The group discussion yielded the overarching understanding of theatre being gamified and moved into a new genre altogether. The panelists discussed how technologies are rapidly evolving to meet demands for creative decisions. Kiira Benzing discussed how difficult it was to mount ‘Love Seat’ in Venice, with a massive point cloud that could, at any time, crash. The shared experience of theatre, in a live setting, pushes the boundaries of technical production for real-time communal energy.

Chris Pfaff welcomes the audience at ‘XR in Theatre: Immersing New Audiences’ at the Verizon 5G Lab, 11.06.19

Chris Pfaff, Mikael Chagnon, from Felix & Paul Studios, and Brandon Powers

Maciej Wisniewski (foreground) and Kris Ramanathan (rear) showcase their EVR1 musical interactive VR piece

Brandon Powers discusses his immersive choreographic work on ‘Queerskins’ and ‘Frankenstein AI’

Kiira Benzing discusses ‘Runnin’ and ‘Love Seat,’ her interactive VR theatre pieces, and her work with Intel Studios

Maciej Wisniewski, from 99 Cent Opera, discusses his use of blockchain to unlock content in the interactive VR musical narrative

David Rodriguez, from Zanni, discusses his use of immersive tools for theatre and opera

The Q&A session for ‘XR in Theatre: Immersing New Audiences’ 

The 100+ audience at ‘XR in Theatre: Immersing Audiences,’ at the Verizon 5G Lab in New York, 11.06.19

David Rodriguez; Brandon Powers; Chris Pfaff; Kiira Benzing, and Maciej Wisniewski, after the ‘XR in Theatre’ panel

VR/AR Association New York chapter members Chris Pfaff, Cindy Mallory, Michael Owen, and chapter president Gordon Meyer

Maciej Wisniewski, Linda Aro, and Kris Ramanathan before the ‘XR in Theatre: Immersing New Audiences’ event

Kiira Benzing, Chris Pfaff, David Rodriguez, and Brandon Powers before the ‘XR in Theatre: Immersing New Audiences’ event

VR/AR Association New York chapter member Chris Pfaff, chapter president Gordon Meyer, and member Gordon Yee before the event