The past five years have seen groundbreaking production of theatre and opera that have integrated augmented reality (AR) and virtual reality (VR) – or, as the industry term refers to all types of immersive technology, XR – to develop new kinds of live and virtual experiences. The Welsh National Opera, in 2017, combined ‘The Magic Flute’ and ‘Madam Butterfly’ for ‘Magic Butterfly,’ an experience that used Google Daydream VR to demystify opera for audiences, in a new venue. Commonwealth Shakespeare Company produced ‘Hamlet 360: Thy Father’s Spirit’ in the past year, working with Google’s AR/VR Lens team to create five seven-minute acts in VR.

Yet, some of the boldest work in XR for theatre is not surprisingly being produced in New York. An event produced by Chris Pfaff, and sponsored by the VR AR Association New York Chapter and Verizon’s 5G Lab, on Wednesday, November 6, 2019 featured some of the leading practitioners of XR in the theatre world who discussed their work and and their platform and technology choices, as well as how to shape narratives for interactive audiences in real-time experiences. Demonstrations of their work, as well as a guest appearance from award-winning XR production firm Felix & Paul Studios, were held prior to and after the main event, which was held in Verizon’s 5G Lab at the Alley powered by Verizon.

The entire event can be viewed at: https://www.youtube.com/watch?v=wQQhDrp0vtQ&t=31s

Brandon Powers, producer/choreographer of ‘Frankenstein AI,’ a piece that premiered at the Sundance Film Festival in 2018, and producer/choregrapher of ‘Queerskins,’ discussed how he worked with the Intel Studios volumetric studio to use his choreographic and immersive theater expertise to move ‘visitors’ through the virtual space. ‘Frankenstein AI,’ which expressed our AI inhabiting a human body to communicate with the audience, and ‘Queerskins’ demonstrate ways for theatre producers to create new work with volumetric capture. In ‘Frankenstein AI,’ the human dancer performed choreography, which was created using a new choreographic notation that Brandon invented. The choreography was manipulated by an AI in real-time based on how the audience responded to the AI’s prompted questions, prompting the dancer to alter her performance for a unique live experience. Brandon is also developing a new episode in the VR experience ‘Queerskins,’ which involves an interactive choreographed duet utilizing volumetric capture which will give a unique experience for each user.

Kiira Benzing, from Double Eye Studios, the renowned producer of ‘Loveseat,’ a VR theatre piece that was featured at the 2019 Venice Biennale, in which two lonely, ordinary people are drawn into a reality show competition to win the love of a Perfect Partner (who looks an awful lot like an empty chair). Part-story, part-interaction between virtual and real worlds; the performers intersect realities in this comedy. Performed simultaneously to virtual and real world audiences. Kiira also discussed her work on ‘Loveseat’ and her more recent work on ‘Runnin,’ which was shot at Intel Studios in Los Angeles. ‘Runnin’ is currently available through the Samsung Gear app through Oculus Quest. Her work in designing virtual worlds has led her to ask ‘what is a stage?’ and ‘what is an audience?’ when audience members are represented by virtual beings and avatars.

Maciej Wisniewski, 99 Center Opera founder, and producer and composer of ‘Edelweiss VR, Act 1,’ or EVR1, the first original VR opera experience, discussed and demonstrated his work in creating a music-driven narrative that enables viewers to unlock hidden aspects of the story. Maciej’s work includes blockchain for encrypted messages that will be retrieved by the players of the EVR1 game, to unlock content. The story of EVR1 takes viewers through an interactive VR story in five acts, based on a true World War II experience of his great-aunt escaping a prison in Poland and becoming a Resistance fighter. Maciej’s work involves breathing light and texture into scenes based on music.

David Rodriguez, founder of Zanni, the first Audience Experience Design Studio, demonstrated the firm’s XR tools, Ovees™ and Zanni™ AXD, for both immersive audiences and theatre producers. David’s work with the Metropolitan Opera led him to consider how opera, and theatre, could evolve through VR and AR tools for both audience participation and narrative extensions, as well as how to enable immersive workflows for theatre/opera producers. Ovees is an XR take on stereoscopic opera glasses – and Zanni AXD – a mixed reality design tool that creates space to view stage production elements, reducing design process time and enhancing stage experience planning for design teams. Zanni is a new company, and the event marked the public debut of Zanni.

The panel discussion delved into how traditional theatre producers, directors, lighting designers, choreographers, and actors themselves are moving to expand the medium through immersive technologies. The group discussion yielded the overarching understanding of theatre being gamified and moved into a new genre altogether. The panelists discussed how technologies are rapidly evolving to meet demands for creative decisions. Kiira Benzing discussed how difficult it was to mount ‘Love Seat’ in Venice, with a massive point cloud that could, at any time, crash. The shared experience of theatre, in a live setting, pushes the boundaries of technical production for real-time communal energy.

Chris Pfaff welcomes the audience at ‘XR in Theatre: Immersing New Audiences’ at the Verizon 5G Lab, 11.06.19

Chris Pfaff, Mikael Chagnon, from Felix & Paul Studios, and Brandon Powers

Maciej Wisniewski (foreground) and Kris Ramanathan (rear) showcase their EVR1 musical interactive VR piece

Brandon Powers discusses his immersive choreographic work on ‘Queerskins’ and ‘Frankenstein AI’

Kiira Benzing discusses ‘Runnin’ and ‘Love Seat,’ her interactive VR theatre pieces, and her work with Intel Studios

Maciej Wisniewski, from 99 Cent Opera, discusses his use of blockchain to unlock content in the interactive VR musical narrative

David Rodriguez, from Zanni, discusses his use of immersive tools for theatre and opera

The Q&A session for ‘XR in Theatre: Immersing New Audiences’ 

The 100+ audience at ‘XR in Theatre: Immersing Audiences,’ at the Verizon 5G Lab in New York, 11.06.19

David Rodriguez; Brandon Powers; Chris Pfaff; Kiira Benzing, and Maciej Wisniewski, after the ‘XR in Theatre’ panel

VR/AR Association New York chapter members Chris Pfaff, Cindy Mallory, Michael Owen, and chapter president Gordon Meyer

Maciej Wisniewski, Linda Aro, and Kris Ramanathan before the ‘XR in Theatre: Immersing New Audiences’ event

Kiira Benzing, Chris Pfaff, David Rodriguez, and Brandon Powers before the ‘XR in Theatre: Immersing New Audiences’ event

VR/AR Association New York chapter member Chris Pfaff, chapter president Gordon Meyer, and member Gordon Yee before the event

The third anniversary of the VR/AR Association (VRARA) New York chapter was a positive snapshot on the growth of the immersive/XR industry in the greater New York area in the past few years. The event was hosted on Tuesday evening, May 21^st, at RLAB (https://www.rlab.nyc), the massive space that NYU and its academic and City of New York partners opened in late-November, 2018. RLAB hosts several companies, including members of its XR Beta program, and is being built out as a larger facility for immersive innovation, including volumetric capture space.

Dex Yee (left), from VRARA, watches as Jason T. Jaslow signs in the VRARA NY 3rd anniversary

Chris Pfaff, VRARA NY Chaper advisor, welcomes the crowd at RLAB

As VRARA has grown globally, so too has its New York chapter, and this mirrors the broader focus on the XR industry in New York, which embraces enterprise and consumer firms alike.

Banu Ozden in discussion with Janice Brown, manager, education and outreach at RLAB

Robin White Owen and Michael Owen, the first XR couple of Brooklyn, at Rlab

Unseen Media demos its soon-to-be-released narrative AR game

Gur Arie Bittan, from Mantis Vision, demos for Banu Ozden and Jeffrey Ginsberg

More than 50 guests attended the mixer event, which featured demos from XR Beta companies, as well as Mantis Vision (http://mantis-vision.com), and remarks by RLAB’s Alexis Seeley and VRARA New York chapter advisor Chris Pfaff. And, of course, numerous demos of mobile AR experiences were shown by VRARA members as well.

Robin White Owen and Michael Owen listen to Tim Meyer, from IBM’s IoT group

Alexis Seeley, director of education and opportunity programs at Rlab, welcomes the crowd

Mantis Vision’s mo-cap installation, and some of its forthcoming collaboration tools, were a major hit at the event. Unseen Media ((https://www.unseenmedia.io), a narrative AR game developer, demonstrated its soon-to-be-released game, while echoAR (https://www.echoar.xyz), and AR-focused CMS and CDN provider, and SIY (Speak it Yourself – https://www.siyvr.com), a VR-based language instruction firm, demonstrated their solutions as well.

Chris Pfaff and Gordon Meyer, one of the Top 5 AR influencers in the industry

Chris Pfaff and Kate Specter, from Toonpack, at Rlab

Alexis Seeley and Janice Brown, with Rlab, prior to the VRARA event

Some of New York’s leading AR technology developers attracted a crowd at The Alley, Verizon’s 5G collaborative lab and work space in Chelsea last Tuesday, May 8^th, at an event sponsored by the VR/AR Association’s New York chapter. ‘AR for Producers: How to Engage Audiences on Multiple Platforms’ was well-titled, as most of the audience was comprised of new media and television producers who are looking at AR for a wide range of production elements, including fan engagement, 2^nd-screen extensions, and location-based entertainment.

JR Dawkins welcomes the crowd to The Alley     Michael and Serge Doudy provide an overview of the VR/AR Association

Eric Schwertzel, head of business development for eyecandylab’s US presence, demonstrated the AugmenTV concept, in which TV content triggers experiences on mobile using an AR app. This is a long sought-after play, and should have traction in sports as well as episodic TV.

Eric Schwertzel discusses AugmentTV

Gordon Meyer, head of marketing for Lampix, demonstrated the company’s projector-based AR solution, which is ideal for retail and public venue experiences.

Gordon Meyer presents the Lampix innovation

JR Dawkins, head of business development for Envrmnt by Verizon, showcased the Envrmnt AR Designer tool, the industry’s first true drag-and-drop AR tool. The simplicity of the tool enables anyone – with no coding experience – to create AR triggers on physical objects, and more.

JR Dawkins presents AR Designer      Ryan Hilla assists with AR Designer demo

Far beyond what previous generations of AR solutions have provided, the presenters discussed the value that 5G will bring to AR, enabling seamless, low-latency experiences with greater connectivity levels.

JR Dawkins, Gordon Meyer, and Eric Schwertzel discuss the AR industry

Chris Pfaff, JR Dawkins, Gordon Meyer, and Eric Schwertzel during the panel discussion at ‘AR for Producers’

For many in the audience, it was their first time in The Alley, which opened last spring, and has gained a following among the technorati of the city.

Lampix demo at ‘AR for Producers’

Chris Pfaff and Ally Perez, event coordinator for The Alley, powered by Verizon