Using mixed reality headsets, the user becomes immersed in a fiercely athletic and romantic dance that permeates an extravagant movie palace where silent films were once shown. Melissa Painter and her key collaborators have crafted a multidimensional experience wherein the user can chase dancers around a pool and on stage, scale through a labyrinth of stunning architecture, and even reach out and hold the dancers in the palm of his or her hand. After the swirl of experience, the user will find that he or she has accomplished a dance of their own.


How did you come up with the idea for the project?

I had been working in emerging media for the previous four years. Throughout that time, I had been witness to capabilities that intrigued me: photogrammetry, highly accurate motion capture, the high-resolution capability of the Nokia Ozo camera, and the ability for a user to move 1:1 through a mixed reality world. I also have both danced, and worked in dance intermittently, my whole life. I knew this was a chance to bring two loves together: emergent technologies and dance. We decided to make this project, and to do so using three forms of media, the moment we first saw Laura Gorenstein Miller’s duet set to “Heroes.”  “Heroes” was the catalyst for the core story elements: the shifts in scale that happen as the dancers go from human to giant, as well as a deep sense of bravery and strength.

Describe the production process.

Heroes was made in partnership between MAP Design Lab, MPC VR, and Helios Dance Theater. We asked ourselves a lot of questions as we started the project: How can emergent storytelling technologies help people see the patterns in the abstraction of movement? How can they help people move their own bodies in relation to the story? How can we best explore the relationship between dancers and space, and between dancers and the audience?

Separately, developing for the HoloLens was new to all of us, and provided some challenges. That said, we were all very enthusiastic about bringing the skills we did have into this new space. We decided to embrace this discovery and invention process that mixed reality demanded.

We had some fundamental design principles we didn’t waver from: the experience should provoke the user to move, allow the user to contribute to the dance, and give the user vantage points on the choreography he or she would not normally have. Key collaborators were Tim Dillon and Jason Schugardt from MPC VR, Laura Gorenstein Miller of Helios Dance Theater, Thomas Wester and Eric Marshall. Joey Verbeke did all of the sound design.

From a production standpoint, the first thing we did was capture the architecture and the dancers in every way that we could: motion capture, rapid avatar scans, photogrammetry, 360 film shoot, stills, traditional film, and sound foley recordings. We pulled all of these points of data into the HoloLens, and started to see what felt right as a user, and what was tantalizing to look at. We wanted to explore as many novel forms of interaction as mixed reality would allow. From this experimentation, we decided early on to rest the UI on the instinctive movement of the user and his or her vocal commands.

What was your biggest hurdle in creating the project and how did you overcome it?

We were spread out over many geographic locations and didn’t have enough HoloLens headsets to go around!   Seriously – in making for an evolving technology, I find that imagination, trust in the team, staying awake to happy accidents and a good sense of humor go a very long way.

What advice do you have for someone who wants to start building for mixed reality?

The best way to learn is by making. The joy of making for HoloLens is that it truly allows you to see into the future – it is liberated from so many of the constraints that come with other headsets. My other piece of advice would be to user test early and user test often. The more iteration, the better!

What excites you most about the future of mixed reality?

I believe that mixed reality is our next computing platform, and that inviting the digital off of the screens and into our lives will lead to a more humanist relationship with technology. I am fascinated by this moment when the tools we hold in our hands are making it easier and easier to capture the real world and place it into the virtual. Motion capture, photogrammetry, 3D scanning… all of these technologies have advanced to the point where we can sketch and prototype with real world elements. As we move into this new medium, I am most compelled by how we protect the imagination (versus overwhelm it), as well as how we remember that interaction, engagement, memory, and welcoming multiple points of view are the keys to learning and being human.

Microsoft provided mentorship and guidance to each creator as part of this initiative.