Thanks to the rapid advancement of technology, life seems to be changing every day. The first of its kind is something we have been saying a lot when it comes to Mixed Reality. The next first of its kind application and use case with Mixed Reality is happening in the space of theater and performing arts.

From October 3rd (Monday) to 25th (Tuesday) 2022, the first use of Mixed Reality in theater and performing arts took place. In Japan, at the Osaka Shochikuza Theatre, a Kabuki (song and dance) performance of a “Japanese ghost story (J horror) Kabuki” seemed to be a normal show for most. However, select participants in the back seats behind the audience experienced the show wearing Microsoft’s Mixed Reality headset, the HoloLens. They were immersed in “a new form of theatergoing”, where subtitles and video production were superimposed on the main stage.

According to Mr. Tsuyoshi Tomita of the production company Shochiku Co., Ltd., the idea came to fruition as they have been “working on a next-generation entertainment creation business that fuses the real and the virtual.” They explored the potential use of AR (Augmented Reality) but did not like the results. When they discovered Mixed Reality, they quickly realized MR was the answer. It allowed them to enhance the viewing experience without taking away from the normal show.

The selection of Mixed Reality was the easy part. The challenge for developers was figuring out how to use MR to add new virtual elements. The show producers hired a couple of companies to help with the MR design and CG production, but they struggled with the first attempt to fuse MR and reality. They tried many different combinations, such as subtitles, videos, and the combination of both subtitles and videos, to improve upon the live performance. Finally, they developed two major systems. One is designed to display visual effects and subtitles, and the other is a simultaneous display system. The reason behind the second application is that each audience member has a different vantage point based on where they sit. To overcome this and create a consistent experience for each user, a custom display had to be designed for each audience member.

During the show, the selected participants started off watching the show without the headset. The reason for this was to allow participants to experience what the show is normally like. After the first act, participants were asked to wear the headset and view the show in Mixed Reality. The results were the best outcome they could have hoped for, as the application and headset did not interfere with the show; instead, they enhanced the viewing experience.