Though virtual reality has been an aspiration for technical professionals and artists alike for decades, it’s only in the last ten years or so that both AR (augmented reality) and VR (virtual reality) have gained significant traction among creators and consumers.
The medium (or group of interrelated mediums) is in fact still sorting out the associated terminology. A somewhat newer term, XR, accounts for any kind of extended reality experience, which can include both AR and VR, as well as other formats that may emerge as the technology that enables these experiences. Regardless of nomenclature, the demand for such experiences continues to evolve.
At the consumer level, the advanced Valve Index offers a high-end experience, while the Oculus Quest 2 from Facebook focuses on streaming content to a standalone headset and has proven to be especially capable when sideloading is used.
But perhaps more interesting than the hardware being used to share these experiences is how designers and artists are now challenging themselves to expand their understanding of what can be created in these spaces that exist beyond the real world, or in addition to it.
This brings us to Shelley Hu, a highly-acclaimed multimedia artist who has won multiple awards for her VR and AR work. With New Reality Co., she has created the VR installation piece Tree VR and the mobile AR experience Rainforest AR.
Her most significant awards thus far include a New York Emmy award and a Horizon Interactive Award for her data visualization for the multimedia piece Finding Sanctuary. Hu’s team also received an Epic Games MegaGrant (a reference to the original company name, Epic MegaGames) to work on Rainforest AR.
But what we discovered while talking with Hu is that she has many insights into the current landscape of XR media and high hopes for what the medium might become in the near future.
A trend toward affordability and accessibility
Public discussions of XR (those not limited to XR industry professionals) are bound to touch on the topic of accessibility and affordability, not only for consumers but for artists and designers hoping to work in this medium.
As for the availability of development tools that enable XR work, Hu notes that the (slowly) decreasing price of headsets and the emergence of advanced development tools are good news for the art form as a whole.
“XR creators in the past have been largely restricted by the price of the hardware and the steep learning curves of the software tools. However, we are currently witnessing increasingly affordable XR headsets, as well as the trend of more XR creator tools becoming open source.”
These open-source tools now include Tilt Brush, powerful software that allows for room-scale 3D painting. This is a major step forward in terms of making it much easier for those interested in XR creation to at least explore the possibilities without having to immediately invest in costly hardware and software.
A parallel can be found in music creation, where extremely accessible software programs designed for creating and mixing music (some of which can be used on a budget laptop or even a smartphone) spurred a new era of musical composition from amateurs and aspiring professionals, who were sometimes referred to as ‘bedroom producers.’
This shift toward greater accessibility to XR creation tools is relatively recent, and it may be years before the industry sees the full results. One thing we likely can expect is an influx of young designers and developers who are ready to bring fresh ideas to a medium that still hasn’t really established its own set of norms.
However, while XR backend tools are becoming more accessible rather quickly, XR accessibility on the consumer end is very much a mixed bag.
On the AR side, virtually all smartphones now have the ability to let users engage with AR content, albeit only through the small phone screen. AR games such a Pokemon Go! have been in the cultural consciousness for years, and new, bespoke AR apps are being released often.
As for VR, it’s still very much a high-end niche form of entertainment, as the most advanced headset systems cost roughly $1,000 US and are only occasionally in stock. These wired headsets also require mounted sensors and a PC capable of rendering high-resolution 360 graphics.
The aforementioned Oculus Quest 2 hopes to make VR an accessible household staple, but having only released in 2020, there’s no telling whether the system will catch on.
Now, getting to the core of the matter, we wanted to know why Hu and other leading multimedia creators like her have gravitated toward XR specifically.
At times, the medium can involve an intimidating learning curve, and artists need to spend quite a bit of time with the associated hardware, software, and applications before really getting an accurate sense of what the medium can accomplish.
In fact, Hu admits that, prior to her own experiences in XR, she had a bias against the medium, seeing it mostly as a novelty rather than as a substantial opportunity for artists and designers.
It was experimenting and getting hands-on experience with XR tools that changed her mind. Hu describes her turning point with XR:
“Instead of another tech fad or selling point, it was indeed an immersive blank canvas, a fresh ‘space’, for any kind of creative expression, particularly those considered impossible for this reality. The moment I put on a headset and dropped into the world I had just created, everything changed.”
Aside from Hu’s personal conversion with regards to XR, this brings up an interesting point when it comes to the adoption of XR in general.
As a quick comparison, consider the move from full HD resolution (1920×1080) to 4K resolution in TVs and monitors. Even without seeing a 4K display in-person, a user will probably still have a fairly good idea of whether they’d be interested in this technology. In the end, it’s a new way to experience pre-existing media formats.
But XR, by definition, has to be experienced firsthand. Describing it or even showing ripped footage doesn’t do a very good job of communicating what it’s actually like.
VR, in particular, is a completely different medium, only borrowing certain elements from other mediums.
In other words, Hu’s own experience of being immersed in her own VR space for the first time is probably a fairly accurate example of how many people would respond to this technology, and for artists like Hu, this novel experience also inspires an endless number of possibilities for creation.
Experiences that work well in VR
Let’s continue with that topic of possibilities. With a medium that’s still so new, how can we tell which ideas and which experiences will work well in VR?
Hu answered this question very succinctly:
“Any interaction that’s considered impossible for our reality would be very fun to create in VR because the technology offers the unique ability to manipulate time and space.”
This philosophy translates directly to Hu’s work on the Tree VR project. Again, describing any VR media will pale in comparison to actually experiencing it firsthand, but to sum up, Tree VR puts the user in the place of a tree in the rainforest, from a very young age. Hu elaborates:
“In just eight minutes, they experience the entire life cycle of the tree, from a seedling to the tallest tree in the rainforest. An experience like this is only possible, and is made particularly moving, through the creative use of immersive technology.”
Of course, a big part of this immersion comes from the fantastic visuals and allowing users to control branches of the tree with their arms, a feature that Hu explained was a big hit at more than 150 showings around the world at venues like the World Economic Forum in Davos, Switzerland and the United Nations General Assembly.
But another element that assists immersion is the stimulation of other senses, beyond just sight and hearing, which leads to the next topic.
Additional sensory elements
Prior to the 2000s, utilization of olfactory (scent-based) elements in entertainment was certainly rare, with one notable exception being filmmaker John Waters’s use of “Odorama” in the theatrical release of his 1981 film, Polyester, which was itself a nod to the theater gimmicks of the 1950s.
However, Hu notes that olfactory involvement has been on the rise in many different areas of entertainment in recent years, including at live concerts and in commercial applications.
Scent was very important in Tree VR, as Hu and the team saw it as adding significantly to the immersion of the piece. Stimulating more than two senses also puts the ‘multi’ in multimedia.
“For Tree VR, we worked with world-renowned perfumers at IFF and developed three unique scents for the experience: soil, foliage, and burning wood. After using a labor-intensive manual method to dispense the scents at first, I collaborated with the olfactory scientists at the University of Sussex to design and implement a fully automated scent delivery system, the first of its kind to be embedded with the game engine.”
This technique was so advanced, in fact, that the system was a finalist at the 2020 Art and Olfactory Awards. The project was also a finalist for the prestigious PGA Innovation Award.
VR already seemed basically limitless in its possibilities, but adding additional sensory elements expands those possibilities by an incredible degree.
In a way, this innovative move is a good representation of Hu’s sense of perspective and creativity. It’s rare to find an artist who doesn’t ask, ‘What are the limitations we need to work around?’ but instead asks, ‘What’s the absolute most we can do with this project?’
It’s much more about possibility and potential than allowing the limitations of old to curtail what could otherwise be groundbreaking work.
XR is about looking to where we can go, not where we’ve already been.