This two-part article describes the state of virtual reality technology and research at the beginning of 2017. I'm writing it for a technical audience. For a more philosophical discussion for a general audience, see my Virtual Reality: The Actuality of Total Cinema talk video.
 |
| Peeking out from under a virtual reality display prototype |
Setting the Standard
Virtual reality (VR) is a collection of technological visions for an immersive computer-driven experience. Different groups have different definitions, and definitions are important for technology--scientists and engineers need to specify what they're building.
Telepresence
I put the base level of VR technology at
telepresence: the point at which the player's senses and instincts treat the virtual world as real. This happens even through the player knows at a cognitive level that it is not. Today's
best VR experiences already achieve this, although many low-end "VR" experiences fall a little short.
Telepresence requires at least real-time 3D rendering, 6 degrees-of-freedom tracking of the player's head (and ideally hands), a wide field of view, and low latency (delay) between motion and the image updating.
It is easy to spot when a first-time VR player is experiencing telepresence. They start laughing hysterically and keep saying, "I'm really in this other place." And it is indeed a magical feeling, even after having experienced it many times.
Holodeck VR
 |
Holodeck VR: The Holodeck fictional virtual reality
technology depicted in Star Trek: The Next Generation |
To me, the ultimate goal of VR is the
Holodeck on
Star Trek: a room that you enter which can become any location.
Objects appear indistinguishable from reality with physical form, weight, smell, and every other property. You can walk without limitation in any direction and don't have to wear any special equipment because it is built in to the room.
The Holodeck is probably long way in the future. The primary challenge is having the player unencumbered.
Matrix VR
 |
Matrix VR: The Matrix is a dystopian film featuring
fictional virtual reality technology and martial arts |
A more near-term goal is something like the VR shown in
The Matrix,
Harsh Realm,
eXistenZ, or
Total Recall, in which the computer simulation interfaces directly with the player's mind.
This avoids the mechanical problems of physical senses by simply bypassing them, while providing an experience identical to the Holodeck.
This kind of technology could be practical within a decade, if there was really demand for it. However, like most people, I have reservations about using technology that directly interfaces with my nervous system.
Avalon VR
 |
Avalon VR: The fictional VR headset from the film Avalon
resembles today's real-world products. |
For a non-invasive VR experience the most practical near-term goal matches the systems described in
Snow Crash,
Ready Player One,
Avalon, or
The Lawnmower Man.The VR systems in those books and films have head-mounted displays, haptic harnesses, and lots of tracking devices. These bear great physical resemblance to products available today. However, the specifications on today's products are far below what is necessary for total immersion.
I hope that in the next five years we'll see consumer technology for VR that resembles this "Avalon VR" vision. Most of this article is devoted to describing progress and future challenges towards this goal.
Players will have seamless and convincing physical interaction with objects through audio, haptic,
and visual feedback; be able to move relatively freely (although perhaps not by walking); make changes to the virtual world; and have virtual bodies. The book
Ready Player One proposes many solutions for heterogeneous input devices, user interfaces, and even economics of such an environment.
Augmented Reality
 |
Visualization of what augmented reality would
seem like, by Magic Leap |
Augmented reality takes the real world and seamlessly adds virtual 3D content. The fictional hologram displays in Star Wars, Minority Report, Iron Man, and Avatar are a weak form of AR.
The Microsoft
Hololens demo is actually a better depiction of the goal of AR than most films. Just beware that this demo as captured on video doesn't really match the experience you'd have today with the product: the camera's limited field of view and editing out of failure cases makes an admittedly impressive technology seem too much like magic.
Arguably, the Holodeck experience is augmented reality, although the point in the Holodeck is to replace everything except the players so I consider it more VR than AR.
Limited AR experiences, such as Google Glass, just provide a kind of heads-up display over the real world. These are useful for maintaining situational awareness and eye contact in scenarios where today we would look down at a cell phone or tablet. However, they are only a small step towards the much greater application of integrating apparently-3D virtual objects into the real world.
What I've been describing so far is free-space AR that can appear anywhere in a room around you and is observed through some kind of special glasses. There are also two variants that are much more limited, but also sufficiently practical as a result of those limitations that they are already widely deployed.
Projection mapping re-colors an existing real-world surface. There are several firms that specialize in applications of this technology, such as
watershed simulation in a real sandbox or
public art shows.
Hand-held display AR superimposes virtual elements onto a live viewfinder video feed, such as on a cell phone.
Pokemon Go was a recent low-end example of this, where the compositing of real and virtual objects was ignorant of lighting and depth.
Ikea's AR tablet and phone app is a more compelling example that allows visualizing furniture in real environments.
Marketing VR
Anything remotely 3D and interactive has probably been marketed as VR in the past. For example, in 1993, the original DOOM game. The practice of using the term "virtual reality" loosely in marketing continues.
 |
| id Software's DOOM in 1993 |
Today, many panoramic, stereo films (also known as "360 video") are marketed as "virtual reality". These lack any interaction, the player has no avatar, and the player can't move in 3D. These are really just personal versions of a stereo OmniMax film.
This is still interesting content to view with a head-mounted display, but it doesn't really capture the full potential of the VR vision. In the extreme, some that don't even use a head-mounted display are called "VR": these are just panoramic videos played through viewers that notice when you've rotated a hand-held mobile device. That's a nice user interface, but I think labeling it "VR" is disingenuous.
Some head mounted displays for watching traditional movies are called "VR" platforms, even though they lack every element of VR--they are really just fixed displays seen through lenses to make them easier to focus on.
Current Technology
Components
A consumer VR system features:
- a head-mounted display (the "goggles") containing a cell-phone panel and two lenses to make it possible to focus on the device up close
- audio, usually through headphones connected to the head-mounted display
- head tracking
- hand tracking, typically through a hand-held controller
- a computer, which is built into the display for mobile
Displays
Current displays provide about 1080p resolution to each eye at 90-120 Hz, which is about 2-3x smoother updates than a typical console game. They offer about 110 degrees for a horizontal field of view, which is only half of what you can see in real life but is enough to at least feeling claustrophobic.
Cables
For high-end, desktop-powered VR platforms, there's a thick cable connecting the display to the computer. This is needed for several reasons.
 |
| Desktop VR requires a thick power and video cable today |
The most significant reason to use a cable is just power consumption. Think about how fast your phone battery runs down if you stream video from the web or run the camera for a long time. Desktop VR content is about ten times as rich as a streamed HD movie, so you'd run out of power in about fifteen minutes with a cell phone-sized battery or about an hour with an external head-mounted power pack. A cable allows the display to be light and run cool.
The image quality and delay in frames can also be several times better when using a wired connection than a wireless one. This is why professional e-sports game players use wired mice and gamepads even though wireless ones are more convenient...and why they use low-latency monitors.
It is
just possible to transmit video to a head-mounted display wirelessly today, but only for short periods and at a risk of increasing latency (some of which can be hidden by software if you are moving very smoothly). The announced
TPCast add-on product for HTC Vive makes it wireless and there are rumors that Hololens and the next Oculus Rift will be wireless as well. However, when VR resolutions and frame rates jump in late 2017 or 2018, wireless solutions will again become impractical...at least until there are comparable advances in that technology.
This arms race will likely continue for a decade for desktop VR. Mobile VR entirely avoids the problem by simply rendering on the display itself, but has quality limitations as a result.
Lenses
 |
The pre-distorted image sent to the display of the Oculus Rift.
When viewed through the lenses, this looks relatively normal. |
The lenses in a VR display allow you to focus at an apparent depth of about a meter even though the device is only centimeters from your eyes. Unfortunately, they also create chromatic aberration (color fringing), barrel distortion (fisheye), and glare. The images shown are pre-distorted to attempt to counter some of this, but at today's resolutions it is impossible to make the entire screen look good.
Latency
To avoid headaches and motion sickness in VR, the platform must provide not only high frame rates and a reasonable resolution and field of view, but also low
latency. Latency is the delay between when you move your head and when the image on the screen updates.
There's a big difference between latency and throughput. Throughput can be measured in frames per second or pixels per second. Latency is just time.
Consider a car assembly line. With 1000 stations of robots and humans working on a car as it moves through the factory, one finished car might emerge every minute from the factory. That's high throughput. It is what we're used to worrying about for 3D games.
Now, think about the path of one individual car. It might take two days for that car to go from raw metal at the front of the factory to a finished car at the back after the 1000 different assembly operations are performed. This is the latency. It doesn't matter how many cars were produced in two days if we're waiting for one specific car.
When you move your head in VR, you don't care about the throughput of how quickly you can see out-of-date image frames update. What you want is for the latency to be very low for the new frame showing the rotated view to display.
That is, in VR, if latency is too high (say, around 50 milliseconds), then it appears that you are dragging the world with you as you move, and it then snaps into place after a moment. This makes your visual system react as if
it is malfunctioning, and you essentially get sea sick. The best VR systems keep perceived latency down to about 10 ms today through a combination of prediction, fast components and algorithms, and warping.
The power of the GPU and quality of the software greatly affect latency as well. Unfortunately, the last two decades of computer architecture all
increased latency to increase throughput. This means that a lot of the image quality that we see in the best 3D games is not possible in VR right now. We have to strip down rendering pipelines to make them render with low latency.
Tracking
 |
Valve's diagram of their inside-out tracking of
the signal from light house beacons |
For the images on the display to update correctly, your head must be tracked very accurately in the real world.
Inside-out tracking systems such as the HTC Vive compute the position of the head and hands by looking at reference points in the world from the head-mounted display.
The HTC Vive specifically uses "light house" laser beacons for a kind of radar-like triangulation, but it is possible to do this based solely on cameras. The laser beacon approach scales well to many objects in the VR space and gives the lowest latency and highest accuracy of technologies available today
Outside-in systems such as the PSVR and Oculus Rift use a camera mounted near the computer to look at the head-mounted display. When properly calibrated, these can be very accurate and have the ability to potentially track individual fingers and whole bodies in the style of Microsoft's Kinect devices. Outside-in systems also don't require power sources for the tracked objects, which is why Oculus Rift's controllers rarely need the batteries changed--they are primarily there for transmitting button presses.
Most tracking systems are augmented by accelerometers in the display and controllers, which drift over time but provide very low-latency updates of rapid or unpredictable movement.
Scales
The best VR experiences today are called "nomad VR" or "
room scale". These allow you to walk freely around an area of at least a few square meters. Cleverly designed applications can make that area feel much larger by adjusting content to steer you back towards the center of the space.
 |
The SteamVR tool accepts both room-scale
and standing configurations |
It is of course hard to create room-scale content because the experience designer doesn't know what size
your room is. Content has to be designed to automatically resize or to choose the smallest common size, perhaps something like 1.5mx1.5m.
Standing VR captures some of the telepresence of room scale by making the player stand in a single place. The player has freedom to duck, jump, lean, and turn, but not to walk. Locomotion must be accomplished by other mechanics such as teleportation or driving a vehicle.
Seated VR is the most restrictive. The player is seated, ideally in a swivel chair to at least allow continuous rotation. Sitting can greatly reduce telepresence and limits player head motion and thus parallax. However, it is a natural fit for experiences and is the most practical to deploy...especially for mobile VR, which tends to lack good absolute tracking today and could leave the player stumbling into real-world objects if standing or walking.
Specific Hardware
The first steps on the path to Avalon VR-style technology began long ago. Stereo viewing glasses are at least 150 years old.
Sutherland and Sproull built a head-mounted computer display with positional tracking in the 1960's.
Even early Holodeck VR technology
began in the 1990's. Technology similar to today's consumer head-mounted displays has been in research labs for over a decade.
However, only recently was this technology available to the general public. Economies of scale from cell phone production drove down the cost of two key technologies for virtual reality: cameras (for tracking) and high-resolution display panels (for the head-mounted display). This made it possible to build consumer VR platforms for about US$500. Simultaneously, US$200 consumer GPUs became powerful enough to drive these displays in real-time.
 |
Sutherland and Sproull's "Sword of Damocles"
...circa 1964 |
In 2016, the first mass-market consumer VR platforms launched. New platforms are now being announced every month, and the existing ones are rapidly being upgraded. At this moment (January 2017), the most significant platforms are:
- HTC Vive, which has the best tracking and display for freedom of movement and visual immersion
- Oculus Rift, which has the best controllers for natural interaction
- PSVR, which has the best movies and games available for it
- GearVR, Google Daydream, and Google cardboard, which are the most affordable and the easiest to use
Expect new devices, such as midrange
Windows 10 head mounted displays, to emerge very soon. Lenovo announced their low-cost headset shortly after this article was first published.
It would be reasonable to assume that every major peripheral vendor and tech company from Amazon and Apple to Dell and Razer has a VR product currently in development. Likewise, expect rapid iteration from Google, Oculus, and Valve (co-maker of the Vive) on their tracking and controllers this year and in the future.
Several augmented reality devices are also coming to market.
Microsoft Hololens is the most sophisticated that has been announced so far, but many more are coming soon, including
CastAR,
Magic Leap,
Meta,
ORA-2,
AiR, and
Moverio. VR is just in its infancy. AR has much farther to go, and few of these are likely to survive the technological and economic hurdles of the market, let alone provide a good consumer experience in the next year. However, they are really important for engineering and research applications.
Content
The virtual reality experiences available are rapidly expanding, with an average of one significant game or film releasing
per day. Some are cross-platform, but most are platform-exclusives funded by the platform vendors.
As a rule of thumb for the high end, right now PSVR has a great lineup of games and films, Oculus Rift is second, and HTC Vive has the smallest library of good content. Expect that to change significantly in the next six months as vendors start creating more of their own content and platform-exclusive deals expire.
Games
 |
The Climb is an action-packed VR game that is
extremely comfortable, self-paced, and nonviolent |
Some fully interactive experiences that I recommend are:
 |
Fans couldn't wait for Valve to make an official VR game
and released their own, quite good, Portal Stories |
- The Lab - Valve's free demo suite provides some of the most comfortable and immersive content available
- The Climb - a beautiful and well-designed arcade interpretation of extreme rock climbing
- Google Earth - fly over the entire 3D planet, from satellite height to street level. It is easy to lose track of hours here with virtual travel despite a confusing control scheme.
- Vanishing Realms - a dungeon-crawling experience with some of the best movement controls and satisfying hack and slash action
- Superhot VR - a fully-formed, native VR experience that is also a great post-modern video game
- Playstation VR Worlds - PSVR's answer to The Lab, which sadly is not free. However, the London Heist and Ocean Descent portions are worth the price of the whole bundle.
- Star Wars: Battlefront X-Wing VR: exactly what you'd hope it would be
- Fast Action Hero - a rapidly iterating shootout simulator available while in development
- Portal Stories - a free fan game set in the world of Valve's Portal games makes clear how great a full-budget PortalVR title could be.
- Robo Recall - the trailer makes this look like yet-another-shooter, but Epic has polished the VR gameplay for two years to make the world feel completely solid and the gameplay maximally fun. Slightly limited by confusing teleportation mechanic and lousy hand tracking on the rift.
- Budget Cuts - James Bond meets satire in this promising demo of an in-production title
- AudioShield - an aerobics class disguised as something reminiscent of TRON
These will soon be eclipsed by many others, but right now offer the best blend of comfort (i.e., not making the player motion sick), interesting content, good controls and mechanics, and quality visuals.
 |
ADR1FT is a great game if your stomach can handle
tumbling in zero-G. Unfortunately, many people's can't. |
There are plenty of other experiences such as
ADR1FT,
EVE: Valkyrie,
I Expect You To Die, and
Edge of Nowhere that have amazing elements, but which I can't recommend to general audiences because of some design flaw or motion sickness issue.
Although there are some VR patches and fan tributes, there are some worlds that haven't yet received full VR releases which I think most gamers are eagerly hoping to see announced. For example,
Portal,
Ethan Carter,
Skyrim,
The Witness, Star Wars, DayZ, Left 4 Dead, Mechwarrior, and
Deus Ex. It is hard to imagine that the owners of that intellectual property won't follow through on this market potential.
Films
 |
| Allumette |
There are thousands of VR "films" available. Some of these, like
Henry and
Allumette, are rendered in real-time so that you can move freely in the environment by walking around. You can't affect the story or change the world, but it feels solid and three dimensional.
The best two real-time non-interactive VR experiences I've had are:
- Pearl - Google's upbeat and interesting father-daughter story, marred only by some awkward cuts and rendering dropouts during them.
- Dear Angelica - Oculus' gorgeous mother-daughter story...with somewhat buggy camera placement and the usual Rift distortion in the perphiphy.
Either would be credible as an animated short were the technology a little more robust (I'm viewing these on top of the line equipment, so the problems are clearly software).
Others are "360 video"...stereo panoramas in which you can look around, but not move from the center of the world. These are the most common, because they are the easiest to produce and can be live action.
White Room 02B3 is a nice example.
A few of the 360 video films allow some limited interaction, changing the story or at least its pace slightly based on your gaze.
Colosse and
WILD - the experience are both in this category. I think we'll see some more of this.
I have yet to see a VR film that wouldn't have been better as a conventional film with faster pacing and tighter editing. However, I expect that to change. Content creators are still bravely exploring the medium, and largely have to move cautiously as they work out the conventions and capabilities of this new medium.
 |
| NextVR's live VR sports viewer |
That said, I think that the killer application of passive virtual reality content will be 360 videos of sports games and theatrical performances. This is content that is relatively cost-effective to capture compared to new narrative experiences.
It is reasonable to expect audiences to pay something like US$10 to see an event in VR that they would pay US$300 per seat for live, when VR can increasingly capture a significant portion of the experience of being present. Apps such as FOX Sports VR, Lionvision VR, NextVR, and VOKE TrueVR are currently available for this purpose, and I think major networks are likely to launch their own native applications in 2017.
For mobile VR, the other great killer application is watching regular non-VR films inside a virtual reality theatre. This allows you to watch any movie on your phone on a bus or plane and feel as if you were in a theatre. Several apps are available for each of the mobile VR platforms today for this experience.
The same technique could theoretically be used to create virtual monitors to provide a desktop-computing experience with only a keyboard, phone, and network connection to a virtualized PC. However, software for doing so in a turnkey fashion doesn't exist--yet.
Part 2 of this article describes the hardware, software, and user interface challenges we face for virtual reality in 2017.
Morgan McGuire (@morgan3d) is a professor of Computer Science at Williams College, a researcher at NVIDIA, and a professional game developer. He is the author of the Graphics Codex, an essential reference for computer graphics now available in iOS and Web Editions.
