How to Get into VR

by Vincent Chen5/3/2017

This is the second edition of Paths, a series outlining emerging technologies with clear steps on how to get started in each field.

This series was designed with makers and aspiring entrepreneurs in mind. We talked to college students interested in engineering, business, and technology to figure out what resources would be most helpful to them. Then, we reached out to experts from academia, industry, or some combination of the two.

We’re excited about the potential for this series to evolve, and we’d love to hear your feedback at Macro@YCombinator.com. What would you like to learn about next?

Today, we’re going to talk about VR.

Science fiction writers and futurists dreamt up virtual reality (VR) decades ago, and hackers have been attempting to build it ever since. Today, the technology is rapidly advancing with the promise to shift modern computing into a new paradigm, not unlike smartphones did one decade ago.

Some people are skeptical about whether VR will stick around, because the technology is still very much plagued by issues like high cost, unwieldiness, and simulator sickness. However, many are excited about VR’s potential to become the most intimate human-computer interface that has ever existed.

In this post, we’ll explore why it’s an exciting time to get into VR now–both for consumers and developers. Then, we’ll discuss how a wide range of interdisciplinary fields have pushed the technology forward. Lastly, we’ll identify concrete ways in which you can get started.

Why Now?
“VR is not a new technology. It just became accessible.” – Jeremy Bailenson (Stanford VHIL)

VR technology has existed for many years, in many forms, from stereoscopes 1 to flight simulators. Academics and researchers in fields like engineering and physics have been working to make the technology feasible. In the last few decades, head mounted displays (HMDs) have emerged as the standard for delivering VR experiences.

During the late 90s/early 2000s, there were several attempts to bring VR to massive audiences. Sega announced (but never released) a headset. Nintendo launched Virtual Boy, a video game console that included a monochrome HMD. Nevertheless, these technologies were held back by a lack of visual fidelity and insufficient processing power.

Only recently have developments in VR become more visible to the public eye. CPU/GPUs have reached a point where they can provide high-fidelity, immersive experiences for reasonable prices. Smartphones have enabled mobile VR as cheaper and more accessible options that don’t require you to be tethered to high-end computer. As more people have an opportunity to try VR, it’s becoming increasingly clear how the technology might reach consumers in a big way.

Virtual Reality and Augmented Reality (AR) are Siblings
VR entails simulated worlds, and AR entails overlaid information in the real world. We can see the difference more clearly if we take the application of surgery as an example. VR can be used to simulate training for surgeons, while AR can be used to superimpose instructions and diagnostics on a live view in real-time surgeries.

Some people believe that virtual reality is a stepping stone to augmented reality. While there exist larger technological leaps necessary to achieving high-fidelity AR, the strides made in VR development can help push us there.

In this post, we avoid discussing whether VR or AR is better, more promising, or more applicable. Other people have speculated on this, so instead we will discuss their joint potential and respective engineering challenges. At this point in time, there is a wonderful opportunity to start developing for both platforms.

VR is a Concrete Engineering Problem
There is no such thing as VR… only tracking, rendering, and display. Tracking is the process of recording the user’s location and orientation in 3D space. Rendering is the process of constructing the appropriate image for a user. Display refers to the fidelity with which the hardware can produce the rendered image.

We need to solve each of these problems well enough that a user doesn’t feel sick or uncomfortable. People experience simulator sickness when cues for self-motion from the visual and vestibular systems don’t match (the same reason for car/sea-sickness).

This is a hard problem. For reference, traditional PC games render at a resolution of 1920×1080 at a refresh rate 60Hz. The Oculus Rift runs at 2160×1200 at 90Hz (over two displays– one for each eye).2 In other words, current VR solutions need to effectively render at 1080p, for two eyes, at a much higher refresh rate than PC games. At the same time, the processor needs to track a user’s location and provide that information to the headset with as little latency as possible.

Even if those specs are achieved, it’s less than ideal! VR displays have yet to cover the full human field of view, and we can certainly improve rendering quality to reach the fidelity of modern retina displays. All together, this suggests that we need 8K rendering for VR!

For now, we have the interesting engineering problem of exploiting limitations in the human visual system to optimize bandwidth and compute power. (e.g. Our peripheral vision is worse than our central vision– so why not try foveated rendering?)

VR isn’t Just for Gamers
Yes, it’s fun to shoot robots in VR.

But VR will also enable immersive concerts, reinvented museums, and live, court-side sporting events. With VR, videoconferencing will improve, with better eye contact and the inclusion of nuanced, non-verbal cues. The cost of training will plummet without the need for human trainers in industries like construction or manufacturing. At the same time, the efficacy of repeatable, hands-on training will increase. Academics will conduct social psychology research with more reproducibility, diverse sample sizes, and day-to-day realism without the need for human confederates. VR will provide a scalable way to introduce true experiential learning into education.

Personally, I’m extremely excited about the impact of VR on healthcare and medicine. The benefits of VR in training/education will also apply to clinicians. In terms of patient care, VR can be used to manage pain, combat addiction, and treat mental health issues.

No Part of the VR Stack is Mature
“The field is a molten landscape.” – Morgan Sinko (NullSpace VR)

There is no standard. No best practices. Everyone is trying something different.

Here are a few technical domains that touch VR, and specific problems within each domain:

  • Human Computer Interaction: How do we implement non-diegetic UI?
  • Optics: How can we fit a tiny projector for your eye on something with the form factor of prescription glasses?
  • Electronics: How can we optimize devices that we put on our faces for batteries, heat, and size?
  • Hardware: How can build haptics for better tracking and feedback on our bodies?
  • Computer Vision: How can we bridge the real-world and VR with 3D reconstruction and scene understanding?
  • Artificial Intelligence/Natural Language Processing: How do we create realistic agents to interact with us in VR?

Even in non-technical domains, there exist many unanswered questions:

  • Psychology: What are the effects of VR on addiction? What are the effects of avatars on identity?
  • Sociology: What will it mean to have transformed social interactions (TSI) (think automatic eye contact)?

We’re Close to Early AR, but Far from Mature AR
In many ways, it is more approachable to start prototyping in AR than it is in VR. With mobile phones, information provided by GPS and cameras can augment our experiences with the world around us (think Pokemon GO).

However, AR faces many challenges that don’t exist in VR. By nature of the experience, AR technologies benefit from being untethered, so you can take advantage of the space around you. What does this mean for computing resources if we can’t carry a high-end gaming computer on our backs?

Other challenges in AR lie in hardware and HCI– how do we create see-through displays with a large field of view? How do we create a form factor that people would actually wear in public? (Hint: remember Google Glass?)

You Can Start Building Content, Right Now
Download a game engine, like Unity or Unreal Engine, and start hacking. If you’ve developed games before, you’ll notice that the process is very familiar, except your headset is rigged to the correspond with the in-game camera.

More generally, these game engines are engineered to be very intuitive and easy to learn. They only require basic scripting and utilize interfaces with shallow learning curves (e.g. drag-and-drop visual scripting).

Here are a few useful resources for getting started:

An Understanding of the Graphics Pipeline Will Help you Appreciate VR’s Constraints and Possibilities
Fundamentally, VR is a cool application of stereo rendering in front of your eyes with head tracking. With a sound understanding of 3D geometries and how they are rendered, you’ll better understand the constraints and possibilities of VR.

Here are some good online resources:

The Fields of Vision and Imaging are Pushing the Frontier of VR Forward
It could be massively rewarding to invest in an understanding of computer vision, optics, imaging, and related topics.

On the learning process: “Approach VR with a rookie mindset.” – Aashna Mago (RabbitHole VR)
Yes, hype is real, which is all the more reason to take a step back from all the noise. Try to embody the perspective of a beginner: be willing to learn and absorb. Don’t start something because you feel like you have to. Now is a great time to learn, experiment, fail and become part of an amazing community. If you believe that you’re late to anything, I haven’t done a good job in this blog post!

Bonus Points: Read Science Fiction!
The development of VR has been surprisingly tied to science fiction. Authors in the field have envisioned the futures that engineers set out to build.3

Fun fact: Neal Stephenson (author of Snow Crash) advises Magic Leap (AR Company) as their Chief Futurist!

Must Reads:

My Personal Favorites:

TL;DR: Build Things. Talk to People.
Get your hands on a headset, and start building in areas where your skills overlap with your interests. VR is a new medium, so creating compelling, shippable content is more of an unsolved problem than hacking on iPhone apps over a weekend. You’ll run into challenges, but in those challenges, you’ll find far-reaching opportunities to fix problems that many people in the field experience.

In addition to building, talk to a wide variety of people. The process of building a beautiful experience requires the work of not only engineers, but artists, designers, and storytellers as well (game developers will understand this). Ask for feedback in online forums. Join clubs. Start clubs! Work at a VR company. The world of VR is still very small, and that’s really special for someone looking to make a big dent in the field.

Here are some online communities that might be worth looking into:

“If you believe that VR is coming…”
…that it’s going to be transformative, that it’s going to be ubiquitous, and very significant for the way that society interacts with one another, then it’s exciting to be at the very early stages.” – Jay Borenstein (Stanford CS)

“The way that people react to VR is so amazing and so visceral. It renews your faith in it, I think, and you know that it’s something special.” -Aashna Mago (RabbitHole VR)

Notes & Refs
1. Stereoscopes are devices that show different images to the left and right eye. When viewed simultaneously, these images appear in 3D.↩
2. Specs from: https://www3.oculus.com/en-us/blog/powering-the-rift/↩
3. For more information about the history of VR and it’s appearance in popular media, here’s a wonderful, in-depth resource from The Verge: http://www.theverge.com/a/virtual-reality↩

Research References
Stanford’s COMM166: Virtual People (Professor: Jeremy Bailenson)
Stanford’s EE267: Virtual Reality (Professor: Gordon Wetzstein)

A special thanks to the following people
Jay Borenstein (Lecturer @Stanford CS, Education Modernizer @Facebook)
Jeremy Bailenson (Founding Director @Stanford VHIL)
Jordan Cazamias (Engineer @MagicLeap)
Aashna Mago (Co-Founder @RabbitHole VR)
Vasanth Mohan (Founder @FusedVR)
Morgan Sinko (CEO @NullSpace VR)
Gordon Wetzstein (Professor @Stanford EE, Lead @Stanford Computational Imaging Lab)
Chris Tan (CEO @ConstructVR) …for lending their time to be interviewed

And Jeremy Bailenson, Neel Bedekar, Niharika Bedekar, Craig Cannon, Jordan Cazamias, Darren Handoko, Jason Zheng, Aashna Mago, Kat Mañalac, and Vasanth Mohan …for reading drafts of this post.

Categories

AdviceEssayPaths

Other Posts

Startups For Students Guide

January 18, 2022 by Tamanna Khemani, Kat Mañalac

Read More

Same, Same but Different with Vanta and Zapier

July 7, 2022 by Y Combinator

Read More

Learnings of a CEO: Matt Schulman, Pave, on Hiring

October 17, 2022 by Lindsay Amos

Read More
Subscribe

Author

  • Vincent Chen

    Vincent Chen is a student at Stanford University studying Computer Science. He is also a Research Assistant at the Stanford AI Lab.