Category: VR

Interacting With X-Plane in VR

I don’t think there’s been a subject that has attracted as much attention on the developer blog as VR – 274 comments on the announcement and another 90 on the next VR topic. We’ve also received tons of bug reports, emails, and Chris has even sat down and watched endless video commentary on Youtube about our VR preview. (Just a reminder, this is a PREVIEW, not a Beta. In a Beta, we expect the features to be mostly complete, though perhaps with some bugs. In this preview, we’re still actively working on the features!)

A lot of the discussion has been about how you interact with the virtual world; Chris and I have spent literally hours discussing this since VR1. In this post, I want to share our thinking and what we’re working on for VR2.

We can view VR interaction in terms of four use cases: fully virtual, virtual with touch and hardware, no touch controllers, and HOTAS. Let’s look at them in detail.

Fully Virtual

The case that works best in VR1 is a “fully virtual” work-flow – take two touch controllers and sit in your room and you can operate the entire aircraft. There are some rough edges that we are looking at.

  • Lots of users complained that the yoke movement in VR is always a joystick motion (rotation controls pitch and roll), even if the aircraft has a yoke you pull toward you to climb. Chris’s always-a-joystick choice was based on ergonomics: since it’s fully based on angle, you can move your hand to rest on something while flying without changing your pitch. This reduces the fatigue that would come with having to hover your hand over the exact position if you were using the yoke in a realistic way. With that said,  we’re looking at providing both a “realistic” mode (where motion tracks the virtual yoke, matching our other manipulators) and an “ergonomic” mode, where the control is always a joystick and you can fly from any virtual hand position.
  • There’s no way to input rudder right now if you don’t own physical pedals. Tyler is working on making the touch controllers configurable as joystick devices. With this, you can map rudder onto your joystick or D-pad, and also use some of those buttons for common functions. (This capability is useful for two-controller use cases; users with one controller and physical hardware will probably want to keep most of the VR controller buttons mapped to VR-specific functions, as some of these cannot be moved to a conventional joystick.)

Controllers and Hardware

We heard about this case in the private beta: users with touch controllers and physical hardware have a problem when the virtual control is located inside the physical control in the real world. This happens if you are flying the Cessna and have a CH Products Yoke, for example.

To solve this, we’re adding the ability to “touch” cockpit elements from a distance via the controller lasers. It’s a little harder to use than grabbing the controls directly (because small wrist motions are amplified by distance) but it lets you get at controls that would otherwise be blocked.

Laser grabbing also turns out to be really handy for airliners – the gear handle is a stretch from the captain’s chair in the 737; now you can grab it by laser.

Laser grabbing should work seamlessly with conventional direct manipulation of controls, so for users who are happy with the direct interaction in the default fleet, there should be no loss here, and no need to revert to a mouse because some of the controls are physically blocked.

Mouse Control to Major Tom

Some users don’t have touch controllers, just an HMD, a mouse and real flight hardware (E.g. a USB yoke or something). For these users, we’re working on 3-d mouse interaction with the sim. I am not sure how capable it will be, but if you used to fly with a monitor, stick, and mouse, now you can fly with an HMD, stick, and mouse; anything you would have done with the mouse before (UI clicking, 3-d cockpit interaction) stays on the mouse.

HOTAS

Some users fly with only physical hardware (E.g. a Warthog or some other HOTAS setup). This is the group of users that Chris and I are, frankly, kind of perplexed by. If you flew X-Plane HOTAS before VR, X-Plane is, right now, just as capable in VR; anything mapped to your USB hardware still works on your USB hardware, and if you have to go pick up a touch controller, you would have had to pick up the mouse anyway.  Once we have mouse support, the mouse can be “the thing you get for UI” instead of the touch controller.

We are looking at HMD-mounted UI selection, e.g. look at the UI and click a button – Rift users may be familiar with this from the “look at this warning message for 2 seconds” notice at startup.

My personal opinion here is: having used half-written mouse code, touch controllers and HMD-aimed selection, personally I thought HMD-aimed selection was not only the worst of the three ways to interact, but it was not in the same league as mouse or controllers in terms of precision. But it does provide a way (not possible right now without VR) to stay HOTAS for the entire use of X-Plane, which could be valuable.

When

I’m not sure how soon we’ll cut VR preview 2 – I wanted to get it done this week, but I think it’s going to be some time next week; we’re working on these issues now, but it’s hard to say when it’s “done” – we might try all of our “really good ideas” and realize they need more work. That’s how user-interface is…you have to try it to know whether the idea works.

Posted in Development, VR by | 156 Comments

Manipulators and VR Part 1: From Mouse to Mechanism

For airplane authors, the biggest question VR brings up is: what do I have to do to my aircraft to make this work? This is the first of three posts explaining not only how manipulation works with VR, but why we made our decisions and how we got here. Bear with me – this will end with specific recommendations for what to do with your aircraft.

A Long Time Ago…

Old-timers like me will remember that when the 3-d cockpit first came out, almost all of the “brains” of it were powered by clicking on the 2-d panel, which was “mapped” to the 3-d cockpit.  X-Plane would find out where your mouse click intersected the 3-d cockpit, find the texture location on that triangle, and map that back to the 2-d panel.

This seemed pretty cool when it came out 15 years ago, but as authors started making more sophisticated 3-d cockpits, it became clear that we needed a way to specify how the mouse worked directly with the 3-d cockpit shape, because the mapping back to the 2-d wouldn’t allow the user to do basic things like dragging a throttle.

Thus in X-Plane 9 we added the first set of manipulators – tags on the 3-d mesh of a cockpit specifying what happens when the user clicks on it. The manipulator could specify a dataref to modify, a command to actuate, and some data about what kind of mouse click operation was happening.

The key words here are mouse click. The original manipulators were designed entirely in terms of a mouse on a 2-d screen; they were designed to be exactly as powerful as a 2-d panel system, which was of coarse also totally mouse-centric. The relationship between the generic instruments and the original manipulators is pretty tight.

Moving to Mechanisms

The good part of the original manipulator system was that it was very flexible – mouse-centric handlers were a low level tool around which authors could do whatever they wanted.

The down-side of this design was that mouse-centric handlers were a low level tool around which authors could do whatever we want. We examined our own default fleet of a dozen or so aircraft and found that no two aircraft’s cockpits operated the same way, and almost all of them had at least some aspect that was hard to use – a poor manipulator choice for the job.

Knobs were, in particular, quite difficult – the original 2-d panel system used click-and-hold over a knob to actuate it, but authors working in 3-d had often used drag actions to do knobs, and the drag actions would respond differently depending on camera angle. We received approximately 8,452,124 bug reports that the knobs in the 747 were hard to use specifically because of this.

So during X-Plane 10, we added some new manipulators to X-Plane, and they had a very different philosophy: the manipulator described what was happening in the simulated aircraft, and X-Plane picked a mouse behavior to make that work. The new manipulators described knobs that turned and switches that either flipped left-right or up-down.  These manipulators reacted to the scroll wheel automatically because X-Plane knows what the knob is and therefore what a reasonable scroll-wheel interaction should be. (By comparison, with the old-style manipulators, the author has to specify what a reasonable scroll-wheel action is.)

Real Interaction Things Before It Was Cool

As it turns out, mechanism-centric manipulators are a lot better for VR than mouse-centric ones. Consider two cases:

  • The 2-d axis manipulator (ATTR_manip_drag_xy) specifies what happens when the mouse moves up-down and left-right, relative to the screen. What on earth does that mean in VR? Even if we made something that tracks up-down and left-right relative to the screens of the HMD, the results would be completely weird because the “thing” in the cockpit would not move the way your hand was moving. With a mouse, a mis-match between mouse and aircraft isn’t surprising, but in VR, it’s weird to reach out and touch something and have it behave the wrong way.
  • The knob manipulator. It describes a knob that does things when turned clockwise or counter-clockwise.  This is pretty easy to deal with – you click it with your controller and turn your wrist clock-wise or counter-clockwise and it turns.  Because we know what the mechanism is (and not just what the mouse should do) we can make a good decision about how to handle this manipulator in VR.

As it turns out, we ran into this problem of not doing what to do with the mouse, and needing to know what the mechanism was before we even started looking at VR. The exact same problem (“I want to touch the 3-d cockpit as if it was a thing and have it respond in the expected way”) exists in VR and on X-Plane Mobile!

Because X-Plane mobile runs on a touch screen and you have your physical finger on a switch, there are a lot of cases where the switch has to track really well and work the right way.  If the switch goes up and down, you want to flick your finger up to turn the switch on; if it goes left-right you want to flick left and right, and anything else is astonishing.

So X-Plane mobile set us on this path toward mechanism-based manipulators, and VR further drove us in that direction, since both have the same physical, non-mouse environment where a user directly interacts with the 3-d cockpit.

Guidance For Authors

So as an author, what should you do when working on a 3-d cockpit? My answer is: use some of the manipulators, but not others, to model the way things work in 3-d.

Use these manipulators ALWAYS

These manipulators are the best way to create certain physical mechanisms – use them every time this case applies to your aircraft.

ATTR_manip_drag_axis. Use this for a physical mechanism that slides along a linear path, like the throttle handle of a Cessna.

ATTR_manip_drag_rotate. Use this for a physical mechanism that rotates around a fixed point, like the throttle handles of a 737, or a trim wheel.  (Prefer this to the old technique of putting a drag axis along the edge of the throttle handles.)

ATTR_manip_noop. Use this to make a manipulator that blocks the mechanism behind it from being touched, e.g. the safety guard plate over a switch.

ATTR_manip_command. Use this for push buttons that actuate momentarily while a button is held down, like a button to motor a starter that you push in and hold down.

ATTR_manip_command_knob, ATTR_manip_command_switch_up_down, ATTR_manip_command_switch_left_right. Use these for knobs and switches with 3 or more positions that rotate or move. Pick the right manipulator for the mechanism!

ATTR_manip_axis_knob, ATTR_manip_axis_switch_up_down, ATTR_manip_axis_switch_left_right. These provide an alternative to the above manipulators when you must use a dataref. We recommend commands over datarefs any time you have a choice; the command system provides better interoperability between custom plugins, custom aircraft, and custom hardware.

ATTR_manip_command_switch_left_right2, ATTR_manip_command_switch_up_down2, ATTR_manip_command_knob2. Use these for knobs and switches with exactly two positions. You can use these to get a simple click-to-toggle with the mouse (quicker and easier for mouse users) while getting real 3-d movement in VR.

Use These Manipulators Sometimes

These manipulators are not perfect fits with physical motions and may require tweaking for VR, but they’re the best options we have now for certain problems. Try to use something from the always list instead if you can. Do not use these manipulators if the mechanism you are simulating matches something from the list above.

ATTR_manip_drag_xy. This is the only 2-axis drag we have, and is the best choice for eye-ball vents and the yoke. The yoke is special-cased in VR and should be based off of a 2-d xy manipulator. We are looking into more powerful multi-dimensional manipulation in VR, but this work won’t be complete for 10.20.

ATTR_manip_command_axis. This manipulator runs a command once based on moving a lever down or up. You should probably use a drag axis or command up-down switch, but there may be some odd mechanisms in aircraft where this manipulator is a good fit. It should not be your first-choice go-to manipulator.

ATTR_manip_push, ATTR_manip_radio, ATTR_manip_toggle. These manipulators can serve as alternatives for push-button style controls when you absolutely have to write to a dataref and not use a command.  WARNING: Do not ever use these manipulators for things that move, e.g. don’t use these for banana switches, spinning knobs, or things like that.

Do not use these manipulators

These manipulators were for mouse-centric movement and should be replaced with knob or switch manipulators.

ATTR_manip_delta, ATTR_manip_wrap. These were designed to simulate knobs by click-and-hold – use the knob manipulators instead.

ATTR_manip_drag_axis_pix. This was designed to simulate a knob by click-and-drag – use the knob manipulators instead.

We Love Commands

As you may have noticed from this list and the ever-growing size of X-Plane’s built-in command list, we love commands – they have become our go-to mechanism for our aircraft. Commands have a few advantages over data-refs:

  1. There is a clear design in the plugin system for changing a command’s behavior. So a third party aircraft can easily “take over” the existing engine start command.  This is very, very difficult to accomplish via datarefs.
  2. Commands can be mapped directly to joystick and keyboard hardware. There is no way to bind a joystick to a dataref.*
  3. Commands can be mapped to sound to capture the act of the user “doing” the thing, regardless of whether it works. For example, if you want the click of the starter switch regardless of whether the starter motor engages (maybe the battery is dead), commands make this easy to do. Often there is no dataref for ‘the user is trying but it’s not working’.

In X-Plane 11 we’ve been moving to commands to get our planes ready for VR, the new manipulators, and FMOD sound all at once.

In the next post I’ll cover some of the issues specific to VR and the 3-d cockpit.

 

 

* Developers who write plugins that provide external interfaces to X-Plane: make sure you have a way to provide access to commands and not just datarefs!  There are things you can do only with commands and not datarefs.

Posted in Cockpits, Development, Plugins, VR by | 12 Comments

You Can Use VR With Any Edition of X-Plane 11

X-Plane 11’s native VR support uses SteamVR, so let me clear this up now:

You will be able to use VR directly in X-Plane 11 with any distribution of our sim:

  • An X-Plane 11 digital download from Laminar Research (or any other reseller) if you have a 24-digit product key.
  • An X-Plane 11 DVD set from Laminar Research (or any other reseller) if you have DVDs.
  • X-Plane 11 on Steam if you bought X-Plane on Steam.

VR will work for X-Plane 11 no matter how you bought it. You do not have to have bought X-Plane from Steam to use VR.

(This post does not contain any guidance on supported hardware – I’d like to wait until we get further into beta and get some feedback on how system requirements affect actual users. This is just a reassurance that you didn’t buy X-Plane “the wrong way” for VR.)

What Is SteamVR

SteamVR is a free download from Steam’s app store that X-Plane (and many other games) use to talk to your VR hardware. We use SteamVR for both the HTC Vive and the Oculus Rift – X-Plane’s native VR support requires that you have both the Oculus drivers and SteamVR.

Read More

Posted in Development, VR by | 100 Comments

X-Plane Public VR Preview Released!

After an intense year of development and a few demos of the technology at various Flight Simulator conferences, it’s finally time to let you all in on a preview of X-Plane 11 with VR support built-in! I will admit that I personally thought this was another technologic fad that was going to fade without ever gaining traction but I’m happy to say that I think I was wrong. Once you try a VR headset, you’ll never want to fly without one again. For the first time EVER, you can fly precisely and accurately by looking around and interacting with the cockpit without being anchored to a narrow field of view and small clickable hotspots viewed from unnatural angles. VR lets you have unlimited freedoms to move your head and body around naturally. You also get a sense of scale for the first time. Objects and manipulators are the right sizes and distances…just stand next to the tires or engines on one of the airliners and you’ll understand what I mean by scale. Read More

Posted in News, VR by | 279 Comments