[This post is a “behind the scenes” look at the tech that makes up the X-Plane massive multiplayer (MMO) server. It’s only going to be of interest to programming nerds—there are no takeaways here for plugin devs or sim pilots.]
In mid-2020, we launched massive multiplayer on X-Plane Mobile. This broke a lot of new ground for us as an organization. We’ve had peer-to-peer multiplayer in the sim for a long time, but never server-hosted multiplayer. That meant there were a lot of technical decisions to make, with no constraints imposed by existing code.
Thanks to the users who filed bugs (especially Bill), we now understand the issues with the HP Reverb G2, but the fix is not in 11.51r1. The fix needs more testing than will fit into this patch. If you have an HP Reverb G2 and haven’t filed a bug, please do, so we can find you to send you a possible test build. In the meantime, we won’t hold up 11.51 and the new Gateway airports; rather we can work on the G2 in parallel.
When we released X-Plane 11.51 beta 2 last week, we included up-to-date Aftermath support. Aftermath is an NVidia driver feature that catches detailed information when the GPU crashes (e.g. “device loss” crashes).
Full Aftermath debugging slows X-Plane down. The sim is still flyable, but you might go from 50 to 35 fps, for example. It’s noticeable, so we didn’t just turn Aftermath on for all eligible users. It’s too big of a perf hit to just leave it on all the time.
Unfortunately, while we know from auto-crash reporting that device loss errors are happening, we also know that no one is using Aftermath to capture detailed information that we could use to find and fix the potential problems in X-Plane.
So: if you hit device loss errors while flying with Vulkan on Windows with an NVidia card, please follow these instructions and run with Aftermath for a little bit. If you can drop us a few auto-crash-reports with Aftermath enabled, it could get us the key breakthrough we need to fix device losses.
Tyler also fixed some low level networking bugs. This doesn’t change how multiplayer fundamentally works – if you can’t do a LAN flight across your WAN or you need command line magic to get the right NIC, that’s all still true and really not in the scope of what we’re fixing in 11.51. This whole beta run is targeted bug fixes.
X-Plane 11.51 Beta 2 is now available. (Release notes here.) Here’s a few more details on bug fixes we are working on.
Device Loss Errors on Windows
A device loss error occurs when shaders running on the GPU crash. In the old days this might hang or blue screen your computer, but fortunately we live in the age of enlightenment – the GPU catches the error, stops running X-Plane’s shaders and leaves a note for the Vulkan driver to tell X-Plane “hey, you your code died.”
NVidia’s “Aftermath” is a diagnostic tool that can tell us why our shaders crashed on the GPU. When we tried to use it in the past it crashed, but NVidia has since updated their drivers, so we are trying again.
Aftermath can collect lightweight or heavy crash info; the lightweight crash info doesn’t hurt FPS, so it is now always on. Heavy crash info significantly lowers FPS, and must be turned on by the command line –aftermath flag.
So…we are looking for a few brave volunteers. If you:
Have an NVidia card with driver 457.09 on Windows 10 and
Sometimes see device loss errors during your flights and
Can live with some FPS loss for a little bit to fix these errors
Please run with
and auto-report any crashes that come up. If you get a device loss error with Aftermath running, the automatic crash report will contain all of the info we need.
Device Lost Errors on OS X
Device loss errors can happen on OS X in Metal, too – the mechanics are the same as Vulkan. We are aware of one AMD driver bug that caused them which we have worked around in 11.51b1. If you still see device loss errors in 11.51 betas, please file a bug, as we don’t have automatic reporting for these.
HP Reverb G2
We are looking into controller problems with the HP Reverb G2. In X-Plane 11.51b2, the grip trigger should start working again, but the default configuration will still be weird.
The problem appears to be that SteamVR identifies the first-gen and G2 WMR controllers the same way; we are still looking into this. If you have these controllers and haven’t yet sent us a bug report, please try them with 11.51b2 then send us your log with the additional diagnostics.
Clicking Settings Crashes on Windows
A few users have seen crashes when opening the settings menu on Windows, before ever turning on Vulkan. As best we can tell, the crash happens when we open the settings screen because we have to go inspect the Vulkan driver to see if it is usable, and that code goes off the rails. We have a possible fix for this; our theory is that it happens to users who have various third party “layers” (layers are basically plugins for the Vulkan driver) that have gone off the rails.
We are still fixing bugs in the Gateway airports export; it’s not ready for beta 2, but it should be in the next beta after this one.
This is a beta so make backups of your work before using!
Custom Spill Lights
Custom Spill Lights are now implemented (#312)! To use, make a light datablock with the XPlane2Blender light type as Custom Spill. Relevant properties are:
RGB color for the light
Spot light direction
Light Spot Size
The width of spot light
Size parameter (in meters)?
Dataref that controls the light
Custom Spill lights can make dataref driven custom omni-directional billboards and directional spot lights. As with Automatic Lights, the goal is What You See Is What You Get.
In this photo the green light is a Custom Spill, where sim/graphics/animation/lights/traffic_light changes the color between green, yellow, and red! The white lights have different widths, all made without doing any math by hand.
X-Plane has pre-made high quality GPS devices that are easily accessible to the artist. Thanks to (#481) using one is as simple as making a mesh, and giving it a material with a Cockpit Device set. Pick the device, at least 1 electrical bus (matching Plane Maker), the lighting channel, and if the screen’s brightness should auto-adjust. You’ll get a fully functional GPS device just like that! You can have multiple devices in the same OBJ, as well as use of the panel texture.
The Cessna’s cockpit provides 2 great examples of using cockpit devices.
Cockpit Features UI
Given the changes and additions to our Cockpit Features, the UI has been changed slightly. On the material Properties tab use the “Cockpit Features” to find “Cockpit Regions” and “Cockpit Device”. The updater should adjust the setting for you if you were using regions.
(#426) Shadow Blend’s Blend Ratio can finally be set. Thanks @kbrandwijk! (#548) Non-Exporting Lights are back in. They’re intended as “work lights” and will never ever show up in the OBJ! Simply set the X-Plane Light Type to “Non-Exporting” and use freely.
Light Level can now be used multiple times in the same .obj without tricks or getting lucky!
Thanks to everyone who downloaded alpha.1! I’m glad that this series has been so successful and I can wait to see what you make with the new light features! As always make backups!
On Tuesday Apple announced new Macs powered by Apple’s M1 chip, a custom ARM system-on-a-chip based on the Apple A-series System on a Chip (SoC) from the iPhone and iPad.
The rest of this post is probably only of interest to Mac users, but for Windows users, it’s worth noting that the M1 chip is fast. It targets laptop and low power use cases, not gamer-class hardware, and it’s not a discrete GPU. Here’s my 27″ iMac – Intel says the i9 in it is a 95W part:
The take-away here is that Apple doesn’t just have fast chips for their new machines, they might have the fastest ones.
Now, how is this going to work with X-Plane and plugins?
X-Plane 11 is an x86_64 app, as are all plugins ever written for it. So if you run it on an Intel Mac, it just works, and if you run it on one of the new ARM Macs, it will run using Rosetta, which will translate the code as you fly.
In the future, we will have an X-Plane build that is “universal”–that is, it contains ARM and x86_64 code, and we will have a plugin SDK that contains both ARM and x86_64 code. At this point, plugin authors can start recompiling plugins to contain both types of code as well. Users with ARM Macs will have the choice to (1) run ‘natively’ in ARM for higher performance and use only plugins that are universal or (2) continue to run x86_64 code under Rosetta, so that all plugins work.
(This option is available for all apps that are universal on an ARM Mac – you turn “Use Rosetta” on or off in the app properties.)
This situation is exactly the same as the PPC->x86 transition we went through years ago.
Plugin developers: once Big Sur and the new X-code are out and we have an ARM plugin SDK, you can add a new architecture to your project and that should be it, as long as you don’t use any x86 assembly code in your add-on.
It’s been a while since I have posted about what the team is working on, and given all that has happened in the last few weeks, it feels like a million years. Here’s a run-down of…stuff.
X-Plane 11: Beta Time
Today we are putting out X-Plane 11.51 beta 1. This is a bug-fix patch for X-Plane 11.50 with a handful of random fixes that we have accumulated over the last few weeks. Release notes here. You will not be auto-notified of this beta–you have to pick it in the installer if you really want it.
I expect the beta to be relatively short, as we’re just trying to put out fixes for things we’ve found since we’ve shipped 11.50, improve diagnostics, reduce crashes, etc.
11.50 beta 1 does not have new Gateway airports. We’ll include them very soon–probably in beta 2–we had a few last minute snags, so I pulled them out of beta 1 to avoid delay.
Road Map: Graphics and Performance
X-Plane 11.50 represents the first step in our long term performance road map: moving to modern, low overhead, high-performance rendering APIs. These APIs are multi-core friendly; for X-Plane 11.50 this results in better overall FPS and smoother performance, but only an incremental increase in multi-core use.
One stealth performance feature in X-Plane 11.50: plugin object instancing. X-Plane has had an instanced drawing API for several years now, but with 11.50 we saw widespread plugin adoption. This is going to be very important for performance going forward; the instancing APIs are designed for efficiency, particularly in a multicore environment.
We have now switched gears and we are working on new features in the engine itself, e.g. we are working on what we draw and not so much how we draw it. In other words, we are working on graphic enhancements, new features, etc.
The new features are, as they are being coded, already taking advantages of new tech made possible by Vulkan and Metal, e.g. GPU compute kernels, GPU-based culling, etc.
Once we finish rendering features, we can pivot back to performance and push hard on multicore. The next multicore goal is to be able to render multiple views in parallel using multiple cores. Parallel rendering has several benefits:
An X-Plane frame often has sub-views rendered to form the main view (e.g. shadows, water reflections, cube maps, in-cockpit cameras, etc.). Any concurrency we expose makes the sim faster in these scenarios, and they are common.
Right now while multi-monitor is possible with X-Plane, it is very expensive performance-wise. Having a frame that can be farmed out to multiple cores would make multi-monitor less of a performance hit.
Note that multi-core multi-monitor would still be single GPU, and it would be a win because right now CPU time limits multi-monitor setups.
What about multiple GPUs? That’s something we’ll have to look at after we have multicore on the CPU–without it second GPU support doesn’t help.
Big Sur & the Mac
There’s been a lot of Apple news this week that’ll have to wait for a separate post. We recommend waiting on Big Sur for a few days until we’ve had a chance to test it a bit. Hopefully that’s an easy ask, as right now the download servers appear to be overloaded.
This alpha contains changes to the updater. Make backups before using!
Global Material Settings -> OBJ Settings
This new version contains one of the most requested fixes (#357, #599) for XPlane2Blender: Normal Metalness, Blend Glass, and Global Tint have been moved out of the Material Properties Tab and into OBJ Settings! The annoying “All Materials in an obj must have the same Normal Metalness/Blend Glass Value” error is gone!
The updater tries its best to guess if you wanted Normal Metalness, Blend Glass, or Global Tint, however it isn’t perfect. If you have to manually correct more than 3 of your OBJ settings, please tell me.
These new settings can be found under the Textures section of the OBJ Settings.
Emissive Panel Texture Only and Panel Mode Selector
#595 Also known as ATTR_cockpit_lit_only, this directive has actually been in X-Plane since 11.10, but now is accessible in XPlane2Blender! It makes a panel only use the emissive “Lit” texture – a great speed boost if that is all you need. This is the perfect feature for computer displays.
For Aircraft or Cockpit export types, look in the Cockpit section for “Panel Mode”. This changes the meaning of “Part of Cockpit Panel” for the whole OBJ. Setting it to “Emissive Panel Texture Only” mode activates the feature. You cannot mix panel modes.
People who use Cockpit Regions will have to manually change “Panel Mode” from Default to Regions to see the UI and have regions export again. A one time fix. A future updater will do it for you. Until then I hope it isn’t too many OBJs to change.
Updater Version History Synchronization
#471 The updater now synchronizes its last version across every scene and new updater functions will not use data cleaning to protect against accidental or purposeful updater re-runs. Simply put this means a safer to use XPlane2Blender (but still make backups). Since this is new updater code, however, it had to be put in an alpha. People with multiple scenes should be especially on the look out for problems. I feel very confident about it however.
With testers and users like the ones XPlane2Blender has we’ll be getting to v4.1.0-rc.1 much much much much faster this time! For the users who requested moving Normal Metalness and Blend Glass, thank you for your patience. I know now just how annoying that error message was! I’ll certainly keep this experience in mind for future decisions and don’t worry, “make it the same across every material” is not going to be chosen again without an extremely important reason!
Back in the day, the way you put in a feature request for X-Plane was to email Austin – his email address was all over the website. So was his phone number – if you really wanted the request you could just call him, but it might hurt your chances. Austin had a big text file where he copied all of the emails, and then he randomly jammed things into X-Plane. It was the wild west.
Fast forward twenty years – we have a development team, we have customer support, we have an art team, and X-Plane has a lot more users than it used to. So we made an official place to record feature requests: feedback.x-plane.com.
The feedback request board allows for voting, so please look through the existing requests and up-vote it if it’s already there–this lets us easily find very popular requests.
The request board covers X-Plane desktop and mobile. You can also request features for end users (“better clouds”) or for third party developers (“scenery packs can edit the mesh around airports”)–it’s all good.
Looking through the “most wanted” right now, it looks like our internal high priority items (most of which are in progress) match the most wanted list pretty well, which I think is a good sign for our upcoming dev work.