Author: Ben Supnik

Help Us Find Those Devices

When we released X-Plane 11.51 beta 2 last week, we included up-to-date Aftermath support. Aftermath is an NVidia driver feature that catches detailed information when the GPU crashes (e.g. “device loss” crashes).

Full Aftermath debugging slows X-Plane down. The sim is still flyable, but you might go from 50 to 35 fps, for example. It’s noticeable, so we didn’t just turn Aftermath on for all eligible users. It’s too big of a perf hit to just leave it on all the time.

Unfortunately, while we know from auto-crash reporting that device loss errors are happening, we also know that no one is using Aftermath to capture detailed information that we could use to find and fix the potential problems in X-Plane.

So: if you hit device loss errors while flying with Vulkan on Windows with an NVidia card, please follow these instructions and run with Aftermath for a little bit. If you can drop us a few auto-crash-reports with Aftermath enabled, it could get us the key breakthrough we need to fix device losses.

Beta 3 Stuff

X-Plane 11.51 Beta 3 is here – here are the release notes. The most notable change is that we now have the new Gateway airports!

Tyler also fixed some low level networking bugs. This doesn’t change how multiplayer fundamentally works – if you can’t do a LAN flight across your WAN or you need command line magic to get the right NIC, that’s all still true and really not in the scope of what we’re fixing in 11.51. This whole beta run is targeted bug fixes.

X-Plane 11.51 Beta 2 Bug Fixes

X-Plane 11.51 Beta 2 is now available. (Release notes here.) Here’s a few more details on bug fixes we are working on.

Device Loss Errors on Windows

A device loss error occurs when shaders running on the GPU crash. In the old days this might hang or blue screen your computer, but fortunately we live in the age of enlightenment – the GPU catches the error, stops running X-Plane’s shaders and leaves a note for the Vulkan driver to tell X-Plane “hey, you your code died.”

NVidia’s “Aftermath” is a diagnostic tool that can tell us why our shaders crashed on the GPU. When we tried to use it in the past it crashed, but NVidia has since updated their drivers, so we are trying again.

Aftermath can collect lightweight or heavy crash info; the lightweight crash info doesn’t hurt FPS, so it is now always on. Heavy crash info significantly lowers FPS, and must be turned on by the command line –aftermath flag.

So…we are looking for a few brave volunteers. If you:

• Have an NVidia card with driver 457.09 on Windows 10 and
• Sometimes see device loss errors during your flights and
• Can live with some FPS loss for a little bit to fix these errors

Please run with

--aftermath

Device Lost Errors on OS X

Device loss errors can happen on OS X in Metal, too – the mechanics are the same as Vulkan. We are aware of one AMD driver bug that caused them which we have worked around in 11.51b1. If you still see device loss errors in 11.51 betas, please file a bug, as we don’t have automatic reporting for these.

HP Reverb G2

We are looking into controller problems with the HP Reverb G2. In X-Plane 11.51b2, the grip trigger should start working again, but the default configuration will still be weird.

The problem appears to be that SteamVR identifies the first-gen and G2 WMR controllers the same way; we are still looking into this. If you have these controllers and haven’t yet sent us a bug report, please try them with 11.51b2 then send us your log with the additional diagnostics.

Clicking Settings Crashes on Windows

A few users have seen crashes when opening the settings menu on Windows, before ever turning on Vulkan. As best we can tell, the crash happens when we open the settings screen because we have to go inspect the Vulkan driver to see if it is usable, and that code goes off the rails. We have a possible fix for this; our theory is that it happens to users who have various third party “layers” (layers are basically plugins for the Vulkan driver) that have gone off the rails.

Gateway Airports

We are still fixing bugs in the Gateway airports export; it’s not ready for beta 2, but it should be in the next beta after this one.

ARM Macs

On Tuesday Apple announced new Macs powered by Apple’s M1 chip, a custom ARM system-on-a-chip based on the Apple A-series System on a Chip (SoC) from the iPhone and iPad.

The rest of this post is probably only of interest to Mac users, but for Windows users, it’s worth noting that the M1 chip is fast. It targets laptop and low power use cases, not gamer-class hardware, and it’s not a discrete GPU. Here’s my 27″ iMac – Intel says the i9 in it is a 95W part:

Single core: 1265 Multi-core: 9414

and here’s a new M1-based MacBook Air, with 8 cores running at ten watts:

Single core: 1732 Multi-core: 7545

That’s…a pretty high score for Apple’s first trip into desktop land. One more for perspective:

AMD’s new Ryzen 5900X, which is a great chip, with a 105W TDP:

Single Core: 1619, Multi-core: 13656

The take-away here is that Apple doesn’t just have fast chips for their new machines, they might have the fastest ones.

Now, how is this going to work with X-Plane and plugins?

X-Plane 11 is an x86_64 app, as are all plugins ever written for it. So if you run it on an Intel Mac, it just works, and if you run it on one of the new ARM Macs, it will run using Rosetta, which will translate the code as you fly.

In the future, we will have an X-Plane build that is “universal”–that is, it contains ARM and x86_64 code, and we will have a plugin SDK that contains both ARM and x86_64 code. At this point, plugin authors can start recompiling plugins to contain both types of code as well. Users with ARM Macs will have the choice to (1) run ‘natively’ in ARM for higher performance and use only plugins that are universal or (2) continue to run x86_64 code under Rosetta, so that all plugins work.

(This option is available for all apps that are universal on an ARM Mac – you turn “Use Rosetta” on or off in the app properties.)

This situation is exactly the same as the PPC->x86 transition we went through years ago.

Plugin developers: once Big Sur and the new X-code are out and we have an ARM plugin SDK, you can add a new architecture to your project and that should be it, as long as you don’t use any x86 assembly code in your add-on.

Stuff We Are Working On

It’s been a while since I have posted about what the team is working on, and given all that has happened in the last few weeks, it feels like a million years. Here’s a run-down of…stuff.

X-Plane 11: Beta Time

Today we are putting out X-Plane 11.51 beta 1. This is a bug-fix patch for X-Plane 11.50 with a handful of random fixes that we have accumulated over the last few weeks. Release notes here. You will not be auto-notified of this beta–you have to pick it in the installer if you really want it.

I expect the beta to be relatively short, as we’re just trying to put out fixes for things we’ve found since we’ve shipped 11.50, improve diagnostics, reduce crashes, etc.

11.50 beta 1 does not have new Gateway airports. We’ll include them very soon–probably in beta 2–we had a few last minute snags, so I pulled them out of beta 1 to avoid delay.

Road Map: Graphics and Performance

X-Plane 11.50 represents the first step in our long term performance road map: moving to modern, low overhead, high-performance rendering APIs. These APIs are multi-core friendly; for X-Plane 11.50 this results in better overall FPS and smoother performance, but only an incremental increase in multi-core use.

One stealth performance feature in X-Plane 11.50: plugin object instancing. X-Plane has had an instanced drawing API for several years now, but with 11.50 we saw widespread plugin adoption. This is going to be very important for performance going forward; the instancing APIs are designed for efficiency, particularly in a multicore environment.

We have now switched gears and we are working on new features in the engine itself, e.g. we are working on what we draw and not so much how we draw it. In other words, we are working on graphic enhancements, new features, etc.

The new features are, as they are being coded, already taking advantages of new tech made possible by Vulkan and Metal, e.g. GPU compute kernels, GPU-based culling, etc.

Once we finish rendering features, we can pivot back to performance and push hard on multicore. The next multicore goal is to be able to render multiple views in parallel using multiple cores. Parallel rendering has several benefits:

• An X-Plane frame often has sub-views rendered to form the main view (e.g. shadows, water reflections, cube maps, in-cockpit cameras, etc.). Any concurrency we expose makes the sim faster in these scenarios, and they are common.
• Right now while multi-monitor is possible with X-Plane, it is very expensive performance-wise. Having a frame that can be farmed out to multiple cores would make multi-monitor less of a performance hit.

Note that multi-core multi-monitor would still be single GPU, and it would be a win because right now CPU time limits multi-monitor setups.

What about multiple GPUs? That’s something we’ll have to look at after we have multicore on the CPU–without it second GPU support doesn’t help.

Big Sur & the Mac

There’s been a lot of Apple news this week that’ll have to wait for a separate post. We recommend waiting on Big Sur for a few days until we’ve had a chance to test it a bit. Hopefully that’s an easy ask, as right now the download servers appear to be overloaded.

Where to Send Feature Requests

Back in the day, the way you put in a feature request for X-Plane was to email Austin – his email address was all over the website. So was his phone number – if you really wanted the request you could just call him, but it might hurt your chances. Austin had a big text file where he copied all of the emails, and then he randomly jammed things into X-Plane. It was the wild west.

Fast forward twenty years – we have a development team, we have customer support, we have an art team, and X-Plane has a lot more users than it used to. So we made an official place to record feature requests: feedback.x-plane.com.

The feedback request board allows for voting, so please look through the existing requests and up-vote it if it’s already there–this lets us easily find very popular requests.

The request board covers X-Plane desktop and mobile. You can also request features for end users (“better clouds”) or for third party developers (“scenery packs can edit the mesh around airports”)–it’s all good.

Looking through the “most wanted” right now, it looks like our internal high priority items (most of which are in progress) match the most wanted list pretty well, which I think is a good sign for our upcoming dev work.

X-Plane 11.50 Is Here

X-Plane 11.50 quietly went final yesterday. Very quietly.

We’re trying something new with this release: a phased roll-out. Here’s how it works:

• 11.50 is final – if you run the installer and request updates, you’ll get 11.50, regardless of whether you check ‘get betas’.
• Only some users are being auto-notified of the update.
• Over the next few days, we’ll slowly crank up the number of auto-notifications until it is everyone.
• Steam users: you still have to opt into the beta – we’ll mark the Steam 11.50 version as final probably tomorrow.

We are trying the phased roll-out to make sure support doesn’t get overwhelmed; some add-ons need to be updated for 11.50 compatibility, and we didn’t want Thomson and company to go down the tubes helping users get their add-ons updated.

X-Plane 11.50 Release Candidate Three

Third time’s a charm? I hope so. We cut a new release candidate this week.

The main reason for the new RC: the very latest AMD drivers (2020.8.3) crash X-Plane in Vulkan. We are working with AMD on this; in the meantime you can run the 2020.8.2 drivers. We wanted to get an error message into the app guiding users – hard crashes were spiking in our auto reporting.

Since we were recutting, we also included fixes for non-working manipulators, and some GL clipping of plugins and the map by airplane wings.

I expect this RC to go final, although I realize I’m tempting fate by putting it in writing. We have a few more fixes we are sitting on for an 11.51 bug fix patch to avoid introducing last minute changes in RC.

X-Plane 11.50 – Release Candidate 2

X-Plane 11.50 release candidate 2 is out. We changed a very, very small amount of code from RC1. If you think your performance changed radically from RC1 to RC2, there’s a good chance it’s an unrelated factor, because we didn’t change code that would affect performance.

The big change in this build is a pair of fixes to two crashes, both of which were related to texture paging and could happen more with orthophotos. One of those crashes was in the betas and was a result of a timing bug; in RC1 I cleverly fixed the timing – causing the crash to happen way more often. This is now fixed, so if RC1 was way worse than the betas for you crash-wise, RC2 should fix it.

The other was a crash we’d see after long flights that has been plaguing mostly Mac users (but sometimes Windows users too) for a while – we finally chased it down.

We do have a handful of issues we are still looking at, and I expect we’ll do an 11.51 bug fix patch, as we have done a bug fix patch on all of our big updates; there’s always something we find after we ship.

X-Plane 11.50 Release Candidate 1

We posted the first X-Plane 11.50 release candidate today. (Release notes here.) X-Plane 11.50 is not final however–to get the release candidate, you still need to have “get betas” enabled.

The changes from beta 17 to RC1 are not very invasive; we’re trying to tweak carefully and not destabilize the build we have. Sidney and I spent most of the last two weeks looking at tons and tons and tons of performance traces from users, looking for performance bugs to fix.

A Note on Addons

A lot of developers have updated their add-ons for 11.50 – either to gain Vulkan compatibility, to modernize old code, or to fix bugs that were revealed when running with X-Plane 11.50.

We still see a lot of crashes in our automatic crash report collector from add-ons where we know that the author has fixed the underlying issue.

If you run an X-Plane installation that has been “heavily enhanced” – you know who you are, with the 1500 scenery packs and the 100 plugins – I’d suggest that X-Plane 11.50 might be a good time to build your system back up from a clean install. This makes it easier to only run the latest versions of add-ons that are Vulkan compatible, rather than having to fish through your full past install to find that one plugin or script that’s causing a problem.

If you use FlyWithLua to whack art controls (or run scripts from people who do this) I’d suggest removing them when you get 11.50 running, then put them back later. A ton of art control tricks simply don’t do anything anymore, and some are now harmful to performance or cause crashes.

Incremental Roll Out

One last note – because 11.50 is such a disruptive release (in that it requires add-ons to be updated to run in Vulkan), we are working on a staged release. While everyone who wants to get the RC can get it, and everyone who wants the final version will be able to get it, we are trying to make the auto-update notice ping the user base incrementally, so that our tech support is not overwhelmed.

If you don’t see the update prompt for a new beta or final release when you launch X-Plane, it may be us testing out this system. Just run the installer manually with the “get betas” box checked to get the latest no matter what.

X-Plane 11.50 Beta 17 – Performance Fixes

X-Plane 11.50 Beta 17 is out now – full release notes here. There are a lot of bug fixes in this beta. We are getting near the end of the 11.50 run and the fixes are becoming smaller and more targeted as we try to lock things down.

We fixed several performance issues on the GPU and CPU, on both platforms. Probably the biggest single fix is Mac cloud performance with Metal and no-anti-aliasing; it turns out that having the rendering surface shared between Metal and OpenGL on a Mac puts it into a layout that slows down the GPU. (This is not an issue on Vulkan.) We’ve fixed this by using separate VRAM for cloud rendering vs plugins; GPU performance with Metal should match OpenGL.

VR users: water reflections are fixed, as well as hopefully the crash on quit with Nvidia + Rift headsets, and we’ve tried to fix the VR mouse not being available until a controller is activated on WMR headsets.

Multi-monitor users: we finally figured out why the horizon line wasn’t lined up – it turns out each monitor was using a different monitor’s height and chaos ensued.

If your bug number isn’t listed in the fixes, you don’t need to tell us it’s still broken, but if we did list a bug as fixed and you still see it, please file a bug and include the XPD number if you know it.

Oh, and Mac Catalina users: you no longer have to reboot your machine after an update. (This was an installer issue – the new installer rolled this week.)

Third party developers: Thomson made this survey form! Basically we’re trying to decide where the best place(s) are to discuss future dev, so we figured we’d get some feedback before proceeding. We’ve had some discussions of future extensions to the SDK on Slack, but not all third party devs have time to monitor a Slack channel, and we don’t want to leave people out.