[This post is a “behind the scenes” look at the tech that makes up the X-Plane massive multiplayer (MMO) server. It’s only going to be of interest to programming nerds—there are no takeaways here for plugin devs or sim pilots.]
[Update: If you’re interested in hearing more, I was on the ThinkingElixir podcast talking about this stuff.]
In mid-2020, we launched massive multiplayer on X-Plane Mobile. This broke a lot of new ground for us as an organization. We’ve had peer-to-peer multiplayer in the sim for a long time, but never server-hosted multiplayer. That meant there were a lot of technical decisions to make, with no constraints imposed by existing code.
Read More
Amazingly, I think the answer (and I realize I am cursing the next beta by typing this) is we’re getting really close. You’d be forgiven for thinking that’s lunacy given beta 15; here’s a little bit of info about the state of the betas.
X-Plane 11.50 beta 15 is out (and marked “unstable”) on Steam and amongst other things, has two big fixes for beta 14:
- Cloud performance should be back to baseline norms for 11.50 – when I fixed a multi-monitor bug in beta 14, I accidentally turned off a major cloud performance optimization,
- Fatal errors while resizing the window should be fixed. These squawks were coming from code that now checks much more heavily for error conditions, and revealed a problem when the OS delivers window resizes to us too slowly.
Note the introduction of new bugs in the process of fixing old ones – this is always a risk when a bug fix is intrusive or complicated; in order to get a shippable product, we have to keep ratcheting down the amount of chaos we introduce per beta, making smaller and more surgical changes.
Beta 15 also tried to fix one last plugin compatibility bug that we discovered very late in the game – under Vulkan, there’s no depth/stencil buffer available to third party plugins, resulting in incorrect drawing for previously working plugins.
I think you know where this is going…this new change introduced new bugs, even as it fixed the problem with the original add-on.
We have been working with third party developers over the last twenty four hours on fixes for the regressions here; our hope is to have a new beta on Monday or Tuesday that fixes these issues and has gotten a good third party once-over.
We’re Going to Have to Close the Door
We are reaching the end of X-Plane 11.50 – at this point the number of remaining bugs to be fixed in this beta is small enough that they aren’t that hard to keep track of. To get to a release candidate though, we’ll need to stop introducing major changes.
I am hoping that we already know about all of the third party incompatibilities, because at this point we have to close the door to complex changes to improve backward compatibility. For beta 16 we are fixing what used to work and is now broken (e.g. yes, SkyMaxx will work again), but if anyone is sitting on an add-on problem they haven’t mentioned, we’re out of time to deal with it.
Plugin Developers: Thread Safety!
Plugin developers: in looking at your compatibility with X-Plane 11.50, please re-read this post and check your usage of threads. We’ve had really helpful responses from third parties who we have notified about threading issues we’ve seen by auto-crash reporting, and I think this will help everyone–third parties, users, and us.
I believe that X-Plane 11.50 is significantly more sensitive to threading violations than 11.40, because the Vulkan driver doesn’t spend CPU cycles protecting itself from abuse. If a plugin calls into us from a thread when it’s not allowed to, this can cascade into crazy X-Plane behavior, which cascades into crazy Vulkan behavior that the driver won’t stop. So careful adherence to the threading rules by plugins is critical to Vulkan stability.
With Vulkan and Metal, X-Plane is now firmly in the driver’s seat for VRAM management. This lets us eliminate stutters that were previously present with OpenGL and almost impossible to avoid. It has, however, one big and noticeable downside: when you run out of VRAM, you get blurry textures.
Of course the goal wasn’t to replace stuttering with blurry textures, and we believe that given the normal work load of X-Plane, you should not be seeing this. The fact that so many users are seeing blurry textures, especially on big cards with lots of VRAM, points to the VRAM code being buggy in all the ways beta code can be buggy.
How much VRAM do I need?
Just to get the obvious out of the way, our system requirements have not changed for 11.50. Our minimum VRAM requirement for X-Plane 11 is still 1Gb. We expect these cards to be able to run 1080p with lowered texture resolution, providing an equal experience to X-Plane 11.41. With 2Gb we expect users to be able to run HDR with medium texture resolution on 1080p systems. 4Gb and higher should be able to run HDR at with high resolution textures with 2k monitor resolutions. With 6Gb and higher, 4k monitor resolution should be possible.
Read More
A week or two ago we had a very dead beta, and posed the question of how to incrementally test betas in the future. We got a variety of responses, ranging from “private test it first” to “roll it out in a wave” to “full speed ahead, we know betas are bumpy.”
Since then, we’ve been doing one of the easiest and probably most useful things we can: posting the betas early to third-party developers who are in our developer Slack channel.
Beta 7/8 had a ton of changes, and our third-party developers found multiple problems, some of which we wouldn’t see in our internal tests. So we held off on releasing betas 7 and 8 to the public while we fixed those issues. Until today.
There are a lot of changes in this one since it’s kind of 3-in-one. Please see the full release notes here.
Are We There Yet?
X-Plane 11.50 has been similar to X-Plane 11.20 (our VR) release and different from what we normally try to do, in that when we went beta (both private and public), the work for Vulkan wasn’t done yet. We had something that you could fly with, that was delightful for some users (and unstable for others), but we also had a big list of things we still needed to do.
So are we there yet?
Read More
This post is just targeted at plugin developers who are modernizing their object drawing – if you don’t write plugin code, the Cincinnati Zoo has been showing their animals on Youtube – it’ll be a lot more entertaining than this post. (An XPLMInstance cannot tunnel down two feet in fifteen seconds – one point for the zoo animals.)
XPLMInstance makes a persistent object that lives inside X-Plane that is visible in the 3-d world. It changes how you draw from “run some drawing code every frame” to “tell X-Plane that there is a thing and update its data every now and then.”
Instancing is actually a lot easier than draw callbacks! But there are two tricky gotchas:
1. You must create the custom DataRefs for your OBJ’s animation before you load the object itself with the SDK. (If the DataRefs do not exist at load time, the animations are disabled as “unresolved to any DataRef”.)
2. When you create the instance, make sure your custom DataRefs are on the list of DataRefs for that instance.
Here’s the really baffling thing: if you create the custom DataRef and then add it to the instance’s list, your DataRef callbacks will not be called.
Wha?
Here’s the trick: the DataRef you register is a global identifier, allowing the object to refer to what it wants to listen to. That’s why you have to create the DataRef – so that the identifier exists.
But when you create an instance, each instance has memory that holds a different copy of those DataRefs.
For example, let’s say you have a truck with four DataRefs, and you make five instances. X-Plane allocates 20 slots (four DataRefs times five instances) to store five copies of each DataRef’s values.
The instances never look at the DataRef itself. They only look at their local copies. That’s why when you push different data to the instance with XPLMSetInstancePosition, each instance animates with its own values – each instance looks at its own local data.
This is also why you won’t see your DataRef callbacks called (unless you use DataRefEditor or some other tool). The object rendering engine isn’t looking at the DataRefs themselves, it’s looking at the local copies.
In other words, XPLMInstance turns DataRefs from the pull model you are used to (X-Plane pulls on your read function to get the value) to a push model (you push set with XPLMSetInstancePosition into the instance’s memory).
This implies two things about your add-on:
- It doesn’t really matter what your DataRef read functions do – they can just return zero, and
- You can’t use tools like DataRefEditor or DataRefTool to debug your animations. (That didn’t work well in legacy code either, but it really won’t work now.)
If you try the obvious optimization of not creating your custom DataRefs (“hey, no one calls them”) before you create your instance, you will find that animation just stops working. This is because we need the DataRef to be that global identifier to match your instance data with the animations of the object itself.
One last note: if your old code used sim/graphics/animation/draw_object_x/y/z to determine which object was being animated (from inside a plugin “get” function) you do not need to do this anymore. Because each instance has its own local copies and your DataRef function isn’t called, this technique is obsolete.
In summary:
- You must register custom DataRefs.
- Their callbacks can just return 0 – they’ll never be called.
- Always list your custom DataRefs for animation when you create an instance.
- Do not use draw_object_x/y/z; use XPLMSetInstancePosition to create per-specific-instance animation.
Sigh. Well, foo.
I was going to write a post about X-Plane 11.50 beta 5 – what’s new in it, the new ways we are debugging GPU crashes, the crash bugs we’ve fixed, etc. A lot of stuff that we thought was pretty good went into beta 5. Cool new technology! Big bug fixes! Lots of winning!
As it turns out, beta 5 is dead. I hit “go” on the release this afternoon, and half an hour ago, I hit “stop.” The auto crash reporter was showing way too many new crashes in memory management that we had not seen before, and this strongly implies a new and serious bug.
Laminar Installer Users: if you were auto-notified to update to beta five and did so, and you are not crashing, you can keep flying! If your beta five is just a smoldering wreckage of crumpled VRAM and GPU parts, you can re-run the installer with “get betas” option checked, and it will take you back to beta 4.
If you were not auto-notified to update to beta five, that’s probably for the best. Please stand by and keep flying beta four; we’ll post a new beta when we’ve gotten to the bottom of this. We have enough captured crash data to investigate.
Steam Users: we did not release the beta five build to Steam and this is probably a good thing; we’ll try again with a new release that isn’t made of plutonium and unicorn hallucinations.
And if you’re going “why didn’t y’all test it before you released it”…we did! None of our machines show these crashes. But we also have probably a dozen PCs total we can run on. Moving to a new driver stack has meant learning about the weird things that happen on your computers and not ours.
Do We Need a Two Tiered Beta System
This came up in our impromptu beta five post-mortem meeting: do we need to bring people into new betas in stages? With code for new drivers, beta five probably won’t be the last beta where we code something we think is helping and discover that it fails catastrophically, but not on our hardware. We need beta victims^H^H^H^H^Htesters to find these bugs, but once we get a dozen crashes, we don’t need anyone else to stub their toe for us to fix our problems.
So we thought about two possible ways to do this:
- A two-tiered system. Early adopters could get an email and hand-update to the new beta before it is put out for auto-update notification.
- Send out the beta update notifications over time, e.g. 10% of users get notified immediately, then another 40% if we don’t see crashes, then the last 50%. (This practice is actually industry standard on mobile apps.)
If you are reading this blog post, this far down, you are probably participating in the beta; I’d be curious what approach you’d find most useful.
We were discussing a particularly exasperated sounding bug report on one of the internal Slack channels when I realized that this might not be obvious: a crash with the error message “pipeline must not be null” – it’s one error message that covers a whole category of bugs. We fixed one major case (skycolors were broken) in b1 and added one major case (custom billboard lights on aircraft) in b2 – conservation of pipeline bugs!
Null pipelines are a new category of crash in X-Plane 11.50, so here are a few notes on what this error is and what you can do to help us fix them (and what you don’t need to bother with).
What Is a Pipeline?
A pipeline is just the Vulkan and Metal term for a shader (plus some extra gak (1)) that we use to do our drawing.
X-Plane 11.41 would ask the OpenGL driver to build shaders as it needed them, and then the driver would turn those GL shaders into hardware pipelines on the fly as it got presented with different scenarios.
Not 11.50. We build everything up front. Vulkan has two rules:
- Using a pipeline is fast.
- Building a pipeline is not fast.
This is a great pair of rules for us – it means if we build our pipelines at load time, we are not going to have stutters mid-frame.
Why Are We Crashing?
There is one down-side to the 11.50 way of doing things: if we don’t build all of the pipelines we need up front during load, then when it comes time to draw, we’re toast. That’s what a “pipeline must not be null” error is – it just means the loading code did not create the pipeline the drawing code needs.
Why not just build every pipeline we could ever possibly need? Load time. X-Plane can build hundreds of thousands of pipelines depending on rendering settings, scenery packs, custom aircraft, etc. We actually did “just build everything” early in our development process and the sim could take half an hour to load.
So we try to build only the pipelines we need. If we build too many, we slow load, and if we build too low, you see this error.
What Do You Do When You See This Error?
On Windows and Linux, it’s really easy: close the alert box and when the auto crash report form comes up, please press “send”. Don’t bother with you email or a message; everything we need to kill this bug is already in the auto report! (Jennifer’s edit: please DO include your email address with any auto report if you want us to be able to confirm we have your specific report! This is the only way we have of identifying who it came from.)
The good news is: the auto crash reports for the pipeline crashes are insanely easy to find and fix.
Mac users: if you see one of these, we need the Apple crash report – please send it in a bug report.
(1) for the plugin developers that know some OpenGL: a pipeline is basically a GLprogram (shader) plus a bunch of the fixed function state that goes with it: blending, depth/stencil, vertex format, FBO format, and some rando stuff thrown in.
The idea is to have the pipeline contain so much information that there is no risk that the driver has to build two hardware shaders for one Vulkan shader (to cope with other fixed function state) no matter how weird the hardware is.
On lots of actual hardware, the pipeline has stuff that’s not actually in the shader, but some surprising things, like vertex format, actually often are.
We have updated the X-Plane plugin SDK and related documentation for X-Plane 11.50. Here are some links:
New Plugin SDK: version 3.02 adds the new modern 3-d drawing callback for OpenGL/Vulkan, and deprecates the drawing callbacks that are unavailable in Vulkan/Metal. Download it here.
(Updating to the new SDK is not mandatory for your add-on to be Vulkan-compatible, but it can help catch drawing callbacks that will not work, and it is necessary to use the new 3-d drawing tech.)
Plugin Compatibility Guide: a short guide that focuses on issues existing plugins might have with X-Plane 11.50. If you are updating a plugin, read this!
OpenGL Drawing Guide: complete documentation for all aspects of drawing using OpenGL with X-Plane. If your plugin uses OpenGL, this is a must-read.
Drawing Over 3-d in 2-d: this sample code shows how to draw in a 2-d window or drawing callback with coordinates that match the 3-d world. Many 3-d plugins that draw need this kind of drawing, e.g. to label routes or aircraft with UI that matches the 3-d world.
Instancing Example Plugin: a sample plugin that draws objects using the XPLMInstancing APIs. Many add-ons will need to transition from object drawing to instancing to be compatible with Vulkan. We strongly recommend moving to instancing – it provides the fastest, most compatible drawing path, particle system support, and in the future will support FMOD sound.
Instancing is actually much easier to use than XPLMDrawObject and bypasses a lot of complexity and chaos. Instancing works with all versions of X-Plane 11, and the sample has everything you need.
X-Plane 11.50 has been out for a little bit more than 24 hours, and things have been a little bit nuts. Here are a few quick notes, in no particular order.
Bug Fixes and Work Arounds
While I don’t have work-arounds for the missing right eye in VR or older NVidia cards that won’t run in Vulkan, the good news is that we have fixes for these already. We are going to start testing beta two on Monday and try to get the fixes we have out as soon as possible. While we don’t have every major reported bug fixed, beta two should make a real difference.
Users who can’t start and have SLI setups: disable SLI in the Nvidia control panel and you will be able to run Vulkan. We are still investigating this – our goal is a bug fix so you don’t need to turn SLI off. (We do not expect X-Plane to leverage both cards – we expect it to run without failing.)
Finally, one thing I should have mentioned in the announcement: if you have scripts that modify art controls, please remove them, and don’t put them back.
The art controls are undocumented and subject to change, and they have changed a lot since X-Plane 11.41. Realistically the authors of these tweak scripts need to go back and re-evaluate every one of their tweaks in 11.50 to see if they actually help or are actually making things worse.
This is not like plugins or scenery; we tell you to get a second installation for the beta not because we want you to run add-on free but rather so that if the beta fails you are not locked out of X-Plane. We expect add-ons to work and we are taking plugin, scenery and aircraft bugs seriously.
By comparison, the art controls are “do what you want, but you void the warranty if you mess with this.” If you are running scripts that hack the art controls, we cannot tell the difference between real bugs in the early betas and art controls screwing things up.
The Road Map For Betas
Looking over the bug reports we have received, I think we are going to take on the 11.50 bugs in three phases:
- Stability and compatibility. We’ll start by making sure that we run Vulkan and Metal on every platform that should be able to run them, with add-ons just working in the cases where we expect them to. We’ll start by focusing on fixing crashes, black screens, device lost, unstable plugins, etc.
- VRAM use. We’ve received a number of reports that make it sound like VRAM management is not working properly. Once we can run, we’ll dig into blurry textures, running out of VRAM, etc. Sidney has built some great tools to get a good picture of how VRAM is being managed. VRAM management is one of the newest and most complex parts of 11.50 so it isn’t surprising that we’ve seen things that look buggy.
- Performance. Once we are running where we should and using VRAM that we should, we can look at the cases where users are not seeing performance benefits from Metal and Vulkan, as well as remaining stutters. Once again, Sidney has built some fantastic tools that should help us dig into this quite efficiently.
This is the only order that we can reasonably approach the bugs. If the app won’t run on all qualifying hardware, we can’t test our VRAM use everywhere, and if our VRAM use isn’t correct, it can bias performance testing.
Today is the day. X-Plane 11.50 public beta 1 is now available for download to everyone. (Steam users: the beta is staged on the servers, and we’ll let it loose this afternoon if it isn’t setting people’s machines on fire.)
I want to extend my appreciation to the 50+ third party developers who participated in the eleven (!) private developer previews. This is probably the most complex single patch we have developed with regards to third party add-ons; their help testing and trouble-shooting issues was invaluable.
Like all public betas with the number “one” in the title, this is a very new, very untested beta; if you wish to try it, maybe keep your regular X-Plane install around. If you post in the comments that you got the beta and now you can’t fly and you are sad, probably the other commenters are just going to say “I told you so” a few hundred times.
Getting The Beta
Non-Steam users can get the update by running the installer, checking the “get betas” box, and running; Steam users will be able to get beta soon by picking public betas in the application properties.
If you can run X-Plane 11, you should be able to run X-Plane 11.50; if you have an old operating system or hardware, you may be stuck running X-Plane 11.50 with only OpenGL as the driver. Here’s a break-down of what you need to run with the Vulkan or Metal drivers:
Operating System:
- Mac users need 10.13 or newer.
- Windows users need Windows 10—we do not support Vulkan on Windows 7.(1)
- Linux users need a distro that can run reasonably recent proprietary NVidia or AMD drivers.
GPU Hardware (Windows and Linux):
- NVidia: GeForce 600 seres or newer
- Any GCN or newer AMD card (E.g. HD7000 series or newer)
GPU Hardware (Mac):
- Any GPU that comes with a Mac that can run 10.13 or newer or
- Any eGPU supported by Apple
- Hackintoshes: we have heard anecdotally that GeForce 10 series on a Hackintosh does work, but we do not have this hardware in the company, so YMMV.
Drivers (Windows and Linux):
- NVidia: 440.26 or newer
- AMD: 19.12.3 or newer
To run with Vulkan or Metal, there is a new check-box in the rendering settings screen – just check the box and restart. If Vulkan or Metal crashes on startup, we will turn the check-box off automatically so you are not locked out of X-Plane.
(1) In theory Vulkan should run on Windows 8, but our devs are all running Windows 10. If you have Windows 7, please update your OS – Microsoft isn’t security patching Windows 7, so it’s more than time.
The First Run Will Take a While To Load
For some users, the first time you start the sim in Metal or Vulkan you may see several extra minutes of load time while X-Plane compiles shaders. This will not happen every time you start the sim, so my advice is: get a beverage appropriate to the time of day and let the sim load.
What you’re seeing is shaders compiling. X-Plane 11.40 compiled shaders while you flew, resulting in stutters. Try this in 11.40: start the sim cold and dark at night, and turn on the battery and landing light. Feel those stutters? That’s the sim going “oh noes, I need the terrain shader with the landing light now”.
X-Plane 11.50 loads every shader it could need up front during preload of scenery, but that’s a lot of combinations. On the first load, the compiled shaders are saved to a disk cache, so load times will be back to normal the next time around.
We are working to make the first load faster, but because it’s a one-time thing it didn’t seem like a good reason to keep everyone out of the beta.
Will My Add-Ons Work?
First, we expect every add-on that does not use undocumented hacking to “just work” in OpenGL – if you find this is not the case, please report a bug.
We expect most add-ons to just work in Vulkan and Metal; we are keeping a list in the release notes of add-ons that we know do not work and require an update from the developer.
Please go easy on your third party developers – in some cases they will need to update code that has “just worked” for over a decade, and the author of a popular add-on will probably hear from users about updating to Vulkan hundreds of times a day. Some add-on authors are part-time or have day jobs, so as a community we will need to be patient.
Third party developers: we should have public docs and an SDK update by the end of the week with complete info about 11.50, but if you were not in the developer preview program and want to dig in now, ping us.
I Found A Stutter!
While we have made tremendous progress with stutters compared to X-Plane 11.41, I can tell you that as of 11.50 beta 1, we are not done, because Sidney is working on fixes for stutters that did not make the first public beta.
The good news is, we’ve reached a point where we can identify and kill stutters in a straight-forward manner – it’s never just “dark matter in the OpenGL driver” anymore. We also have tools built into the sim to collect stutter reports from users – we’ll help you use that once we kill off the remaining known ones (which appear to be in Windows itself – that’s how far we’ve come).
I Found A Bug!
The release notes do list some known bugs, some of which we already have fixed for an upcoming beta 2. If you find something you think is a bug, please file it. Commenting about the bug here, emailing us, and discussing it on a forum do not count! If you file it, it goes into our tracking system so we don’t lose it.
