X-Plane 11.50 release candidate 2 is out. We changed a very, very small amount of code from RC1. If you think your performance changed radically from RC1 to RC2, there’s a good chance it’s an unrelated factor, because we didn’t change code that would affect performance.
The big change in this build is a pair of fixes to two crashes, both of which were related to texture paging and could happen more with orthophotos. One of those crashes was in the betas and was a result of a timing bug; in RC1 I cleverly fixed the timing – causing the crash to happen way more often. This is now fixed, so if RC1 was way worse than the betas for you crash-wise, RC2 should fix it.
The other was a crash we’d see after long flights that has been plaguing mostly Mac users (but sometimes Windows users too) for a while – we finally chased it down.
We do have a handful of issues we are still looking at, and I expect we’ll do an 11.51 bug fix patch, as we have done a bug fix patch on all of our big updates; there’s always something we find after we ship.
We posted the first X-Plane 11.50 release candidate today. (Release notes here.) X-Plane 11.50 is not final however–to get the release candidate, you still need to have “get betas” enabled.
The changes from beta 17 to RC1 are not very invasive; we’re trying to tweak carefully and not destabilize the build we have. Sidney and I spent most of the last two weeks looking at tons and tons and tons of performance traces from users, looking for performance bugs to fix.
A Note on Addons
A lot of developers have updated their add-ons for 11.50 – either to gain Vulkan compatibility, to modernize old code, or to fix bugs that were revealed when running with X-Plane 11.50.
We still see a lot of crashes in our automatic crash report collector from add-ons where we know that the author has fixed the underlying issue.
If you run an X-Plane installation that has been “heavily enhanced” – you know who you are, with the 1500 scenery packs and the 100 plugins – I’d suggest that X-Plane 11.50 might be a good time to build your system back up from a clean install. This makes it easier to only run the latest versions of add-ons that are Vulkan compatible, rather than having to fish through your full past install to find that one plugin or script that’s causing a problem.
If you use FlyWithLua to whack art controls (or run scripts from people who do this) I’d suggest removing them when you get 11.50 running, then put them back later. A ton of art control tricks simply don’t do anything anymore, and some are now harmful to performance or cause crashes.
Incremental Roll Out
One last note – because 11.50 is such a disruptive release (in that it requires add-ons to be updated to run in Vulkan), we are working on a staged release. While everyone who wants to get the RC can get it, and everyone who wants the final version will be able to get it, we are trying to make the auto-update notice ping the user base incrementally, so that our tech support is not overwhelmed.
If you don’t see the update prompt for a new beta or final release when you launch X-Plane, it may be us testing out this system. Just run the installer manually with the “get betas” box checked to get the latest no matter what.
X-Plane 11.50 Beta 17 is out now – full release notes here. There are a lot of bug fixes in this beta. We are getting near the end of the 11.50 run and the fixes are becoming smaller and more targeted as we try to lock things down.
We fixed several performance issues on the GPU and CPU, on both platforms. Probably the biggest single fix is Mac cloud performance with Metal and no-anti-aliasing; it turns out that having the rendering surface shared between Metal and OpenGL on a Mac puts it into a layout that slows down the GPU. (This is not an issue on Vulkan.) We’ve fixed this by using separate VRAM for cloud rendering vs plugins; GPU performance with Metal should match OpenGL.
VR users: water reflections are fixed, as well as hopefully the crash on quit with Nvidia + Rift headsets, and we’ve tried to fix the VR mouse not being available until a controller is activated on WMR headsets.
Multi-monitor users: we finally figured out why the horizon line wasn’t lined up – it turns out each monitor was using a different monitor’s height and chaos ensued.
If your bug number isn’t listed in the fixes, you don’t need to tell us it’s still broken, but if we did list a bug as fixed and you still see it, please file a bug and include the XPD number if you know it.
Oh, and Mac Catalina users: you no longer have to reboot your machine after an update. (This was an installer issue – the new installer rolled this week.)
Third party developers: Thomson made this survey form! Basically we’re trying to decide where the best place(s) are to discuss future dev, so we figured we’d get some feedback before proceeding. We’ve had some discussions of future extensions to the SDK on Slack, but not all third party devs have time to monitor a Slack channel, and we don’t want to leave people out.
X-Plane 11.50 beta 16 is now out. Probably the most important bug fix is the set of fixes for broken plugin drawing–this affected SkyMaxx Pro and a number of add-ons in pop-out windows. Full notes here.
At this point we think we are good in terms of plugin compatibility APIs – if your add-on still acts funny in beta 16, please let us know ASAP.
Performance Investigations
Sidney and I spent a bunch of time over the last week looking at performance. Here’s what we have found so far:
X-Plane 11.50 beta 14 made cloud performance significantly worse on the GPU. I accidentally turned off optimized off-screen clouds. This is fixed as of beta 15.
X-Plane 11.50 beta 15 made CPU performance worse on AMD cards on Windows during panel drawing – the same bugs that caused the incorrect plugin drawing hit performance as well. This is fixed as of beta 16. (XPD-10943)
X-Plane 11.50 beta 15 leaks VRAM when scenery or the aircraft gets loaded. That doesn’t necessarily hurt performance but it’s definitely not good. This is fixed in beta 16 (XPD-10941). If you had blurry textures, definitely re-test now.
NVidia cards are losing about 1.5 ms of GPU time as a result of the bug fix for wrong reflections in Vulkan. We found the cause of the performance loss but we have not fixed it yet. This was introduced in beta 12 when we fixed the reflections. (XPD-10953)
Temporary slow frame rate or stutters when turning your head with Vulkan. If your frame rate is smooth and good when flying and then temporarily slows down only when your view changes as lot (e.g circling, turning your head, etc.), that’s this bug. The cause is slower draw time when we draw 3-d scenery out of system memory while we wait for the DMA to VRAM to finish. (XPD-10898)
From what we can tell, a huge percentage of the blog comment complaints about performance are all XPD-10898.
At this point, with beta 16, we do not want any additional performance bug reports. Given that we have two performance issues fully diagnosed but not fixed, there’s no point in collecting more data – the two existing problems would mask other ones. We are also drowning in performance bug reports at this point; we don’t have bandwidth to triage additional ones right now.
Amazingly, I think the answer (and I realize I am cursing the next beta by typing this) is we’re getting really close. You’d be forgiven for thinking that’s lunacy given beta 15; here’s a little bit of info about the state of the betas.
X-Plane 11.50 beta 15 is out (and marked “unstable”) on Steam and amongst other things, has two big fixes for beta 14:
Cloud performance should be back to baseline norms for 11.50 – when I fixed a multi-monitor bug in beta 14, I accidentally turned off a major cloud performance optimization,
Fatal errors while resizing the window should be fixed. These squawks were coming from code that now checks much more heavily for error conditions, and revealed a problem when the OS delivers window resizes to us too slowly.
Note the introduction of new bugs in the process of fixing old ones – this is always a risk when a bug fix is intrusive or complicated; in order to get a shippable product, we have to keep ratcheting down the amount of chaos we introduce per beta, making smaller and more surgical changes.
Beta 15 also tried to fix one last plugin compatibility bug that we discovered very late in the game – under Vulkan, there’s no depth/stencil buffer available to third party plugins, resulting in incorrect drawing for previously working plugins.
I think you know where this is going…this new change introduced new bugs, even as it fixed the problem with the original add-on.
We have been working with third party developers over the last twenty four hours on fixes for the regressions here; our hope is to have a new beta on Monday or Tuesday that fixes these issues and has gotten a good third party once-over.
We’re Going to Have to Close the Door
We are reaching the end of X-Plane 11.50 – at this point the number of remaining bugs to be fixed in this beta is small enough that they aren’t that hard to keep track of. To get to a release candidate though, we’ll need to stop introducing major changes.
I am hoping that we already know about all of the third party incompatibilities, because at this point we have to close the door to complex changes to improve backward compatibility. For beta 16 we are fixing what used to work and is now broken (e.g. yes, SkyMaxx will work again), but if anyone is sitting on an add-on problem they haven’t mentioned, we’re out of time to deal with it.
Plugin Developers: Thread Safety!
Plugin developers: in looking at your compatibility with X-Plane 11.50, please re-read this post and check your usage of threads. We’ve had really helpful responses from third parties who we have notified about threading issues we’ve seen by auto-crash reporting, and I think this will help everyone–third parties, users, and us.
I believe that X-Plane 11.50 is significantly more sensitive to threading violations than 11.40, because the Vulkan driver doesn’t spend CPU cycles protecting itself from abuse. If a plugin calls into us from a thread when it’s not allowed to, this can cascade into crazy X-Plane behavior, which cascades into crazy Vulkan behavior that the driver won’t stop. So careful adherence to the threading rules by plugins is critical to Vulkan stability.
An ongoing question in the comments section has been: “when will this beta be released for Steam?” It’s a good question! In the old days, the answer was “it’ll be a few days” because building a beta for Steam was a separate process from building a beta for the Laminar Research installer.
We solved that problem a few months ago; when we create a beta, we create the beta for both installers at the same time in one coordinated symphony of automatic scripts and command line witchcraft. But there is still some delay in making the Steam beta be available to users – we usually wait a few hours to make sure the build isn’t crashing for a big portion of our user base.
Now you don’t have to wait! If you are a Steam customer you can get the very latest beta even if it’s broken and unusable.
What Is an Unstable Beta
An unstable beta is a beta build of X-Plane that has some kind of relatively serious problem, or that might have a problem we don’t know about because it hasn’t been rolled out to the whole community.
Why would you ever want that? Sometimes the unstable beta has a bug fix you really need. Maybe the crashes affect common hardware but not your hardware. Maybe you just like really new things.
How To Get Unstable Betas for Steam
Our application now has two choices for betas – if you go to the betas tab in the X-Plane app properties in Steam, you now have both the choice of:
“Public Beta” – this is the beta you’ve been using for months – it’s a beta, so it may be buggy, but we don’t release it until we have at least a little bit of evidence that it mostly works.
“Unstable pre-Release Beta” – this is the very latest beta even if it’s broken.
You can choose which one you want, or even switch between them.
For example, right now if you pick public beta you’ll get beta 9, which, for better or worse, has been the current beta for several weeks. If you pick “unstable pre-release beta”, you get beta 10, even though it crashes in HDR with some third party aircraft (a new crash to beta 10) and people tell us that sometimes it hangs on load.
Should I Use an Unstable Beta?
Probably not? There are two cases I can imagine where the unstable beta would be more useful than the normal beta:
The current stable beta doesn’t work for you, so you might as well try something new.
You are waiting on a specific bug fix and the unstable beta has it.
What About Customers Not Using Steam?
We don’t have this multi-beta capability for our home-grown installer, but someday I hope to add it – I think it’s a really useful capability to be able to define multiple tiers of beta quality.
We had a long discussion a few weeks ago about ways to deal with broken betas and a lot of people though that rolling out betas incrementally would be a good idea. Having multiple beta tiers can help this.
Normally I try to not mention specific add-ons when talking about problems; it’s not fair to the add-on maker, it’s really hard to know what the real problem is without knowing everything about the bug, the problems I blog about usually affect a wide array of add-ons, and I don’t want to throw add-on makers under the bus. It’s not fair to them, particularly in this case.
In this case, however, approximately everybody knows that the Zibo was missing its wings in 11.50 beta 9 – it’s a very widely used add-on, and it’s a perfect illustration of the problems with third party content validation, which is what this post is about.
So…gather round children, and I will tell you the tale of how the Zibo lost its wings.
A week or two ago we had a very dead beta, and posed the question of how to incrementally test betas in the future. We got a variety of responses, ranging from “private test it first” to “roll it out in a wave” to “full speed ahead, we know betas are bumpy.”
Since then, we’ve been doing one of the easiest and probably most useful things we can: posting the betas early to third-party developers who are in our developer Slack channel.
Beta 7/8 had a ton of changes, and our third-party developers found multiple problems, some of which we wouldn’t see in our internal tests. So we held off on releasing betas 7 and 8 to the public while we fixed those issues. Until today.
There are a lot of changes in this one since it’s kind of 3-in-one. Please see the full release notes here.
Are We There Yet?
X-Plane 11.50 has been similar to X-Plane 11.20 (our VR) release and different from what we normally try to do, in that when we went beta (both private and public), the work for Vulkan wasn’t done yet. We had something that you could fly with, that was delightful for some users (and unstable for others), but we also had a big list of things we still needed to do.
Well, that was something. I had a very nice post written up last week on the state of beta. We had spent a week very carefully trying to improve stability and then…beta 5 exploded on the launch pad.
So…let’s try this again. But before we get into beta 6, a few graphs:
Crash reports per 12 hours – the big spike is X-Plane 11.50 beta 1
That’s a graph of auto-reported crashes over time – the big spike up is April 2nd when 11.50 came out. The gap in the timeline at the end is when our crash reporter temporarily was shut off for exceeding quota! From this I can take derive two take-away points:
A lot of people are really excited to try the 11.50 beta even though it’s early and unstable and
The 11.50 beta crashes a lot.
The silver lining is that the crashes we have been collecting are very very informative so it’s been a really great data stream.
Here’s one more graph:
Bug reports per day – the big spike is X-Plane 11.50 beta 1.
That’s bug reports and they’re up something like 1000% – we have received close to 1800 reports since then. Of these reported bugs, over 500 are in the category of “it crashed” or some other similarly catastrophic, bad thing happened.
So with those graphs in mind, let’s talk about where we are at with the beta.
This post is just targeted at plugin developers who are modernizing their object drawing – if you don’t write plugin code, the Cincinnati Zoo has been showing their animals on Youtube – it’ll be a lot more entertaining than this post. (An XPLMInstance cannot tunnel down two feet in fifteen seconds – one point for the zoo animals.)
XPLMInstance makes a persistent object that lives inside X-Plane that is visible in the 3-d world. It changes how you draw from “run some drawing code every frame” to “tell X-Plane that there is a thing and update its data every now and then.”
Instancing is actually a lot easier than draw callbacks! But there are two tricky gotchas:
1. You must create the custom DataRefs for your OBJ’s animation before you load the object itself with the SDK. (If the DataRefs do not exist at load time, the animations are disabled as “unresolved to any DataRef”.)
2. When you create the instance, make sure your custom DataRefs are on the list of DataRefs for that instance.
Here’s the really baffling thing: if you create the custom DataRef and then add it to the instance’s list, your DataRef callbacks will not be called.
Wha?
Here’s the trick: the DataRef you register is a global identifier, allowing the object to refer to what it wants to listen to. That’s why you have to create the DataRef – so that the identifier exists.
But when you create an instance, each instance has memory that holds a different copy of those DataRefs.
For example, let’s say you have a truck with four DataRefs, and you make five instances. X-Plane allocates 20 slots (four DataRefs times five instances) to store five copies of each DataRef’s values.
The instances never look at the DataRef itself. They only look at their local copies. That’s why when you push different data to the instance with XPLMSetInstancePosition, each instance animates with its own values – each instance looks at its own local data.
This is also why you won’t see your DataRef callbacks called (unless you use DataRefEditor or some other tool). The object rendering engine isn’t looking at the DataRefs themselves, it’s looking at the local copies.
In other words, XPLMInstance turns DataRefs from the pull model you are used to (X-Plane pulls on your read function to get the value) to a push model (you push set with XPLMSetInstancePosition into the instance’s memory).
This implies two things about your add-on:
It doesn’t really matter what your DataRef read functions do – they can just return zero, and
You can’t use tools like DataRefEditor or DataRefTool to debug your animations. (That didn’t work well in legacy code either, but it really won’t work now.)
If you try the obvious optimization of not creating your custom DataRefs (“hey, no one calls them”) before you create your instance, you will find that animation just stops working. This is because we need the DataRef to be that global identifier to match your instance data with the animations of the object itself.
One last note: if your old code used sim/graphics/animation/draw_object_x/y/z to determine which object was being animated (from inside a plugin “get” function) you do not need to do this anymore. Because each instance has its own local copies and your DataRef function isn’t called, this technique is obsolete.
In summary:
You must register custom DataRefs.
Their callbacks can just return 0 – they’ll never be called.
Always list your custom DataRefs for animation when you create an instance.
Do not use draw_object_x/y/z; use XPLMSetInstancePosition to create per-specific-instance animation.
