As we move forward with the beta, we’ll write up some posts on X-Plane 11.10’s the new features – there are a lot of them, to the point where I’ve lost track of what’s actually in the release. Right now we’re in the “does it work” phase, trying to get a beta that works for everyone without crashing. (Beta 1 worked fine for everyone in the company, but often our internal machines are very similar to each other, so early betas catch things we missed.)
So what follows is a big pile of nerdy stuff…I’m going to add random images in to make the post less boring. So picture is not related.
Where Do The Bugs Go
Planet Ferrari
When you file a bug with our bug report form, it goes to a shared email in-box that someone triages – usually it’s Jennifer, but sometimes it’s me or Philipp. The person triaging the bug will then forward it to our internal bug tracking system (that’s where those XPD-123 numbers come from) or possibly forward it to tech support – we do get a lot of “my sim is broken, help!” in the bug reporter.
This is why I always jump up and down and go “file a bug!”: everything in the bug reporter gets looked at, and everything that is then filed is permanently tracked. At the end of 11.10 we can check the bug list and see what hasn’t been fixed. Like all software, not all bugs will be fixed by the end of 11.10, but if the bug is tracked, this avoids us losing the bug.
Here are some ways to report a bug that are not tracked and are extremely likely to get lost:
Posting in the blog comments section. We do not ‘scrape’ the blog for bug reports. If you write a blog comment and do nothing else, your bug will not be fixed.
“Reporting” the bug on a public forum (x-plane.org, reddit, etc.). We don’t scrape those either, so it’s quite possible no one will see it.
Emailing an engineer at Laminar Research directly.
Night Vision Deserves a Quiet Night
The problem with direct email is that it doesn’t go through the tracking system, you might not get the right engineer, and again, you’re bypassing the mechanisms we have in place to not lose things. If you email someone organized and responsible like Tyler, he might file the bug for you. If you email someone like me, whose in-box looks like a grenade went off, there’s a very good chance I lose your report.
Jennifer tries really hard to list every item that is fixed in each beta, so that you can tell when your issue is fixed and it’s worth re-testing/re-reporting.
This system is far from perfect, and the number one request we get which is reasonable is better linkage between the internal tracking and the user reporting. I sympathize because we have the same problem of “it’s a black box” with the technology vendors we depend on (Apple, Microsoft, Intel, AMD, NVidia).
Changing How We Build
737 With Canards
With X-Plane 11.10 we moved to cmake for our build system. Previously we would maintain separate project files for all three operating systems. That’s about 3x the amount of work it should be, so for 11.10, Sidney and Jörg moved us to cmake, which lets us manage X-Plane as a project once.
The down-side is that there have been a number of serious one-time bugs due to projection configuration problems, since using cmake meant rewriting all of the configuration info for building X-Plane. This is what caused the Linux bug in beta 1 that we required libc++ (fixed during 11.00’s beta process, it popped back up when we replaced makefiles) as well as a number of other random bugs we fixed before public beta started.
In the long term cmake is a win – having gone through the pain of the migration, it’s just quicker for us to administer X-Plane’s project files, which means more coding and less fighting with X-Code, MSVC, etc.
More Robots
The Sky Shader is Out Of This World
In the last few months we’ve automated our build and our packaging process, as well as the amount of testing done automatically by the build system. Hopefully this will turn into bugs being caught earlier, and it makes it easier to cut new betas – getting quick betas out to fix crash bugs wasn’t time consuming. I expect the beta count to be higher for 11.10 than in the past due to the lower cost of recutting the build.
(If I had to cut five betas in a week for 11.05 I’d be pretty cranky at everyone – five betas in a week is totally doable now.)
Graphics Cards That Remember
You Put Your Map In My Cessna
Before X-Plane 11.10, X-Plane would set up shading for a given material by telling the graphics card about every aspect of the material. Over and over. Every time the material was used. For every single frame. For every single view in the frame.
X-Plane 11.10 now stores parts of the materials in VRAM, so they can be referred to as needed. This is part of our restructuring of the rendering engine for VR, Metal and Vulkan.
Monkey See, Monkey Do Silly Things With Substance Painter
This was also the cause of the “invalid UBO” errors in beta 1 – now that we are saving materials and not just reiterating them per frame, we have to get the book keeping right. Sidney set the sim up to crash if the book keeping is done wrong. (This is a good thing – the sim is going to crash anyways if the book keeping is wrong — at least this way the auto-report explains exactly what happened, rather than us having to autopsy the resulting carnage and squint to find the root cause.)
AMD Users: this code is not working on AMD cards right now, which means you aren’t seeing some performance improvements. We’re working this week to see if we can get this going on AMD cards too – stay tuned.
I just cut a quick new beta build of X-Plane 10.11 – Jörg fixed the Linux crash. The crash was a problem with the plugin .so’s and we don’t even know how the bug happened, but it’s fixed now.
Linux users: since the crash is at startup, you won’t be prompted to auto update (because X-Plane crashes before the auto-update check can run). So to fix your beta installation, re-run our installer and pick “update x-plane” with the beta box checked. You’ll get beta 3 and be back in business.
We’ll fix more bugs next week, and the rate of betas may slow down, but we’ll try to patch crashes as quick as we can.
If you have 11.10 beta 1, you’ll be auto-notified to update. (If you’re locked out, e.g. for users on Linux who didn’t have libc++, use the installer and check “Get Betas”). See the release notes for bugs fixed in this beta.
Thanks to everyone who reported bugs! If your bug is not yet fixed, don’t panic – we cut b2 quick to fix the pile of “OGL_ubo_is_valid(s_environment_ubo)” crashes. If you have one of these in b2 please re-report it (and auto-report it!)
To get it, make sure you have an X-Plane install you can scrap, have a stiff drink, and check “get betas” in the installer.
Release notes are here. I’ll try to post more over the weekend about some of the new stuff, but there’s a lot, and we’ve probably left things off the initial notes. For now, since it’s beta 1, our main concern is “does it work at all?”
We’ve been working on bug fixes like crazy, and over the last week or two we’ve run a small internal beta of 11.10 to try to get the biggest, nastiest bugs out. So far it looks like we’re on schedule to put out a public 11.10 this week.
Besides fixing bugs, there are still a few irons in the fire for 11.10 that might go into the beta after public beta 1:
Sidney and I have one more set of performance optimizations that are a “maybe” for 11.10 – we’ll try them and if they blow up the beta, they can wait for 11.20.
Jörg has fixed a lot of weapon-related issues – you can actually author fighter planes now – but weapon cam is still inoperative and probably will be in beta 1.
We are working on VR in parallel to 11.10, and we may be able to release some authoring interfaces for aircraft in 11.10 to get aircraft developers a head start.
A bunch of stuff for developers and artists is already done and ‘in the can’ for beta 1:
The new plugin API 3.0 will be fully available in beta 1.
The G1000 will be available in beta 1, and the G1000-based Cessna is included in the install.
New art for airport authors is in place.
In preparation for this beta, Jennifer ran off to Las Vegas. We might reasonably disagree on how to interpret that.
This is a library path that refers to a file outside its own pack by using ../ to go up a directory and hopefully find that OpenSceneryX is installed, at which point it assumes that it knows the internal layout of OpenSceneryX and exports the object.
Do not do this.
This will become a warning as soon as I can write some code.
This will eventually stop working completely, because it’s a terrible idea that I have said multiple times should never be done.
This has one problem: every plugin on your install is either named mac.xpl, win.xpl, or lin.xpl.
And as it turns out, pretty much every debugging tool ever assumes that each DLL will have its own unique name because it’d be crazy to do otherwise. The decision to make the file name in the fat plugin structure the OS was a really dumb one by me.
With version 3.0 of the SDK, you can now pack like this:
my_plugin/mac_x64/my_plugin.xpl
and with that format, each plugin’s name makes its DLL unique.
This should fix a bunch of things:
You’ll be able to start X-Plane via X-code to debug your plugin without all hell breaking loose.
Back traces on Windows debugging tools should make the plugin executing clear even without symbols.
Any kind of memory map dump (including those in Apple crash reports) will unambiguously show what plugin code is loaded into what memory address.
The existing 2.0 format (fat plugins shown above) and 1.0 format (thin plugins, supported only in the global plugins folder) will continue to work indefinitely – they’re still there.
But if you are going to target X-Plane 11.10 and use the new plugin system features, you may want to use the new packaging and save yourself some debugging chaos.
Tyler posted about some of the new features coming to the X-Plane SDK version 3.0 API (available with 11.10 once we find the last bug and move it somewhere else). Here’s how compatibility works for the windowing system.
First, since the 2.0 SDK, XPLMCreateWindowEx has taken all of its arguments in a structure, a XPLMCreateWindow_t. That structure, in turn has a structSize variable that is meant to be initialized like this:
Now when we revise the SDK, the size of the XPLMCreateWindow_t structure grows when you #define XPLM300. This results in new plugins compiled with the new SDK sending in a large struct (with more callbacks) and older plugins compiled against the old SDK sending in a small struct.
This is how the SDK “knows” what SDK you are using and provides new behavior to new plugins but provides compatible behavior to existing plugins, compiled years ago.
(This technique isn’t in any way new or unique to the X-Plane SDK – you can tell by the coding conventions that it is cough, cough, borrowed from the Win32 SDK.)
These both come from third party developers – one I was aware of and one was completely new to me.
The Debug Heap Is Slow
Veteran Windows C++ developers know this, but if you don’t spend a lot of time with MSVC, this might be new.
The default configuration of MSVC is to force use of the debug heap when the debugger is attached; this applies even to release builds of binaries that aren’t yours.
What this means is: if you do nothing else but start your plugin “inside” X-Plane, X-Plane will use Microsoft’s debug heap.
We avoid doing this internally because it is really, really, really slow. The debug heap provides some great tracking for finding memory problems, but the cost is really, really slow load time.
The fix is simple: add _NO_DEBUG_HEAP=1 to the environment variables for your debugger. You can do this in the control panel as shown in the linked article, or you can add it to MSVC’s debugger properties.
Other Things To Go Fast
As a side tangent, when debugging a plugin, find ways to load X-Plane faster. A few things that can help:
Turn off AI aircraft.
Pick a simple aircraft to fly (e.g. not the 747).
Go somewhere without scenery (or without much scenery, e.g. PMDY).
If you have to have scenery, turn 3-d all the way off.
My Plugin Won’t Reload
We receive periodic bug reports that plugins won’t reload on Windows. This is miserable for you as a third party developer – it means rebooting the sim instead of just reloading your plugin. But why does it happen?
I thought we had a bug, but a third party developer found this:
I did searched the registry for the names I’ve used for X-Plane 11 folder and found this items at
HKEY_CURRENT_USER\SOFTWARE\Microsoft\Windws NT\CurrentVersion\AppCompatFlags\Layers
Each name used was included there with “$IgnoreFreeLibrary<win.xpl>”
value, after removing them and restarting X-Plane I got able to compile the plugin using the names again.
I have not found any official Microsoft documentation on this, just a lot of ranting from really annoyed programmers who want to know why the hell Microsoft would do this. If anyone has better Google-Fu than me and can find official docs, please do share. So what follows is my wild speculation of what’s going on.
In some old version of Windows, FreeLibrary probably didn’t actually (or immediately) free a DLL under certain circumstances. Application developers then wrote buggy apps that relied on this behavior, e.g. they freed a library while it was in use and their app didn’t crash.
Microsoft updated Windows and made FreeLibrary do what it says on the label, always, and old buggy apps started crashing left and right.
Microsoft, being Microsoft, did not say “you app developers are idiots, why don’t you go rewrite your buggy apps”. Instead, they added a feature to the App Compatibility Wizard that watches your app and, if it crashes after a DLL was unloaded and the moon is the right shade of Rondele, it adds a registry key marking this library as “never to be unloaded, even if this app says so”.
From that point on, the app doesn’t crash because the old semantics of FreeLibrary (“ha ha ha, you don’t mean that do you”) are restored in that one case.
The above is entirely made up – it’s my mental model for what might have happened, unless Raymond Chen someday gives us some back story.
Still, you can see what has gone wrong here – Windows decided that for compatibility reasons, plugins should not actually be unloaded, and since all of your DLLs are named win.xpl, this one plugin crash has caused plugins to never be unloaded. Thus the plugin unload/reload never releases the DLL and you can’t swap in new code.
Nuke the registry key and you’re back in business, at least until you crash again.
