X-Plane 11.10 brings a few changes to how airports, the airport gateway, and navdata interact.
Many pilots who try to fly realistic IFR operations with the X-Plane built-in GPS or FMS will have encountered this dreaded window already:
runway 12L not found at Vero Beach Municipal airport
The reason for this is that coded instrument flight procedures (CIFP) come from very reliable sources – Jeppesen or LIDO (depending on whether you get your data updates from Navigraph or Aerosoft), while the runways on X-Plane’s airports come from a community driven, open database: The X-Plane airport gateway.
Unfortunately, the airport gateway community is not always fast when it comes to runway renames or airport expansions, which happen all the time all over the world. The most common reason for a runway rename is a shift in magnetic variation: Runways are named for their cardinal direction relative to magnetic north. While the runway’s orientation with regard to true north is fixed[citation needed], the orientation measured against magnetic north changes over time, as the magnetic pole moves and local magnetic declination changes. Now when the magnetic course of runway 11L changes from 114 to 115 degrees, airports paint new numbers on their runways. 11L-29R becomes 12L-30R. Jeppesen knows about this and changes the runway name in all their data, which ends up in a data update for X-Plane. Meanwhile, the scenery author community over at the airport gateway of course has more exiting things to develop then a runway rename.
To make things worse, runway renames are super annoying in WED. After you renamed the runway from 11L to 12L, you had to go through ALL your flows, ALL your taxiroutes, and ALL your airport signs to change the name EVERYWHERE.
In the past, we have partially solved this problem by running mass renames of runways in the gateway database rather than through WED. If you see a change on an airport made by a user named “WEDbot” (like at this airport) that is usually such a batch-rename.
With X-Plane 11.10 and WED 1.7 there are some big changes that greatly improve the interaction between X-Plane airport data, navdata, WED, and the airport gateway.
Easy runway rename in WED
WED 1.7 has a function that changes all flows, routes and signs for you when you rename a runway end. This makes bringing an airport up-to-date a nearly foolproof operation even for a WED-dummy like me. You don’t need to be a scenery wizard to simply fix an airport anymore.
Silent runway rename in X-Plane
If you have navdata from Aerosoft or Navigraph, and a runway in the X-Plane airport matches a runway coming from the navdata, but the name has changed, X-Plane 11.10 now silently renames the runway at runtime for you. Which means, even if a 11L is painted on the runway, the FMC can load the procedure for 12L and get you there. This only works if the scenery is properly georeferenced and the runway is actually in the right spot – if the scenery was made incorrectly and the runway is not at the right coordinates, this obviously doesn’t work.
Silent threshold fix in X-Plane
Not all scenery authors correctly place displaced thresholds. A bit of confusion exists over when to use the white arrows or the yellow chevrons – and which counts into the runway length and which doesn’t. I teach my student pilots “the only thing you can do on yellow chevrons is crash – anything but crashing on that area is illegal.” Hence this area doesn’t count for runway length. Again, if you work off a properly georeferenced orthophoto, you won’t have any problems. Unfortunately, if you misplace where the (displaced) threshold is, this coordinate problem can feed back into the instrument procedures of this runway. For example, for many non-precision approaches the MAPt of the procedure coincides with the runway threshold, so if those coordinates are off, so will be your missed approach point. With X-Plane 11.10, if a runway in the airport scenery matches a runway coming from your updated navdata, but the threshold is laterally offset from where it should be according to instrument procedure data, X-Plane silently moves the threshold coordinates the GPS/FMS works off to the correct location. This works if the scenery is “good enough” in that the majority of the runway pavement is where it should be, and the thresholds are only off in the direction of the runway. If the whole scenery is ill-referenced, meaning the runway is off other than along its major axis, this obviously doesn’t work.
Silent and not-so-silent feedback
If you have enabled anonymous data collection in X-Plane, whenever your X-Plane silently applies a runway name or runway threshold location fix in the background, it also sends a packet of data to our analytics server, telling us the airport you were approaching and what was up with the runways. Collecting this data from a wide range of X-Plane 11 users will allow us to generate a heatmap, i.e. the most important airports that need the gateway communities’ love. Note that this data is collected only if you are running navdata that is current – we are not collecting reports based on historical data.
Only if both of the above fail, which means the airport has both a problem with its runway numbering and is ALSO poorly georeferenced (runways are in the wrong location geographically) the situation is beyond fix for the new runway logic. Only in this case you will see the dreaded dialog, because the runway simply does not exist in X-Plane, at least not where it should be. In this case, you will be able to submit an automatic report to the gateway website if the problem exists with current navdata. Note that this dialog will come up whether you have enabled data collection or not – but you can still chose to close it without actually posting the report if you don’t want to.
Only this kind of “all is lost” reports are actually visible on the gateway website and the XSG bug database. This allows artists to see the only airports that are actually so outdated that they cannot be fixed automatically. The automatically fixable scenery errors no longer clutter up the gateway airport bugbase.
Any downsides?
The downside to all these changes is that they all actively work to keep the X-Plane default scenery up to speed with the airport changes in the real world. This means that over time, as our global airports follow the real world in terms of runway renames, airport construction, expansions, etc… it will become less useable without up-to-date navdata. That’s the price we have to pay for “as real as it gets”.
Break ALL the scenery!
Poorly georeferenced scenery has a problem beyond affecting the missed approach points of non-precision approaches. It also affects the ability to use the new SBAS (satellite based augmentation system) approaches that are comparable in accuracy to ILS. I always prefer to fly the LPV approach if given the choice. However, the FAS block (final approach segment) comes from the navdata, which means it guides you precisely to where the runway is in the real world. If the X-Plane scenery is poorly referenced, the approach will dutifully fly you into the grass in X-Plane, if this is where the runway would have been in the real world. This is obviously a problem for serious training scenarios. Therefore, X-Plane 11.10 can be started with the commandline option –accurate_runways which will dynamically rewrite the actual scenery in X-Plane after loading an approach, both moving the runway into the correct geo-location and also changing the numbers written on the runway if needed. This obviously only works on default scenery with the procedurally generated runway textures. It will not change custom scenery that uses draped polygons for photorealistic runway textures. Moving the runway into the correct location will obviously also disconnect it from any incorrectly placed taxiways. Also, using this option increases load times for selecting an instrument procedure significantly, since it has to rebuild the airport scenery. So this option is really only there to help you keep limping along with broken scenery, if your operation absolutely requires accurate runways and you can live with some broken taxiways. It is therefore not available as an “official” setting. Do not come to us to complain about the jarring results – make a proper fix in WED instead! The results can be quite disruptive, but at least the approach won’t guide you into the grass:
This LPV approach required the runway to be moved quite dramatically. See the taxiway that was parallel to the runway in the scenery.A closer look at the situation above. This is an extreme example. This airport scenery had the orientation of the runway badly wrong. Note where the threshold was originally placed by the author where the taxiway now ends in the grass.
This has one problem: every plugin on your install is either named mac.xpl, win.xpl, or lin.xpl.
And as it turns out, pretty much every debugging tool ever assumes that each DLL will have its own unique name because it’d be crazy to do otherwise. The decision to make the file name in the fat plugin structure the OS was a really dumb one by me.
With version 3.0 of the SDK, you can now pack like this:
my_plugin/mac_x64/my_plugin.xpl
and with that format, each plugin’s name makes its DLL unique.
This should fix a bunch of things:
You’ll be able to start X-Plane via X-code to debug your plugin without all hell breaking loose.
Back traces on Windows debugging tools should make the plugin executing clear even without symbols.
Any kind of memory map dump (including those in Apple crash reports) will unambiguously show what plugin code is loaded into what memory address.
The existing 2.0 format (fat plugins shown above) and 1.0 format (thin plugins, supported only in the global plugins folder) will continue to work indefinitely – they’re still there.
But if you are going to target X-Plane 11.10 and use the new plugin system features, you may want to use the new packaging and save yourself some debugging chaos.
XPlane2Blender’s new version number system will allow us to debug .blend and .obj file even faster. It should also make making updates to the data model easier to implement.
Every exported .obj file’s footer contains information about the addon version, when it was compiled, and why. For example: 3.4.0-beta.5+1.20170922151901
A break down of the components
3.4.0: The traditional Blend addon number
beta: The type of build cycle we’re in. Other choices you may see are dev (a developer build), alpha, rc (for full release)
5: the build type version we’re on. Since this is beta 5, the build type version is 5.
The elements after the + are generally less meaningful to the average user
1: The version of the data model (the properties and settings for XPlane2Blender, used for comparing changes
20170922151901: The “build number” date this source code was packaged and released in the form of YYYYMMDDHHMMSS in UTC.
Communication
The build version number is partially shown (elements after the + are hidden) in the scene settings under Composite Normal-Textures, potentially along with warnings about the stability of the build you are using.
Examples
When starting this version of the beta, you will see this:
A future stable version of 3.4.0 for the general public will show this:
Notice the green check mark and lack of any warnings.
In a future pre-alpha cycle for 3.4.1, two types of people will see the following:
A developer writing and testing unreleased code
Someone who didn’t head the warning against using such code, or get scared off by the nuclear symbol and extra warning about a lack of a build number
It is the worst case scenario for stability and traceability, hence the nuclear symbol.
Why the extra warning about a lack of a build number?
A lack of a build number indicates that you do not have a good dialog (e-mail, chat, this release page, or other channels) with a knowledgeable and attentive developer. This means your work is more likely to be run through a bad version of the code and damaged, or your bugs will be harder to diagnose and repair.
In this case, despite the code appearing to come from a stable era of the code near a release, there is potential for something to be wrong and have very poor ways of tracing what it could be – hence the lack of green check mark and big red warning symbol.
If you see some new pre-alpha feature you’d like to try, just ask us about it first. Going forward, we want to start with a dialog about potential dangers, testing, and intentions of an incomplete feature rather than an autopsy later on. Don’t be scared, we always love hearing from users before there is a problem, not after!
Build Version History
Also, all .blend files will be keeping a log of every different version of XPlane2Blender that they are opened and saved with. This is automatic and needs no involvement from the user. Those who are curious can look in the Plugin Development Tools section at the bottom of the scene settings and see the history of their file.
Note: This does not record any history data about Blender versions, other addon versions used, timestamps opened or saved, or changes made to it (including XPlane2Blender settings changes). It is purely useful as a debugging tool, and is not fool proof.
This .blend file started as a legacy 3.4.0 pre-beta.5 file, and was then with a copy of 3.4.0-beta.5, with no build number. Then it was used with 3.3.12, then finally, used with a build of 3.4.0-beta.5 created on 09/18/2017 at 01:27:30am.
One could use this information to guess what transformations the data could possibly have gone through along each step of its journey.
Tyler posted about some of the new features coming to the X-Plane SDK version 3.0 API (available with 11.10 once we find the last bug and move it somewhere else). Here’s how compatibility works for the windowing system.
First, since the 2.0 SDK, XPLMCreateWindowEx has taken all of its arguments in a structure, a XPLMCreateWindow_t. That structure, in turn has a structSize variable that is meant to be initialized like this:
Now when we revise the SDK, the size of the XPLMCreateWindow_t structure grows when you #define XPLM300. This results in new plugins compiled with the new SDK sending in a large struct (with more callbacks) and older plugins compiled against the old SDK sending in a small struct.
This is how the SDK “knows” what SDK you are using and provides new behavior to new plugins but provides compatible behavior to existing plugins, compiled years ago.
(This technique isn’t in any way new or unique to the X-Plane SDK – you can tell by the coding conventions that it is cough, cough, borrowed from the Win32 SDK.)
These both come from third party developers – one I was aware of and one was completely new to me.
The Debug Heap Is Slow
Veteran Windows C++ developers know this, but if you don’t spend a lot of time with MSVC, this might be new.
The default configuration of MSVC is to force use of the debug heap when the debugger is attached; this applies even to release builds of binaries that aren’t yours.
What this means is: if you do nothing else but start your plugin “inside” X-Plane, X-Plane will use Microsoft’s debug heap.
We avoid doing this internally because it is really, really, really slow. The debug heap provides some great tracking for finding memory problems, but the cost is really, really slow load time.
The fix is simple: add _NO_DEBUG_HEAP=1 to the environment variables for your debugger. You can do this in the control panel as shown in the linked article, or you can add it to MSVC’s debugger properties.
Other Things To Go Fast
As a side tangent, when debugging a plugin, find ways to load X-Plane faster. A few things that can help:
Turn off AI aircraft.
Pick a simple aircraft to fly (e.g. not the 747).
Go somewhere without scenery (or without much scenery, e.g. PMDY).
If you have to have scenery, turn 3-d all the way off.
My Plugin Won’t Reload
We receive periodic bug reports that plugins won’t reload on Windows. This is miserable for you as a third party developer – it means rebooting the sim instead of just reloading your plugin. But why does it happen?
I thought we had a bug, but a third party developer found this:
I did searched the registry for the names I’ve used for X-Plane 11 folder and found this items at
HKEY_CURRENT_USER\SOFTWARE\Microsoft\Windws NT\CurrentVersion\AppCompatFlags\Layers
Each name used was included there with “$IgnoreFreeLibrary<win.xpl>”
value, after removing them and restarting X-Plane I got able to compile the plugin using the names again.
I have not found any official Microsoft documentation on this, just a lot of ranting from really annoyed programmers who want to know why the hell Microsoft would do this. If anyone has better Google-Fu than me and can find official docs, please do share. So what follows is my wild speculation of what’s going on.
In some old version of Windows, FreeLibrary probably didn’t actually (or immediately) free a DLL under certain circumstances. Application developers then wrote buggy apps that relied on this behavior, e.g. they freed a library while it was in use and their app didn’t crash.
Microsoft updated Windows and made FreeLibrary do what it says on the label, always, and old buggy apps started crashing left and right.
Microsoft, being Microsoft, did not say “you app developers are idiots, why don’t you go rewrite your buggy apps”. Instead, they added a feature to the App Compatibility Wizard that watches your app and, if it crashes after a DLL was unloaded and the moon is the right shade of Rondele, it adds a registry key marking this library as “never to be unloaded, even if this app says so”.
From that point on, the app doesn’t crash because the old semantics of FreeLibrary (“ha ha ha, you don’t mean that do you”) are restored in that one case.
The above is entirely made up – it’s my mental model for what might have happened, unless Raymond Chen someday gives us some back story.
Still, you can see what has gone wrong here – Windows decided that for compatibility reasons, plugins should not actually be unloaded, and since all of your DLLs are named win.xpl, this one plugin crash has caused plugins to never be unloaded. Thus the plugin unload/reload never releases the DLL and you can’t swap in new code.
Nuke the registry key and you’re back in business, at least until you crash again.
11.05r2 is now available – if you have 11.05r1, you’ll be notified to auto-update. We fixed the broken SIDs/STARs at KMCO, removed a few custom Aerosoft lego brick airports to reveal the gateway one, and we have some updates to KLAS from Julian.
This should come out on Steam in a day or two if no one finds something on fire and hopefully be final this week.
We’re working full time to get 11.10 ready for beta, hopefully very shortly after 11.05r2 is final.
Update: Steam users can get 11.05r2 as well by opting into betas.
We’re continuing to integrate X-Plane 11.10 (with mixed success!); we’ll get more info on 11.10 posted once we’re in beta. 11.05r2 will come out as soon as I get my hard drive fully restored – from the looks of the comments this should be Real Soon Now™.
This post is for plugin developers – if you don’t write plugins, I suggest cat pictures instead.
For as long as there has been OBJ animation, X-Plane has called the XLMGetDataXXX APIs from inside the OBJ engine as drawing is taking place. These calls eventually show up in your plugin in your dataref read callback. You might even see X-Plane code on the stack if you put a break-point in your read accessors.
This design has two major draw-backs:
It’s bad for multi-core. Since the plugin API is single-threaded, operations requiring us to know the animation state of an object (drawing, collision testing, etc.) require us to be on the main thread. So this doesn’t scale well.
It’s inefficient. We might have to draw your aircraft several times – one or more times for reflections, one or more time for cascading shadow generation, and once to draw (or more for multi-monitor). The work to fetch those datarefs is done multiple times, even though we don’t want different answers between drawing passes.
(In fact, it’s worse than that – if you have the same dataref used in multiple animations, we just request the dataref over and over…not real clever.)
X-Plane 11.10 changes this for the first time: we are starting to collect the entire set of datarefs we need to draw a model in a single burst. We can then save these dataref results and reuse them.
This is just the beginning of a series of changes to make the rendering engine multicore-friendly. Hopefully it won’t affect any add-ons* – this should have no effect unless you are doing highly illegal things with the SDK.
I am hoping we’ll see some performance benefit by not spending as much time in plugin code, but I don’t have numbers yet.
* Plugins that read the object-draw-location datarefs to do multiplex their datarefs on multiple models should still work; we set those object-location datarefs before we collect all of the datarefs for your model.
TL;DR version: my iMac’s fusion drive “lost its marbles” right before I went on vacation. This has delayed cutting an 11.05 release candidate 2 with a few scenery fixes, but we should get to it next week. In parallel, we’re working furiously to get all of the code locked down for 11.10.
Everything else that follows is really, really, really, really boring. I’m writing it only because some of my co-workers watched this slow motion car crash and tightened up their backup game a bit. If my drive fail can shake you out of complacency, read on.
Basically: my iMac is my main development machine, and the data is backed up and/or duplicated in a bunch of different places: a USB time machine archive, a Backblaze cloud backup (both are “full machine”), DropBox for virtually all of my documents, and my work for Laminar is kept on Laminar’s source control servers. Data loss was never a huge risk here.
Time loss, however, is a real risk! My goal was to lose as little work time to fixing my machines as possible. So my plan was: restore from time machine disk backup, request a cloud backup restore via hard drive, return the hard drive. The total cost would be a few hours of disk copying and less than an hour of my time. My development machine would be usable for new work while waiting for the cloud backup to arrive.
This has not gone as well as I had hoped! You can learn from my fail here — a few notes.
Your backup might as well not be a backup if you have not checked that the backup contains the data you think it contains. It turns out that both the cloud backup and time machine backup were missing files! I’m very lucky that they weren’t missing the same files.
Time machine sometimes decides not to back stuff up. OS X has a hidden per-file/directory attribute that can exclude a file from backup without showing it in the Time Machine UI! Once you check your time machine backup and find a folder is missing, from terminal you can do tmutil isexcluded <file path> to see if the file has been explicitly excluded. If it is, tmutil removeexclusion <file path> fixes this.
Backblaze ships with a bunch of file exclusions too – mostly designed to not archive stuff that isn’t your data. But beware – stuff you care about might not be on the list. (For example, virtual disks in a virtual machine are excluded by default.) I had to add back .iso files to the backup list. Backblaze backups are also not bootable. This is something I can live with, but always read the fine print about what’s in the backup.
The Backblaze data restore has been very slow – over ten days for less than half a terabyte and it’s still “in progress”.* While they haven’t exceeded the maximum restore time they advertise, it’s slow enough that the delay matters.
One other note on Backblaze: I saw major performance problems on my iMac while Backblaze was running, even when a backup was not running (since they were scheduled for overnight). I do not think this is necessarily Backblaze’s fault – it may be a problem with CoreStorage (which “runs” the fusion drive) or even a fault with my drives. From what I can tell, cloud backup exacerbated it by putting a lot more file traffic on my system.
A possible danger if (like me) you keep documents on DropBox to have them everywhere: when I restored my iMac from Time Machine, I was exposing DropBox to my data from a week ago. I didn’t wait to see if DropBox would figure out what happened; I unlinked my iMac while it was offline after the restore, then re-established DropBox and let it download my data. Better safe than sorry.
I have been backing up to portable 2.5″ USB drives because they’re cheap and really convenient, but they have a down-side: the mechanisms can easily fail and take your whole backup down. I have five of these drives and one has failed in a three year period.
I’m really unhappy with CoreStorage, to the point where I would not recommend a fusion drive anymore. CoreStorage is an Apple virtual-volume technology (similar to soft-RAID) that makes one small SSD and one large HDD look like a single unified volume, with some of the data “cached” on the SSD for performance. CoreStorage is a lot newer than HFS, so when things go wrong, most disk utilities you would go to just don’t work.
I actually ended up in a state where (after wasting almost an entire day) I could see my data, but only in single-user mode with a read-only file system. I might have been able to directly copy the data, but I picked to format the drive and restore from the backup to save more of my time and get back to coding X-Plane. My suggestion for developers getting iMacs: get an internal SSD (whatever storage size you can afford) and supplement with a fast external hard drive over Thunderbolt.
Going forward, I am replacing the portable backup drives with a Synology NAS RAID device – this gets me high performance, high capacity backup (about 10 TB) with redundant drives. I picked HGST drives because they’ve had a good track record for reliability. With a large network attached storage server, I can have all of my machines backing up in the house all of the time, and have that be the primary way of getting my data back. I’m keeping cloud backup as a last-resort-the-house-burned-down kind of thing.
If my cloud backup hasn’t shipped Monday, I will rebuild the setup I use to cut builds by hand (it’ll take a few hours but it’s doable) and we’ll cut 11.05r2 that way. If the drive comes, I can get the last of my data back and we’ll get to 11.05r2 the easy way. Either way, we’ll get things moving again.
* I opted for a hard drive restore, which should have one day of shipping time, instead of a download; a smaller restore based on download made clear that the transfer speeds would be slower than FedEx for that quantity of data.
If you have been doing work with manipulators during beta.1, please download this new one and review the notes! Download the latest here: io_xplane2blender_3_4_0_b3.zip
Fixes
#ATTR_layer_group_draped causing KeyErrors
Reverts bad change to manipulators properties*
Adds
“.beta.3” will be printed on all OBJs. This is not permanent, expect something better in the future
What happened to the manipulators?
A poor decision to change the Manipulator Type (Drag XY, Push, Command Knob, Toggle, Delta, etc) means that if you changed the type of manipulator between beta.1 (or 7fe534ad1f906b7853827 if you constantly check out the latest cutting edge commits [which I do not recommend for this exact reason]), this beta will make those changes irrelevant.
Example, suppose on 8/1/2017 you create a switch and set the manipulator type to Toggle, then with beta.1 you change the type to Push. Beta.3 will show the type as Toggle. All you need to do is go back to the switch and change the type back to Push. The rest of the values you set will still be there. If you created manipulators during the beta, they will be set back to the default “Drag XY” type and will need to be fixed.
This is an unfortunate lesson to be learned, and if you are facing a lot of tediousness (30 manipulators to hunt down and change) or potential errors (you can’t remember how many or which ones you changed) please contact me!
I will personally help you recover the changes you made during beta.1 – present.
