One of the most well known errors you will encounter in X-Plane is the infamous device loss error. But it’s also one of the least well understood errors and one of the hardest to debug, so in this blog post I want to talk about what device losses are, what we have done to fix them and what you can do to help us investigate these issues. Let’s actually start with the latter because if there is one takeaway from this blog post for anyone, it’s the following:
In case of device loss
If you encounter a reliable device loss error, please run X-Plane with --aftermath in the command line options, or run X-Plane via the X-Plane_aftermath.bat script from the Support folder inside of your root X-Plane folder. This flag currently works with both Nvidia and AMD hardware, but it will come with a bit of a performance hit. It’s best used in cases where you already know you are going to run into a device loss and want to provide additional diagnostics. I would not recommend just running with this flag, in particular because it does not work around device losses in any way – It just gathers additional data to make device loss debugging easier. When running with Aftermath, next time you encounter a device loss, X-Plane will no longer call it that but instead say “Encountered a GPU crash!”. Please submit those crash reports! If you see this message it means that X-Plane was able to gather diagnostic data that can help identify what happened.
What is a device loss anyways?
Everyone knows that applications can crash. The human species is incapable of writing bug free code and some of these bugs lead to crashes, while others just result in broken logic ranging from mildly entertaining to downright annoyingly broken. But crashes are definitely the worst kind of bug: If you have invested 7 hours into a flight and then suddenly X-Plane crashes, it’s probably one of the worst experiences in flight simming. I know it has put me off from virtual flying for weeks in the past. Luckily, when applications crash, it’s possible to capture the current state of execution and everything that lead up to it. As programmers, we get to see the callstack which tells us not only in which function we are currently in and where, but also what function has called it and in turn which has called that function and so on. We can also see what thread executed it, what the other threads were up to and even the state of the CPUs internal registers that hold part of the current program state. While this isn’t always enough to create a fix, it points in a very specific direction and lets us investigate the code and, together with the log, piece together what happened. Essentially, when you submit a crash report, you send us the black box of your flight. It contains forensic information that can be analyzed to figure out what happened, but we don’t look as cool as NTSB investigators.
Device losses in reality are just crashes as well, but crashes on the GPU. GPUs these days, and really since a long time, are almost fully programmable and just execute arbitrary code. Programs that execute on a GPU are called shaders, a leftover from the olden days when they could exclusively be used to shade vertices. But these days, shaders are just tiny programs that execute on potentially tens of thousands of GPU cores. Shaders are responsible for basically everything that X-Plane puts up on the screen, from mundane tasks like transforming vertex data into screen space, calculating the colour of every single pixel all the way to culling tens of thousands of trees per frame. X-Plane makes use of a lot of shaders because GPUs are incredibly flexible and a lot of workloads lend themselves really well to their highly parallel nature. Counting shaders is surprisingly hard, X-Plane bundles shaders into what we call “modules” which contain variants of similar shaders. But doing a count of just modules, X-Plane 11 shipped with 29, while X-Plane 12.1.3 had 88 modules and 12.2b4 ships with 91. Each shader module contains between a handful to thousands of distinct shader variants, so the real runtime count of compiled shader programs is much higher, somewhere in the mid to high 5 digit range. Quantity is of course not a measure of quality, but X-Plane 12 as well as any modern game wouldn’t be possible without the ability to flexibly run millions of shader invocations every single frame.
Just fix the crashes then
While shaders are no doubt one of the greatest additions to computer graphics, they come at a cost: They are incredibly hard, near impossible, to debug properly. On the CPU side you can attach debuggers to your code and then step through the code as it executes while seeing the result of each operation. On the GPU that is not a possibility, you can’t inspect the state of a running shader, only the results of the execution after the fact. And if nothing shows up on screen, then good luck trying to figure out where your triangles went. And if it crashes, you are even more toast because now you only got broken pieces to look at left. There are tools to make this a bit easier and it definitely isn’t as bad as it used to be, which is a big reason why it’s possible to put a lot of shaders into the field now: The tooling around this is has gotten a lot better. But for crashes, the GPU once again turns into a terrible blackbox.
The first hurdle is that the CPU and GPU run asynchronous from each other. The CPU encodes operations for the GPU into a command buffer that is then submitted to the GPU for execution, so right there you already have a source of latency. In practice, the CPU is usually at least a frame ahead of the GPU in terms of what it is computing. Detecting a crash also involves latency, often the OS/driver has to recover the GPU from its current state and put everything back together. Since the GPU is responsible for getting anything onto the screen, the OS and driver are much more interested in getting you back into a state where you can interact with your computer. But eventually, X-Plane will catch up to the fact that the GPU crashed, because at some point a call into Vulkan will produce the infamous error code VK_ERROR_DEVICE_LOST. This is why it will often feel like your computer is glitching out prior to a device loss, everything stops working for a second, the OS and driver recover the GPU, your windows and displays might flicker around for a second and only then will X-Plane go “yeah by the way, stuffs broken, yo”.
This huge latency between cause and effect is one of the big issues with device loss debugging. By the time the CPU side of X-Plane realizes the GPU is dead, it’s too late to gather data anymore: Something we did god knows how long ago crashed and that’s all the information we have. This is also why the log file tends to just not be very useful for device loss triage. This is especially bad when the device loss is hard to reproduce and happens only once every blue moon. My favourite kind of device loss is the one where you can get X-Plane crash really fast and reliable, in those cases it’s possible to start toggling things on and off and compare what happens to zero in on what is the actual cause. I have spent many a day having X-Plane take down my window manager over and over again.
What’s being done about device losses then?
Over the years we’ve fixed a bunch of device losses. 12.06 was probably the biggest release here, cutting device losses by about 75%. 12.1.0 also reduced device losses by a large fraction by working around a bug in the Nvidia driver. But of course new code gets written and new code can always be buggy. Way back in 11.50 we also added support for Aftermath, which is a library from Nvidia that helps gather crash dump data off of the GPU. In 12.2, I have massively reworked Aftermath support and also added support for AMD GPUs. While AMD does not actually use Nvidia’s Aftermath library, for simplicity sake the same command line option is re-used and my hope is to also add support for this for Intel GPUs in the future.
With 12.2 and Aftermath enabled, X-Plane now injects per draw/dispatch checkpoints into the command stream. In the event of a device loss, these can then be analyzed to recover the GPUs program state. Because it is very fine grained, this data is incredibly valuable and has actually helped fix two device losses already. The downside is that, because it is so fine grained, it also comes with some overhead. After all for every draw or dispatch command, X-Plane has to stash away some data. But the goal was to keep this as lightweight as possible, the current implementation stores just a couple of bytes of data and defers resolving all of it until after a device loss. Under the hood, this is implemented with Aftermath on Nvidia and with buffer markers on AMD, plus logic inside of X-Plane for the post mortem resolving of data.
And now it’s your turn. Got an annoying device loss? Run with Aftermath, submit your crash reports and hopefully in an update coming to your install soon, that device loss is gone. And maybe by this time next year, I can sip Martinis on a tropical island because my work is finally done.
Misconceptions
One thing I want to make clear though, device losses and running out of VRAM are two entirely different issues. There is a persistent rumour that low VRAM can cause device losses, but this is not the case. The other thing I also frequently see is the advice to uninstall scenery or plugins. Scenery and/or plugins don’t get access to X-Plane’s Vulkan command stream so they can only very indirectly cause device losses. For example, it’s possible that art controls modified by plugins can cause device losses by enabling shader paths that are not normally taken and thus aren’t fully tested. But in general, it feels like there are a lot of snake oil fixes out there, so please be wary. Neither the log nor the alert box have enough information to triage a device loss to even remotely claim that X is the cause of it. That being said, it never hurts to do some simple A/B testing with things disabled, although this might just mask the problem instead of actually resolving it. A/B testing is particularly useful in the case of a repeatable device loss because you can get a much stronger signal from that test, so you can pass that information along with your bug report and make reproducing the issue much easier for us. For random device losses however it’s near impossible to get a clear signal from such A/B tests.
The X-Plane 12.2.0 beta is finally here! This update has been a labour of love from the team for many months, and we’re excited to see so many users enjoying the changes. The stars of this show go to Daniel, Daniela and Maya for their meticulous commitment to enhancing the atmospherics of the sim. Alas, this is a developer website. So let’s glance at some of the changes relative to our third parties.
1. Changes that concern third parties
Rolling back the time machine, Ben once discussed plugins using private art controls here and here. And that time has come once again because we have changed ALOT.
Developers who make use of private art controls in their work (that includes Aircraft, Plugins and Scenery alike) will need to review/remove them from their product.
Manipulating cube map positions (which if you used to attempt to solve dark cockpits… may now inversely create dark cockpits). Please return them to their default position. Cockpit lighting now honours the aircraft geometry, eliminating errors such as “glowing orange cockpit at sunset”
Atmospheric/Cloud datarefs will behave significantly differently. Cloud scattering and shading is improved, amongst changes in cloud layer presets.
Dark cockpits should now be resolved. Aircraft texture artists will need to review albedos, including attempts to brighten cockpits to compensate for previous versions of X-Plane.
Terrain roughness (and subsequent brightness) has been improved. However older generated content may appear differently. These will need to be updated with proper roughness and normal direction
We have a new AGX tonemapper. This has significantly improved the range of colors we can render, and subsequent “details,” bringing a gorgeous new saturation to the sim.
However, this will mean that colors* will shift in appearance. So please review your work. This concerns all parties (Artists, livery painters, scenery designers, system developers etc)
*As a Brit, it annoys me deeply having to constantly change “colours” to “colors.” So I hope you all appreciate this noble sacrifice 🤣
^ This is X-Plane’s lighting in 12.1.4. Notice how almost uniform the shading appears!
^ And this is 12.2.0. Not only are the color shades “deeper”, but colors also desaturate under direct illumination (like real life.)
2. Things we’re working on
Beer Shadow Maps – They are currently too sharp, and we are attempting to soften them up.
Updated cloud darkness – There are still a few cases where clouds are rendering a bit too dark, and in some cases still causing a minor amount of dark cockpits.
X-Plane2Blender Update
The above will be addressed in 12.2.0
In addition, we are also aware of the following
SSR
Cloud formations/Square clouds (This has been a persistent thorn in our sides, and won’t be addressed in 12.2.0. However, we are planning more improvements to this later this year)
3. Brakes!
We’ve radically expanded our brake system, with a plethora of different braking options and behaviours now available. Tom has ensured new documentation is available, which can be found here!
We’ve also introduced a new chocks mechanic, which is part of X-Plane’s ground handling. Only a select few default aircraft currently have this feature which includes:
Lancair Evolution
Van’s RV-10
Boeing 737
Cirrus SR-22
Aero-works Aerolight 103
Beechcraft Baron 58
How do I get chocks onto my aircraft?
We will have a new version of X-Plane2Blender out in the near future, along with documentation (and we will let you know) that allows for chock annotation, similar to particles are referenced. X-Plane provides the model for you, and references the position from the OBJ… not Planemaker. *On the subject of documentation, Tom has been hard at work re-vamping all of our docs. I think many will be pleased at the changes, which we hope to deploy this year!
5. New Gateway cut
We’ve included a new Gateway cut in 12.2.0. Thank you to all our Gateway artists! We’ve left a rough list of changed airports below.
Some of you may also remember a recent request-for-comment/poll for WED assets. We’ll have a lot more to talk about after the conclusion of 12.2.0. But the spoiler here is: we listened!
Following the success of versions 12.1.2 and 12.1.4 with our scenery and gateway artists, the art department is eager to understand which airport 3D/2D assets are at the top of your wishlist, and we would appreciate your input.
Guidelines:
Please focus exclusively on airport assets.
Limit suggestions to new static 3D/2D assets. This is not a place for new WED features or graphical (GFX) updates.
Please be succinct and polite.
Please provide references where appropriate and be as accurate as possible. Saying “I want a new bollard” is not good enough. Is there a size, shape, or colour that you need?
Understand that these are suggestions only, or “a request for comment”. There is no confirmation/timeline. Also understand that assets need to be accessible to all, not specific. So that weird hyperspecific 2-wheeled pushback truck at your local airport is not going to be appropriate.
We will close off comments on the 9th February
We encourage you to share your thoughts in the comments below or join the discussion on our *Discord server.
Many thanks, we look forward to your ideas!
*To access, click on this link here, and make sure you pick up the “creator” role. There you should see a channel called #rfc-wed-assets
Hello everyone, and welcome back to the blog post (quite some time right?). We have recently released 12.1.4 as a beta, and we highly encourage you to check it out here. There is some nice content released, that makes for an exciting appetizer for the new year. Whilst we don’t expect to be on this beta for too long (famous last words), most of the new additions will likely be subject to long-term feedback. Please message in support!
One of the major additions for this release is more content and updates for our scenery gateway artists!
New Default Assets
For those unaware, back in November, we welcomed Justin (MisterX) to the Laminar team. He likely needs no introduction, as his scenery work is the stuff of legend amongst the community. So we’re really excited to have him join the team, and Justin represents another infinity stone amongst our third-party hires!
Over the winter period, Petr (Our in-house legendary artist) and Justin went to town on some new scenery assets for X-Plane. Some of which, were highly nominated on our Discord feature requests. These assets include;
Trucks
Emergency/Fire Training Equipment
Airside Operations
De-Icing Equipment
Hopefully not to state the obvious, but these are not dynamic. So please don’t start placing these in areas that will obstruct the “simulator flow”.
We will have some more inbound assets over the coming months, so keep an eye peeled.
X-Plane Gateway
Whilst we didn’t highlight each airport in our initial release posts for 12.1.4, we did have a staggering 1400+ updated entries that we’ve included in our latest cut of the gateway. Take a look below (note, there are approximately 100 airports missing from this list)
Because of our new semantic versioning, internally we’re looking at having these cuts more frequently. Some of you have already noticed our update cadence has been picking up, and hopefully cycling in these updates more regularly will be refreshing to our gateway artists.
Last week Marco and I were on a live stream and had a chance to talk about some of the upcoming developments we’re working on; next month Marco will be full time with us, but for now here’s a few notes on things making progress in the lab. I’m sure publicly discussing them on Friday the 13th will have no unexpected consequences. (Knocks on wooden desk repeatedly.)
We should hopefully have a release candidate for 12.1.2 next week – this week the team was fixing crash bugs, and we’re down to trying to finishing up stability. Crash rates in the beta look good enough to ship. (Please consider turning on analytics – we don’t collect your personal data but we do record crashed vs non-crashed sim runs – this is how we know whether stability is getting better or worse!)
12.2.0
Dark Cockpits: Art team is evaluating Maya’s exposure fusion work. Generally things look better than the shipping product, but we discovered yesterday that the indirect lighting environment is pretty weird on cloudy days. We’ve been fixing bugs, removing old hacks we no longer need, and I’m optimistic that the update should be both better looking and closer to reality.
We also have some fixes to the PBR material model that may get X-Plane closer to Blender and Substance Painter 2. This is a double-edged sword: we don’t want to change the material model such that everyone has to repaint everything (that would be too much both for our team and for add-on makers) but we do want X-Plane to be as close to WYSIWYG possible to make development faster. Art is evaluating these changes too.
We’re targeting 12.2.0 (a “major free update”) for these changes so we can run a substantial alpha and beta program and give add-on makers time to report bugs.
12.1.X
ATC, Weather: we have two minor releases before 12.2.0 making their way through testing. The first is ATC with SIDs and STARs – as of now it looks like that will ship next, but all releases are subject to change based on bugs.*
The other update is a weather update, and it looks like it may have some substantial features:
New in-cockpit weather radar, with SDK support.
Fixes for real weather bugs, including jumps in pressure altitude (which are driving the autopilots crazy) and some fixes to cloud visuals.
Plugin-controlled weather across multiple locations.
Better weather sync over the internet.
As of now the weather branch also has network sync of trucks and jetways to external visuals. This only affects users using external visuals but for those users, it’s a big feature, and it’s also an important foundational technology for us.
* In the past we have announced “X” is next and if that code was buggy, we’d just delay shipping anything until we fixed it. We’re trying to be more flexible and ship what’s ready first, so good code doesn’t have to wait for unrelated bugs.
World Editor 2.6r1 is now available. This is the first release candidate, for which we encourage you to explore before we lock it down for release! Version 2.6 packs quite an extensive feature list. Features include :
New Shape Entities
Allowing users to define lines and polygons, and then export to .kml or .osm for use in third-party tools!
Polygonal Exclusions
A highly requested feature! (Note, these do not work on forests in current-gen scenery. The default behaviour will instead treat the defined exclusion as one large bounding box)
Terrain Objects
Some of you may have noticed that the terrain for TNCS and TFFJ in 12.1.2 looks a bit more defined. This is somewhat of an illusion, as we are still using the same base mesh that shipped in 12.0.9.
Terrain objects are designed to interface with orthophotos and create small scale 3D details.
*This is NOT a replacement/new/next-gen system for mesh-editing. But rather an additive tool that can help define more awkward structures
Dual-docking Jetway support
Another highly requested feature, WED can now support dual-docking jetways. Existing sceneries are NOT automatically ready, and must be defined by an artist first. Thanks to Michael, in a future version of X-Plane, we will push a new Gateway cut with some demo airports that are ready for dual jetways. These include (*subject to change):
Strings can now be defined down to the centimeter, instead of the meter… allowing for more accurate object spacing.
For WED newcomers
In the past, whenever users have attempted to open a scenery pack without an .XML file, all you got was an empty project. This was obviously confusing to users who did not know how to import the scenery files in. Now in WED 2.6, we analyze what is inside your scenery pack and offer to import the relevant files for you. If starting from a completely blank project folder, we now also give you the option to download directly from the gateway! These changes should help newcomers and seasoned artists get stuck into a project with ease!
Misc
In addition, a number of UI enhancements for users have been made available.
X-Plane 12.1.1 is out! This is a quick “bug fix” patch on X-Plane 12.1.0. If you could not load the sim due to problems with XPLM_64.dll, please update using the installer to fix it. Full notes here.
Websocket-To-Me Time
Besides fixing bugs, X-Plane 12.1.1 enables X-Plane’s web API*. Basically we were asked for this so many times at FSExpo that Chris and Daniela put the remaining work on the fast track.
Our plan is to build several services into the web server; the web APIs will not be exact mirrors of the plugin SDK, but will cover enough functionality to build wide range of apps. We have plans for:
Datarefs (shipping now)
Commands (in developement)
Initiializing Flights (in development)
Documentation on the new web APIs can be found here.
Doc-It-To-Me Time
X-Plane alum and fellow X-Plane-10-survivor Tom Kyler is back working on developer documentation for us, and has finished a first release on some of the new X-Plane 12.1.0 features:
Tom has big plans for X-Plane’s documentation; one thing that’s great to see in these docs is really good illustrations based on real sample 3-d models and test projects.
The Devil Is In the Details
A detail texture is a repeating, high detail pattern that adds high-res detail to an otherwise lower resolution texture. Detail textures let us add fine detail at high resolution without using a ton of extra VRAM efficiently. We use detail textures with bits of “gravel” to make the runways look more realistic.
We introduced detail textures over a decade ago in X-Plane 10 to enhance ground detail around airports and in the new (at the time) autogen cities.
What we did not do was give them a good name. At the time we called them “decal” textures. This has turned out to be incredibly confusing, because in every other game engine ever, a high-res texture applied over the entire material (which is what we have) is called a detail, and a decal basically means “sticker that you stick on somewhere”. Detail textures make the ground look more rocky, decals add blood spatter and bullet holes to the walls in your zombie-shooter game.
So moving forward, we are going to try to call detail textures “detail textures” wherever possible, to be consistent with the norms of content creation. Tom and Maya will be updating labels in Blender and doc to be consistent about this.
For X-Plane 12.1.1, Maya has extended the detailing system in two ways:
Detail textures work on OBJs and not just in the scenery system, so aircraft authors can add detail too.
Detail textures can affect the bumpiness of normal maps and not just the color of albedo textures, so detail textures now interact with the lighting model in realistic ways.
* Please note that while the dataref API does use websockets, most of our web APIs are conventional REST requests over HTTP. I did consider “The X-Plane developers never REST..until now” as a section title.
The X-Plane 12.1 release candidate is out today!* The last twenty four hours have been a mad dash to get the release candidate posted before we head to Las Vegas for FlightSimExpo 2024. If you are attending, stop by the X-Plane booth – there will be a bunch of us there including myself, Austin, and Marco, our new release manager.
All of this is a little bit exciting because we can’t recut the release candidate while in Las Vegas – one of the machines involved still requires a human to sit in front of it. We’ve been racing to get the RC done while we still have access to those machines, and the build system responded as you’d expect – by failing in all sorts of new ways. My plan is to just not breathe until I’m on the plane tomorrow.
(*) Because this is a release candidate you still have to check “get betas” in the installer to opt in to getting the RC; once the candidate has been out for a week, if nothing huge goes wrong, we will declare it final and it will become the official update for everyone.
If you’re at FSWeekend, say hey to our peeps there! In the meantime, we’re working to kill off the last 12.1.0 features. A few internal pics* from killing off the last features:
Author-controlled de-icing has had a series of bugs in 12.0.x. For 12.1.0 we’ve gone over it with a fine tooth comb; Alex ran the above tests with our Kingair, which has the unusual case of two overlapping de-icing zones. We’ll have updated docs, Blender exporter support, and builds for authors.
Some tests of real weather. In the first pic, the sky says clear but there’s a distant cloud…because the weather report is local.
Water turbidity is fixed for 12.1.0 and I am finishing documentation for authors. It looks like Oscar’s work on Ortho4XP is on GitHub. Please note that X-Plane 12.1.0 does not improve X-Plane 11 orthophoto water compatibility; v11 packs will only have 2-d water because their meshes are not triangulated properly for 3-d waves.
Based on the progress we done this week, I am hoping we will be able to test these features next week and start a private alpha in our developer lobby.
* Please note: these are internal pics from the developers that I am posting while the marketing team is too far away to object – literally what we were passing around while discussing the features. Don’t panic over FPS; the sim is running a debug build in those real weather pics, for example.
