Since X-Plane 10.30 shipped, we have received a lot of feedback about the stability of X-Plane 10.30 – or rather, the lack thereof.  We are getting more bug reports that the sim crashes, more tech support requests, and more discussion of crashing in the forums.

We don’t have a good way to characterize with hard data how much less stable 10.30 may be – unfortunately even the crash reports we gather don’t give us a statistically clear picture.*  But with this much user reporting, stability is our first line of inquiry. 10.31 will come out shortly (and will fix a few issues) and we’ll keep investigating problems and pushing small patches (to fix stability issues without introducing new ones) until we’re back on solid ground.  The rest of this post discusses some of what I’ve seen so far looking into stability issues.

How To Use the Automatic Crash Reporter

Since X-Plane 10.10, we’ve had the ability to send crash reports** directly to Laminar Research.  Here are a few suggestions:

  1. If you need help, contact tech support!  This is the most important thing. We do not read every single crash report!  Instead we use statistical queries to locate and identify common crashes.  If you crash and write in the comment field “please help me”, your cry for help will probably be lost.
  2. Please do report every time X-Plane crashes, even if the crash fields are blank.  The reporter can handle huge quantities of crashes, and we’re trying to gather a statistical picture of crashes.  If you get a crash every day, we want to know that it wasn’t a one-time thing.
  3. If you hit a reproducible crash (e.g. every time you follow a simple set of steps the sim crashes with a crash-auto-report dialog box), please file a bug and include the crash report (from Output/crash_reports).  Reproducible does not mean “fly the sim and it crashes”.  It means “start the sim with no prefs, open the C172, flip the avionics switch 3 times and the sim crashes” – specific steps with a known crash outcome every time.

Reproduction steps are gold to us – sometimes we’ll see a crash and the proximate cause is “hrm – the sim crashed drawing an OBJ”.  Sadly this doesn’t tell us much about how to reproduce the crash – and clearly the sim does not crash every time you draw an OBJ; it wouldn’t start up.

The Problem with OpenGL Drivers and 64-Bit

Our crash reporter can only trace the origins of a crash through X-Plane code.*** Unfortunately, of the top 5 Windows 64-bit crashes in 10.30r2 (and this represents most of the crashing going on), 87% of the crashes originated in an OpenGL driver from AMD, NVidia or Intel.  (This does not mean the driver is bad – it could mean that X-Plane sent the driver junk data.  Or it could be the driver’s fault.  We don’t know.)

Unfortunately that means that for the vast majority of 64-bit crashes, we have no idea what the sim was doing when it crashed – the trace from the driver will contain junk.  All we know is that we were, in fact, in the driver at the time of death.  (We also have the log.txt but often it just indicates the user was flying.)

This is a long term problem that we need to address as developers with our crash report technology; we have some ideas on how to fix it but they will take time to implement. Clearly we’ve hit a point where the low hanging fruit (crashes that affect a lot of users but are entirely in X-Plane code) are gone.

Coping With Add-On Crashes

We do not have good statistics (yet) for what percentage of submitted crashes were on X-Plane installations heavily augmented with add-ons, but it is not uncommon to see a user with a custom aircraft running a custom plugin, an online flight plugin, tons of custom scenery packs, and general utility plugins.

This kind of customization is great!  We wouldn’t have added the SDKs if we didn’t want people to use them.  But when it comes to isolating a crash it does make things quite a bit more difficult.

My goal for X-Plane is to not have X-Plane crash due to bad third party add-on data, no matter how badly it is formed.  To that end, when I find cases where bad data will crash the sim, I am adding code to protect, document, and shut down based on the error.  (Some errors are recoverable, some are not, but the important thing is to give the author some feedback as to what is going on.)

In the long term I think we need to add better diagnostics to our crash reporting to show what add-ons are involved in a crash. If you have 1000 scenery packs, I can’t go download them all in an attempt to reproduce a crash.

In the short term, if you are seeing instability and you have third party add-ons installed:

  • Please run without the third party add-ons – the goal here is not to have a more boring flight, and it is not a cure – it is a diagnostic tool.
  • By running with different sets of add-ons you can isolate which add-on causes a problem with a given flight.
  • If you find a problem with a plugin (whether it’s in an aircraft or the add-on itself is a plugin), you probably need to contact the plugin author – the plugin contains its own code that we don’t have source to.
  • If you find a problem with data (e.g. a scenery pack or airplane without plugins), please file a bug, including where to get the scenery pack or aircraft, and how to reproduce the bug.

Reproduction steps are gold to us!  The combination of the add-on and simple steps to reproduce let us find and fix problems, even with third party add-ons.  If you have a flight that crashes, please try to simplify it.  Try going directly to the location that you were at where you crashed, starting with the engines on, etc.

Yosemite

While we’re talking about stability, I have to mention Yosemite: on OS X the vast majority of bug reports we’re getting that involve “weird stuff” and crashes involve Yosemite.  We’re still trying to sort out the issues, but Yosemite is definitely the most disruptive OS X major version release we’ve had in a long time, and because it’s free, a lot of people downloaded it.

  • If you haven’t updated to Yosemite on you’re using X-Plane, consider waiting until things settle down.  We’ll work around problems, and Apple will issue patches.
  • If you have an older machine, consider never updating.  Perhaps this is heresy but often the maximum OS version your machine can run is limited by hardware requirements, e.g. this is the OS that just about knocks your machine over.  How well is it going to run X-Plane?  How much of the hardware resources are left for us?
  • When a new OS comes out, don’t just click “download” even if it’s free; wait a week or two and see how well it works for other people first.

A major OS upgrade is when the OS vendor can make major changes, drop old technology, etc.  On OS X it also means new OpenGL drivers (because the drivers are always packed with the OS), so it’s a potentially disruptive update.

Update: it turns out Yosemite also broke our automatic crash reporter, which would explain why none of the common Yosemite crashes show up in our top crash list. 🙁  Some users have sent me the Apple crash reports that appear (since our crash reporting doesn’t run) so that we can diagnose what’s going on.

Next Steps

10.31 should be out in a day or so (including Steam); as we continue to find and fix stability problems we’ll continue to issue small patches.  We’re looking to do a 10.35-type release with small features and new airports; the current plan is that Julian and Tyler will do a public beta of the new airports (as a single separate scenery pack) so that people can try the new airports and spot problems before we get into beta.

 

* Crash reports don’t tell us how many times the sim was run overall; users with stable systems don’t send us any data, so an increasing crash report incidence would be meaningless – we don’t have data about increasing or decreasing use of the sim.

** Here we’re talking about true program crashes – the uncontrolled exit of the sim for unknown reasons.  If you have a controlled failure, e.g. you load scenery and it’s missing objects and the sim refuses to run, you don’t get a chance to auto-report that.  We also don’t view this as a stability issue – X-Plane can be very unforgiving about broken third party content, and we can have a separate argument about whether this is a necessary evil or terrible user experience (or both!), but there’s no diagnostic information to gather.

Update 2: Blog comments are now working again.

*** The problem is that for 64-bit there is no standard for back-tracing a stack without the debug symbols to decode the stack linkage; there is no standard link register.  I’m guessing that the ABI designers thought they were helping the compiler guys optimize, but in practice it makes debugging other people’s DLLs miserable.  Microsoft publishes debug symbols for every DLL they make ,which is a big help, but the driver writers do not.

About Ben Supnik

Ben is a software engineer who works on X-Plane; he spends most of his days drinking coffee and swearing at the computer -- sometimes at the same time.

19 comments on “Diagnosing Crashes in X-Plane 10.30

  1. Good post Ben, I don’t feel so bad now. 10.31r2 started total lockups on my mac and my pc was already flaky. I have Yosemite and an AMD 7970 which it don’t like, so I’ve got a GTX970 AMP on the way. When I make the swap and see if that had anything to do with the lockups I’ll file a more detailed bug report. The lockups are repeatable but I know the AMD drivers ain’t doing what they should either, ie safari jitters badly and don’t draw pages like it should,(new java crap I’m sure) but my little 640 nvidia is smooth as it should be so it’s all nvidia from here on out. I’ll let you know the outcome when the beast comes and I might go ahead and file a report now that I know it’s not just me..
    Again, thanks for this post

    Thomas Kirk

    1. Generally speaking, if you upgrade your system software and everything involving the Window manager and simpler apps gets worse, the odds of X-Plane working well is slim. We put more pressure on the drivers, and we have higher system performance demands.

      1. Perfectly understood, Ben. I’ll see if a one video card driver system dose and go from that. Never liked AMD/Nvidia together anyway, to many workarounds.

  2. I run two computers with a crossover ethernet connection, Win 7. I have noticed crashes on both, usually BSOD. Blue Screen app reading .dmp file on both fingers ntoskrnl.exe as the culprit with reference to Memory Management. Memory checks out OK with Memtest, so thats doesn’t seem to be the problem. Maybe a clue there, I can provide the .dmp file if it is of interest…

      1. New user here with about 80 hours in the sim. I’ve had one or two BSOD’s…one for sure was memory management. I down-clocked my RAM from 2133 to 2000 and while I’m not sure that was the issue, it has not recurred. Granted I’m pushing my RAM a good bit as it’s spec’d at 1600 (no BSOD’s prior to XP). One comment about Memtest. I’ve passed Memtest without issue only to find a memory failure in prime95. Try that stress test with your ram and see if you have issues there. I’ve found memtest to not be as reliable as I once hoped and thought it was. FWIW.

        1. I may be conservative, btu if your RAM is spec’d at 1600 and you have a BSOD…well, if it was me I’d clock all the way down to 1600 as a start. 🙂

          1. “well, if it was me I’d clock all the way down to 1600 as a start.”

            Crazy talk. 😉 It’s actually that magic low-profile/low voltage samsung memory that was made a few years ago, 1.35v spec up to 1.5v (so running in spec with adjusted timings). I don’t believe the BSOD was RAM related. The system is solid at 2000 and was at 2133…i think driver related. My R9 280x throws artifacts with the Carenado B200 and some different artifacting (more of a flashing box rather than streaks across the screen) with the FlyJSim Q400. Annoying to be sure, but still usable. Running 2560×1440 and video temps at 60C to 65C with load at 60%. CPU load of around 40 to 50% with low temps. Getting 18 to 26 FPS normally depending on clouds.

            I can run FurMark on the video card to get temps well above 70C with no artifacts. Video is not overclocked. This seems to point to drivers but I can’t be sure.

  3. Excellent Road Map. But my “IMHO” tell me: “Let’s try to send an informative crash reports. Even you have more than 1000 custom scenery, plugins or other bullshit. Winners will be named on the .org, still too lazy to remove.”
    Cheers
    Youri

  4. Guess I should really feel lucky, been running 10.31r2 and r3,64bit, with Yosemite, and have had no crashes or issues whatsoever, with a ton of scenery (with zero errors in the log.txt,all fixed when spotted), and the HD mesh V2.

  5. Mid 2010 27″ iMac, 2.93GHz i7, 16GB RAM, Radeon 5750 1024GB
    Yosemite: every beta, and release
    X-Plane: every 10.x currently 10.31r3
    no problems

  6. I run a copy of the X-Plane DEMO in the same settings as my main X-Plane app (I have only added in the world scenery to make it a full X-Plane world simulator and switched it out of the demo mode), but the demo is totally clean (no custom scenery, add-ons or plugins), so if I get a crash I will only transfer the aircraft flying and see how it then performs in the DEMO version. With this you can then deduct again if it is the aircraft or the app?

    Again the DEMO is handy if X-Plane is in a beta run, I update the main app but keep the DEMO still clean if I need to test or check between the main app and the original DEMO version, If the main goes down or becomes corrupt I then rebuild a clean beta as a new main app… I rarely have to do this, but it saves a lot of hair pulling 🙂

    I haven’t updated to Yosemite yet, I now take the advice and now wait and is not anymore a first responder, Apple and co now use you as a beta tester for the software and I never now buy anything that is brand new but wait now for version two or even three before purchasing or upgrading software.

    1. If the crash report doesn’t clearly show a problem with the ACF (sometimes it’s obvious) then the problem has to be diagnosed by picking a different airplane.

      1. Usually it is the aircraft I am testing, not X-Plane in the different apps… but it does work like I noted if running between a beta and a stable XP version. It is always good to have the clear difference between the two apps 🙂

  7. RC3 works fine here with no crashes. The only recurring problem I have seen in the last few releases is random switches to “map”. Really increases the pucker factor during approaches. I have a button which corrects it quickly. No, I have not filed a report as it has only recently surfaced, is minor, and I figure it will go away when all the other bugs you have to work on go away. I appreciate all your efforts – greatly!

    John

    1. Just figured out the “map problem”. Not a problem with XPX – a problem with Track IR. Reinstalled, recalibrated, realigned, and everything appears to be back to normal. I had noticed that the problem seemed to occur when I was turning my head to look out to the side. Apparently, the alignment has somehow become more critical in recent updates. I hope this might help someone else who may have experienced a similar problem.

  8. FWIW — I’m on a mid 2011 iMac running Yosemite and have about 15 hours on RC3 with no problems.

    AM

Comments are closed.