Lori and I are hooked on the TV show “House”, where Hugh Laurie plays a really grumpy doctor who solves bizarre medical cases more or less by ESP. The characters are well written, but the medical plot line is somewhat predictable: there are four quarters to the show – in each one House except the last, House will make the wrong diagnosis and the patient will get worse right before the commercial break. (Usually this involves massive bleeding or cardiac arrest going into the long commecial block at 0:30.) None of the symptoms fit until the very end when House finds the simple right explanation they just couldn’t see.
This set of nvidia crash bugs felt a lot like that – we had multiple attempted fixes, some of which didn’t help at all, until finally after multiple tries I found a bug that explained all of the otherwise completely weird behavior we were seeing.
But I must admit – I have brought shame on the house of X-Plane…the buggy code was mine and the mistake was really stupid. Why nVidia on Windows? As far as I can tell the optimizers present in most OpenGL engines can change whether (and how) the bug manifests itself – different OS/driver pair: different engine with different optimizers.
Now that (at risk of massively jinxing ourselves) we have the crash bug fixed, I will resume performance work. Once we get a build done with all of the immediate performance items I want to cover, we’ll start collecting user reports on in-field performance. So I should have more specific instructions on what you can do to help us isolate performance problems in the next few days.
(Probably I’ll be blogging a lot today…the load/change-planes/crash/recompile cycle I am going through while working on the crash bug is a slow one – my old Dell is long in the tooth…it leaves me a lot of time to post.)
Beta can be frustrating for both users (why don’t they fix the bugs I reported) and programmers (I need more details in my bug report). Here are a few thoughts on what makes an initial bug report useful:
- Precision of reproduction. This is probably the most important thing – we get a lot of “open an airplane”-type instructions. Which airplane? It turns out that a lot of bugs depend on the particular content being used. So if you know how to make a bug happen, please describe it in the most painfully precise steps possible!
- Short is beautiful. We must know precisely how to reproduce a bug, but a procedure that takes two hours kills our productivity. So please try to determine how to reproduce the bug with the minimum number of precise steps.
- Clean system. Bugs that involve only the default content shipped with the sim are more useful for us because they’re quicker to find and more likely to be due to a bug in the sim itself.
- Nuke the prefs. Bug reports that start with “delete your preferences” are good because it means the bug procedure starts from a known state (the sim defaults). We get bugs that we can’t reproduce because something is subtly different in our system. Killing prefs is the quickest way to eliminate this case.
As an example, the cleanest, simplest version of the nvidia crash bug would be:
1. Delete prefs.
2. Start the sim.
3. Open the C172 using the “open aircraft” dialog box.
Result: unexpected program termination before the terrain is visible.
Back in the good old days (that would be X-Plane 6), X-Plane’s framerate would suffer in two ways as you cranked up the rendering options:
- For most features (more visibility, more autogen) as the CPU and GPU became more heavily loaded, the framerate would gradually decrease.
- If you ran out of VRAM (that is, the working set of textures needed per frame was more than your card’s VRAM) framerate would really die fast – think 2 fps.
The reason for this second behavior was that computers couldn’t shuffle textures between main memory and VRAM fast enough to render a frame in a 30th of a second.
As computers have gotten faster, this second behavior has gone away – modern cards, with fast PCIe 16x busses, can transfer textures from system memory to VRAM pretty fast – fast enough to have the working set be (slightly) larger than VRAM and still fly. So as texture memory increases, framerate decreases more gradually.
However, a new behavior has emerged: “glitching”. You may have noticed that when you’ve got your computer set to the ragged edge of the rendering settings, as you turn the camera, the framerate will stutter for a few frames, then return to a relatively high rate (40-50 fps).
What’s happening is: the working set of textures and geometry needed by X-Plane just barely fit in VRAM. But when you turn your head, a different set of textures and geometry are needed. While the card sorts out what is needed and what isn’t, it spends some time needlessly shuffling textures, and eventually reaches stability, with only what’s needed in VRAM, and framerate stays high.
Glitching has emerged as a mode of performance degredation because over time we’ve cut down the amount of “stuff” (textures and geometry) x-plane needs to draw a frame to only what’s really absolutely needed. This means there is less intersection between the working set in one view and another, and it also means you can get closer to the edge of your hardware.
So my view on glitching is basically “too bad”. If the working set weren’t as carefully trimmed, you wouldn’t have glitches, but the framerate would be entirely low, not entirely high. The only solution is to turn down settings that increase the working set (object density, world LOD, tex res, forests…) until the computer can run without glitches.
An even stranger variant of this: users sometimes report framerate getting “stuck” at 19 fps and then coming back when they change apps. The problem is that the driver doesn’t know exactly what the best order of textures to keep in VRAM vs. shuffle is…as the view changes, sometimes the driver ends up with a non-optimal decision about what stays in VRAM and what goes, causing framerate to drop. Changing which app is in the foreground fixes this by temporarily pushing a lot of items out of VRAM, at which point the driver makes a different decision by luck.
Again, the solution is simple: turn down rendering settings to get the working set smaller than VRAM.
Basically, if the working set is smaller than your amount of VRAM, you should have even framerates, proportional to rendering load.
If the working set is greater than VRAM, the driver may find an optimal way to shuffle things and only decrease fps a little, or it may find a non-optimal way to shuffle and you’ll get terrible fps, and that shuffle can change on the fly, causing framerate to fluctuate all over the place.
I would have to say my track-record at predicting hardware developments in the sim world has been pretty poor. But these two factors seem like they won’t change for a bit:
- The amount of working data a system can crank through in one frame keeps increasing while the total amount of virtual address space stays the same (3 GB).
- The gap between the fastest and slowest systems from a finite time period keeps widening.
To elaborate on this first point, as video cards get faster (both GPU and internal bus/RAM speeds), system RAM and buses get faster (graphics slot changes are rare, but with enough VRAM this is relatively moot) the amount of texture and geometry data that X-Plane can tell the card to draw in a 30th of a second keeps going up. So users are running with more trees, roads, 3-d, etc. than in the past.
But all of this stuff lives in memory, and even if a user has 8 GB, X-Plane can’t load more than 3. Imagine what will happen when a graphics card can draw 3 GB of data in one frame? X-Plane will have to use all of its available memory for things you see right now or it will waste graphics power. This would mean purging from memory anything that isn’t on screen!
On the second point, video card power doubles every, um, six to twelve months (perhaps more like 18, now that the card makers are hitting the same fabrication and power limits that the CPU makers have already hit, but this is all seat of my pants). So even if we only supported the last three generations of cards (we support at least seven!!) that gap in performance doubles!
This means that every year it requires a more flexible rendering engine to make a sim with decent frame-rates for old computers and up-to-date graphics on new ones.
X-Plane 9 is currently in beta 2. Long-time X-Plane users know that a lot of bugs get fixed between early and late betas, and they also know that a good number of bugs get added between the early betas.*
If all goes well, X-Plane 9 beta 3 will be out in a few days. My advice is: don’t panic. X-Plane beta 2 crashes for a number of users, so our top priority is to fix the crash and get the fix out ASAP. If your bug doesn’t get fixed in beta 3, it’s probably because we’re still working on it but didn’t want to delay getting the crash fix to users.
Similarly, beta 3 will include some performance improvements, but more are coming. Beta 3 doesn’t represent the end of our performance tuning, it represents the first beta that we can do serious analysis with. We only have a fraction of all of the supported video cards within the company, so if your computer is having performance problems, well, we’ll figure out what’s going on in beta 3 and then fix it.
* Our approach to bug fixing is: if a piece of code is buggy because it’s subsystem has a design problem, we go in and fix the design problem, even if we’re in beta. Other companies might say “no fixing design problems (which changes more code) during beta.” But the way I look at it, badly designed code is just going to cause problems all the time until it’s fixed, and it has to be fixed some day, so why wait.
(Why would there be badly designed code in X-Plane, or any computer program? Computer programs change over time, and the functionality they perform changes and grows. As this happens, the designs of the past no longer make sense for future requirements. In my experience most design problems come from code “outgrowing” its framework.
So our approach is to upgrade the framework as soon as it shows signs of growing pains, rather than jamming as many features into the existing framework as possible until it becomes so overcrowded that we can’t get anything done.)
Read the info here:
//www.xsquawkbox.net/xpsdk/phpwiki/index.php?NewVersionAnnouncement
Please note that if you are a scenery author and not a programmer, this will not directly affect you at all. (But see the section on compatibility – old plugins will still work!)
Version 9 will also feature a lot of new datarefs – this should make things a lot easier for authors doing animation.
The new API will allow programmers to access the terrain mesh. This means that add-ons that want to add dynamic graphics content by drawing (e.g. drawing in baggage trucks) will be able to place that content “on the ground” no matter what the surface of the airport area. I think this will allow for some interesting new add-ons.
There have been plenty of rumors and semi-official posts regarding the upcoming major revision of X-Plane (X-Plane 9). I have been trying to keep my mouth shut about it…the problem with pre-announcing anything is that the upside to us is small (at best we do what we said) and the downside is large (at worst we don’t do what we said and people get grumpy). No one complains if XP9 turns out to have no-pause scenery load and it’s a surprise…but plenty of people complain if we say “there won’t be pauses” and then they are.
But…the situation is becoming mildly absurd…plenty of info is out there, and saying “the upcoming major release”, etc. just feels political and weaselly. Austin would be disgusted.
So listen: I am going to try to provide some info on X-Plane 9. This info is subject to change. This is what we think is going to happen to the best of our knowledge. The release is still a ways away and enormous changes will happen. When things change, do not bitch to me that “you promised X” would happen. I do not promise anything. This info is provided to try to help those making add-ons for X-Plane plan appropriately.
With that in mind…I will try to post some more details on the authoring environment in the next few days. For now, here’s some very basic guidance on compatibility and hardware requirements:
- The hardware requirements will be at least as high as X-Plane 8. If your machine is gasping and wheezing on 8, it’s not going to be any better on 9.
- X-Plane 9 will depend more heavily on pixel shaders. If your hardware doesn’t have pixel shaders (GeForce 2-4, Radeon 7000-9200) or has really crappy shaders (GeForce FX series) you will miss out on a lot of the cool stuff in v9, and possibly have the scenery look worse (but faster) than v8 (as we move features from the CPU to pixel shaders).
- Scenery that opens in x-plane should open in X-Plane 9 unmodified – if the scenery works in 8 but not 9, report it as a bug!
- Plugins that work in v8 should work in v9 without modification.
Finally, we are trying to finish up X-Plane 861…this is a bug-fix patch for version 8 – it contains no new features, but it does fix a few nasty bugs, some of which cause crashes. So if there is any new feature, it’s coming in 9, not 861. Version 8 has been out for a very long time, so I will accept no argument that v8 should have more features than it does now. It’s been a long run!
(One of my main goals with 861 is to try to fix any weird behavior for third party scenery add-ons, so that a scenery add-on looks the same in v8 and v9. If we left the bug in 861 and fixed it in v9, authors would have to hack the scenery to make it work with v8, and then remove the hack and republish for v9. By trying to fix the authoring bugs in v8, at least when possible, it lets authors publish one package for both versions. Of course, v9 will have new features, so I expect some v9-only scenery will emerge pretty quickly.)
In my previous post I described the underlying problems that make translucency in airplane cockit objects tricky. That gives us the background to understand how X-Plane draws cockpit objects and why.
The actual draw order for airplane-related objects is this:
- Far away weapons.
- All of the airplanes that we are not inside (this means the user’s plane in an external view, and AI planes all the time). For each of these planes, the external cockpit object is drawn first, then the parts, then the attached objects in order.
- Clouds and puffs and other such environmental phenomena.
- The geometry of our plane, if it is to be drawn and we are in an inside view.
- The attached objects in order for our plane, if internal geometry is drawn and we are in an inside view.
- Weapons that are very close to the plane.
- The inside cockpit object, if we are inside the plane.
A few things to note about this draw order:
- Note that weapons appear twice in the category of “near” and “far”. This is all about the clipping plane – if a weapon is close to us, it must be drawn with the cockpit object, late in the draw cycle, when we are doing “close” things. Weapons are treated dynamically – they change where they are in the draw cycle depending on their position in space. This is necessary because a weapon starts out real close (just off your wing) and then goes real far away.
- The user’s airplane is special-cased when we are in an internal view … at that point if external things are being drawn, they are done in the later “close” view with the cockpit.
- The external cockpit object is (strangely) quite early in the draw order. There’s really no good reason for that. What’s particularly annoying is that it’s inconsistent with the internal draw order. The main point of internal/external cockpit objects is to let you simplify your cockpit object in external view for performance.
Is there a sane way to set up an X-Plane 860 aircraft with translucent geometry? I’m not 100% sure about this, but it looks to me from the code like the correct thing to do might be:
- Put any translucent windshield/canopy/etc. in the last attached object.
- Put only the interior panel (and not the canopy) in the cockpit object.
This works because in the external view the canopy is drawn last, and in the internal view, the panel is more or less guaranteed to be closer than the canopy, which is good enough.
Today is Rosh Hashanah, the Jewish New Year. In Jewish tradition it is also thought of as the birthday of the world. If you’re reading this blog, you probably have some interest in how X-Plane attempts to simulate the world on your computer. The attempt over the years to create a higher fidelity simulation has led me to a deeper appreciation of the subtlety, complexity, and beauty of the real thing.
(Nothing makes you realize how rich and intricate the world is than trying to model it with a few million triangles and ending up with something that looks completely crude.)
X-Plane’s digital world isn’t the only way that we interact with a proxy instead of the real thing. When we drive instead of walk, eat packaged food from a supermarket, talk on the telephone instead of talk in person, our technology becomes a proxy for our relationship with our direct natural environment – the planet, plants, animals, and other human beings.
Now I’m not saying that any of these things is bad. I’m not about to become a dairy farmer, and without the internet we couldn’t create X-Plane at all. But I think it’s important on this day, and hopefully every day to take time for activities that put us in direct contact with the world. Consider a few questions:
- How does what I eat affect the world?
- How does my travel affect the world?
- What impact does my home have on the world?
- Am I leaving the world in better condition for the next generation or worse?
Please … take a few moments to consider the world, the only home we’ve ever had.
When I was in the Dolomites a few years ago with Sergio we were looking at the dolomites from his friend’s balcony – mile after mile of beautiful mountains and rolling hills. I looked at him and said “God has more polygons than we do.” It was a joke at the time, but I think that the act of really observing the real world and realizing that the digital reality and technology we create can be a proxy and an addition but never a replacement is critical to understanding the responsibility for stewardship we have over the planet. Are we taking good care of our most precious gift?
Side note: please do not post tech support questions as blog comments – please use the x-plane-scenery yahoo list or the x-plane.org scenery forum. I would like to keep tech support discussions in easily search-able public places so future users can get answers quickly.
I saw a post on x-plane.org referring to the process of creating translucency in objects attached to airplanes as a dark art. There is definitely a lot of weirdness in how X-Plane draws airplane objects. I will try to shed some light on what’s really going on and how to deal with it. For this first post, I’ll explain the requirements of the hardware, which shape the ensuing chaos.
First, and most important, in order for X-Plane to render translucent geometry (objects or otherwise), the geometry must be drawn from farthest from the viewer to closest. This is in stark contrast to normal operation — for transparent or opaque geometry, we can draw the closest objects first, and the graphics card makes sure the far away objects don’t “paint over them”. But the technology that does this (z-buffering) doesn’t work when the closer geometry is translucent. (If the translucent geometry is drawn first, it acts opaque to what is behind it.) For more info on this phenomenon and what to do about it, look here.
So in order for our translucent cockpit objects to draw correctly they need to be drawn from “back to front”. But note that this term is relative to the camera — which object is closest will depend on where the camera is located!
The other cause of cockpit object weirdness is the near clipping plane. At any time when X-Plane is drawing, there are two limits on where we can draw:
- The near clipping plane defines an invisible wall – anything closer to the viewer than this distance will not be drawn.
- The far clipping plane defines the other invisible wall – anything farther from the viewer than this distance will not be drawn.
The far clipping plane is usually set far enough away that objects disappear into the fog before hitting it. The near clipping plane is usually set close enough that by the time an object hits it, your plane has crashed.
Now here’s the rub: the quality with which the graphics card can discriminate which polygon is closer (via z-buffering) goes down as the ratio of the far to near clipping plane gets larger!
Take a second to think about that. Basically if we want to increase the visibility in X-Plane without losing z-buffering fidelity ,we need to move the near clipping plane farther away from the user.
The real problem is this: X-Plane, with its sometimes long visibilities (when you get up into orbit, you can see a long way!!) really stretches the z-buffering fidelity of even modern cards. So we have to keep the near clip plane fairly far from the viewer in order to have the world look reasonable. But that distance might be a lot larger than the distance from the viewer to the interior of the cockpit!
We work around this by having two separate drawing passes. We first draw the “outside” world, with a near clipping plane that is fairly far away. (Every now and then a user tells me that this clip plane is causing scenery not to be drawn.) We then draw the cockpit interior and the user’s plane, with the near and far clip plane both reset to be a lot closer. This way we can use our “z-buffer precision” in different ways for different geometry.
(It should be noted that z-buffering does not correctly handle the relationship between near and far objects when we reset the near and far clip plane. This technique works because we assume that everything drawn in the “cockpit” view will draw over everything drawn in the “outside” view.)
In my next post I will explain what X-Plane actually draws. For now suffice it to say that X-Plane has the task of drawing from farthest to nearest, but also the task of drawing in two phases: a far-away outside view and a close inside view.
