It’s been a tough year for X-Plane authors. Key framing, manipulators, global lighting, baked lighting control, generic instruments, normal maps…version 9 has extended the possibilities for airplanes in a huge way. This is great for X-Plane users, and great for any author who wants to push the envelope, but there is a flip side: the time investment to make a “cutting edge” X-Plane aircraft has gone way, way up.
Here’s the problem: as hardware has been getting faster, the amount of data (in the form of detailed airplane models) needed to keep the hardware running at max has gone up. But the process of modeling an airplane hasn’t gotten any more efficient; all of that 3-d detail simply needs to be drawn, UV mapped and textured. Simply put, NVidia and ATI are making faster GPUs but not faster humans.
That’s why I was so excited about order independent transparency. This is a case where new graphics hardware and nicer looking hardware means less authoring work, not more. (The misery of trying to carefully manage ordered one sided geometry will simply be replaced by enabling the effect.)
My Daddy Can Beat Up Your Daddy
Cameron was on FSBreak last week last week discussing the new CRJ…the discussion touched on a question that gets kicked around the forums a lot these days: which allows authors to more realistically simulate a particular airplane…X-Plane 9 or FSX.
This debate is, to be blunt, completely moot. Both FS X and X-Plane contain powerful enough add-on systems that an author can do pretty much anything desired, including replacing the entire host simulation engine. At that point, the question is not “which can do more” because both can do more than any group of humans will ever produce. As Cameron observed, we’ve reached a point where the simulator doesn’t hold the author back, at least when it comes to systems modeling.
(It might be reasonable to ask: which simulator makes it easier to simulate a given aircraft, but given the tendency to replace the simulator’s built-in systems on both platforms, it appears the state of the art has gone significantly past built-in sim capabilities.)
Graphic Leverage
When it comes to systems modeling, the ability to put custom code into X-Plane or FS X allows authors to go significantly beyond the scope of the original sim. When it comes to graphics, however, authors on both platforms are constrained to what the sim’s native rendering engine can actually draw.
So if there’s a challenge to flight simulation next year, I think it is this: for next-generation flight simulators to act as amplifiers for the art content that humans build, rather than as engines that consume it as fuel. The simulator features that get our attention next year can’t just be the ability for an author to create something very nice (we’re already there), rather it needs to be the ability to make what authors make look even better.
(This doesn’t mean that I think that the platforms for building third party “stuff” are complete. Rather, I think it means that we have to carefully consider the amount of input labor it takes to get an output effect.)
This is getting embarrassing – I’m at risk of getting shut out. I was able to fix the “Null texture, how” error users were seeing on NVidia hardware.
It turns out it was an uninitialized variable in code that was never used until NV changed their drivers. As far as I can tell, NV dropped support for FSAA in 16-bit mode a few months ago, at least on some of their newer GPUs. (It is also possible that the incantation necessary to get FSAA has changed a lot and I simply don’t know what it is.)
So the dialog between X-Plane and the video card ran something like the Monty Python cheese shop sketch:
X-Plane: So … can you do full screen anti-aliasing?
GeForce 8: Oh yes, of course! (Please, I’m a GeForce 8 card.)
X-Plane: Splendid! So…how about 16x FSAA?
GeForce 8: Sorry, can’t I can’t do that.
X-Plane: Ah. How about 8x FSAA?
GeForce 8: Sorry, can’t do that either.
X-Plane: I see. Well then, how about 4x FSAA?
GeForce 8: Nope.
X-Plane: 2x FSAA?
GeForce 8: No way.
X-Plane: Ah. I see.
At this point in the dialog X-Plane would promptly lose track of what it had been doing in the setup process, throw out its notes on the GPU setup, and then freakout a bit later when it realized its note taking left something to be desired.
This is the first case I’ve hit where a video card advertises FSAA and can’t actually do it.
Anyway, if you have hit this bug:
-
Update to 941 final – it should fix it.
-
Stop trying to run with FSAA and 16-bit color. This is a somewhat crazy combination. FSAA attempts to clean up rendering artifacts at the cost of fill rate. 16-bit color creates artifacts to save fill rate. If your GPU needs 16-bit color to run at high framerate, it’s time to turn FSAA off.
(I realize that 16-bit color and aliasing are different kinds of artifacts, and some users might prefer harsh color transitions to harsh polygon transitions. But I still say, go for 32-bit color, no FSAA. When the sim is running in 16-bit mode, a good chunk of the sim still runs in 32-bit mode because 16-bit RGB surfaces only have 1 bit of alpha.* So you’re not quite getting universal savings but you get 16-bit output colors, so the results look universally bad.)
*This assumes 5551, or 565 pixels. There is a 4-bit alpha 16-bit color format, cleverly called 4444, but if you thought 16-bit looks bad…
To get this out: I have no idea if/whether/how much SLI or Crossfire improve or hinder X-Plane’s framerate. None of my development systems have such hardware, and I spend no time either optimizing for SLI/CrossFire or testing. If you have done tests with SLI enabled and disabled, I would like to know the results!
I have read the white papers on how to optimize an application for SLI/CrossFire, and while X-Plane isn’t quite a laundry of SLI/CrossFire sins, we’re definitely an application that has the potential for optimization.
Now normally more hardware = faster framerate. In particular, the limiting factor of filling in a big display with high shader options and full screen anti-aliasing can be the time it takes to fill in pixels, and more shaders mean more pixels filled in at once.* Why doesn’t having an entire second GPU to fill in pixels allow us to go twice as fast?
The answer is: coordination. Normally the process of drawing an X-Plane frame goes a little bit like this:
- Draw a little bit more of the cloud shadows into the cloud shadow texture. (This is a gradual process.)
- Draw the panel into the panel texture.
- Draw the world (as seen from below the water) into the water reflection texture.
- Draw the airplane shadow into the airplane shadow texture.
- Draw the entire world using the above four textures.
Notice a trend? Those four textures are dynamic textures, meaning they are created by the viedo card drawing into its own texture, once per frame. Dynamic textures are generally a good thing, because they let us create texture content much more rapidly than we could with the CPU. (There is no way we could prepare the panel texture once per frame if we had to do it on the CPU.)
In fact, the total dynamic textures can be more so – if you use panel regions, there are 2 panel textures per region, and if you use volumetric fog, there are two more textures with interim renderings of the world, used to create fog effects.
Okay, so we have a lot of textures we drew. What does that have to do with multiple video cards?
Well, one reason why dynamic textures are normally fast is because, when a dynamic texture is finished, it lives exactly where we want it to live: on the video card!
But…what if there are two video cards? Uh oh. Now maybe one video card drew the water, and another drew the clouds. We have to copy those textures to every video card that will help draw the final frame.
There is a sequence to draw the right textures on the right card at the right time to make X-Plane run faster with two video cards…but the video drivers that manage SLI or CrossFire may have no way to know what that sequence is. The driver has to make some guesses, and if it puts the wrong textures in the wrong places, framerate may be slower, due to the overhead of shuffling textures around.
So SLI and CrossFire are not simple, no-brainer ways to get more framerate, the way having a faster GPU might be.
* If you have a huge number of objects, your framerate is suffering due to the CPU being overloaded, and this is all entirely moot!
The X-Plane version 8/9 default scenery uses raster land use data (that is, a low-res grid that categorizes the overall usage of a square area of land) as part of its input in generating the global scenery. When you use MeshTool, this raster data comes in the .xes file that you must download. So…why can’t you change it?
The short answer is: you could change it, but the results would be so unsatisfying that it’s probably not worth adding the feature.
The global scenery is using GLCC land use data – it’s a 1 km data set with about 100 types of land class based on the OGE2 spec.
Now here’s the thing: the data sucks.
That’s a little harsh, and I am sure the researchers tried hard to create the data set. But using the data set directly in a flight simulator is immensely problematic:
- With 1 km spatial resolution (and some alignment error) the data is not particularly precise in where it puts features.
- The categorizations are inaccurate. The data is derived from thermal imagery, and it is easily fooled by mixed-use land. For example, mixing suburban houses into trees will result in a new forest categorization, because of the heat from the houses.
- The data can produce crazy results: cities on top of mountains, water running up steep slopes, etc.
That’s where Sergio and I come in. During the development of the v8 and v9 global scenery, Sergio created a rule set and I created processing algorithms – combined together, this system picks a terrain type from several factors: climate, land use, but also slope, elevation, etc.
To give a trivial example, the placement of rock cliffs is based on the steepness of terrain, and overrides land use. So if we have a city on an 80 degree incline, our rule set says “you can’t have a city that slanted – put a rock face there instead.”
Sergio made something on the order of 1800 rules. (No one said he isn’t thorough!!) And when we were done, we realized that we barely use landuse.
In developing the rule set, Sergio looked for the parameters that would best predict the real look of the terrain. And what he found was that climate and slope are much better predictors of land use than the actual land use data. If you didn’t realize that we were ignoring the input data, well, that speaks to the quality of his rule set.
No One Is Listening
Now back to MeshTool. MeshTool uses the rule set Sergio developed to pick terrain when you have an area tagged as terrain_Natural. If you were to change the land use data, 80% of your land would ignore your markings because the ruleset is based on many other factors besides landuse. Simply put, no one would be listening.
(We could try some experiments with customizing the land use data..there is a very small number of land uses that are keyed into the rule set. My guess is that this would be a very indirect and thus frustrating way to work, e.g. “I said city goes here, why is it not there?”)
The Future
I am working with alpilotx – he is producing a next-gen land-use data set, and it’s an entirely different world from the raw GLCC that Sergio and I had a few years ago. Alpilotx’s data set is high res, extremely accurate, and carefully combined and processed from several modern, high quality sources.
This of course means the rules have to change, and that’s the challenge we are looking at now – how much do we trust the new landuse vs. some of the other indicators that proved to be reliable.
Someday MeshTool may use this new landuse data and a new ruleset that follows it. At that point it could make sense to allow MeshTool to accept raster landuse data replacements. But for now I think it would be an exercise in frustration.
You know it as the Radeon 9700 (and its later cheaper 9600XT, as well as the X300, X500 and X600). The R300 was a definite victory for ATI – when it came out it was vastly superior to the NVidia GeForce FX series, and for a video card it held up pretty well.
From my perspective as an application’s developer, however, the R300 has some fine print that makes it difficult to deal with:
- It features only 24-bit floating point precision (as opposed to 32-bit precision in all other shader-enabled hardware from ATI or NV). This is why the reflective water looks square and pixelated up close on these cards.
- It has a 96 instruction limit per shader (as opposed to the 1024 instruction or larger limit in all other shader-enabled hardware from ATI or NV.) X-Plane 9’s current water shader is right on the bleeding edge of exceeding this limit. In fact, the water pattern is simplified for this set of GPUs to stay within the 96 instruction limit.
- Since the cards were really quite decent for their time, they are still in field and in use.
(This is a different problem from the Geforce FX/5 series…those GPUs were fully capable of running advanced shaders, they just couldn’t do it quickly. In practice this is better for an application’s developer, because users are more likely to replace a slow GeForce FX than a capable but internally limited 9700.)
This shows up in X-Plane as a pile of special cases…X-Plane 9 productizes 2.5 renderers:
- A no shader renderer for old GPUs and buggy drivers.
- A shader-based renderer for modern hardware.
- A special-case on the shader based GPU to meet the limits of the R300.
I’m not sure what we will do about this in the future. But as I go through the shader code, this special casing sticks out. We don’t usually special case X-Plane for a particular set of hardware, but in the case of the R300, it was for a while so widespread that it made sense to special-case for its limitations.
Let me set the record straight on this: you should not need to re-save an airplane to have it work in a newer version of X-Plane. If you have to do this, X-Plane is broken … please report a bug!
(In the case of 940 – there is a big fat bug – see the end of the post.)
Here’s a little bit more about what’s going on under the hood.
When Austin creates a new revision of the acf format (which happens in virtually every major patch), he handles backward compatibility with old aircrafts in one of two ways:
- He sets the default value of a setting to match the “unused” value in the old ACF file and sets this default value to match the legacy behavior. This naturally initializes all newly introduced functionality to its “backward compatible default” for old airplanes.
- Where this is not possible, he writes some conversion code that maps old ACF values to new ACF values. This mapping tries to re-create the old systems functionality as closely as possible.
This forward conversion code runs in two cases:
- When you open the airplane in Plane-Maker.
- When you open the airplane in X-Plane.
Plane-Maker will resave the plane in the newest format, with the automatic system updates in place. But this should not be necessary because X-Plane applies the same automatic process on airplane load. This is why you should not need to resave – X-Plane will do the upgrade “on the fly”.
Now about that bug…it turns out that 940 incorrectly updates 930 airplanes – the generator amperage is not correctly initialized. This is why 930 planes will run their batteries down in 940. (This bug was fixed in 941 beta 2, btw.)
What was strange was that, because of the way Plane-Maker’s code was structured, this code was failing in X-Plane but succeeding in Plane-Maker. This doesn’t happen very often (usually the code fails everywhere) but the result was authors noticing that their planes would start working if resaved in PM.
And that brings me back to the beginning of the post. If Plane-Maker can update the airplane but X-Plane cannot, that’s a bug! Please report it as such.
I want to make sure people realize that auto-update should work, and that resaving in Plane-Maker should not be necessary. Otherwise authors will start silently resaving their airplane instead of reporting the bugs, and we’ll never find them.
(Systems bugs sometimes only show up with a particular combination of systems settings. So while I do hope that we can catch all such bugs in beta, it is always possible that one peculiar model will induce a bug once the sim is released.)
I don’t usually link to non-X-Plane blogs, but I really liked this pair of posts:
//distractible.org/2009/11/05/top-10-ways-to-annoy-your-doctor/
//distractible.org/2009/11/08/top-10-ways-doctors-can-annoy-patients/
If you live in the US, you’ll definitely appreciate it…the lists are funny and yet have a seed of painful truth in them.
So I decided to try to create my own lists.
I am only tangentially related in tech support – Randy takes on most of the work with some help from Jack. Sometimes very weird reports get escalated to me. (And most of the “let the report sit for a week” comes from me not having time to dig in.)
Anyway, please take these with a grain of salt – they’re meant to be funny and exagerated. Most of our users are very, very helpful in tech support calls, despite the fact that, if you are talking to tech support X-Plane is already hosed. And Randy puts forth some amazing acts of patience in the face of some of the requests he gets. My hope here is only to show that there are two sides to the frustration in a tech support incident, and we’ll all be happier if we can see the whole picture.
Five Things You Can Do To Annoy Tech Support
1. Be As Angry as Possible
Threaten to switch to Microsoft Flight Simulator. Drop the F word a few times. KEEP CAPS LOCK DOWN FOR THE ENTIRE EMAIL. Tech support definitely responds better to users who are angrier – you don’t want to get sub-standard service because you were too nice, right?
2. Omit Information
If you have a second graphics card made in Kazakhstan, over-clocked and running hacked drivers you got off of the pirate bay, don’t tell us. If your computer regularly catches on fire, be sure not to mention that. Did you recompile the Linux Kernel yourself after letting your pet monkey edit the thread scheduler? It’s best we not know.
Extra credit: report a truly bizarre problem, provide no details on your customized configuration, wait a week and tell us how you fixed it by removing a third party program that “enhances” sound or graphics. Priceless!
3. Don’t Include Past Emails In a Thread
Be sure to delete any past information from your email. Change the subject of the email so we can’t tell what the original issue was. If you have more than one email, send replies from different addresses. A perfect reply would be “That didn’t work” sent from an email address that you haven’t used before, without your name included.
4. Email the Last Person You Talked To.
If you just finished up sorting out a shipping problem with the shipping guy, ask him how to create a plugin. If you just got info from the developers about UDP, ask them why your credit card was charged the amount it was charged.
5. Bring Up New Issues In the Middle of Old Ones.
To do this just right, wait until the thread between you and tech support is pretty deep into the meat of a complex issue. Then throw in another paragraph about something else that’s gone wrong. To perfect this technique, try to pick a new problem that the person who you are emailing with isn’t equipped to handle (see point 4) and keep the report vague (see point 2). You can repeat this technique to stretch out a tech support incident indefinitely.
Five Ways Tech Support Can Annoy You
1. Make the User Reinstall the OS
Reinstalling the operating system fixes approximately 0% of user problems, but it takes a really long time, and is almost guaranteed to screw something else up, usually something that wasn’t broken and isn’t related to X-Plane. If a user is a little bit annoyed, this is a great way to pour gasoline on the flames.
This is really a special case of the general strategy “ask the use to do something time consuming, annoying, and unlikely to help.”
2. Forward the User a Huge FAQ, None of Which is Relevant to the Problem
Everyone likes form letters and impersonal service. The FAQ should be badly written, badly formatted, confusing to read, and preferably not accidentally contain the real solution to the problem. If the solution to the problem is in the FAQ, don’t tell the user where in the FAQ to look.
3. Wait a Long Time Between Replies
Tech support incidents are like fine wines – they get better with age. To allow the user’s annoyance to bloom into a finely honed rage, be sure to let each email ‘sit’ for a week before replying. This works especially well if your response is just to ask another question, setting the user up for another week’s delay.
4. Blame Some Other Component
The modern PC is built by approximately 600 different vendors. Blame one of them. The beauty of this strategy is that it is one that can be used by every vendor who provided software or hardware for the PC. Also, because quite often the problem really is with another component, you can claim this with a straight face.
Tip: blame the graphics card maker – ATI and NVidia do not have the resources to pursue every complaint that an over-clocked graphics card running the latest patch to some simulator written by two guys in their bedrooms crashed with the drivers visible somewhere in the callstack. Put the blame on the GPU makers – they don’t have the resources to refute you, no matter how bogus your claim.
5. Forward the User’s Issue Around the Company Until It Gets Lost and Dropped
Everyone in the company has to be in on this strategy for it to work – if one of your idiot coworkers actually solves the user’s problem, well that defeats the purpose. This strategy can be combined with (3) and is sort of a riff on (4) – once the user complains that they got dropped, blame everyone else in the company for the mis-communication.
First, I want to thank the X-Plane users who have gone through piles and piles of regression tests to isolate some of the peskier bugs. Often a bug only occurs on hardware that we don’t have in-house, so this detailed reporting is a huge help in shipping a program that has to run on a huge number of configurations.
Sometimes a user will offer to work around a bug by changing an add-on, or just dropping the add-on. But…I have learned the hard way: never ignore a bug you don’t understand.
First, the bug might run much deeper than the reported use. Perhaps the bug is actually affecting dozens of other add-ons.
If we don’t understand the bug, how can we say “this is so unimportant that it can be ignored”?
Now some bugs, once diagnosed, may prove to be not worth fixing. But…until the bug is fully understood, we have to take the time to dig in. We can’t just give up because the bug seems unimportant.
I’ve had a few inquiries about environment maps, normal/specular maps in more places than just OBJs, etc. The short answer is: a lot of these future rendering engine enhancements are good ideas that we like, but we have other features that are already partially implemented that we need to productize first.
In particular, one of those half-finished features not only could use to be shipped, but also affects the way just about every other rendering effect works. So better to get these features finished first, and build new effects within the context of these “new rules”.
Here are some of the ideas that I’ve heard kicked around:
- Environment maps on the plane – I like it, it’s not that hard to do, and the framerate hit could be ramped up and down with detail. See above about finishing other features first.
- Next-gen texturing on runways (wet runways, environment maps, bump mapping) – I like all of it. The runways really need to be addressed comprehensively, not piece-wise, in order to find a rendering configuration that meets our scalability and efficiency needs. (In other words, we can’t just burn tons of VRAM on the runways, and we need a way to render them that works on low and high end computers with one set of art assets.)
- Normal mapping on the ground. I like it, but I wonder if this isn’t part of a bigger idea: procedural texturing on the ground. E.g. if we want to add detail on the ground, can we add it with multiple layers at different resolutions with a shader adding yet more detail.
Generally speaking, it’s not that hard to push a feature out from one part of the rendering engine to the other. For example, normal maps on facades (if anyone cares 🙂 would make sense and be a fairly trivial change. I hesitate on polygons only because the ground might require something a little bit more sophisticated.
Going Crazy With Choices
Here’s a straw man of why polygons might be different: draped polygons can’t have specular shininess right now, and I don’t think anyone is complaining. So it’s a bit of a waste to use the alpha channel of a normal map for shininess. Perhaps it could be used for something else like an environment mapping parameter.
Hrm…but we would like to have environment mapping on airplane objects too someday. Well, we could go two ways with that. We could just use the alpha channel for shininess and environment mapping…not totally unreasonable but it wouldn’t let us have a glossy non-reflective material, e.g. aluminum vs. shiny white paint.
Math nerds will realize that the blue channel in normal maps can be “reconstituted” from the red and green channels by the GPU (at the cost of a tiny bit of GPU power). That would give us two channels to have fun with – blue and alpha. Perhaps one could be shininess and one could be an environment mapping material.
Well, shininess is still no good on the ground. But…perhaps that would be a good place to store dynamic snow accumulation? Hrm…
My point of this stream-of-consciousness rant is that the design of any one rendering engine feature is heavily influenced by its neighbors. We’ll get all of these effects someday. If there are features that are really easy, we can get them into the sim quickly, but the only obvious one I see now is using bump maps on other OBJ-like entities (which at this point would mean facades).
In response to so many bug reports I say:
Please temporarily remove your third party scenery, airplanes and plugins and retry. Do you still see this bug?
And in response I get:
No – the bug was with XXX. Thanks!
But…but…but…we’re not done yet! The point of removing the third party add-on is divide-and-conquer: to see if the bug is in the core of X-Plane or is caused by an X-Plane-add-on interaction.
If the add-on used to work but no longer does, I want to know where I can get the add-on, who created it, and when it used to work, so I can investigate further! If an add-on stops working because X-Plane was updated and the add-on is not doing anything naughty, that is still an X-Plane bug.
So please: don’t give up on the bug just because removing the add-on makes it go away…there is still more to debug!
