Two bugs we resolved today …
One user couldn’t use our installer – it always failed on the file FA-22_cockpit_OUT.obj.zip. After reporting this to us, he figured out that parental-access controls in his router were banning the file due to a certain 4 letters in its name. I feel lucky that the user diagnosed the problem because I never would have figured that one out!
Also we’ve had reports that X-Plane 832 is slower than 816 when used in a two-monitor instructor/visual configuration. Finally we figured out what this was – a bug in my code: I programmed X-Plane to reduce CPU usage when a “dialog box” is up (the instructor station is one such dialog box) to play nice with other programs. But of course in two monitor mode the dialog box is up and you’re flying, so reducing CPU usage kills framerate. This will be fixed in 835.
What’s funky about this second bug is the reason why Austin and I could not reproduce it: the bug only happens if you do not touch the mouse! Austin and I, while trying to debug, would mouse around and click on all sorts of things, and this would hide the bug. Real customers, in real applications, don’t touch the mouse while doing commercial training. (UI programmers who understand how mouse cursor updates are done in Win32 and Mac OS will understand this behavior.)
If there’s a moral to this post (and there probably isn’t), it’s that bugs are hard to catch and reproduce because there can be system differences between users and developers that are very subtle – assumptions so basic that they don’t get questioned.
Last night I was looking at what it will take to put animated cars back on the roads in X-Plane. (I can’t say if or when they will come back…that’s a subject for another blog entry.) I found something that’s going to be a little tricky.
Simply put, the road data we use in the US (Census TIGER data) does not contain directional information. It might tell us a street is one-way, but it doesn’t tell us which way it goes!
In fact, the data doesn’t even say which highways connect to each other! The road data is “flat”. If two highways cross each other we see this as a four-way intersection. In the original US DSFs the roads were built with this flat pattern – sometimes you’d see a 10-way intersection of limited access highways.
In an attempt to clean up the highways for the global render I wrote some code that attempts to evaluate junctions for traffic flow. Basically it knows that two limited access highways at right angles cannot intersect and it moves one up in the air as a bridge. This process iterates a bit in an attempt to make a junction that is fully real-world navigable.
One advantage of processing the roads in this manner is that often it will remove bogus junctions, allowing us to continue straight roads. This both reduces file size and improves the look of the roads. (Our road junctions still look terrible – a throwback to a time when road-creation paused flying and therefore speed was at a premium over visual quality.) Unfortunately this “layering” code sometimes goes haywire and does some pretty strange things:
- A highway may be put in a high layer in one junction and a low layer in an adjacent one; the result is a highway ramp that “dives” down at a steep angle to try to meet all of its crossing restrictions. The layering code isn’t smart enough to find optimal junction ordering when there are many junctions nearby.
- Sometimes the number of layers just gets huge, creating a massively tall junction. (But then, in Los Angeles perhaps this is realistic!)
How does this relate to cars? Well, what is significant here is that the algorithm does not consider directionality of one-way roads since we do not know it. Furthermore, the DSF optimizer, because it was written under the assumption that we really have no way to know which way a one-way street is going, will flip the direction of one-way roads to reduce file size. (Sometimes changing the direction of a road segment allows for tighter file compression.)
The good news is: DSF files do have a notion of direction, so if we ever get better road data, the sim and file format will be ready. The bad news is: the current DSFs have a road grid that may be similar to driving in Boston.
(As a side note, I grew up driving in Boston, so none of those rules seems even slightly unusual to me. What did surprise me was: when I moved to Washington, DC I found the rule that the cheaper car has the right of way turned on its head. I’ve never been cut off by more reckless BMWs and Mercedes in my life. You can speculate about what this says about the culture of our capital.)
Clearly X-Plane’s car engine needs to address this in some manner. There are two cases that are tricky: one way roads that end at the destination of other one-way roads (leaving the car with no legal route) and one-way roads whose origin is not connected to anything but the origins of other one-way roads (giving cars no way to reach the road). The later case is not really a problem – it just means the road will have no traffic; the later is more of a problem as it requires the car to do something illegal to get out of the situation. I am still looking at what we can do about this but my guess is that cars are going to have to employ some Boston driving to get around the X-World.
Here’s a short list of some of the features of OBJ8 you should never use:
Don’t use ATTR_no_depth. This attribute should probably never have been invented and will be removed someday. There is no situation I can think of in the sim where this attribute is a good idea. (If you do need it, please let me know – tell me why!) Use ATTR_poly_os to put your object on top of a runway or the ground.
Don’t use ATTR_ambient_rgb or ATTR_specular_rgb. These attributes are no-ops and have never worked.
Don’t use ATTR_diffuse to change the color of your object. Use a small solid-colored patch of your texture – it’s much faster. The only reason for ATTR_diffuse is in conjunction with ATTR_emission. I will explain this special usage in another post.
Don’t use ATTR_shade_flat. You can represent any shading, smooth or flat, with ATTR_shade_smooth. In AC3D this means clicking the ‘smooth’ button and using the crease-angle to create the appropriate look for your object.
Don’t use ATTR_poly_os with huge offsets. Even if you stack several layers of offset polygons on top of each other, you do not need to make each offset larger and larger; the offset only needs to be large enough to put the polygon over the base terrain. An offset of 1 or 2 is almost always adequate.
A few other warnings: Don’t rely on your objects being drawn in a certain order in the sim – this will change! Don’t put lights in the middle of your objects – when textured they will only look right at the end of the object.
I sometimes get requests for two features:
- The ability to use more than one texture per OBJ.
- The ability to not use a texture per OBJ.
My answer to both is the same: one texture per object forever!
The issue with the first request is simply framerate – see my previous post on the “crayon rule”. X-Plane has to do a certain amount of setup work for each OBJ it draws. Therefore we have two ways to implement multiple textures per object.*
If we want to obey the crayon rule, we would draw the parts of the objects that use the first texture first for all objects, then change textures once and draw the second texture. But this isn’t so good – if you are using polygon offset or translucency or any other feature that requires the object to be drawn in order, those features won’t work. Also we do a certain amount of setup work per object – we now have to do that work twice! This is about the same as doubling the number of objects. Ouch!
Or we can disregard the crayon rule and change textures once per object. That would be performance death . No optimization has more impact on framerate than minimizing texture changes! This is why X-Plane keeps moving from multiple textures to one texture – in both airplanes and objects! We do it for speed!
So that’s why we require one texture. But why not zero textures? Well, there are a few issues with allowing untextured objects:
- The request for an untextured object almost always includes a request to use colors on the object instead. This would actually possibly be slower, not faster. Color information is twelve bytes per vertex, while texture is only eight. Each vertex per object is now 32 bytes, a wonderful power of 2 that happens to be very efficient on certain modern hardware. With color we hit 36 bytes. Ouch! Remove the color (not useful anyway) and we have 24 bytes. (Why not use 4-bytes instead of 12 per color? The hardware works better with color as floating points, not bytes – surprising but validated by performance testing.)
- X-Plane would have to take time off from drawing to turn texturing off and then turn it on again. So unless we’re going to stop texturing for MOST of the sim, this “state change” hurts us.
- The actual process of drawing with a texture is basically free on modern hardware. Either the chip has dedicated hardware to do the texturing that would be idle otherwise or, if it’s a pixel-shader chip, the ability of the card to texture is so far ahead of what X-Plane needs that the issue is moot.
So we don’t allow color-instead-of-texture because it would be slower! Instead simply make a small “color palette” in your object texture and use that part of the texture stretched over your object to make your colors.
We do allow you to tint your object with ATTR_diffuse, but do not do this! The performance of the diffuse attribute is way worse than using a bit of your texture for color. ATTR_diffuse only has one legitimate use – paired with ATTR_emission to allow for the creation of objects that are self-lighting in low-light daytime conditions. I will post about this in a future entry.
* For OpenGL geeks: technically with pixel shaders you could bind multiple textures simultaneously and write a shader to select one, perhaps by mapping ST coordinates from a virtual texture space. So perhaps someday when we require pixel shaders for X-Plane and they are ubiquitous we can re-examine this.
I’m sitting in a lab at Apple with Austin; that screenshot is X-Plane 835 running on a 2 ghz DuoCore Intel Macintosh in a 20-inch iMac, 512 MB of RAM. That’s 25 miles of visibility, insane roads, mega-tons of objects, 30 fps.
One reason why X-Plane runs so fast on the new iMacs is that the graphics card is an ATI X1600 – that’s a big step up for the iMac.
When will the Universal X-Plane be available? I can’t say yet but we are making progress!
I’ve been getting a lot of bug reports where X-Plane quits with a message about a missing .ter file. The problem is almost always installation related:
The only X-System folders that can support global scenery DSFs are the ones that started from the X-Plane global scenery DVDs.
Therefore if you want to use X-Plane 832 with global scenery, you must first install X-Plane 821, then run the updater to upgrade to 832.
If you simply start with 832 and try to copy files then you may miss some of the artwork and the missing .ter file message will appear. I will try to improve the error message with some clear instructions in the future, but only a proper install can remedy the situation.
There may be new apt.dat features in a future version of X-Plane. I really can’t comment on what will be in future releases. But I can ssure you this much:
You will not have to change existing scenery to work with the new version of X-Plane! No upgrade will be required.
When we introduced DSFs we continued to support ENVs; when we introduced OBJ8 we continued to support OBj7 (and OBJ2!)…whatever we do to the apt.dat file, whether it be a new version or just new features, X-Plane will continue to work with existing apt.dat files as it always has.
Austin announced that X-Plane runs on Intel-based Macintoshes, and Steve Jobs announced that the Intel-based iMacs are 2-to-3 times as iMac G5s. But here’s the catch:
X-Plane 832 is not a universal binary – it has no native Intel code in it. So it runs under emulation on the Intel-based Macintoshes, which really hurts performance. So do not expect god framerates on Intel Macintoshes in 832 – you’ll have to wait for a universal binary version of X-plane for that.
Hopefully I’ve dispelled some rumors. This won’t dispell anything, but..I am writing this post from Cupertino.
Quick note: the region system will go in a later version of X-Plane, not 8.30. It’s coming soon though! I will try to post a spec soon…I am way behind on my documentation.
Anyway, the “Crayon Rule”: the worst thing you can do for frame-rate in X-Plane is change textures.
The name comes from a conversation almost three years ago when Austin explained to Sergio the cost of changing textures like this: drawing in OpenGL is like drawing with a crayon. Changing textures is like going back and getting a different crayon. It’s a lot faster to do all the red in your picture and not have to change between red and blue crayons all the time.
The crayon rule continues to hold true and it’s understandable why:
- The texture you are using to draw is usually in VRAM. But the texture you are changing to could be in physical RAM (and must be transferred to the card), or worse it could have been paged out and must come from disk.
- The drawing pipeline must be stopped and reconfigured with the texture change.
- These things involve the host machine and CPU. Graphics cards are getting faster but CPUs are not – at least not much faster. So while the new graphics cards will be able to draw twice as many triangles, they will not be able to change textures twice as fast. In other words, we’re stuck with the crayon rule for a long time.
The crayon rule is why Austin and I insist on using one texture per object, and always recommend that you use a few large textures rather than many smaller ones. Fewer textures means fewer texture changes.
So the big things you can do as an author to obey the crayon rule are:
- Use fewer large textures instead of many small ones.
- Group your polygons that use the panel texture together – switching to the panel texture is a crayon change.
The crayon rule makes it more difficult to make scenery, but that extra effort has a huge payoff in framerate.
I like to pretend I am a responsible software engineer, but…I am not. X-Plane is RC2 right now but I have just hacked in the “region system”.
Put simply, the region system allows you to make collections libaries of objects (in scenery packages) to customize the look of the scenery, that only apply to a region of the world, rather than globally. This lets you make sure your custom Igloos are not used in the Bahamas.
The reason for this impulse is that we are setting the stage for non-US global scenery objects. I will try to post some docs soon and a tutorial showing how to customize X-Plane’s objects.
You may have seen a crpytic message like “async task jammed” or “thread timed out” with X-plane 8.20 or 8.30, after which the sim quits. Here’s what’s going on and what we’re trying to do to fix this.
X-Plane offloads part of the scenery load process to a second processor. (If you don’t have a second CPU, the first one does the work, but the operating system switches between the two tasks automatically.) For example, X-Plane “outsources” the work of turning a road (which is just a line segment in a scenery file) into real 3-d roads using the second CPU.
X-plane puts a request for the roads out to the second CPU, and then every frame it checks to see if it’s done. If the roads are processed, it can then send them to the video card and go on to the next thing to be done. If the roads are not finished – no biggie – it just checks again on the next frame.
Now here’s where the trouble begins: sometimes X-Plane needs the outsourced work to be done immediately. For example, when we load new scenery we have to recycle the memory from the old scenery. This requires the work of building the roads on the old scenery to be done! When this happens, X-Plane waits (up to 2 minutes) for the background work to finish.
If after 2 minutes the work is not done, X-Plane gives up and puts up a cryptic message like “async task jammed” or “thread timed out”. (In 830 beta 11 the message should be less cryptic.) I must admit: I do not understand how this condition happens; it has never happened on my machine or Austin’s.
Here’s what we’re doing to alleviate the problem: in X-Plane 830 you can turn off the “load scenery in the background” option. When this happens, rather than outsource jobs to the second CPU, X-Plane will do the work immediately. This will have two effects:
- There will be slight pauses as you fly while x-plane stops and builds roads, etc.
- There is no way that we can get stuck waiting for the second CPU, because it won’t be used.
Besides turning off this option, you can reduce the amount of stuttering (either with background loading on or off) by decreasing the amount of roads and objects. For example, on my G5, the sim flies pretty smoothly on “default” settings with background loading on or off. But on “insane” settings the sim is still smooth with background loading on and stutters visciously with background loading off.
I hope to get the background code to be more reliable in 840, but in the meantime you should be able to fly (etiher with or without background loading) in 830 with reasonable results if you set your rendering settings appropriately.
