Chris and I bashed at the developer site (that is, this site today); sorry about any dead URLs – things should be more stable now.
I have (finally) posted a new build of XPTools; there are two items of note:
We’ve been making steady forward progress with the scenery tools over the last few weeks; finally there is some stuff to see.
First: we’re moving the scenery tools into WordPress. All of the new tools and docs are hosted at developer.x-plane.com; over the next few weeks I’ll try to make the theme less bare-bones. As I update documentation to version 10, I will move it here.
What about the Wiki? We are retiring it as our content management system. MediaWiki makes everything possible, but it makes nothing easy. Well, that’s not true – it makes it easy to collect link spam.
We can possibly keep the Wiki open on a smaller scale for developers, but my preference is strongly in favor of a mailing list.
The first binary tools release is a final version of WED 1.1. If you have been using a WED 1.1 beta, get the final one – it fixes some bugs and adds control over object density in custom scenery.
I will have binary releases of the base scenery tools pack including DSFTool and a WED 1.2 preview over the next few days.
The DSF specifications are updated – see here. The DSFTool binary will let you create version 10-style DSFs. I will continue to update the documentation over the next few weeks.
The scenery tools source code is now natively hosted in GIT – previously it was hosted in CVS and was bridge to GIT for public preview; the bridge broke down during v10 development a few months ago.
Chris and I are now using GIT internally, putting us in sync with the Linux community; if you want to work on the scenery tools or modify them beyond patching, you can clone the repository and merge in changes.
10.04 Beta 4 was released last night. The changelog can be found here as usual. Remember, you have to run the updater with the box checked that tells it you WANT to participate in the beta testing in order to get this version. Bug reports go here!
I originally meant to write a post about deprecation and feature evolution a while ago when I accidentally (and temporarily) broke the way textures are located in scenery packs and put it back. In X-Plane 10.04 betas we are now fighting with apt.dat validation, so the topic is perhaps again relevant.
X-Plane is continuously patched. We put out about 4 major patches a year plus numerous bug fixes for the life of the product. One of the implications of this is that even if we are planning to drop a feature, we can’t just go from “it’s there” to “it’s gone” in a patch, because users get grumpy if their content doesn’t work after a free (and hard to avoid) update. No one wants to pick between getting the latest perf tuning and having working content.
The way we try to cope with this is by easing in sim changes that are necessary in the long term but disruptive in the short term. We try to make the stop light go yellow before it goes red. Here are some examples:
I originally meant to post this back in 10.02 betas when I removed the code to handle the old (X-Plane 7) texture file location system and discovered that a bunch of otherwise current scenery packs still use it. The new (art asset relative) system has been available since version 8 (that’d be 7 years) but X-Plane never warned that obsolete packages were doing something that was going away, so I put the code back with a warning.
We’re running into another version of this with custom airports that put ramp starts, windsocks and beacons in strange places. I suspect that some of this represents accidental bad data and some of it represents intentional authoring – attempts to use X-plane in a manner that we didn’t quite expect. The need for data validation comes from the ATC system – with ATC and AI planes using apt.dat, weird data causes significantly more problems than it used to.
I don’t know what the final outcome will be – we’re still looking at some of the apt.dat files that fail. But my expectation is that:
X-Plane 10.04 beta 2 is out now. Release notes here; like all betas, you’ll need to run the updater and click “get betas” to see this – it won’t auto-update until we go final. Report bugs here. Do not post bug reports on the blog.
A Temporary Outage
The beta was actually ready last night, but iWeb, one of our hosting providers, had a fairly major outage in the data center that the update servers are hosted in. The servers are now back on the net, so updates should work normally.
Our overall strategy for updates is to spread out the update pool over a few providers so that temporary outages only reduce capacity, not availability. In the short term, however, we have a lot of capacity at iWeb because their pricing is really aggressive and we had to push a lot of bits for the X-Plane 10.0 roll-out. This is the first outage they’ve ever had; if we see continued reliability problems we’ll get more aggressive about rebuilding our demo/update mirror.
Controlling Scalability
With 10.04 beta 2, the attached objects on the parking garage facade now shows more or less cars based on the object rendering setting. This means that if you have a computer that cannot instance and you run with lower object density, the KSEA is going to be a lot less expensive than it used to be. The facade engine also allows more control over object placement, which Tom was able to use to further reduce overall object count.
Our plan is to include this kind of object density control on all parts of the airport lego brick library so that the amount of “ramp clutter” is variable by user settings. Chris and I were in the terminal at KCLT waiting for a flight and looking out across the ramp, one thing was clear: there’s a ton of crap floating around the ramp at a major airport. With variable density objects, authors can put down major “mini-scenes” (e.g. a jetway plus docking position) and we will be able to scale from the minimum objects to make the sim functional all the way up to a ton of clutter for machines with good instancing performance.
What’s On the White Board
I have a white board on my wall with some of the highest priority bugs scribbled all over the place; I like it because it gives me an immediate view of what can’t get ignored, and when I am on the phone I can simply write down an issue that someone brings up. So far the white board issues tend to be in a few categories: crashes and serious bugs, performance, authoring platform features, and rendering artifacts.
With 10.04 beta 2 I finally had a chance to start killing off artifacts; the taxi lines should (previously they’d get all bent out of shape when they went around sharp curves) and the water shader is better. (The water shader still isn’t done, but there should be no more hard lines in it.)
At this point rendering errors and performance still dominate the board and my next few weeks of coding.
Last week I received a number of emails about the technical details of v10 scenery. My short answer has always been “please wait for the docs” – that is, I am trying to spend more time writing docs quickly for everyone and less time writing one-off emails. I was hoping to be further along in the doc process, but at least I can say I had a little bit of time to work on WED last week.
So I apologize to everyone who is stuck waiting, but we are getting closer to a useful package of “stuff” (that is, tools, source code and docs). I think you’ll need both tools and source to take advantage of X-Plane 10.
I spent a few hours last night working on X-Plane’s facade engine – that is, the part of X-Plane that builds buildings, fences and structures from their shape traced on the ground. Besides fixing some v9 compatibility bugs, I had a chance to add in a few last features that didn’t quite make the 10.00 cut.
I used Tom’s “classic”-style parking garage (you’ve seen it – it’s the one used at KSEA in the demo area!) to test some of my work; some of the work on the facade engine illustrates three different ways to optimize a program.
Be More Efficient
The first way to optimize is to find ways to accomplish the same work in less time – that is, to write more efficient code. This is certainly the least risky form of optimization: there’s no trade-offs, you just get more for your hardware dollar.
In the case of Tom’s parking garage, after putting geometry in place for the roof of each floor (to fix a rendering bug) I noticed that we were building the mesh for each floor twice – once for the roof of the level and again (but facing in the opposite direction) for the floor of the next level. I added some code to the engine to recycle the mesh, but create duplicate indices; now two-sided roofs in facades use about half the vertex count.
There are still a number of places in X-Plane 10 where we can be more efficient. But rewriting code to be better takes time; these performance changes will come out incrementally, and any one change may not be that noticeable. What is important though is the net effect of improving efficiency over and over again.
Cheat!
This sounds negative, but when it comes to computer graphics, cheating is preferable. Clouds behind the panel? Just cheat – don’t draw them! Trees too far away? Use a billboard! No airplanes in the game? Use a painting for the sky and don’t bother with 3-d clouds. What I am referring to here is finding ways to deliver the appearance of eye popping graphics for significantly less work. This kind of optimization is critical to making a competitive rendering engine.
In the case of Tom’s garage, the facade engine puts the same number of cars on every level of the garage. This is hugely wasteful; while you might want cars on the roof, you don’t need very many on the lower levels. The lower levels are only visible from the side, and from the side even a few cars are enough to create the appearance of a non-empty garage.
So I am adding to the facade engine the ability to specify mid-level objects and roof objects separately; Tom can use that to “thin out” the objects in the garage. The result will be a big reduction in OBJ count for almost no visual quality loss.
I think there’s a lot of fine tuning we can do like this to make sure that we spend our hardware budget only on things that really matter most.
Scale Down
This isn’t really a type of optimization, but it’s important to make sure that the rendering engine can throw visual detail out to meet the requirements of older computers. One of the problems with X-Plane 10 is that some parts of the rendering engine only have one mode: “maxed out”.
10.04 beta 2 will allow facades to specify attached objects (like the cars on the parking garage roof) that decrease in density with the rendering settings. The result will be a full garage when objects are on “insane” but a mostly empty one with just a few cars when objects are on “default”.
Tom and I are working to put this kind of object density control in all of Seattle (as well as the airport library it uses and the rendering engine components that the library uses); the result should be a Seattle demo area that becomes significantly cheaper when the rendering settings are dialed back just a bit.
There are three nasty bugs haunting X-Plane 10 that are on my plate right now; these are serious bugs that pretty much ruin the X-Plane experience for those who hit them. They are marked in the bug base as “hair on fire” – the highest priority we have for a bug. (Yes, the bug base literally has a priority category called “hair on fire”!)
Every morning I have to ask “can I close any of my hair on fire” bugs, but these three have been nasty enough that a status update is warranted, even before we have a fix.
Bug 1: Poor Performance with ATI Hardware on Windows
A code path used by the cars and clouds is stalling in the ATI driver on Windows. I don’t know why this is, but we are trying to follow up with ATI directly on this. The result is pretty unfortunate: if you have any cars (even “siberian”) or any clouds (even scattered) framerate absolutely tanks.
As it turns out, if you set weather to clear and turn off cars completely, ATI performance on Windows is pretty good! Obviously that’s not a useful work-around, but if you have ATI hardware and are wondering what kind of fps you’ll get once we fix this, those are the two features to turn off. (I suspect rain may hit this code path too.)
As with all driver-related problems, I must point out: I do not know if it’s our fault or ATI’s, and the party at fault may not be the party to fix it. Apps work around driver bugs and drivers work around illegal app behavior all the time…that’s just how it goes. Philosophically, the OpenGL spec doesn’t require any particular API calls to be fast, so you can’t really call a performance bug a bug at all. It’s ATI’s job to make the card really fast and my job to speculate which subset of OpenGL will cause the card to be its fastest.
I have a Radeon 7970 and will write up some of my performance findings later. Unfortunately with the clouds affected by the problem code path, it’s hard to see how the card will really do.
(OpenGL nerds: the problem is that glBufferData with an in-flight VBO and a NULL pointer is stalling. This particular use of glBufferData is meant to tell the GPU that we want a new fresh buffer, with the old buffer being asynchronously discarded once the GPU has completed drawing. This “orphan the buffer” pattern is the standard way to do high performance vertex streaming in OpenGL prior to glMapBufferRange.)
Bug 2: Hang on Start with NVidia Hardware (Windows)
A small percentage of Windows users get a hang on start with NVidia hardware. We’ve had this bug for years, but it’s more of a problem in version 10, where command-line work-aronuds are not available. We have identified four third party programs (Omnipage, Matrox Powerdesk, Window Blinds, and TeamViewer) that all seem to cause the problem at least some of the time, but some users have this problem with no obvious cause. There isn’t a hardware or driver pattern to it either.
Until recently, we could never reproduce this bug in-house; it was only through the patience of users that we were able to characterize exactly what was going wrong via copious logging. But a few days ago we got a break: a user reported Window Blinds to be hanging X-Plane, and unlike before, Window Blinds hangs our machines up too! (That’s a huge step forward in getting a bug fixed.)
At this point we’re working directly with NVidia to try to find out what’s going on, as the hang occurs inside their driver code. The same rant applies here as above: until we know exactly what’s going on, we can’t say whether this is a driver bug or an app bug.
Bug 3: Crashes Near Munich on Linux
Linux users see crashes near EDDM with memory exhaustion, even at very low rendering settings. This bug appears to be Linux specific and we do now know what’s going on.
For reasons that I cannot yet fathom, the math deep inside the ATC airport taxi layout auto-generator goes haywire on Linux, resulting in an infinite loop that allocates memory, eventually crashing X-Plane.
What’s infuriating about this bug is that it is a Heisenbug: all of my attempts to diagnose or “prove” what is going on causes the math to stabilize and the app to function. The bug does not appear when the optimizer is off; it certainly looks like an optimization bug but I hate to call foul on the compiler guys without a ton of evidence. It’s just too easy to write C++ floating point code with sharp edges.
I don’t have a good time frame for the two video card related bugs; I consider us lucky to be able to communicate with ATI and NVidia at all. I don’t have insight into their development process and if I did, it would be covered under our NDA. The only thing I can say is that these are top priority bugs.
The Linux bug is still in my court; I last put it down in a state of GCc-induced rage. I believe I will be able to get it fixed some time during the 10.04 beta process.
Final unrelated thought: all of the GPUs in my office, listed in performance order…
GeForce 5200 FX, GeForce 6200, Radeon 9600 XT, Radeon 9700, Radeon X1600M, GeForce 8800 GT, Radeon HD 4870, GeForce 285 GTX, Radeon HD 6950, GeForce 580 GTX, Radeon HD 7970
A question came up in the comments to my previous post:
Any suggestions on the best way to edit DDS files on a Mac?
The answer is simple: do not edit DDS files!!!
The problem is simple: DDS is a lossy compressed format. Every time you go from an uncompressed/losslessly compressed format (like PNG, TIFF, or BMP) to a lossy format (like DDS or JPEG) you are going to lose a little bit of image quality. So if you re-open your DDS, edit it, and resave, your graphics program is going to make a round trip and reduce quality.
Therefore the work flow should be:
So each time you work, you work from a lossless or uncompressed master (like PNG or a gimp/photoshop file) and then re-export to DDS.
Thanks to Captain Peter Hager for sending me his A340 as a nice test case for texture compression. Peter (like many aircraft authors) has never been very happy with the artifacts introduced by DXT compression.
First the basics: DXT1-5 are a series of texture compression formats. They store the same number of pixels in a smaller number of bytes via clever encodings of the pixels.
When your image has no alpha channel, the savings from DXT1 compression are 8:1. (Modern graphics cards store RGB images in 32 bits for alignment, so the 4-bits-per-pixel DXT encoding is an 8:1 savings, not the 6:1 you’d expect on paper.) RGBA images take 8 bits per pixel, for a 4:1 savings. That’s as much as reducing the image size by a factor of 2 in each dimension.
DDS is an image format that contains a compressed image. DDS files load faster (because the image is already compressed), look better (because the image is compressed using a slower, higher quality process before sim load) and let you pick the DXT algorithm choice. (In some cases, the choice between DXT3 or DXT5 matters!) However, once an image is compressed into a DDS file, the pixels lost in compression never come back – DXT is a lossy compression format! (This is just like saving a JPG as a PNG – if the JPG quality was too low, the PNG will contain the same blocky artifacts that the JPG had.)
Compression is useful for more than just saving VRAM – because the amount of data needed to texture a given pixel is smaller (4 bits per pixel instead of 32) compressed textures can speed up framerate if the memory bus between the GPU and its own VRAM is congested.
Now let’s look at some pictures.
This is the Swiss Air livery at extreme res, uncompressed, sourced from a PNG. This is our reference image for how good the original image can look before we start trading off quality for VRAM.
This is the same image, extreme res, texture compression on, sourced from a PNG. It just looks awful. The graphics card has done the compression, and it has done a fast job, not a high quality job. (In the driver’s defense, X-Plane asks for a fast compression rather than a high quality one to avoid slowing the sim down.)
This is also compressed!!! But in this case the livery was pre-compressed using XGrinder/DDSTool (which in turn uses libSquish). DDSTool sets libsquish for maximum quality and then takes quite a while to grind each texture, but the results are better use of those compressed bits. (DXT is a lossy compression, so more compute time can find a better use of the finite quality available in the compressed texture.)
Finally, this is the uncompressed texture at “very high” res – that is, taken down one resolution level. Compare this image to the compressed one above. For an alpha-blended texture, the trade-off is one “size” increment (2x smaller on both sides for 4x VRAM savings) vs. compression. I submit that the compressed version is a better way to save VRAM than down-sizing.
In X-Plane 9 you had to make a trade-off: ship a compressed image in a DDS file and have the very best look be extreme res compressed or ship a PNG file, and have the texture look terrible when texture compression is on. You can see from the above images what the options looked like.
In X-Plane 10 you can do both. If you ship a DDS and a PNG file, X-Plane will load the PNG file when texture compression is off and the DDS file when it is on. The result is a pre-compressed texture for users who need texture compression and an uncompressed texture for those who can afford the VRAM.
One last note on texture resolution for users: most drivers use virtual address space to load textures, and thus texture memory eats into X-Plane’s 2/3/4 GB address space limit. Therefore with high VRAM cards (e.g. a 3 GB card) it probably isn’t possible to absolutely max out texture res until we go 64 bit. For example, on the Mac (where the driver uses a lot of address space and we only get 3.5 GB) running the sim on extreme + uncompressed texture res will almost always run you out of memory.
One thing I have noticed from looking at user’s settings and our own machines is that the textures resolution selector goes in pretty big jumps. Each resolution change cuts VRAM by 75%, so if you are over VRAM (and having performance problems because of that), the next setting down will probably leave VRAM unused.
So I have a todo item to look at putting incremental texture resolutions in that derez the scenery but not the airplane, for example, to hit some in-between points. In that context, we may be able to support mixed compression, where some airplane elements remain uncompressed, but the scenery is compressed.
The recommended practice for developers remains the same: