This is an authoring “gotcha” in X-Plane 10:
Do not use a degenerate UV map if you are using bump maps. If you use both techniques, your object will have black splotches in HDR mode.
Let me define what I mean:
A degenerate UV map is a UV map where the “U” and “V” axis (that is, the horizontal and vertical axes of your texture) are not moving in different directions across the face of a single triangle.
The most common case is when all three vertices of a triangle are mapped to a single point in the texture, giving you a “solid color” triangle (that is, the whole triangle picks up the color of the single point of the texture.
You can also have this problem if you use “1-dimensional” texturing – that is, all three texture points for a triangle are along a line in the UV map or two of the three points are colocated.
If you need to use bump maps for your OBJ and you need a “single color” spot, separate the UV map of each vertex of the triangle by a little bit (even just one pixel).
You’ll know you have this problem when you see some of your polygons show up as pure black when HDR lighting spills on them. Here the top of the APU/air conditioner on the jetway has a degenerate UV map. In HDR mode it shows as a black area despite the spill light. This kind of artifact will show through any geometry that should normally cover the affected area.
Why This Happens
(What follows is strictly for the computer graphics nerds – you don’t need to understand this to fix your OBJs).
For the computer graphics nerds: in order to apply a tangent space bump map, you need three basis vectors to create a 3-d space for the normal vectors; the tangent space bump map is then essentially a perturbation of the regular normal vector. (Or put another way, the tangent space coordinate space system defined by U, V, N is used to transform the tangent space bump map back into world space. For the degenerate case of no normal map, the normal map can be thought of as 0,0,1 and the transform is a no-op from what we would have had anyway.)
Some engines require that the U and V axes be specifically passed per vertex. X-Plane, however, reconstructs the U and V map from the UV map of the model. When the UV map of the model is degenerate (e.g. it doesn’t form distinct non-zero length basis vectors along the output triangle) the coordinate system of the tangent space normal map ends up degenerate and the output normals suffer from a divide-by-zero.
The divide-by-zero is burned into the G-Buffer and the artifacts are thus persistent.
Fill rate: when we talk about fill rate, generally what we’re talking about is your GPU’s ability to fill in individual pixels to make the final image you see while flying. (The actual components of GPU throughput get a lot more complex, but we’ll keep things simple for now.) Two key things to note about fill rate:
- It only comes from your GPU – not your CPU.
- The bigger the size of X-Plane’s window (or the higher the full screen monitor res), the more of it you use.
Therefore I can offer this very simple test as to whether you are fill-rate limited:
If you are running in windowed mode and making the window smaller improves frame-rate, you are fill-rate limited.
(There is actually one important exception to the above test: if you are running out of VRAM, making the window smaller uses less VRAM for the main window + off-screen buffers, which can help fps. So be sure to see whether cutting your texture res helps fps – if it does, you are out of VRAM.)
Fill rate gets used up by:
- A larger rendering window. This is the number one consumer of fill-rate. Double the size of the window in each dimension, you use 4x the fill-rate. That adds up quick!!
- Clouds tend to be fill-rate intensive.
- HDR mode is significantly more fill-rate intensive than non-HDR mode.
- The 3-d cockpit is more fill-rate intensive than 2-d and external views, particularly with HDR enabled.
- Shadows, to a limited extent.
X-Plane 10 contains a stat called “cpu load” in the framerate output data line; I will describe its full meaning and implication in another blog post, but generally low CPU load (e.g. < 0.7) means you are fill-rate limited. If your CPU use for one core is 100% (this means 25% in the task manager for Windows users on a 4-core i5, but 100% in top for Mac/Linux users no matter what the number of cores) then you are almost certainly not fill-rate limited.
Optimizing Fill Rate
X-Plane 10.03r1 will post in a little bit – this is our first attempt to finalize X-Plane 10.03. It is not the end of the bug fix and performance tuning work for X-Plane 10. But if we can get a stable 10.03 we can use it as a baseline and limit betas to those who check the “get betas” check-box in the installer.
X-Plane 10.03r1 contains fill-rate optimizations for the 2-d and forward-no-HUD views; we’ll have to see how much they help people, but they should help you a bit if you are fill rate limited, flying a plane with a 2-d cockpit, and are seeing low fps with clouds and/or HDR.
Once 10.03 is final, we’ll start 10.04 and I will attempt to make a similar set of optimizations for the 3-d cockpit. Note that the 2-d optimizations do not apply if you have a plane that uses the 3-d cockpit for the 2-d view. (Since all of the new X-Plane 10 default planes do this, we’ll need the 3-d optimizations in 10.04 to see the performance benefit with the 747, Baron, and other new default planes. The Cessna, Cirrus Jet, and Piaggio from X-Plane 9 should all benefit immediately.)
Your Mobile GPU Is Light But Not Fast
Here is a table I culled from Wikipedia: it is a table of GPU performance for the “top end” cards for ATI’s last 4 generations of hardware.
| CARD |
DATE |
GP |
GT |
BW |
GFLOP |
| 4870 |
Jun 08 |
12 |
30 |
115 |
1200 |
| 6970M |
Jan 11 |
21 |
32 |
115 |
1305 |
| 5870 |
Sep 09 |
27 |
68 |
153 |
2720 |
| 6970 |
Dec 10 |
28 |
84 |
176 |
2703 |
| 7970 |
Jan 12 |
29 |
118 |
264 |
3789 |
GP = gigapixels/second of fill, GT = gigatexels/second of texture, BW = GB/second of memory bus bandwidht, and GFLOPS = gigaflops/second of MADD. At least I hope – see the original table.
Now I have inserted one mobile GPU (the 6970M) into this chart. So the first thing you can see is: the 6970M provides in a laptop the power that was available in a desktop about two years earlier, more or less. That’s a big step back in time.
The reason is simple: mobile GPUs are typically cut down in their core configuration – there’s no way you can run a 150W full power GPU with those 3 fans in a laptop.
If you have an iMac, you have a mobile GPU. Even the older iMacs which didn’t have the “M” designation on their chip, those were mobile too. In fact, you’ve got a mobile GPU on a huge screen. (Super-size to the 27″ iMac to get the 4x fill rate boost and you pick up 1.7x the pixels; if you get the big iMac with the cheap GPU you have a mid-range mobile GPU on a 2560 x 1440 screen!)
Some Kind Of Summary
I’m not entirely sure what my main point here is…perhaps the executive summary is:
- We (LR) are working to optimize fill rate in X-Plane 10. Fill rate is not the only performance bottleneck we are working on, but it is a very important one. Some of this is in X-Plane 10.03r1, some will come in later updates.
- You can determine whether you are fill-rate limited by resizing your window.
- If you have a mobile GPU in a laptop or iMac you may have less fill rate than you would expect.
At this point I am mostly done investigating weird video card behavior: these two pages are now reasonably accurate for X-Plane 10.
We are still trying to resolve as many of the known issues as we can, but now that we know what cards are in and out, it gives me more time to work on performance and scenery tools.
Happy New Year! I will try to get a few more posts out over the next few days; we spent a lot of December with our heads down trying to fix as many bugs as possible despite the chaos that the holidays always inflict on any company.
We had a slight fire-drill with beta 8 last night; it turns out it crashes on Windows, so we pulled back to beta 7. X-Plane 10.03 Beta 9 will come out today and fix the Windows crash and 1 or 2 other issues.
Right now we are “pushing” betas to everyone; this is not the norm and we will go back to the normal process (where users opt in to betas) once the 10.03 beta process completes. Since 10.03 has so many stability fixes, we want to make sure everyone gets them. We’ll lock 10.03 down before proceeding to additional performance tuning.
For December I had four things I was trying to work on: stability fixes, backward compatibility, performance enhancements, and third party support. I was only able to make progress on some fronts. At this point we have most of the hardware support issues stabilized and we’re close to closing up the crashes. (There is no short term fix for running the sim out of address space unfortunately.) We’ve done some performance tuning and a lot of analysis, but a lot of the work to act on the analysis is yet to happen.
The front where I made no progress is in third party support. So starting this week I am going to try to dedicate one day a week to some kind of third-party-only activity, whether it is getting out the docs on the new v10 authoring features, getting out scenery tools, etc. We have a lot of “almost ready things” and I don’t want to keep authors waiting.
I had a chance to catch up with Robin the other day and discuss airports with him. Here’s the basic road map for airports in X-Plane 10.
In version 9, an airport consisted only of taxiway layout data. Robin collected and managed a big database of contributed airport taxiway layouts, which are available under GPL and ship with the sim. WED 1.0 lets you edit these layouts.
In X-plane 10 we now have three categories of airport data:
- The taxiway layout – this lives in the apt.dat file.
- The ATC taxi layout and flow information. This also lives in the apt.dat.*
- Lego brick building placements. This lives in an overlay DSF.
Our plan for the version 10 run is to collect all of this data together and redistribute it all, just like we do the current apt.dat file.
We also plan to build a few airport building layouts ourselves, using the existing lego bricks, without custom elements. This will help us further debug the bricks, get users some more airports quickly, and help us understand the authoring process. We have some of Tom’s time earmarked for this.
WED 1.2 will support taxi layouts and airport building placement. Based on my talk with Robin, I believe I will also need to build a more specialized “Send to Robin” export function that pre-checks and packs the submission to streamline the process; since an exported airport will include DSFs and apt.dat files (and should not contain custom OBJs) having the packing be automatic will save everyone a lot of time.
What about autogen buildings? I don’t know. We wanted to try this before we shipped and ran out of time. I think we could autogen buildings for small airports that just have buildings next to the autogen taxiways, but for custom layouts I don’t know that autogen will ever be good enough to make humans happy.
Over the next few weeks I’m hoping to have more time to roll out tools and documentation to move this process forward. The library is already complete, so the quicker we can get everyone using it, the better.
* If you have an airport layout wihtout taxi layout and flow information, X-Plane automatically generates default flows and layouts.
In my previous post I announced that we have the ability to recut DSF tiles in an update and that we have already started doing this. I pointed out that we’re not signing up to recut the whole planet because we don’t how we will manage distribution yet.
This sets us up to talk about OpenStreetMap (OSM). First the basics:
- The X-Plane 10 global scenery gets its vector data from OSM – that would be roads and water.
- X-Plane 10 does not get its raster data from OSM – we get land class and elevation data from a composite of sources , both integrated by Alpilotx using a mix of tools I wrote and some GDAL/GRASS voodoo. (I think he may have had to sacrifice a live goat. I try to not ask questions about what goes on with the raster data!)
- OSM is the only source of water body data in X-Plane 10.
The result is a lateral move for water body data quality and an ironic one: for the first time the US isn’t the area of the world with the best data sources. OSM coverage in Western Europe is often near complete. By comparison, the US is still quite sparse.* The result is that water bodies that were present in the US in version 9 are now missing in version 10.
(To make matters a bit worse, the OSM extract for the global scenery is from July. I had to take an extract and stop updating to stabilize the scenery development process. We’ll take new extracts in the future but what you see now is months older than the current OSM data, not weeks.)
Why didn’t we use multiple water data sources? There were two problems:
- We ran out of time. It’s really tricky to integrate multiple data sources that don’t line up spatially, and while we had some promising tech, it made the DSFs worse as often as it made them better.
- Because all other sources of water were not in OSM, we had the potential for road/water conflicts, which makes the integration very tricky. In the old days when we used terrible VMAP0 data for roads, we could just nuke the road if the water didn’t match. But OSM road data is quite accurate and we didn’t want to manually edit it.
So my thinking is that what we really need to do is make OSM as good as possible and then recut tiles. Improving OSM improves the global scenery in the long term, everyone can participate, it helps everyone using OSM, and once OSM has the correct water data, getting it into a DSF is just a matter of recutting the tile. If the water data is all in OSM, then we don’t have to worry about water/road conflicts. This is the true power of using OSM: for the first time we can all focus on the source data, because the source data is public and community owned.
I did some work a few months ago to prepare NHD (the US national hydrography dataset) for OSM use. I am hoping to find a little bit of time to continue with this effort – my goal was to make it easier for people to import high quality NHD water into OSM, but there were some unsolved workflow problems when I left off to focus 110% on getting v10 out the door.
What Other OSM Data?
One question that comes up over and over is: what other OSM data will you use? The short answer is that I don’t know, but I think individual building shapes is not on the short list. The problem is that I don’t think we can adopt our autogen at the quality level we like to such arbitrary shapes.
That doesn’t mean that other third party scenery can’t use the data now, or that we won’t use it someday. But it’s not next on my list.
One type of data that I do think we could make good use of is “point of interest” data – e.g. we there is a school tagged in the OSM data, we could drop in a school autogen art asset for a given location – this kind of “shaping” of the autogen could, I think, provide another level of richness and accuracy while leveraging the existing autogen and DSF generation technology.
OSM vs. Airport
One last note on OSM and recuts: in the past we have cut out roads near airports. Our older road data was often very inaccurate and would run right over a runway. The cutting process tried to minimize the insanity.
OSM data is often very good and there are many cases where the OSM data could be left alone for perfect perimeter roads. I do not yet know how we will modify the cutting code to ‘preserve’ such road data, but I would like to get this right in future renders.
One problem that makes this difficult: often the on-airport roadways are modeled in OSM, but without any tagging. The result (when road cutting is off) is a city street down the middle of the tarmac, often where the vehicle paths should be. So we need a way to know whether a given OSM road is okay or not.
* The US road grid was first seeded by integrating TIGER road data into OSM. This gives the illusion of total mapping when in fact some areas have never been hand edited. The TIGER water was not brought in during the initial import, hence areas with roads but no water.
With X-Plane 8 and 9 we had a perpetual problem: the airport layouts ship with X-Plane and are updated frequently by Robin; if an airport layout is changed significantly or a new runway is installed, the airport layout may conflict with the base terrain and autogen from the underlying DSF that was cut when the sim was first shipped.
This resulted in a huge amount of stress for users who were building airport layouts. When is the cutoff for new layouts? What if I don’t make it? It will be years before the DSF is fixed.
I think the solution for version 10 is actually quite simple: we will recut individual DSF tiles and “push” them in sim updates.
In fact, we are already doing this. Five tiles, including the tiles containing Paris and Denver, rolled off the production line broken and we didn’t detect the problem until after we burned the DVDs. The patch is already online, and if you are running with current betas and installed Denver and Paris from DVD, you should have the corrections.
I think we’ve reached a point in X-Plane’s development where we need to evolve the global scenery from a static product that never changes to an ongoing one that can be updated in pieces, on the fly.
The History
I have been involved in X-Plane since version 6; for as long as I’ve been around, the product has been updated frequently with frequent patching. Starting with X-Plane 8 we developed a proprietary installer that let us easily push patches to three operating systems, with users downloading only the changed bits of the sim.*
The problem with the update strategy was that the global scenery was always too big. And what we found was that if we perpetually recut the global scenery, it made everyone who had already bought the scenery crazy mad. The last thing you want to do is buy the sim and discover it would have been better if you waited.
The idea of the “update” strategy is that you can buy into a major version run as soon as you want or late, and if you jump in early you get all of the future updates for the version run for free over the net. So we would update the sim and leave the global scenery alone. Thus there was never an update to the version 9 global scenery.
The Brave New World
The problem with not updating the scenery is that it takes us years to fix bugs, share improvements with users, and integrate changes from users (e.g. better airport layouts). Using OpenStreetMap raises the stakes: anyone can improve raw vector data using OSM at any point. How long do we wait to ship the improvements to everyone?
So I think it’s time to start recutting tiles and shipping them in updates.
Now to be blunt: I have no idea how we are going to go about doing this. So far recut tiles come down in free updates to users who have them installed, but this is just a beginning; we don’t have scalability problems yet because we have recut only 5 tiles.
Airport Anomolies
In the long term I don’t know how many tiles we will recut, when we will recut them, or what percent of the Earth we might recut in a given version run. We are not promising to recut any particular area or set of tiles. Please read that sentence twice. Don’t come back to this post and say “but you promised me free global scenery.” I don’t know how much recutting we can do or what the terms will be. I just know that it is the direction we need to investigate and move in.
With that disclaimer in mind, I do think we will be able to recut tiles that have airport anomalies. For example, I have two bug reports on class B airports. KSFO has an incorrect coastline, and KORD has a collision between autogen and runways. In both cases, I don’t know what went wrong and will need to investigate. And in both cases, pushing the fixed tile in an update seems like a reasonable solution.
* As far as I know, no commercial solution would give us three operating systems, deltas from any past version, while making it quick to cut a patch. Since we use the installer for every beta, the initial development time was worth it to make patching quick for users and for us.
My preferred way to fix X-Plane problems is to fix the problem first, then post second. That way there is no question that what we ship is what we announce.
But there’s a bunch of issues floating around with X-Plane 10, some of which will either take a while to fully fix, or which will be fixed in steps, with some steps happening now and some happening over a much longer time frame as the sim evolves. So I’m going to write some of them up now and get the information out there. A lot of users have taken the time to send in good bug reports, so it’s not fair to then sit on the plan.
A lot of these plans are just that: plans. Plans change. I believe that one of LR’s strengths is that we can change the plan quickly – if we come up with a plan and it turns out to be dumb we don’t have to just go through with it, we can come up with something better.
With that in mind, I had the following conversation with Austin yesterday – over a rather poor cell phone connection:
Me: Heya! Do you want to talk about BAD ALEX?*
Austin: Wait, what? What did Alex do???
Me: Oh…no no. Bad ALLOCs! A-L-L-O-C-S. Running out of memory…
A number of you have seen this error. When X-Plane runs out of memory one of three things usually happens:
- Some low level part of the sim seg faults because the request for memory failed and wasn’t properly detected.
- The OpenGL driver blows up because it can’t function without memory.
- The sim tries to allocate memory and throws an “uncaught exception: std::bad_alloc” error.
So first, let’s make one thing clear: we are running out of virtual address space, not physical RAM or VRAM. X-Plane is a 32-bit process, and thus it can only access 2-4 GB of virtual memory (address space) regardless of what is in your computer. The specific limits:
- 32-bit Windows: 2 or 3 GB, depending on OS settings.
- 64-bit windows: 4 GB
- OS X: 3.5 GB
- Linux: 3 GB – I think.
This address space has to contain a few big things:
- A copy of every texture X-Plane is using. (Most OpenGL drivers will “back up” textures in virtual memory at least some of the time if not all of the time.)
- Every mesh that will be drawn for all scenery, planes, etc.
- The weather system (physical and drawn).
- The physical mesh that is collision checked.
- The sim itself.
- etc.
So if you increase the combination of screen size, texture res, turn on HDR (which allocates a bunch of textures to render to), crank up the autogen, the sum can exceed X-Plane’s virtual address space limit.
The Short Term: Surviving With 3 GB
In the short term there are a few things that you can do and a few things that we can do. If you are seeing crashes or bad_alloc errors:
- Turn down rendering settings. The major ones are: airport detail (set to default), forests (set to something moderate if you are using autogen too), and texture res (first run with compressed textures). Don’t use 4x SSAA when in HDR mode – use FXAA instead.
- If you are on 32-bit Windows, consider moving to 64-bit Windows, at least in the long term.
- If you are on 32-bit Windows with 2 GB per process, increase that limit to 3 GB as shown here.
Note that buying more RAM will not help! We are out of virtual address space, not physical RAM chips!!
In the short term we can:
- Improve the error reporting to make it more obvious what has happened.
- Attempt to detect a lack of memory earlier and give warnings.
- Tighten our memory belts – I need to go on another memory hunt. Unfortunately we went on a pretty serious one before we shipped so there may not be any quick wins.
- Detect 32-bit Windows that are running without the 3 GB warning and flag this for users clearly with help instructions. If you’re running with 2 GB on 32-bit Windows it’s hard to see this in the OS itself, but getting that extra GB makes a big difference!
The Long Term: 64 Bits
The long term solution is clear – we need to migrate X-Plane to 64 bits. This is not going to be particularly easy or fun.
- On Mac, we have to port all of our platform specific Window/UI setup code from Carbon/C++ to ObjectiveC/Cocoa because Apple doesn’t provide our APIs in a 64-bit variant. Thanks Steve.
- On Windows we have to change compilers, as the one we use now is old and doesn’t build x64 apps.
- Sandy and I will have to make a 64-bit variant of plugins.
- 32-bit plugins won’t run in a 64-bit app. As far as I know none of the three operating systems provides a way to bridge DLLs like this, so plugin authors will need to create 64-bit compiled versions of their plugins.
It’s not a trivial amount of work. But it is what we need to do. It’s crazy to have a video card with 3 GB of VRAM and an app with only 3 GB of address space – in that situation by definition an app could only fill VRAM with textures if it had no code (let alone data). For years the cards, CPUs, buses, everything has been getting faster, bigger, more, except for virtual address space limit, which we have finally smashed into face first.
There have been a bunch of posts in which I have said “we know we need to do 64 bit” but now we’re saying “it really needs to be on the next train that leaves the station.” We will probably start the porting work once we get the 10.0.x series of patches stabilized.
X-Plane 10.03 beta 4 is out – just a few unrelated notes:
- Legacy scenery packs should work – I’ll write that up in detail in another blog post.
- If you have Paris or Denver installed, you need to update using the newest updater (version 3.02). If you update from the About Box, you get this version automatically. But if you have a version of the updater already on your hard drive, get a new one here.
- If your third party add-on worked in v9 and is still not in working in v10, please file a bug if you have not done so already. If you did file a bug, please bear with us – we’re killing off compatibility bugs with each patch, but there are still some major ones out there. (For example, I have on my plate to fix some bugs with OBJ materials that affect a lot of airplanes.)
And I meant to link to this: everything you ever wanted to know about global scenery but were afraid to ask.
I have a number of technical issues to cover relating to v10 scenery, so I’ll try to increase the blog post frequency to get through it all over the next week or two.
The short answer is: I think it will be fixed for everyone in 24 hours, but less if you want to get it by hand. Here’s the whole story:
The Paris DSF rendered incorrectly when we cut the global scenery. We discovered this after we cut the global scenery DVDs, but before they shipped out to users. I found the bug in the DSF generator (it was an error in airport processing) and fixed Paris and 5 other tiles. This was a top priority fix – it’s been fixed for probably two weeks now.
The Paris DSF is included in the X-Plane 10.02 beta series – that is, it has been available for over a week!
But here’s where things go sideways: since we’ve never updated global scenery before, you need a new updater to actually get the file. And what I am just realizing now is: while you need the X-Plane 3.02 updater to get the file, everyone is using the 3.01 updater, which doesn’t know to grab the Paris fix if you have Paris installed from DVD.
Hang on one second while I bash my head on my desk.
Okay. So there are two ways you can get your Paris fixed.
The Fast Way
If you want Paris fixed now, do this:
- Get the X-Plane Updater from here.
- Run the updater on your copy of X-Plane.
Paris should be fixed, at least I think.
The Easy Way
If you have already installed global secnery for Paris and you do nothing now, but get beta 4 when it comes out, X-Plane will use the latest updater to get beta 4 and you’ll get Paris anyway. I expect beta 4 tomorrow…maybe Monday at the latest.
If you need to use bump maps for your OBJ and you need a “single color” spot, separate the UV map of each vertex of the triangle by a little bit (even just one pixel).