Category: File Formats

Don’t Change lights.txt

Some very advanced users have asked: can we change lights.txt. The answer is: please don’t.

Lights.txt is not a part of the “scenery SDK”, that is, it’s not a file whose format we will keep the same and allow you to modify. (The fact that it isn’t accessible via the library system is an indication of our intention NOT to make it part of the scenery system.)

The problem is basically this: lights.txt translates named lights into the inputs to our pixel shaders for the hardware-accelerated lights. That pixel shader is really new and likely to change a few times. If the shader changes, we might need new parameters not in lights.txt, requiring a fundamental format change.

For example: those who have poked in the named lights file have noticed that the hardware lights can either have directional or flashing properties. This is because they run on two different shaders, each taking only four input values. This was done a while ago, when we were using low level assembly language shaders. In the future we might merge the two shaders and have 8 params per light. This would give us more flexibility (directional flashing lights), more bus usage (pushing 8 params per light instead of 4) and fewer state changes (we have to change shaders right now).

My point is: we can’t predict what will happen, so we can’t safely expose these parameters. The best thing to do is: email me and request named light types. We can easily have hundreds of named-light types (see how many there already are just for airports!).

Named lights make our lives easier because it tells us WHAT to draw but not HOW to draw it. So when we put in the next evolution of the lights code, we can remap the named lights to look the best they can for the new technology, instead of worrying about how to map the old params to the new ones.

(I appreciate the input from the users who emailed me about this — it gives me more insight into what extensions to the scenery system would be useful.)

Posted in Development, File Formats, Scenery by | 1 Comment

Liveries Part III: Image Contracts

The livery system I described in my previous blog entry (proposed independently by several users) would imply a contract between the aircarft creator and livery creator. The way I look at it:

  • The livery creator agrees to utilize the texture conventions established by the aircraft modeler.
  • The aircraft modeler recognizes that changes in the fundamental texturing scheme of the airplane become much more “expensive” because they potentially invalidate a whole set of liveries.

This is an exceptional situation in X-Plane…we have avoided contracts regarding the layout of images in all previous cases. (Examples: you can’t use the library to replace an OBJ’s texture. You specify terrain via a text file that references the image file, not the image file itself, so information on how to use the texture isn’t part of the contract.)

But in the case of liveries, I think the exception is the inevitable outcome – a livery quite literally is a repainting of an aircraft, and the X-Plane community has been living with the limits of this kind of add-on for a while.

What this means is that it would be appropriate to override object textures in a livery system, but this feature is unlikely to appear anywhere else in the sim.

Posted in File Formats, Scenery by | Comments Off on Liveries Part III: Image Contracts

Liveries Part I: Contracts

Any time two independently varying components need to interoperate, a contract is required.

Because X-Plane varies independently of third party add-ons (we release patches, you make new add-ons), any extensible part of X-Plane implies a contract. That contract basically says what legal things the add-on can do and how X-Plane will react.

We have to consider this “contract” with third party add-ons any time we modify the sim, and in some cases it means we can’t change things. How this applies to liveries will become clear later.

Libraries and PNGs

There was some discussion on about whether it should be possible to share and/or override PNGs via the library system independent of their objects. I say “no” for this reason:

The library system connects independent third party add-ons, that is, components that vary separately. Therefore there needs to be a “contract” any time the library is used, between the package requesting something from the library, and the package fulfilling it.

My concern with PNGs is that a library PNG would have to have a fixed layout. But realistically the kinds of PNGs that people want to share get reorganized on a regular basis. In particular, people want to reference the default sceney PNGs. But where we have a contract, we are limited in what we can change, and reorganizing the default scenery is critical to our ability to grow our content. I agree with Aussie’s comments on that fixed-layout PNGs don’t add a lot of value to the library system.

File System Vs. Text Files

There are a number of design trends in all of X-Plane’s third party add-on systems. One is the use of the file system to specify modifications to the sim. For example:

  • Put a “cockpit” folder in your aircraft, we’ll try it first.
  • Some files in the aircraft folder must have certain names.
  • Sometimes putting _LIT after a texture causes it to be used as a lighting map.

On the other hand, I’ve gone a very different route with the scenery system:

  • Libraries are looked up via a text file, not via filename.
  • In all scenery cases, the lit texture is specified in a text file, not by filename.

What’s the difference? Well, the file system way seems to be simpler for most users to understand. The text file mechanism is a lot more flexible. (Consider: we have versioning info in a text file, we can have each line in the text file clearly identify a feature, and so there is no risk of a file name being mistaken for a feature.)

Whether we use file names or text files to control extensions, doing so creates a contract, so we must ask: can we easily extend or safely modify the contract later? Can we express what we want using filenames (or text files)?

In the case of the scenery system, I think we need the full expressiveness of text files. Imagine if we wanted to provide a new kind of lighting that uses a different format of texture. Or a way to control at what time of day the lights get turned on? Or what if we want seasonal varying textures, and the rules for how seasons affect the light map aren’t simple? In all of these cases, a file name convention rapidly becomes unworkable.

On the other hand, the airplane system has done reasonably well with filenames. Based on what I see, the problems with airplane distribution mostly come from a lack of features (no livery support, no plugins built into airplanes), not file names as a convention running out of flexibility.

Posted in Development, File Formats, Scenery by | Comments Off on Liveries Part I: Contracts

Color Matching in X-Plane

Sergio sent me a scenery package with the question: why don’t these two textures appear as the same color in X-Plane (they did in Photoshop). The answer is: gamma correction.

Some background (with much hand-waving): gamma refers to how bright the mid-tones of an image appear. (It’s a lot more complex than that, but that’s what Wikipedia is for!) Basically Macs and some other computers adjust the colors in an image to compensate for the deficiencies of CRTs, while PCs leave them alone. The result is that the same numeric color levels, when sent to Mac and PC hardware, result in brighter images on the Mac than the PC.

Since X-Plane is authored almost entirely on the Mac, the old complaint (around the time of X-Plane 6, with BMPs) was that X-Plane looked too dark on Windows. PNG addresses this issue: the gamma curve of the system an image was created on can be written into the PNG, allowing X-Plane to adjust the colors (making them brighter or darker) depending on what destination system we are running on.

Unfortunately, X-Plane isn’t too brilliant about this, in two ways, one of which isn’t our fault:

  • X-Plane assumes the platform default gamma for both Mac and PC (that would be 1.8 for Mac and 2.2 for PC). If you read that background article, you know that this isn’t real clever of us. But it actually is better than doing nothing at all.
  • Gamma in PNG files is optional; if we get a PNG file with no gamma information, we make the rather arbitrary assumption that it came from a Mac. Since X-Plane used to be authored on Macs, this seemed like a reasonable thing and in the case of no gamma information, we’re going to be wrong half of the time no matter what.

If you open up the default scenery PNG files in Preview, you can see the gamma inforation – 0.4545. This is 1/2.2, meaning the files are encoded to PC gamma standards. It turns out that one of the textures Sergio sent me had no gamma information, so X-Plane assumed Mac gamma (0.5555, or 1/1.8). Thus the brightness of these textures were being adjusted by different amounts.

My recommendation to authors is simple: make sure that all of your PNG files always have gamma values written into them. Otherwise there is a risk that the default gamma guess that X-Plane makes will not be the one you authored under, causing color shifts.

Posted in Development, File Formats, Scenery by | Comments Off on Color Matching in X-Plane

Crash Bang Goes the Airplane

Ari asked a good question regarding sloped runways and the new apt.dat 850 format:

Am I understanding it right that airport taxiways, ramps and runways are from now on going to be merged into one big mesh, instead of bunch of rectangle pieces overlapping each other? If yes, will this finally allow us turning on sloped runways option in X-Plane without any of the current side effects?

This brings up some interesting questions. First the most basic answer:

  • apt.dat 850 prrovides curves, irregular taxiway shapes, which will allow you to create complex taxiway shapes with only one piece of pavement, rather than many overlapping ones.
  • X-Plane still honors the order of apt.dat 850 for drawing, so you can also overlap and get visually consistent results.
  • We recommend using a smaller number of curved taxiways rather than many overlapping rectangular ones because X-Plane can handles this case more efficiently. It is not necessary to build the entire airport out of one taxiway though.

Now the second part of the second of this question is a little more complex, because the cause of bumps in the scenery changed.

Bumpy Runways in the Good Old Days

Back in X-Plane 806 there was a fundamental problem with the way we did sloped runways that made them virtually unusable: while the corners of each rectangular piece of pavement would sit directly on the terrain (no matter what the terrain’s slope), the area of the taixway was formed by a flat plane. This means that the middle of the taxiway might be above or below the terrain.

Now the real problem comes when we have two taxiways that overlap. Because they are only aligned to the terrain at their corners and not centers, there may be differences in their height when one taxiway’s corner hits another taxiway’s center (which happens a lot). As the airplane travels from one taxiway to another, the elevation of the ground changes instantly, inducing a major jolt to the suspension. At high speeds these damage the airplane’s suspension.

Bumpy Runways Now

In X-Plane 850, we break all runways and taxiways (new and old) into multiple pieces each tiem the terrain underneath them has an edge. The resulting taxiways are then aligned to the mesh at their corners. But since no taxiway center goes over a mesh corner, the taxiway “hugs” the mesh perfectly. And since all taxiways hug the mesh in the same way, there is never a height gap between taxiways.

It’s the Mesh, Stupid

So why do we still have bumps in X-Plane 850 if we so carefully make sure the taxiways exactly reflect the mesh height? Well, you’re effectively driving on the terrain, so any bumps are ones from the terrain. Simply put, even with the new system the usability of sloped runways is only as good as the underlying terrain.

Now our meshes come from SRTM data, which is radar data – it naturally has a certain level of noise and “speckle” which makes it pretty unusable for airports…airplanes are very sensitive to even small bumps during takeoff.

We attempt to “condition” the elevation data for airport use, smoothing out hills and bumps. Unfortunately our algorithm doesn’t always work right. The X-Plane 8 US scenery was way too bumpy to be usable. The 7-DVD set is better, but still makes bumpy airports in a few cases:

  • If there was no airport in the apt.dat file at the time of scenery creation, no conditioning was applied, and the underlying terrain is probably inappropriate for draping.
  • The scenery creator has a bug that causes airport flattening to fail when it’s very close to water. For example, the big bump in the runway at KLGA is due to a water-airport interaction.
  • I think that flattening across DSF tiles can have problems too.

If you’ve been thinking — wow, the diagram for 850 has a lot more triangles (10 vs 4) than the one for 806, you are right. Fortunately, the number of triangles, all part of one taxiway, in an airport layout, doesn’t really affect frame-rate, since this is handled by the GPU.

But this is also a case where a few curved polygons can be much more efficient than several overlapping ones – when we cut up the taxiway based on the mesh, if there is ovelapping pavement, each overlapping taxiway must be cut, multiplying the effects of the mesh on triangle count.

It also turns out that in the real case this is somewhat moot: because X-Plane smooths the airports and then induces triangle borders around the edge of the airport. Since the interior area is so flat, it doesn’t require a lot of triangles, and therefore the trianglse inside an airport tend to be big, so the number of times we have to cut an actual layout is quite small.

Posted in Development, File Formats, Scenery by | Comments Off on Crash Bang Goes the Airplane

Look Mom, No Tiles

Since before I’ve been involved in X-Plane, scenery has been broken into tiles. You know the naming scheme…+42-072.XXX is some kind of file for Boston. Perhaps it’s a .env or a .dsf, but what is unchanged is that the world is broken up into bite-sized pieces.

Tiling is very necessary for X-Plane…it allows the sim to rapidly load only the information we are interested in. The world is too big to go fishing for the relatively small amount of data that is loaded at one time.

The source data that is used to build global scenery is also tiled – if you look at the SRTM files, they often have names like N42W072. It’s no surprise that global data requires tiling to make it manageable.

When we produce global scenery, we work on a per-tile basis. We load the raw tiled data* into source XES files (one per tile), then process and convert that XES file into a DSF. By working one tile at a time, we limit how much RAM we need.

But if you used WorldMaker on an airport or city that spanned a tile boundary, you know how annoying tiling can be. With WorldMaker you couldn’t see both halves of an airport at once.

WED will not use tiles. For custom scenery packages, the total data is not so large that we have to tile. When you work on a scenery package in WED, you work on the entire package at once as a single seamless workspace. When you export your work, WED will split the data into tiles as needed. This will mean:

  • You do not have to decide in advance where you are going to work. All scenery packages cover the entire planet, and DSFs are created only as needed.
  • You will be able to work on more than one “tile” of area at a time, because there are no tiles.
  • You will be able to work on airports that span tile boundaries seamlessly.
Posted in File Formats, Scenery, Tools by | 1 Comment

You Can’t Unbake a Cake

First, I just want to be clear: I am not announcing any future features for the scenery tools. I am not saying when they will be released, and I am not saying what they will do, because honestly I do not know. There have been too many cases when users have emailed me and said either “I thought you guys were going to do X” or “you guys promised you were going to do X”, so now I am officially paranoid.

So this blog post is not about what the future scenery tools will do – it is simply a discussion of the difference between editing source data and compiled DSFs.

When we make the global scenery, we start with a bunch of source data that roughly consists of: road maps, coastlines, elevation, landuse, climate data, airport locations, etc. When we build a DSF out of it, we “bake” these items together into a single file. During this baking process, our tools apply some “integration effects”. Here are a few of the more obvious examples:

  • Terrain under airports is forced to be an airport grass texture, appropriate for local climate data.
  • Roads are removed from airport areas.
  • Intersections are computed for highways – that is, a highway and city street form an overpass, but two city streets become a real intersection.
  • Generated Buildings are put around the roads, not under them.
  • Buildings are oriented to “face” the slope of the ground they are under, based on their shape.

That’s not a complete list, but it gives you an idea of some of what goes into making a DSF. All of this information is precomputed and represented in the final DSF. But consider this last point: the DSF contains the actual orientation (north/south/east/west) of each building. It does not contain the slope underneath the building and it contains no information about which buildings need reorientation. In other words, the results of the process, not the inputs to the process, are present.

So consider what would happen if you could simply edit the data in the DSF:

  • If you moved an airport, the airport grass would stay in its own location.
  • If you moved an airport over a road, the road would still be there.
  • If you changed a city street to a highway, it would not form an overpass.
  • If you moved a road on top of a generated building, the building would remain in place.
  • If you changed the ground elevation, buildings would not change their orientation to face the new slope.

If you changed the source data and re-ran the DSF building process, these effects would occur. But remember, we need elevation, land use, etc. to build a DSF from scratch, and the DSF itself doesn’t contain its source data.

So we have two possible strategies for editing DSFs:

  1. We could build a DSF “retouching” tool that let us make very small changes to the existing DSF without having any source data. None of these effects would “work” so authors would have to make very small changes and then hand-fix any problems that appeared.
  2. We could build a DSF “rebuilding” tool that let authors make new DSFs from source data. All of the effects would look good, but authors would have to get some of the source data. (We could post our source data, or provide links to places where it can be downloaded.)

Note that we can’t have our cake and eat it…we cannot get the “integration effects” listed above unless we go back to the source data. If I can stretch my cake metaphor to the breaking point, once we make our cake, we can’t easily remove the flour and add another egg – we need to start over with the raw ingredients.

Which strategy will the scenery tools use? I don’t know yet. I am focusing on airport and overlay editing, which sidestep this issue a bit (we can easily edit an apt.dat 850 or overlay from the final product). We may do a bit of both strategies – it depends on what users want and what we can code efficiently.

Why don’t the finished DSFs contain everything we need to edit them? The answer is size. The finished global scenery was about 56 GB of DSF. When I last checked, the raw data that forms the DSFs was at least 100 GB or more. So for each DVD we shipped of scenery, we’d have to ship two more DVDs of source data, for a 21 DVD set, most of which wouldn’t be useful to most users.

Posted in File Formats, Scenery, Tools by | 5 Comments

European Buildings Everywhere

A user reported a bug: the 3-d buildings in all parts of the world outside the United States* are European-style, even in Australia, Canada, China, etc.

It takes two things to see 3-d objects in X-Plane:

  1. Location data in the DSF you are flying over.
  2. 3-d models (stored in OBJ files) that match the definitions called for by the location data.

Now we have location data for our “generic” objects (objects that are placed pseudo-randomly based on local topography and roads, to synthesize cities) for all urban areas that are covered by the 7-DVD global scenery set. But we only have two sets of artwork: a set of US-style buildings for use in the US, and a set of European-style buildings for the rest of the world.

So we had an unfortunate dilemma. Either we could have European style buildings everywhere, which looks funny, but at least it’s 3-d. Or we could use the European-style buildings only in Europe and have no 3-d at all everywhere else. Both options make some users unhappy and neither is very good. We chose to have the European buildings everywhere – my guess is that it generated slightly less complaints.

Well, the silver lining is: it’s actually really easy to remove the European-style builldings from the rest of the world (Canada, Australia, China, Russia, Africa, etc.). All you have to do is make a custom scenery pack that applies to the regions where you don’t want the buildings and maps them to a blank object. I have built such a custom scenery pack and posted it here. If you install this scenery pack, then Canada will be devoid of 3-d, but at least there won’t be mismatching architecture.

This begs the question: why not use the US buildings to populate some of those regions? The answer is that the US buildings are built to a different spec than the European ones, because the road grid data for the US is more detailed than the rest of the world. So amongst our current artwork, only the European buildings will work globally.

* When I refer to the United States, I really mean the continental United States. Sorry to everyone in the beautiful states of Hawaii and Alaska. This is just an artifact of how the data was imported – I could not get the Census data for Hawaii, Alaska, or the territories to import, so I reverted to the generic “global” data.

Posted in Development, File Formats, Scenery by | 2 Comments

The Sordid History of ATTR_poly_os

I’ve blogged in the past about ATTR_poly_os…it’s a tricky topic. ATTR_poly_os is
a feature of the OBJ file format designed to let authors fix z-buffer thrash problems. Unfortunately, the cause of z-buffer thrash is pretty complex. To make things worse, it turns out I never finished my intended documentation on the subject. (I’m an idiot!)

The fundamental problem I think is that what we have now (ATTR_poly_os and ATTR_layer_group) provide a mechanism to correctly fix z-buffer thrash, but they don’t in any way enforce good behavior over bad behavior. The two attributes are very flexible, and if used together, can do all sorts of bad things. The problem is that ATTR_poly_os was thought up years before the layer-group mechanism, and thus they don’t really reinforce each other.

So…here are a few simple rules to help with z-buffer thrash in X-Plane 860:

  1. Never use the names of objects or their order in the DSF to accomplish anything. X-Plane ignores both names and orders when processing your scenery.
  2. Do not move your polygons above the terrain to fix z-thrash. This won’t work.
  3. When possible, divide your objects into ones that are 100% on-the-ground (and thus may z-thrash) and ones that are 100% 3-d above the ground (and will not thrash). I realize that more objects means slower fps…so this applies best when you have many objects and can pick how you divide them up.
  4. Always use ATTR_poly_os for any polygons that lie along the ground. Use the smallest number you can to fix the thrash.
  5. If you have an object with ATTR_poly_os geometry and non-poly_os geometry, make sure the ATTR_poly_os geometry is first!

Those five rules should keep you out of trouble.

What about ATTR_layer_group? Well, secretly X-Plane 860 will change the layer group of an object that is ATTR_poly_os for you. So as long as your object contains only offset geometry (this is what I recommend in rule 3) it wll always be drawn before the rest of the objects, preventing artifacts.

You’ll need ATTR_layer_group if you want to put objects underneath runways, or underneath taxiways, for example.

I am working on more comprehensive documentation on the topic, and appreciate any feedback on stuff that I’ve written that’s unclear…the rules are complicated!

Posted in Development, File Formats, Scenery by | Comments Off on The Sordid History of ATTR_poly_os

Using Layer Groups

I can’t say enough good things about Jonathan Harris (Marginal) — his work on X-Plane has been fantastic, he is one of the most advanced third party scenery authors I know of, and when he sends us a bug report, it is usually so perfectly patched up that I’m looking at the bad line of code in minutes! (One of these days I’ll post one of his bug reports — he always isolates the bug in a simple package that makes it very clear, with no extra “stuff”.)

He emailed me a while ago with some questions about layer groups, and I saved them to rewrite into a blog entry. A little bit of background:

X-Plane 850 and 860 introduce the concept of “layer groups”, which provide a way to control the draw order of scenery to some extent. Objects naturally fall into a layer group based on their type (e.g. objects go into the “object” layer group by default, and runways always go into the runway layer group). However, some scenery elements let you customize their layer-group placement in two ways:

  • By changing which layer group the element goes to entirely, or
  • By providing a “bias number”, which indicates that within the catagory of scenery elements, this one must be drawn early or late.

Layer groups let you do a number of useful things:

  • Make sure that polygons and objects are drawn under runways or over taxiways when needed.
  • Make sure that runway markings with polygon offset are drawn before 3-d objects.

Simply changing the order of objects in the DSF is not a reliable way to control draw order! Layer groups are. You can read about layer groups in the OBJ8 spec.

With that in mind, some Q and A. (I will elaborate on my answers from what I originally sent Jonathan.)

J: I assume that objects and polygons within a single layer can be drawn in any order – ie there’s no defined drawing order between different scenery types.

B: You assume correctly! Within a layer group, X-plane is free to reorder to improve fps. So you cannot rely on the draw order of any two scenery elements without assuring that they are in different layer groups, either by using different group names, or different relative offsets.

J: But is there are defined drawing order between objects/polygons and apt.dat-generated scenery?

B: Yes because airport scenery goes into specific layer groups! (In fact, all scenery has a “default” layer it goes into, and they usually vary by type of scenery element.)

J: eg if I have an object or polygon with ATTR_layer_group runways 0 will it be guaranteed to be drawn after the runway?

B: You’re close. The runways go into the runways group so

ATTR_layer_group runways -1

will always be before runways and

ATTR_layer_group runways 1

will always be after.

ATTR_layer_group runways 0

is the default layer group for runways, so your object or polygon would be in the layer group with all of the other runways, and X-Plane would be free to change the order amongst your object and runways in any way that would optimize framerate.

J: If not, is there any other way to insert objects and/or polygons between the taxiways and the runways?

B: The numeric offset is provided for this. The “spacing” of the layer group numbers is such that you can have up to 5 groups before and after each “named” group. So anything from runways -5 to runways +5 is fair game. (In other words, you can separately control up to 5 different “layers” of elements with a well-defined order for any layer group name.)

Posted in File Formats, Scenery by | Comments Off on Using Layer Groups