It seemed like opposite-day at Laminar Research…Austin saying that an error shouldn’t quit the sim and me saying the error was never okay, ever. Well, I relented: with X-Plane 930 beta 8, if your scenery pack is missing objects, you can still fly. Instead you get a single error message like this:
That is the “non-fatal” error dialog box – you will see it only once for each scenery pack with a problem for each time you run the sim. It means that at least one thing is wrong with your scenery pack, but you need to look at Log.txt to see what’s wrong. For example, you might see this in the Log.txt file:
Failed to find resource 'KSBD_example.obj' at 'Custom
Scenery/KSBD Demo Area/KSBD_example.obj'
Failed to find resource 'KSBD_example.obj' at 'Custom
Scenery/KSBD Demo Area/custom objects/KSBD_example.obj'
Failed to find resource 'KSBD_example.obj' at
'Resources/objects/KSBD_example.obj'
Failed to find resource 'KSBD_example.obj'
at 'Resources/KSBD_example.obj'
***Error with scenery file "Custom Scenery/KSBD
Demo Area/Earth nav data/+30-120/+34-118.dsf"
(/Volumes/RAID/code/design/HLutils/Files/io_dsf.cpp: 503.)
Unable to locate object: KSBD_example.obj
In this case, X-Plane couldn’t find the object KSBD_example.obj – the sim is also listing all of the places it looked. Note that only the first location is a good location – the other 3 are legacy search paths that date all the way back to version 6. It is likely that in the next major version we will trim down our search paths significantly.
A few comments on this whole situation:
-
Authors, do not ignore error messages like the dialog box above – every one of them indicates a condition serious enough that we think you should fix it. Non-fatal errors like these may crash future versions of the sim, or your content may simply stop working.
If you file a bug against a future version of the sim saying your scenery pack used to work and is now broken, and we find that the old scenery pack had errors, we’re not going to fix the bug – we’re going to laugh maniacally and dance around you in a circle while singing “told you so”.
Okay – we’re very unlikely to do that – but if you have errors in your scenery pack, you’re doing something wrong and you need to fix it – treatment of illegal data is not stable between versions of the sim!!
-
I was never very sympathetic to this whole bug report because X-Plane has never accepted a DSF with missing objects – this has been a fatal* error since X-Plane 8.0 when DSF was introduced. So I simply don’t understand why there are any scenery packs floating around with objects missing. Why would you place an object if you don’t want to see it? The whole issue strikes me as a total failure to check quality by authors, since even running your pack once would reveal this kind of problem every time!
-
The motivation to make missing objects illegal comes from version 7 and ENVs. When looking at ENV scenery, I found that a large number of ENV scenery packs were missing at least some of their objects (an error that was silently ignored in ENV). It seemed like we were hiding an error and the result was authors not noticing simple mistakes that might “lose” an object (e.g. renaming an OBJ file).
Hence the “harsh” policy for DSFs – it was in response to a real problem with existing scenery!
- You don’t need to have missing objects just because you use library objects from another scenery pack (that might not be around). Use the EXPORT_BACKUP command in your library and a single blank OBJ as a place-holder for the objects you want from a library that might be missing. OpenSceneryX provides a stub library that authors can include so that their scenery will load without errors even if the OpenSceneryX library is not installed.
Anyway, Austin was right to make the error non-fatal. Besides being a little bit nicer for users who don’t know (or care) why their pack is gone) it lets authors get a list of all missing objects with only one run of the program.
* Fatal? In computer terms, a fatal error is one that makes the program quit, e.g. the error is fatal because it kills the program.
I have said this before, but now it’s finally true: new file specifications are subject to change in the middle of beta!
In particular ATTR_light_level has changed slightly from beta 7 to beta 8. If you are using this feature in your objects, you will need to update your objects.
A new ac3d beta will be posted later today that supports the updated syntax.
You can read about the syntax here.
I get asked about the maximum mesh density in X-Plane a lot.
First, I must note that X-Plane’s mesh is adaptive – the triangles are not arranged in a grid, but rather they’re arranged to maximize the quality of the mesh with minimal triangle count, while preserving the outlines of water bodies and airports. These pictures show a dynamic mesh fit to a section of the grand canyon (both with and without the shaded DEM for reference):
Second, the actual maximum mesh density is almost certainly limited by system resource constraints. The maximum spatial resolution in a DSF is about 0.6 millionths of a foot. You’re going to exceed the maximum number of vertices, or much more likely, the maximum amount of available memory in X-Plane long before you have a grid at that resolution.
(That doesn’t meant that resolution isn’t usable – remember, the idea of the adaptive mesh is to have very high densities in very improtant places, but not everywhere.)
But in practice this is all moot if you don’t code your DSFs by hand – the question that really matters is: what is the highest density mesh that you can use with the current scenery tools.
Until recently, MeshTool was limited to a 1201×1201 (90 meter) input DEM. The input DEM is a simple grid, and the finished DEM is never higher resolution than the input. So until now, it would have been just about impossible to make a really high res mesh, even if X-Plane could handle it.
The most recent changes to MeshTool fix this – MeshTool can now handle a 10801 x 10801 (10 meter) input DEM, which X-Plane should be okay with as well.
This new version will hopefully go into beta relatively soon.
What is the difference between a dataref and a command? They serve different purposes in X-Plane, but it’s easy to get them confused, especially because the names can look so similar. If you only take one thing away from this comparison, it should be:
- Datarefs are information.
- Commands are actions.
Datarefs
A dataref is a single bit of published information. For example, the user’s indicated airspeed, as seen by the pilot, is a dataref, stored in:
sim/cockpit2/gauges/indicators/airspeed_kts_pilot
Datarefs have names that do not change. Datarefs made available by X-Plane start with sim/ while datarefs made available by plugins start with another prefix. Datarefs have been in X-Plane since the release of the plugin system in version 6.70.
You can always read a dataref, but sometimes you can change it. Trying to change a dataref usually has one of three actions:
- If the dataref is not writable at all, nothing happens.
- If the dataref is writable, it will change.
- Sometimes a dataref may be writable, but only after changing some other sim configuration. For example, you can only “write” to the control surface deflection datarefs after setting the control surface override dataref to 1. (If you don’t set this override, X-Plane will constantly write its own ideas of the control surface positions to the control surface datarefs and your changes will be lost.)
You can read and write datarefs:
Commands
A command is an action that the sim can take on your behalf. For example, the command
sim/autopilot/altitude_arm
arms the autopilot for altitude hold.
Like datarefs, commands have permanent names, starting with sim/ for X-Plane or other prefixes for plugins. Commands have been available in X-Plane since version 9.0.
You can always actuate a command, but there is no guarantee that it will do anything. For example, the engine starter command won’t start the engine if the plane has electrical starters and the battery is dead.
You can use commands by:
Plugins
Plugins can add both new datarefs and new commands to the sim. Plugins can also change the behavior of all built-in sim commands, and can change the information in some datarefs.
Where Do I Find Datarefs And Commands
X-Plane’s default commands and datarefs are listed in the text files Commands.txt and Datarefs.txt in the Resources/plugins folder. (Note: providing the command list is new to X-Plane 930.) The dataref list is also available on the X-Plane SDK Wiki.
Up next: when should I use a command and when should I use a dataref?
I’m always a little bit nervous about posting grand new initiatives … if we don’t actually do the initiative, invariably someone comes out of the walls later to say that we “promised” a feature that we didn’t do. But road maps are important. So, bearing in mind that this is not an official announcement, and that nothing has been decided yet, here are some areas of active investigation:
- An airport art asset library. Sergio already started this process by making some aircraft OBJs and other elements (like custom pavement types) available in the libraries that ship with X-Plane. We are looking at extending this over time to include more useful elements for building airports.
- Sharing airport building placements the way we do airport layouts. Right now, airport layouts are shared in a communal database under the GPL license. An airport building database would work the same way – it would be a collection of placements of airport buildings, GPLed and redistributed with X-Plane. The idea would be to make it easy for people to add simple buildings to their local airport and share the results with everyone.
- Using OpenStreetMap (OSM) for roads. We’ve been looking at OSM for a while, but it’s too soon to announce a plan.
- Sharing obstacle data with FlightGear. We already share airport layout data with flight-gear; this would be a similar initiative. We looked at OSM for this, but FlightGear’s data needs are a lot closer to ours. This is still in discussion; the FG guys are a sharp bunch, so I think we’ll able to work something out.
One of the common threads for all of these ideas is that X-Plane community members have dug into them before we have. This is not surprising, and I think it is a good thing!
Another common thread is that these are all open data sharing initiatives. Collaborative data sharing has come a long way since we redesigned the scenery system, starting in version 8.0. My hope is that over the next several months we can make some of these ideas a reality.
But first I have to fix my 930 beta features. 🙂
When you make an airplane, you customize some of the images and sounds by simply putting a wav or png file in your aircraft package, with the same name as X-Plane’s default. This is “file name” substitution.
What is good about file name substitution is that it is very, very simple.
But file name substitution has some limitations:
- If you need to provide multiple versions of a file, there is no way to do this. You can at most replace one file with one other file.
- There is a risk of “file name collision” – so file name substitution is only appropriate when we can be sure that a folder is only used for one simple purpose.
- The file name cannot easily encode a lot of information about how the file is to be used. We use _LIT to indicate an emissive texture, but imagine trying to encode every aspect of a .ter file in a file name (all of the projection parameters, physics parameters, texture clamping and alpha managemnent, paged texture loading). You’d end up with a file name like my_tex_na_42.23E_72.32W_conc_pd_5000_4000_nw_LIT.png. You can’t tell me that that’s an easy file to work with!
The scenery system hits all three of these limitations, and it deals with them in two ways:
- Texture files are almost always referenced from a text file. The text file provides a place to put all of the important parameters about the texture.
- The library system maps art assets to virtual file paths, avoiding collisions and allowing multiple files to be mapped to one virtual path.
Cristianno wrote this awesome tutorial, which shows how the library system works.
Microsoft extended Windows XP sales yet again, more or less. But rather than rant about how the Vista user experience makes me want to tear my fingernails off or how brain-damaged it is to try to put DRM into drivers, let me instead focus on Windows as an example of the cost of contracts.
I have
ranted in the past about how the boundary between X-Plane and a third party, or the plugin system and third parties, or even two third parties, is a contract. Consider:
- The named light list forms a contract between X-Plane and objects, e.g. X-Plane guarantees that there will be a named light called “airplane_landing” and that it is a good choice for landing lights. (This implies that X-Plane won’t change what it looks like to be inappropriate for landing lights, and that third parties won’t use it for inappropriate uses, like airport apron lights.)
- XPLMGetDataf forms a contract between the plugin SDK and plugins, guaranteeing that there will be a function in the XPLM called “XPLMGetDataf” that takes a dataref and returns its value. (This implies that Sandy and I won’t rename it or change its arguments or remove it all together, and that plugins won’t pass non-datarefs in as arguments.)
- Even a paint kit forms a contract – the airplane maker is essentially saying “the tail will be mapped to the upper left of the texture, and I won’t remap my UV” and the livery maker is saying “I will put an image in the upper left corner that looks like a tail.”
By comparison, the clouds are not a contract – there is no way third parties can customize the look of the clouds, so we can change the algorithm by which we create them pretty much at will. We could switch to a volumetric approach for all clouds or even go back to 2-d without worrying about third party interaction.
Okay – that’s a lot of words about contracts…what does that have to do with Vista? Well…
The Cost of Changing the Contract
Two major aspects of why Vista has been a worse experience for users than Windows XP come directly to the need for Microsoft to change contracts.
- For years, applications have run on Windows with admin rights. This is not good – it means that any process can do serious damage to the system if hijacked – and on Windows processes get hijacked on a fairly regular basis!
- For years, audio and video drivers have run pretty much unprotected. This was good from a performance perspective, but also caused a lot of BSODs. On Windows, drivers are third party components and are quite possibly not checked by Microsoft (especially video drivers) so letting them run without protections is risky.
In both cases, the problem is that the old contract is both (1) poorly designed* and (2) being used by a lot of third parties. What choice did Microsoft have? Continue to let apps run in admin mode and hijack the whole machine any time the user picked up a virus? Or change apps to run in user mode and hope that the applications didn’t depend on this guarantee?
(At this point,
Raymond might go ballistic and point out that the Windows API doesn’t really promise admin rights and apps should not have been doing all of these naughty behaviors in the first place. I don’t know what the Win32 API declares…the difference between what a platform declares and what it does is important enough to warrant another post. Certainly with X-Plane we have to worry a lot about third parties depending on behavior that was unintentional but turned out to be useful.)
Vista has been a difficult transition because it changed a bunch of rules (that needed to be changed). In the long term, I am sure that both of these decisions are for the better — eventually applications won’t be counting on administrator rights, so we won’t have to fume about UAC (or shut it off), and a hijacked web browser won’t be nearly as dangerous.
On the video driver front, the Vista experience is pretty reasonable now – there has been a lot of improvement since Vista first came out. I expect applications and UAC to take a lot longer – video drivers get revised quite frequently; applications seem to linger around forever.
I’m Not Signing That
If we end up with a situation like this in X-Plane (the contract is used heavily by third parties and not well designed) we only have two options, and they’re both bad:
- Break the contract. Third party content stops working, users are angry, authors are angry.
- Stick with the contract and mitigate as best we can. Usually this means writing more code (slows down new features), using a less optimal implementation (lowers frame rate), etc.
This is why my first reaction to any file format extension is: “is this going to be a PITA in a year”? The benefit might be visible now, but the cost could plague us indefinitely.
What You Want, Not Where You Want It
If you would like to request a feature, tell me what you want, not where you want it implemented. I bring this up because many of the feature requests I get are very specific and describe an implementation, not a goal. (To draw an analogy, it’s as if I call a general contractor and say “dig a big hole right here” without telling him “it is for a swimming pool”.)
The reason “what not how” is so important is because many of the “how” implementations that people send me involve creating new contracts with third parties. I am going to try to design the feature with the minimum contractual obligations – that is, to do just what is intended and hopefully not much more.
But if I can’t tell what you are trying to do, I can only say “I won’t code this implementation – the cost of long term support due to contractual obligations outweighs the usefulness.” It might be that there is another way to implement the feature that would not put a long term burden on the scenery system or airplane SDK and still provide all of the benefits.
* Poorly designed? Or perhaps well designed for a previous problem – if the problem changes, the design might not be appropriate. Or perhaps not even designed at all – sometimes contracts evolve without a lot of central planning. All of these things have happened in X-Plane. In the case of Windows, I suspect it’s the previous-problem case — that is, what made sense for much smaller computers where the scope of what could be done was quite limited no longer makes sense for big modern computers that are capable of a more expensive and robust solution…just my speculation.
Some coding problems are stubborn – I find myself looking back at a week of working realizing that all I really did was prove that a bunch of theoretical improvements don’t work in practice.
Improving OBJ throughput is one of those problems. On a high-end machine, even drastic changes to the OBJ engine make only the slightest difference in throughput – 2 or 3% at best. Every improvement counts, but a 3% improvement doesn’t change the game for how we draw scenery.
There is at least one route I haven’t had time to go down yet: object instancing. The theory is that by making many objects appear with only one object drawn, we get a multiplier, e.g. a 2x or 4x or larger amplification of the number of objects we can have.
In practice it won’t be that simple:
- To get such an amplification we have to recognize groups of the exact same object. Grouped objects will have to be culled together. So we might get a hit in performance as we draw more objects that are off-screen, just to use instancing.
- It may be that the grouping requirement is so severe that it is not practical to find and group objects arbitrarily (instead we would have to group objects that are built together, like clusters of runway lights). That might limit the scope of where we can instance.
- The objects have to look more or less the same, so some categories of very complex objects won’t be subject to instancing. (E.g. objects with animation where each object might look different.)
- I have already coded some experiments with geometry shaders, and the results are just dreadful – geometry shaders simply don’t output a huge number of vertices efficiently, so they don’t help us increase our total vertex throughput. The experience has left me with a “prove it” attitude toward GL extensions that are supposed to make things faster.
When will we know whether instancing can help? I don’t know — I suspect that I won’t be able to find time for code experiments for a bit, due to other work, particularly on scenery creation and tools.
I was going to post some pictures of the newly fixed “auto-vary” feature, but before I can do that in a way that makes any sense, I need to explain how X-Plane deals with texture repetition.
Texture repetition is the inevitable result of using “landuse-style” texturing (that is, a repeating single texture representing a type of land). Typical X-Plane land use textures are 1024 x 1024 at max res and repeat about every 3-5 km. Unfortunately, our brains are pattern-recognizing machines, and the result of this texturing scheme is that the “grid lines” of texture placement become apparent over wide views.
We use a number of techniques to minimize this problem.
Lots of Land Uses
Our main tool to combat repetition is to not use a given land use for too large of an area. This has the advantage of efficiently using the entire set of textures, and (because terrain textures change based on an irregular grid, based on elevation) the changes to textures are both irregular in shape and “plausible” in placement.
In this picture, you can see that the urban residential land use has been interrupted by various forest and grass textures. This is intentional – Sergio tuned hte land use rules to make sure that we wouldn’t have large regions of one land use type. Those blobs match the irregular grid, which gets its shape from the terrain’s elevation.
Variation
In the above picture, you can still see the repeating grid of the residential terrain; observe the right side – you’ll see the same repeating vertical pattern of road repeating over and over. In order to further hide repetition, we use the same texture multiple times, but in offset locations.
Here you can see that the vertical line on the right side has been broken up a bit.
More VRAM
In some cases, a terrain covers such large areas despite the rule set (e.g. for really flat areas) that we use two separate textures and can vary between them. Here you can see both input textures for our dry square crop land use, as well as the combined results.
In summary, we have three techniques:
- Add more rules to prevent large spans of a single land-use.
- Use a texture with multiple offsets (variation)
- Use two textures and vary between them.
The apt.dat 850 file format defines a polygonal “airport boundary”. But what exactly does it do? It does different things when creating DSFs and when rendering them.
Inside X-Plane it has relatively little effect:
- It is one of many elements that counts toward the land area that will be flattened. (Runways and taxiways are also used.)
- We do not actually render any special terrain or fences.
The airport boundary has more of an effect during scenery creation.
- If an airport has an airport boundary, we do not calculate the airport’s boundary ourselves – instead f we use the specified boundary.
- All land intersecting the airport boundary is turned to airport terrain.
- The DEM is flattened within the airport boundary to reduce the slope of high frequency bumps.
There is one aspect of airport creation that the airport boundary is not involved in: filling in water to make land under runways. When we create an airport, we actually calculate three boundary polygons:
- The inner ring is closest to runways and taxiways (very close) with very little simplification. It is filled in with land if it is wet.
- A second ring slightly farther from runways and taxiways is also filled in with land if it is wet.
- The outermost ring is a lot farther out, but does not fill in Water. This is the ring that the airport boundary can replace.
Why do we need two inner rings? Well, if an airport is next to the water but not at sea level, we need to induce two sets of mesh points, the outer ones which drop down to sea level and the inner ones which are at airport level. You can see the importance of this at KLGA, where one of the runways dropped to sea level at its midpoint in version 8, but not version 9.
Since the airport boundary polygon provides only one ring, it cannot be used for this purpose. At some point in the future, we might use the airport boundary for rings 2 and 3, or use a smaller version of the airport polygon for rings 1 and 2.
For now, my recommendation is: the airport boundary should trace out the entire airport premesis, not including water.
