Category: Development

Art Controls Are An Active Volcano

I received an email yesterday that I found a little bit concerning; an author wrote this:

I hope 10.30 is not going to change any of the datarefs we as a community have identified and are using to make the sim better and if it does I will work to modify the scripts to make them work again.

Here’s the problem: his add-on is one of the new Lua scripts that changes the internal art controls for atmospheric scattering – it is almost certainly not built on public x-Plane SDK datarefs, it is built upon finding, locating, and hacking the internal art controls in the sim.

So I hope the author isn’t going to mind working to modify his scripts again, because those art controls will change.

Simply put, building an add-on based on art controls is like building a house next to an active volcano – it might be a nice place to live, but you’re going to have to rebuild your house from time to time.  Be prepared to do your work over, because art controls will change – it’s a matter of when, not if.  The rest of this post is some warnings, scare tactics, and an explanation of why the art controls change.

Scary Warnings, Etc.

In my past comments, both public and private, I have tried to be clear that the art controls are not a stable interface, they are subject to change, and that you can’t build an add-on that will keep working while we patch if you use them.  (I’d also hope that the fact that every art control starts with sim/private would get people’s attention.)  But I apparently never wrote one big public global “no warranty” message*, so…um…here we go.

The art controls are not a public interface to make X-Plane add-ons.  They are an internal development tool.  They are unsupported, undocumented, unsafe, and most importantly subject to change with every patch of X-Plane.  If you create an add-on that requires reading or writing the art controls, you can expect that your add-on will stop working when X-Plane is updated.  When your add-on breaks, please do not complain or file a bug.

So hopefully that makes things clear.  Art controls will change,you’ll have to do your work over, and we’re telling (and have been telling) you that since people first figured out how to access them.  If you learned about art controls from someone other than me and weren’t aware of this, the person who taught you forgot to tell you the most important thing.

What Is An Art Control

An art control is an internal hook into our engine that we use to tune and calibrate algorithms, measure performance, and debug problems.  I started adding art controls so that:

• Our art team could “tune” X-Plane without having to have the source code or recompile the sim and
• So I could see the results of changes to the engine without having to quit and restart X-Plane.

They are a critical part of X-Plane development.

They are not, however, some kind of public interface for third party development.  We have public datarefs, listed on the X-Plane SDK website, that let you replace the flight model, customize the control surfaces, etc.  Those datarefs are a real “interface” to the sim – even when we modify X-Plane, we try to keep that interface working.  Many of the public datarefs are over ten years old now.

By comparison, the art controls aren’t an interface – they are deeply tied to the how of the engine, e.g. the specific algorithm we are using today.  If we come up with a better algorithm, those tuning controls lose relevance and the art control goes away.

Why Don’t You Make the Art Controls Public

The immediate question that most developers have when they hear my rantexplanation of art controls is: why don’t you make the art controls public so we can use them?

It’s a good question, particularly when the art controls are useful.  The answer is that the art controls don’t represent a specification for what the sim does, they represent side-effects of the algorithm we are currently using.  Let me use atmospheric scattering as an example.

Atmospheric scattering is essentially a mathematical formula (run on the GPU) that makes the color of pixels “more blue” when they are farther away.  The art controls for scattering become constants in that formula – thus you can change the constants (but not the overall shape) of the formula via art controls.

But here’s the problem: the formula I use isn’t very good.**  The result is that when the constants are high (for “heavy” scattering) the far view turns a really rich, electric blue that just doesn’t look anything like the planet we actually live on.

As long as we keep this formula, we’re stuck with two options, neither of which are much fun:

1. Keep scattering turned way down.  This is what we ship with; clearly users want more scattering – you can see that by all of the positive reactions to add-ons that crank the art controls up.
2. Turn up the scattering and try to ignore the blue.  This is what most of the scripts that hack the scattering art controls do, and I would say it’s clearly a trade-off.

So I am working now on a better formula, one that won’t produce incorrect colors at high scattering levels.  But when that formula is done, it will be a different formula, with different constants in different places in the equations.

And that is why art controls change. The very code that gave birth to them changes, and they get deleted and replaced with new art controls.  The only way to keep the scattering controls forever is to never fix the flaws in the current formulas.

What Art Controls Will Change, and When

First, I hope that if I’ve made one thing clear, it’s this: if you want your add-on to keep working with future X-Plane patches, you need to avoid the art controls.

Realistically, the art controls tend to change when there is major development in the sub-system that they are part of.  So the shadow art controls will change any time we try to make the shadows better; the scattering art controls will change any time we try to make scattering better.

So if you are just tinkering with X-Plane, the good news is: if your art controls have changed, there’s a chance that you may be able to achieve a better effect using the new code.

One final note: if you want to build an add-on that needs to work past version updates (e.g. a payware add-on where having to follow our patch schedule is not a good idea) and the only way you can accomplish your goals is with art controls, send me an email.  We may be able to add a real interface to replace art control hacking.  This won’t always be possible, but sometimes it is, and the results will be a lot less work for your add-on in the long term and happier customers.

* See Resources/settings.txt for the warranty on hacking settings.txt.

** In my defense, the real look of the sky comes from a huge number of photons crashing into a huge number of molecules a huge number of times – we’re not even remotely close to having enough computing power to solve that honestly, so all scattering code is based on gross simplifications of gross simplifications.

A New Particle System Is In the Roadmap

I think I have randomly mentioned this to various developers, so I might as well randomly mention it to all developers.  A new particle system for X-Plane is in our road-map – that is, a particle system where authors can control the properties and graphics in detail (similar to what you’d get in one of the AAA game engines).

If you have feature requests or ideas for things you need for a particle system, please write something short and coherent and email it to me; I can file it with my design notes for later integration.

X-Plane Platform Breakdown

When X-Plane checks X-Plane.com for updates, it calls the server with an identifier for itself that contains a little bit of information about the host machine it is running on: operating system type and version, whether it is the 64-bit or 32-bit version, and whether it is running as a demo.*  (X-Plane does not send any personally identifying information about you, but the server can see your IP address because all servers can see the IP addresses of all incoming network traffic.)

I sometimes get asked by third party developers: what percentage of users are using 64-bit, or what percentage of users are on Mac or Linux.  So I wrote a script to analyze the incoming data and break it down by platform, etc.  Here are the results.  (I have done this kind of analysis before, but this is the first time I wrote a good script to remove possibly confounding results.)

Platform Breakdown

The data set consists of 24,917 unique IP addresses that ran a non-demo, global X-Plane 10.25r1 in the last week of 2013.  This excludes users who have the regional version, a demo, don’t have their DVD in the drive, are running an old version, or who don’t have net connectivity.  So that’s a big enough sample to get good data, even though it’s only a fraction of the total X-Plane 10 copies sold.  Here’s the platform break-down:

IBM: 65.7%
APL: 32.2%
LIN: 2.0%

This matches  the number we’ve seen ever since FSX development was halted: growing market share for Windows. (We used to be 60-40 Windows/Mac back in the day).  Since X-Plane 10 is selling better than X-Plane 9, I believe that what we are seeing is growth, and the growth is disproportionately among Windows users.  The Linux share appears to have shrunk, but it’s hard to tell since past data wasn’t as carefully analyzed.  (The highest percent I have ever seen for Linux was 5%, but from this data I estimate the error bars on old data might be +/- a few percent, so who knows.)

64-Bit Adoption

Here’s the rate of 64-bit adoption for each OS.

All OSes: 82.0%
APL: 85.1%
IBM: 80.7%
LIN: 72.7%

It doesn’t surprise me that OS X has the highest 64-bit adoption – every Mac is running a 64-bit operating system and OS X has the least available address space.  What does surprise me is that Linux has the lowest 64-bit adoption rate, since Linux users have had the strongest desire for 64-bit.  (This desire is, I think, driven by the difficulty of setting up cross-64/32-bit operation on modern distributions.)

Operating System Breakdown

We don’t have a break-down of Linux distros or Kernels – the Linux version of the sim doesn’t report that, but we do have operating system versions for Windows and Mac.  On Windows the _32 and _64 bit suffixes tell whether the user is running the 32 or 64 bit “edition” of the OS.

The numbers include the 64-bit adoption rate for that particular OS; naturally the 64-bit adoption rate on the 32-bit editions of Windows is 0% because those OSes can’t run 64-bit X-Plane.  Fortunately 64-bit editions of Windows are becoming the norm – of the users running Windows 8, over 98% have the 64-bit edition.  On OS X, every version of the OS is 64-bit capable.

Windows:
5.1_32: 1.8% (64-bit: 0.0%)
 5.1_64: 0.2% (64-bit: 0.0%)
 5.2_64: 0.1% (64-bit: 100.0%)
 6.0_32: 0.9% (64-bit: 0.0%)
 6.0_64: 0.8% (64-bit: 77.8%)
 6.1_32: 4.1% (64-bit: 0.0%)
 6.1_64: 66.0% (64-bit: 87.2%)
 6.2_32: 0.4% (64-bit: 0.0%)
 6.2_64: 25.7% (64-bit: 87.3%)
OS X:
 10.6.5: 0.1% (64-bit: 42.9%)
 10.6.6: 0.0% (64-bit: 100.0%)
 10.6.8: 8.8% (64-bit: 78.9%)
 10.7.0: 0.1% (64-bit: 28.6%)
 10.7.2: 0.1% (64-bit: 100.0%)
 10.7.3: 0.0% (64-bit: 100.0%)
 10.7.4: 0.1% (64-bit: 81.8%)
 10.7.5: 11.3% (64-bit: 78.7%)
 10.8.0: 0.0% (64-bit: 100.0%)
 10.8.1: 0.0% (64-bit: 0.0%)
 10.8.2: 0.4% (64-bit: 88.2%)
 10.8.3: 0.3% (64-bit: 88.0%)
 10.8.4: 0.7% (64-bit: 84.7%)
 10.8.5: 15.0% (64-bit: 82.0%)
 10.9.0: 8.0% (64-bit: 84.3%)
 10.9.1: 54.7% (64-bit: 88.5%)
 10.9.2: 0.2% (64-bit: 100.0%)

Hopefully this is useful for third parties in deciding what operating systems and platforms to support.

* This is a standard practice – the update check runs over HTTP, just like your web browser.

Let’s All Try to Fight Log Spam

In a past life, before I worked for Laminar Research, I was a plugin developer.  So I’ll admit up-front that I might be the most guilty individual when it comes to this particular problem: log spam.

What do I mean by log spam?  By log spam I mean printing information to Log.txt that isn’t necessary.

Log.txt exists so that we (and third party developers) can look at a single machine-generated file and get a strong idea of what a user did with X-Plane for diagnostic purposes.  Therefore log.txt needs to contain three kinds of information:

1. Basic configuration information.  We print your graphics card, your OS, etc.  (In 10.30 we’ll even get your CPU type right on OS X. 🙂  It’s more reliable to log this than to send a user into ten different control panels to gather information.
2. Very basic event logging, e.g. what did the user do?  Where did he start his airplane, what scenery was loaded.
3. All abnormal conditions, including error alerts that came up.

In particular, what we do not need to do is log the detailed blow-by-blow progress of loading in excruciating detail.

If I may use my own sins as an example, older versions of XSquawkBox shipped with debug code on that dumped the contents of its entire ICAO dictionary – thousands of lines of text.  This information isn’t configuration (it’s not unique to particular installs), it isn’t an event – it is part of the event of loading XSquawkBox, and its not an error condition. What a mess!  (Fortunately Wade has fixed this and turned such logging off in the newest XSB betas.)

Programmers write this kind of logging code to help trace program execution, but it’s important to turn it off – if we don’t, the log files get so verbose that everyone spends a ton of time just fishing through ten tons of junk.

Some more guidelines:

• Log the presence of files or sub-modules that are user-configurable, but do so briefly.  It’s reasonable for SASL or Gizmo to list scripts that do not come with the base package, or for XSquawkBox or Pilots Edge to list CSLs found.  (This information lets a developer see what a user has done to modify the plugin.)
• Don’t log the presence of required files unless an error condition occurs.  An empty log can simply mean “everything went okay.”  (The sim will already confirm successful plugin loading on a per-plugin basis.)
• Log rare events, e.g. a flight starting, a network connection being opened.  Don’t log events that occur at high frequency, like receipt of a single network packet.

X-Plane is guilty of log-spam too; I’m slowly trying to cut down on this.  For example, spurious autogen-tree warnings are now gone from the log in 10.25r1.

Support Debugging

Sometimes you need all of that logging.  My suggestion is: if your add-on is complicated, make a specific interface to get verbose diagnostics that users can turn on as needed.  For example, X-Plane has an option in “operations and warnings” to dump verbose play-by-play network data to Log.txt.  When you enable this, you can get a ton of information about network operations, but the log will also be totally over-run.  The option is meant only for developers who are integrating with X-Plane’s UDP networking, and thus it has to be turned on by hand.

The OBJ engine actually also has such a trick – if you put the single word DEBUG (all caps) at the end of your .obj file, the entire OBJ contents will be output in detail to Log.txt.  Again, this is a ton of information, available when needed, but off by default.

You can support verbose logging in your add-on with a script variable, a dataref, a change to a preferences file.  Pick an easy mechanism, so that you can have users turn on verbose logging only when they need it.

Not Everyone Has Grep

Finally, a quick aside to my fellow professional programmers.  You might be rolling your eyes and going: “Ben you idiot, it’s easy to filter the log.  Everyone should just prefix their log entry and use grep.”

If we were all programmers, I would agree 100%.  But please understand that aircraft and scenery authors have to look through the log to get diagnostic information about art-asset problems, and this is an audience that is sometimes not comfortable with command-line tools, huge 25MB text files, massive amount of text, etc. So we are trying to keep the log.txt more human-readable than, for example, dmesg on Linux.

SkyMaxx Pro Draws in 3d; You Can Too

X-Aviation just posted an update to SkyMaxx Pro – the new 1.1 patch brings big performance improvements and fixes rendering problems with HDR. From the user reports I’ve read, performance in HDR mode is good.  (I hate to see users have to pick between HDR and third party add-ons; we want HDR to be the basis for superior next-generation aircraft and scenery.)

Getting a plugin that draws in 3-d to work with HDR mode requires some caution; the plugin APIs for drawing were designed in X-Plane 6, when ‘deferred rendering’ didn’t even exist as a concept.

I have updated the X-Plane SDK tech-note on 3-d drawing to contain modern guidance on how to cope with 3-d drawing callbacks.  In particular, you get called twice when your plugin requests to draw in 3-d:

1. The first callback is for drawing “solid” stuff that will be affected by spill lights; the only safe thing you can do from this callback is to call XPLMDrawObject.
2. The second callback is after lights have been mixed; at this point normal OpenGL drawing works reliably (albeit without spill lights being mixed in).  This is the time to draw translucent prop discs, coach marks and labels, clouds, smoke and particle systems, etc.

If your plugin does any 3-d drawing (e.g. custom particle system drawing or any kind of effects code), please review the tech note, and email me if you have questions.  The next-gen CSL code sample that is linked from the article is tested and works correctly too.

HDR Isn’t Just HDR: What Does That HDR Check-Box Do?

The HDR check-box turns on HDR mode in X-Plane 10. But what is HDR mode?

High Dynamic Range( HDR) mode in X-Plane actually enables many features at the same time. We put in a single check-box because the features all fit well together, and we wanted to keep the interface simple.

• High-dynamic range rendering.  In HDR mode, the world is rendered across a wider spectrum of brightness (just like real HDR photography)*, to better capture lighting levels.  The result is then compressed to low dynamic range (which is what your monitor does).
• Tone mapping with real-time exposure.  The HDR image is converted to LDR via a tone mapping algorithm that attempts to preserve the character of the image while keeping the dynamic range within limits of your screen.  The level of exposure changes in real-time: if you look at a bright scene and then duck the camera behind a building, you’ll see your eyes slowly adjust up, then everything gets dark when you stare at the sun.  (The level-adjustment algorithm runs entirely on the GPU for performance.)
• X-Plane uses deferred shading for all rendering except for translucent glass in aircraft. Deferred shading is a 2-pass approach to rendering that is commonly used in today’s shooter games, because it allows us to efficiently draw huge numbers of real 3-d spill lights.  In the case of X-Plane, you get a lot more night lighting.
• The individual puffs in our 3-d clouds have soften intersection with the terrain to avoid ugly cloud-ground collisions in low-cloud conditions.  (See page 1 and 4 of the article for before/after pics.)
• We emit heat blur from the engine particles.  The ability to add blur effects is a capability of the deferred rendering engine; something I’d like to do more with in the future.
• Optional atmospheric scattering.  This effect enables a more advanced lighting and fog shader that runs on the GPU and models the diffusion of light through the atmosphere as it hits oxygen and water particles.  Atmospheric scattering is what makes the far view ‘bluer’ than the near view.

So the moral of the story is: the HDR check-box actually enables a lot of effects!

Why Did You Put Everything in One Check-Box?

We had a few reasons to put a lot of effects into one bucket:

• Some of those effects require other effects.  For example, you can’t have the heat blur or soft cloud puffs without the deferred renderer.
• We wanted to keep the UI simple.  Our view is that the rendering settings screen is way too complicated and hard to tune, something we’re still looking to improve.  So any time we add features, we have to ask: can we make this simpler?**
• There are fewer combinations for us to test if there are fewer settings.  Sometimes we can have bugs based on combinations of features, so having the features all enable at the same time simplifies development of testing, which is extra important when we have to test GPU features against different hardware and drivers already.

* GPU nerds: The final render surface is in 16-bit floating point format.

** If you really want to get tweaky, most of the sub-effects within HDR mode can be accessed by settings.txt.  But if you edit settings.txt and your computer transforms into an alien cyborg and destroys humanity, don’t hold me responsible.