X-Plane 10.30 Beta 3 Is Out

Beta 3 just went live.  Release notes are still here – the document grows!  When the beta is done I’m going to have it printed and use it as a weapon to squash house flies.  Bug reporter is still here.  Bug reporter is still not the comments section in this post.*

The short summary is: Philipp crushed a huge pile of GPS bugs, and I crushed none of the cloud-related bugs.  So please read the bug fix list carefully before you report.  We try to make sure there’s a bug list bullet for every single fix in the beta, so that you don’t have to waste your time (or ours!) re-reporting a bug that isn’t fixed.  If you reported (hypothetically :-) that cloud performance makes you weep (and not in a good way), there’s no need to re-report, I haven’t gotten to it yet.

If you have not tested your payware add-on with X-Plane 1030, please do so ASAP!  We do not have a pile of known issues with third party drawing, so if you see something, say something.  (We have a number of possible bugs that aren’t reproducible yet – so your bug report might be what lets us get to the bottom of things efficiently.  Only you can prevent forest fires!)

* This post comes with an extra helping of snarkiness…Chris and I made the really poor decision to try to debug part of the X-Plane airport gateway deployment from 11:30 PM to 3:30 AM land night.  Since we both have small children who didn’t get the memo and woke up at their normal time, I am, at this point, pretty much unaware of what I am typing.

  • Facebook
  • Reddit
  • StumbleUpon
  • Twitter
  • Google Buzz
  • LinkedIn
Posted in News | 31 Comments

X-Plane 10.30 Beta 2 is out

I’m a little slow at blogging this, but 10.30 beta 2 is out – beta 1 users received an auto-update notice.

Please report bugs here.  Starting now I am going to delete comments that are bug reports.  I really can’t be any more clear about this: please report bugs in the bug report form and not on the blog post.*

The release notes are here.  I broke up the bug fix section by beta so you can see what is new in beta 2.

I have already received reports that the “fix” to the slow local map is not actually a fix at all. Unfortunately my C drive lost sectors (again) so I am behind in investigating both this and general performance complaints.  I’ll need to rebuild the machine before I can get to these things.

* Why am I being such a jerk about this?  Two reasons:

  1. I don’t want bug reports to get lost.  Sending me a private email, posting on a blog, etc. — these are all recipes for bugs to get lost.
  2. Efficiency.  I just spent an hour triaging X-Plane 10.30 bug reports and that was only bugs that actually went into the bug reporter.  Those are the fastest bugs to triage because we have an efficient system to deal with them.  Every bug that is in the wrong place takes twice as long for me to deal with.

So bugs posted to forums, blogs, email, these are at best slowing down the beta process and at worst getting lost.  (Especially forums – since I don’t read them, they are by definition lost.)

And as a final rant: emailing with “should I file a bug about X” is inefficient for everyone. It takes as long for you to write the email as to file the bug.  It takes us time to read the email, answer, then get the bug.  Just file the bug!

  • Facebook
  • Reddit
  • StumbleUpon
  • Twitter
  • Google Buzz
  • LinkedIn
Posted in News | 24 Comments

Some Notes From Beta 1

The next X-Plane 10.30 beta should be out tonight (or maybe tomorrow morning); the release notes are already updated with the latest fixes, and we’re just waiting on upload.

There were some “boneheaded” bugs in beta 1 – things that just didn’t work, should have worked, and were easily fixed; we have addressed all of those that were reported for beta 2.  Despite about half a dozen boneheaded bugs, I actually think the quality of public beta 1 – relative to the amount of source change – was pretty strong.  Remember that 10.30 has perhaps 2x the amount of code change of a normal major patch, so if there isn’t too much nuclear then I think we’re doing okay.

Aircraft Authors: please test beta 2!  I think we may be able to keep 10.30′s beta period down to 4 weeks, which is a lot shorter than normal.  And the remaining bugs are all in clouds and performance, so please do not wait to test your aircraft.  Eugeny contacted me on problems with the A320 Neo on beta 1, and we have X-Plane fixed for beta 2. Don’t wait!

Performance: there have been all sorts of performance comments from users.  I think there are three separate things going on:

  1. For general use, X-Plane 10.30 is faster than 10.25.
  2. X-Plane 10.30′s clouds are definitely slower/more fill rate intensive than 10.25.
  3. There’s something wrong with some combination of AMD drivers and 10.30 that can cause total framerate death.

I am investigating both points 2 and 3, but neither are fixed for beta 2; we wanted to get some of the stupid things fixed ASAP (e.g. the ATIS not working).

To make matters worse, my PC’s SATA controller seems to have finally lost it’s mind. Philipp had to cut the Windows build, and this has halted me from investigating AMD performance problems.  (In my initial tests I couldn’t repro anything but now I’m stuck.)

Clouds: The other area of bug fixing on my plate that simply isn’t addressed in beta 2 is the cloud locations; several users have correctly reported that the bases of the clouds are simply not where they expect.  Since there is an unfortunate link between rendering settings and fixing this bug (if the puffs we use to visualize clouds change, it effectively moves the cloud bases) I have to fix both cloud performance and cloud bases at the same time.

The expected behavior for the clouds, I think, is that if you set a stratus layer with no storms and fly an ILS, you should “break out” (that is, make visual contact with the runway environment) right around the elevation set in the weather screen.  (This is not what is happening now, which is why there is an open bug.)  If you set more ‘creative’ weather, X-Plane will start to vary things and you won’t get “reliable” break-outs.

METARs: One last note on weather: if you find that X-Plane has misinterpreted a METAR (and my sympathy is with X-Plane, because METARs just contain the most random stuff in them sometimes), then please be sure to include in your bug report:

  1. The actual METAR.rwx file!  No METAR file, no way we can possibly debug.  To everyone who has sent METAR files, thank you!
  2. The location of the aircraft.  A fairly precise latitude/longitude, or a FIX or navaid that you are over is good.  The easiest is when you are at an airport and can send us an ICAO code.
  3. A description of what weather you actually got.

X-Plane 10.30 does have a new METAR parser so we’re trying to catch the bugs.

Airports and DSFs: we are looking at including both some fixed DSFs and additional 3-d airports in 10.30.  Both depend on the tech being ready; the airport gateway is in the deployment phase and if it goes smoothly, we could be ready to post airports.  I’m still trying to get to the bottom of DSF bugs, but I am close.  If either of these content updates miss 10.30, we’ll release them in a small patch as soon as possible.

  • Facebook
  • Reddit
  • StumbleUpon
  • Twitter
  • Google Buzz
  • LinkedIn
Posted in Development | 25 Comments

X-Plane 10.30 Public Beta 1 Is Available

The first public beta of X-Plane 10.30 is here.  To get it, go to update an installation of X-Plane and click the “get betas” check-box of the X-Plane installer or demo installer.  (As this is a first beta, you should probably run 10.30 on a copy of your main X-Plane folder; it’s premature to run 10.30 as your main install.)

Release notes here.  Bug reporter here.  Do not report bugs on the blog comments.

We do not read the various third party X-Plane forums.  If you find a bug and you discuss it on the org or avsim or fs.com for 10 pages and do not report the bug in the form, we do not know about it.  Please do not assume that someone else will report the bug – you’d be amazed how often everyone thought someone else would make the real report.* Also, while I’m ranting, please do not link to forums that require logins or memberships with bug reports; a forum discussion is not a clean bug report!

What’s In The Beta

The release notes are four pages long, so let me try to summarize what we’ve done.  There are really four major areas of this beta:

  1. The new GPS.  The new GPS is the driver of this beta – it’s why Austin called me from the airport this morning to find out “is it out yet”?  It’s the big news of the beta, and it’s something we’ve wanted to bring to X-Plane for quite a long time.
  2. A pretty long list of bug fixes and usability improvements.  We held back on a lot of these to get 64-bit out quickly, so with 10.30 we’re finally putting out the door a lot of work we’ve done.  There’s a lot of attention to detail in these fixes.
  3. Improvements in clouds and sky.  This work is only partly done – I have some fog work that is not in the beta because I did not want to hold the new GPS (see point 1) up.  It is even possible that if everything else is done, we do a 10.35 for fog or something.  The GPS and fog work are independent from each other (and driven by different engineers), so my thinking is that if one is done and the other is not, we should not hold back the GPS unnecessarily.
  4. Lots of under the hood changes.  This isn’t particularly interesting for users or authors, but it is necessary for the long term development of the sim.  I mention it to try to give you a sense of the scope of the changes.

Here are some numbers: from X-Plane 10.11 to X-Plane 10.25, we had 404 commits in GIT, including betas and patches.  From X-Plane 10.25 to X-Plane 10.30 beta 1, we have 1113 commits, and we haven’t even run the public beta.  This is a big release in terms of work!

Aircraft Authors

If you are making a third party aircraft, please do test your aircraft with 10.30 and report a bug if we broke something!  You can also begin work on supporting the new GPS.

But, for the love of all that is beta, please do not release 10.30 aircraft!  We’re in public beta and the way the new features work are likely to change.  During 10.20 we had developers release 64-bit plugins mid-beta, and they had to immediately redo their release because the beta changed.  The safe time to release a 10.30 aircraft will be when 10.30 goes final – no sooner!

(If you are releasing a new product during beta, release it for X-Plane 10.25, then please report a bug if it doesn’t work with the beta.)

* If you are part of a discussion with ten people on a website and you all see a bug, you can pick one person to be ‘the reporter’ – what drives me nutty is when everyone assumes someone else will report and, with no coordination, no one ever does.  There have been a number of times recently that I’ve discussed a bug with a third party author and heard “we’ve known about this on XYZ website for months.”  I’m sorry, but there are too many third party web-sites and not enough hours in the day for the dev team to scan every forum post for hidden bug reports.

  • Facebook
  • Reddit
  • StumbleUpon
  • Twitter
  • Google Buzz
  • LinkedIn
Posted in Development, News | 67 Comments

X-Plane 10.30 is Coming Real Soon Now™

I was going to write this post yesterday, and I’m glad I didn’t, because my estimated release time was off by a day.  But 10.30 is now cut and is being mirrored out to our servers.  If things go without hitch, we can hit “go” tomorrow.

In the meantime, you can read about what we did here - after a little bit of cleanup I was able to cut the notes down to ten pages.  Aircraft authors, you’ll want to read the docs on the new GPS and other cockpit issues.

(10.30 is probably better documented as an early beta than any previous beta we have done, both in terms of completeness and detail of changes and release notes, and in terms of new docs on new features.)

  • Facebook
  • Reddit
  • StumbleUpon
  • Twitter
  • Google Buzz
  • LinkedIn
Posted in Development, News | 24 Comments

Physics and Magic

Perhaps I am too optimistic, but: rendering solid, opaque objects that look realistic in real-time (read: for games and rendering engines) is rapidly becoming a solved problem. I look at the bleeding edge of the top-tier shooters as a preview of the direction for the field of real-time rendering*, and for the last year, physically based rendering (PBR) has been front and center.

I’ll write more about PBR in another post, but the short version is that, in PBR, the equations for light interacting with materials in a rendering engine are modeled more closely on real-world physical equations; the goal is something that is very accurate to real world optics, and produces realistic results over its entire range of inputs.

PBR is made possible by increased computing power on the GPU.  (And here when I say computing power I really do mean computing, or ALU power; PBR requires more math per pixel to calculate for a wider range of optical effects, often with more expensive but better approximations of the underlying optics.)  We now have big enough GPUs to do “real physics” (well, real-ish) when it comes to light and solid objects.  This is a huge step forward when you consider that a decade ago, calculating lighting equations per-pixel was marginal and had to be done in the wrong color space for speed.**

So for solid objects like airplane fuselages, runways, buildings, baggage trucks, etc., we have physics.

Then there are clouds, and the air, and rain and fog.  For those we have magic.

By magic, I do not mean something truly magical, something outside the realm of what is possible with computer graphics.  Rather, I mean the definition a professional magician like Penn Jilette might use: trickery.  To quote Penn***:

The only secret in magic, there’s only one, and that is, the secret must be ugly.  You cannot have a beautiful secret…So, in magic, what you want is an idea that is not beautiful…If I have to say, he’s lying about that, and there’s gaffer’s tape over behind there, and, ah, they’re not actually telling you the exact truth here, and it gets so…you don’t get an a-ha.

But there’s another side to magic: to make the trick look magical, the magician has to practice – the trick is a craft, and it can be finely honed.  That’s why, before Teller does the Red Ball trick, Penn has to tell you how it’s done; you can’t appreciate the workmanship if you don’t know how much workmanship went in.  The secret of the trick may be ugly, but the craftsmanship can be beautiful.****

Clouds, fog, atmospheric scattering – in X-Plane (and pretty much all rendering engines), these are not physics, they are magic, meaning they are tricks.  The sky is a hemisphere around your head, like in The Truman Show.  The color of the clouds are just painted onto textures with photoshop.

It’s trickery, but I feel no shame in these things, because the real physics are approximately a gajllion times more expensive than we can possibly afford on today’s hardware.  A single square meter of ground (on an overcast day) might be lit by 10^18 photons per second – a lot more when the sun comes out, and each of those photons can take a different path through the atmosphere, bouncing off molecules multiple times before arriving at a surface.  The actual optics of clouds and the atmosphere is incredibly complex.

The hard work and craft of the trickery is in making all of the tricks work together to make a unified effect, and this is where some of my time has been spent over the last few days.  It is frustrating work because there isn’t a right answer (the way there might be in the programming of physics equations); instead it is a series of tricks and compromises that try to make an illusion as convincing as possible given limited resources.

Whenever I describe the code behind atmospheric scattering, clouds and fog to Chris, he is always horrified by the complexity, by the layers of tricks.  But such complexity matches Teller’s the definition of a functional magic trick:

Make the secret a lot more trouble than the trick seems worth. You will be fooled by a trick if it involves more time, money and practice than you (or any other sane onlooker) would be willing to invest.

If clouds and scattering weren’t so much trouble, they wouldn’t be magic.

YouTube Preview Image

* The field of real-time rendering includes computer programs that produce graphics at high framerates, like games and CAD programs.  This contrasts it with the movie industry, where they can spend two years with a giant render farm to produce 90 minutes of final output.

** In 2004, we had the GeForce FX series and the Radeon 9700.  The 9700 was insanely powerful for its time, but it only had a 24-bit internal floating point path, a 96 instruction shader limit, etc.; you could do per-pixel calculations but burning shader instructions to work in linear space wasn’t on anyone’s mind.  The Geforce FX series was significantly less powerful.  This chapter wasn’t published until 2007 – 3 years later.

*** The link is to a Radio Lab story about a man who tries to understand how his grandparents performed a mentalism act on the radio – it’s a good listen!

**** In Fool Us, many times Penn and Teller knows exactly how the trick was done, but still praise the magician for their presentation – because they know how the trick was done, they know how hard it is to do it well.  Within the show, they seem to have a preference for tricks that are not easy and require skill to perform.

  • Facebook
  • Reddit
  • StumbleUpon
  • Twitter
  • Google Buzz
  • LinkedIn
Posted in Development | 7 Comments

No OpenSceneryX For the Airport Gateway

Robin, Tyler and I have been going through the airports that have been sent in since X-Plane 10.25; the good news is that there are over 350 batches sent in!*

But – I had to throw thirteen of them overboard; they were using OpenSceneryX.  For airports that are going to be sent to the gateway and become part of the default airport file for X-Plane, you can’t use third party libraries!  Everything you use has to ship with X-Plane.

WED 1.3 has user interface features to help hide third party libraries, but in this case you’d think that having the objects in the opensceneryx/ folder would have been a hint.

It’s not a coincidence that this was the same thirteen entries not sent in the right format; WED 1.2.1 won’t do an export for the global database with third party libraries, and this will be true with WED 1.3 as well, where the export is uploaded directly to the gateway.

  • Facebook
  • Reddit
  • StumbleUpon
  • Twitter
  • Google Buzz
  • LinkedIn
Posted in Development, Scenery | 15 Comments

Overwriting X-Plane Files Is Like Smoking In A Fireworks Factory

My previous post described the dangers of relying on art controls for an add-on that needs to be forward compatible with future versions of X-Plane; the short version is that art controls are an internal tool and not a public SDK, so we don’t try to maintain compatibility with updates.

So I figured I’d blog about the other dangerous way to build an add-on: creating an add-on by modifying X-Plane’s files.

All of the well-supported forward compatible ways to make add-ons (scenery packs, aircraft, and plugins) work by adding new files to X-Plane, usually in a folder specific to your add-on (a scenery pack, an aircraft folder, or a fat plugin).  This is very intentional – it ensures that add-ons don’t clash with our installer/updater over ownership of specific files.

But if you make an add-on that requires the user to replace one of our files with your own, all bets are off, and future patches of X-Plane will probably both screw up your add-on and be an updater nightmare for your users.  The best bet is to avoid modding our files.

Use the Library

The library system lets you virtually replace our files without actually touching them by mapping your own art assets to a given library path.  This is the right way to customize the scenery; editing the files on disk is not a good choice.  We may move files around within our own art assets in an update, but the library paths remain the same.  Thus building a library replacement is relatively safe; hacking the library scenery packs inside “Resources/default scenery” is not.

You should also use the library when you want to use our art assets; do not copy our art assets out of X-Plane into your scenery pack.  If you use the library, improvements to the art assets are automatically transferred to your scenery pack.

Some Dangerous Files

Here are some files that really aren’t meant to be modified in an add-on:

  • Our shaders
  • Random bitmaps in Resources/bitmaps, including sky and cloud textures
  • Lights.txt (the configuration file for named lights)

These files are all really part of X-Plane’s implementation and change regularly as we try to improve the sim.

You Can’t Share Naked Textures

You might notice that there is no way to directly use X-Plane’s textures in your add-on. At best you can use the .ter or .pol file that references them.

This is intentional!  Our artists may need to remap the location of elements within a texture as part of a future sim change, and when this happens, add-ons reference the texture would show the wrong part of the image.  That’s why add-ons can only reference some asset that itself references the texture; if we change the texture, we change our .pol files or .ter files or what-have-you to compensate for the UV map change.

X-Plane Is Not a Library

Austin and I get a steady stream of “can we just get access to X for our add-on” where X is some part of the sim that the add-on maker would like to re-use.  Unfortunately most cases where internals aren’t public are because we can’t safely share them; X-Plane is an application first and a library or platform second, and in many of these cases, to share these internal parts of X-Plane would be to commit to not ever making them better, which we don’t want to do.

  • Facebook
  • Reddit
  • StumbleUpon
  • Twitter
  • Google Buzz
  • LinkedIn
Posted in Development | 17 Comments

Art Controls Are An Active Volcano

I received an email yesterday that I found a little bit concerning; an author wrote this:

I hope 10.30 is not going to change any of the datarefs we as a community have identified and are using to make the sim better and if it does I will work to modify the scripts to make them work again.

Here’s the problem: his add-on is one of the new Lua scripts that changes the internal art controls for atmospheric scattering – it is almost certainly not built on public x-Plane SDK datarefs, it is built upon finding, locating, and hacking the internal art controls in the sim.

So I hope the author isn’t going to mind working to modify his scripts again, because those art controls will change.

Simply put, building an add-on based on art controls is like building a house next to an active volcano – it might be a nice place to live, but you’re going to have to rebuild your house from time to time.  Be prepared to do your work over, because art controls will change – it’s a matter of when, not if.  The rest of this post is some warnings, scare tactics, and an explanation of why the art controls change.

Scary Warnings, Etc.

In my past comments, both public and private, I have tried to be clear that the art controls are not a stable interface, they are subject to change, and that you can’t build an add-on that will keep working while we patch if you use them.  (I’d also hope that the fact that every art control starts with sim/private would get people’s attention.)  But I apparently never wrote one big public global “no warranty” message*, so…um…here we go.

The art controls are not a public interface to make X-Plane add-ons.  They are an internal development tool.  They are unsupported, undocumented, unsafe, and most importantly subject to change with every patch of X-Plane.  If you create an add-on that requires reading or writing the art controls, you can expect that your add-on will stop working when X-Plane is updated.  When your add-on breaks, please do not complain or file a bug.

So hopefully that makes things clear.  Art controls will change,you’ll have to do your work over, and we’re telling (and have been telling) you that since people first figured out how to access them.  If you learned about art controls from someone other than me and weren’t aware of this, the person who taught you forgot to tell you the most important thing.

What Is An Art Control

An art control is an internal hook into our engine that we use to tune and calibrate algorithms, measure performance, and debug problems.  I started adding art controls so that:

  • Our art team could “tune” X-Plane without having to have the source code or recompile the sim and
  • So I could see the results of changes to the engine without having to quit and restart X-Plane.

They are a critical part of X-Plane development.

They are not, however, some kind of public interface for third party development.  We have public datarefs, listed on the X-Plane SDK website, that let you replace the flight model, customize the control surfaces, etc.  Those datarefs are a real “interface” to the sim – even when we modify X-Plane, we try to keep that interface working.  Many of the public datarefs are over ten years old now.

By comparison, the art controls aren’t an interface – they are deeply tied to the how of the engine, e.g. the specific algorithm we are using today.  If we come up with a better algorithm, those tuning controls lose relevance and the art control goes away.

Why Don’t You Make the Art Controls Public

The immediate question that most developers have when they hear my rantexplanation of art controls is: why don’t you make the art controls public so we can use them?

It’s a good question, particularly when the art controls are useful.  The answer is that the art controls don’t represent a specification for what the sim does, they represent side-effects of the algorithm we are currently using.  Let me use atmospheric scattering as an example.

Atmospheric scattering is essentially a mathematical formula (run on the GPU) that makes the color of pixels “more blue” when they are farther away.  The art controls for scattering become constants in that formula – thus you can change the constants (but not the overall shape) of the formula via art controls.

But here’s the problem: the formula I use isn’t very good.**  The result is that when the constants are high (for “heavy” scattering) the far view turns a really rich, electric blue that just doesn’t look anything like the planet we actually live on.

As long as we keep this formula, we’re stuck with two options, neither of which are much fun:

  1. Keep scattering turned way down.  This is what we ship with; clearly users want more scattering – you can see that by all of the positive reactions to add-ons that crank the art controls up.
  2. Turn up the scattering and try to ignore the blue.  This is what most of the scripts that hack the scattering art controls do, and I would say it’s clearly a trade-off.

So I am working now on a better formula, one that won’t produce incorrect colors at high scattering levels.  But when that formula is done, it will be a different formula, with different constants in different places in the equations.

And that is why art controls change. The very code that gave birth to them changes, and they get deleted and replaced with new art controls.  The only way to keep the scattering controls forever is to never fix the flaws in the current formulas.

What Art Controls Will Change, and When

First, I hope that if I’ve made one thing clear, it’s this: if you want your add-on to keep working with future X-Plane patches, you need to avoid the art controls.

Realistically, the art controls tend to change when there is major development in the sub-system that they are part of.  So the shadow art controls will change any time we try to make the shadows better; the scattering art controls will change any time we try to make scattering better.

So if you are just tinkering with X-Plane, the good news is: if your art controls have changed, there’s a chance that you may be able to achieve a better effect using the new code.

One final note: if you want to build an add-on that needs to work past version updates (e.g. a payware add-on where having to follow our patch schedule is not a good idea) and the only way you can accomplish your goals is with art controls, send me an email.  We may be able to add a real interface to replace art control hacking.  This won’t always be possible, but sometimes it is, and the results will be a lot less work for your add-on in the long term and happier customers.

* See Resources/settings.txt for the warranty on hacking settings.txt.

** In my defense, the real look of the sky comes from a huge number of photons crashing into a huge number of molecules a huge number of times – we’re not even remotely close to having enough computing power to solve that honestly, so all scattering code is based on gross simplifications of gross simplifications.

  • Facebook
  • Reddit
  • StumbleUpon
  • Twitter
  • Google Buzz
  • LinkedIn
Posted in Development | 29 Comments

Catalyst 14-4 Drivers Are Good

A few weeks ago AMD posted a beta driver for newer Radeon HD cards that fixed some drawing artifacts with X-Plane.  They have now released an official, non-beta WHQL that works with X-Plane.  So if you have a newer AMD card on Windows, I suggest updating to the new Catalyst 14-4 driver.

  • Facebook
  • Reddit
  • StumbleUpon
  • Twitter
  • Google Buzz
  • LinkedIn
Posted in Hardware, News | Comments Off