Cockpit Panel Regions, the Untold Story

A third party developer recently asked me what the purpose of panel regions was, and how they were meant to be used.

Now I blogged about this before, and if you’ve read the post you are probably thinking: “Ben, were you high when you came up with panel regions?”

Sadly, the answer is no, but that doesn’t make panel regions any less weird.  This post explains what happens.  (Sound nerds: see the end for the moral of the story.)

One thing to note: the panel texture is expensive.  Because the sim re-draws it every frame, it has to sit in VRAM all of the time – thus it puts more pressure on VRAM than regular OBJ textures (which can be paged out when not used).  No matter how you do your panels (the old way, the byzantine way, or the new way), make your panel texture as small as you can – it’s worth it!

The Short Version

The short version is this:

• Always use the 3-d panel.
• Keep your 3-d panel as small as possible – pack it tight!
• Make your 3-d panel a power of 2 in each dimension, e.g. 2048 x 1024.
• Use one cockpit region in your object, that matches the size of the 3-d panel.

The long version explains how we got here.

You Might Ask Yourself, “How Did I Get Here?”

In order to understand how we ended up in this situation, remember that panel regions were invented before the 3-d panel existed, and they were invented for X-Plane 9, which runs on some fairly old video cards.

Going into X-Plane 9, we wanted to simultaneously fix a bunch of problems with using panels as textures for 3-d cockpits.

1. The 2-d panel supports alpha; the technology to do this was at the time expensive and hurt performance of the 3-d cockpit as a whole.
2. The 2-d panel contains a lot of wasted space.  The texture space dedicated to the cockpit windows is not useful in a 3-d model.
3. The 2-d cockpit might be huge; version 9 introduced 2048 x 2048 panels.  Authors have a motivation to make the 2-d panel huge to cover a large monitor.
4. The 2-d cockpit includes 2-d spot light and shadow effects, which are expensive to prepare for the 3-d panel texture.
5. The 2-d cockpit might not be a power of 2 – at the time, this meant wasted VRAM.
6. The panel texture was drawn in 3-d by simply drawing the 2-d panel.  Thus the panel textured area of 3-d cockpits didn’t respond to real-world directional lighting.

The panel region system fixed all of these things:

• A panel region is a sub-area of the panel, and it must be a power of 2.  This fixes items 2, 3, and 5.
• A panel region ignores alpha and ignore spot lights.  This fixes items 1 and 4.
• A panel region is drawn with correct additive lighting, fixing item 6.

In other words, ATTR_cockpit_region fixed everything wrong with ATTR_cockpit.

3-D Panels Make For Chaos

It’s about at this point in our story that things start to go south design-wise.  At the time (near version 9 beta) the panel system was in sort of a never-never land, not really my code, not really Austin’s code, being driven by one-off requests.  There was no road map.

Shortly after we got the panel region system working, we added the “3-d panel”.  The 3-d panel is an entirely separate parallel panel whose only role was to be “the panel texture” for 3-d cockpits.  We realized we needed this because often the 2-d cockpit is useless for 3-d texturing.

Consider the overhead panel.  If you have an overhead panel in 2-d, it’s going to be drawn at a crazy perspective, to make it look sane in a forward view.  But to texture a real 3-d overhead panel, you need an orthographic texture, taken straight on.

Once we had a 3-d panel, a lot of the reasons for panel regions were gone.  The 3-d panel could be tightly packed, with no windows, no alpha, and it didn’t need to be enormous.  In other words, the 3-d panel solved most of these problems better than panel regions did.

Had I used my brain, at this point I would have simply merged the two features together and gone home and life would have been really simple.  But instead, I let the two features co-exist.  This makes for four combinations – we can have the 3-d panel without panel regions (generally a poor idea) and we can have the 2-d panel with panel regions (which is arguably unnecessary).  And I let the 3-d panel be as unrestricted on the 2-d panel, which was just silly.

It should also be noted that having no alpha in the panel and a power of 2 panel size aren’t as important as they used to be, and can be overcome with modern hardware.  They were important for version nine though.

So the bottom line is: you still have to use a panel region to tell X-Plane to use real additive lighting, and to guarantee that you’re going to use a power of 2, no-alpha texture.  But the “region”-ness (the ability to pick out parts of the panel) isn’t useful since you can now create a separate packed panel for 3-d cockpits.

Thus the recommended practice: use the 3-d panel, pack it tight, and use one region to get additive lighting.

Clean-Up?

Can anything be done to simplify and clean up the panel creation process?  I’m not sure.

For the authoring environment, there are three things we can do to make life easier for panel authors:

1. Introduce new commands that are simpler.  For example, we could set up a single OBJ command that uses the entire 3-d panel with additive lighting – this command would provide the interface that always should have been there.  Of course, this would be yet another command, making the “full” system (including backward compatibility) even more confusing.
2. Simply change the behavior of existing commands.  I am always hesitant to do this, because it has the potential to break older airplanes.  But the current system does provide a number of feature combinations that make very little sense.  There might be some things that we could do that could simplify the system for authors. (For example, we could use new modern correct additive lighting even for ATTR_cockpit if the 3-d panel is used, since it’s almost always the right thing to do.)
3. We can provide Plane-Maker “hints” to encourage people not to use the strange un-recommended paths.  This would hopefully start to shape the set of airplanes to use the features we want without breaking any airplanes.

None of these options is great, but my gut feeling is that if we start with 3, we might be able to go to 2 – that is, we can start encouraging people to use the 3-d panel with one full-size cockpit region, the preferred way to use the system.

Epilogue: What Have We Learned?

If there’s a moral of the story it’s this: when it comes to features that affect how third party content is created, think twice, cut once.  Once you design a bizarre interface and everyone starts using it, it’s nearly impossible to fix, and the “cost” of maintaining backward compatibility in the sim goes up for years.

This is why I have said no to any and all incremental proposals for sound features in X-Plane.  There is no question we need better sound support.  But I believe that the future format of sound is (1) not particularly obvious in terms of today’s sound system and (2) not likely to look anything like what we have now.

If we put in incremental sound features one at a time, every one of them will have to be heavily modified to work with future sound features.  There’s no easy way to add just a little bit here, just a little bit there.  In other words, it’s the panel system all over again.

When there is a mature, modern design, it is possible to add incremental features quickly and cheaply.  This is the case with OBJs – we’ve been able to add new features (mostly in the form of new attributes) one at a time and it more or less just works, because the basic OBJ design is solid, and adding new attributes doesn’t radically change the system.  The same thing is true for adding new properties to generic instruments.

But when a feature fundamentally changes the structure of authoring, there’s a big penalty for not getting it right the first time.  The panel system continues to be a pain in the ass to maintain and confusing to use because some major features were thrown together ad hoc without a good plan.  I don’t want the sound system to be like that.

NVidia Driver Update

In a previous post I discussed some of the hang-on-start problems we were seeing on NVidia cards.  Since these started relatively recently, but the sim code hadn’t changed much, we thought maybe there was a driver issue.

Since then we’ve collected a few more data points:

• Some users with older cards are seeing problems starting X-Plane – there are shader-compile errors in the log file when this happens.  Our shaders haven’t changed.
• Some users with newer cards see rendering artifacts: typically the cloud shadows will darken some terrain triangles, but not others, which make the terrain look very weird with broken cloud cover.
• Some users still see a hang on startup which is “fixed” with –no_fbos.

Now here’s the interesting thing: the shader compile errors and rendering artifacts appear to be happening with the 270.61 drivers; going to the 260.99 drivers seem to help.

So if you have NVidia hardware, there are a few data points I am looking for.  Please post a comment if you meet any of the following:

• If you have the 270.61 drivers and see either visual corruption, crashes or hangs, please try going back to 260.99 and post your results, including what card you have.
• If you have the 270.61 drivers and a new card (GeForce 400 or 500) and you do not see corrupt cloud shadows, please post so – we need to know if the bug only affects some users.
• If you have the 270.61 drivers and an old card (GeForce 7xxx or older) and you can run without crashing, please post so – we need to know if the crash bugs only affect some users.

As always, we don’t know if this is a driver bug or an X-Plane bug, and who “fixes” the bug may not be an indication of whose fault it is.  We will work around the bug even if it is in the drivers, and sometimes this kind of problem is due to X-Plane doing something naughty that some but not all drivers tolerate.  We won’t know if this is really a driver bug until we have a full diagnosis.

Edit: These bugs are Windows/Linux specific; Macintosh users will not see them, nor do they have driver version numbers that match the normal NVidia scheme.

That’s One Powerful Landing Light

A while back I described some of the new lighting features in version 10, including lighting calculations done in linear space.  The very short version of this is: X-Plane 10’s lighting will be more physically realistic, and this is important when you have a lot of lights shining all over the place.

The change doesn’t affect how you author content (you still texture your models and we shine the sun on them) but it has been a source of bugs, as we find all of the different parts of the sim that use lighting equations.  In the picture on the right, the landing light hasn’t been updated, and as a result, it lights up the city 2 miles away.  The picture on the left has the landing light turned off.

I kind of like the right hand version, but that’s why I’m not in charge of the overall “look” of the sim.

(Will the runway lights look that “splattery”?  Probably not, but I don’t know; Alex will be in charge of the final decision.  That picture is zoomed in, which makes the lights look a lot bigger, but also the tuning of the runway lights for size is only partially done right now.)

Why not GPGPU?

A commenter asked if we were planning to use a GPGPU API in X-Plane 10 or beyond.  I really can’t answer for the far future, but I can say that we aren’t planning to use GPGPU for X-Plane 10.  This post will explain a little bit about what GPGPU is and why we haven’t jumped on it yet.

Every new technology adoption has an adoption cost.  So the question of GPGPU isn’t just “will it help” but “is it better than the alternative uses of our time”.  For example, do we spend time coding GPGPU, or do we spend time optimizing the existing code to run faster on all hardware?  But this post is about GPGPU itself.

GPGPU stands for General Purpose programming on Graphics Processing Units – the Wiki article is a bit technical, but the short of it is: graphics cards have become more and more programmable, and they are highly powerful.  GPGPU technologies allow you to write programs that run on the GPU other than graphics.

There are two major APIs for writing GPGPU programs: OpenCL and CUDA.  OpenCL is designed to be an open standard and is heavily backed by Apple and ATI; CUDA is NVidia specific.  (At least, I don’t think you can can get CUDA to run on other GPUs.)  I believe that NVidia does support OpenCL with their hardware.  (There is a third compute option, DirectCompute, that is part of DX11, but that is moot for X-Plane because we don’t use Windows only technologies.

If that seemed confusing as hell, well, it is.  The key to understanding the alphabet soup is that there are API standards (which essentially define a language for how a program talks to hardware) and then there are actual pieces of hardware that make applications that use that language fast.  For drawing, there are two APIs (OpenGL and Direct3D) and there are GPUs from 2+ companies (ATI, NVidia, and those other companies whose GPUs we make fun of) that implement the APIs with their drivers.

The situation is the same for GPGPU as for graphics: there are two APIs (CUDA and OpenCL) and there is a bunch of hardware (from ATI and NVidia) that can run some of those APIs.*

So the question then is: why don’t we use a GPGPU API like OpenCL to speed up X-Plane’s physics model?  If we used OpenCL, then the physics model could run on the GPU instead of on the CPU.

There are two reasons why we don’t use OpenCL for the physics engine:

1. OpenCL and CUDA programs aren’t like “normal” programs.  We can’t just pick up and move the flight model to OpenCL.  In fact, most of what goes on in the flight model is not code that OpenCL would be particularly good at running.
2. For a GPGPU program to be fast, it has to be running on the GPU.  That’s where the win would be: moving work from the poor CPU to the nice fast GPU.  But…we’re already using the GPU – for drawing!

And this gets to the heart of the problem.  The vast majority of the cost of the flight model comes from interaction with the scenery – a data structure that isn’t particularly GPU-friendly at this point.  Those interactions are also not very expensive in the bigger picture of X-Plane, particularly when the AI aircraft are threaded.

The biggest chunk of CPU time is being spent drawing the scenery.  So to make X-Plane faster, what we really need to do is move the graphics from the CPU to the GPU – more time spent on the GPU on less time on the CPU for each frame of drawing we run through.

And the answer for why we don’t use OpenCL or CUDA for that should be obvious: we already have OpenGL!

So to summarize: CUDA and OpenCL let you run certain kinds of mathematically intense programs on the GPU instead of the CPU.  But X-Plane’s flight model isn’t that expensive for today’s computers.  X-Plane spends its time drawing, so we need to move more of the rendering engine to the GPU, and we can do that using OpenGL.

* Technically, your CPU can run OpenGL via software rendering.  The results look nice, but aren’t fast enough to run a program like X-Plane.  Similarly, OpenCL programs can be run on the CPU too.

Real Physics on an Airplane

A while ago I wrote two posts trying to explain why we would use real physics for the AI planes.  Looking back over the comments, I think my message missed the mark for a number of readers.  The basic idea was this:

• It’s quicker to get the new ATC system done by using the existing physics model than by inventing a brand new, parallel “fake physics” model for AI planes.  So using real physics lets us focus on other features.
• The physics model is not more expensive than a fake physics model would; the few things that actually take CPU time on the real physics model are things the fake model must do: check for ground collisions, etc.

In other words, using real physics doesn’t hurt the schedule and it doesn’t hurt fps.  I followed that up with a bunch of talk about how you incrementally build a complex piece of software, and off we went.

What I didn’t do is make the argument for why real physics might be better than fake physics.  So: I made a video of the 777 taxiing and taking off.

Some disclaimers: this isn’t a marketing video, it’s what was on my machine at this instant.  This is an older v9 plane with v9 scenery*.  Version 10 is in development and has plenty of weird stuff going on, and the AI still needs a lot of tuning. Anyway:

With the video compression, calm conditions, v9 airplane, etc. it’s a bit tough to see what’s going on here, but I’ve seen a few AI takeoffs as I run X-Plane in development and it seems to me that the real physics model provides a nuance and depth to how the planes move that would be impossible to duplicate with a “fake” AI (e.g. move the plane forward by this much per frame).  When the airport isn’t flat, the plane sways depending on its landing gear, weight, and wheelbase.  The plane turns based on its rotational inertia, easing into the turn (and skidding if Austin dials in too much tiller).  When the plane accelerates, the rate of acceleration takes into account drag, engine performance, and wind.

* Except for Logan – that’s George Grimshaw’s excellent KBOS version 2 – version 3 is payware and I hope they’ll some day bring it to X-Plane.  Unfortunately there is some Z-thrash in the conversion.

Hardware Buying Advice (or Lack Thereof)

If I could have a nickel for every time I get asked “should I buy X for X-Plane 10”, well, I’d at least have enough nickels to buy a new machine.  But what new machine would I buy?  What hardware will be important for X-Plane 10?

The short answer is: I don’t know, it’s too soon.  The reason it’s too soon is because we have a lot of the new technology for version 10 running, but there’s still a lot of optimization to be done.

As I have posted before, the weakest link in your rendering pipeline is what limits framerate.  But what appears to be the weakest link now in our in-development builds of X-Plane 10 might turn out not to be the weakest link once we optimize everything.  I don’t want to say “buy the fastest clocked CPU you can” if it turns out that, after optimization, CPU is not the bottleneck.

One thing is clear: X-Plane 10 will be different from X-Plane 9 in how it uses your hardware.  There has been a relatively straight line from X-Plane 6 to 7 to 8 of being bottlenecked on single-core CPU performance; GPU fill rate has stayed ahead of X-Plane pixel shaders (with the possible exception of massive multi-monitor displays on graphics cards that were never meant for this use).  X-Plane 10 introduces enough new technology (instancing, significantly more complex pixel shaders, deferred rendering) that I don’t think we can extrapolate.

To truly tune a system for X-Plane 10, I fear you may need to wait until users are running X-Plane 10 and reporting back results.  We don’t have the data yet.

I can make two baseline recommendations though, if you are putting together a new system and can’t wait:

1. Make sure your video card is “DirectX 11 class”.  (This confuses everyone, because of course X-Plane uses OpenGL.  I am referring to its hardware capabilities.)  This means a Radeon HD 5000 or higher, or an NVidia GeForce 400 or higher.  DirectX 11 cards all do complete hardware instancing (something X-Plane 10 will use) and they have other features (like tessellation) that we hope to use in the future.  We’re far enough into DX11 that these cards can be obtained at reasonable prices.
2. Get at least a quad-core CPU.  It won’t be a requirement, but we have been pushing to get more work onto more cores in X-Plane 10; I think we’ll start to see a utilization point where it’s worth it.  The extra cores will help you run with more autogen during flight, cut down load time, and allow you to run smoother AI aircraft with the new ATC system.

Finally, please don’t ask me what hardware you need to buy to set everything to maximum; I’ve tried to cover that here and here.