apt.dat 1050 Proposed Format Changes Posted
We’ve posted a Request for Comments for the apt.dat 1050 file format. You can see the proposed changes. Please comment on that article with feedback about the file format.
We’ve posted a Request for Comments for the apt.dat 1050 file format. You can see the proposed changes. Please comment on that article with feedback about the file format.
First, we are making progress toward X-Plane 10.50. I sent out a private beta a few days ago; how soon we get to public beta will depend on how buggy the private beta turns out to be. (I am assuming that it will have zero bugs, because I have decided to not write any more bugs in my code. So…there, I fixed it.)
Coming in 10.50: X-Plane supports new manipulators.
I want to make two things clear here:
We’ve had a long term internal goal of making the 3-d cockpit as usable and user-friendly as possible. We’ve reached a point where 3-d cockpits are the norm, and all of our new aircraft development is 3-d only. But you can’t deny that 2-d cockpits are incredibly usable from a user interface standpoint. How do you make 3-d usable?
We already have view presets to let you get to a particular cockpit view easily. Once you have a good view of the surface you want to interact with, the next step is to have a mouse gesture that’s easy to use.
This is where the new manipulators come in. Rather than describing a mouse gesture (drag, click, etc.) they describe a type of physical hardware, one of a rotating knob, left-right multi-way switch or up-down multi-way switch.
Because the manipulators describe the physical hardware and not how the UI should work, we can change the interaction mode based on user preferences, available hardware, etc.
One of the big UI questions is: should you change a knob by clicking and holding or dragging? I built a test airplane with some of each and the company was split down the middle. By using knob manipulators (and not click or drag manipulators) we can change the way knobs work in the cockpit for all airplanes with one change of the code, rather than having our art team redo every single manipulator in every single airplane.
I don’t know whether we’ll use dragging or clicking-based knobs – or whether a user preference will be exposed. But by having a higher level manipulator type we make this a small programming problem and not an unmanageable authoring problem.
(The new knob manipulator automatically uses the scroll wheel – since we know it’s a knob, we can figure out how the wheel should work. The new command to add scroll wheel to legacy manipulators is necessary to provide the data the sim needs to make the user interaction good.)
The new higher level manipulators also give us flexibility for new input devices. For example, if we someday want to support multi-touch tablets and track-pads, we could map a pinch-rotate gesture to a knob – since we know the manipulator really is a round knob (and not just ‘something that you drag’) we can know that a rotating gesture makes sense.
The new manipulators are already supported in code in the Blender 2.7, Blender 2.49 and and AC3D. The manipulators are released in the newest Blender 2.7 exporter; I need to post newer 2.49 exporters. I do not know if/when there will be a public release of the AC3D exporter, but I can probably find the time at some point to do one more compile.
All aspects of the new manipulators are opt-in; if you don’t change your airplane, its functionality should not change.
I’ve buried the lead a little bit here: X-Plane 10.50 will have a heavily upgraded Kingair C90 and Baron B58 that will use the new manipulators. We have not (yet) upgraded the rest of the fleet, and this upgrade will happen over a very long time-span. The goal of 10.50 is to lay the groundwork and make cleaner user interfaces possible by making the new manipulators available to our art team and third parties.
The guidelines I’ve been advocating for our artists are:
There’s no reason why third parties have to use these guidelines; I post them only to show how we are thinking about usability. A more “3-d” approach is possible, e.g. drag a banana switch up and down to toggle it; my view is that for Laminar’s default fleet, which are the first aircraft users are going to use, single clicks provide a more direct and less surprising user experience.
For power users, having single clicks for switches may also mean you can get more done in the cockpit faster. (In real life, a pilot can reach up and flip four metal banana switches on the overhead panel very, very quickly.) We’ll still use dragging for big 3-d motions.
*This feature targets one-wheel clicking Windows-style mice. 2/3 of our users are on Windows, and Apple has shipped a wide variety of strange input devices, so the one-axis clicking mouse wheel is the most consistent hardware target for us to aim at. And frankly, I think it’s the most useful since you get clear ‘detents’ as you scroll things – good for changing a radio by one notch.
That’s the new taxi sign editor in WED.
Signs in the hierarchy are shown as a rendered preview of what the sign will look like.
When you click on it (as if it was text) you get a WYSIWYG editor with two “text” fields where signs can be directly typed, and a palette where signs can be added by clicking.
I just fixed a problem in DDSTool: DDSTool was calculating mipmaps in sRGB space instead of linear space. This was wrong, but didn’t make a huge difference when grinding day-time textures.
It does make a big difference for LIT textures though. Let me show you.
That’s a night LIT scene (at PHNL I think) with some new autogen from X-Plane 10.50. (To answer your question, some of the AG has LIT textures, but not 100% of objects do.)
In the lower left corner are thumbnails of the raw textures – what is interesting about them is that they show the smaller-sized images in the mipmap pyramid of the image. And you can see that in the old image, the small thumbnail of the buildings is almost completely black – the building light has been lost with down-sizing.
Note that the buildings are a little bit darker in the old grind on the left but the tiny thumbnail is much darker. That’s because making mipmaps is a recursive process – each next-smallest-size mipmap is processed from the previous image; by the time we get to a really small size image, if we have lost brightness, the effect has really built up.
The result was that in the old grinds, as you flew away from the buildings, they’d get darker. In the new grinds, they stay at consistent brightness.
Here’s the funny thing about images in sRGB color space (which is what X-Plane uses) – the mid-point between black (0) and white (255) is not 127 or 128. That’s too dark! The actual right color is 188 – pretty bright!
This happens because sRGB images are encoded in a non-linear manner for even perceptual spacing for humans, and our eyes are more sensitive at lower light levels. But the average between two pixels should be the average of the actual amount of light coming from the pixels. The half-way point of sRGB is too dark because a lot of numeric space in sRGB is spent in the dark regions.
The problem is that lit textures are made up of a few really bright pixels and a lot of black; when averaged together the wrong way the smaller sized images end up too dark, making slushy, dark, munged night lit textures at small sizes.
The solution (in the new DDSTool code) is to convert the colors to linear space, down-size the image and convert them back. There’s no change to how you use the tool – the images just look better at small size.
This is a tiny feature coming in X-Plane 10.50, but it will make a big difference for a few key plugin use cases: X-Plane will provide the current world model-view, projection, and aircraft model-view matrices to plugins via datarefs.
If you aren’t a plugin developer, or you don’t like matrices, or you don’t write your own OpenGL code, then the rest of this post is going to be boring, so go watch FrooglePete interviewing two funny looking guys. [Edit: link fixed – why that was pointing to airport flattening is totally beyond me!]
For the three of you still here: three matrices will be available:
You already are using two of these, the third is new. So who would even care? There are two use cases for this.
Calling glGetXXX is bad. Modern Windows GL drivers send GL commands to a worker thread for execution, freeing up the rendering thread to keep going, improving framerate. But every time you ask OpenGL a question, our thread has to wait for that driver thread to catch up, slowing everybody down. The rule is simple: don’t call glGetXXXX.*
But there’s one case where your plugin might really need glGet: to get the current matrices for culling 3-d drawing.
This is where the datarefs come in – by reading our datarefs instead of calling glGetFloat, you can get the transform matrices without stalling the driver.
(If you have other glGetXXX that you can’t get rid of, ping me and I’ll try to find a work-around.)
The third matrix (the aircraft model-view matrix) is the matrix you would get if you translated to the aircraft’s location and then rotated around its orientation.
Here’s the key difference: when we do that calculation, we do the calculation in double-precision. We do this because the OpenGL origin can be tens of thousands of meters away from the aircraft, which in turn can be very, very close to the camera. In that situation, precision loss from single-precision floating point when multiplying together two very large matrices (that result in a very small matrix) results in apparent jitter when drawing.
The problem is: even if you wanted to do this calculation yourself in double precision, you can’t – the world matrix isn’t available to plugins in double precision.
So starting in X-Plane 10.50, the aircraft matrix is simply provided.
I did an experiment where I attached a green cube to an aircraft and then drew a blue cube at the same location from a plugin. With glTranslate/glRotate there was a ton of Z artifacts and slop that changed per frame because the two cubes were in slightly different positions. With the aircraft matrix, the green cube was invisible (except when disabling the plugin) because the blue cube overdrew it pixel-perfectly.
I’ve been way behind on blog posts over the last few weeks. Basically, the more we are doing, the less I manage to blog about what we are doing. (Instead I put off writing a blog post until it’s really late and then decide at 11 pm that I’m too tired and I’ll do it tomorrow.)
So here’s a partial list of X-Plane 10.50 features. This is not even remotely complete, it’s just some “headline” features that I can think of now; the release notes will be comprehensive.
New Autogen: We have new US tall building art assets that will make cities look better.
apt.dat 1050 with Static Aircraft and New Models: A new revision of the apt.dat file provides information on parking spots so that we can place static aircraft models inside X-Plane, based on the library and rendering settings. X-Plane 10.50 will ship with a bunch of additional static aircraft models, and third parties can add more via the lbirary. WED 1.5 will have new features to edit this information.
For airports created before WED 1.5 (which is nearly all of the approved airports), we are working on tech to auto-upgrade the gateway airports to use the new static aircraft; third party scenery will simply not participate in the new feature until updated by the authors.
Global Winds Aloft: X-Plane 10.50 will use global NOAA data for winds aloft, rather than a US-only data source.
ATC Fixes: X-Plane 10.50 has a number of ATC bug fixes to make ATC a lot more usable. No more “you are off course!”
New Manipulators: We have added a few new manipulator types as part of an effort to make 3-d cockpits more usable. Yes, the scroll-wheel is accessible. (We have not rebuilt every 3-d cockpit in our fleet. The feature here is the capability in the engine, for us and third parties to use.)
Update King-Air and Baron: we have, however, redone the Kingair and Baron, fixing a number of issues and getting them to a whole new level for IFR flight. These planes use the new manipulators for their 3-d cockpits.
More Airports from the Gateway: as with all releases, we’ll include the latest airports from the X-Plane airport gateway.
Those are just the “big” things – there’s some huge number of other changes, some of which may be really important to some of our users. I still have about half a dozen items on my todo list to get to a 10.50 beta, so I haven’t worked up complete release notes yet.
In my previous post, I posted a screenshot of X-Plane 10.50 with the new autogen update we are working on. This is an AG update that includes a bunch of buildings, particularly tall ones; I’ll post a lot more detail no that in a separate post.
One of the main questions in the blog comments was: “how are you getting this pic with 30 fps? Did you get new hardware?”
Well, I did get new hardware; a few months ago I finally swapped in the first-gen 5K iMac with the R295X GPU or whatever it is. (I thought this was latest but I see now Apple has revised the machine and bumped the GPU up a generation.) So that’s a machine with a bit more kick than my older 2008 Mac Pro.
But more importantly: X-Plane’s autogen engine is really fast. It hits the instancing path almost all of the time and as a result it can draw a lot of autogen at reasonable fps.
When X-Plane 10.0 shipped, I could run the autogen + roads maxed out my old Sandy Bridge PC – that was before optimization on hardware five years ago. I think this is one of the things that X-Plane 10 does really well that people tend not to notice; that code path was hot from day 1!
There’s a third thing going on here: I picked my rendering settings to promote AG + framerate; you can do that too. I see a lot of users running with relatively low autogen density and poor fps due to other settings. And clearly people like lots of buildings.
So here are some tips on how to get a lot of autogen and decent framerate.
The other day I was forwarded a very (cough) angry customer email; the customer had a gamer-class PC, turned everything in X-Plane up to max, got about 5 fps, and went absolutely ballistic. We see emails like this over and over and over and over again. “How much @#$@# money do I have to spend to turn everything up?”
In the long term, our plan is to change the rendering settings so a high-end gamer-class PC can run with every single slider topped out; if you want some of the super high settings in some areas of the sim by turning down other areas, this will be possible only by hacking the config files, not by the UI. The current rendering settings scheme is simply not friendly to users.
With that in mind, I look at what combinations of settings we can max out in hardware (to the point where the artists intended things to be) and still have good fps. My current view is:
* Computers also just get faster over time, so I suppose I could solve this by taking a ten year sabbatical. 🙂 Seriously though, if the cars would run at playable fps at medium res, I’d be okay to let hardware improvements get us to really dense traffic.
Update: some answers about performance, hardware and settings.
A user asked me to write a little bit about Vulkan. My first reaction was to not post anything for a simple reason: Vulkan is a feature for me, not for you. That is, Vulkan does not make you (the user of X-Plane’s) life directly better; instead it makes my development job easier and it makes it possible for me to create a better X-Plane.
Some day if we end up running on multiple drivers (e.g. Vulkan and OpenGL and Metal), you may not be able to tell the difference between the Vulkan and OpenGL version; the sky won’t be more blue, the clouds won’t be more puffy. I can’t think of any features exposed by Vulkan that aren’t in OpenGL 4.x with extensions now — we already have sparse memory, tessellation and compute in OpenGL. (But then, you might actually be able to tell because framerate might be higher.)
Anyway, having finished reading the (700+ page – that’s what down time in the airport is for) Vulkan spec last week, the rest of this post is my view on Vulkan as a 3-d graphics developer. This is definitely an “inside baseball” kind of post, so if you want to go surf cat videos, I won’t be offended!
For an application like X-Plane, Vulkan is an improvement over OpenGL; to understand why we have to look at the problems with OpenGL as an API that allows applications to communicate with 3-d hardware drivers. There are a few:
How Vulkan Helps: The design principles behind the Vulkan API address all of these issues.
Vulkan is a smaller, lower level API with clear performance guidelines and a focus on multicore from day one. This is a good fit for what we need with X-Plane.
The OpenGL ecosystem is the collection of all of the companies and programmers working with OpenGL. This includes the vendors of graphics chips who create OpenGL drivers (NVidia, AMD, Intel, Imagination, Qualcomm, ARM), OS vendors who provide OpenGL interfaces (Apple and Google), the major game engine developers that have the ear of the hardware vendors (think Unreal Engine 4, Unity, etc.), the major CAD application developers, etc.
More serious for OpenGL than the problems of the API is the state of the ecosystem.
Given the lack of specifics, application developers and driver writers end up locked in a sort of death-spiral:
How Vulkan Helps: Vulkan helps by being much more highly specified in terms of both conformance and performance.
For smaller OpenGL application developers like Laminar Research, I can think of just one down-side; it’s one that I haven’t seen a lot of application developers talk about, probably because it requires admitting that we (the app developers) might not be as smart as those driver guys.
OpenGL is a higher level API; OpenGL applications leave some of the hard problems of 3-d graphics up to the driver. This means that some of these very hard, very important performance problems are being solved by a team of engineers from the company that built the hardware. They have resources, they know everything about the particular hardware they are coding for, and performance is job one.
Vulkan is a low level API; with Vulkan, some of those hard problems will be solved by Chris.
Ha ha…no, I’m totally kidding. We don’t let Chris play with pointers or any other sharp objects. That code will be written by me, and it’s a safe bet that I know less about the hardware than the driver team and have less time to do pure performance work then all of the engineers at Nvidia or AMD who work on the OpenGL stack.
This is a calculated risk for Vulkan as an eco-system. The hope is that (1) with specific information about performance as part of an API, us application developers won’t screw things up too badly and (2) because we (the application developers) know more about our specific applications, we can optimize performance in ways the driver guys can’t because they don’t have the bigger picture. One of the hardest problems in a graphics API is conveying intent; OpenGL drivers spend a ton of code trying to guess what the application is trying to do. Vulkan solves this problem by letting the application make performance decisions itself.
If you know OpenGL but don’t know a lot about how 3-d graphics hardware really works, reading the Vulkan spec is a little bit like believing that Santa Claus brings your presents down the chimney and then [spoiler alert] one day reading a 700+ page PDF that explains how your parents actually buy the presents a month or two in advance, wrap and hide them, and then sneak them under the tree while you are asleep.
For X-Plane, the Vulkan spec makes clear what types of operations future drivers will support, what will be done for us, and what we have to do ourselves. This gives us a good framework to then incrementally build next-generation rendering code in X-Plane that is “Vulkan-friendly” – even if it is still using OpenGL. My guess is that this new code will be faster than the code it replaces even before we change drivers.
Once we have the rendering code restructured and modernized, we can then set it up to run on Vulkan or OpenGL, taking advantage of the Vulkan pathways where they exist.
One final thought: I have no idea how the actual experience of coding for Vulkan will be. It may be that everything “just works”, or it may be an exercise in frustration. Once we’ve been through a full port I’ll post a post-mortem, but I think it’s too soon for anyone out there to have good realistic feedback on a full Vulkan porting experience from OpenGL.***
* And when people suggest ground up rewrites I usually link to this.
** For what it’s worth, the driver shader bugs I see (and the ones that really take up my time) are bugs in the back end of the compiler, where optimized machine code for the GPU are generated. Vulkan isn’t going to fix this; Vulkan removes the front end of the compiler from the driver but not the back end.
*** Yes, there are games running on Vulkan now. But if your game is already running on multiple APIs, that’s different from porting a straight OpenGL app. The final spec and roduction drivers haven’t been out long enough for anyone to be able to say “I ported half a million lines of OpenGL code to Vulkan and here’s the result.”
Ondrej has posted a public thread about the new version of the Blender 2.7 exporter here.
The 3.3 release is the first release where we (Laminar Research) have worked closely with Ondrej to build what we hope will be one of the future cornerstones of modeling for X-Plane.
If you are currently using Blender 2.49 or AC3D, I expect that 2.73 will provide the best way forward for modeling in X-Plane.
The new release has a few major features, all aiming to create a new modern work-flow:
The goal of many of these is to hide the idiosyncrasies of X-Plane’s modeling format and provide a cleaner, more artist-friendly view of modeling.
Regarding instancing: the model we have adopted is the one we used internally on the 2.49 exporter: you (the artist) tag an export as either “instanced” or not.
The win here is that you don’t have to know the (complicated) instancing rules; set instancing for the scenery objects you need to make fast in bulk (e.g. a luggage cart, a house, something that will be used many times in a small area) and you’ll get optimal output.
The goal of the 3.3 release was to set up an environment where authors could work using the new work flow. We have not finished optimizing the OBJ output.
If you are using the existing 2.7 exporter for airplane work, the output should be similar in performance. If you are using the 2.49 exporter, the new output is (like the current 2.7 exporter) not as well optimized.
In a future update, we will tighten up the generated OBJ code; this should not affect anyone other than producing better OBJs.
The new exporter should be fully work-flow compatible with the previous Blender 2.7 exporter; if you find your existing project does not work, please file a bug!
Our plan is to create a migration tool for Blender 2.49 projects; this will forward-convert the datarefs on annotations, manipulators, and object properties from 2.49 to 2.73 format. This lets authors using 2.49 move forward to 2.73 and have a migration path for existing content. (This work is not started yet, but is planned – we have our own aircraft we need to keep working on.)