I’m going to keep going with “random stuff I looked at today” and see if there’s something for authors mixed in.
I spent part of today measuring shader and texture changes in our engine under heavy load – my goal was to get a sense of how much data we’d be pushing through a next-gen API (e.g. Vulkan, Metal, DX12) if we just did a straight port of the engine. This was just a fact finding mission.
The only problem was: the numbers were just absolutely awful. Terrible. Unusable!
Then I noticed the other thing: the entire area of KSEA was littered with thousands of Fed-Ex trucks. Just tons and tons of them! Only Fed-Ex trucks on the road, and only Fed-Ex trucks parked on the street.
Wha?
Leftovers For Lunch
The Fed-Ex trucks were a left-over. I do this to myself all the time: I create a dumb custom scenery pack to test some part of the sim and then forget to remove it from my working X-Plane folder.
Before X-Plane 1040 there was a bug where cars and trucks on the road could crash the sim if you viewed them across a DSF tile boundary and the 3-d models were not instanced. This last point is why the bug went unfixed for so long; the car set we ship with is entirely instanced for performance.
So I built a library with a Fed-Ex truck that was intentionally not instanced to reproduce the bug and forgot about it. The custom scenery pack was why my traffic looked silly, and the non-instanced traffic was why my stats showed the rendering engine doing 4x more on the CPU work than it was supposed to.
(Since X-Plane was in debug mode, the framerate was expected to be terrible due to unoptimized code and debug checks running on an old laptop with the scenery cranked to the max.)
So there’s a take-away here and it is:
OBJs in a Custom Vehicle Pack Should Be Instancing-Friendly
There are a few custom vehicle packs for X-Plane floating around the web, and the quality of the objects with regards to performance is mixed and overall not very good – probably some of these packs pre-date X-Plane 10.
Instancing is the ability for X-Plane to draw more than one OBJ in a single instruction to the GPU. We implement instancing by observing your OBJ as we load it and noting whether the OBJ contains anything complicated (dataref usage, animation, lots of material changes) or if it is more or less just a big pile of triangles.
If we have the latter case, then we mark the object as instancing friendly, and when it is drawn, all objects of that type are collected and drawn at once. The instancing code path in X-Plane is thus separate and much faster for both X-Plane itself and the driver.
Since you can have a lot of the same car on the roads, even with a varied collection, it’s worth it to be instanced!
How to Tell If Your Object Is Instance-Friendly
To see if your object is instancing friendly:
- Make a custom scenery pack and place ten of the objects very close to each other (e.g. at an airport).
- Load the airport in X-Plane and in DRE set the art control “terrain/kill_clusters” to 1.
When you do this, all of the instanced objects that come from DSFs will disappear, and all of the non-instanced ones will remain.
Your object will be instance-friendly if:
- It uses no animations
- It uses no ATTRibutes mid-object – use the new GLOBAL properties instead
- For cars, LOD is okay (but non-additive LOD will make the WED test fail). For cars you should only use one LOD anyway.
- Only some named lights are instancing friendly; fortunately the headlight/taillight ones are.
Draped geometry is instancing-friendly, but don’t use it for vehicles.
In the new Blender 2.7 exporter (and our branch of the Blender 2.49 exporter) instancing is made quite easy: you mark an object as “instanced” and one of two things will happen:
- Blender will write only stuff that is instancing friendly or
- The export will fail with an error telling you what you did wrong.
Thus when you -need- something to be instanced (scenery objects, etc.) you just tell the exporter to enforce it.
Here are some things where instancing really matters:
- Repeated buildings in autogen.
- Static aircraft that repeat a lot.
- “Clutter” on the ramp (baggage trucks, etc.).
- 3-d modeled vegetation that gets repeated.
- Cars (both parked and moving)
Here are some cases where it does not matter:
- Aircraft-attached objects. Since aircraft attached objects aren’t usually repeated and almost always have a lot of complicated stuff, instancing doesn’t matter or work here. Instancing is really for scenery.
- Single extremely complicated objects like the Nimitz.
Right now objects drawn with the XPLMDrawObjects API call do not benefit from instancing, but this is probably something that will change in the future, as long as every “instance” is fed through a single XPLMDrawObjects call.
First, X-Plane 10.45 release candidate 2 is out. There was a bug in RC1 where the instructor’s operator station (IOS) couldn’t change aircraft over the network. I’m hoping we don’t have any other last-minute fire-drills; the IOS bug was reported the day I was going to call RC1 final.
I keep on telling myself during the day “I’ll blog some developer news later when I’m too tired to code” and (shockingly) at 10 pm, after the second round of coding after the kids have gone to bed, I find myself too tired and go “meh, I’ll blog tomorrow.”
So in an attempt to post something rather than keep delaying…
Spock And Friends
The Vulkan API was released today. Vulkan is the third new next generation 3-d graphics API (the other two are Metal for OS X and iOS and DirectX 12 for Windows). Vulkan should be available on Windows, Linux and Android devices. The specification is 651 pages long, so I have some light reading to do.
Faster Cockpit Lights
FlightFactor sent me a test case to look at a while ago – a possible bug in the Blender 2.49 exporter. While exploring the performance of the test case, I discovered that the code that runs ATTR_lit_level in a cockpit OBJ was a lot less performant. (If you don’t create 3-d cockpits, ATTR_lit_level lets you turn the brightness of a 3-d model’s night texture up and down based on a dataref; it is crucial for creating cockpit lighting animations, including annunciators.)
I ripped out a pretty huge pile of code in the process, and I am hoping we’ll see a performance boost. On the mobile product (this code is shared on mobile and desktop) the changes cut the number of driver calls we make down by about 25%. (That doesn’t mean a 25% speed-up – not all calls are equal, but it shows that there was a real win in there.) My measurements of the ATTR_lit_level test case run the OBJ itself 10x faster. (Again, that doesn’t mean 10x the framerate – it means this one very small part of X-Plane runs 10x faster). Once the work is done I’ll post some performance numbers. This code is targeted at X-Plane 10.50.
Posted in News
by
Ben Supnik |
X-Plane 10.45 release candidate 1 is out (and will be available on Steam for users who have selected to get betas within a few days). If you aren’t running betas and would like to try it, run your X-Plane Installer, pick update and check “Get Betas”.
If you are the developer of an add-on aircraft and you haven’t tried 10.45 yet, please try it as soon as possible! We caught a few surprising systems-related bugs and fixed them for RC1, but overall beta bug reporting has been quite low; we can’t fix what we don’t know about, and some bugs might only be reproducible using your add-on.
X-Plane 10.45 is a “small” patch for us in that it doesn’t contain major changes to the code, but it contains a number of features you might like:
- An update to the global airports.
- The global airports won’t conflict with custom scenery – new exclusion technology prevents overlapping buildings.
- Nav data has been significantly updated – lots of bug fixes are in 10.45.
- The GPS data for the G430 now contains IAFs, so you can fly GPS approaches with proper transitions.
- Proper prop torque for aircraft that opt in.
I keep meaning to post about other stuff we’re working on; I’ll post an update soon. But for now, try 10.45 before it goes out the door!
In my previous post I mentioned that re-saving your aircraft in Plane-Maker from X-Plane 10.45 would opt you in to the new fixed torque model.
I’ve talked about this before, but it might bear repeating now:
Please do not ship an aircraft saved with a beta version of Plane-Maker. Instead wait for X-plane to go final or use the previous non-beta version of X-Plane.
The problem is that file formats change during beta and if they do we do not provide compatibility; the file format is only “for real” when beta is over.
So if you need the torque bug fix, plan to ship your aircraft in a week or two when 10.45 beta is over. If you need to ship now, use Plane-Maker 10.42. Test against the beta, but don’t save using it.
X-Plane 10.45 beta 1 fixes a bug in X-Plane’s flight model where props applied too much torque to the aircraft. The effect is most noticeable on single engine aircraft where the prop torque is applied along the center of the fuselage.
Authors: in order to have the new correct torque applied to your aircraft, you must resave your aircraft in Plane-Maker 10.45. When X-Plane finds an aircraft authored with an old Plane-Maker, which includes all existing aircraft already shipped, we preserve the old torque behavior.
We put in this compatibility because some aircraft have plugins and other work-arounds that try to get correct flight dynamics assuming X-Plane will apply too much torque. If we were to apply the fix to these planes, they’d then have too little torque (because we’d apply less torque and the work-around would already be in place). So this is an opt-in fix.
Authors: if you use Plane-Maker 10.45 to work on your aircraft, you will be opted in to the fix! There is no way to work on a new aircraft and preserve the old torque. Our thinking is: the old torque calculating is physically wrong, so we’re not letting anyone preserve this mistake when working on a new aircraft.
So when you bring your aircraft in for its next feature update, note that you will be getting the torque fix and you should make sure the results are useful.
MH1212, developer of the Prefab Airports for X-Plane, requested this feature, and it looks like we are going to be able to sneak it into X-Plane 10.45. (If we hit bugs, it might get pushed out to 10.50, but so far things look okay.) The feature is: the ability to exclude objects by airport ID without using exclusion zones.
Right now when a custom scenery pack replaces an airport (via apt.dat), the old apt.dat is completely ignored. But the DSF-based overlay objects, facades, etc. are included; the custom scenery pack has to use exclusion zones to kill them off.
With this extension, the DSF-based overlay objects in a scenery pack can act as if they are in the apt.dat file, disappearing when the apt.dat airport is replaced. This means that when you replace an airport (via apt.dat file) not only do the runways go away, but so do the overlay elements.
Here’s the win: we can export our global airports from the X-Plane Scenery Gateway this way, and custom scenery will remove the overlay elements automatically just by replacing the apt.dat, even if no exclusion zones are present.
Here are some details:
- The scheme works if the underlying airport is correctly marked as having its objects and facades “inside” the airport. So unlike exclusion zones, this scheme works if the underlying airport is modified, not the overlaying airport.
- This is a new scheme – no existing scenery already does this; scenery must be re-exported to gain access to this functionality.
- The functionality requires the latest version of X-Plane, but is harmless to old X-Planes – the DSFs will still load.
- Exclusion zones still work and are still recommended; if you are making custom scenery and you are on top of autogen or an old scenery pack that is not modified using this new scheme, only an exclusion zone will save you.
There are two big advantages of this scheme:
- We (Laminar Research) re-export our collection of thousands of airports on a regular basis, so we can tag the entire set of global airports using this new scheme and have them ready for by-airport-ID exclusion very quickly. So this scheme will start to help conflicts immediately.
- The scheme doesn’t require modifying the overlying scenery at all. There are freeware airports that are effectively orphaned – the author cannot be found and the license doesn’t allow the community to modify them*. Since these airports cannot be legally redistributed with exclusion zones, this technique will help.
Once X-Plane has this extension and the global airports are re-exported using it, global airports will fully disappear when any custom scenery pack replaces the airport by apt.dat, even if the custom scenery pack doesn’t have good exclusion zones.
This functionality will be available to third parties in WED 1.5 when it goes beta. In WED 1.5, if an overlay element is in the folder for an airport, it will be excluded when that airport is excluded. If an overlay element is ‘loose’ in the outer level hierarchy, it will not be excluded by airport (but will be affected by other pack’s exclusion zones).
Since gateway airports already have the objects “in” the airport folder, they are already authored to make this scheme work.
If you create your own DSFs using a low level tool like DSF2Text, the DSFLib source code, or something else crazy, I have posted the proposed schema here. That’s a technical link for people tinkering with the DSF format itself, but if you’re in that category, please do ping me to get early test materials. (The new code is also on GitHub.)
Two warnings to custom scenery authors: if you are creating a custom airport scenery pack, especially payware, please read these very carefully:
- This is not an invitation to stop using exclusion zones. There are plenty of scenery packs in the wild that do not have their objects tagged by airport, as well as autogen and all sorts of junk that can be under your payware. If your airport doesn’t have an exclusion zone and it conflicts with another pack, it is your fault. Go add exclusion zones, like I told you to do years ago.
- If a scenery pack from X-Plane Airport Gateway is removed by your pack (by airport ID) then our exclusion zones are removed too. This means that if trees on the runway have been removed by an X-Plane Airport Gateway airport, you will no longer get those exclusion zones for free. You must exclude those trees yourself! (If you put exclusion zones in from point 1 this is a non-issue.) Test your airport with global airports enabled and disabled to make sure your pack is good.
* As a side note I consider it a real problem that airports get uploaded and shared with the community under licenses that don’t allow for abandon-ware to be maintained. It’s clearly the right of the author to use any license you want, but as a community I hope we can encourage freeware authors to use a permissive open license.
X-Plane 10.45 Beta 1 is out – here are the release notes. This is a small update in terms of code change; the new feature is updated global airports and nav data, as well as a bunch of bug fixes; the beta period should be relatively short.
To get the new beta, run the X-Plane installer, update your copy of X-Plane and make sure “get new betas” is checked.
Prop Torque
Prop torque is fixed in X-Plane 10.45 beta 1; the calculation was incorrect (causing too much torque from props) in previous versions of X-Plane.
In order to get the new correct physics, you must resave your aircraft in Plane-Maker 10.45 beta 1.
We have heard of authors doing a number of things to lower the effects of prop torque in old versions of X-Plane, including having plugins apply a counter-torque and tweaking the physics parameters of the aircraft itself. Because we cannot know if an aircraft has such work-arounds applied, the prop torque fix is applied only to aircraft resaved in 10.45 beta 1 or newer. This way the fix takes affect when an aircraft author can remove work-arounds.
WorldEditor 1.4.1 is out – it’s a very small patch to 1.4.0 that increases the strictness of validation to catch a number of errors we found in the last export from the X-Plane Scenery Gateway.
Please upgrade! WED 1.4.1 will become the mandatory version to upload to the X-Plane Scenery Gateway in a few days if we don’t find any major bugs.
64-Bit WED
Last week I completed most of the work to modernize WED and the scenery tools for modern OS X/X-Code, which finally lets us build 64-bit Mac scenery tools. (This also lets anyone compile WED without having to find a Mac from the dark ages.)
WorldEditor 1.4.1 will be the last 32-bit build of WorldEditor. The next update (WED 1.5) will be 64-bit on all three platforms: Mac, Windows and Linux. This transition will also raise WED’s minimum operating system for OS X to OS X 10.7 (Lion).
Having WED be 64-bit on all platforms should solve problems that advanced scenery authors are seeing where WED runs out of memory when editing a custom scenery pack with a lot of textures or reference imagery.
WED 1.5 Features
I am not sure of the feature list for WED 1.5 yet; 1.4.1 has been kept intentionally small to get a quick patch out. (The longer we have a WED that does not properly validate airports, the more weird stuff ends up in the scenery gateway.) Two features are very likely for WED 1.5:
- 64-bit on all operating systems.
- Support for apt.dat file format extensions to coincide with X-Plane 10.50.
My plan is to have WED 1.5 ready for beta before X-Plane 10.50 is ready for beta, so we can test X-Plane’s new capabilities with WED.
We have internal notes on the apt.dat 1050 extensions; I’ll get an RFC posted in the next few days.
We’ve been doing some work behind the scenes to modernize our servers; as part of that, I have moved the scenery tools code to GitHub.
The GitHub public repositories are the official repositories for WED and the other scenery tools that live in the XPTools code base; I push my changes directly there.
using GitHub means much faster clone times (since the server we had the code on was getting long in the tooth), better public browsing, and you can do all of the normal GitHub things that you’d want, like forking, making pull requests, etc.
If you are using our CGIT browsing (at http://dev.x-plane.com/cgit/cgit.cgi/) please switch to GitHub; CGIT will be going away soon. Similarly, if you have the code checked out, you should change your remote’s URL to https://github.com/X-Plane/xptools.git.
I mentioned in a previous post that we are working with Ondrej on a next-generation Blender 2.7 OBJ exporter. I am excited for this work because I believe it will greatly improve the artist interface to X-Plane. Let’s use instancing as an example to explain the idea.
X-Plane 10 will sometimes draw scenery objects using hardware instancing. When an object is drawn with instancing, performance is a lot better than without it; the huge numbers of objects you can see in the autogen scenery are due to instancing. But the conditions for instancing are, to put it mildly, complicated.
- The object has to be attached to a DSF. Well, that’s not crazy.
- There’s a giant pile of things you’re not allowed to do in an instanced object – animation, material attributes, smoke puffs, bla bla bla. (The object has to be simple so that the GPU can draw a big pile of objects without the CPU intervening to handle things like smoke generation on a per-object basis.)
- Buuuuut…not every ATTRibute is a problem. You can use ATTR_hard in an instanced object, no problem! But don’t use ATTR_shiny_rat – that’ll kill instancing.
- But wait, there are these weirdo new attributes like GLOBAL_specular that seem to replace ATTR_shiny_rat that do work with instancing.
If you model, and this seems confusing, it’s because it is. You were never supposed to see this!
File Formats for Machines and Applications For Humans
My view is this: the file formats of X-Plane are meant to be seen by programmers, not artists. When was the last time you had to hand-edit an .acf? You just use Plane-Maker. You don’t hand-edit a PNG file or worry about its internal compression – you use PhotoShop.
OBJ should be the same way – you should export from a 3-d tool and never have to look inside the OBJ itself. This means the OBJ8 format can serve two purposes:
- Help X-Plane load models quickly and draw good looking content and
- Maintain compatibility so older scenery continues to load.
Note that “be really easy to understand in Notepad” is not on the list!
Blender the Transformer
Now here’s the trick: the rules for OBJ that a 3-d modeler can present can be different from the rules that the OBJ8 format itself has. And that is exactly what we are trying to do in the new Blender exporter.
Rather than expose global attributes and non-global attributes and all of the crazy complexity of OBJ8, our strategy with the new exporter is this:
- When you export a scenery object, you specify if you want the object to be instanced or not.
- If the object is not instanced, Blender creates the best OBJ it can for this case.
- If the object is instanced, Blender tries to create the best OBJ it can for this case.
- If the object is supposed to be instanced but you’ve done something that is not instancing-friendly, the exporter tells you, in terms you can understand.
So for example, if you make an animated radar dish that spins and attempt to export with instancing, the exporter can say “you cannot have animation in an instanced object.” Now you, the scenery designer, can make a decision: is it more important to have the radar dish draw really really fast, or is it more important that it spins?
This is the strategy we used for our modifications of Marginal’s 2.49 exporter* – the artist specifies instancing and gets a check that instancing requirements have met. This means instance-tagged objects will be fast. (It is possible to check instancing within the sim with DataRefEditor but it’s not easy, especially for large projects.)
Don’t Expose the Weird
This idea that Blender can transform your work, and thus hide weird X-Plane rules, is not specific to OBJ, not new to X-Plane, and not even specific to X-Plane. When you look at major 3-d game engines (e.g. for combat games and racing games) a ton of code goes into the “art asset pipeline”, transforming the artist’s original work into something already digested for the 3-d engine.
Another example of this is WED. DSFs have an annoying rule that polygons can’t cross DSF boundaries. There are also some even weirder exceptions for a few special cases, introduced in X-Plane 10.20.
WED knows all of these rules; you draw polygons wherever you want and set the export target for the version of X-Plane you want to fly in, and WED figures out the best way to export your work, chopping up polygons as needed.
Shedding Light on Things
Our current work on the Blender exporter focuses on WYSIWYG animation and the material model (which includes instancing support). There is one more area that I think will make a huge difference to authors that we haven’t started working on yet: lighting.
The lighting values for commands like LIGHT_PARAM are, to put it mildly, completely goofy. If you’ve ever tried to read this, you know that even when things are explained, they make no sense.
No author should ever have to write a LIGHT_PARAM. This is a classic case where the exporter should do the transformation for us. What I would like to get into Blender 2.7 is the ability to place a cone light in Blender and have the exporter write the spill light automatically, e.g. filling in the parameters from the position and shape of the light. You light your scene and you see the same thing in X-Plane.
For years we’ve been behind the curve on user experience when it comes to 3-d modeling. I think we’re starting to fix some of those problems in a major way, which should help make the power of X-Plane 10’s rendering engine quite a bit more accessible.
*As a side note, I have no idea what the current 2.49 script status is. We view the 2.7 scripts as the long term way forward, due to Blender 2.49’s not being maintained, so our effort on a truly user-friendly tool is going into 2.7. But we also maintain updates to 2.49 to support our internal team. X-Plane 10.50 will feature a few new OBJ commands, so I will try to post our branch of the 2.49 scripts (which do have this new support) for anyone whose aircraft is in 2.49.
In the long term the 2.7 scripts will feature a migration tool, but I expect we will have a useful 2.7 release (for 2.7 authors) without migration first, and then migration will come later.
