This alpha contains changes to the updater. Make backups before using!
Global Material Settings -> OBJ Settings
This new version contains one of the most requested fixes (#357, #599) for XPlane2Blender: Normal Metalness, Blend Glass, and Global Tint have been moved out of the Material Properties Tab and into OBJ Settings! The annoying “All Materials in an obj must have the same Normal Metalness/Blend Glass Value” error is gone!
The updater tries its best to guess if you wanted Normal Metalness, Blend Glass, or Global Tint, however it isn’t perfect. If you have to manually correct more than 3 of your OBJ settings, please tell me.
These new settings can be found under the Textures section of the OBJ Settings.
Emissive Panel Texture Only and Panel Mode Selector
#595 Also known as ATTR_cockpit_lit_only, this directive has actually been in X-Plane since 11.10, but now is accessible in XPlane2Blender! It makes a panel only use the emissive “Lit” texture – a great speed boost if that is all you need. This is the perfect feature for computer displays.
For Aircraft or Cockpit export types, look in the Cockpit section for “Panel Mode”. This changes the meaning of “Part of Cockpit Panel” for the whole OBJ. Setting it to “Emissive Panel Texture Only” mode activates the feature. You cannot mix panel modes.
Cockpit Regions
People who use Cockpit Regions will have to manually change “Panel Mode” from Default to Regions to see the UI and have regions export again. A one time fix. A future updater will do it for you. Until then I hope it isn’t too many OBJs to change.
Updater Version History Synchronization
#471 The updater now synchronizes its last version across every scene and new updater functions will not use data cleaning to protect against accidental or purposeful updater re-runs. Simply put this means a safer to use XPlane2Blender (but still make backups). Since this is new updater code, however, it had to be put in an alpha. People with multiple scenes should be especially on the look out for problems. I feel very confident about it however.
With testers and users like the ones XPlane2Blender has we’ll be getting to v4.1.0-rc.1 much much much much faster this time! For the users who requested moving Normal Metalness and Blend Glass, thank you for your patience. I know now just how annoying that error message was! I’ll certainly keep this experience in mind for future decisions and don’t worry, “make it the same across every material” is not going to be chosen again without an extremely important reason!
(This post was edited to reflect a discussion Ben and I had)
A constant request for XPlane2Blender and other exporters maintained by Laminar Research is an import OBJ feature. So, one last time, here is the news: by now an importer is basically never going to be a part of XPlane2Blender, but it could be its own project! It could take years of development to make it very well polished, however.
There are a few projects started on the forums you can checkout if you’d like!
Why Is This Such A Hard Problem?
It is simple to read the vertex table, attributes, animation keyframe table, and turn that into Blender data, some projects can already do this, in fact. The trouble is getting the exact or nearly the exact .blend file back.
OBJs Don’t Save Everything You Want And Need As An Artist
Some of the things things you could never import from an OBJ
Window Settings
Text Blocks with scripts, notes, or annotations
Any non-XPlane2Blender Blender settings we don’t care about
Object, layer, material, and texture names
A bunch of settings that can’t easily be inferred (more on that later)
Blender’s parent child relationships
If the OBJ was produced with comments and correct indentation, you might be able to get some of these things back (likely just a few names.) A large complex Blend file without this stuff is a nightmare of an unorganized mess and would make the rest of the manual reverse engineering process even harder.
Multiple .blend files Can Produce The Same OBJ Content
If A.blend, B.blend, and C.blend can produce the same OBJ, which one should the OBJ be reversed engineered back into? The relationship between XPlane2Blender settings and what appears in the OBJ can be very esoteric and there is not always a 1:1 relationship. Some ATTR_s only appear when certain combinations of settings are used. You may find there are absolutely no good defaults.
A Moderately Smart Importer Would Need To Know Massive Amounts of History Of All Exporters
In order to perfectly solve these ambiguities and produce .blend files that are similar to what originally exporter the OBJ file, it would be incredibly useful to know about the behavior of the exporter that exported the OBJ in question. This means an importer would need to know all the bugs and features of every exporter, and we don’t even know that after developing the exporter. Bugs are waiting to be discovered, or used by artists until they have to be turned into a feature. Our exporter currently struggles to take into account past bugs, and that’s the exporter!
This turns into it’s own ambiguities to solve: “Is this OBJ’s mention of deprecated ATTR_s a choice the artist made. despite the deprecation warning, or was it a bug that this was still getting written, or was this OBJ written before it was deprecated?” Now you’re messing not only with valid or not, but what the OBJ is supposed to look like.
Optimizations Create Further Challenges
Exporters often take optimizations to improve an OBJ’s performance and quality. For instance:
Ignoring or appending ATTRs to handle deprecations or obsolete OBJ features
Now you will have even less information or, now, seemingly incorrect information! OBJs are meant for X-Plane, not humans. As such exporters can take many liberties with the content of OBJs as long as they match what the artist meant. This can result in very complex optimizations that might even break our own guidelines, all to deliver the best (and deliver on time) to the consumer and our artists. This makes developing an importer that reproduces the importer OBJ, either exactly or simply visually matching (let alone animated or textured properly) even harder.
In a raw .blend file, objects are like Lego bricks which get baked into one solid piece. Going in reverse will likely not get the same neat separation. Blender is not smart enough to tell what is a wing shape, what is a wheel shape, what is a throttle level shape. It may be impossible to separate the vertices back into these distinct meshes (especially after optimizations)! Making a “really smart” importer that is shape aware is a brutally hard algorithm that the world’s best computer scientists are still attempting to solve. It may not be challenge you want to take on.
We May Build A Reasonable Importer One Day!
With all these challenges, an importer would have to be willing to not handle all edge cases and not attempt to reverse engineer an OBJ back to the exact .blend file that made it. For instance, a 3 year old OBJ would not be reversed engineered into the .blend file that produced it 3 years ago, especially since an artist would want to then update their assets anyway! Having art assets “marooned” on other exporters or “dead” is a pretty terrible waste of time, likely be more painful than hand fixing all lot of little things. As Ben pointed out “If we can’t make a direct import [.skp->.blend, .ac->.blend] path, OBJ import is the least bad option.
2.49 to 2.7x Converter
A 2.49 converter is a much more manageable project (far on the back-burner.) The complexity of this tool is much more manageable, because the 2.49 is not in active development and you are converting .blend to .blend, not .obj to .blend. Blender to Blender is something which Blender is already very good at.
While everyone looks at the new UI in awe, X-Plane 11 also had a few important changes under the hood. With Aerosoft Navdata Pro now also supporting X-Plane 11 beta, let’s talk about one of the most boring technical features of X-Plane 11: The completely redesigned database for navigational data, which makes it much easier for data providers like Aerosoft and Navigraph to supply data updates for the X-Plane navigational facilities, while preserving scenery compatibility.
The most important goal when designing the new database was to eliminate the duplication between data in X-Plane’s world and X-Plane’s navigation systems to leave less room for subtle inconsistencies. I also wanted to address compatibility of navdata updates and global scenery (mostly concerning localizers at airports). Other improvements were the integration of SBAS path points (needed for LPV approaches) and RNP service volumes. Last but not least I wanted the ability to work with ARINC424 data directly, and eliminate most of the subtle encoding differences that result from different providers generating files with slightly different converters.
The specification of the database was finalized in September, and both Navigraph and Aerosoft were provided the tools they needed to create navdata for X-Plane 11 in the new format. Actually, we are not limited to those “big two” – as the tool is available for everyone, open-data purists can actually generate their own navdata for the US and Canada using the FAA’s file.
With the great power of the unified database comes great responsibility: The navigational data can only be as good as the world scenery it is placed in, especially the airports. Some of X-Plane’s airports in the default scenery have not kept up with the pace the real world is evolving at: runways are renamed (due to magnetic shift), extended, built or closed and X-Plane’s airport scenery is only as good as the community who cares for it. To make their life easier, we are currently working on a big automated scenery update on the server side. We will rename several thousand runways all over the world on the scenery gateway soon, and this will solve the most annoying issue people are currently facing with the new database: runways not being found because they have been renumbered.
This automatic scenery update is however only part of the solution – because we can only rename runways we have! If an airport is extended in the real world because new runways are built, we rely on the scenery gateway and its incrediblecommunity for updated airports.
I took the time to write an even more boring technical article on how everything works together in X-Plane 11: Navdata in X-Plane 11. If you are an end-user, you don’t need to bother, because here’s the TL;DR: It’s awesome. It gives you RNP approaches for airliners, and LPV approaches for GA aircraft.
Writing this article though, when I compare it to the new UI, I can’t help but feel like the poor guy in this webcomic because this is exactly how the end-user will experience the change: Most won’t even notice.
A user asked me whether we would ever have a conditional directive in the OBJ format to let an author vary the look of the object based on weather. My answer:
Definitely not – this is definitely not the intention of conditionals!
The goal of conditionals is to allow authors to cope with multiple “rendering modes” of the engine, where:
The look of the sim is radically different between the rendering modes and
The performance cost of working around the lower rendering mode is expensive.
So for example, when shadows are off, you have to use some ATTR state change to drop a fake shadow on an object. This kills instancing, so when shadows are on (and the sim needs more performance), leaving that double shadow is not only ugly, but slow.
The conditional works by fully “stripping” those commands for shadowing from the object, then re-analyzing it.
So the win of conditionals is that you do not pay for what you don’t use, because everything is done on load – it’s like shipping two highly optimized objects.
But the down-side is: conditionals require scenery to reload, which is slow, disruptive and annyoing. This is why we would never use them for something that is dynamically changing in the sim, like weather or time of day.
We do want to have weather effects, but our approach is going to be very different: to have a “weather map” texture on the object (same UV as the object) indicating what sections of the texture will receive changes to their material due to weather effects. This will allow the object to look progressively more wet over time without a reload as the rain falls.
X-Plane 10.45 fixed one of the big “ecosystem” problems with the global airports by excluding airport objects by ICAO code. This makes the global airports much less likely to conflict with custom scenery, even if that custom scenery lacks exclusion zones.
For 10.50, I am looking at another change to how we manage airports that should help gateway authors: per airport control of flattening.
As of X-Plane 10.45, airport flattening is a user setting; a user can pick to have all airports flat, or all airports follow the terrain contours.
This is a rather silly setup. One size does not fit all for airport flattening, and the author of the airport is more likely to know how the airport will look with/without flattening than a user who is flying to that airport. It certainly isn’t efficient to have everyone in the X-Plane community set flattening individually without authors being able to set up their airport the way they want.
From what I’ve seen, there are two legitimate uses of airport flattening.
To fix bugs in the underlying terrain, e.g. if the DSF is screwed up, then the airport may need to flatten it to make the airport usable at all. We want this to be the exception, not the rule. (X-Plane 10 is more buggy than past X-Plane releases WRT bumpy runways, so this may sound funny right now, but overall we aim to have users be able to fly with non-flat runways.)
To accommodate highly customized airports where the 3-d models depend on a flat surface.
In X-Plane 10.50, it should be possible (using a new version of WED) to mark an airport as “always flatten”. The expected usage is:
Users leave “runways follow terrain contours” on, all of the time.
Authors mark individual airports as needing flattening, e.g. to fix bugs or accommodate custom scenery.
There is one use case that isn’t handled well by this: if an airport needs different flattening based on different base meshes, there is no way to tag that in the airport. But I view this as a fairly difficult problem to solve with existing technology – we would need a 2-d grid matching every custom version of an airport against every version of the mesh ever to exist.
Fixing the Mesh
Please note that this feature is not a replacement for being able to customize mesh elevation at a local airport from an overlay. “Mesh patching” is what we want to be able to do in the long term, but for X-Plane’s engine, it also means a lot of complicated internal changes to how the rendering engine works; customizing flattening is something I can ship now in 10.50 that at least helps.
Flattening is also not a replacement for fixing bugs in the base meshes themselves; one trend I have observed over my decade+ working on X-Plane (!) is that the accuracy of source data keeps getting better. Ten years ago it would be silly to say “how about if everyone just uses the real world values for their scenery and it will just work”. At this point that’s not actually a pipe dream, it is often completely manageable. So my hope is that someday we can reach a point where the terrain is just accurate and everyone uses it.
As of right now I have this code working in X-Plane but I do not have a build of WED that supports it. We are still in the middle of working on 10.50 apt.dat features, so I’m hoping to post something on that soon.
Ondrej has posted a public thread about the new version of the Blender 2.7 exporter here.
The 3.3 release is the first release where we (Laminar Research) have worked closely with Ondrej to build what we hope will be one of the future cornerstones of modeling for X-Plane.
If you are currently using Blender 2.49 or AC3D, I expect that 2.73 will provide the best way forward for modeling in X-Plane.
New Stuff
The new release has a few major features, all aiming to create a new modern work-flow:
All animation bugs are fixed. (At least,we think – if you find one, please report it ASAP!) This means animation is WYSIWYG. Armatures are supported for animation as well as animated data block containers.
The exporter understands all modern OBJ features, including ones scheduled for release in X-Plane 10.50.
Instancing is handled via a single option with exporter validation, so you don’t have to know how instancing works to get instancing.
The material rules are validated, and materials are found automatically; you can simply model as you want and Blender will do its best to export it or tell you if there is a problem.
Specular and Normal maps are ‘merged’ together from two separate sources. This lets you set specularity and normal maps in separate material channels in Blender for a more WYSIWYG approach. It also means no more messing with Photoshop to combine these layers.
The goal of many of these is to hide the idiosyncrasies of X-Plane’s modeling format and provide a cleaner, more artist-friendly view of modeling.
Regarding instancing: the model we have adopted is the one we used internally on the 2.49 exporter: you (the artist) tag an export as either “instanced” or not.
If instancing is on, the exporter writes only data to the OBJ that will be hardware-instanced by X-Plane. If you do something that cannot be instanced, like animation, you get an export error telling you what you have to remove.
If instancing is off, the exporter writes the object normally.
The win here is that you don’t have to know the (complicated) instancing rules; set instancing for the scenery objects you need to make fast in bulk (e.g. a luggage cart, a house, something that will be used many times in a small area) and you’ll get optimal output.
Optimization – Coming Soon
The goal of the 3.3 release was to set up an environment where authors could work using the new work flow. We have not finished optimizing the OBJ output.
If you are using the existing 2.7 exporter for airplane work, the output should be similar in performance. If you are using the 2.49 exporter, the new output is (like the current 2.7 exporter) not as well optimized.
In a future update, we will tighten up the generated OBJ code; this should not affect anyone other than producing better OBJs.
Compatibility
The new exporter should be fully work-flow compatible with the previous Blender 2.7 exporter; if you find your existing project does not work, please file a bug!
Our plan is to create a migration tool for Blender 2.49 projects; this will forward-convert the datarefs on annotations, manipulators, and object properties from 2.49 to 2.73 format. This lets authors using 2.49 move forward to 2.73 and have a migration path for existing content. (This work is not started yet, but is planned – we have our own aircraft we need to keep working on.)
MH1212, developer of the Prefab Airports for X-Plane, requested this feature, and it looks like we are going to be able to sneak it into X-Plane 10.45. (If we hit bugs, it might get pushed out to 10.50, but so far things look okay.) The feature is: the ability to exclude objects by airport ID without using exclusion zones.
Right now when a custom scenery pack replaces an airport (via apt.dat), the old apt.dat is completely ignored. But the DSF-based overlay objects, facades, etc. are included; the custom scenery pack has to use exclusion zones to kill them off.
With this extension, the DSF-based overlay objects in a scenery pack can act as if they are in the apt.dat file, disappearing when the apt.dat airport is replaced. This means that when you replace an airport (via apt.dat file) not only do the runways go away, but so do the overlay elements.
Here’s the win: we can export our global airports from the X-Plane Scenery Gateway this way, and custom scenery will remove the overlay elements automatically just by replacing the apt.dat, even if no exclusion zones are present.
Here are some details:
The scheme works if the underlying airport is correctly marked as having its objects and facades “inside” the airport. So unlike exclusion zones, this scheme works if the underlying airport is modified, not the overlaying airport.
This is a new scheme – no existing scenery already does this; scenery must be re-exported to gain access to this functionality.
The functionality requires the latest version of X-Plane, but is harmless to old X-Planes – the DSFs will still load.
Exclusion zones still work and are still recommended; if you are making custom scenery and you are on top of autogen or an old scenery pack that is not modified using this new scheme, only an exclusion zone will save you.
There are two big advantages of this scheme:
We (Laminar Research) re-export our collection of thousands of airports on a regular basis, so we can tag the entire set of global airports using this new scheme and have them ready for by-airport-ID exclusion very quickly. So this scheme will start to help conflicts immediately.
The scheme doesn’t require modifying the overlying scenery at all. There are freeware airports that are effectively orphaned – the author cannot be found and the license doesn’t allow the community to modify them*. Since these airports cannot be legally redistributed with exclusion zones, this technique will help.
Once X-Plane has this extension and the global airports are re-exported using it, global airports will fully disappear when any custom scenery pack replaces the airport by apt.dat, even if the custom scenery pack doesn’t have good exclusion zones.
This functionality will be available to third parties in WED 1.5 when it goes beta. In WED 1.5, if an overlay element is in the folder for an airport, it will be excluded when that airport is excluded. If an overlay element is ‘loose’ in the outer level hierarchy, it will not be excluded by airport (but will be affected by other pack’s exclusion zones).
Since gateway airports already have the objects “in” the airport folder, they are already authored to make this scheme work.
If you create your own DSFs using a low level tool like DSF2Text, the DSFLib source code, or something else crazy, I have posted the proposed schema here. That’s a technical link for people tinkering with the DSF format itself, but if you’re in that category, please do ping me to get early test materials. (The new code is also on GitHub.)
Two warnings to custom scenery authors: if you are creating a custom airport scenery pack, especially payware, please read these very carefully:
This is not an invitation to stop using exclusion zones. There are plenty of scenery packs in the wild that do not have their objects tagged by airport, as well as autogen and all sorts of junk that can be under your payware. If your airport doesn’t have an exclusion zone and it conflicts with another pack, it is your fault. Go add exclusion zones, like I told you to do yearsago.
If a scenery pack from X-Plane Airport Gateway is removed by your pack (by airport ID) then our exclusion zones are removed too. This means that if trees on the runway have been removed by an X-Plane Airport Gateway airport, you will no longer get those exclusion zones for free. You must exclude those trees yourself! (If you put exclusion zones in from point 1 this is a non-issue.) Test your airport with global airports enabled and disabled to make sure your pack is good.
* As a side note I consider it a real problem that airports get uploaded and shared with the community under licenses that don’t allow for abandon-ware to be maintained. It’s clearly the right of the author to use any license you want, but as a community I hope we can encourage freeware authors to use a permissive open license.
This may have happened to you: you import the latest approved airport scenery pack from the X-Plane Airport Gateway and modify it. When you go to export your improvements back to the gateway, WED says your work is invalid and has a problem — but not in the changes you made!
Huh? If this was the approved airport on the Gateway, how can it also be invalid in WED? How did that other author upload the airport before?
You Get a Free Pass
What do we do if we find that scenery and airports have bad data that is causing bugs? What if we can’t just fix the scenery pack for you? The policy we’ve been using is: “you get a free pass until your next scheduled maintenance; then you have to fix the bug.”
The X-Plane Scenery Gateway is a good example of this. Sometimes we find new categories of authoring mistakes – I write improvements to WED’s scenery validation function to catch these authoring errors. These are errors like:
Typos in taxiway signs – the old syntax makes a sign with incorrect letters.
Overlapping ATC routes – with the overlapping routes the AI aircraft do not taxi correctly.
In other words, these are situations where the WED scenery pack, if left alone, is causing clearly bad things to happen inside X-Plane. This isn’t a case of “old style or new style”, it’s a case of “broken or fixed.”
So what we do is we set WED to reject these scenery packs on future uploads, but we do not delete the airports from X-Plane itself. In other words, the airports with serious mistakes get a free pass until the next time an author does scheduled maintenance.
Then when an author is working in WED and is going to update the airport anyway, the author must fix the problems we have found.
No One Likes a Fire Drill
This strategy is a compromise between two extremes:
If we force you to fix your airport right now (e.g. “fix the airport or we remove it from X-Plane”), that’s not fun, that’s a fire drill. And having been exposed to plenty of fire drills myself dealing with the new OS X 10.11 and Windows 10 releases, I’m sympathetic to authors not wanting to have to stop everything to deal with an issue ASAP.
If we never require that these kinds of problems be fixed, they simply won’t be fixed. The overwhelming evidence that I have seen from working on authoring tools for X-Plane for over a decade is that some authors won’t fix problems unless the tools force them to do it.**
So requiring the change when the author does maintenance but not requiring the existing shipped scenery pack be modified strikes me as the best possible compromise: we still get quick responses to serious bugs, but we don’t force anyone to drop everything.
* I do realize that some authors are incredibly diligent about getting their scenery to be correct even if the tools don’t force them to do so! But the goal here is to have all scenery be correct; it only takes one broken scenery pack in the hundreds a user installs to crash X-Plane.
I’ve been meaning for weeks now to write up some notes about 10.40. I’ve also been trying to put the hood back on 10.40 so we can get it to public beta; instead the last few weeks have been a flying circus of driver bugs, expiring certificates, etc.
But we are making progress, so I’ll start off in this post by describing a change in how DSFs are loaded and the wide DSF box.
A Tangent: Stuttering and Pauses
A post that involves X-Plane pausing during flight is going to invariably bring up a bunch of blog comments: “X-Plane stutters while I fly on my really expensive machine!” This is not good, and from what I’ve been hearing, it sounds like something got worse recently; this is something that we are investigating now.
But I also want to mention that historically, X-Plane has never been “no-pause”. What we have done is periodically made the pauses much shorter; our goal is to get down to zero pauses, ever, but this will happen by finding every source of pausing and fixing them one at a time. In other words, we’re in the middle of a process of improving smoothness, but even if one source of pausing is fixed, another source might still be causing problems.
DSF Loading: the Old Way
X-Plane has, since X-Plane 9, loaded DSFs in the background on a second core while you fly. This cuts down the amount of time it takes to change scenery. (Older versions of X-Plane would pause while scenery was loaded and shifted.)
The old DSF loader did have a few weaknesses though:
The sim pauses while DSFs are deleted. As DSFs become bigger, this pause is becoming more noticeable.
If the loader ever gets behind by two scenery shifts, it just waits until it catches up. This is where the dreaded “Async load timed out after 30 seconds” comes in – it indicates that the DSF loader was so far behind that it locked up for half a minute and didn’t catch up.
The old DSF loader loads one DSF at a time, tops.
DSF Loading the New Way
X-Plane 10.40 has the new DSF loader, which both loads and unloads DSFs on worker threads to keep flight smoother. It also will load more than one DSF at a time, limited only by the requirement that adjacent DSFs not be loaded at the same time.
X-Plane 10.40 also has the option of an extended DSF scenery region for sharper terrain; with this option off, two DSFs are loaded at one time during sim boot and one or two are loaded at a time while you fly. With the extended DSF region on, up to four DSFs are loaded at once during sim boot and one or two are loaded while you fly.
The extended DSF scenery region is optional; don’t use this if you’re using HD meshes or you’re short on RAM. The new DSF loader is always on.
