We’ve been doing some work behind the scenes to modernize our servers; as part of that, I have moved the scenery tools code to GitHub.
The GitHub public repositories are the official repositories for WED and the other scenery tools that live in the XPTools code base; I push my changes directly there.
using GitHub means much faster clone times (since the server we had the code on was getting long in the tooth), better public browsing, and you can do all of the normal GitHub things that you’d want, like forking, making pull requests, etc.
If you are using our CGIT browsing (at http://dev.x-plane.com/cgit/cgit.cgi/) please switch to GitHub; CGIT will be going away soon. Similarly, if you have the code checked out, you should change your remote’s URL to https://github.com/X-Plane/xptools.git.
I mentioned in a previous post that we are working with Ondrej on a next-generation Blender 2.7 OBJ exporter. I am excited for this work because I believe it will greatly improve the artist interface to X-Plane. Let’s use instancing as an example to explain the idea.
X-Plane 10 will sometimes draw scenery objects using hardware instancing. When an object is drawn with instancing, performance is a lot better than without it; the huge numbers of objects you can see in the autogen scenery are due to instancing. But the conditions for instancing are, to put it mildly, complicated.
- The object has to be attached to a DSF. Well, that’s not crazy.
- There’s a giant pile of things you’re not allowed to do in an instanced object – animation, material attributes, smoke puffs, bla bla bla. (The object has to be simple so that the GPU can draw a big pile of objects without the CPU intervening to handle things like smoke generation on a per-object basis.)
- Buuuuut…not every ATTRibute is a problem. You can use ATTR_hard in an instanced object, no problem! But don’t use ATTR_shiny_rat – that’ll kill instancing.
- But wait, there are these weirdo new attributes like GLOBAL_specular that seem to replace ATTR_shiny_rat that do work with instancing.
If you model, and this seems confusing, it’s because it is. You were never supposed to see this!
File Formats for Machines and Applications For Humans
My view is this: the file formats of X-Plane are meant to be seen by programmers, not artists. When was the last time you had to hand-edit an .acf? You just use Plane-Maker. You don’t hand-edit a PNG file or worry about its internal compression – you use PhotoShop.
OBJ should be the same way – you should export from a 3-d tool and never have to look inside the OBJ itself. This means the OBJ8 format can serve two purposes:
- Help X-Plane load models quickly and draw good looking content and
- Maintain compatibility so older scenery continues to load.
Note that “be really easy to understand in Notepad” is not on the list!
Blender the Transformer
Now here’s the trick: the rules for OBJ that a 3-d modeler can present can be different from the rules that the OBJ8 format itself has. And that is exactly what we are trying to do in the new Blender exporter.
Rather than expose global attributes and non-global attributes and all of the crazy complexity of OBJ8, our strategy with the new exporter is this:
- When you export a scenery object, you specify if you want the object to be instanced or not.
- If the object is not instanced, Blender creates the best OBJ it can for this case.
- If the object is instanced, Blender tries to create the best OBJ it can for this case.
- If the object is supposed to be instanced but you’ve done something that is not instancing-friendly, the exporter tells you, in terms you can understand.
So for example, if you make an animated radar dish that spins and attempt to export with instancing, the exporter can say “you cannot have animation in an instanced object.” Now you, the scenery designer, can make a decision: is it more important to have the radar dish draw really really fast, or is it more important that it spins?
This is the strategy we used for our modifications of Marginal’s 2.49 exporter* – the artist specifies instancing and gets a check that instancing requirements have met. This means instance-tagged objects will be fast. (It is possible to check instancing within the sim with DataRefEditor but it’s not easy, especially for large projects.)
Don’t Expose the Weird
This idea that Blender can transform your work, and thus hide weird X-Plane rules, is not specific to OBJ, not new to X-Plane, and not even specific to X-Plane. When you look at major 3-d game engines (e.g. for combat games and racing games) a ton of code goes into the “art asset pipeline”, transforming the artist’s original work into something already digested for the 3-d engine.
Another example of this is WED. DSFs have an annoying rule that polygons can’t cross DSF boundaries. There are also some even weirder exceptions for a few special cases, introduced in X-Plane 10.20.
WED knows all of these rules; you draw polygons wherever you want and set the export target for the version of X-Plane you want to fly in, and WED figures out the best way to export your work, chopping up polygons as needed.
Shedding Light on Things
Our current work on the Blender exporter focuses on WYSIWYG animation and the material model (which includes instancing support). There is one more area that I think will make a huge difference to authors that we haven’t started working on yet: lighting.
The lighting values for commands like LIGHT_PARAM are, to put it mildly, completely goofy. If you’ve ever tried to read this, you know that even when things are explained, they make no sense.
No author should ever have to write a LIGHT_PARAM. This is a classic case where the exporter should do the transformation for us. What I would like to get into Blender 2.7 is the ability to place a cone light in Blender and have the exporter write the spill light automatically, e.g. filling in the parameters from the position and shape of the light. You light your scene and you see the same thing in X-Plane.
For years we’ve been behind the curve on user experience when it comes to 3-d modeling. I think we’re starting to fix some of those problems in a major way, which should help make the power of X-Plane 10’s rendering engine quite a bit more accessible.
*As a side note, I have no idea what the current 2.49 script status is. We view the 2.7 scripts as the long term way forward, due to Blender 2.49’s not being maintained, so our effort on a truly user-friendly tool is going into 2.7. But we also maintain updates to 2.49 to support our internal team. X-Plane 10.50 will feature a few new OBJ commands, so I will try to post our branch of the 2.49 scripts (which do have this new support) for anyone whose aircraft is in 2.49.
In the long term the 2.7 scripts will feature a migration tool, but I expect we will have a useful 2.7 release (for 2.7 authors) without migration first, and then migration will come later.
If you have a computer that can handle more detailed base meshes, AlpilotX’s free HD and (for those with serious hardware, UHD) meshes are a really great looking enhancement to X-Plane. Alpilotx just released a new HD version of South America with more up-to-date land class and OSM data.
Here are comparison screenshots.
Update: the comments form is fixed – sorry about that!
I’ve been working a bit on scenery tools this week. Here’s a road map of some of what’s coming up.
WED and the X-Plane Scenery Gateway
We are working on a release of the latest scenery gateway airports for X-Plane 10.45. Like all releases, we’ve found problems with airports that WED does not detect. So I will try to release a new version of WED with stricter validation soon.
Airport Parking Spots
Austin is working on new code to place static aircraft at unused ramp parking spots. I’l describe this in more detail in another post, but here are some key points:
- As an airport author, you simply place ramp starts, not static aircraft.
- X-Plane will feature some new static aircraft in the update.
- Third parties can further add to the static aircraft and have them be used via the library system.
- There will be a revision to the apt.dat format that stores more data per ramp spot and a WED update to support this new data.
- Since there may be scenery now with static aircraft on top of ramp starts*, we will auto-remove static aircraft from ramp starts in the gateway export as a temporary measure until authors can resolve the situation and post the fixes to the gateway.
- We are not removing the old static aircraft from the library, but we will be hiding them from WED’s library so that authors use ramp starts instead of placing them by hand.
Static aircraft in parking spots will ship next year, not this year, but is planned as a v10 update.
Scenery Tools on OS X
I made significant progress this week porting WED for OS X to 64-bit and modern Mac APIs. This effort basically meant rewriting the OS-level UI code in WED to be new AppKit based calls instead of the old Carbon API.
What this means is that developers will be able to build all of the scenery tools on a modern Mac running Yosemite or El Capitan with X-Code 7. Since Chris has already updated the projects to compiled WED in MSVC, a developer can build all of the common scenery tools from the most widely used open source IDEs.
It also brings us closer to a 64-bit WED on Mac, which is desperately needed since OS X provides the least amount of usable address space to 32-bit apps. I am not sure of the 64-bit status of the Windows WED build; it may need more work on the libraries.
Will It Blend
This might be the most significant scenery tools development: we’ve been working with Ondrej (der-On on Git-Hub) on new features for the Blender 2.7 exporter. This will include:
- Much better animation support. Complete WYSIWYG animation of data blocks and bones, with no work-arounds needed. Bone animations match Blender 2.49 so you can move your projects forward.
- Modernized material support including instancing for a really clean work-flow that takes advantage of X-Plane 10’s features.
- Updates for the latest OBJ features.
- Our plan is to create a migration script to bring 2.49 projects into 2.7, converting the special tags used for an X-Plane 2.49 export into 2.7.
Many of our artists still use 2.49, but there’s no question that 2.49’s days are numbered. It’s a question of when it stops working on OS X, not if. With the migration step, we can move our projects forward without artists having to redo work.
Like the previous 2.7 exporter, this one is open source and will be free, and should be able to export existing 2.7-type projects with no modifications – we are trying to maintain full backward compatibility.
* This situation is slightly silly – if the user picks the ramp start to fly, the user will be on top of a static aircraft.
This may have happened to you: you import the latest approved airport scenery pack from the X-Plane Airport Gateway and modify it. When you go to export your improvements back to the gateway, WED says your work is invalid and has a problem — but not in the changes you made!
Huh? If this was the approved airport on the Gateway, how can it also be invalid in WED? How did that other author upload the airport before?
You Get a Free Pass
What do we do if we find that scenery and airports have bad data that is causing bugs? What if we can’t just fix the scenery pack for you? The policy we’ve been using is: “you get a free pass until your next scheduled maintenance; then you have to fix the bug.”
The X-Plane Scenery Gateway is a good example of this. Sometimes we find new categories of authoring mistakes – I write improvements to WED’s scenery validation function to catch these authoring errors. These are errors like:
- Typos in taxiway signs – the old syntax makes a sign with incorrect letters.
- Overlapping ATC routes – with the overlapping routes the AI aircraft do not taxi correctly.
In other words, these are situations where the WED scenery pack, if left alone, is causing clearly bad things to happen inside X-Plane. This isn’t a case of “old style or new style”, it’s a case of “broken or fixed.”
So what we do is we set WED to reject these scenery packs on future uploads, but we do not delete the airports from X-Plane itself. In other words, the airports with serious mistakes get a free pass until the next time an author does scheduled maintenance.
Then when an author is working in WED and is going to update the airport anyway, the author must fix the problems we have found.
No One Likes a Fire Drill
This strategy is a compromise between two extremes:
- If we force you to fix your airport right now (e.g. “fix the airport or we remove it from X-Plane”), that’s not fun, that’s a fire drill. And having been exposed to plenty of fire drills myself dealing with the new OS X 10.11 and Windows 10 releases, I’m sympathetic to authors not wanting to have to stop everything to deal with an issue ASAP.
- If we never require that these kinds of problems be fixed, they simply won’t be fixed. The overwhelming evidence that I have seen from working on authoring tools for X-Plane for over a decade is that some authors won’t fix problems unless the tools force them to do it.**
So requiring the change when the author does maintenance but not requiring the existing shipped scenery pack be modified strikes me as the best possible compromise: we still get quick responses to serious bugs, but we don’t force anyone to drop everything.
* I do realize that some authors are incredibly diligent about getting their scenery to be correct even if the tools don’t force them to do so! But the goal here is to have all scenery be correct; it only takes one broken scenery pack in the hundreds a user installs to crash X-Plane.
I’m declaring WorldEditor 1.4 to be final. From now on, you must use WED 1.4 to upload to the X-Plane Scenery Gateway. If you have not upgraded yet, please do; you can download it here!
I cut a new release candidate for WorldEditor: WED 1.4r2 – please update to this latest build. (1.4r1 had a crash-on-export under a relatively rare configuration of lines in a scenery pack.)
I’ve been meaning for weeks now to write up some notes about 10.40. I’ve also been trying to put the hood back on 10.40 so we can get it to public beta; instead the last few weeks have been a flying circus of driver bugs, expiring certificates, etc.
But we are making progress, so I’ll start off in this post by describing a change in how DSFs are loaded and the wide DSF box.
A Tangent: Stuttering and Pauses
A post that involves X-Plane pausing during flight is going to invariably bring up a bunch of blog comments: “X-Plane stutters while I fly on my really expensive machine!” This is not good, and from what I’ve been hearing, it sounds like something got worse recently; this is something that we are investigating now.
But I also want to mention that historically, X-Plane has never been “no-pause”. What we have done is periodically made the pauses much shorter; our goal is to get down to zero pauses, ever, but this will happen by finding every source of pausing and fixing them one at a time. In other words, we’re in the middle of a process of improving smoothness, but even if one source of pausing is fixed, another source might still be causing problems.
DSF Loading: the Old Way
X-Plane has, since X-Plane 9, loaded DSFs in the background on a second core while you fly. This cuts down the amount of time it takes to change scenery. (Older versions of X-Plane would pause while scenery was loaded and shifted.)
The old DSF loader did have a few weaknesses though:
- The sim pauses while DSFs are deleted. As DSFs become bigger, this pause is becoming more noticeable.
- If the loader ever gets behind by two scenery shifts, it just waits until it catches up. This is where the dreaded “Async load timed out after 30 seconds” comes in – it indicates that the DSF loader was so far behind that it locked up for half a minute and didn’t catch up.
- The old DSF loader loads one DSF at a time, tops.
DSF Loading the New Way
X-Plane 10.40 has the new DSF loader, which both loads and unloads DSFs on worker threads to keep flight smoother. It also will load more than one DSF at a time, limited only by the requirement that adjacent DSFs not be loaded at the same time.
X-Plane 10.40 also has the option of an extended DSF scenery region for sharper terrain; with this option off, two DSFs are loaded at one time during sim boot and one or two are loaded at a time while you fly. With the extended DSF region on, up to four DSFs are loaded at once during sim boot and one or two are loaded while you fly.
The extended DSF scenery region is optional; don’t use this if you’re using HD meshes or you’re short on RAM. The new DSF loader is always on.
It’s come to my attention that the Scenery Gateway has a serious shortcoming:
There’s currently no good way to get feedback from other users.
This is a shame, because when you upload scenery to a site like X-Plane.org, you get to hear from real people using your scenery—they might tell you how much they appreciate your work, or suggest ways you could improve it.
There’s a tradeoff here: because all airports on the Gateway are periodically exported to X-Plane, users don’t have to seek out your scenery in order to enjoy it; they get it automatically.
That means your work has a much wider impact—it benefits hundreds or even thousands of times as many users compared to posting to a download site. But, it also means people who use your scenery probably don’t know your name.
So, if you’re a Gateway artist, I’d like to hear your thoughts on a couple things:
- How do you feel about the current situation? Do you want a way to get feedback (and kudos) on your Gateway submissions?
- If so, what would be the best way(s) to give that feedback? A few possibilities I can think of include:
- A “like” button for your scenery pack or the airport as a whole
- Pros: Easy to understand, zero friction for people leaving feedback (means more people are likely to use it)
- Cons: Impersonal (compared to text-based comments like “I love this scenery!”), doesn’t solve the problem of hearing from X-Plane users who never visit the Gateway
- The ability for other users to leave comments on your scenery pack
- Pros: Very personal, the “standard” way to get feedback on other download sites
- Cons: Higher friction (fewer people will use this compared to a “like” button, for instance), doesn’t solve the problem of hearing from X-Plane users who never visit the Gateway
- Automatic estimates of how many X-Plane users see your scenery pack each month/year—for instance, you submit a scenery pack for KBOS, and you see that x,000 people fly there each month
- Pros: Gets “feedback” (such as it is) from users who never visit the Gateway, gives a very clear indication of how many users you impact
- Cons: Impersonal
- Something else entirely?
Disclaimer: As with any discussion of future features, I can’t promise we’ll implement any of the above… but I can tell you we’ll consider it.
So, drop a comment below and let me know what you think!
UPDATE: As of July 11, we now have a basic discussion system on the Gateway which integrates with the existing bug reports. Give it a try and let me know what you think!
I just posted WorldEditor 1.4 – “WED”, as we seem to call it most of the time – release candidate one for Mac, Windows and Linux – see the WorldEditor page for download links, report bugs on the gateway bug reporter.
Besides fixing a few bugs, this build has a new certificate so uploading/downloading from the gateway will work.
WED has a validate function that finds authoring problems in your scenery packs; two notes on this function:
First, I have been trying to improve the validation every time an issue comes up with a scenery pack. Therefore WED 1.4’s validation is quite a bit more strict than 1.3’s validation.
Once WED 1.4 goes final, it will become the required version to upload to the gateway. This ensures that we have the strongest validation, and saves Julian from having to hand check authoring issues.
This means that when you go to make a modification to your existing work, you might not be able to resubmit it until you fix existing problems. Your project didn’t change, the validation just became tighter.
Second, I have added the ATC layout error checks to the validation step. These checks (select crossing routes, select doubled nodes, select degenerate route lines) have been available since WED 1.2, but you now have to have a ‘clean’ layout to build your project. I’m hoping that this stricter validation will help fix layout bugs; when the layout is wrong, the AI aircraft’s taxi routes become even weirder than normal.