Category: Scenery

A New Way To Exclude

MH1212, developer of the Prefab Airports for X-Plane, requested this feature, and it looks like we are going to be able to sneak it into X-Plane 10.45. (If we hit bugs, it might get pushed out to 10.50, but so far things look okay.)  The feature is: the ability to exclude objects by airport ID without using exclusion zones.

Right now when a custom scenery pack replaces an airport (via apt.dat), the old apt.dat is completely ignored. But the DSF-based overlay objects, facades, etc. are included; the custom scenery pack has to use exclusion zones to kill them off.

With this extension, the DSF-based overlay objects in a scenery pack can act as if they are in the apt.dat file, disappearing when the apt.dat airport is replaced. This means that when you replace an airport (via apt.dat file) not only do the runways go away, but so do the overlay elements.

Here’s the win: we can export our global airports from the X-Plane Scenery Gateway this way, and custom scenery will remove the overlay elements automatically just by replacing the apt.dat, even if no exclusion zones are present.

Here are some details:

  1. The scheme works if the underlying airport is correctly marked as having its objects and facades “inside” the airport. So unlike exclusion zones, this scheme works if the underlying airport is modified, not the overlaying airport.
  2. This is a new scheme – no existing scenery already does this; scenery must be re-exported to gain access to this functionality.
  3. The functionality requires the latest version of X-Plane, but is harmless to old X-Planes – the DSFs will still load.
  4. Exclusion zones still work and are still recommended; if you are making custom scenery and you are on top of autogen or an old scenery pack that is not modified using this new scheme, only an exclusion zone will save you.

There are two big advantages of this scheme:

  1. We (Laminar Research) re-export our collection of thousands of airports on a regular basis, so we can tag the entire set of global airports using this new scheme and have them ready for by-airport-ID exclusion very quickly. So this scheme will start to help conflicts immediately.
  2. The scheme doesn’t require modifying the overlying scenery at all. There are freeware airports that are effectively orphaned – the author cannot be found and the license doesn’t allow the community to modify them*. Since these airports cannot be legally redistributed with exclusion zones, this technique will help.

Once X-Plane has this extension and the global airports are re-exported using it, global airports will fully disappear when any custom scenery pack replaces the airport by apt.dat, even if the custom scenery pack doesn’t have good exclusion zones.

This functionality will be available to third parties in WED 1.5 when it goes beta. In WED 1.5, if an overlay element is in the folder for an airport, it will be excluded when that airport is excluded. If an overlay element is ‘loose’ in the outer level hierarchy, it will not be excluded by airport (but will be affected by other pack’s exclusion zones).

Since gateway airports already have the objects “in” the airport folder, they are already authored to make this scheme work.

If you create your own DSFs using a low level tool like DSF2Text, the DSFLib source code, or something else crazy, I have posted the proposed schema here. That’s a technical link for people tinkering with the DSF format itself, but if you’re in that category, please do ping me to get early test materials. (The new code is also on GitHub.)

Two warnings to custom scenery authors: if you are creating a custom airport scenery pack, especially payware, please read these very carefully:

  1. This is not an invitation to stop using exclusion zones. There are plenty of scenery packs in the wild that do not have their objects tagged by airport, as well as autogen and all sorts of junk that can be under your payware. If your airport doesn’t have an exclusion zone and it conflicts with another pack, it is your fault. Go add exclusion zones, like I told you to do years ago.
  2. If a scenery pack from X-Plane Airport Gateway is removed by your pack (by airport ID) then our exclusion zones are removed too. This means that if trees on the runway have been removed by an X-Plane Airport Gateway airport, you will no longer get those exclusion zones for free.  You must exclude those trees yourself!  (If you put exclusion zones in from point 1 this is a non-issue.) Test your airport with global airports enabled and disabled to make sure your pack is good.

 

 

* As a side note I consider it a real problem that airports get uploaded and shared with the community under licenses that don’t allow for abandon-ware to be maintained. It’s clearly the right of the author to use any license you want, but as a community I hope we can encourage freeware authors to use a permissive open license.

Posted in Development, File Formats, News, Scenery, Tools by | 15 Comments

A New WED Release and 64-bit WED

WorldEditor 1.4.1 is out – it’s a very small patch to 1.4.0 that increases the strictness of validation to catch a number of errors we found in the last export from the X-Plane Scenery Gateway.

Please upgrade!  WED 1.4.1 will become the mandatory version to upload to the X-Plane Scenery Gateway in a few days if we don’t find any major bugs.

64-Bit WED

Last week I completed most of the work to modernize WED and the scenery tools for modern OS X/X-Code, which finally lets us build 64-bit Mac scenery tools. (This also lets anyone compile WED without having to find a Mac from the dark ages.)

WorldEditor 1.4.1 will be the last 32-bit build of WorldEditor. The next update (WED 1.5) will be 64-bit on all three platforms: Mac, Windows and Linux. This transition will also raise WED’s minimum operating system for OS X to OS X 10.7 (Lion).

Having WED be 64-bit on all platforms should solve problems that advanced scenery authors are seeing where WED runs out of memory when editing a custom scenery pack with a lot of textures or reference imagery.

WED 1.5 Features

I am not sure of the feature list for WED 1.5 yet; 1.4.1 has been kept intentionally small to get a quick patch out. (The longer we have a WED that does not properly validate airports, the more weird stuff ends up in the scenery gateway.) Two features are very likely for WED 1.5:

  • 64-bit on all operating systems.
  • Support for apt.dat file format extensions to coincide with X-Plane 10.50.

My plan is to have WED 1.5 ready for beta before X-Plane 10.50 is ready for beta, so we can test X-Plane’s new capabilities with WED.

We have internal notes on the apt.dat 1050 extensions; I’ll get an RFC posted in the next few days.

Posted in Development, Scenery, Tools by | 5 Comments

Scenery Tools Source Code on GitHub

We’ve been doing some work behind the scenes to modernize our servers; as part of that, I have moved the scenery tools code to GitHub.

The GitHub public repositories are the official repositories for WED and the other scenery tools that live in the XPTools code base; I push my changes directly there.

using GitHub means much faster clone times (since the server we had the code on was getting long in the tooth), better public browsing, and you can do all of the normal GitHub things that you’d want, like forking, making pull requests, etc.

If you are using our CGIT browsing (at http://dev.x-plane.com/cgit/cgit.cgi/) please switch to GitHub; CGIT will be going away soon. Similarly, if you have the code checked out, you should change your remote’s URL to https://github.com/X-Plane/xptools.git.

Posted in Development, Scenery, Tools by | 7 Comments

Making 3-d Modeling Less Weird

I mentioned in a previous post that we are working with Ondrej on a next-generation Blender 2.7 OBJ exporter. I am excited for this work because I believe it will greatly improve the artist interface to X-Plane. Let’s use instancing as an example to explain the idea.

X-Plane 10 will sometimes draw scenery objects using hardware instancing. When an object is drawn with instancing, performance is a lot better than without it; the huge numbers of objects you can see in the autogen scenery are due to instancing. But the conditions for instancing are, to put it mildly, complicated.

  • The object has to be attached to a DSF. Well, that’s not crazy.
  • There’s a giant pile of things you’re not allowed to do in an instanced object – animation, material attributes, smoke puffs, bla bla bla. (The object has to be simple so that the GPU can draw a big pile of objects without the CPU intervening to handle things like smoke generation on a per-object basis.)
  • Buuuuut…not every ATTRibute is a problem. You can use ATTR_hard in an instanced object, no problem! But don’t use ATTR_shiny_rat – that’ll kill instancing.
  • But wait, there are these weirdo new attributes like GLOBAL_specular that seem to replace ATTR_shiny_rat that do work with instancing.

If you model, and this seems confusing, it’s because it is. You were never supposed to see this!

File Formats for Machines and Applications For Humans

My view is this: the file formats of X-Plane are meant to be seen by programmers, not artists. When was the last time you had to hand-edit an .acf? You just use Plane-Maker. You don’t hand-edit a PNG file or worry about its internal compression – you use PhotoShop.

OBJ should be the same way – you should export from a 3-d tool and never have to look inside the OBJ itself. This means the OBJ8 format can serve two purposes:

  • Help X-Plane load models quickly and draw good looking content and
  • Maintain compatibility so older scenery continues to load.

Note that “be really easy to understand in Notepad” is not on the list!

Blender the Transformer

Now here’s the trick: the rules for OBJ that a 3-d modeler can present can be different from the rules that the OBJ8 format itself has. And that is exactly what we are trying to do in the new Blender exporter.

Rather than expose global attributes and non-global attributes and all of the crazy complexity of OBJ8, our strategy with the new exporter is this:

  • When you export a scenery object, you specify if you want the object to be instanced or not.
  • If the object is not instanced, Blender creates the best OBJ it can for this case.
  • If the object is instanced, Blender tries to create the best OBJ it can for this case.
  • If the object is supposed to be instanced but you’ve done something that is not instancing-friendly, the exporter tells you, in terms you can understand.

So for example, if you make an animated radar dish that spins and attempt to export with instancing, the exporter can say “you cannot have animation in an instanced object.” Now you, the scenery designer, can make a decision: is it more important to have the radar dish draw really really fast, or is it more important that it spins?

This is the strategy we used for our modifications of Marginal’s 2.49 exporter* – the artist specifies instancing and gets a check that instancing requirements have met. This means instance-tagged objects will be fast. (It is possible to check instancing within the sim with DataRefEditor but it’s not easy, especially for large projects.)

Don’t Expose the Weird

This idea that Blender can transform your work, and thus hide weird X-Plane rules, is not specific to OBJ, not new to X-Plane, and not even specific to X-Plane. When you look at major 3-d game engines (e.g. for combat games and racing games) a ton of code goes into the “art asset pipeline”, transforming the artist’s original work into something already digested for the 3-d engine.

Another example of this is WED. DSFs have an annoying rule that polygons can’t cross DSF boundaries. There are also some even weirder exceptions for a few special cases, introduced in X-Plane 10.20.

WED knows all of these rules; you draw polygons wherever you want and set the export target for the version of X-Plane you want to fly in, and WED figures out the best way to export your work, chopping up polygons as needed.

Shedding Light on Things

Our current work on the Blender exporter focuses on WYSIWYG animation and the material model (which includes instancing support). There is one more area that I think will make a huge difference to authors that we haven’t started working on yet: lighting.

The lighting values for commands like LIGHT_PARAM are, to put it mildly, completely goofy. If you’ve ever tried to read this, you know that even when things are explained, they make no sense.

No author should ever have to write a LIGHT_PARAM. This is a classic case where the exporter should do the transformation for us. What I would like to get into Blender 2.7 is the ability to place a cone light in Blender and have the exporter write the spill light automatically, e.g. filling in the parameters from the position and shape of the light. You light your scene and you see the same thing in X-Plane.

For years we’ve been behind the curve on user experience when it comes to 3-d modeling. I think we’re starting to fix some of those problems in a major way, which should help make the power of X-Plane 10’s rendering engine quite a bit more accessible.

 

*As a side note, I have no idea what the current 2.49 script status is. We view the 2.7 scripts as the long term way forward, due to Blender 2.49’s not being maintained, so our effort on a truly user-friendly tool is going into 2.7. But we also maintain updates to 2.49 to support our internal team. X-Plane 10.50 will feature a few new OBJ commands, so I will try to post our branch of the 2.49 scripts (which do have this new support) for anyone whose aircraft is in 2.49.

In the long term the 2.7 scripts will feature a migration tool, but I expect we will have a useful 2.7 release (for 2.7 authors) without migration first, and then migration will come later.

Posted in Development, Modeling, Tools by | 30 Comments

Scenery Tools Update

Update: the comments form is fixed – sorry about that!

I’ve been working a bit on scenery tools this week. Here’s a road map of some of what’s coming up.

WED and the X-Plane Scenery Gateway

We are working on a release of the latest scenery gateway airports for X-Plane 10.45. Like all releases, we’ve found problems with airports that WED does not detect. So I will try to release a new version of WED with stricter validation soon.

Airport Parking Spots

Austin is working on new code to place static aircraft at unused ramp parking spots. I’l describe this in more detail in another post, but here are some key points:

  • As an airport author, you simply place ramp starts, not static aircraft.
  • X-Plane will feature some new static aircraft in the update.
  • Third parties can further add to the static aircraft and have them be used via the library system.
  • There will be a revision to the apt.dat format that stores more data per ramp spot and a WED update to support this new data.
  • Since there may be scenery now with static aircraft on top of ramp starts*, we will auto-remove static aircraft from ramp starts in the gateway export as a temporary measure until authors can resolve the situation and post the fixes to the gateway.
  • We are not removing the old static aircraft from the library, but we will be hiding them from WED’s library so that authors use ramp starts instead of placing them by hand.

Static aircraft in parking spots will ship next year, not this year, but is planned as a v10 update.

Scenery Tools on OS X

I made significant progress this week porting WED for OS X to 64-bit and modern Mac APIs. This effort basically meant rewriting the OS-level UI code in WED to be new AppKit based calls instead of the old Carbon API.

What this means is that developers will be able to build all of the scenery tools on a modern Mac running Yosemite or El Capitan with X-Code 7. Since Chris has already updated the projects to compiled WED in MSVC, a developer can build all of the common scenery tools from the most widely used open source IDEs.

It also brings us closer to a 64-bit WED on Mac, which is desperately needed since OS X provides the least amount of usable address space to 32-bit apps. I am not sure of the 64-bit status of the Windows WED build; it may need more work on the libraries.

Will It Blend

This might be the most significant scenery tools development: we’ve been working with Ondrej (der-On on Git-Hub) on new features for the Blender 2.7 exporter. This will include:

  • Much better animation support. Complete WYSIWYG animation of data blocks and bones, with no work-arounds needed. Bone animations match Blender 2.49 so you can move your projects forward.
  • Modernized material support including instancing for a really clean work-flow that takes advantage of X-Plane 10’s features.
  • Updates for the latest OBJ features.
  • Our plan is to create a migration script to bring 2.49 projects into 2.7, converting the special tags used for an X-Plane 2.49 export into 2.7.

Many of our artists still use 2.49, but there’s no question that 2.49’s days are numbered. It’s a question of when it stops working on OS X, not if. With the migration step, we can move our projects forward without artists having to redo work.

Like the previous 2.7 exporter, this one is open source and will be free, and should be able to export existing 2.7-type projects with no modifications – we are trying to maintain full backward compatibility.

 

* This situation is slightly silly – if the user picks the ramp start to fly, the user will be on top of a static aircraft.

Posted in Modeling, Scenery, Tools by | 22 Comments

Scheduled Maintenance and Free Passes

This may have happened to you: you import the latest approved airport scenery pack from the X-Plane Airport Gateway and modify it. When you go to export your improvements back to the gateway, WED says your work is invalid and has a problem — but not in the changes you made!

Huh? If this was the approved airport on the Gateway, how can it also be invalid in WED? How did that other author upload the airport before?

You Get a Free Pass

What do we do if we find that scenery and airports have bad data that is causing bugs? What if we can’t just fix the scenery pack for you? The policy we’ve been using is: “you get a free pass until your next scheduled maintenance; then you have to fix the bug.”

The X-Plane Scenery Gateway is a good example of this. Sometimes we find new categories of authoring mistakes – I write improvements to WED’s scenery validation function to catch these authoring errors. These are errors like:

  • Typos in taxiway signs – the old syntax makes a sign with incorrect letters.
  • Overlapping ATC routes – with the overlapping routes the AI aircraft do not taxi correctly.

In other words, these are situations where the WED scenery pack, if left alone, is causing clearly bad things to happen inside X-Plane. This isn’t a case of “old style or new style”, it’s a case of “broken or fixed.”

So what we do is we set WED to reject these scenery packs on future uploads, but we do not delete the airports from X-Plane itself. In other words, the airports with serious mistakes get a free pass until the next time an author does scheduled maintenance.

Then when an author is working in WED and is going to update the airport anyway, the author must fix the problems we have found.

No One Likes a Fire Drill

This strategy is a compromise between two extremes:

  • If we force you to fix your airport right now (e.g. “fix the airport or we remove it from X-Plane”), that’s not fun, that’s a fire drill. And having been exposed to plenty of fire drills myself dealing with the new OS X 10.11 and Windows 10 releases, I’m sympathetic to authors not wanting to have to stop everything to deal with an issue ASAP.
  • If we never require that these kinds of problems be fixed, they simply won’t be fixed. The overwhelming evidence that I have seen from working on authoring tools for X-Plane for over a decade is that some authors won’t fix problems unless the tools force them to do it.**

So requiring the change when the author does maintenance but not requiring the existing shipped scenery pack be modified strikes me as the best possible compromise: we still get quick responses to serious bugs, but we don’t force anyone to drop everything.

 

* I do realize that some authors are incredibly diligent about getting their scenery to be correct even if the tools don’t force them to do so! But the goal here is to have all scenery be correct; it only takes one broken scenery pack in the hundreds a user installs to crash X-Plane.

Posted in File Formats, Scenery, Tools by | 3 Comments

X-Plane 10.40: The New DSF Loader

I’ve been meaning for weeks now to write up some notes about 10.40. I’ve also been trying to put the hood back on 10.40 so we can get it to public beta; instead the last few weeks have been a flying circus of driver bugs, expiring certificates, etc.

But we are making progress, so I’ll start off in this post by describing a change in how DSFs are loaded and the wide DSF box.

A Tangent: Stuttering and Pauses

A post that involves X-Plane pausing during flight is going to invariably bring up a bunch of blog comments: “X-Plane stutters while I fly on my really expensive machine!” This is not good, and from what I’ve been hearing, it sounds like something got worse recently; this is something that we are investigating now.

But I also want to mention that historically, X-Plane has never been “no-pause”. What we have done is periodically made the pauses much shorter; our goal is to get down to zero pauses, ever, but this will happen by finding every source of pausing and fixing them one at a time. In other words, we’re in the middle of a process of improving smoothness, but even if one source of pausing is fixed, another source might still be causing problems.

DSF Loading: the Old Way

X-Plane has, since X-Plane 9, loaded DSFs in the background on a second core while you fly. This cuts down the amount of time it takes to change scenery. (Older versions of X-Plane would pause while scenery was loaded and shifted.)

The old DSF loader did have a few weaknesses though:

  • The sim pauses while DSFs are deleted. As DSFs become bigger, this pause is becoming more noticeable.
  • If the loader ever gets behind by two scenery shifts, it just waits until it catches up. This is where the dreaded “Async load timed out after 30 seconds” comes in – it indicates that the DSF loader was so far behind that it locked up for half a minute and didn’t catch up.
  • The old DSF loader loads one DSF at a time, tops.

DSF Loading the New Way

X-Plane 10.40 has the new DSF loader, which both loads and unloads DSFs on worker threads to keep flight smoother. It also will load more than one DSF at a time, limited only by the requirement that adjacent DSFs not be loaded at the same time.

X-Plane 10.40 also has the option of an extended DSF scenery region for sharper terrain; with this option off, two DSFs are loaded at one time during sim boot and one or two are loaded at a time while you fly. With the extended DSF region on, up to four DSFs are loaded at once during sim boot and one or two are loaded while you fly.

The extended DSF scenery region is optional; don’t use this if you’re using HD meshes or you’re short on RAM. The new DSF loader is always on.

Posted in File Formats, News, Scenery by | 60 Comments