I fear one of the main points of my last post was perhaps lost in the excited discussion of how to cope with conflicting crowd-sourced airport submissions. The engineering principle is:
Fix real bugs, not hypothetical bugs. Solve real problems, not hypothetical problems.
By this I mean: it’s invaluable to wait until you actually have evidence of a problem before you design the solution. Without a real data point showing what the bug is, the solution might not actually fit the problem as it actually happens. In a program as complex as X-Plane, particularly when third party content is thrown in, it’s hard to predict in advance what is going to go wrong.
In the previous case (coping with conflicts) the key here is that I want to see a few conflicts before I choose a plan. I see proposals for a revision system, public logs, a registery, a voting system…any one of these ideas might be the right solution, but we don’t know yet because we don’t have two people submitting buildings.
The same principle applies to another issue that was brought up a while ago:
Won’t all of the default airports look the same since they all use the same library?
First, to be clear: we’re not trying to make the default library airports look like the real world airports, we’re not trying to duplicate custom scenery, and if people submit buildings for all 20,000 airports in X-Plane there’s no way we’re going to have 20,000 control tower libraries in the library.
But there is a valid issue: what if people heavily use the assets? Won’t we start to get a “sameness” either between two airports or within an airport as we use the library more and more?
This brings up the second half of “fix real bugs”:
Identify possible solutions, but don’t use them until necessary.
I am not worried about collisions of airport because there are a huge number of possible solutions, many of which are easy to implement and have been proven to work well in the real world in other problems. In other words, the problem of merging crowd-sourced data is a solved problem, and we can use the right existing solution when we have enough data to pick that solution.
The same thing applies to repetition in the library: we have ways of fixing repetition. I want to develop a few airports and see what looks repetitive before asking our artists to spend their time making more stuff* but we at least know how we can fix the problem when the time comes.
Options to Deal With Repetition
So what are the safety valves here? How can we deal with repetition in the library? There are a few ways:
- Every object in the library can have multiple real OBJ models for each “virtual” object. X-Plane picks randomly among the choices, so we can always add more real objects. We already do this for cars. (There are something like a dozen cars and trucks backing up one or two library entities.)
- Some of the library elements are .agp (autogen-point) files – these files are mini-scenes made up of other objects, and the objects referred to within the scene are also subject to library variation. So we can have a mini-scene where some of the decorations are constantly changing.
- The facade engine contains some fairly flexible rules for picking which segments of walls go where, so there’s a lot of potential to break up the order of facade walls.
- We can add more types of facade walls to a given facade without hurting X-Plane’s performance.
All of these changes can be put into an X-Plane update.
In summary, there are a lot of ways we can make the library more diverse and less repetitive, and as we see the library get used (and certain elements become over-used) we can add more variation to keep things looking fresh.
* For example, it wouldn’t make sense for me to ask Tom to make 50 variants of the administration building if almost no one places it in any airport. We need to see what kinds of assets get used the most and then make decisions.
WED 1.2 is in beta now — thanks to the users who have taken the time to submit WED bugs (which go here, btw). Tyler is working on an update to the manual, and I’ll try to find some time to squash bugs.
But how do you share an airport with terminals with the rest of the X-Plane community? At the X-Plane developer’s conference, we reiterated that we want to extend our crowd-sourced airport data collection to include terminal buildings and ATC routes, as well as the pavement layouts that Robin already curates for us.
To be clear: you do not have to share your WED creation to use WED! WED can be used for custom scenery and your own private creations; submitting your work for sharing with the community is optional!
Here is our thinking on submitting airports:
- Submitted airports will in the future contain three types of data: taxiway layouts (in an apt.dat), ATC taxi route data (also in the apt.dat) and library airport building placements.
- We will not accept custom art assets! The public database of airports is meant to be location data only! Custom scenery is great, but the purpose of the public database is to populate the world, not to crowd-source the art-creation process.
- WED will have a “package for submission” export option that properly bundles everything into a single file in a format useful for Robin.
- The package for submission option will do some basic validation, to catch problems like custom objects in the library, or possibly erroneous data.
You will also still be able to say “Build Scenery Pack” to create a custom scenery pack for your own use – WED 1.2 is meant to support both the crowd-sourcing and custom scenery work-flows.
The “package for submission” work is still in progress; Robin and I will test this out privately while WED is in beta so that test submissions don’t flood the data collection process.
This is why.
Let me anticipate the inevitable “why don’t you just use Google Maps” question: to the best of my understanding, tracing your airport vectors off of Google Maps imagery is an unlicensed use of their service and would result in a derived product owned by Google. That’s pretty much a deal breaker.
OSM’s use of background imagery to trace (in Potlatch) comes from specific license agreements between OSM and the image provider – in other words, the image providers are “donating” to OSM by giving them a specific license to trace without ownership issues.
Anyway, I am open to other ideas. (Except for “no, you didn’t read the license right, Google really doesn’t mind if you use their imagery for free to create your own data set”. I am not open to that! 🙂
I have updated the WorldEditor page to link to the new builds of WED 1.2 beta 1. This is the first beta of WED 1.2, which we showed at the X-Plane developer’s conference in SC a week ago. A few hilites:
- Finally, a real airport import dialog box – when you import a dialog box, WED will let you pick a subset of airports to grab. So now you can open the entire apt.dat file from Robin and just pick the airport you want.
- There is now a library preview pane for objects, autogen scenes (.AGP files) and draped polygons.
- WED 1.2 supports all of the new v10 overlay editing extensions, e.g. forests as tree lines, facade wall-type control, etc.
Tyler and I are working on a 1.2 manual but it’s still chaotic enough that I think it isn’t even ready as a beta document.
Last weekend I was in Columbia, South Carolina for the second X-Plane developer’s conference – that is, the US meeting. I’ll try to write that up later, but first a quick note on WED 1.2. We had Tom Kyler on site, and his demonstration of WED 1.2 with the new lego brick objects turned out to be the surprise hit of the weekend. It’s one thing to say “we’re going to crowd-source airports” – it’s another to show the pieces in action.
WorldEditor 1.1 is released, and WED 1.2 is available as a “developer preview” – it’s that developer preview we showed at the conference. A real “beta 1” will be out shortly.
What’s a developer preview? It’s an incomplete beta. In this case we didn’t have all of the features of 1.2 in place, but I wanted to get it out there for people to poke at. Tom set up Seattle using the developer preview, so clearly it’s “usable” – albeit with a heavy dose of caution.
My plan is to get WED 1.2 beta 1 released some time this week, with every planned feature except for “submit-to-Robin”, which he and I should probably test internally a bit while we make sure that WED’s output is solid.
Note that if you are making custom scenery, there’s no reason why you can’t use WED 1.1 for now – it’s finished and stable. WED 1.2 has usability and v10 updates, but overlay editing has been available in WED 1.1 for a while.
After WED, the next scenery tools priorities will be:
- Getting ac3d and Blender 2.49 caught up for version 10 technology. For Blender 2.49, I am trying to merge my
hackingchanges with Jonathan’s, so that users of the public scripts can use my v10 mods without having to rework content. I will send these to Jonathan, and also post some of our newer scripts (e.g. autogen-editing).
- MeshTool 2.x will write v9 meshes, but a new 3.0 version will be needed to make “v10”-style DSFs; this is on my todo list.
What About That Program That Simulates Airplanes?
It’ll take a bit to get through the scenery tools because we are also mid-patch for X-Plane. 10.10 will be the next “patch” with new features. It looks like we will post a 10.06 first with some new translation files that have come back to us from a number of sources.
Here are three obscure Plane-Maker/X-Plane features that can save you time if you developer complex aircraft.
Plane-Maker Will Copy Your Instruments
You may know that in Plane-Maker, you make your own copies of X-Plane’s PNG files to customize the look of the instruments. But did you know that Plane-Maker will copy the images for you?
Select an instrument and type Control-P (which is the default for the command binding “pln/panel/copy_instrument_png”). Plane-Maker will copy all PNGs for that instrument into your aircraft’s cockpit or cockpit_3d folder. This can save you the time spent wading through X-Plane’s cockpit folder to find the right PNG files.
X-Plane Can Make a Panel Image for UV-Mapping
When you are making a 3-d cockpit, you use the 3-d panel as a texture. But how do you know how to UV-map this texture in your cockpit? Often the panel background (panel.png) is blank.
X-Plane can make a snapshot of your panel for you, in the exact size you need to UV map. Use Control-Alt-Shift-Space (Command-Option-Shift-Space for Mac nerds) to run the “sim/operation/make_panel_previews” command in X-Plane. It will make a PNG file in your aircraft called Panel_preview.png – it’s just like Panel.png but with the instruments drawn in – perfect for UV mapping.
Plane-Maker Will Tell You What’s Wrong
That sad face icon in top bar of the Plane-Maker panel editor enables “warning” mode. In warning mode, every instrument that has a potential problem gets a red outline. Select one instrument with a red outline and in the upper left corner of the panel you’ll see a description of what’s wrong.
This picture on the left is from Plane-Maker editing a 3-d panel. (That’s why it is just a “packed” set of instruments with no background; this panel is used as a texture for a 3-d cockpit – each instrument is UV-mapped into the right location.)
The air-speed indicator has been flagged as having a problem, and the text shows it. In this case, the lit texture has an alpha channel, which causes the lit texture to draw incorrectly. Fix the texture and the warning will go away.
I strongly recommend checking all Plane-Maker “red boxes” on your plane – most of the warnings will tell you about problems that would otherwise be very hard to detect.
I realized the other day that while Chris and I have discussed cities with a bunch of people, I left out the city plan from my series of “road map” blog posts a while back.
The picture on the right is downtown Seattle from the upcoming X-Plane 10.04 beta 7, which includes an update to the urban art assets: new facades, a bunch of lit textures, and careful library tuning by Propsman. (There’s also some clever use of spill lights next to the tall buildings.) This update is part of ongoing work to build out our new city autogen; this post will explain the road map for that work.
A Radical Approach to Cities
Before I describe some of what’s failing in the current city scheme, how it’s supposed to work, and what we’re doing to fix things, let me take a few paragraphs to describe the “big idea”. Cities in X-Plane 10 are completely different from X-Plane 9, and they’re completely different from any other flight simulator that I’m aware of.
Before X-Plane 10 there were three approaches to cities that I am aware of:
- Land class tiles. You create repeatable square orthophoto textures of cities and put matching 3-d on top of them. What’s good about this technique is that it runs on really minimal hardware, it looks good without using a lot of resources, and it’s fairly easy to code. The down side is that you will not get accurate cities. There is no way to use additional data to put roads or buildings in their actual correct places. You will always get a plausible but non-accurate city, a “city in theory”. This is the technique X-Plane 6 and 7 used, and the technique FSX uses.
- Vectors over land class tiles. You create repeatable square textures of city (just like above), but instead of attaching the 3-d to them, you build your 3-d off of real vector data. This is what X-Plane 8 and 9 did. What’s good about this technique is that it runs on modest hardware, and it puts your roads and buildings exactly where they should be. The down side is that the 3-d is completely misaligned with the textures underneath, creating a “stew” effect when viewed up close. (Some users consider this mismatch absolutely unacceptable; others seem to not care.) There’s no question that the texture mismatch is not plausible, but the shape of the city is accurate. This is the technique X-Plane 8 and 9 use.
- Fully custom cities. If you can afford custom non-repeating orthophotos of the real city and you have the matching 3-d data, you simply build the city and everything matches up. This is the ideal way to build a city, but it assumes custom data for every city; this is great for custom scenery but not scalable to a global product. It is plausible and accurate but not global.
For X-Plane 10 we wanted a way to have it all: a city that was plausible (e.g. looks good and doesn’t have weird artifacts) but also accurate (e.g. shaped like the real city, using data that’s now available – we can know where every road is, and even some of the buildings) and for the default sim, our approach has to be global; we can’t simply build a custom city for every city on the planet. And when we looked at the technology out there, it looked like it was for the first time possible.
Here’s what we came up with: in X-Plane 10, the unit of autogen is not the landclass orthophoto tile (a 1 km x 1 km square of terrain whose entire interior is defined by the texture and 3-d). The landclass tile has wonderful properties, but it’s just too big to be accurate for a real city.
Instead X-Plane 10’s unit of autogen is the city block. Each city block is an individual autogen unit, and the autogen is capable of flexing and shaping itself to fit the demands of a real road grid based on real data. This means we can have plausible autogen with tons of detail while sitting inside the real road grid of a real city.
This approach is significantly harder than using landclass tiles. Each autogen primitive needs to be able to resize and contort itself to meet the demands of real world data while still looking like it was meant to be there. The road grid has to look really good even as it is built from vector data, because we can’t just bake the pictures of the roads into the terrain. The terrain has to contain no city details in the near view (because the autogen defines where there is city), and the autogen buildings have to have “skirts” of orthophoto that they drop down to put their driveways and other ground details in place.
Then the rendering engine has to somehow take all of these disparate parts and render them at high speed!
So Does It Work?
When the new system works, it really does work and we get plausible and accurate cities with good performance on a global scale. But this system is entirely new – because it is such a radical change ,we couldn’t recycle any art assets or code from previous versions of X-Plane and thus it is very new, and frankly a bit raw. So when it fails, the artifacts are sometimes quite spectacular.
As the system ships now, a few things tend to go wrong; I will describe what’s going on under the hood and what we expect to do to fix it.
Crazy, Deformed Roads. This is the most common reported bug. (Hint: please stop reporting this. I know about it already.) We feed extremely detailed road data from OpenStreetMap into X-Plane; unfortunately due to bugs in the code, sometimes when X-Plane analyzes an overpass, it can’t figure out how the roads should stack up, and it instead creates some kind of ridiculous bridge, e.g. a road that shoots straight up into the sky or a giant 1000-foot tall arch.
The good news is: this is just a bug in the rendering engine; when I fix the bug (which I hope to do for the 10.05 patch), the artifact should go away without the need to redo any DSFs.
I brought this bug on myself by insisting that the roads be draped on the terrain. X-Plane 9 placed roads at absolute altitudes in 3-d space, which was easy for X-Plane, but meant that roads weren’t easily used in an overlay. For X-Plane 10 I wanted to make it easier to work with roads directly; once the road bridging bugs are fixed, this should be doable.
The road system is already doing a number of things we are pretty happy about though:
- Roads are made of bezier curves. Once we go 64 bit we’ll be able to crank up the smoothness factor for users who have a lot of RAM.
- The road junctions are flexible – that’s how we get clean, real, custom-looking junctions out of raw vector data. Over time we can add more junction definitions to make nicer looking overpasses.
- City roads drape on the ground, avoiding Z thrashing and some of the other ugly version 9 artifacts.
Where Are the Buildings? The second bug report we get is that cities don’t contain enough big buildings, or enough sky scrapers. Most commonly the problem is residential houses appearing downtown.
The problem here is that the new autogen requires new art assets; we couldn’t just reuse our substantial pile of buildings from X-Plane 9. So we’re building more buildings; what you see in the meantime is the existing art assets shoved into the wrong kind of library slots. Once we have more appropriate buildings for a given library slot, things should look better.
We have also discovered some cases where the placement of autogen in the DSF isn’t very good. This was the first global render where we built the new autogen and there are some clear cases where we can do better.
So when it comes to building placement, things should get better by putting more art assets in a net update; we’ll probably recut some major city DSFs to get even better use of that autogen. (One nice thing about cities: there aren’t that many of them. There are over 18,000 DSFs in the global scenery, but the top 100 cities world-wide..well, that’s probably about 100 DSFs.)
There are two aspects of the buildings that are already working right now:
- There is a ton of detail. If you get in close to the autogen, you’ll see a scene similar to what you might get in a custom scenery pack.
- It’s fast. You can max out the road and building density on today’s hardware and get 25-30 fps. Compare this to X-Plane 9 where even four years after the product shipped, “insane” objects is often not reachable.
I put Seattle from v9 on an i5 machine for testing and compared it to v10 with insane roads, insane autogen, moderate forests, and shaders. I got > 25 fps in v10 vs. approximately 5 fps in v9. We’re going to try to keep that kind of capacity for huge amounts of autogen with high performance as we add more building types.
One last note about autogen and buildings: in X-Plane 10 the autogen can optionally be “height sensitive” – in this case the buildings in the autogen block take on an AGL height based on the DFS. We only use this for appropriate autogen types. For example, a dense urban city block might be height sensitive (since the DFSs contain heights for skyscrapers) but a block of residential houses is not height sensitive – they’re always just little houses.
In most cases the height data is already in the DSF; once we get the right autogen art assets in place, they should start repsonding to the height data.
Green Terrain. The third bug report I hear about is green terrain – that is, a city is filled with nothing but a big green expanse. To understand this bug you have to understand how our autogen creates the look of the terrain.
Austin posted in one of his early
rantsmanifestos that we would build the terrain from the ground up, with green grass for the ground and no cities if 3-d is not enabled. This is a massive simplification of what actually actually happens.
The terrain is a composite: the actual base terrain is indeed devoid of any recognizable city details – because they will come from overlays. The ground instead contains “natural turf” patterns that try to impart some detail while accepting overlays.
The road grid then puts down a layer on top of the ground, with sidewalks and the actual pavement.
On top of that, the autogen system puts down a ground tile for each building. (One autogen block consists of multiple tiles, and the tiles are moved around to fit the block shape.) The tiles are mostly transparent, but contain details that mask parts of the ground, such as sidewalks and driveways.
Thus the combination of three layers (ground, road and autogen ground tiles) composite together to form the kind of detail that you would normally get in an orthophoto.
So when we have green terrain, there are actually several problems conspiring to ruin this “composite orthophoto” effect:
- If we don’t have the right autogen buildings (see above) we won’t have the right ground tiles. More buildings will help fix this.
- It’s actually a bit tricky to build the ground terrain that can sit under the autogen; the only really complete climate we have supported right now is a northern climate (which works well in Seattle). So as we add more urban base terrains the bottom layer will start to work better.
- When you turn down autogen density, some autogen city blocks disappear. This is a hold-over from version 9. What we want to do is keep the base tiles but remove the 3-d when you turn the rendering settings down. This will lighten the rendering road, but the 3-layer orthophotos will still look good. (Similarly, we’d like to reduce detail but keep all roads for lower road settings.)
In other words, once we have more buildings, more ground terrain, and a better way to turn down detail, the terrain should look more like a real city under a wide range of rendering settings.
(If you’ve really watched the sim carefully, you may have noticed that the ground textures actually crossfade from “just grass” to orthophoto-like as you get farther away from them.
Everything listed above represents incremental improvement of our cities – more art assets, bug fixes, smarter LOD and rendering settings. But that’s just the beginning. I believe that what we have on our hands is a fairly big fundamental improvement in how cities are handled: we have a way to build cities that provides a huge amount of up-close detail while working from detailed data (and representing that data), and it can run at good fps on today’s hardware.
Down the road I think we’ll be able to integrate and utilize OSM data to get even more detail into our cities and further push the envelope for how much detail, realism and accuracy we can get into cities at a global scale.
Announcements on the plethora of betas that went up this weekend.
Edit: beta 5 is temporarily offline while we resolve a missing scenery texture. So you’ll get beta 4 if you update.
X-Plane 10.04 beta 5 is now available. Release notes here. I think we will be winding down the 10.04 patch in the next week and going on to 10.05. There are a number of cosmetic bugs that I hope to get fixed shortly; it looks like they’ll have to go into 10.05 and not 10.04.
What’s the logic behind this patching? Basically we’re going to keep doing small patches until we get a certain set of bugs fixed – bugs that we think are important and that we can fix without tearing the sim to bits. The patch runs sometimes have to be closed off to burn new DVD masters* (this is the case with 10.04). In all cases, if you don’t want to deal with beta chaos, you can ignore the betas and get a new patch every few weeks with a little more performance and a few more bugs fixed.
I spent some time yesterday retuning cloud performance and the effect of the cloud rendering setting. Besides trying to find a better performance-quality trade-off point, the old slider had way too much range on it. It defaulted to 50%, but that 50% point really represented “a good setting for a high end gaming desktop”; going past that would put you into the range of “this isn’t ever a good idea, but the engine can do this without crashing”. The side effect was that running at 25% clouds (a very reasonable setting) would feel demoralizing because the setting scale was so vast.
The new scale marks a “100%” point at about where I would put it for high quality if you’re just sitting on GPU power (e.g. small screen, nice GPU). You might still want to turn it down for performance reasons. The range from 100%-150% is highly experimental land…if you want to turn it up to eleven, we’re not going to stop you.
WorldEditor Developer Preview
There is now a binary build of WED 1.2 (download here). This is a developer preview: WED is still in environment and doesn’t meet the criteria for beta software. Basically: do feel free to poke around with it, but don’t use it on your real work. Don’t edit your scenery packs with it – make a copy first. WED 1.2 has not been tested enough to use for production yet!
The main feature of interest is ATC taxiway editing: WED 1.2 contains complete support for taxiway networks, as well as “flow” information to control the direction of operations. A number of weird ATC bugs actually stem from the automatically generated taxi layouts that X-Plane produces when the apt.dat layout contains no taxi information. By providing a custom layout, you can get the AI planes to behave quite a bit better.
* We periodically re-burn the disc 1 master to put the latest sim on it whenever we are preparing new DVD inventory. If you have an old DVD, please do not panic! Once the updater is run, the sim looks the same no matter which DVD you have.
Chris and I bashed at the developer site (that is, this site today); sorry about any dead URLs – things should be more stable now.
I have (finally) posted a new build of XPTools; there are two items of note:
- The new build of DSFTool, which can read and write X-Plane 10’s DSF raster data layers.
- The new DDSTool can write gamma 2.2 DDS files for X-Plane 10. This should provide better color fidelity for content that targets only X-Plane 10.
I had a chance to catch up with Robin the other day and discuss airports with him. Here’s the basic road map for airports in X-Plane 10.
In version 9, an airport consisted only of taxiway layout data. Robin collected and managed a big database of contributed airport taxiway layouts, which are available under GPL and ship with the sim. WED 1.0 lets you edit these layouts.
In X-plane 10 we now have three categories of airport data:
- The taxiway layout – this lives in the apt.dat file.
- The ATC taxi layout and flow information. This also lives in the apt.dat.*
- Lego brick building placements. This lives in an overlay DSF.
Our plan for the version 10 run is to collect all of this data together and redistribute it all, just like we do the current apt.dat file.
We also plan to build a few airport building layouts ourselves, using the existing lego bricks, without custom elements. This will help us further debug the bricks, get users some more airports quickly, and help us understand the authoring process. We have some of Tom’s time earmarked for this.
WED 1.2 will support taxi layouts and airport building placement. Based on my talk with Robin, I believe I will also need to build a more specialized “Send to Robin” export function that pre-checks and packs the submission to streamline the process; since an exported airport will include DSFs and apt.dat files (and should not contain custom OBJs) having the packing be automatic will save everyone a lot of time.
What about autogen buildings? I don’t know. We wanted to try this before we shipped and ran out of time. I think we could autogen buildings for small airports that just have buildings next to the autogen taxiways, but for custom layouts I don’t know that autogen will ever be good enough to make humans happy.
Over the next few weeks I’m hoping to have more time to roll out tools and documentation to move this process forward. The library is already complete, so the quicker we can get everyone using it, the better.
* If you have an airport layout wihtout taxi layout and flow information, X-Plane automatically generates default flows and layouts.