X-Plane 10.35 is now officially released! If you run X-Plane, it will prompt you to update. Lots of details here!
I’m still trying to get WED 1.4 to public beta – that’s high priority right now!
X-Plane 10.40 is currently in development. Austin has some new features that he’s letting people test-drive; I’ll post more about 10.40 in a few weeks.
Note: Oops – I wrote this a week ago and forgot to post it. 10.35 will probably “go final” shortly.
If you haven’t tried X-Plane 10.35, please do. Check “get betas” in the updater to get it. Just one bug fix and a license file in the release candidate. A Steam RC should come soon.
Chris took a sledgehammer to the Windows build materials for the scenery tools and pretty much rebuilt everything for Windows; I’ll have betas in a day or two.
UPDATE: The RC is available on Steam also. To get it, right-click on X-Plane in the Steam library, click the “BETA” tab, and then select the beta to opt-in to.
The X-Plane scenery tools are open source; their bug database is open too, so that anyone working on the project can see the status of all bugs.
We recently merged the scenery tools bug base into the gateway bug base; in the process I audited about 70 open WED bugs, and finally closed all of the bugs where the original filer did not provide adequate materials to reproduce the bug. Typically these bugs had been sitting, waiting for user feedback for at least a year.
I’m looking at the feature request list next; it is also about 70 items long and needs some auditing. The old bug base had about a million “levels” of bug status, so it was easy to leave a bug in some partial state and ignore it forever. The new bug base does not, so I think I need to either set the feature request as something we want or kill it.
One problem: a lot of the WED feature requests aren’t feature requests at all; they are requests for changes in the underlying scenery system. E.g. if you say:
Allow WED to edit the height of the underlying terrain of the global scenery.
The only possible response is “unable”; WED cannot do this because the DSF file format cannot include this information; if you had some way to edit such data in WED, there would be no place to save it in the scenery pack that is built.
Really the request needs to be two-part:
- Allow overlay DSFs to change the underlying mesh heights of the global scenery underneath them.
- Provide an interface and exporter in WED to export the new “mesh modifier” added to the scenery in point (1).
The first change is a change to X-Plane and its file formats; the second one is to the scenery tools.
I don’t blame authors for filing the bugs against WED – as far as authors are concerned, WED is their interface to the scenery system; they want to see something new in WED, and if I do (1) and not (2), the job is not done.
But…the bug base is also my todo list, and having an item on the wrong todo list is a good way for it to get “lost”. If I look at mid-term feature requests for the scenery system in X-Plane, I will not find anything in WED.
I haven’t figured out what I am going to do with this, but one possibility is to simply move all valid feature requests on the underlying scenery system out of the WED bug base and into the X-Plane one. This will have the side effect of taking them private, but if the feature request is a legitimate one, I don’t think the original poster will need to provide more information.
For about a year there has been a subtle bug in how X-Plane draws taxiways: if you build an S-curve shape out of a single bezier and it is almost perfectly symmetrical, X-Plane would go “nah, why bother” and replace it with a single straight line segment.*
So in 10.30 I fixed it, and the result was a bunch of broken airports with missing pavement. (YMML is high on this list!)
It turns out that these airports have authoring errors – typically a segment of pavement that should be straight instead being formed by two overlapping beziers. This is definitely wrong, but due to the bug, X-Plane would simplify the overlapping S curve into a single straight segment and the layout would work. Only now that X-Plane correctly renders the S curve does the taxiway fail (because taxiways may not have overlaps). So two wrongs may not make a right, but they do make a “hey, that looks okay, let’s ship it”.
Note that the overlaps depend on the rendering setting of X-Plane – a different S curve is formed at different rendering settings; the overlap that causes the taxiway to disappear may only appear at a particular rendering setting.
For 10.31 I am going to undo my bug fix. This doesn’t make me happy, but I think it is necessary:
- We have no idea how many airports have their taxiways broken by this bug.
- Authors have no easy way to detect this problem, other than re-testing every airport at every rendering setting.
- Even if an airport looks okay at all rendering settings, future rendering settings may cause the problem.
This is too much uncertainty to solve ‘by hand’. So my plan is:
- Undo the code change for 1031. YMML and friends comes back.
- Develop validation code in WED to detect this kind of authoring error.
- Ship that version of WED so new authoring work will be checked.
- Run the WED code on all airports and make a list of ones that need repair.
- Fix all of the known problems in the airport gateway.
- Redo the code change so X-Plane is correct.
This isn’t going to be a quick process, but then it can’t be, because third parties have apt.dats shipping now that only work when X-Plane has the buggy taxiway code in place. So we need to ship WED and then give third parties enough time to go back and check their layouts and fix them if necessary.
I expect to get a 10.31 beta with the taxiway code changed back this week.
For WED validation, I have some test code to detect errors but it isn’t ready yet. The problem is that it’s not good enough to detect errors with overlapping beziers**; we have to consider two bezier curves near enough to each other that with the error in rendering introduced by WED’s rendering settings, we get an overlap. (So authors, better be safe than sorry in creating your pavement.)
If there’s a moral to this story, I think it’s this: when we (LR) don’t provide good tools for authors to validate that their work is correct, the resulting body of work will end up with subtle errors.
* X-Plane renders beziers by measuring how far the mid-point of the curve is from the average of the ends. As long as this distance is ‘too far’ and the iteration count isn’t too high, X-Plane divides the bezier in half and repeats the process. In this way beziers are broken into enough line segments to approximate the bezier within a minimum error limit. The rendering settings control the error limit.
The bug: if the curve was a ‘balanced’ S curve the mid point of the curve was the average of the end points and X-Plane went “great, no error” and stopped dividing.
** Which is already not an easy problem – the analytical solution for bezier intersection is a 9th order polynomial!
Users with newer Ubuntu versions have reported they can’t get X-Plane to start after the update to 10.30, while it worked fine with 10.25.
Since 10.30, X-Plane links to libudev to discover devices like the Oculus Rift on Linux, and that has caused a few hiccups with some of your Linux installations out there.
No, this post is NOT about the Oculus Rift on Linux!! If you want to know the current state of Oculus Rift development, go and read this one. Though there’s a little update: At OC1, Oculus confirmed they still want to support Linux. They didn’t say when, though.
Back to libudev. X-Plane for Linux is built on a very old Linux distro, Ubuntu 10.04LTS server, which is horrendously outdated by now. But it has the advantage that binaries built on that an old version, will work with basically ANY distro out there today. Basically, the older the distro is we choose for building, the more distros users can run the binary on.
The problem with libudev0 though is, it is so old, that modern distros just don’t ship it anymore! You can only get the newer libudev1. As a work-around, you can simply sym-link libudev.so.0 to libudev.so.1 to make X-Plane find the newer version.
Starting with X-Plane 10.31, we will remove the load-time dependency on libudev again so everything is back to working like it was on 10.25.
In the future, we will load libudev dynamically based on the version the Oculus SDK requires (This is when an Oculus runtime is available for Linux, which currently isn’t).
- X-Plane 10.30: you need to create a symlink if it doesn’t work
- X-Plane 10.31: no need for a symlink because we won’t depend on libudev at all
- X-Plane 10.x: X-Plane will ONLY require libudev when you are using the Oculus Rift
As for the sym-link work-around, avid Linux user and plugin developer Bill “Sparker” has created a thread on X-Plane.org where the appropriate paths for the symlinks are posted for a variety of different Linux distros.
UPDATE: The method described here works just as well and has the benefit of limiting the change to one application only.
One of the major initiatives of X-Plane 10 was to add buildings to our airports – terminals, hangers, warehouses, control towers, etc. In X-plane 8 and 9 we had detailed taxiway line markings, lights, and signs, but most airports were devoid of structures. The plan we came up with went something like this:
- Create a library of “lego brick” airport buildings. The library provides visually high quality, high performance art-work with moderate variety and ships with the sim.
- Extend WorldEditor to edit DSF overlays as well as airports.
- Extend the global airport database that Robin Peel maintains to include buildings and not just apt.dat data, so that we can collect, share, and redistribute the buildings.
- Release airport building layouts (created by users, contributed to the database, using the lego brick buildings) as part of the sim.
In other words, take the process we have been doing all along for taxiways, and extend it to buildings. (Some would call this “crowdsourcing” the airports, but Robin has been collecting user-created airport layouts for us since long before such terms existed.)
So where are we on the project? As of now, it is more or less completed!
- X-Plane 10 shipped with the airport library.
- WED 1.2 provides the tools necessary to create airports using those lego bricks.
- We have a provisional process to collect those layouts.
- The first 250+ airports are now included in X-Plane 10.25 beta 1.
So what do we do next? We are working on a few separate initiatives.
- Documentation, Instructions, and Tutorials. We are working on videos, tutorials, etc. to help explain the process. Some of the docs are done now – for example, Marty and Nick just finished their 13-part WED tutorial, which is a great introduction.
- Automation and infrastructure. We’re looking into what we can do to provide a more automated submission process, real-time viewing of what has been submitted, etc. We haven’t made any decisions yet; we’re evaluating options.
- More library variety and WED features. We wanted to wait on library extensions until we could see how people were using the existing elements; it makes sense to add more variety in popular categories. WED 1.3 has additional usability features to make navigating the library easier.
How to Participate Now
We’ll have better documents on “how to participate” in the near future, but for now if you want to create your own airport and share it with the community, the basic process is:
- Get WorldEditor 1.2.0 (or 1.2.1 beta).
- Watch the tutorial on how to create an airport.
- Use “Export for Global Airports” to export the airport – you’ll get a zip file.
- Email that zip file to robin at x-plane.com for inclusion in the sim.
I just made the X-Plane 10.10 beta live only to realize that: some of the airplanes are missing. We are reorganizing where the fleet is stored, and the installer didn’t kit them properly. So:
- If you ran the update and are now missing some aircraft, you can update back to beta 9 to get them, or wait until tomorrow.
- A new beta will be posted tomorrow with the aircraft all present, just in their new locations.
- Beta 9 is “live” again until I can get things repacked – should be less than 18 hours.
Sorry about the confusion!
Edit: And…it looks like there’s something weird going on between our software that creates the patches and Code Signing (which we’re just starting to do for Mountain Lion users). So…it might take a little longer than expected to get this beta kitted.
Having such a small development team, it’s very difficult for us to test ALL conditions in which X-Plane is being used. Each of our offices are littered with joysticks, laptops, desktops, loose video cards, hard drives and various other hardware. We do our best to always be making steps in the right direction but it’s a losing battle to keep our testing internal. A prime example is the recent Joystick debacle. I rewrote the entire Joystick subsystem on 3 platforms and tested it on all 3 platforms on all joystick hardware I own, and Ben and Austin did the same. We worked out all known bugs and then shipped it…and immediately we found hardware that caused crashes 100% of the time. Once those were fixed, other devices were still broken…some devices didn’t even meet USB compliance and we had to work around that issue as well. I can’t possibly own every USB Joystick device on the market so how on earth do we get the product to be as stable as everyone wants it to be?
The answer is simple…we rely on ambitious and courageous users to take the plunge and help us test it. Being a Beta tester is a bit of a thankless job. Sure you get an early look at the new goodies but it can be very frustrating since often, you can’t enjoy them without crashes, artifacts and other annoying bugs. Most users are happy to provide their feedback via the bug form, and now with the new automatic crash reporting system, we’re finding and fixing crash bugs in record time. A prime example is the Joystick crash. Within minutes of releasing 10.10Beta 1, I was aware of the crash and had all of the information that I needed to solve it. And within 24 hours, we had a new build that took care of it. But that’s only useful for crashes, it doesn’t help us learn about visual artifacts or other problems. That’s where we need YOU.
After reading through the forums, it’s become clear to me that many people are just not exactly sure what it means to Beta test software so I’ve come up with my own list.
- Keep two copies of X-Plane. If you care to ever fly X-Plane for enjoyment, ALWAYS keep two copies of the simulator on your hardware. One on a stable release that you use for enjoyment, the other on the latest beta for test purposes only. Your sanity will thank you later.
- Don’t use the beta for recreation time. If you’re only in the mood to fly and have fun that night, stay away from the beta copy unless you don’t mind your flight getting cut short. Too many users expect to fly for 6 straight hours and become enraged when the sim crashes on short final. That’s Murphy’s law!
- Always stay on the LATEST beta. Users often say “The latest beta broke XYZ, how do i go back to the previous beta?”. The answer is you don’t and you shouldn’t. That defeats the whole purpose of beta testing! During a beta process, the code is often changing very rapidly. If we’re on Beta 6 and you’re still submitting bugs on Beta 2, you’re wasting your own time and you’re not helping the product. It’s important you keep up with development.
- Blame us, not your system. If it worked in Beta 3 but it’s broken in Beta 4, do NOT tear your machine to pieces trying to figure out the cause of the problem. Report it and let US figure it out. Time after time, I see users change their drivers, install service packs, uninstall and reinstall X-Plane, update their OS etc trying to solve the bug…when it’s probably OUR bug…not your computer’s.
- Beware of the placebo effect! _EVERYTHING_ affects frame rate. Time of day, clouds, location, aircraft, view angle, view direction, addons, plugins etc etc. Sometimes we’ll release a simple patch that does nothing but fix one bug that was unrelated to the rendering pipeline. I’ll poke my nose in the forums to get some feedback and see 10 users who all of a sudden claim to see some massive fps increase in performance and 10 users who see a massive decrease in performance. I role my eyes at the claims because it’s not a controlled experiment. For example, if you’re the type of person who flies with real world weather enabled, flying one day with clear skies may get you 60fps while the next day you may see 40fps with overcast. Heck, even departing from a different runway than the previous day can result in different frame rates. Running with the command line fps_test IS a controlled experiment. THAT’s the only way to be sure you’re seeing a change in performance. Don’t trust your instincts, gather data in controlled experiments.
- Always read the release notes (http://wiki.x-plane.com/beta) for each new version. They’ll tell you what bugs are supposed to be fixed as well as what new features have been added. If you see a bug that’s claimed to have been fixed yet it’s still happening to you, that’s when you write a new bug report. If we didn’t claim to fix it, it’s probably not fixed in that particular beta.
- We need a Log.txt! When submitting a bug, ALWAYS ATTACH the Log.txt from THAT sim run. Even if you don’t think the Log.txt is applicable, attach it anyway. Tell us in as much detail as you can what you were doing at the time.
- We already know about your crash. Because of the new automatic crash reporter, there’s no need to submit a manual bug report for crashes! Let the system do its thing. If you remembered some details after the fact that you want to tell us, then go ahead and file a report. Of course, if the auto-crash reporter didn’t load up for whatever reason, that’s a good time to file a manual report.
- Tell us who you are and what you know. We really appreciate users who enter their email and comments into the crash report form. It gives us a way to contact you to get more information. This was invaluable to me when attempting to fix the joystick issues earlier in 10.10. A handful of really helpful users made the research a breeze.
- Be objective, not emotional. No one hates bugs more than we do but writing a rant instead of a bug report may make you feel better, but it makes me grumpy. And when I get grumpy, I attend seminars…and when I attend seminars, I feel like a winner. And when I feel like a winner, I go to Vegas…and when i go to Vegas, I lose everything. And when I lose everything, I sell my hair to a wig shop….Don’t make me sell my hair to a wig shop.
- Stop staring at the fps counter! It seems that many users are obsessed with their fps counter. “It went up this beta. It went down this beta. It stayed the same this beta.” Yes, the framerate is ONE useful metric to determine the software’s performance but it’s not the ONLY one and often it’s not the right one. Turn it off and fly! Enjoy the simulator. Sometimes there’ll be a bug in one beta that increases fps as a side effect because the sim is no longer drawing what it’s supposed to. Suddenly, some users are thrilled to see a 20% boost in performance and perhaps they don’t notice that half of the usual streets aren’t being drawn any longer. Ben fixes the streets and in the next Beta, all of a sudden they lose their 20% performance gain and are outraged that we “broke things again.”
I have spent almost the entire last week looking at ATI performance on Windows in depth; this post summarizes some of my findings and what we are now working on. Everything on this post applies to ATI hardware on Windows; the ATI Mac driver is a completely separate chunk of code.
Forgive me for quoting an old post, but:
As with all driver-related problems, I must point out: I do not know if it’s our fault or ATI’s, and the party at fault may not be the party to fix it. Apps work around driver bugs and drivers work around illegal app behavior all the time…that’s just how it goes. Philosophically, the OpenGL spec doesn’t require any particular API calls to be fast, so you can’t really call a performance bug a bug at all. It’s ATI’s job to make the card really fast and my job to speculate which subset of OpenGL will cause the card to be its fastest.
This proved to be true – most of the ATI performance problems on Windows involve us leaning heavily on an API that isn’t as fast as we thought it was, but that’s not really a bug, it’s just the particular performance of one driver we run on. The solution is to use another driver path.
I’m going to try to keep this post non-technical; if you are an OpenGL nerd you can read more than you ever wanted to know here.
With 100,000 cloud puffs (this is a typical number for one broken layer, looking at an area of thicker clouds) we were seeing a total cost of about 9 ms to draw the clouds if we weren’t GPU bound, compared to about 2 ms on NVidia hardware and the same machine.
Is 7 ms a big delta? Well, that depends on context. For a game developer, 7 ms is a huge number. At 15 fps, saving 7 ms gets you to 16.7 fps, but at 30 fps it takes you up to 37 ms. That’s one of the crazy things about framerate – because it is the inverse of how long things take, you get bigger changes when the sim is running faster. For this reason I prefer to think in milliseconds. If we can get a frame out in 20 ms we’re doing really good; if it starts to take more than 50 ms, we’re in real trouble. You can think of 50 ms as a budget, and 7 ms is 15% of the budget – a number you can’t ignore.
The ATI guys pointed us to a better way to push the cloud data through to the card, and the results are better – about 3 ms for the same test case. That should make things a bit better for real use of the sim, and should get clouds out of the “oh sh-t” category.
Now there is one bit of fine print. Above I said “if we weren’t GPU bound”. I put the sim through some contortions to measure just the cost of the geometry of clouds, because that’s where ATI and NV cards were acting very differently. But for almost anyone, clouds eat a lot of fill rate. That fill rate cost is worse if you crank the rendering setting, run HDR, run HDR + 4xSSAA, have a huge monitor, or have a cheaper, lower compute-power card. So if you were CPU bound, this change will help, but if you don’t have enough GPU power, you’re just going to be blocked on something else.
(A good way to tell if you are fill rate bound: make the window bigger and smaller. If a smaller window is faster, it’s GPU fill rate; if they’re the same speed it’s your CPU or possibly the bus.)
At this point I expect to integrate the new cloud code for ATI Windows into the next major patch.
Performance Minus Clouds
I took some comprehensive measurements of framerate in CPU-bound conditions and found that with the “penalty” for the existing clouds subtracted out of the numbers, my machine was about 5% faster with NV hardware than ATI hardware. That may represent some overall difference in driver efficiency, or some other less important hardware path that needs tuning. But the main thing I would say is: 5% isn’t that much – we get bigger changes of performance in routine whole-sim optimization and they don’t affect all hardware in the same way. I have a number of todo items still on my performance list, so overall performance will need to be revisited in the future.
The other code path in the sim that’s specifically slower on ATI cards is the cars, and when I looked there, what I found was sloppy code on my part; that sloppy code affects the ATI/Windows case disproportionately, but the code is just slow on pretty much any hardware/OS combination. Propsman also pointed me at a number of boneheaded things going on with the cars, and I am working to fix them all for the next major patch.
So my advice for now is to keep the car settings low; it’s clear that they are very CPU expensive and it’s something I am working on.
One of the problems with poor CPU performance in a driver is that you never get to see what the actual hardware can do if the driver can’t “get out of the way” CPU-wise, and with clouds having a CPU penalty, it was impossible to see what the Radeon 7970 could really do compared to a GTX 580. Nothing else creates that much fill rate use on my single 1920 x 1200 monitor.*
I was able to synthesize a super-high fill-rate condition by enabling HDR, 4x SSAA, full screen, in the 747 internal view. This setup pushes an astonishing number of pixels (something that I am looking to optimize inside X-Plane). I set the 747 up at KSEA at night so that I was filling a huge amount of screen with a large number of flood lights. This causes the deferred renderer to fill in a ton of pixels.
In this “no cloud killer fill” configuration, I was able to see the 7970 finally pull away from the 580 (a card from a previous generation). The 7970 was able to pull 13.4 fps compared to 10.6 fps, a 26% improvement. Surprisingly, my 6950, which is not a top-end card (it was cheaper than the 6970 that was meant to compete with the 580) was able to pull 10.2 fps – only 4% slower for a significantly lower price.
In all cases, this test generated a lot of heat. The exhaust vent on the 7970 felt like a hair dryer and the 580 reached an internal temperature of 89C.
CPU Still Matters
One last thing to note: despite our efforts to push more work to the GPU, it’s still really easily to have X-Plane be CPU limited; the heavy GPU features (large format, more anti-aliasing, HDR) aren’t necessarily that exciting until after you’ve used up a bunch of CPU (cranking autogen, etc). For older CPUs, CPU is still a big factor in X-Plane. One user has an older Phenom CPU; it benches 25-40% slower than the i5 in published tests, and the user’s framerate tests with the 7950 were 30% slower than mine. This wasn’t due to the slightly lower GPU kit, it’s all in the CPU.
The executive summary is something like this:
- We are specifically optimizing the cloud path for ATI/Windows, which should close the biggest performance gap.
- We still have a bunch of performance optimizations to put in that affect all platforms.
- Over time, I expect this to make ATI very competitive, and to allow everyone to run with “more stuff”.
- Even with tuning, you can max out your CPU so careful tuning of rendering settings really matters, especially with older hardware.
* As X-Plane becomes more fill-rate efficient it has become harder for me to really max out high-end cards. It looks like I may have to simply purchase a bigger monitor to generate the kind of loads that many users routinely fly with.
Working on the installer is neither exciting nor glamorous work, but it is inescapably important. If the installer doesn’t do its job right, nothing else in X-Plane matters, because you can’t fly the sim until you have the sim. This is particularly true since we push out regular patches during the life of the product.
X-Plane 10.04 will complete it’s beta cycle in the next week, and this update includes a new installer with a fairly major change: the DVD installer (which also adds and removes scenery) and the net updater are being merged. (The demo installer remains separate.) The rest of this post explains what’s going on and why.
The updater and DVD installer were originally separated a few years ago so that each update/install product could do exactly one thing. We kept getting tech support calls where users tried to update and instead installed a demo copy. By having each tool do one task and only one task, it helped users to do that one task without error.
Fast forward a few years. People have faster broadband, we have more server bandwidth, and we’re using OpenStreetMap for our scenery; scenery updates are going to be part of the product. (We have already patched 5 DSFs that had major defects when originally produced.)
So now when you go to add or remove scenery, running a net update may be a necessary step to finish the scenery. That is, you need to get new scenery from you DVD, and then grab any newer patches off the web. By merging the updater and installer, we can build an installer that provides updates immediately, eliminating a second step. (This also means that when you first install the sim, we can update to latest before you run.)
The new installer/updater will also remove the “repair installation” option (which is now replaced with “update installation). The old repair option restores an installation back to the DVD version, which is almost always older than the net version, and thus undoes bug fixes (which is hardly a “repair” at all). A net update should be more useful.
Update from the about box is still supported – when you update from the about box, the sim downloads the latest installer/updater and runs it, skipping the main menu and going directly into an update.
The new installer can be found here for Mac, Win, and Linux. The demo installer is here. Links to the updater now simply grab the installer.