One last comment about driver crashes: I try to be fair about the distinction between causation and correlation when it comes to driver bugs; the piece of software that “fixes” a bug may not be the piece of software that contains a bug, and work-arounds happen in both directions between drivers and apps.
Similarly, seeing “ATI” or “NVOGL” in your crash report does not imply a driver bug. It does imply that the crash happened in the driver, but the driver crash may be a proximate cause, while the ultimate cause may be in X-Plane. (Sometimes the stack crawl is also just plain wrong, depending on the severity of the crash.)
OpenGL drivers need to be very, very fast; the whole point of buying that shiny new GPU was to get faster framerates. So driver writers have to make trade-offs about how much error checking they do when an application passes in junk data. If I were writing a driver and debating an error check that would slow performance and prevent a crash of an application that passed illegal data (but not the machine), I might very well skip the error check and say “if the application doesn’t want to crash, it should pass better data.”
Of course, it’s not up to me, and it’s not always up to the driver writers; the OpenGL specification actually dictates what outcome are acceptable in various error conditions. Quite often, “program termination” (read: the application crashes but the machine does not) is an allowed response to certain kinds of bad data from the application to the driver, particularly where checking the data would slow performance for all applications and the application should know better.
My last blog post may have expressed some negative views toward software upgrades:
I know I’ll take some slack for this, but: why did you all go and grab an OS update so fast? What have you gained? The down-side is lost time and application compatibility problems…was it worth it?
That blog post was of course not my real opinion, but rather a recreation of the evil Ben. Always upgrade your software to the newest version whenever it comes out!
My friends say I have become a technological curmudgeon…whenever a new gadget or device or operating system comes out, I just grumble about how that it just means more bugs and mutter about how a byte only had 7 bits when I was young.
How did this happen? Pretty simple. New driver comes out, new bugs and interactions with X-Plane, rinse, wash, repeat. This is not a criticism of the driver writers. Those guys have an impossible job: write code that is super high performance in an environment that is highly unforgiving (an X-Plane crashes X-Plane; a video driver bug crashes your entire machine), then do it again when the next hardware comes out 9 months later. As the apps guy integrating on top of this ever-running treadmill of new hardware and drivers (think I Love Lucy) I sometimes dream of a week when none of the software that X-Plane runs on top of changes.
This is not one of those weeks. Snow Leopard came out, and it looks like Apple has been quite successful in keeping the price low; I’m seeing plenty of bug reports against 10.6 and it’s only been a few days. Apple ships video drivers with the OS, so Snow Leopard isn’t just a new OS, it’s new drivers too, as well as possible changes to the OpenGL “middle” layer that Apple provides.
Now about my pants…over the last few years I have been accumulating graphics card + operating system combinations, so I can see a higher percentage of hardware-related bugs “in the lab”.* But my wife and I are in the process of relocating to Boston, and this has temporarily separated me from a significant subset of my hardware. The result: driver bugs are coming in and I can’t easily reproduce them.
To paraphrase Airplane, I picked the wrong week to quit sniffing OS betas.
(I know I’ll take some slack for this, but: why did you all go and grab an OS update so fast? What have you gained? The down-side is lost time and application compatibility problems…was it worth it?)
*Just because it’s hardware related doesn’t mean it’s a driver bug – X-Plane can be the broken app too! Sometimes drivers work around X-Plane bugs and sometimes X-Plane works around driver bugs. I mention this because plenty of apps developers complain on forums about “driver bugs” – but how many of them repost when they discover a null ptr in their code? A driver and an app are two pieces of software that have to be integrated, and bugs crop up on both sides.
Beta users, see the bottom of this post for how scalability turns into possible bugs.
In computer science, a program or architecture is scalable if it doesn’t totally vomit up a lung as its constituent parts become bigger. For a cleaner definition, see Wikipedia, source of all internet knowledge.
An ant is not scalable – if you made an ant 100 times larger in every dimension, its tiny legs would break under its new weight. (An ant is not scalable because its weight grows faster than its structural strength. Thus elephants are not built like ants.) Geeks: scalability is to computer science as marginal cost is to economics.
Before X-Plane 940, the apt.dat file was distinctly not scalable. The entire file was loaded into memory; as users created more and more taxiway lines and signs and details, we simply used more and more memory. This approach isn’t very scalable because authors have the potential to grow the apt.dat file faster than our system requirements can increase.
X-Plane 940 fixes this by not loading the entire apt.dat file into memory. Instead, only essential airport information is loaded into memory, along with a note as to where in the file the airport lives. Whenever an airport actually has to be built into a 3-d mesh while you fly, we go back to the apt. file and load the rest of the data for the one airport we are building, use it, and throw it out. Since 3-d airport meshes are built on a second core, the cost of loading one airport off disk is pretty harmless.
The problem with this fix is that it introduced a new scalability problem. Consider:
- Meshes in 940 are built on as many CPU cores as you have – some users have 8!
- Each CPU core could be working on a different airport, depending on how many are nearby.
- Each airport has to load up the apt.dat file to get the extra airport data.
This means that at times on an 8 core machine we could easily have allocated 50 * 8 = 400 MB of memory just to temporarily hold 8 copies of the apt.dat file.*
This is of course completely silly – there’s no reason to load the whole apt.dat file to get one airport, and the fix that is going into beta 8 lets the airport loader surgically grab just one airport. Thus we will be scalable again, because adding more cores won’t cause memory usage to go up.
Beta Users: Please keep an eye out for X-Plane running out of memory – if it starts to do so in the next beta it means that some part of this code change munged memory management. We’re running stress tests on the sim now, but touching the low level memory nd file handling code late in beta isn’t something I like to do.
* While loading 8 copies of the apt.dat file is wasteful of memory, it is not slow; X-Plane uses memory mapped file I/O, so reading a small part of a large file is very fast – just not very virtual-memory efficient.
A while ago, Austin was on FSBreak, and I wrote this post as commentary on his interview. The main point I meant to make was this: from what I have heard from other engineers and seen in my own experience, most software companies prefer to develop new features on top of implementations that are known to have architectural problems. At LR, we fix the implementation architectural problem first, and that has been a net win for us.
Now that’s basically a statement of my opinion on software engineering – in hindsight it probably belongs on my programming blog and not here. Unless you develop X-Plane plugins, you’re not a programmer; I will try to constrain future scenery blog posts to things that non-programming X-Plane users will notice. If you are a plugin developer, you might want to look at the “Hacks of Life” posts tagged with OpenGL.
Anyway, back to the story…the responses to that blog post were all well thought out comments on X-Plane’s quality control process. At the time my immediate reaction was: that’s totally off topic – I’m commenting on architecture and they’re talking about QA. I do think the authors made fair points.
But in hindsight, I think that there’s a deeper issue: one of verifiability. Simply put, my statement that we (LR) rewrite stale implementations is nearly impossible to verify without source code access, something that you can’t get for X-Plane. So from the perspective of anyone outside the company, my original statement is not falsifiable (it cannot be proven false) and thus rather useless as a statement of fact. Even though I claim that we make rapid progress on features by keeping implementations clean, you as a user don’t care how we develop our features – clean architecture, more developers, or the use of time travel and voodoo dolls, it’s a bit moot.
Thus the comments were off topic, but also they were moving away from an unverifiable topic and toward one that users can measure, namely the quality of X-Plane’s betas.
There’s a fair amount of marketing that gets put out in the tech and games industry. It’s a slippery slope from giving a new, real technology a whiz-bang name (e.g. HyperZ is a real technology, and it is good for your frame-rates) to using techno-babble to make the bad seem good (e.g. HyperMemory just means that your video card lacks VRAM and is going to be slow). When new products come out, the feature list is parroted, but it’s not always clear whether the new features turn into real benefits.
So what I’m going to try to do with the scenery blog is: try to keep the blog limited to verifiable, measurable aspects of X-Plane. If we ship X-Plane with “psychoactive rendering*”, I’ll try to explain what the heck that is and why you’d want it, and how you might notice that it’s working.
* X-Plane does not have psycho-active rendering, except possibly when I make a mistake in the shaders and everything turns purple.
X-Plane 940 beta 7 is out. Now is the time to test your airplanes, scenery, and plugins. We’ve reached a point where we think we have the new systems code working. Please try the new beta, test your add-ons, and if something works in 930 but is broken in 940, file a bug immediately!
Two warnings about normal maps:
-
Make sure that the RGB color underneath transparent sections does not turn black or white! Some image editing programs (in particular Photoshop) will lose the color beneath a transparent area.
With a normal map, this is very bad – black and white are not legitimate normal map colors, and the result will be bogus normal vectors under the non-shiny part. Normal maps affect more than just specular shiny hilights – the normal map affects all lighting, so having black or white under your transparent (non-shiny) parts is bad news.
To check whether this has happened, I recommend Graphics Converter, which will show you your alpha and RGB channels separately, exactly as they are in the file.
-
Make sure your RGB value are normalized. The “length” of the normal (as encoded in RGB) must come out to a distance of 1. This is virtually impossible to do using PhotoShop or an image-oriented program…I suggest you use a real plugin to PhotoShop or Blender to create normal maps that are correctly “normalized”.
It is also very possible that X-Plane’s gamma correction is distorting normal maps, but that’s one for me to fix.
I just found out from a user that Chase View Delux doesn’t work with X-Plane 940 beta 4. Here’s what’s going on:
- Plugins often end up running X-Plane code – when you make a call to the X-Plane SDK, the SDK sometimes passes the call right to X-Plane.
- X-Plane 940 beta 4 has heavy error checking on all of the X-Plane code.
Put these together and…X-Plane 940 beta 4 is putting heavy checking on plugins that wasn’t there before.
Typically the kinds of problems caught by the error-checking code inside a plugin are not critical, but aren’t bad things for authors to fix. Since we will turn off heavy error checking as soon as we get the systems code fixed, plugins will start working again..there’s no need for plugin authors to recut anything.
I’m driving at 55 mph on the highway. I drive over a nail, lose a tire, skid off the road, crash through the guard rail, plunge off a cliff, and die. That’s not much fun. But when you get to the accident site, you’ll probably be able to piece together what happened. The skid marks, the nail, the hole in the guard rail, the car wreckage below – you can connect the dots.
Now…let’s say I’m driving 500 mph. Same nail, same out of control crash. But this time it’s going to be a lot harder to tell what happened. Lord knows where the nail ends up, the distance from the nail strike to exiting the highway is going to be a lot bigger, and the car is going to be in smaller pieces scattered over a wider distance. It’s going to take a lot longer to piece together what went wrong.
That, in a nutshell, describes the motivation for an X-Plane beta with all of the debug and safety checks on. X-Plane’s normal operation is like the car doing 500 mph – when it crashes and dies, there’s very little left that can be used to figure out what went wrong. When there is a bug in the code that destabilizes the sim, finding it via crashes in release builds takes a lot of developer time and slows the whole beta down.
With the safety checks on, X-Plane still crashes when something goes wrong – but the bodies and wreckage are all a lot closer to the scene of the crime, and the evidence left around is in much better shape.
One of the side effects of the safety builds is that they catch “harmless” coding mistakes – (harmless in quotes – the bug might seem harmless but who knows if that will always be true). XSquawkBox now quits the sim with an ugly alert box because it reads off the end of a piece of the airplane data structure via the plugin system. This hasn’t hurt things in the past, but it’s not really correct. Beta 5 should fix the underlying problem, letting you run XSquawkBox again. (The fix will be in X-Plane, not XSquawkBox.)
X-Plane 940 has a new electrical systems model and it has a few important differences from 930:
- You must specify exactly how many buses your plane has. 930 provided two buses but then did a bunch of cross-tying behind the scenes in case you didn’t have enough power sources.
- X-Plane 940 requires that each battery and generator be on exactly one bus.
- X-Plane 940 will not allow systems to be powered by non-existent buses.
Now this can have some strange side effects. Consider the default king-air with two generators and one battery:
- In 930 it has two buses, battery feeds both buses, and each generator feeds one. You could have systems split by bus and they would work unless you lost one generator and the battery.
- 940 defaults this plane to one bus, because on battery power only one bus will be fed.
- This means that in 940 all of your systems will be reset to bus 1.
But wait…if you import into Plane-Maker, the import will trash your system bus selections before you can increase the number of buses to 2. What can you do?
Here’s a work-around: before you update your plane, make sure you have two battery and two generator switches on your panel. Then open in 940. The import will set 2 buses and your systems will be preserved.
Of course, by the next few betas this may all be moot because we may get something less crazy in there.
