Tag: installer

Fun With Servers

Just an FYI: when it rains it pours. Normally betas increase load on our set of update servers. To compound this, one of them is suffering a midlife crisis^H^H^H^H^H^H^H^Hhard drive failure.* We’re working on it now; hopefully it will be resolved in the next 24 hours.

EDIT: the update server is back up – our host not only swapped out the drive, but the whole box. We’ll have to take it down one more time in the future, but for the most part I think we’re out of the woods.

Another note on servers: Chris has restructured X-Plane for Android to separately download the art assets from our servers, rather than contain all art assets in the actual download. What he found after several painful weeks was that the Android store is not yet reliable for large apps. While the official app size limit is 50 MB, many phones have problems with their configuration that cause downloads to fail. When the user buys our app and the download fails, they get angry at us. (X-Plane may have been, until it was restructured, one of the largest Android game APKs. The other games with large amounts of 3-d content were already doing separate downloads.)

We originally wanted to build a monolithic app (everything in the APK) because we thought that this would provide the simplest, easiest configuration to maintain, and thus hassle-free installation for our users. You get the APK, you install it, you fly! Unfortunately, the Android Market isn’t reliable for such a large download, so we had to re-evaluate.

The new system downloads only the core app from the Android Market and then pulls the art assets from one of our servers. So far this appears to be an improvement. If/when Google provides an integrated solution, we will probably switch back to it to simplify the process again (right now we have two points of failure: the Android Market and our server farm, which, per the above notes, sometimes does fail). But for now, we’ll host the apps and try to give people the best download experience we can.

Finally, I will try to roll out at least a beta of new installers some time this week. The new installer simultaneously downloads from multiple servers, with a more efficient HTTP implementation; this should hopefully result in better download times and also lower server load per demo.

* Chris pointed out: most normal humans don’t know what this ^H^H^H^H is about…it’s nerd-speak for the delete key, e.g. to undo a text. ^H is control-H, which you may find works just like the delete key. Yes, I’m a huge nerd.

Posted in Android, Development, News by | 1 Comment

Updater Design Decisions

My blog postings have been a little thin – but not because nothing is going on. Rather it’s a combination of working on next-gen stuff that is still in heavy development, travel, and (perhaps due to the travel, perhaps due to lack of sleep, or perhaps due to certain forms of Brandy brought on site by a certain British developer who won’t be named, but whose handle on the org starts with a P and rhymes with “Cropsman”, cough, cough) a minor fever that I’ve just about recovered from.

A user asked me about the design of the X-Plane updater – so this post is only going to be of interest to the few authors out there who are creating installers for their add-ons and need to update. We try to keep the file formats for X-Plane simple, e.g. “this one folder is your airplane”, so that if you don’t make an installer, the user won’t be overwhelmed with a complex unpack-and-install routine. But if you do need to install or update, well, here’s what we were thinking when we developed our installer.

We had a few needs for our installer:

  • It had to be cross-platform to Mac, Windows and Linux.
  • It had to update existing products as well as install new ones.
  • The updates had to be minimal deltas, e.g. a demo can be 600 MB but an update should only be the 30 MB of files that really changed.
  • We wanted the updater to do delta updates from any old version. (Some update systems require installing a number of patches in sequence. We have users who buy the DVD and then update a year later; we wanted to let them update in one shot.)
  • Building the patches had to be really, really fast. We use our updater to publish our betas. So while X-Plane probably has 2-3 major patches in a single year and maybe 5 or 6 bug fix patches, we will (due to betas) cut perhaps 100 or more actual “updates”.
  • On that last point, cutting separate updates by platform was a deal breaker, as it would triple the number of updates we had to cut. Having a demo install and an update be the same on the server was a big win.

Some of those needs are pretty specific to Laminar Research, but some might apply to a third party. There are commercial installers that will let you generate patches. If you don’t generate a lot of patches, you can probably use existing tools – basically you’ll spend a lot less time up front (since the tool set exists) but you might spend more time in the long run cutting separate patches by OS.

One of the first things we decided was to not use server-side technology or special protocol. That is, we don’t need any kind of smart servers to run the updater – the updates are just files hosted over HTTP on standard rented or owned apache servers. All of the work is done in the client. We did this for a few reasons:

  • It lets us throw up the install anywhere – we don’t have complex needs for what’s on the server. Virtually any server will do. (As the company has grown, our server needs have grown too, so this is less of a concern now, but back then we had fewer servers in service, and were buying less overall server capacity.) If a server goes down, we can press another one into service about as quickly as we can move the file set to the server.
  • We’re not server programmers. Coding the installer/updater on the client side let us leverage existing company technology, etc.

Here’s what we came up with. The basic idea of the installer is very simple. Note that since the installer uses HTTP files, you can “watch” the installer simply by downloading the files that it downloads.

  1. First the installer goes to a master server and gets a “version list”, which tells it what it’s going to actually get and what (multiple) mirrors are available. This one master file is the only file that must be hosted at X-Plane.com, and it allows us to change a very small file to press mirrors into service.
  2. Installs and updates are actually the same to us – it’s a set of files that the user should have once the install or update finishes. This starts with a file directory that lists every file (by path) that needs to be downloaded, as well as its MD5 signature.
  3. The directory also contains the MD5 of every old version of any file in the version, and lists files that have to be deleted to update the version (e.g. the file panel.png used to have MD5 signature 2934b..23abc2 but is now removed).
  4. The client, once armed with the file directory, can now download all files in the directory (install) or compare the existing files on disk to the directory by MD5 and only download changes.
  5. Individual files are on the server in zip form for download and decompression.

That’s pretty much it – you could write an installer in a scripting language using a tool like CURL or WGET – it wouldn’t be very complex because the installer is really just a clever, painted download+unzip tool.

Dealing With Modified Files

There is a bit of complexity in this design that you might not need for a third party installer: handling modified files. Note that the directory contains the MD5 not only of the current version of the file (to detect when no update is needed and save bandwidth) but also to see when the user has modified a file that is “managed” by X-Plane.

Before the installer overwrites/deletes a file that you own, it compares the file’s MD5 to the entire known list of MD5s for the entire version. If this file doesn’t match a known old version, it puts up the warning dialog box that you have modified a file that is about to be overwritten.

While we recommend that add-ons use scenery packs and other “safe” ways to customize the sim, this helps detect when a user has gone in and edited the cloud textures in Photoshop, and prevents us from overwriting them without warning.

Posted in Development by | 4 Comments

1,2,3,4,5…6,7,8,9,10,11,12

Surely I’m not the only one with that song embedded in some deep part of my psyche, right? One thing we seem to have down with 940 is counting upward…945 is in testing now.

There are two things driving the perpetual 940 bug fix releases:

  1. Third parties are using the sim heavily, so the bug fix releases have been vehicles for fixing a number of small bugs that are only reproducible with custom add-ons. (The fix for night lighting and orthophotos is an example of this.)
  2. We’ve played a real game of whac-a-mole with the throttles. (The bug is that X-Plane will lock the two throttles together. I believe that the new 945 release candidate finally fixes this.) Having done a bit of a 3 stooges routine on this bug, we’ve tightened up our internal procedures a bit.

As always, if your third party add-on broke with 940, please file a bug report! The only case of this I know of is the throttles bug – 945 should fix the BK-117, the Mu, and any other plane suffering from throttle-glue.

Will there be a 946? There very well might be, and I sincerely hope it will not be because 945 missed its target. But if, two months from now, we’ve collected another half-dozen fixes for third party authoring, I don’t see why we wouldn’t release them.

Could The Name Beta Be Any More Confusing?

The version number gets bumped when we have a release and it still needs patching. Of course, some bugs get reported after we ship, and some address issues not even under consideration during release. Had we not had any fire drills, we would probably still be on 942 or 943 right now, because we’ve had third party reports with very specialized authoring cases after we finalized the build.

Here’s what makes things even more confusing: that check box “Check for new betas as well as updates” in the installer. At any one time we may have two versions of X-Plane availalbe for download:

  1. The official version (currently 944) for everyone.
  2. Whatever the latest in-test build is (currently 945).

(You can even see which versions are current here.) What’s confusing is that while this check box says “get betas”, in truth it gets any build that is “in staging”. At this instant that I blog, X-Plane 9.45 final is in staging. That is, we believe 9.45 fixes our two known bugs, we think it’s done, it’s a release candidate, we think it is the next American Idol^H^H^H^H^H^H^H^HX-Plane. But until we certify it, you still have to check “get betas” to download it.

What we probably need to do is re-label the check-box in some way; the “staged” approach where every build is available for test first is, I think, a valuable tool to let third party authors confirm that we fixed their bugs before the entire world gets the app.

Posted in Development by | Comments Off on 1,2,3,4,5…6,7,8,9,10,11,12

Scalability and apt.dat

Beta users, see the bottom of this post for how scalability turns into possible bugs.

In computer science, a program or architecture is scalable if it doesn’t totally vomit up a lung as its constituent parts become bigger. For a cleaner definition, see Wikipedia, source of all internet knowledge.

An ant is not scalable – if you made an ant 100 times larger in every dimension, its tiny legs would break under its new weight. (An ant is not scalable because its weight grows faster than its structural strength. Thus elephants are not built like ants.) Geeks: scalability is to computer science as marginal cost is to economics.

Before X-Plane 940, the apt.dat file was distinctly not scalable. The entire file was loaded into memory; as users created more and more taxiway lines and signs and details, we simply used more and more memory. This approach isn’t very scalable because authors have the potential to grow the apt.dat file faster than our system requirements can increase.

X-Plane 940 fixes this by not loading the entire apt.dat file into memory. Instead, only essential airport information is loaded into memory, along with a note as to where in the file the airport lives. Whenever an airport actually has to be built into a 3-d mesh while you fly, we go back to the apt. file and load the rest of the data for the one airport we are building, use it, and throw it out. Since 3-d airport meshes are built on a second core, the cost of loading one airport off disk is pretty harmless.

The problem with this fix is that it introduced a new scalability problem. Consider:

  • Meshes in 940 are built on as many CPU cores as you have – some users have 8!
  • Each CPU core could be working on a different airport, depending on how many are nearby.
  • Each airport has to load up the apt.dat file to get the extra airport data.

This means that at times on an 8 core machine we could easily have allocated 50 * 8 = 400 MB of memory just to temporarily hold 8 copies of the apt.dat file.*

This is of course completely silly – there’s no reason to load the whole apt.dat file to get one airport, and the fix that is going into beta 8 lets the airport loader surgically grab just one airport. Thus we will be scalable again, because adding more cores won’t cause memory usage to go up.

Beta Users: Please keep an eye out for X-Plane running out of memory – if it starts to do so in the next beta it means that some part of this code change munged memory management. We’re running stress tests on the sim now, but touching the low level memory nd file handling code late in beta isn’t something I like to do.

* While loading 8 copies of the apt.dat file is wasteful of memory, it is not slow; X-Plane uses memory mapped file I/O, so reading a small part of a large file is very fast – just not very virtual-memory efficient.

Posted in Development, Scenery by | 8 Comments

New Installers

The 2.05 installer is now released – this is a multi-language update that fixes a few bugs and makes the map interface more usable. If you have existing DVDs, you may want to use the new 2.05 DVD installer because it scans the scenery folder very quickly when you pick “add-remove scenery”.

(The old installer would check the signatures of all scenery files – this one simply checks whether they exist.)

No URLs have changed – installers will always have the same file names and live at the same URLs; you can pick “Get Info” or “Properties” to tell their version (or run with –version on Linux).

I have a ton of emails I need to get through, so if you emailed me, I do apologize; I will try to clean out the pending tech support issues, etc. in the next week.

Posted in Development, News by | Comments Off on New Installers

Releases, Bugs, and the iphone

Sometimes I have one of those weeks where all I do is look at crashes and weird behavior.  This is turning into one of those weeks.  So here’s some status on the various bugs floating around.

I should say: you’ll find a lot of developers blaming the technology providers for bugs (just look at how many OpenGL developers have blamed ATI for their apps crashing).  Sometimes it’s the app, sometimes it’s the driver.  More importantly:
  • You don’t know who’s fault it is until you fully understand the bug.
  • The fix for a bug might not be in the broken code.  That is, one piece of code can work around a bug in another.

So…you can’t necessarily tell whose fault it was from new drivers coming out, us changing the sim, etc.  But…when it’s my stupid code, I’ll admit it openly – no one should think that a bunch of other smart programmers are screwing up on my behalf.  (This is also useful to other apps developers, who can know that my bug isn’t the same as their bug, since my bug was in my code.)

X-Plane: we’re happy with 921r2 finally…the final bug (crash on startup on the Mac) was due to an incompatibility between Apple’s OpenAL implementation and special memory allocator (which is really just a wrapper around NEDMalloc).  I still don’t know exactly what the rules should be (you try reading the C++ spec!) but for now we turned the allocator off on Mac.
(This brings up another issue about bugs – you can’t tell whose bug it is by whose code crashes, since one piece of code can sabotage another.)
So the next X-Plane release will probably be 930, with new features.  We may have a few more language patches if needed.
iPhone: the 9.01 iPhone patch is out, and it improves framerate a bit.  We are still seeing crashes on startup for users who have just downloaded the app.  Rebooting the phone will fix this, but please see this post for more info!  We need your help to collect more data.
Radeon 9800 on Windows: for the longest time we’ve had users reporting “framebuffer incomplete” errors when using catalyst 8.x drivers, an R300 chipset, and Windows with X-Plane 9.x.  I have been trying to reproduce this problem “in the lab” off and on for months, but finally saw it this week.  From what I can tell, we’re getting into some low-memory condition and the driver is freaking out in various ways.  The command line options people sometimes use to get past this are probably rearranging memory, not saving it.  I don’t know why the Catalyst 7.x drivers don’t have this problem.  But…at least I am making more progress than I was before.  Please see this post for more info.
Installers: I am working on the 2.05 installer.  I have seen a number of users report problems running a full install from DVDs, so I am just starting to investigate that.  I will post more when I have something to test.  Unfortunately the problems reported are not something we see here.
Posted in Development, News by | 1 Comment

Bad Alloc, New Patch, New Installers

The 921 patch is now available. This fixes the bad-alloc problems relating to complex airports.

Please note: if you have a bad-alloc crash, it could be because you are out of memory. Make sure you have virtual memory turned on, your page file is large enough, your disk isn’t full, etc. If you have a bad alloc error, try the sim without third party add-ons to see if you really are running out of memory. If you are running Vista or XP, use /3GB or the BCD – see this for more info. Basically when you are running out of memory, you either can crash on “bad alloc” if we need memory for the CPU or “we ran out of video card memory” if we can’t map geometry into virtual memory.

If you get a crash with 921 on OS X, please let me know by email! I’ve seen one of these reports.

I also have cuts of the new installer/updater suite, version 2.05 – if you are going to update, tryo ne of these:

//dev.x-plane.com/update/installers9/stage/

The main features of the new installer are:

  • Support for French, Spanish, Italian, German, and Russian.
  • Clearer colors on the world map.
  • Scanning the global scenery folder to add/remove scenery is much faster.

I recommedn the new installers for that last point alone.

Posted in Development, News by | 1 Comment

Why Is Installer Scanning Slow?

When you update X-Plane, it scans about 10,000 files before it runs the update. This scan checks every single byte of every file. As a result, it’s a little bit slow – it takes a few minutes.

With the new installer, when you add and remove scenery, we do the same scan on the installed scenery. This scan is very slow – 78 GB is a lot of data!

When the next installer comes out (2.05, which will be cut after 920 goes final), the add-remove scenery function will do a quick scan to see what files exist. This is a lot faster – even with full scenery the scan takes only 10-20 seconds on a slower computer.

I can’t pull the same trick for updates – we need to detect any possible defect in a file during update to ensure that the install is correct!

Posted in Development by | 4 Comments

Installer Chaos

I spent part of the morning tracking down installer issues.  Here are some things I have learned:

  1. If you have Windows Vista and User Access Control, there’s a good chance that the installer may get caught on permissions problems.  I don’t fully understand what’s going on, but I don’t think it’s a coincidence that it is US GraphSim customers who see this.  

    Basically those DVDs, which went to press a little bit early, default to installing X-Plane on the C drive, which is not a generally accessible area.  UAC then makes some guesses and allows some but not all operations (or something equally weird) and the user gets stuck.

    I am still investigating, but I think the fix is to install to your home folder.

  2. Some users have bad DVDs.  It happens, and I think it happens more now that we’re dual layer.  There’s also a variance in the sensitivity of drives – some users send us back damaged disks and they look bad but we can still read them.

    Tech support will tell you this, but if you can’t install, one test is to simply copy the entire DVD to your hard drive; if the copy fails mid-way with “I/O error” or some other message about damaged media, call up tech support, they’ll swap you a new disk.

    We try to find duplication facilities that do high quality work, but bad disks do happen.  Tech support should be able to make this right for you.

  3. I’ve gotten a wide variety of weird reports on Linux – there are a lot of distros out there, so potentially a lot of versions of the OS code base.  For Linux users I strongly recommend the X-Plane Wiki, where we are trying to accumulate all of the fringe cases.
Posted in Development by | 7 Comments

901 – Stealth Release

901 is out now…basically it’s just 900 + French and German localizations. Also a few minor tweaks:

  • We added a few datarefs and commands by request – check DataRefs.txt.
  • The plugin font size is back to 6 pixels, like v8. It looks a bit ugly now, but we’ll put a new font in place soon. The important thing is: 6-pixel fonts for v8 and v9, so one plugin can operate everywhere.
  • A few plugin bug fixes – see the SDK known bugs page.
  • Linux won’t complain if your disk is mounted as iso9660. Since the v9 final disks are all mastered as ISOs and not UDF, asking you to mount them as UDF is asking the impossible.
  • If you have a demo install, it will add the word “Demo” to its name in the startup screen. This is to help users with more than one install identify which one they are running.

If this all seems minor and tweaky, well, it is. The main point of 901 was to get the localizations in. In the future we’ll have a feature patch that brings new airplane features in and other cool stuff.

901 is freely available using the X-Plane web updater…the download size is only about 14 MB.

Finally, I’ve posted all new DVD installer/updater/web demo installer apps – version 2.04 localizes to French and German too.

Posted in Development, News by | 2 Comments