Beta users, see the bottom of this post for how scalability turns into possible bugs.

In computer science, a program or architecture is scalable if it doesn’t totally vomit up a lung as its constituent parts become bigger. For a cleaner definition, see Wikipedia, source of all internet knowledge.

An ant is not scalable – if you made an ant 100 times larger in every dimension, its tiny legs would break under its new weight. (An ant is not scalable because its weight grows faster than its structural strength. Thus elephants are not built like ants.) Geeks: scalability is to computer science as marginal cost is to economics.

Before X-Plane 940, the apt.dat file was distinctly not scalable. The entire file was loaded into memory; as users created more and more taxiway lines and signs and details, we simply used more and more memory. This approach isn’t very scalable because authors have the potential to grow the apt.dat file faster than our system requirements can increase.

X-Plane 940 fixes this by not loading the entire apt.dat file into memory. Instead, only essential airport information is loaded into memory, along with a note as to where in the file the airport lives. Whenever an airport actually has to be built into a 3-d mesh while you fly, we go back to the apt. file and load the rest of the data for the one airport we are building, use it, and throw it out. Since 3-d airport meshes are built on a second core, the cost of loading one airport off disk is pretty harmless.

The problem with this fix is that it introduced a new scalability problem. Consider:

  • Meshes in 940 are built on as many CPU cores as you have – some users have 8!
  • Each CPU core could be working on a different airport, depending on how many are nearby.
  • Each airport has to load up the apt.dat file to get the extra airport data.

This means that at times on an 8 core machine we could easily have allocated 50 * 8 = 400 MB of memory just to temporarily hold 8 copies of the apt.dat file.*

This is of course completely silly – there’s no reason to load the whole apt.dat file to get one airport, and the fix that is going into beta 8 lets the airport loader surgically grab just one airport. Thus we will be scalable again, because adding more cores won’t cause memory usage to go up.

Beta Users: Please keep an eye out for X-Plane running out of memory – if it starts to do so in the next beta it means that some part of this code change munged memory management. We’re running stress tests on the sim now, but touching the low level memory nd file handling code late in beta isn’t something I like to do.

* While loading 8 copies of the apt.dat file is wasteful of memory, it is not slow; X-Plane uses memory mapped file I/O, so reading a small part of a large file is very fast – just not very virtual-memory efficient.

About Ben Supnik

Ben is a software engineer who works on X-Plane; he spends most of his days drinking coffee and swearing at the computer -- sometimes at the same time.

8 comments on “Scalability and apt.dat”

  1. Hey Ben: Does this mean that scenery developers can finally reload airports without having to reboot the app? Or, at least, that that capability may be realistic to attain?

  2. We could someday do apt reload but for now it's worse than it used to be – if you resave the apt.dat after loading, the in-memory index of where the airports live could be wrong and cause serious problems. Better not to try it for now.

  3. It sounds to me like you need to MD5 hash or even just re-check the modification timestamp of the apt.dat file before re-opening the file for a read, and then go through all the re-indexing if it has changed. (I'd think you'd want to do this for safety anyway) But since I don't know the X-Plane code, I can't say what unknown hitches to that plan arise, nor what the performance loss is. 🙂

  4. hey ben ! Mate I am getting this memory error in X-Plane 9 . Whenever I switch from one airport to another I get this freaking awesome error "Xmapped file ran out of memort". what the hell is this ? I have got a Core 2 quad q9550 @ 2.83ghz and 8800gtx with 768mb memory and 4gb ram..I shouldnt get this stupid memory errors right ? Please advise

  5. Aon – you probably have exactly what the post says:

    "This means that at times on an 8 core machine we could easily have allocated 50 * 8 = 400 MB of memory just to temporarily hold 8 copies of the apt.dat file.*"

    It's a bug in the 940 betas, the patch is not out yet. Wait for beta 8. And 4 GB of RAM doesn't make a difference – you're running out of physical address space – more RAM chips never help that.

  6. so how can i increase the physical address space ? I used to get the same error in X-Plane 9.31 sometimes aswell..I am Currently running X-Plane on Windows Vista 32bit ? Will changing my windows to XP32-bit can remove or atleast reduce this problem ?

  7. Getting the XMapped ran out of memory error in 9.40RC16 in mid-air. Custom mesh and scenery – hope that helps. What i noticed is that prior to the error pop-up my HSI filled with missing VOR/NDB's (prolly not loaded properly earlier due to memory shortage?)

Comments are closed.