I have a lot to blog about this week, but before I can get into any of it, let me describe the process of recutting DSFs.

Alpilotx and I are now working on DSF recuts.  The recuts will incorporate a few important changes:

  • Bug fixes to the OSM import code.  Some of the major cases of ‘missing water’ (e.g. Botany Bay) were due to a bug in the code that imports vector water day from OSM into our DSF generator. Once we fix this code, recut DSFs will get back water that sould have been there but was “lost on import.”
  • Newer OSM data.  The OSM data in the simulator now comes from approximately July of 2011.  OSM is growing fast and being improved every day; for the recuts, we will take the latest data.  If water was missing because it was not in the map in 2011, but that water is present now, recut tiles will get the new water.
  • Airport boundaries from the latest airport file from Robin.  Due to a last-minute data screw-up by me, many airports in the original DSFs were cut with very old airport data, and thus the airport boundary is wrong or missing.  Even the airports that were cut with current data have the airport areas from November of 2011.  Recut tiles will use the lastest (March 2013 when I last checked) data.
  • Better land-class allocation.  While the source land-class data that alpilotx prepared for me is very, very good, the resulting land class in the DSFs is sometimes inaccurate due to the limits of the algorithm I apply to it.  I am working on improvements to that algorithm.  The details of these changes are probably too technical for this blog, but the short of it is that I am trying to fix as many of the algorithmic problems that we’re not happy with as I can in the short amount of time I have been allocated.

A Tactical Recut

At this point you are asking the two most important questions: which tiles will you recut and how will you get them?  The answer to both is: we are starting the process by recutting a small number of tiles that we think are most important to recut, and pushing them out over the network as part of the update process.

At this point we have enough bug data from users that we have a pretty good idea of which tiles need to be recut first.  For example, trees on Chicago’s runway, Botany Bay not existing, and Edwards AFB containing a real (and not dry*) lake have all been reporting about 5,290 times each.  So we will pick the first set of recut tiles ourselves to fix the most visible, highly reported bugs.

But the goal here is also to start an assembly line that can mostly run by itself.  That is, once we get these bugs fixed, we can potentially recut a fixed number of DSF tiles every month and include them in an update, which will move us closer to having the scenery be “live” like the sim and not frozen for the entire version run.

I think moving to a live model for the base DSFs is an important step that we have to make.  Back in X-Plane 6 when I first started flying X-Plane as a user, Austin sending out patches with a new version of the sim every month or two was viewed as crazy.  Look at how far we’ve come: at this point application updates are built into the Apple app store on iPhone (and the Mac app store), and if you don’t push updates, users want to know if you’ve been hit by a bus.  Updates are the norm.

That change is completely logical: back when I started working on commercial software (in the 20th century) distribution was by CD-ROMs, which were mostly sold in stores.  Once your software went out, that was it, so “done” was “done”.

Now software is mostly distributed over the internet; with constant connectivity and increasing bandwidth, it would be crazy for a company to not respond to its customers needs and requests by issuing updates.  One of the most important properties of software as a “building material” is its flexibility — if users want software to do something different, you can easily change that software.  With widespread internet connectivity we have the matched distribution to go with the flexibility of the software itself.

This recut is not a global world-wide recut; I do not know when or even if this will happen. Instead the recut is a first step into keeping DSFs “current”, and the beginning of an on-going process.

The Next Train Leaves in 10 Minutes

The other question that I hear a lot (besides which DSFs and how will I get them) is “when do I need to finish my OSM/apt.dat mods to get them into the recut.”

My goal with incremental periodic recuts is to avoid this issue entirely.  Once we recut a DSF, the cost of recutting it again is very low — we just hit “go” on the DSF generator and replace the files in the update.

In other words, don’t worry about when you “have” to get your updates in by.  We will recut tiles, and if you make a useful change after the tile is recut, we’ll recut it again.  Recuts should be like a train that leaves the station every 15 minutes, not once a week.

(With Robin republished the apt.dat on a regular basis again, apt.dat updates are already like this.)

Executive Summary

TL;DR?  Here you go:

  • We are recutting a few tiles at a time, not the whole world.
  • Recut tiles will come as part of the automatic X-Plane update process.
  • Tiles may be recut multiple times to gather new improvements, so don’t worry about “missing” an update when you improve source data.
  • We are fixing import/programming problems in the first set of recuts.
  • We have enough bug reports on tiles now, we don’t need more yet.

* I’m still not sure what happened at AFB, but my guess is that our importer saw “lake”, got very excited, and ignored the word “dry”.

About Ben Supnik

Ben is a software engineer who works on X-Plane; he spends most of his days drinking coffee and swearing at the computer -- sometimes at the same time.

29 comments on “DSF Recuts – Some Basics

  1. Oops, typo:

    “lot on
    import.”
    … loSt … I guess?

    By the way that’s great news Ben!
    Cheers.“

  2. I assume in bullet 3 you actually mean the airport data from March 2013, not 2012?

    Good news indeed.

    Can you pass along word of how (*if at all) you’d like users to lobby for certain geographic areas to have work done? I’ve done lots of water related OSM edits (and WED airports with proper boundaries) in my flying region which (in my opinion, of course) need to get into X-Plane, and I’m sure you’d prefer not to use the blog comments as that vehicle for that.

    1. 2013, yes – typo corrected, thanks.

      Re: how to lobby…for now the answer is “please do _not_ lobby”. The reason for this is that we need to go through our big pile of bug reports for now and we have plenty to do for the very first round. There will be instructions on how to lobby effectively (and I will be biased toward recutting areas where the recut request contains all of the information we want 🙂 but I’ll post that later once we’re ready to receive the input.

      Considerations for recuts will include the amount of improvement we’ll get in the area, the data size of the recut, the importance of the area to the entire flight sim community, dependencies on the area for third party products, and the amount of recut work people put in, plus a fair amount of arbitrary selection. It may be possible to get an area considered for recut via bribery.

      1. Does this means that small towns with local airports are likely to not get updated? I’m asking this because i’m much of a VFR person and would like to fly around my region with accurate towns and such. Currently there is nothing there besides the airport, but the OSM data is pretty much 90% accurate now and i’d like to be able to use it.
        It’s Rio Grande/Brazil.

        1. No, not exactly. It means they aren’t as likely to get updated in the very first batch. But I think they are likely to get updated at some point. It’s too soon to tell how quickly we’ll get through airports, etc.

  3. “Better land-class allocation … I am trying to fix as many of the algorithmic problems that we’re not happy with as I can in the short amount of time I have been allocated.”

    Since you are re-cutting individual areas are there plans to eventually include OSM building data?

    1. No. I don’t know what future data we will take — OSM is rich with all sorts of tagged stuff — so anything is possible, but generally we do not expect to use the -specific- footprint shapes of polygonal buildings to create autogen for the default scenery.

      Autogen is a trade-off of accuracy vs. visual quality vs. scope, and we think we’d have to go too far backward on the visual quality to get this minor increase in accuracy while maintaining our global scope.

      I think we could use info derived from those buildings, e.g.
      “hey, there is a big building in this block!” and
      “hey, it is a hospital” and
      “hey, it is 200m tall” and
      “hey, it’s big – it’s footprint area is 20,000 square meters.”
      All of those _derived_ parameters could be used to make the autogen process more accurate.

      But when you actually take the exact polygonal footprint, you give up the ability to have things like a really clean visual link between the road grid and the building itself, or good integration between the shape of the building and the side textures, etc.

      Fortunately, global scenery isn’t the only way to get OSM data – you can use something like Benny’s osm2xp to convert OSM building data into facades. So we (LR and friends) aren’t looking to do this, but it can certainly be done!

      1. > But when you actually take the exact polygonal footprint, you give up the ability to have things like a really clean visual link between the road grid and the building itself, or good integration between the shape of the building and the side textures, etc.

        Ben, please can you explain a bit more about this? I just discovered OS2XP, and I was wondering whether its worth the effort. If the files I generate using this software are going to be used to create some kind compromised graphical impression in the sim, as you hint, then I’d be more inclined not to bother.

        On the other hand, OS2XP seems to be very popular, so I’m sure I’m missing something here!

        Thanks

        1. I can’t go into it in a ton of detail now. But basically what I am saying is that:
          – The osm2xp facades are higher in accuracy – with osm2xp the building follows the shape the data dictates to the maximum extent.
          – The v10 autogen is much looser in placement, but higher in visual detail – you have driveways connecting houses to the sidewalks and buildings that form complete blocks with no ‘wasted’ space.

          So it’s a trade-off – there isn’t a right or wrong. The LR approach (autogen) makes sense for what we try to solve with the global scenery.

          It was an intentional design decision of the scenery system that the global scenery is NOT the only way that OSM data (or any data) can be brought into the scenery system; the scenery system is very flexible and can do lots of things that we (LR) will never do with it.

          1. Thanks for that, Ben.

            I applaud you for not just writing an interesting blog, but for engaging in dialogue with us lot. Its refreshing to get personal responses from a company distributing a mass, worldwide product.

  4. you mentioned chicago’s runways having trees. ORD, MDW, smaller airports in the Chicago area, kcgx

    – wil recuts address why some cities are empty of buildings basically, such a Denver ?

    thanks

  5. Interesting.
    Yet I wonder if we will (in this version cycle – V10) eventually move to a more “live” DSF update policy. I don’t mean periodic. Not even the “train every 15 minutes” example you gave.
    I mean actually “live” (maybe filtered to actually do it once per fly session, or once a day or something like that).
    I can see this done in two ways in the future:
    – Have a server that does this on demand on client update request when there is a flight plan (for those tiles) or at least a starting airport (for those tiles and around).
    – Eventually move the translation code IN the program and actually let it request OSM and convert them to DSF.

    This of course will present problems, but the… “PROPER” problems; i.e. making the translator better, not manually debugging all translation. Of course manual intervention (by Laminar) will probably still need to be there, but this should already be in the internal file tags (like “don’t touch this tile” or “this tile has priority such from Laminar”). I hope I am getting through with what I mean (sometimes language barrier is harder to overcome, esp. stupid Mondays with little sleep last night).

    Something else. I realise that terrain generation cannot include actual building data, but as you already said can and will take “advice” on what to render based on those data. This is “good enough” for the trade between realism and performance indeed. YET I strongly believe world landmarks (haven’t looked if OSM can actually classify something like that or relative to that), SHOULD be in the stock world scenery. I think I am saying this for years actually. Even with render engine all notches down and completely stock, you cannot have an NYC without ESB or Chrysler building, or Paris without Eiffel tower or Athens without Acropolis. They are actually used for visual orientation, so I consider them PART of the simulation (I wonder how you are not).

    Anyway keep up the good work.

    1. > Something else…. world landmarks

      +1 to the question. There’s got to be a better way of incorporating significant landmarks in the scenery, than relying on users to import models made by other users manually. Ben?

  6. Very exciting new indeed! Of course, the only bit missing is from this roadmap is a time scale. Are we looking at May or June or July for the first recut to appear? Should I hold my breath?

  7. Hi Ben,

    great news! Just my two cents on the subject: since a relevant part of the scenery updates comes from the contribution of users all around the world, and their submission is “asynchronous” with respect to Your update schedule… I wonder if it would be convenient to keep the scenery update and main application update as two separate processes. Maybe enhancing the X-Plane installer, in order to offer the possibility to look for updates to either scenery, application or both of them could be viable?

    Best regards,
    Filippo

  8. *Reads the interesting new post*

    *Comes across the section: “This recut is not a global world-wide recut; I do not know when or even if this will happen.”*

    *Stares at the monitor and starts to cry*

    Seriously, no? :O No, this can not be true? Since 2006 I have probably spent hundreds after hundreds of hours mapping Sweden for better quality in X-plane. I’ve always gotten down to detail with regard to how sweet the experience would then be in this future global scenery update. Now, it will not be global? Oh my god, Ben 😀

    Pretty please, pretty pretty please do eventually a global refresh of everything. I spent days doing just northern very high res fjords, lakes and rivers. I would cry a river not to see them included in the next scenery update :O

    Not to mention hundreds of effective hours just editing, researching and trimming:
    //osm.org/go/0NcqV8
    //osm.org/go/0NctR2K

    Kindly, Ben, please give a global DSF recut another thought. Just the thought of having spent so much spare time at OSM and risk not having it “prioritized” in the scenery updates makes me want to just give it all up. I just loved the thought of having all the new edits updated in X-plane. This is just a total anticlimax.. At the same time I feel I should just shut myself because I know in the end you guys are smart and are pretty surely doing the right thing anyway… I just so badly was looking forward to at last see all those nice updates I’ve done all over the contry…

    1. Hi,

      First: why would we not recut Sweden? I think Sweden is a valid recut target…I am told the land of Sweden has people living there…they call them…Swedes — and I have even met one or two of these legendary creatures. (They look nothing like the muppet chef!)

      So a recut not being global isn’t really an issue for a European or Scandanavian country. It might be an issue for one of those random 50m rocks in the middle of the pacific that seems to crop up in the SRTM, or perhaps some of the middle of the Sahara (as long as we out ‘sand’ there it can stay).

      But please: no lobbying for a given area just yet. I can’t promise specific areas right now and I can’t take more bug reports than we have.

      Cheers
      ben

  9. In this DSF restructuring flurry I am wondering if the difficulty with autogened roads and their overlay of sections of airport taxiways, ramps, and aircraft parking pads is being addressed. The Excludes which are available do eliminate these interferences from sight, but sadly, any aircraft taxiing over these excluded roads’ edges cause a crash and a restart of the simulator. We BADLY need a method to both eliminate them from sight AND the means for eliminating their hard surface actions which are the cause of the crashes. Am I missing some means that exists from doing this?? For an example, see the airport at KPGD…

    1. The exclusion zone should remove hard surfaces. Please file a bug – attach a MINIMAL scenery pack that can repro this (it should literally just take one exclusion zone) and the runway/taxiway to drive on.

  10. Hello Ben,

    Do you know if the buggy mesh of Rio de Janeiro will be fixed? It has terrible problems since v9.

    1. I do not want to promise to much, or jump the gun … but a test with Rio showed (after changing elevation data source) that it has vastly improved (yes, I know how bad the original XP10 version looks).

  11. Are differential file patches unworkable for DSFs? I’m assuming the algorithms used to convert the various geo data sources (including OSM) need a clean run at tile generation and the differences between old and new so great that diffing isn’t an option? It’d be ideal if the update engine didn’t need to replace entire DSF tiles that have changed but I can see why it needs to.

    1. I’m not sure – we’d have to try it. But my guess is that the patches will be low efficiency — final DSFs are heavily bit-packed and even small changes in source data can ripple through and bit-pattern changes in the entire file.

      Jonathan Morton used deltas on the v6 ENVs to make a patched update cheap to download, but ENVs are a much simpler format and the data changes were localized (in a format where the locality was preserved).

  12. …Due to a last-minute data screw-up by me, many airports in the original DSFs were cut with very old airport data …
    U-huh! Guessed as much when I discovered that the boundary data that I had submitted for all Irish airports ahead of Robin’s deadline was not reflected in the v10 DSF cut. I also notice that the edge of the airfield ground texture can be somewhat soft/blended. In some of the small grass strips I’ve submitted (and no doubt others), the boundary is wrapped fairly tightly around the runway(s), within maybe 2-3 metres at times. This reflects real world cases where the runway may just be a stretch of grass inside an electric fence in the middle of an agricultural field. Will the algorithm throw a hissy fit at this? An example would be EIKH.
    (and if you have to recut the Irish tiles sooner rather than later to test this, so be it, there’s lots of updated OSM data there too 😉 )

Comments are closed.