Blog

All Your VRAM Is Belonging To Us (and Plugins)

One thing I have learned during my time here in the matrix is that users often find new bugs to be significantly more annoying than existing ones; trading an existing bug for a new one is often seen as a big step backward, even if the new bug isn’t that bad. I think this is just human nature – we get used to things – it’s what humans do – and this takes the sting off of some bugs that have been around a while.

I bring this up to put in context the problem of texture paging, blurry textures, and VRAM management in 11.50. To understand what makes VRAM paging hard, you have to understand how the OpenGL driver solved the problem, and ask yourself: what was the old bug, what is the new bug?

What Is Texture Paging?

Texture paging is when textures are moved between the memory on your graphics card (VRAM) and the memory in your computer (“system memory”). At any given time, X-Plane may have more textures loaded than you can fit on your card, but it doesn’t need them all at once. As you fly, different textures are needed, and the textures have to be moved around.

If this sounds a lot like moving data from RAM to your hard drive (virtual memory), it is. When I get the question “how can X-Plane be using 2.5 GB of texture memory when I only have a 2 GB card” the answer is…texture paging.

How Did the OpenGL Driver Do It?

In an OpenGL based app, paging textures to VRAM and back is entirely up to the OpenGL driver. The algorithm goes something like this:

  1. Start drawing.
  2. Realize something you need is in system memory.
  3. Move something you haven’t used in a while out of VRAM.
  4. Move the thing you need into VRAM.
  5. Continue drawing.

The OpenGL driver pages during drawing when we need something new, and the result is a stutter – the frame takes longer because it cannot be completed until the texture is moved to VRAM.

Now NVidia, AMD, and Apple have spent a lot of time and effort to make this work as well as they possibly can, and I’m definitely selling short the complexity and sophistication of the drivers as they try to guess really well what should be moved out of VRAM. Still, two key facts will be true:

  1. If we need to page stuff into VRAM to draw, and we don’t figure that out until right when we need it, there’s going to be a stutter. Maybe a really short one you barely notice, maybe a really bad one, it’s a question of “how much”, not “if”.
  2. What you see on screen always looks fantastic – never blurry – because the driver won’t cut resolution to save VRAM.

So…the old bug in this case is stutters due to VRAM containing the wrong stuff.

How Does X-Plane 11.50 Do It?

With Metal and Vulkan, it’s up to us, the app, to decide what goes in VRAM and what doesn’t, and to schedule the movement of data between VRAM and system memory. Our strategy is:

  1. Keep an eye on how much memory is free.
  2. If it looks like it’s getting too close to full, shrink textures to a smaller size. Prefer to shrink textures you’re not looking at. Do the shrinking in the background.
  3. If it looks like VRAM has empty space free, grow textures, preferring ones you are looking at now. Do the growing in the background.
  4. If we have to load new textures in the background (e.g. new DSFs as you fly), wait until old textures can be shrunk.

We have a huge advantage over the drivers in this game: we know the textures we might need in the future, long before we actually need them. We can also shrink textures. This means we don’t have to stop and shuffle things around, and that means we can avoid stutters. So we can fix the old bug: stutters while you fly.

(To be clear: this is not the only source of in-flight stutters while you fly – OpenGL and 11.41 are full of them.)

The down-side is: if VRAM gets tight, we’ve made some of the textures lower resolution. This is a new behavior (blurry textures), so my fear is: users aren’t going to like it.

In particular, the OpenGL driver uses the exact optimal set of textures for every single frame – at the cost of stuttering. We may not have things shrunk and grown in exactly the best way while you fly, so for the same VRAM and no stutters, we may not be as visually sharp.

What About Plugins

Add-ons that use OpenGL (e.g. aircraft with custom glass displays) use VRAM too – this is one reason why we can’t pack 100% of VRAM with our textures; we need to have enough free that if a plugin needs VRAM, it won’t crash us.

My expectation is that fighting blurry textures and crashes due to running out of VRAM is going to be a long, iterative process and one of the main reasons to have beta testing. It’s one thing to have an algorithm that works good in the lab or on paper – it’s another to have thousands of hours of test time in real-world conditions on user machines with lots of add-ons and anarchy.

So when 11.50 is public beta and you see your blurry texture, don’t panic. VRAM management is a very new part of X-Plane, and it’s a very important part, and it’s something we will iterate on over time.

(Finally, I want to mention that while the port to Vulkan/Metal has been a joint venture that Sidney and I have worked on together, there are certain parts of it where Sidney basically did 100% of the work, and my only contribution was to complain about bugs. The shader compiler, the OpenGL plugin bridge, and the VRAM pager are all big lifts that Sidney has done on his own that make the Vulkan/Metal port possible.)

Posted in Development by | 148 Comments

The Vulkan/Metal Public Beta Will Be in 2020

As I announced in my previous post, X-Plane 11.50, featuring native Vulkan and Metal driver support, is in a private beta made up of people in our dev-rel program (E.g. third party developers). We’ve gotten some really good bug reports from them – the reports are good, not the bugs – and based on that, it’s clear that 11.50 will not be ready to go public this year (meaning tomorrow).

This is disappointing to us – we burned candle at both ends to try to get all the way to public beta this year, but at this point the build just isn’t ready yet.

I’ll describe in detail below what the one bug is that is really holding back public beta, and I’ll try to do a general FAQ about Vulkan and Metal tomorrow; based on the 150+ comments from the last post, I think it’s clear that we have a lot of basic info to get out there, some of which has only been shared with third party developers until very recently.

Please don’t bombard me with “can I please be in the private beta” (or even “you’re a giant pile of moose poop for not letting me into the private beta”). Here’s the thing about the private beta: our goal is to kill off the bugs so everyone can use the beta, and to kill them off as rapidly as possible.

At this point the dozen or so third party developers who are banging on the beta are generating bug reports faster than Sidney and I can fix them – if we add more people to the beta, our bug fix rate will slow down as we have to spend more time triaging the reports coming in. So as long as we’re fixing the worst bugs productively, more people in the private beta means everyone waits longer to get to public beta. I think the best thing here is for Sidney and I to just try to fix things ASAP.

(I think there’s also a win to getting this into the hands of third party developers – in some cases we’ve seen add-ons that accidentally use the wrong APIs for processing and are thus incompatible with Vulkan when they don’t need to be – developers can get a head start on updating this code.) If you are a third party developer, we can get you into the beta program – I don’t want to favor some third party developers over others.

Blurry Textures

A short summary of performance feedback from our internal beta might go like this: smoothness and FPS are good, VRAM management is not.

Most of the crashes while flying we have seen look like the sim runs out of VRAM and isn’t able to recover, and for users with smaller GPUs (e.g. 4 GB of VRAM) X-Plane gets into a tight VRAM situation and solves it by making everything on screen low res and blurry.

The VRAM management path, which is entirely new code written for the Vulkan/Metal back-ends, is…well, it’s entirely new, and it’s going to need a bunch of testing, debugging and iteration to get solid. On an 8 GB card, VRAM is so plentiful that X-Plane’s somewhat naive VRAM allocation scheme works fine. On a 4 GB card, we’re just a tiny bit short (a few hundred MB) but the result is an alarming loss of texture resolution.

This is something we can make better! It is under our control and is thus being worked on now. (By Sidney that is – I’m writing this blog post while he gets the real work done here.) But until we have better VRAM management, it’s too soon to go public. If we did, we would just be inundated with hundreds of “my textures are blurry” reports. It’s a bug too noisy to ship with.

Why Wasn’t This a Problem on OpenGL?

One might ask: why don’t you just do whatever OpenGL did to manage VRAM? The problem is: OpenGL’s solution is to stutter.

The OpenGL driver swaps textures between VRAM and system memory based on what you really need to draw now. If you are looking away from Seattle, the space needle’s texture can live in system memory, but you might need mountain textures for Rainier.

We try to do that too – the big difference is: OpenGL will stop rendering while it moves the textures around, and we will not. The result is that OpenGL always shows you a perfect image at the texture resolution you picked – no blurred images. But you might have stutters! You can see this if you’re on the margins of VRAM by flying and then changing views or circling the camera – if your FPS go “chunk chunk chunk VROOOM” you’re eating some stutters while textures move around, then you go fast again.

The OpenGL way is a solution that we never thought was acceptable – our goal is the best image quality without sacrificing smoothness. It’s going to take a little more time to iron out low VRAM situations, but once we get there I think it will be worth it.

Posted in Development, News by | 116 Comments

Vulkan and Metal: Testing and Bug Fighting

Last week we sent out our first build of the Vulkan/Metal ready X-Plane (which will be version 11.50) to an external sight, and last night we sent out the first private beta.

Our plan is to move to an open public beta soon once we have some feedback on how well the build is working. When that is will depend on the bug reports; I’m crossing my fingers.

How It Works

In X-Plane 11.50, there is a check-box in the rendering settings that enables Vulkan or Metal; you’ll have to restart for the change to take effect. If the sim crashes really early (e.g. before the main menu) it will revert to OpenGL to keep you from getting “locked out” but random crashes while flying won’t do this. There are also command line overrides to control the driver for people running performance testing scenarios.

I should also mention that we’ve changed what the reflections slider does – it’s now like X-Plane 10 and does not attempt to update the environment cubemaps in real-time. (If you really like flying at 5 fps, you can re-enable this in settings.txt – the underlying tech has not been removed.)

What we have found over the last few years is that users will drag every slider all the way to the right, and then be surprised that the sim has poor performance. While I have in the past dismissively claimed that they are doing it wrong (“don’t put every topping on your pizza”) the inevitable truth is that we are violating an industry UI norm by having the highest possible settings run beyond the scope of the fastest possible computers our users can get their hands on.

Now that we have Vulkan and Metal running, we can make some judgments about what will and won’t be real-time. Full scenery shadows are still available and are helped by the Vulkan and Metal driver; a decent machine should be able to run full autogen + scenery shadows at 30 fps.

The Vulkan and Metal back-ends change their approach to running out of VRAM. With the OpenGL driver, when VRAM is exhausted, textures are moved to system memory, and the application stutters while the driver makes this mad last-minute shuffle.

When running with Metal and Vulkan, X-Plane will reduce the resolution of your textures while you fly, in the background, without stutters, preferring to reduce resolution on textures that are not being used right now. The result should be smoother and generally less disruptive. You can still lower the overall texture resolution to better fit your card’s budget.

This new “on the fly paging” is…well, it’s completely new, and I suspect it will need a lot of debugging during beta. It’s an area where we just need to collect use-case data to better tune the algorithm.

X-Plane 11.50 no longer allows for uncompressed textures – if a DDS is present on disk, we will use it. I have to recommend texture compression in the strongest possible terms; while the artifacts are annoying, when we turn it off, everything looks worse due to the VRAM pressure – uncompressed just isn’t a good way to use our VRAM budget. Pre-compress your textures for best results!

A Detailed View

One of the great things about running with the Vulkan/Metal back-end is that there isn’t any dark matter in the rendering path – everything that takes time takes that time when we say so. This means that when we have a stutter we can take the frame apart and figure out exactly why that happens.

One of the last fire-drills before sending out a build happened last week – with the Vulkan back-end and memory paging working correctly, we would still see these violent stutters flying over downtown LA. As it turned out, the code to delete autogen as you flew away from it was too slow and was causing hitching every time a 3 km AG square had to be torn down.

That code is now rewritten and the results are much smoother. Here’s what winning looks like:

This is my old iMac running at 60 fps – it can’t do that with everything maxed out, but when it does hit 60 fps you’ll note the dead even orange bar at the top – that’s consistent framerate. The bottom of the screen shows a block diagram of some of the tasks that went into the current frame render.

With “weird driver behavior” removed, when something stutters, so far we’ve been able to identify it, e.g. the autogen stutters from last week. Here’s what stutters look like:

The orange spikes you see are the stutters – I induced these by command-tabbing to other apps while flying; when the window manager swaps apps it’s pretty disruptive. That big blank space in the middle of the screen is time spent doing something (probably in the window manager) that we do not yet track.

One last show-and-tell pic: for some reason when running at 45 fps, the sim is actually running at a mix of 60 and 30 fps every other frame, which is not good. We can easily see this with our tools:

Here we can see two frames, a long one on the left and a short one on the right. The difference: “acquire surface” is taking a ton of time. The long frame is stuck waiting for the window manager to give us some VRAM to draw into. With this kind of detail, we can at least understand the bug and try to fix it.

Next Steps

At this point the only thing left to do is fix bugs, some of which are still open even as the private beta goes out. We will move to public beta once we have something where we think the initial experience isn’t a blue screen and tears.

Posted in Development by | 202 Comments

XPlane2Blender v4.0.0-alpha.4

Download from GitHub here!

XPlane2Blender v4.0.0-alpha.4

As always, make backups!

Over the past month people have reported to me all sorts of weird things:

  • OBJ geometry moving with every export
  • Blender Objects moving after every export
  • Animations spiraling out of control
  • Even simple renaming Blender Objects and restructuring of the Blender hierarchy might trigger a bad export.

And the only cure was to apply all transformations or restart Blender!

Thanks to your bug reports and e-mails I was able to track down the source: all these problems were facets of the same bug!

To make a long story short: I have a fix for XPlane2Blender, but other Blender exporters and users may not be so lucky. I’ve filed a bug with Blender itself, but that is nothing to worry about for X-Plane artists.

Please continue to send me your bug reports! I do read every one of them, even if I don’t get back to you right away.

Posted in Aircraft & Modeling, News, Scenery, Tools by | Comments Off on XPlane2Blender v4.0.0-alpha.4

XPlane2Blender v4.0.0-alpha.3, aka Exportable Collections!

Visit the Github release page to download!

XPlane2Blender for 2.80 has crossed a new milestone! Exportable Collections are in! They work just like root objects (which we are now calling Exportable Objects), the settings for them are just where Layers Mode settings used to be, and Layers Mode projects keep their OBJ settings automatically. More details down below.

As always, make backups! This is still an alpha and how collections work could change!

New Name For Root Objects

As mentioned, Root Objects – the concept – has been renamed to “Exportable Objects”. Everything is the same, it is only the name that is changing in the UI and the docs. This was done to let people know there really isn’t much difference between using the new Collections and the old Root Object system. Half the name is the same! Otherwise it sounds like “Root Objects vs Exportable Collections” are as different as how it used to be with “Root Objects Mode” vs “Layers Mode”. From here on I will use the new names – Exportable Collections and Exportable Objects.

New Features

Exportable Collections

We’re very excited about this: Collections are in! Many thanks to the people who talked about it on #450. I think the feature will be able to do all or most of what you wanted. Here’s how to use them.

  1. When you open a 2.79 file in 2.80, the contents of Layer 1 will be put into a collection called “Layer 1_{the scene’s name}”, the contents of Layer 2 will be put into a collection called “Layer 2_{the scene’s name}” and etc.
  2. Blender has new and exciting ways of using Collections and the 3D View. If Show is on but you can’t see the contents of a collection, try right clicking on it in the outliner, go to Visibility and ensure Render and Viewport are enabled.

I highly recommend reading
New Outliner Controls
Collections Overview. Complex instancing is planned but not supported: #495 and #452.
View Layers

  1. You’ll find the exact same OBJ settings in the exact same layout where Layers Mode used to put them.
  2. The updater will have copied the previous settings if the layer had some object in it or the OBJ settings was changed in some way.
  3. Collections can be marked as “Exportable” or not. Non-Exportable Collections can be used as a way to organize your project, similar to using Empties. Unlike empties
    • An important terminology note: Datablock Objects (Meshes, Lights, Armatures, Empties) can have 1 Object parent and many Object children. Collections can have 1 Collection parent and many Collection children. This means a datablock objects can have 1 parent and be in multiple collections. When an object is linked to a collection, it is in that collection and all that collection’s chain of parents.
    • Collections do not have a location or rotation.
    • A mesh, armature, or other objects can have a blender parent and be in multiple collections. A great opportunity for re-use that XPlane2Blender supports. Also key for another new feature called “Split Animations”.
    • Every object inside the exportable collection and its chain of child collections is collected and exported together as one OBJ. Any children of those objects found outside the exportable collection are ignored, just like how Layers Mode used to work.
    • Each object is included once. Attempting to make duplicates by putting an object in multiple sub-collections of the exportable collection won’t work. In this example Cube only appears once in the OBJ.
  4. What is exported only depends on what objects are in what collection and the value of that collection’s Exportable Collection checkbox. Viewport visibility doesn’t matter.
  5. Everything inside the exportable collection and any of its child collections will be exported. If any object has a child that is outside of the exportable collection, that child and all its children will be silently ignored.
  6. There is no more “Layers Mode vs Root Object” divide, and projects can use Exportable Collections and Exportable Objects at the same time (just not nested).

Here is a picture showing a side by side comparison of the relevant UI in 2.79 and 2.80, using the BD-5J Microjet. After opening the file I hit export and it just worked!

image

In this picture I’ve unchecked Exportable Collection on Layer 2_Scene. Now it won’t export.

Split Animations

Split Animations is one of our most exciting new features – something never before possible in XPlane2Blender! See the guide in the manual, click here to find more.

Other Bug Fixes

  • #493 Unit test system slowly healing and getting back to usable.

Caveats

Things Move On Every Export

There appears to be a Blender bug in 2.80 where sometimes after every export (of any exporter) the objects drift apart. It is unknown why this happens but there are two ways to fix it.

  1. Restart Blender and try again and hope it works
  2. Select all objects and bones and use Apply > Transformations > Scale. And if that doesn’t work, Apply > Transformations > All Transformations. These are drastic, but, if it works, please e-mail and help confirm this bug that we can report to Blender.

Updater Changes

The updater has had some changes and not all of them tested 100 times over like I normally like. I don’t think it will be a problem however. Please e-mail me if you have trouble. If you have a multi-scene project, double check your updated OBJ settings and tell me if you see an automatically made collections with names like “Collection.001”. They should have been renamed.

For those of you patiently waiting for an e-mail back from me about a problem you’ve had with your .blend file, thank you for your patience. I’m going to try to respond to all of them this week!

Posted in Aircraft & Modeling, Scenery, Tools by | 1 Comment

X-Plane 11.41r1

The initial beta for X-Plane 11.41 is now available for LR customers, and we expect Steam to be available within 24 hours or so.

This is a small patch that focuses on fixing a security vulnerability. Small bug fixes are also included, and you can see the full release notes here.

Posted in News by | 67 Comments

X-Plane 11.40 is Final

X-Plane 11.40 is now final! You will be prompted to update to X-Plane 11.40 when you start X-Plane; Steam users will receive the update automatically via Steam. Here’s what’s next in the pipeline:

We are working on a bug-fix update (11.41) to catch one or two bugs that didn’t make the RC, as well as the inevitable bug that will be reported after go final. I expect to cut an 11.41 release candidate some time next week, and it should be a pretty quick release.

In the meantime, we are pushing hard to get Vulkan/Metal ready so that we can do an X-Plane 11.50 beta. We may start private testing of Vulkan and Metal before 11.41 is done, depending on what gets fixed first.

The limiting factor on getting to a public Vulkan/Metal beta will be bug fixing – it doesn’t make sense to go public beta with known bugs that will be reported over and over and make the beta unusable. There are also two remaining areas of development we need to close up:

Load Times

Vulkan and Metal fight stutters by pre-loading everything they need to render the frame – this means whatever we need to draw, it’s ready. But this also means X-Plane needs to prepare every shader it might need to render a given aircraft and scenery pack.

Because X-Plane’s shaders are so flexible, this can mean tens of thousands of shaders, and that hurts load time. We have new code that tries to analyze exactly what we’ll need for a given frame. For example, if an object comes from a DSF, we never need an “interior” shader for it because the scenery object can never be inside an aircraft. By carefully applying this kind of logic, we can cut down the number of preloaded shaders and speed up load times.

Sharing Memory

With Vulkan and Metal, X-Plane manages its own use of VRAM – this is a totally new capability that we have had to code from scratch for the Vulkan/Metal port. We watch our total memory usage and reduce texture resolution dynamically (hopefully on less important and less used textures) to stay within budget.

While this code is working, it hits a wrinkle on Windows:

  • Third party add-ons that chew up VRAM via their OpenGL VRAM usage can use up part of our budget. For example, if you change to an aircraft with a plugin that allocates its own textures to render to, we can’t use that VRAM anymore.
  • The Vulkan version of X-Plane allocates VRAM in big blocks for efficiency. If we reduce our texture res to make room for the plugin, we might end up with a bunch of big pre-allocated blocks (that are mostly unused) inside Vulkan – the memory still isn’t usable by the OpenGL plugin.

Sidney is working on compacting our textures so that some of the big blocks can be given back to OpenGL – yet more new memory management code for Vulkan.

There Be Dragons

Once we have the number of shaders cut down and memory compacting working, we’ll begin heavier testing of the Vulkan/Metal build – I can’t say how close to public beta we are because we haven’t gotten the initial wave of bugs from this testing.

Posted in News by | 37 Comments

X-Plane 11.40r2

Release Candidate 2 is now available for LR customers, and we expect Steam to be available within 24 hours or so.

This is a very small patch with a few minor fixes. At the risk of sounding like a broken record: if you haven’t tried your add on aircraft in 11.40 yet…do it now! The reports have been pretty quiet the last couple weeks, and nobody enjoys a last minute surprise bug.

Release notes here.

Posted in News by | 16 Comments

X-Plane 11.40 RC 1

Release Candidate 1 is now available for LR customers, and we expect Steam to be available within 24 hours or so.

We are moving towards the end of the beta period, so we’re minimizing the number of code and flight model changes at this point. If you haven’t tried your add on aircraft in 11.40 yet…the time is now!

This update contains mostly tweaks to AI & ATC interaction to improve landings, replay sound improvements, and includes a few new datarefs requested by devs.

Release notes here.

Posted in News by | 26 Comments