It has been almost a month since my last post on Vulkan and the state of development; a lot has happened since then, both for Laminar Research and the world.
COVID-19
First, at this point, no one working for Laminar Research has COVID-19. We have people working on X-Plane in at least six countries, including northern Italy. The team news on Slack as of Tuesday was that everyone is in some degree of lockdown and/or voluntary social distancing, varying by country. Schools in the US are temporarily closed, so both Chris and I have the kids at home.
Being sequestered at home has not affected our internal pace of development because our entire team is distributed and works at home; we have no office to close. This work-at-home pattern started almost twenty years ago when Austin had people like me working on part-time contracts to modify specific parts of the sim – it never made sense to relocate people to South Carolina and open up an office. Being remote also lets us hire people anywhere in the world who have the unique blend of skills we’re looking for.
At this point we haven’t seen any operational issues either; mostly running our business requires that our devs have internet and electricity and that the data centers we use for our cloud servers remain open and operational.
So in summary, we are very fortunate that the new coronavirus is not affecting our development and operations; other industries are paying a high price to stop the spread early. We have several things we are working on right now, and this spring is a critical time in our development schedule.
Vulkan and Metal
Update: I have gone in and changed Vulkan to Vulkan/Metal in a bunch of places – Mac users were confused as to whether we had silently dropped support for Metal at the last minute – we have not!
We are posting developer preview nine to our private testers today; hopefully this will be the last private build before a public beta candidate. We acknowledge we’re reaching public beta much later than we had hoped/wished/anticipated, but we are now aiming for a public Vulkan/Metal beta by the end of March. If you are already rolling your eyes at a public beta date that is less than two weeks away, I don’t blame you; having been this late, it’s on us to post the beta to show progress. With that in mind, I am going to describe what we’ve been doing for the last three months and why it has taken so long to get here.
The Vulkan/Metal public beta is several months later than we had anticipated because of “scope growth” – that’s manager nerd speak for “we added more stuff to it than we originally planned.” Scope growth (adding more features/code/tricks than you originally planned) is one of the big ways that projects miss original deadlines, so the big question is: what did you add and why did you add it?
The first thing we added was better handling of plugin drawing. The rewritten plugin compatibility layer provides better drawing compatibility for more plugins, including weather plugins on Windows. We went with the rewrite mostly because of the level of bugginess we saw with third party add-ons in the early developer previews.
Plugin drawing was definitely a case where we learned how to do a better job from the first version of the feature (plugin compatibility that went into the first private beta); if we had received that second-generation design via time machine we probably could have shipped faster. Adding weather support was pure feature creep–a new thing we didn’t plan on, but something that we thought was worth extra schedule time.
The second thing we added was better handling of texture paging. Once again, this was a feature where we had to rewrite the feature (multiple times in fact!) based on the feedback we got from our testers to really come up with something solid.
Our first generation texture paging around the first private developer preview was very simple: most stuff lived in VRAM, with a little bit of code to move unused stuff out of VRAM. It was a minimalist strategy that let us develop the rest of the sim and worked great on high-end cards. It was clear from day one that it wouldn’t be good enough for public beta.
Our second generation strategy added automatic movement of texture res up and down with VRAM pressure and code to page out textures that weren’t being used. This shipped about half way through the private beta, and was better, but suffered from one fatal flaw: as you turn your head, the stuff behind you isn’t being used. Under heavy memory pressure users would constantly turn their head and see blurry textures that would then “res up”. The results were distracting and unacceptable quality-wise.
We now have finished our third generation strategy: besides automatic texture res control based on VRAM pressure, we now set the relative resolution of non-orthophoto textures by distance to the aircraft. A background task on another core “crawls” the scenery near your aircraft and reevaluates the texture res of nearby scenery constantly, effectively transferring VRAM to where it is needed most. This process is completely transparent; authors do not need to modify scenery in any way for it to work, and since it runs on another core (as does the paging), it does not affect framerate.
In these pictures, you can see the new pager at work. I have parked my aircraft on the ramp at SeaTac and moved the camera across the city, so that the distant autogen (from the pilot’s perspective) is now close and the airport is in the background.
Texture Paging – Camera moved to reveal “far away” areas
In this first picture, green represents full-res textures, with the next levels down being yellow, then magenta. You can see some nearby autogen down one level and far-off autogen down two levels; the loss of res is put far from the user.
Why is there so much green (full res) near the magenta autogen? Texture reuse. If a texture is used near the user and far from the user, it gets high res because it might be seen up-close.
Texture Paging – VRAM artificially reduced to simulate low-end GPU
In this next picture, I artifically lowered my machine from 4 to 1 GB of VRAM using a developer tool. Now you can see magneta autogen close to the airport, yellow even closer, and some really dark green (the next level down) where we had magenta. In other words, everything got a little bit lower res because we were tight on VRAM, but again nearby sceney is prioritized.
It looks like this third strategy is a keeper – it combines the careful management of VRAM to use it where it is most needed with a stable heuristic that isn’t constantly changing, which avoids chaos and unreliable behavior.
So at this point we are just fixing bugs. This is a big step forward for the private beta because past betas were dominated by coding new things, which in turn creates new bugs. At this point we just have to kill bugs off to the point where the build is high enough quality for a public beta.
I cannot promise a particular build number or date will be the public beta full stop, because I do not know yet what other bugs we may find. But I can say that we are in the part of bug fixing where things are getting locked down.
Vulkan is Not the Only Iron In the Fire
A flight simulator is more than its rendering engine (or at least, people tell me that). While we are pushing as hard as we can to get Vulkan and Metal to public beta, we also have several other irons in the fire. This includes new features for mobile, next-gen features for desktop in several feature areas, and even a few extra non-Vulkan/Metal features that have snuck into 11.50. We’ll post more about the mobile and 11.50 features as they become available; the rest is still in stealth mode. Often when the dev blog is quiet, it means we’re working on new things and putting features in the bank; this is the case right now.
One thing I have learned during my time here in the matrix is that users often find new bugs to be significantly more annoying than existing ones; trading an existing bug for a new one is often seen as a big step backward, even if the new bug isn’t that bad. I think this is just human nature – we get used to things – it’s what humans do – and this takes the sting off of some bugs that have been around a while.
I bring this up to put in context the problem of texture paging, blurry textures, and VRAM management in 11.50. To understand what makes VRAM paging hard, you have to understand how the OpenGL driver solved the problem, and ask yourself: what was the old bug, what is the new bug?
What Is Texture Paging?
Texture paging is when textures are moved between the memory on your graphics card (VRAM) and the memory in your computer (“system memory”). At any given time, X-Plane may have more textures loaded than you can fit on your card, but it doesn’t need them all at once. As you fly, different textures are needed, and the textures have to be moved around.
If this sounds a lot like moving data from RAM to your hard drive (virtual memory), it is. When I get the question “how can X-Plane be using 2.5 GB of texture memory when I only have a 2 GB card” the answer is…texture paging.
How Did the OpenGL Driver Do It?
In an OpenGL based app, paging textures to VRAM and back is entirely up to the OpenGL driver. The algorithm goes something like this:
Start drawing.
Realize something you need is in system memory.
Move something you haven’t used in a while out of VRAM.
Move the thing you need into VRAM.
Continue drawing.
The OpenGL driver pages during drawing when we need something new, and the result is a stutter – the frame takes longer because it cannot be completed until the texture is moved to VRAM.
Now NVidia, AMD, and Apple have spent a lot of time and effort to make this work as well as they possibly can, and I’m definitely selling short the complexity and sophistication of the drivers as they try to guess really well what should be moved out of VRAM. Still, two key facts will be true:
If we need to page stuff into VRAM to draw, and we don’t figure that out until right when we need it, there’s going to be a stutter. Maybe a really short one you barely notice, maybe a really bad one, it’s a question of “how much”, not “if”.
What you see on screen always looks fantastic – never blurry – because the driver won’t cut resolution to save VRAM.
So…the old bug in this case is stutters due to VRAM containing the wrong stuff.
How Does X-Plane 11.50 Do It?
With Metal and Vulkan, it’s up to us, the app, to decide what goes in VRAM and what doesn’t, and to schedule the movement of data between VRAM and system memory. Our strategy is:
Keep an eye on how much memory is free.
If it looks like it’s getting too close to full, shrink textures to a smaller size. Prefer to shrink textures you’re not looking at. Do the shrinking in the background.
If it looks like VRAM has empty space free, grow textures, preferring ones you are looking at now. Do the growing in the background.
If we have to load new textures in the background (e.g. new DSFs as you fly), wait until old textures can be shrunk.
We have a huge advantage over the drivers in this game: we know the textures we might need in the future, long before we actually need them. We can also shrink textures. This means we don’t have to stop and shuffle things around, and that means we can avoid stutters. So we can fix the old bug: stutters while you fly.
(To be clear: this is not the only source of in-flight stutters while you fly – OpenGL and 11.41 are full of them.)
The down-side is: if VRAM gets tight, we’ve made some of the textures lower resolution. This is a new behavior (blurry textures), so my fear is: users aren’t going to like it.
In particular, the OpenGL driver uses the exact optimal set of textures for every single frame – at the cost of stuttering. We may not have things shrunk and grown in exactly the best way while you fly, so for the same VRAM and no stutters, we may not be as visually sharp.
What About Plugins
Add-ons that use OpenGL (e.g. aircraft with custom glass displays) use VRAM too – this is one reason why we can’t pack 100% of VRAM with our textures; we need to have enough free that if a plugin needs VRAM, it won’t crash us.
My expectation is that fighting blurry textures and crashes due to running out of VRAM is going to be a long, iterative process and one of the main reasons to have beta testing. It’s one thing to have an algorithm that works good in the lab or on paper – it’s another to have thousands of hours of test time in real-world conditions on user machines with lots of add-ons and anarchy.
So when 11.50 is public beta and you see your blurry texture, don’t panic. VRAM management is a very new part of X-Plane, and it’s a very important part, and it’s something we will iterate on over time.
(Finally, I want to mention that while the port to Vulkan/Metal has been a joint venture that Sidney and I have worked on together, there are certain parts of it where Sidney basically did 100% of the work, and my only contribution was to complain about bugs. The shader compiler, the OpenGL plugin bridge, and the VRAM pager are all big lifts that Sidney has done on his own that make the Vulkan/Metal port possible.)
As I announced in my previous post, X-Plane 11.50, featuring native Vulkan and Metal driver support, is in a private beta made up of people in our dev-rel program (E.g. third party developers). We’ve gotten some really good bug reports from them – the reports are good, not the bugs – and based on that, it’s clear that 11.50 will not be ready to go public this year (meaning tomorrow).
This is disappointing to us – we burned candle at both ends to try to get all the way to public beta this year, but at this point the build just isn’t ready yet.
I’ll describe in detail below what the one bug is that is really holding back public beta, and I’ll try to do a general FAQ about Vulkan and Metal tomorrow; based on the 150+ comments from the last post, I think it’s clear that we have a lot of basic info to get out there, some of which has only been shared with third party developers until very recently.
Please don’t bombard me with “can I please be in the private beta” (or even “you’re a giant pile of moose poop for not letting me into the private beta”). Here’s the thing about the private beta: our goal is to kill off the bugs so everyone can use the beta, and to kill them off as rapidly as possible.
At this point the dozen or so third party developers who are banging on the beta are generating bug reports faster than Sidney and I can fix them – if we add more people to the beta, our bug fix rate will slow down as we have to spend more time triaging the reports coming in. So as long as we’re fixing the worst bugs productively, more people in the private beta means everyone waits longer to get to public beta. I think the best thing here is for Sidney and I to just try to fix things ASAP.
(I think there’s also a win to getting this into the hands of third party developers – in some cases we’ve seen add-ons that accidentally use the wrong APIs for processing and are thus incompatible with Vulkan when they don’t need to be – developers can get a head start on updating this code.) If you are a third party developer, we can get you into the beta program – I don’t want to favor some third party developers over others.
Blurry Textures
A short summary of performance feedback from our internal beta might go like this: smoothness and FPS are good, VRAM management is not.
Most of the crashes while flying we have seen look like the sim runs out of VRAM and isn’t able to recover, and for users with smaller GPUs (e.g. 4 GB of VRAM) X-Plane gets into a tight VRAM situation and solves it by making everything on screen low res and blurry.
The VRAM management path, which is entirely new code written for the Vulkan/Metal back-ends, is…well, it’s entirely new, and it’s going to need a bunch of testing, debugging and iteration to get solid. On an 8 GB card, VRAM is so plentiful that X-Plane’s somewhat naive VRAM allocation scheme works fine. On a 4 GB card, we’re just a tiny bit short (a few hundred MB) but the result is an alarming loss of texture resolution.
This is something we can make better! It is under our control and is thus being worked on now. (By Sidney that is – I’m writing this blog post while he gets the real work done here.) But until we have better VRAM management, it’s too soon to go public. If we did, we would just be inundated with hundreds of “my textures are blurry” reports. It’s a bug too noisy to ship with.
Why Wasn’t This a Problem on OpenGL?
One might ask: why don’t you just do whatever OpenGL did to manage VRAM? The problem is: OpenGL’s solution is to stutter.
The OpenGL driver swaps textures between VRAM and system memory based on what you really need to draw now. If you are looking away from Seattle, the space needle’s texture can live in system memory, but you might need mountain textures for Rainier.
We try to do that too – the big difference is: OpenGL will stop rendering while it moves the textures around, and we will not. The result is that OpenGL always shows you a perfect image at the texture resolution you picked – no blurred images. But you might have stutters! You can see this if you’re on the margins of VRAM by flying and then changing views or circling the camera – if your FPS go “chunk chunk chunk VROOOM” you’re eating some stutters while textures move around, then you go fast again.
The OpenGL way is a solution that we never thought was acceptable – our goal is the best image quality without sacrificing smoothness. It’s going to take a little more time to iron out low VRAM situations, but once we get there I think it will be worth it.
Last week we sent out our first build of the Vulkan/Metal ready X-Plane (which will be version 11.50) to an external sight, and last night we sent out the first private beta.
Our plan is to move to an open public beta soon once we have some feedback on how well the build is working. When that is will depend on the bug reports; I’m crossing my fingers.
How It Works
In X-Plane 11.50, there is a check-box in the rendering settings that enables Vulkan or Metal; you’ll have to restart for the change to take effect. If the sim crashes really early (e.g. before the main menu) it will revert to OpenGL to keep you from getting “locked out” but random crashes while flying won’t do this. There are also command line overrides to control the driver for people running performance testing scenarios.
I should also mention that we’ve changed what the reflections slider does – it’s now like X-Plane 10 and does not attempt to update the environment cubemaps in real-time. (If you really like flying at 5 fps, you can re-enable this in settings.txt – the underlying tech has not been removed.)
What we have found over the last few years is that users will drag every slider all the way to the right, and then be surprised that the sim has poor performance. While I have in the past dismissively claimed that they are doing it wrong (“don’t put every topping on your pizza”) the inevitable truth is that we are violating an industry UI norm by having the highest possible settings run beyond the scope of the fastest possible computers our users can get their hands on.
Now that we have Vulkan and Metal running, we can make some judgments about what will and won’t be real-time. Full scenery shadows are still available and are helped by the Vulkan and Metal driver; a decent machine should be able to run full autogen + scenery shadows at 30 fps.
The Vulkan and Metal back-ends change their approach to running out of VRAM. With the OpenGL driver, when VRAM is exhausted, textures are moved to system memory, and the application stutters while the driver makes this mad last-minute shuffle.
When running with Metal and Vulkan, X-Plane will reduce the resolution of your textures while you fly, in the background, without stutters, preferring to reduce resolution on textures that are not being used right now. The result should be smoother and generally less disruptive. You can still lower the overall texture resolution to better fit your card’s budget.
This new “on the fly paging” is…well, it’s completely new, and I suspect it will need a lot of debugging during beta. It’s an area where we just need to collect use-case data to better tune the algorithm.
X-Plane 11.50 no longer allows for uncompressed textures – if a DDS is present on disk, we will use it. I have to recommend texture compression in the strongest possible terms; while the artifacts are annoying, when we turn it off, everything looks worse due to the VRAM pressure – uncompressed just isn’t a good way to use our VRAM budget. Pre-compress your textures for best results!
A Detailed View
One of the great things about running with the Vulkan/Metal back-end is that there isn’t any dark matter in the rendering path – everything that takes time takes that time when we say so. This means that when we have a stutter we can take the frame apart and figure out exactly why that happens.
One of the last fire-drills before sending out a build happened last week – with the Vulkan back-end and memory paging working correctly, we would still see these violent stutters flying over downtown LA. As it turned out, the code to delete autogen as you flew away from it was too slow and was causing hitching every time a 3 km AG square had to be torn down.
That code is now rewritten and the results are much smoother. Here’s what winning looks like:
This is my old iMac running at 60 fps – it can’t do that with everything maxed out, but when it does hit 60 fps you’ll note the dead even orange bar at the top – that’s consistent framerate. The bottom of the screen shows a block diagram of some of the tasks that went into the current frame render.
With “weird driver behavior” removed, when something stutters, so far we’ve been able to identify it, e.g. the autogen stutters from last week. Here’s what stutters look like:
The orange spikes you see are the stutters – I induced these by command-tabbing to other apps while flying; when the window manager swaps apps it’s pretty disruptive. That big blank space in the middle of the screen is time spent doing something (probably in the window manager) that we do not yet track.
One last show-and-tell pic: for some reason when running at 45 fps, the sim is actually running at a mix of 60 and 30 fps every other frame, which is not good. We can easily see this with our tools:
Here we can see two frames, a long one on the left and a short one on the right. The difference: “acquire surface” is taking a ton of time. The long frame is stuck waiting for the window manager to give us some VRAM to draw into. With this kind of detail, we can at least understand the bug and try to fix it.
Next Steps
At this point the only thing left to do is fix bugs, some of which are still open even as the private beta goes out. We will move to public beta once we have something where we think the initial experience isn’t a blue screen and tears.
Here’s a feature that went into X-Plane 11.35 that will be really exciting to a very small number of scenery authors: in X-Plane 11.35, line files (.lin) can have custom painted end-caps on each end, and the texture repeats can be aligned to a grid.
Scenery authors who have gotten deep into X-Plane’s scenery system will already know from that first paragraph why this is useful, but for normal people who have seen the sun in the last 30 days, .lin files are the art files that define the look of thin linear features. X-Plane uses them for the painted yellow and white lines in the airport environment, but a developer can use them for anything linear.
Line files repeat, tile, and can follow the ground in a bezier curved path, which makes them great for curved taxiway lines. Their achilles heal up to now has been that when the line ends, the line just … stops. This makes them inappropriate for really thick uses, where that hard “cut” at the end of the line would be really noticeable and ugly.
In X-Plane 11.35 you can provide a start and end cap for each line definition. Like the line itself, it can be mulit-layered if desired. So, for example, you could use it to make dirt paths between buildings in a rural airport – where the path ends you can have a “soft” ending to the path and not have to worry about tucking the line under another scenery element to hide the cut.
We’re not using these in the default art yet, but the code is done, and I have updated the .lin file format specification with the new syntax.
X-Plane 11.35 beta 4 is now live – run our updater and check “get betas”. Release notes here.
Steam users: we’ll release this tomorrow if there isn’t a sign of a huge fire overnight – it’s already uploaded to the servers.
Add-on Developers: if you haven’t run your add-on in 11.35, please put it through its paces now. Everything that might be risky or weird is already in and we’re just killing off bugs now. If your add-on has a compatibility problem, we need to know now if we’re going to fix the problem during beta. While we’ve tried to make everything “just work” with old add-ons, there’s always the chance that your add-on does something special we didn’t think of. Please test now!
Here’s our keynote talk from FlightSimExpo 2019 last week!
FSExpo 2019
Evans video team really did a fantastic job with the feed this year – everything was live-streamed in great quality, and the talk has a direct feed to the slides and good quality audio. Totally what we were hoping for!
We’d like to point your attention especially to the new “2D” or “3D” meta-tag for export targets of 11.30 or higher. This tag tells the X-Plane GUI to list the airport as “2D” or “3D”.
Before X-Plane 11.35, the information in the “Features” column in the X-Plane airport selection menu came from any scenery in a user’s Custom Scenery directory, rather than being an entry in the system-wide apt.dat in the Resources/default scenery/default apt dat/ directory. The assumption was any addon airport would always be 3D. But starting with X-Plane 11.33, all global airports, even those with 2D-only layouts, were now included in the Global Airports –which made the X-Plane GUI effectively list every airport as “3D”.
At export, WED 2.1 will analyze the scenery for 3D content and look for the existence of the meta-tag “GUI label”. If the export target is 11.30 or higher it will warn if the tag is missing or improperly set, ultimately leaving it up to the designer how they want the airport to be listed in X-Plane. If the target is “Gateway”, it will forcibly create or update this meta-tag to always be set correctly according to the actual scenery exported.
In order to fix the 2D/3D display in X-Plane 11.35 and later for any scenery not submitted via the Gateway, authors will need to manually add the tag via “Airport->Add Meta Data” and then re-exported to a target of X-Plane 11.30 or higher.
Since my last Vulkan update, we now have the full sim running natively with Vulkan and Metal! There is still a pretty big list of random things turned off or bypassed to make this happen, but we can fly in the cockpit and use the sim.
Here’s some stuff that is now working:
Using regular and HDR rendering, and SSAO on metal.
Flying with Vulkan on AMD, NVidia, and Intel drivers.
Hardware stereo rendering on OS X – this never existed before Metal because the Mac GL drivers didn’t support it. Hardware stereo rendering is necessary for VR support.
MSAA on Metal – with the restructuring of our code, you’ll be able to change MSAA settings without restarting. I don’t know if this code works on Vulkan right now; it might.
Here’s some stuff that does not work yet:
Plugins are bypassed right now. We have not yet written the plugin-OpenGL-interop layer.
VR only works when using OpenGL as the driver; we need to write some new VR code to pass Metal and Vulkan frames directly to the OVR and Rift APIs.
Screenshots/Movie capture are a work in progress – that’s what I’ve been working on this week.
We still have a number of visual bugs, so screenshots are an important feature so we can run our automated test system. The test system takes hundreds of screenshots of the sim in many configurations and compares them to 11.30 to catch bugs introduced by the new Vulkan and Metal back-ends. Clearly I’ll have to fix my color problems first.
One fun aspect of this port: Metal and Vulkan copy Direct3D’s convention where the viewport Y axis points down and not up. This resulted in a whole series of weird “it’s upside-down again” bugs, most of which have been fixed.
The world is drawn correctly but the atmosphere around it is upside down.
(That airplane might look silly, but at least all of the image is consistently upside down. That’s because the only remaining upside down code is the movie/screenshot capture code itself.)
We should have some performance numbers to post next week; right now our primary focus is fixing bugs, particularly bugs that bring the whole machine down.
Here are a few more pics of things that have gone wrong during development.
X-Plane 11.33b1 is an incremental update that includes updated Gateway airports and translations, crash fixes and diagnostics, minor UI improvements, and bug fixes such as:
XPD-9441 Weight, balance and fuel- Total weight (lbs) does not include weapons.
XPD-9683 Fix dupe banks in FMOD crashing the sim.
XPD-9882 Fixed KOAK nav data.
XPD-9919 Lightning appears in cockpit in VR.
XPD-9991 Crashes on older Intel GPUs.
XPD-9992 Default FMC, Holding Pattern will only be flown once.
XPD-10003 Landing lights has no effect on battery amperage draw.
Just a reminder that the sim will not prompt you to install this beta–you will need to launch the installer manually and opt into betas to get this version.
We are putting more effort into cleaning up frequent crashes. We recently upgraded our back end crash reporting to a fancy paid service that we hope will allow us to gain a lot more insight into what is going wrong. This beta includes additional crash logging for that purpose.
Keeping in the crash-fixing vein, we believe we also fixed two notable crashes in this beta: issues with Intel GPUs and with duplicate FMOD sound banks. We heard from a lot of Intel users about these crashes, so Sidney took a look and was able to find a fix for it. The FMOD crash has been a round for a while and was caused by duplicating an aircraft folder that included FMOD. When an FMOD bank had the same contents on disk at a different file path, FMOD wouldn’t load it. We now handle this case gracefully.
Update: Steam users can get 11.33b1 by selecting the public beta under application properties.
