The next X-Plane update will focus primarily on flight-model and systems, plus external-visual networking and some ATC features. While that update is in beta, we can work in parallel on real weather and graphics.
But there are two graphics bugs we already have fixed in-house which should relieve some 12.06/12.07 pain:
Running out of memory mid-frame. Turns out if you got into a fairly tight situation VRAM wise (and we try to do that to max out the texture res you can have) then X-Plane might run out of memory trying to draw trees and…melt down like a toddler who can’t have any more candy.
We have an interim fix: allocate memory statically so we always have it. In a future update we’ll reuse memory from other parts of rendering to be more efficient.
Popping out a window causes a big slow-down. When the arrangements of windows changes, we might need to allocate more VRAM for rendering. This is not a fast process – we have to halt all rendering, throw out the old memory, compact things, allocate new memory, and if a DSF is loading while this happens, the DSF loader is using up memory as we are trying to reallocate the windows, which can mean compacting memory again, paging down textures…you get the idea.
The fix is pretty simple: don’t do this if the popped out window doesn’t require more VRAM. Most of the time, this is the case, so this is an easy fix for a silly bug.
Integration work for the next update is going on now and I’m hoping it will be done next week. More details soon!
Editor’s note: what follows is very, very, very, very, very silly.
Last week we shipped X-Plane 12.06; since then we have found a few straggler bugs; like typical lARRge patches from the days of X-Plane 11, a few bugs escaped us until after final, including some crashes we could see in the auto reportARR.
While it’s not great to be shipping a “hot patch” to our release, it is pretty fantastic to announce X-Plane 12.07 ARRsea 1 on International Talk Like a Pirate Day (the 19th of septembARR). The rest of this blog post only gets worse; you’ve been wARRned.
12.08 Is The New 12.07
Internally, nobody is happy about this, but the hot patch bumped all of our version numbers. So the new road map looks something like this:
12.07: hot patch of 12.06.
12.08: coming soon. Flight model and systems, ATC and Networking
12.09: more graphics and real weathARR fixes.
12.08 (né 12.07) is almost completely integrated and should be ready for private testing as soon as 12.07 is settled.
Why Wasn’t I Notified
Since 12.07 is a new version, you won’t be auto-updated to upgrade from 12.06 while 12.07 is in testing; run the installer, update with “get betas” checked and you can get 12.07.
12.06 shipped with more sensitive internal detection for numeric problems. This is how we run the sim internally all the time – the goal is to find and squash bugs fast.
Unfortuntately there are sections of X-Plane’s simulation that are:
Not present in an aircraft (e.g. the real aircraft doesn’t have X)
Not expensive to run, CPU-wise
Not visible to the user (since the aircraft has e.g. no gauges to show these systems.
As it turns out, these systems are often happily running away in the background producing absolutely bonkers values; with sensitive numeric checking, the sim can crash due to problems in a system that nobody cares about.
For 12.07, I solved the problem by muting the alarm. My expectation is that we’ll have sensitive numerics in most betas, turn them off in the releases, and work through the systems code over time. (The priority on this isn’t super high because the systems aren’t consuming framerate – if they were we’d see them in our profiling.)
The other area where we saw increased crashes was with TCAS plugins. 12.07r1 has better logging and shouldn’t crash as much – the goal here is to not brick third party add-ons.
Other Bug Fixes
Multi-monitorARR: data output was not working in full screen configurations, fixed now, and manipulators work for multi-monitor systems.
OpenXARR: HP Reverb fixed, and we finally figured out why the white board would sometimes disappear. (It wasn’t gone, it was just 20 km from the hangARR.)
X-Plane does not have a built-in way to remove the bezels from the pop-out avionics. To solve this, some of our user and some add-ons edit the ARRtwork inside X-Plane’s “resources” folder, setting the bezels to clear.
X-Plane 12.06 introduced new bezel variants for the G1000 to cover all of the real-world panel button configurations. This fixed the bug “the real aircraft does not have these buttons, but when you pop the panel out, the buttons appear and do nothing.”
A side effect of this is: those aircraft using the new bezels use new art files, so the hacked up no-bezel art files on longer work.
This is not a bug for us to fix; if your add-on works by modifying the internals of X-Plane, we cannot guarantee that it will keep working even when we change the sim. (The only way to make that work would be to never change the sim.) You can work around this by modifying the new bezel files.
In the future, our plan for this is to provide a real way to hide bezels in the sim as a built-in part of the UI, which should make hacking unnecessary.
What Comes Next
Hopefully 12.07r1 will be “one and done”; if so, we’ll move on to private testing of 12.08 almost immediately.
If you are a third pARRty and your add-on has problems with 12.06, please tell us four weeks agoas soon as possible!
X-Plane 12.04b3 ships with a new feature called “Zink”, which we hope is going to alleviate a lot of the long standing issues that many users, especially with AMD GPUs have suffered from. This hopefully provides a work around for both flickering plugin drawing, eg. EFIS screens that either flicker or are missing altogether, but also potential improvements in performance by decreasing driver overhead.
If this sounds like you, you are welcome to give this a try right now by upgrading to 12.04b3, going into the Graphics settings and checking the “Use Zink plugin bridge” check box. Restart X-Plane and if everything goes to plan, X-Plane should look exactly the way you are used to but most importantly your plugins should look right and your performance might be better too. Because Zink is built on top of Vulkan, it’s only available to Windows and Linux users and is not applicable to macOS and Metal.
If you only take one thing away from this blog post, I hope it’s this image showing the difference between native Vulkan/OpenGL interop as well as Zink interop gives you an idea of why we are excited about this:
Now that I hopefully have your attention, let me elaborate how we got here and what this means for both users and developers. First of, Zink is actually a graphics driver that sits between plugins and X-Plane and translates plugin OpenGL rendering into native Vulkan commands that get executed by the same Vulkan device that X-Plane is using for rendering. Of course, X-Plane itself no longer uses OpenGL, but it’s is still very important for plugin development. A long time ago, when X-Plane was still OpenGL based and the plugin SDK was first made, it seemed like an absolute no-brainer to just expose the X-Plane OpenGL context to plugins directly and let them handle all their drawing needs themselves. It gives the most flexibility and control over everything, and it also makes it easy for the SDK because there is no need to make fancy drawing routines to be used by plugins.
Fast forward to the Vulkan and Metal port and we suddenly have an issue: Neither Vulkan nor Metal is OpenGL, but there are now hundreds of plugins out there that all assume X-Plane uses OpenGL and the OpenGL context is there. Killing OpenGL for plugins would mean countless plugins would stop working, requiring potentially lengthy update processes or maybe they’ll just never work at all again because the author lost interest in X-Plane or developing the plugin. This is how we ended up with the OpenGL Bridge in X-Plane 11.50: We create a real OpenGL context, share some memory from within Vulkan to that OpenGL context, then let plugins continue to draw how they used to do. X-Plane takes care of all of the heavy synchronization and resource creation rules under the hood and everyone is happy. Everyone? Well, no, not quite…
The problem with Vulkan/OpenGL interop
The big problem with Vulkan/OpenGL interop is that it relies on the driver to support this properly. As it turns out, driver support for this is flaky at best, if you own an AMD card you can probably tell long stories about issues. The most prominent one is flickering or altogether missing rendering, in which case plugin drawing is either completely missing or flickering in and out of existence making it basically impossible to use plugins. The other issue is performance, captured in the screenshot above, where the OpenGL bridge adds almost 10ms per frame, although I have seen numbers as bad as 30ms per frame. For reference, 33ms is the time one frame can take at most to reach 30 FPS! This wasn’t always the case, it used to run much better, but driver regressions are a real thing and this isn’t really something the hardware vendors test. Not that I can be too mad about this, X-Plane is one of the very few applications that uses this feature at all. Despite having filed numerous bug reports with the vendors, unfortunately the problems were never completely fixed, so even during the 11.50 days we started thinking about alternatives. During the X-Plane 11 days it was at least somewhat easy to tell users to switch back to OpenGL to avoid the flickering, even though that usually meant a hit in performance because the Vulkan backend was just so much faster.
There are of course other open source OpenGL drivers out there that aren’t Zink. The most well known one is probably Angle, a implementation of OpenGL ES by Google. None of them work for us however, because of another terrible decision made many years ago: X-Plane has always used a compatibility profile OpenGL context. What that means doesn’t matter, but the end result is that X-Plane and by extension plugins, rely on the existence of OpenGL commands from the 90s! Most modern implementations of OpenGL will happily ignore everything from the compatibility profile because it’s a nightmare to implement and doesn’t map very nicely to modern hardware at all. But a lot of plugins make use of these old features and rely on the fixed function pipeline. Which, by the way, is also the reason we didn’t just expose Vulkan or Metal to developers. Writing either a Vulkan or Metal renderer requires a ton of overhead, is deeply complicated and error prone. Writing correct code requires testing on different hardware, reading a few thousand pages of specification text and having a good understanding of the underlying hardware. Most plugin developers don’t venture past the fixed function pipeline, it would be absolutely unreasonable to expect them to spend the time and effort involved in this.
What’s a Zink?
Luckily we don’t have to throw any developers into the deep end because there is Zink. Zink is part of the Mesa open source project and is a backend for the Gallium driver, which is the OpenGL frontend provided by Mesa. Okay, lots of big words here, let’s simplify this a bit: Gallium implements OpenGL 1.0 to OpenGL 4.6, which basically means “all of OpenGL”. But it doesn’t actually do anything with it, it just does all of the heavy lifting required to make OpenGL work. Backends are what actually turn the intermediate data that Gallium produces into real commands to be executed by a GPU. There are a lot of these backends: For example there are multiple software renderers that just use the CPU, there are implementations that talk to AMD GPUs, Nvidia GPUs, Intel GPUs, things I never even heard about, it’s all there. But among all of those backends there is also Zink. Zink just takes what Gallium produces and builds Vulkan commands out of it to be executed by any Vulkan capable hardware. This is what makes Zink so great for us, it’s a real OpenGL implementation that includes all of the hard and complicated bits we need to make plugins work.
Of course I’ve made it sound easier than it really is, “just” writing a Gallium backend that magically speaks Vulkan is a lot of hard work. The unsung hero of all this is Mike Blumenkrantz, head honcho and lead developer of Zink. He’s spent years getting Zink to the point it is at and is contentiously working on improving it, both in terms of supported features as well as performance. He’s also an absolute legend and none of this Zink interop X-Plane stuff would exist if it wasn’t for his help, not just in getting Zink to where it is but also answering tons of our questions during the integration. Big shout out to Mike! While I’m doing shout outs, it would be unfair of me to not also mention AMD. While they are partially the reason we are here today, they also provided us with help and driver updates to make Zink work together with X-Plane and every engineer of theirs that I have met along this journey has been nothing but kind and helpful.
I’m a user, how does this affect me?
If you are on X-Plane 12.04b3, open your graphics settings and enable the Zink backend. By default we run with the native OpenGL backend for compatibility reasons, although the long term goal is to switch to Zink exclusively, but not anytime soon. There is a chance some or all of your plugins will explode, we’ve done testing with third party developers on this, but the area touched by this immense and we can’t test every plugin out there. If it breaks, no worries, please file a bug report and let us know, fall back to the native OpenGL backend and try again in the future. The idea behind Zink is to make it as much a “it just works” thing as possible, but it’s the first step and if it isn’t perfect right out of the gate, I hope you have some patience with us. I’m saying this because like I said, the surface area touched is immense here and shipping a whole graphics driver with an application isn’t really something super common and, as I have found out, sometimes breaks stuff.
The good news is that Zink is open source and we are building it ourselves, which means that for the first time we can actually see under the hood when things explode. I have very high hopes that this will help us track down issues much faster than usually. Normally drivers come with all debug data stripped, so a few times I have seen bugs that just end somewhere in the driver with no way to tell what even happened or how to reproduce it. Sometimes we don’t even have any information at all because it got so thoroughly muddied.
I’m a developer, how does this affect me?
Well, same as above: Test your plugins, file bugs, be patient. But also, there are a few things you might want to be aware of in general:
First, X-Plane now finally lets you enable a debug GL context. If you run X-Plane with --debug_gl, X-Plane will make a debug GL context for you and then wire up the necessary callbacks. By default it’ll print all messages into stdout and errors get a big in sim pop up. But you can always redirect it to something of your choosing by calling glDebugMessageCallbackARB() with your own callback. Please don’t ship plugins with a handler enabled though, because it might lead to other developers not being able to set their own callback as there can only be one. The debug GL context is available when running through Zink and also the native GL driver.
Gotcha wise, there are two that I’m aware of:
In theory you can make a shared GL context for background GL processing, but it might not be 100% stable. I’m still trying to track down an issue somewhere in the stack that leads to resources being incorrectly deleted, so if you run into issues with a shared GL context, this might be it. Please try it though!
Don’t disable and/or enable GL_FRAMEBUFFER_SRGB, this will lead to all subsequent rendering to disappear. This bug I have tracked down further into Mesa and the problem is that the driver will create a copy of the image to add or remove the sRGB bit, but it never ends up copying the result back over. In theory this should be handled by just a view on the same image without any copies, but either way this is currently broken.
If you want to test Zink as part of some CI solution, you can also run X-Plane with --zink or --no_zink through the command line and it’ll override whatever was set in the graphics settings.
I’m on macOS and I feel left out
First of all, I’m sorry to hear that. 12.04 has great things for macOS users as well that I’m sure you’ll love. Long term, we are having ideas of running Zink on macOS too, just this time on top of Metal. GL on macOS is already emulated through Metal, it’s just done by Apple without visibility or ways to tap into that. We have a bit of a pipe dream of running plugins on top of Zink on top of MoltenVK on top of Metal, but this is purely in the ideas bin right now and no code for this has been written. There would be two advantages for this approach though: First, Apple has deprecated OpenGL already in macOS and while I do believe they will continue to support it, lest their pro users will climb on their roof and shout bloody murder, they clearly have no interest in expanding OpenGL support any further. Secondly, if we get OpenGL bridging down to just Zink on all platforms, it’d be a boon for developers because they can target just one OpenGL implementation and cover all platforms in one go. It’d also finally bring OpenGL 4.6 support to macOS, which is currently held back to OpenGL 2.1.
First, I appreciate everyone’s cooperation with the RFC on scenery; we’ve had an ongoing discussion in our developer Slack as well as the comment section, and I don’t think I had to nuke any off-topic comments. The feedback was wide-ranging and there’s no one clear answer but it does give us a really good picture of how the scenery system is working (and isn’t working).
It’s Friday, so let’s do something completely different – her’s some show and tell from a few things people have been working on things week.
Light It Up
Alex has been recalibrating the runway and airport lights for the new photometric lighting engine. This spurred an internal discussion about how best to calibrate artificial light sources. Does the author specify the luminance of the bulb before a tinted plastic housing goes on top (this way is good if you have the bulb specs from the internet) or based on what you’d measure when the finished light is tested? (This way matches FAA specs for airport lights.)
After going back and forth a few times, our answer is “well, both”, and we have a system that now allows this, which should solve use cases for both aircraft (where often the bulb properties are known because you can look up replacement parts) and for airports (where the FAA has standards for the light’s final results).
Something to keep in mind: urban airports are quite dark compared to their surroundings. Ther are very few light sources near the runway that aren’t tightly controlled for brightness and direction. I used to fly over KLAX on a regular basis at cruise altitudes (commuting from San Diego to San Francisco for work) and KLAX was always an inky black void in the sea of lights that is the LA basin; at 34,000 feet no runway lights are pointed up at us.
Petr and Sidney have been working on the weather surface shader, which applies water and other weather effects to surfaces. This is how we dynamically make the pavement wet when it rains.
The shader is tricky because the effect of a surface being wet changes a lot once the water forms a real puddle. When I took my kids to their swim lesson, I couldn’t help but notice the useful reference material all over the place.
Stop Writing on the Windows
I must be a dad, because I get annoyed when my kids get finger prints all over the windows when they “write” things in the frost on a cold day.
Turns out Sidney does the same thing.
What you’re seeing there is programmer art. Programmer art is when the programmers make their own texture files to test code. In this case, Sidney is testing the defrosting system for windscreens, which use a special texture to specify the pattern of defrosting. This lets artists control the defrosting effect and get faster defrosting near vents.
Another “behind the scenes” thing you can see here: that popup window is a set of internal controls for testing, debugging and developing the windscreen effects. The parts of these internal controls that are generally useful will become third party developer tools (like the texture browser and particle system editor in X-Plane 11).
Cessna In Spaaaaaaaace
Daniel rewrote the planet shader. In X-Plane 12, water is treated separately from land (so that it can be 3-d). The new planet shader shows a far view of water and a far view of land at the same time and correctly shows atmospheric scattering, which is normally pre-calculated in a special “froxel cache” for regular scenery.
If you haven’t noticed the pattern, it’s that the art team’s screenshots all tend to look good enough to ship, and the programmer’s screen shots tend to be very, very silly. In this case, the Cessna in space is pretty silly, but what we were looking for was the smooth atmospheric effects all the way out to the horizon.
Here’s one more goofy programmer screenshot:
I was calibrating the runway lights according to Alex’s spec, and typed an extra 0 into one of the internal art controls by accident. The result was this fantastic screen shot.
What you’re seeing is: the billboards for the runway environment are accidentally huge and are filling up the entire reflection cube map. The reflective underside of the Cessna wing picks up this blue lights and it looks like a rave.
[This post is a “behind the scenes” look at the tech that makes up the X-Plane massive multiplayer (MMO) server. It’s only going to be of interest to programming nerds—there are no takeaways here for plugin devs or sim pilots.]
In mid-2020, we launched massive multiplayer on X-Plane Mobile. This broke a lot of new ground for us as an organization. We’ve had peer-to-peer multiplayer in the sim for a long time, but never server-hosted multiplayer. That meant there were a lot of technical decisions to make, with no constraints imposed by existing code.
Amazingly, I think the answer (and I realize I am cursing the next beta by typing this) is we’re getting really close. You’d be forgiven for thinking that’s lunacy given beta 15; here’s a little bit of info about the state of the betas.
Cloud performance should be back to baseline norms for 11.50 – when I fixed a multi-monitor bug in beta 14, I accidentally turned off a major cloud performance optimization,
Fatal errors while resizing the window should be fixed. These squawks were coming from code that now checks much more heavily for error conditions, and revealed a problem when the OS delivers window resizes to us too slowly.
Note the introduction of new bugs in the process of fixing old ones – this is always a risk when a bug fix is intrusive or complicated; in order to get a shippable product, we have to keep ratcheting down the amount of chaos we introduce per beta, making smaller and more surgical changes.
Beta 15 also tried to fix one last plugin compatibility bug that we discovered very late in the game – under Vulkan, there’s no depth/stencil buffer available to third party plugins, resulting in incorrect drawing for previously working plugins.
I think you know where this is going…this new change introduced new bugs, even as it fixed the problem with the original add-on.
We have been working with third party developers over the last twenty four hours on fixes for the regressions here; our hope is to have a new beta on Monday or Tuesday that fixes these issues and has gotten a good third party once-over.
We’re Going to Have to Close the Door
We are reaching the end of X-Plane 11.50 – at this point the number of remaining bugs to be fixed in this beta is small enough that they aren’t that hard to keep track of. To get to a release candidate though, we’ll need to stop introducing major changes.
I am hoping that we already know about all of the third party incompatibilities, because at this point we have to close the door to complex changes to improve backward compatibility. For beta 16 we are fixing what used to work and is now broken (e.g. yes, SkyMaxx will work again), but if anyone is sitting on an add-on problem they haven’t mentioned, we’re out of time to deal with it.
Plugin Developers: Thread Safety!
Plugin developers: in looking at your compatibility with X-Plane 11.50, please re-read this post and check your usage of threads. We’ve had really helpful responses from third parties who we have notified about threading issues we’ve seen by auto-crash reporting, and I think this will help everyone–third parties, users, and us.
I believe that X-Plane 11.50 is significantly more sensitive to threading violations than 11.40, because the Vulkan driver doesn’t spend CPU cycles protecting itself from abuse. If a plugin calls into us from a thread when it’s not allowed to, this can cascade into crazy X-Plane behavior, which cascades into crazy Vulkan behavior that the driver won’t stop. So careful adherence to the threading rules by plugins is critical to Vulkan stability.
With Vulkan and Metal, X-Plane is now firmly in the driver’s seat for VRAM management. This lets us eliminate stutters that were previously present with OpenGL and almost impossible to avoid. It has, however, one big and noticeable downside: when you run out of VRAM, you get blurry textures.
Of course the goal wasn’t to replace stuttering with blurry textures, and we believe that given the normal work load of X-Plane, you should not be seeing this. The fact that so many users are seeing blurry textures, especially on big cards with lots of VRAM, points to the VRAM code being buggy in all the ways beta code can be buggy.
How much VRAM do I need?
Just to get the obvious out of the way, our system requirements have not changed for 11.50. Our minimum VRAM requirement for X-Plane 11 is still 1Gb. We expect these cards to be able to run 1080p with lowered texture resolution, providing an equal experience to X-Plane 11.41. With 2Gb we expect users to be able to run HDR with medium texture resolution on 1080p systems. 4Gb and higher should be able to run HDR at with high resolution textures with 2k monitor resolutions. With 6Gb and higher, 4k monitor resolution should be possible.
A week or two ago we had a very dead beta, and posed the question of how to incrementally test betas in the future. We got a variety of responses, ranging from “private test it first” to “roll it out in a wave” to “full speed ahead, we know betas are bumpy.”
Since then, we’ve been doing one of the easiest and probably most useful things we can: posting the betas early to third-party developers who are in our developer Slack channel.
Beta 7/8 had a ton of changes, and our third-party developers found multiple problems, some of which we wouldn’t see in our internal tests. So we held off on releasing betas 7 and 8 to the public while we fixed those issues. Until today.
X-Plane 11.50 has been similar to X-Plane 11.20 (our VR) release and different from what we normally try to do, in that when we went beta (both private and public), the work for Vulkan wasn’t done yet. We had something that you could fly with, that was delightful for some users (and unstable for others), but we also had a big list of things we still needed to do.
This post is just targeted at plugin developers who are modernizing their object drawing – if you don’t write plugin code, the Cincinnati Zoo has been showing their animals on Youtube – it’ll be a lot more entertaining than this post. (An XPLMInstance cannot tunnel down two feet in fifteen seconds – one point for the zoo animals.)
XPLMInstance makes a persistent object that lives inside X-Plane that is visible in the 3-d world. It changes how you draw from “run some drawing code every frame” to “tell X-Plane that there is a thing and update its data every now and then.”
Instancing is actually a lot easier than draw callbacks! But there are two tricky gotchas:
1. You must create the custom DataRefs for your OBJ’s animation before you load the object itself with the SDK. (If the DataRefs do not exist at load time, the animations are disabled as “unresolved to any DataRef”.)
2. When you create the instance, make sure your custom DataRefs are on the list of DataRefs for that instance.
Here’s the really baffling thing: if you create the custom DataRef and then add it to the instance’s list, your DataRef callbacks will not be called.
Here’s the trick: the DataRef you register is a global identifier, allowing the object to refer to what it wants to listen to. That’s why you have to create the DataRef – so that the identifier exists.
But when you create an instance, each instance has memory that holds a different copy of those DataRefs.
For example, let’s say you have a truck with four DataRefs, and you make five instances. X-Plane allocates 20 slots (four DataRefs times five instances) to store five copies of each DataRef’s values.
The instances never look at the DataRef itself. They only look at their local copies. That’s why when you push different data to the instance with XPLMSetInstancePosition, each instance animates with its own values – each instance looks at its own local data.
This is also why you won’t see your DataRef callbacks called (unless you use DataRefEditor or some other tool). The object rendering engine isn’t looking at the DataRefs themselves, it’s looking at the local copies.
In other words, XPLMInstance turns DataRefs from the pull model you are used to (X-Plane pulls on your read function to get the value) to a push model (you push set with XPLMSetInstancePosition into the instance’s memory).
This implies two things about your add-on:
It doesn’t really matter what your DataRef read functions do – they can just return zero, and
You can’t use tools like DataRefEditor or DataRefTool to debug your animations. (That didn’t work well in legacy code either, but it really won’t work now.)
If you try the obvious optimization of not creating your custom DataRefs (“hey, no one calls them”) before you create your instance, you will find that animation just stops working. This is because we need the DataRef to be that global identifier to match your instance data with the animations of the object itself.
One last note: if your old code used sim/graphics/animation/draw_object_x/y/z to determine which object was being animated (from inside a plugin “get” function) you do not need to do this anymore. Because each instance has its own local copies and your DataRef function isn’t called, this technique is obsolete.
You must register custom DataRefs.
Their callbacks can just return 0 – they’ll never be called.
Always list your custom DataRefs for animation when you create an instance.
Do not use draw_object_x/y/z; use XPLMSetInstancePosition to create per-specific-instance animation.
I was going to write a post about X-Plane 11.50 beta 5 – what’s new in it, the new ways we are debugging GPU crashes, the crash bugs we’ve fixed, etc. A lot of stuff that we thought was pretty good went into beta 5. Cool new technology! Big bug fixes! Lots of winning!
As it turns out, beta 5 is dead. I hit “go” on the release this afternoon, and half an hour ago, I hit “stop.” The auto crash reporter was showing way too many new crashes in memory management that we had not seen before, and this strongly implies a new and serious bug.
Laminar Installer Users: if you were auto-notified to update to beta five and did so, and you are not crashing, you can keep flying! If your beta five is just a smoldering wreckage of crumpled VRAM and GPU parts, you can re-run the installer with “get betas” option checked, and it will take you back to beta 4.
If you were not auto-notified to update to beta five, that’s probably for the best. Please stand by and keep flying beta four; we’ll post a new beta when we’ve gotten to the bottom of this. We have enough captured crash data to investigate.
Steam Users: we did not release the beta five build to Steam and this is probably a good thing; we’ll try again with a new release that isn’t made of plutonium and unicorn hallucinations.
And if you’re going “why didn’t y’all test it before you released it”…we did! None of our machines show these crashes. But we also have probably a dozen PCs total we can run on. Moving to a new driver stack has meant learning about the weird things that happen on your computers and not ours.
Do We Need a Two Tiered Beta System
This came up in our impromptu beta five post-mortem meeting: do we need to bring people into new betas in stages? With code for new drivers, beta five probably won’t be the last beta where we code something we think is helping and discover that it fails catastrophically, but not on our hardware. We need beta victims^H^H^H^H^Htesters to find these bugs, but once we get a dozen crashes, we don’t need anyone else to stub their toe for us to fix our problems.
So we thought about two possible ways to do this:
A two-tiered system. Early adopters could get an email and hand-update to the new beta before it is put out for auto-update notification.
Send out the beta update notifications over time, e.g. 10% of users get notified immediately, then another 40% if we don’t see crashes, then the last 50%. (This practice is actually industry standard on mobile apps.)
If you are reading this blog post, this far down, you are probably participating in the beta; I’d be curious what approach you’d find most useful.