Yesterday I went off on a rant about how the OpenGL 3.0 spec doesn’t matter a lot because OpenGL grows via a la carte extension. And I mentioned that this creates a dilemma for anyone designing a rendering engine that has to run on a wide range of hardware, with very different underlying capabilities.
Back in the old days of X-Plane 6, there wasn’t a lot of variety in hardware. Some cards were faster than others, but the only real difference in capability was how many texture units you had. X-Plane’s job was simple…
- First we have a runway texture.
- Still got a second texure unit? Great! Skid marks!
- Still got a third texture unit? Great! Landing light!
- Got another one? Etc…
Other than texture stacking, there wasn’t much to do.
Since then the rendering engine has become a lot more complex, as have OpenGL cards. To try to keep the combinations down, I tried to use a “bucketing” strategy for X-Plane 9. The idea of bucketing is to group cards into major buckets based on whole sets of common functionality, so that we only have to test a few configurations (the “low end” bucket and “high end” bucket), rather than having to test every single card for its own unique set of features.
The obvious bucketing choice was pixel shaders – given a card with shaders and a few other features, we can render all of the new effects. A card without shaders basically gets X-Plane 8.
So what went wrong? Driver compatibility, that’s what. Ideally we don’t want to allow every single underlying rendering engine feature to be turned on or off individually because the combinations are uncontrollable.* But in practice, being able to turn features on and off is necessary to trouble-shoot drivers that don’t always do what we expect them to.
With the GeForce 8800 and Radeon HD cards, there is a potential third bucket for DirectX-10 capable cards, capable of significantly more advanced pixel shading effects. But time will tell whether we can actually make a bucket, or we have to look at each feature individually. My suspicion is that even if we organize the shipping features into buckets, we’ll have to support a lot of combinations under the hood just to trouble-shoot our own application.
*Example: cross a standard vs. oversized panel with the presence or absence of advanced framebuffer blending, crossed with whether render-to-texture works. That’s 8 possible ways just to render the 3-d cockpit panel texture. Cross that with panel regions and 3-d cockpits and the new panel spotlighting and you have 64 configurations. Ouch!
I just received a series of reports today that certain converted scenery will cause X-Plane to crash with a “bad alloc” error. Basically, this couldn’t have hit us at a worse time. The final 920 was cut a week ago. We physically can’t recut; Austin is on the road, and I am knee deep in it. But there is a possible work-around, and there will be a patch. Here’s the whole situation.
What is a Bad Alloc?
A bad alloc error is an error that comes up when X-Plane runs out of memory. This can happen for two reasons:
- We have run out of address space – that is, there is no more virtual memory left, or
- We have run out of page file/physical memory – that is, we can’t back that virtual memory.
The first case is by far the most common – you’d only hit the second if you are on Windows with a fixed-size (but small) page file. (Hint: if you have a fixed size page file, make it big!)
X-Plane can run out of memory for many reasons – everything that runs in the sim uses memory, and the amount used depends on what area you are in, what rendering settings you pick, and what third party add-ons you use. While I’d like to someday reach a point when the sim tells you gracefully that it’s out of memory, it will always be a fact of life that at some point (hopefully an absurdly high one) the amount of stuff you’ve asked X-Plane to do will exceed how much memory you have.
(If you are thinking 64 bits, well, that will just change the problem from a crash to a grinding halt when we run out of physical memory.)
We see bad allocs when there are too many third party add-ons installed (XSquawkBox is a particular pig because it loads every CSL on startup), too complex scenery, and it can also be caused by drivers not efficiently using memory. (This is particularly a problem on Vista RTM.)
The Bug
When X-Plane creates a curved airport taxiway, it allocates a temporary memory buffer to hold the intermediate product of the pavement. The size of that buffer depends on the complexity of the curve it is processing and a constant, based on the maximum curve smoothness.
In 920 I provided an option to crank up the curve smoothness in X-Plane. In the process, I increased that constant factor by 4x, which causes X-Plane to hit its memory ceiling on layouts that used to be acceptable. You’ll see this problem more often on:
- Bigger, more complex layouts.
- Configurations that were already chewing up a lot of memory.
- Machines with less address space (Windows without /3GB, older Mac OS X operating systems.)
What really suckered us about this bug was that it comes in a form that looks almost the same as a driver issue we’ve seen with ATI drivers on Windows — we’ve seen strange forms of memory exhaustion on ATI when shifting scenery with high rendering settings. So we didn’t realize that this was something new until G5 users reported the bug (making us realize it wasn’t a driver thing).
What To Do
The bad news is that we can’t do an RC5 – we’re out of time. But – there will be a patch – relatively soon. This bug is on the short list for a patch to fix 920.
In the meantime, there is actually a work-around. By coincidence, some of the internal rendering engine constants are viewable via the “private dataref” system — basically a series of datarefs in the sim/private/… domain that I use for on-the-fly debugging. The dataref that matters here is:
sim/private/airport/recurse_depth
If you load up DataRef Editor you’ll see it has a value of 12 . That’s too high. Changing it to 10 will allow otherwise problematic airports to load.
I will try to post a plugin in the next 10 days that sets this dataref to 10 on startup, effectively patching the problem. This will also limit the maximum smoothness of curves – but my guess is that if you see the crash (not all users do) then you can’t run on the max airport curve setting anyway.
Of course the next patch will contain a real solution: a more efficient memory allocation scheme!
Some users reported during 920 beta that X-Plane would sometimes not detect its DVD – a condition that would come and go. Tonight I figured out what is happening.
- In order to validate the DVD, X-Plane decompresses part of its contents into the preferences folder. Why preferences? There is no good reason – it’s historic.
- X-Plane will create a preferences folder if there is not one. But it does not do that until you quit.
- The X-Plane installer will not make directories unless they contain files.
So put these three things together: on the first run of a new install, there is no preferences file, so the DVD check fails since the directory that will contain some temporary files is missing. Run a second time, and the directory is there and the DVD check succeeds.
The next patch of the sim will fix this, but in the meantime, if you delete your preferences, leave the empty directory in place!
I don’t usually blog about plugin issues, but this one falls into limbo, somewhere between plugins, aircraft design and OBJ animation. This is written for plugin developers; if you don’t write plugins, you can tune out.
Plugin Drawing
Plugin drawing is documented in a few tech notes – I strongly recommend reading them all.
The basic idea is this: to draw in a plugin, you register a callback. X-Plane tells you “it’s time to draw” and you draw.
The first rule of drawing callbacks is: you only draw. You don’t do anything else.
The second rule of drawing callbacks is: you only draw. You don’t do anything else.
There are a number of good reasons for this.
- You don’t know how many times, or in what order the callbacks will be called! You might be called four times per frame or none. So doing flight model calculations in a draw callback is a bad idea.
- Drawing has to be fast. In the drawing code we are trying to stuff the GPU to the gills with work to do. Drawing time is not a good time to go off and do other expensive calculations.When we look at the interaction of the CPU and GPU, we know that the flight model takes some pure CPU time, and we can improve efficiency by queuing an expensive OpenGL operation before we hit that CPU-only phase. If your plugin is doing its real calculations during draw time, our pipelining gets thrown off.
- We’re continually increasing the parallelism of the sim via threads to take advantage of multiple cores. But plugins are single threaded by definition. This means that plugin interaction points will (in the future sim) halt parallel operation and slow overall performance. When we design parallel operation, we can try to optimize our design to avoid this “halt” case, but if you are doing something strange (like flight calculations in a draw loop) you’re more likely to fall off the “fast path”.
Surprise
Now here’s the surprising thing: your dataref callback is actually a drawing callback, sort of!
Object datarefs are called back during object drawing. Therefore they have all of the above problems that drawing callbacks have. Here is my recommendation:
If you create a custom dataref that is used by an OBJ for animation, return a cached value from a variable in the dataref, and update that variable from the flight loop.
This avoids the risk of your “systems modeling” code being called more than once per frame, etc.
I commissioned
PropsMan to make me a test bed for some of the new 9.20 airplane features. Clearly he has never worked in a furniture store — this is what he sent me!
In his defense, it’s very comfy, flies surprisingly well, and is a great comprehensive test of a huge pile of advanced aircraft features. Yes, you can close and open the laptop with the mouse in the 3-d cockpit.
(One of the new features in 920 is “
no-op manipulators” – that is, the ability to make 3-d geometry “eat” clicks before they get to the panel, without having to use panel texture as the consuming surface. So when the laptop is closed, you cannot click on the buttons that are exposed when it is open.)
My real question is: how did he know about Laminar’s new secret test vehicle?
I’ve been reading Fooled by Randomness (highly recommended – it will either change your life or you won’t finish it because Taleb’s writing style annoys you) – and it has me thinking about the nature of certainty in software development. Consider two approaches to uncertainty and how they are completely at odds with each other.
No Weird Behavior
The “No Weird Behavior” approach goes like this: you never want a harmless behavior inside your code that you don’t understand. The reason is that if you don’t understand the behavior, you don’t really know that it’s harmless!
In fact this isn’t just a theoretical problem – I only truly started to believe in “No Weird Behavior” after fixing several very hard to find, hard to reproduce crash bugs, only to discover (once the analysis was done) that the broken code also produced a very frequent, less harmful behavior. Had I taken the time to investigate the “weird behavior” first (despite it not being of high importance) it would have led me to a ticking time bomb.
No Weird Behavior implies that a bug isn’t fixed until you know why it’s fixed, and that randomly changing code until the behavior goes away is absolutely unacceptable. This shouldn’t be surprising; if a bridge was
swaying would you accept an engineer who fixed it by randomly tightening and loosening screws until it stopped?
Wait And See
I get a lot of email with bug reports, questions, and reports of strange sim behavior. These emails have some unfortunate statistical properties:
- A good number of them are resolved by the user who emailed within a day or two.
- A certain percentage are simply never resolved. (Often I email a question that would point to a clear diagnosis and the user never writes back. I can only speculate that the user answered the question, found the problem, fixed it, and promptly forgot about our emails.)
- A certain percentage are solved by the user, who tells me what the problem was, and it was something I would never, ever be able to help them with. (Things like “I have this program called WickedMP3Player – it turns out if I set its visualizer setting to ‘Rainbow’ then X-Plane stops crashing” when I’ve never even heard of the WickedMP3Player program to begin with.)
- There is a high correlation between bug reports reported by a very small number of users and a resolution from the above categories, or a resolution by the user changing his or her local configuration.
So playing the odds, the right thing to do when presented by a third party with weird behavior is to wait and see who else reports it; the frequency of reports gives us information about the likely resolution.
(And for those of you who think our tech support are being lame when they ask if you’ve updated your drivers, they are playing the odds to the hilt – they ask you about drivers first because updating drivers fixes an absurdly huge number of the tech support calls we get.)
Getting It Wrong
So we have motivation to investigate everything (no weird behavior), motivation to ignore everything (wait and see) and a rule of thumb that the frequency of reports can help us pick which strategy is best. Of course, sometimes the rule of thumb is completely wrong.
One user reported a crash using the current web updater (version 2.04) – I had not seen this crash anywhere and it was inside the operating system UI code, so I assumed it was a local problem, e.g. some kind of extension or add-on that caused the OS problems.
The problem, it turns out, is simply low frequency – as the incorrect code made it into 902b1, I got a few reports from more users and realized that this wasn’t a local problem, it was weird behavior! (The bug will be fixed in the 205 installer and 902b2, both of which will be out in June.)
Consider this: if you make a measurement of a known quantity with a dubious measuring device, you learn more about the measuring device than the object you are measuring. (If you have a ruler and you don’t know the units, you could determine them by measuring yourself.)
In a number of cases, we have seen serious “should-happen-all-the-time” crash bugs that get reported by very few users. (Once we know the actual root cause of the bug, we can deduce logically whether the bug should happen to all users with the configuration or be intermittent.) We can then look back at the actual number of reports to make a judgement call on how much testing is really happening.
For example, we can make some guesses about how quickly a new Macintosh have saturated the X-Plane user base when there is a hardware-specific serious bug in just that machine.
We had this with the iMacs (where the runway lights were broken) and we could watch the machines sell – slowly at first, but then quite quickly. (BTW, I think that 10.5.3 fixes this and anti-aliasing problems.) We can even see who runs with anti-aliasing when there is a setting-specific bug (a lot of you do)!
In the end, I think the right approach to balancing “no weird behavior” and “wait and see” is to remember a truth about uncertainty that is very hard for humans to grasp:
The most likely outcome of an uncertain situation is not guaranteed to happen – in fact a lot of the time it won’t.*
So we can play the rule of thumb and wait and see, but we always have to keep one eye toward the improbable, which is still possible!
* Blatant blog cross-promotion…the point of my
big rant here was essentially the same…it’s easy to expect the most likely outcome, but the unlikely outcome will happen some of the time.
Now that multi-core machines are mainstream, you’ll hear a lot of talk about “threads”. What is a thread, and how does it relate to using more cores?
Definitions
A “core” is an execution unit in a CPU capable of doing one thing. So an 8-core machine might have two CPUs, each with four cores, and it can do eight tasks at once.
A “thread” is a single stream of work inside an application – every application has at least one thread. Basically a two-threaded application can do two things at once (think of driving and talking on your cellular phone at the same time).
Now here’s the key: only one core can run a thread at one time. In other words, if you have an eight core machine and a one-thread application, only one core can run that application, and the other seven cores have to find something else to do (like run other applications, or do nothing).
Two more notes: a thread can be “blocked” – this means it’s waiting for something to happen. Blocked threads don’t use a core and don’t do anything. For example, if a thread asks for a file from disk, it will “block” until the disk drive comes up with the data. (By CPU standards, disk drives are slower than snails, so the CPU takes a nap while it waits.)
So if you want to use eight cores, it’s not enough to have eight threads – you have to have eight unblocked threads!
If there are more unblocked threads than cores, the operating system makes them take turns, and the effect is for each of them to run slower. So if we have an application with eight unblocked threads and one core, it will still run, but at one eighth the speed of an eight core machine.
It’s not quite that simple, there are overheads that come into play. But for practical purposes we can say:
- If you have more unblocked threads than cores, the execution speed of those threads slows down.
- If you have more cores than unblocked threads, some of those cores are doing nothing.
Trivial Threads
When a thread is blocked, it does not use any cores. So while X-Plane has a lot of threads, most of them are blocked either most or all of the time. For all practical purposes we don’t need to count them when asking “how many cores do we use”. For example, G1000 support is done on a thread so that we keep talking to the G1000 even if the sim is loading scenery. But the G1000 thread spends about 99.9% of its time blocked (waiting for the next time it needs to talk) and only 0.1% actually communicating.
What Threads Are Floating Around
So with those definitions, what threads are floating around X-Plane? Here’s a short list from throwing the debugger on X-Plane 9.0. (Some threads may be missing because they are created as needed.
- X-Plane’s “main” thread which does the flight model, drawing, and user interface processing.
- A thread that can be used to debug OpenGL (made by the video driver, it blocks all the time).
- Two “worker” threads that can do any task that X-Plane wants to “farm out” to other cores. (Remember, if we want to use more cores, we need to use more threads.)
- The DSF tile loader (blocks most of the time, loads DSF tiles while you fly).
- At least 3 threads made by the audio driver (they all block most of the time).
- At least four threads made by the user operating system’s user interface dode (they block most of the time).
- The G1000 worker thread (blocks most of the time, or all the time if you don’t have the G1000 support option).
- The QuickTime thread (only exists when QuickTime recording is going on).
So if there’s anything to take away from this it is: X-Plane has a lot of threads, but most of them block most of the time.
Core Use Now
So how many cores can we use at once? We only need to look at threads that aren’t blocked to add it up. In the worst flying case I can think of:
- The main thread is rendering while
- The DSF tile loader is loading a just-loaded tile while
- One of the pool threads is building forests while
- You are recording a QuickTime movie (so the QT thread is compressing data).
Yep. If you really, really put your mind to it, you can use four cores at once. 🙂 Of course, two cores is a lot more common (DSF tile loading or forests, but not both at once, and no QuickTime.
Core Use In the Future
Right now some of X-Plane’s threads are “task” oriented (e.g. this thread only loads DSF tiles), while others can do any work that comes up (the “pool threads”, it’s like having a pool car at the company, anyone can take one as needed). The problem with this scheme is that sometimes there will be too many threads and sometimes too few.
- If you have a dual-core machine, having forests building and DSF loading at the same time is bad – with the main thread that’s three threads, two cores; each one runs at two-thirds speed. But you don’t want the main thread to slow down by 66%, that’s a fps hit.
- If you have a four-core machine, then when the DSF tile is not loading, you have cores being wasted.
Our future design will allow any task to be performed on a “pool thread”. The advantage of this is that we’ll execute as many tasks as we have cores. So if you have a dual-core machine, when a DSF tile load task comes along while there is forests being done, the one pool thread will alternate tasks, leaving one core to do nothing but render (at max fps). If you have a four-core machine, the DSF load and forests can run at the same time (on two pool threads) and you’ll have faster load times.*
* Who cares about load time if you’re flying? Well, if you crank up the settings a lot and fly really fast, the loader can get behind, and you’ll see trees missing. X-Plane is always building trees in front of you as you fly and deleting the ones behind you. So using more cores to build the forests faster means you’re less likely to fly right out of the forest zone at high settings.
I just finished about 15 pages of emails, mostly to Andrew McGregor (who is the very first MeshTool user) and also Benedikt Stratmann (whose x737 is on the bleeding edge of plugin-based aircraft) and AlpilotX (we all know about his forests). Probably all three are wondering how the hell I have time to write so much on weekends. (The answer is of course that my frisbee game got rained out. Foo!)
In the meantime, probably about 300 other people who have emailed me in the last few
months are wondering why the hell they have heard nothing from me. My in-box looks like a mail server exploded. It’s not pretty.
So let me blog for a moment about the “relationship problem”. Simply put, there are two of us (Austin and myself) and about a thousand of you (third party developers doing cool and interesting things with X-Plane) plus significantly more users, some of whom have some very weird tech support problems.
In this environment, our algorithm for who gets “developer attention” is pretty broken and subject to total thrash…there is a huge element of random luck (who emails me when I am recompiling the sim vs. debugging a nasty bug).
I’m aware of both how hard the task Austin and I face and how frustrating it is for a third party developer because I’ve been on both sides. Before I worked for LR, I was a third party and I was always astounded that Austin couldn’t remember what we talked about last week.
Then I started working for the company and saw what it’s like. Imagine sitting at a train station watching the trains go by* (at full speed, not stopping) and someone says “last week I waved to you out the window and you waved back, remember me?”
So I would advise three things to the neglected third party:
- Be firm – you may need to ping us again because at busy times we can’t always keep track of who has asked for what.
- Be patient – if you need something the week we’re burning DVD masters for a second time (because the first set failed at the factory) then you’re going to have to wait.
- Don’t take it personally…a lack of a response usually indicates overload inside the company, not a poor opinion of your work!
This blog post has rambled enough, but it may feed well into the next one.
* This analogy is totally stolen from “How Doctors Think” by Jerome Groopman – he uses it to describe the task of primary care physicians trying to spot the early signs of a very rare illness among a fast-moving train of patients who are almost entirely healthy. I strongly recommend this book particularly for Americans – we need to understand the forces at work in shaping the quality of our medical care!
One of the reasons why the X-Plane 9 betas have had so many more crash bugs than version 8 is that we introduced loading DSFs on a second core. This feature makes scenery loads much slower and (by using the second core) impacts fps less while they happen.
The problem is that I’m still fumbling with code that will allow this in all cases. (That would be three operating systems, two hardware vendors, and a myriad of drivers, some new, some quite prehistoric.)
Beta 22 will be out soon, and will contain the fourth major rewrite of the OpenGL setup code for X-Plane 9. So far the initial tests look good, but we never know until we let a lot of users try the code and find the new edge cases.
It’s relatively easy to tell if your instability is related to the use of OpenGL with threads: simply run the sim with the –no_threaded_ogl option. If things become a lot more stable, it’s a threaded GL problem. Mind you –no_threaded_ogl is more of a diagnostic than a workaround; without threaded OpenGL, the sim will pause when loading scenery.
(Also, to clarify, you’ll find talk on discussion groups and game forums about “threaded drivers”. Threads are a programming abstraction that can utilize multiple cores. What I am talking about is X-Plane using multiple threads to load scenery – in our case this requires interfacing with OpenGL. But a threaded driver is different – it’s just a graphics driver that’s been optimized for multcore machines. These two concepts are totally different; you don’t need a threaded driver to use X-Plane 9, and a threaded driver won’t make X-Plane 8 load without pauses.)
A number of Windows and Linux GeForce 7 users have discovered that the command-line option –no_fbos improves their pixel-shader framerate a lot. Windows and Linux Radeon HD users have also discovred that –no_fbos cleans up artifacts in the water. Here’s what’s going on, at least as far as I can tell. (Drivers are black boxes to us app developers, so all we can do is theorize based on available data and often be proved wrong.)
Warning: this is going to get a bit technical.
FBO stands for framebuffer object, and simply put, it’s an OpenGL extension that lets X-Plane build dynamic textures (textures that change per frame) by drawing directly into the texture using the GPU. Without FBOs we have to draw to the main screen and copy the results into the dynamic texture. (You don’t see the drawing because we never tell the card “show the user”.)
We like FBOs for a few reasons:
- Most importantly, FBOs allow us to draw big images in one pass even if the screen is small. For example, if we have a 1024×1024 dynamic texture but the screen is 1024×768, then withou FBOs we have to draw the image in two parts and stitch it together. That sucks. With FBOs we can just draw straight to the texture and not worry about our “workspace” being smaller than our texture. This is going to become a lot more important for future rendering features where we need really-frickin’ big textures.
- It’s faster to draw to the texture than to copy to it.
- If you’re running the sim with FSAA, then we end up using FSAA to prepare all of those dynamic textures. In virtually all cases, we don’t need the quality improvements of FSAA, so there’s no point in taking the performance penalty. When we render right into the texture, FSAA is bypassed and we prep our dynamic textures a lot faster.
Since copying to a texture from the screen predates these new-fangled FBOs by several years, most drivers can copy from the screen to the texture very quickly; however we have hit at least one case where FBOs are much faster than copy-from-screen. That’s really a rare bug, and as you’ll see below, we see more weird behavior with FBOs.
When do we use FBOs vs. copying? Well, it’s pretty random:
- Pixel shader reflective water and fog use FBOs.
- Cloud shadows and the sun reflection when pixel shaders are off do not use FBOs.
- The airplane panel uses FBOs if the panel is 1024×1024 or smaller; if the panel is larger than 1024×1024 we draw from the screen and patch things together. So the P180 and the C172 are using different driver techniques!!
When you run X-Plane with –no_fbos, you instruct X-Plane to ignore the FBO capability of the driver, and we use copy-from-screen everywhere.
Mipmapping
There is one more element: mipmapping. A mip map is a series of smaller versions of a texture. Mipmapping allows the video card to rapidly find a texture that is about the size it needs. Here’s an example: imagine you have a building with a 128×128 texture. If you park your plane by the building, the building might take up about 100×100 pixels on the screen; your 128×128 texture is a good fit.
Now taxi away from the building and watch it get smaller out your rear window. After a while the building is only taking up 8×8 pixels. What good is that 128×128 texture? Its’ much too big for the job. With mipmapping, the card has a bunch of reduced-size versions of your texture laying around…64×64, 32×32,16×16, 8×8, 4×4, 2×2, 1×1. The video card realizes the building is tiny and grabs the 8×8 version.
Why not just use the 128×128 texture? Well, we’d only have two options with this texture:
- Examine all 16384 pixels of the texture to compute the 64 pixels on screen. That sucks…we’re accessing VRAM sixty four times for each pixel. Accessing VRAM is slow, so this would kill performance.
- Simply pick 64 pixels out of the 16384 based on whatever is nearby. This is what the card will do if mipmapping is not used (because option 1 is too slow) and it looks bad. Thsoe 64 pixels may not be a good representation of the 16384 that make up your building side.
So mipmapping lets the video card grab a small number of pixels that still capture everything we need to know about the building at low res.
We don’t mipmap our dynamic textures very often; the only ones that we do mipmap are the non-pixel-shader sun reflections and the pixel-shader sun reflections.
ATI
As far as we can tell, the current ATI Catalyst 8.1 drivers do not generate mipmaps correctly for an FBO-rendered texture. This is why without –no_fbos ATI users on Windows or Linux see very strange water artifacts. –no_fbos switches to the copy path, which works correctly.
At risk of further killing my track record of driver bugs in v9, we do think this is a bug. We have good contact with the ATI Linux driver guys so I have hopes of getting this fixed.
nVidia
It appears that the process of creating mipmaps for FBO textures is not accelerated by the hardware on the GeForce 7 GPU series. This is why GeForce 7 users are seeing such poor pixel shader performance, while GeForce 8 users aren’t having problems.
Now poor performance is not a bug; there’s nothing in the OpenGL spec that says “your graphics card has to do this function wickedly fast”. Nonetheless, what we’re seeing now is unusably slow. So more investigation is needed — given that the no-FBO case runs so quickly, I suspect the hardware itself can do what we want and it’s just a question of the driver enabling the functionality. But I don’t know for sure.
