X-Plane 11.10 is the first major version release with a lot of code changes and new features. By design, this version has a lot of big changes in it that we’ve held back for weeks or even months. This also means it is more likely to be unstable and buggy, especially for the first few beta releases.
We’ve recently progressed to internal beta testing. Now we should have a public beta sooner rather than later, so I wanted to bring back out some great tips on beta testing that Chris wrote way back in 2012. Nearly all of the info remains as relevant today as it did 5 years ago, and I’ve included his best practice tips below.
- Keep two copies of X-Plane. If you care to ever fly X-Plane for enjoyment, ALWAYS keep two copies of the simulator on your hardware. One on a stable release that you use for enjoyment, the other on the latest beta for test purposes only. Your sanity will thank you later.
- Don’t use the beta for recreation time. If you’re only in the mood to fly and have fun that night, stay away from the beta copy unless you don’t mind your flight getting cut short. Too many users expect to fly for 6 straight hours and become enraged when the sim crashes on short final. That’s Murphy’s law!
- Always stay on the LATEST beta. Users often say “The latest beta broke XYZ, how do i go back to the previous beta?”. The answer is you don’t and you shouldn’t. That defeats the whole purpose of beta testing! During a beta process, the code is often changing very rapidly. If we’re on Beta 6 and you’re still submitting bugs on Beta 2, you’re wasting your own time and you’re not helping the product. It’s important you keep up with development.
- Blame us, not your system. If it worked in Beta 3 but it’s broken in Beta 4, do NOT tear your machine to pieces trying to figure out the cause of the problem. Report it and let US figure it out. Time after time, I see users change their drivers, install service packs, uninstall and reinstall X-Plane, update their OS etc trying to solve the bug…when it’s probably OUR bug, not your computer’s.
- Beware of the placebo effect! _EVERYTHING_ affects frame rate. Time of day, clouds, location, aircraft, view angle, view direction, addons, plugins etc etc. Sometimes we’ll release a simple patch that does nothing but fix one bug that was unrelated to the rendering pipeline. I’ll poke my nose in the forums to get some feedback and see 10 users who all of a sudden claim to see some massive fps increase in performance and 10 users who see a massive decrease in performance. I roll my eyes at the claims because it’s not a controlled experiment. For example, if you’re the type of person who flies with real world weather enabled, flying one day with clear skies may get you 60fps while the next day you may see 40fps with overcast. Heck, even departing from a different runway than the previous day can result in different frame rates. Running with the command line fps_test IS a controlled experiment. THAT’s the only way to be sure you’re seeing a change in performance. Don’t trust your instincts–gather data in controlled experiments.
- Always read the release notes for each new version. They’ll tell you what bugs are supposed to be fixed as well as what new features have been added. If you see a bug that’s claimed to have been fixed yet it’s still happening to you, that’s when you write a new bug report. If we didn’t claim to fix it, it’s probably not fixed in that particular beta.
- We need a Log.txt! When submitting a bug, ALWAYS ATTACH the Log.txt from THAT sim run. Even if you don’t think the Log.txt is applicable, attach it anyway. Tell us in as much detail as you can what you were doing at the time.
- We need different things for crashes on Mac vs Windows. Because of the automatic crash reporter, there’s no need to submit a manual bug report for crashes on Windows. If you remembered some details after the fact that you want to tell us, then go ahead and file a report. Of course, if the auto-crash reporter didn’t load up for whatever reason, that’s a good time to file a manual report. Mac users on the other hand, need to send us a copy of the Apple crash report. These files are located in your username/Library/Logs/DiagnosticReports/. The file name will have X-Plane & the date in it, and will end in .crash.
- Tell us who you are and what you know. We really appreciate users who enter their email and comments into the crash report form. It gives us a way to contact you to get more information. A handful of really helpful users can make the research a breeze.
- Be objective, not emotional. No one hates bugs more than we do but writing a rant instead of a bug report may make you feel better, but it makes us grumpy. It also makes us unlikely to take you seriously or follow up on your issue.
- Stop staring at the fps counter! It seems that many users are obsessed with their fps counter. “It went up this beta. It went down this beta. It stayed the same this beta.” Yes, the framerate is ONE useful metric to determine the software’s performance but it’s not the ONLY one and often it’s not the right one. Turn it off and fly! Enjoy the simulator. Sometimes there’ll be a bug in one beta that increases fps as a side effect because the sim is no longer drawing what it’s supposed to. Suddenly, some users are thrilled to see a 20% boost in performance and perhaps they don’t notice that half of the usual streets aren’t being drawn any longer. Ben fixes the streets and in the next beta, all of a sudden they lose their 20% performance gain and are outraged that we “broke things again.”
Being a beta tester is a bit of a thankless job, but we rely on our ambitious and courageous users to take the plunge and help us test each new version of X-Plane. Beta testers help us find those subtle or rare issues so that the sim is polished and enjoyable for everyone.