Having such a small development team, it's very difficult for us to test ALL conditions in which X-Plane is being used. Each of our offices are littered with joysticks, laptops, desktops, loose video cards, hard drives and various other hardware. We do our best to always be making steps in the right direction but it's a losing battle to keep our testing internal. A prime example is the recent Joystick debacle. I rewrote the entire Joystick subsystem on 3 platforms and tested it on all 3 platforms on all joystick hardware I own, and Ben and Austin did the same. We worked out all known bugs and then shipped it...and immediately we found hardware that caused crashes 100% of the time. Once those were fixed, other devices were still broken...some devices didn't even meet USB compliance and we had to work around that issue as well. I can't possibly own every USB Joystick device on the market so how on earth do we get the product to be as stable as everyone wants it to be?
The answer is simple...we rely on ambitious and courageous users to take the plunge and help us test it. Being a Beta tester is a bit of a thankless job. Sure you get an early look at the new goodies but it can be very frustrating since often, you can't enjoy them without crashes, artifacts and other annoying bugs. Most users are happy to provide their feedback via the bug form, and now with the new automatic crash reporting system, we're finding and fixing crash bugs in record time. A prime example is the Joystick crash. Within minutes of releasing 10.10Beta 1, I was aware of the crash and had all of the information that I needed to solve it. And within 24 hours, we had a new build that took care of it. But that's only useful for crashes, it doesn't help us learn about visual artifacts or other problems. That's where we need YOU.
After reading through the forums, it's become clear to me that many people are just not exactly sure what it means to Beta test software so I've come up with my own list.
- Keep two copies of X-Plane. If you care to ever fly X-Plane for enjoyment, ALWAYS keep two copies of the simulator on your hardware. One on a stable release that you use for enjoyment, the other on the latest beta for test purposes only. Your sanity will thank you later.
- Don't use the beta for recreation time. If you're only in the mood to fly and have fun that night, stay away from the beta copy unless you don't mind your flight getting cut short. Too many users expect to fly for 6 straight hours and become enraged when the sim crashes on short final. That's Murphy's law!
- Always stay on the LATEST beta. Users often say "The latest beta broke XYZ, how do i go back to the previous beta?". The answer is you don't and you shouldn't. That defeats the whole purpose of beta testing! During a beta process, the code is often changing very rapidly. If we're on Beta 6 and you're still submitting bugs on Beta 2, you're wasting your own time and you're not helping the product. It's important you keep up with development.
- Blame us, not your system. If it worked in Beta 3 but it's broken in Beta 4, do NOT tear your machine to pieces trying to figure out the cause of the problem. Report it and let US figure it out. Time after time, I see users change their drivers, install service packs, uninstall and reinstall X-Plane, update their OS etc trying to solve the bug...when it's probably OUR bug...not your computer's.
- Beware of the placebo effect! _EVERYTHING_ affects frame rate. Time of day, clouds, location, aircraft, view angle, view direction, addons, plugins etc etc. Sometimes we'll release a simple patch that does nothing but fix one bug that was unrelated to the rendering pipeline. I'll poke my nose in the forums to get some feedback and see 10 users who all of a sudden claim to see some massive fps increase in performance and 10 users who see a massive decrease in performance. I role my eyes at the claims because it's not a controlled experiment. For example, if you're the type of person who flies with real world weather enabled, flying one day with clear skies may get you 60fps while the next day you may see 40fps with overcast. Heck, even departing from a different runway than the previous day can result in different frame rates. Running with the command line fps_test IS a controlled experiment. THAT's the only way to be sure you're seeing a change in performance. Don't trust your instincts, gather data in controlled experiments.
- Always read the release notes (http://wiki.x-plane.com/beta) for each new version. They'll tell you what bugs are supposed to be fixed as well as what new features have been added. If you see a bug that's claimed to have been fixed yet it's still happening to you, that's when you write a new bug report. If we didn't claim to fix it, it's probably not fixed in that particular beta.
- We need a Log.txt! When submitting a bug, ALWAYS ATTACH the Log.txt from THAT sim run. Even if you don't think the Log.txt is applicable, attach it anyway. Tell us in as much detail as you can what you were doing at the time.
- We already know about your crash. Because of the new automatic crash reporter, there's no need to submit a manual bug report for crashes! Let the system do its thing. If you remembered some details after the fact that you want to tell us, then go ahead and file a report. Of course, if the auto-crash reporter didn't load up for whatever reason, that's a good time to file a manual report.
- Tell us who you are and what you know. We really appreciate users who enter their email and comments into the crash report form. It gives us a way to contact you to get more information. This was invaluable to me when attempting to fix the joystick issues earlier in 10.10. A handful of really helpful users made the research a breeze.
- Be objective, not emotional. No one hates bugs more than we do but writing a rant instead of a bug report may make you feel better, but it makes me grumpy. And when I get grumpy, I attend seminars...and when I attend seminars, I feel like a winner. And when I feel like a winner, I go to Vegas...and when i go to Vegas, I lose everything. And when I lose everything, I sell my hair to a wig shop....Don't make me sell my hair to a wig shop.
- Stop staring at the fps counter! It seems that many users are obsessed with their fps counter. "It went up this beta. It went down this beta. It stayed the same this beta." Yes, the framerate is ONE useful metric to determine the software's performance but it's not the ONLY one and often it's not the right one. Turn it off and fly! Enjoy the simulator. Sometimes there'll be a bug in one beta that increases fps as a side effect because the sim is no longer drawing what it's supposed to. Suddenly, some users are thrilled to see a 20% boost in performance and perhaps they don't notice that half of the usual streets aren't being drawn any longer. Ben fixes the streets and in the next Beta, all of a sudden they lose their 20% performance gain and are outraged that we "broke things again."