Beta Testing 101

X-Plane 11.10 is the first major version release with a lot of code changes and new features. By design, this version has a lot of big changes in it that we've held back for weeks or even months. This also means it is more likely to be unstable and buggy, especially for the first few beta releases.

We've recently progressed to internal beta testing. Now we should have a public beta sooner rather than later, so I wanted to bring back out some great tips on beta testing that Chris wrote way back in 2012. Nearly all of the info remains as relevant today as it did 5 years ago, and I've included his best practice tips below.

  1. Keep two copies of X-Plane. If you care to ever fly X-Plane for enjoyment, ALWAYS keep two copies of the simulator on your hardware. One on a stable release that you use for enjoyment, the other on the latest beta for test purposes only. Your sanity will thank you later.
  2. Don't use the beta for recreation time. If you're only in the mood to fly and have fun that night, stay away from the beta copy unless you don't mind your flight getting cut short. Too many users expect to fly for 6 straight hours and become enraged when the sim crashes on short final. That's Murphy's law!
  3. Always stay on the LATEST beta. Users often say "The latest beta broke XYZ, how do i go back to the previous beta?". The answer is you don't and you shouldn't. That defeats the whole purpose of beta testing! During a beta process, the code is often changing very rapidly. If we're on Beta 6 and you're still submitting bugs on Beta 2, you're wasting your own time and you're not helping the product. It's important you keep up with development.
  4. Blame us, not your system. If it worked in Beta 3 but it's broken in Beta 4, do NOT tear your machine to pieces trying to figure out the cause of the problem. Report it and let US figure it out. Time after time, I see users change their drivers, install service packs, uninstall and reinstall X-Plane, update their OS etc trying to solve the bug...when it's probably OUR bug, not your computer's.
  5. Beware of the placebo effect! _EVERYTHING_ affects frame rate. Time of day, clouds, location, aircraft, view angle, view direction, addons, plugins etc etc. Sometimes we'll release a simple patch that does nothing but fix one bug that was unrelated to the rendering pipeline. I'll poke my nose in the forums to get some feedback and see 10 users who all of a sudden claim to see some massive fps increase in performance and 10 users who see a massive decrease in performance. I roll my eyes at the claims because it's not a controlled experiment. For example, if you're the type of person who flies with real world weather enabled, flying one day with clear skies may get you 60fps while the next day you may see 40fps with overcast. Heck, even departing from a different runway than the previous day can result in different frame rates. Running with the command line fps_test IS a controlled experiment. THAT's the only way to be sure you're seeing a change in performance. Don't trust your instincts--gather data in controlled experiments.
  6. Always read the release notes for each new version. They'll tell you what bugs are supposed to be fixed as well as what new features have been added. If you see a bug that's claimed to have been fixed yet it's still happening to you, that's when you write a new bug report. If we didn't claim to fix it, it's probably not fixed in that particular beta.
  7. We need a Log.txt! When submitting a bug, ALWAYS ATTACH the Log.txt from THAT sim run. Even if you don't think the Log.txt is applicable, attach it anyway. Tell us in as much detail as you can what you were doing at the time.
  8. We need different things for crashes on Mac vs Windows. Because of the automatic crash reporter, there's no need to submit a manual bug report for crashes on Windows. If you remembered some details after the fact that you want to tell us, then go ahead and file a report. Of course, if the auto-crash reporter didn't load up for whatever reason, that's a good time to file a manual report. Mac users on the other hand, need to send us a copy of the Apple crash report. These files are located in your username/Library/Logs/DiagnosticReports/. The file name will have X-Plane & the date in it, and will end in .crash.
  9. Tell us who you are and what you know. We really appreciate users who enter their email and comments into the crash report form. It gives us a way to contact you to get more information. A handful of really helpful users can make the research a breeze.
  10. Be objective, not emotional. No one hates bugs more than we do but writing a rant instead of a bug report may make you feel better, but it makes us grumpy. It also makes us unlikely to take you seriously or follow up on your issue.
  11. Stop staring at the fps counter! It seems that many users are obsessed with their fps counter. "It went up this beta. It went down this beta. It stayed the same this beta." Yes, the framerate is ONE useful metric to determine the software's performance but it's not the ONLY one and often it's not the right one. Turn it off and fly! Enjoy the simulator. Sometimes there'll be a bug in one beta that increases fps as a side effect because the sim is no longer drawing what it's supposed to. Suddenly, some users are thrilled to see a 20% boost in performance and perhaps they don't notice that half of the usual streets aren't being drawn any longer. Ben fixes the streets and in the next beta, all of a sudden they lose their 20% performance gain and are outraged that we "broke things again."

Being a beta tester is a bit of a thankless job, but we rely on our ambitious and courageous users to take the plunge and help us test each new version of X-Plane. Beta testers help us find those subtle or rare issues so that the sim is polished and enjoyable for everyone.


  • Facebook
  • Reddit
  • Twitter
  • LinkedIn

About Jennifer Roberts

Jennifer came to X-Plane to update the manuals and stayed for the bug testing. You'll most likely see her on the Q&A site or answering bug reports.
This entry was posted in Testing. Bookmark the permalink.

86 Responses to Beta Testing 101

  1. Bruno says:

    Nice post Jennifer, thanks.

  2. Steve.Wilson says:

    I agree with Bruno... must link to this directly from the .Org!

    It sounds like 11.10 is going to really knock our socks off, or knock our heads off our shoulders. (At least at first... 😉 ) Let the games begin (when yer ready)!

  3. Christian Novak says:

    Excellent post, thank you.

  4. Dan / DKFlight says:

    Well you have my attention...... Great Info!

    When ever your ready to let the code out the cage for the wild simulator animals to tear into like it's the last meal they "may or may not get to eat just yet," let us know 🙂 ..... I'm looking forward to test diving again and sending in reports as needed.

  5. Eric....... says:

    Hurry Ben, release the beta !!!!

  6. Vimal Gopal says:

    Are there are expected to be any more releases prior to 11.1? Or is 11.1 the next release?

    • Tyler Young says:

      11.10 is the next release—no 11.06 or anything like that. 🙂

    • wigry says:

      By the way, X-Plane always has two decimal places in the version number and hence version 11.1 does not make sense. If you think about the first version after 11..00 then that was 11.01 Now we are already at 11.10 while previous one was 11.05

      Also if we will reach version lets say 11.20 and 11.30 then they are not versions 11.2 or 11.3, they are versions 11.20 and 11.30

  7. Dainius says:

    Brilliant post!

  8. steman says:

    Let the show begin.... 🙂
    cheers

  9. Dominik Pospisil says:

    Can we expect that the new SDK will be released simultaneously with the beta? Many thanks!

  10. Gianni Caruso says:

    ready for testing 🙂

    Will the new "profiles" feature also allow to save "Control Response" and "Stability Augmentation" on a per aircraft basis?

  11. Ronny Heinz says:

    The biggest tease in history are the protected release notes 😉

  12. Peter andrego says:

    I don't know how you do it! The combinations and permutations of variables that can affect performance and 'bug free' operation is absolutely mind boggling! You not only have to fine tune your product but also ensure that it works in harmony with L.R and the myriad of other third party addons. It's like being a family member in counselling. Everyone needs to do their part to make things go smoothly and if one doesn't it often screws it up for everyone else!
    You have a very interesting job though and I am grateful for your endeavours!

  13. Moyses Torres says:

    I want to be a "Beta Tester" .

    • Jennifer Roberts says:

      We're trying to work more kinks out with a small, private beta right now. Once we release the public beta, anyone will be able to test it by running the installer and checking the box to get betas. Steam users can opt into betas in the X-Plane 11 Properties > Betas section.

      • ulises legarreta says:

        How can we keep two copies of XPlane if we are Steam users?

        might be a dumb question.. 😛

        • You can't, really, but Steam allows you to quickly switch between the release and the beta version, and the Steam client will switch you reliably. With the X-Plane installer, down-dating from beta to stable is a hassle you don't want to do.

  14. Manuel says:

    Ben said he could probably talk abut what is coming to 11.10. Do you still have plans to share a xp 11.10 overview about all changes, features in advance? or should we wait for the first beta release notes?. Very excited about 11.10.

    • Jennifer Roberts says:

      I don't think we will have many more announcements before the first public beta. We have some marketing plans lined up to cover some features more in-depth as we go through the beta process.

  15. Mihaly says:

    Just. Can't. Wait. 🙂

  16. Mike says:

    Could you remind users how to run two X-Plane versions with one copy of the scenery.

    • Aiden V says:

      You could copy your X-Plane folder and then put a symlink to the custom scenery folder in one to another. You'd use the mklink command on Windows.

  17. Mario says:

    Please work on the multiplayer connection please,some of us will fly with friends,like we used to in FSX, so make multiplayer more friendly rather then opening ports 49000 etc, please make multiplayer like any other platform ,look for a server and boom join in or create our own server in a way that anybody could join in without the need of knowing each other IPs
    Not like Vatsim,Ivao,Pilotedge etc,for that the average guy like me needs a lot of studying and time to actually enjoy flying in it,most of the time a bunch of friends only wants to get together and enjoy the xp11
    Reply

    • Tyler Young says:

      Internet multiplayer is unfortunately not going to be supported any time soon. It's a big project, and one we'd like to tackle in the future, but it's not on the short list.

      • Marius says:

        Might not be for you to answer, but didn't Austin say that one of the two new Laminar Research hires' focus on networking/multiplayer?

        Obviously does not mean that this is the focus of development anytime soon though.

        • Tyler Young says:

          Indeed! Jörg is our new resident networking expert. He's been killing off networking bugs—mostly surrounding the weapons systems—for the better part of 3 months or so.

          • Marius says:

            Awesome, thanks!

          • Elios says:

            he might want to look at why xp11 doesn't work with the android version of flight plan go wile xp10 does

          • Willzah says:

            Brief question on this, as was briefly addressed in one of the Q&A's, will the issue with multiplayer DE-SYNC be worked on before the major overhaul of multiplayer? Many of us have been enjoying a port forwarded multiplayer environment, however since v11 release there has been positional sync issues for the visual representation of other players that was not present in most of the v10 run. It is not a connection issue as Austin seemed to suggest in the Q&A, it is some other as of yet unknown (to us mere mortals) issue. Has Jörg made any headway on tracking that one down? Its a rather big buzzkill for lots of us who like to fly in formation 😉 Cheers and keep up the good work!

          • Tyler Young says:

            I'm not aware of anyone looking into this.

            The thing is... there are soooo many things that can go wrong between your computer and one across the internet. We don't offer any official support for internet multiplayer for that reason. If you can track down the issue and show it's not due to "the usual suspects" (e.g., high ping times between you and your internet-connected peers, DNS lookups times if applicable, etc.), we can look into it.

          • Ben Supnik says:

            Hi Willzah,

            If you can run the X-Plane 10.50 demo and latest X-Plane 11 on the same machines and you see different sync in formation flight between the two, please email Austin _directly_ to discuss what info he needs to sort this out.

          • Terje says:

            Hi, I saw there is a discussion on desync in multiplayer. I can confirm i have also seen this happen to me both with systems and aircraft position, those i've flown with have seen the same.

          • Willzah says:

            Hello again! So I did what I would consider conclusive testing that it isn't an issue with connections. I just tried it on all 3 home computers (1 MAC and 2 WIN) with clean demo installs V11 on our LAN (gigalan) all 3 systems had fps over 40, and default transmissions per second of 20 set in networking. The same exact issues are present. Desync for sure due to some unknown bug(other bugs present as well, altimeter/compass jittering, see here for example someone else's vid https://youtu.be/f1-kojVE2FY?t=355). The strange thing is that when all clients pause, the aircraft go to their actual synced positions, but when the simulation is running it is very out of sync, and visibly stuttering. With regards to v10 I used it for hundreds of hours with many friends some as far as thousands of miles away (Norway, UK, ect. while I'm in USA) in the past and had no sync issues(albeit depending on the release version, there were "phases" of bugs there too). So I don't really feel that would be necessary to test v10 again personally, because when it worked, it worked beautifully. Since others in my crew of about 10-20 online flight sim enthusiasts have certainly confirmed v11 has this issue independently of my own multiplayer testing, I think that the "cat's out of the bag" for us at least. If you would still like myself and/or others to e-mail Austin directly about this we can. Thanks again guys!

          • Tyler Young says:

            I'll look into the issues with positions not being synced across the network.

  18. Brillett says:

    any eta on 11.10 ? im really looking forward to this update 🙂

  19. Milan Popovic says:

    One question... When 11.10 will come out?

  20. riccardo says:

    x-plane 11.10 relased this weekend ? thank you .

  21. Marius says:

    I wanted to ask: has the new particle system made enough progress that we may be seeing it in 11.10 or is that almost definitely not going into this update?

    I would say it is now on of the few jarring things remaining in XP11's otherwise beautiful world.

  22. Mike Meriden says:

    What’s the password for the release notes, where can we find it?

    Also does that mean the public beta is released this weekend or as soon as you work out the kinks?

    • Jennifer Roberts says:

      You can't! 🙂 The people with private access to the beta at the moment also have access to the release notes. Everything will become public once we have a public beta. It doesn't look like that will be this week at the moment however.

      • Steve.Wilson says:

        It will be ready when its ready, Jennifer. No worries. 🙂 It's going to be "fun" enough with the bugs that do get into 11.10 for the user community to find. Stomp away!

        "We will release no beta before it's time." (With apologies to Paul Masson.) 😉

  23. David says:

    Hi, I can oly assume WED 1.7 will be released at same time, looking forward to teh new building blocks.

    Thanks

  24. Hiya, I noticed in the Cosford presentation there is mention of utilising Vulkan moving forward. For MacOS users, has any consideration been given to utilising a framework such as Molten (https://moltengl.com) to allow Vulkan apps to run on MacOS? Just a thought 🙂
    Thanks!

  25. Christopher Hallam says:

    hello, will the landmarks shown months ago finally release? ie Chicago

  26. Khaled says:

    Dir those who can’t wait (like me)
    Here is a nice preview https://youtu.be/xK2DItuHPws

    P.s. he said: 11.10 public beta next week! 🙂
    VR until end of the year.

    • vonhinx says:

      Geezes! Someone's been feeding Ringler! Ambitious plan--thanks for the link. Missed it as my main machine is down 🙁

  27. Richard says:

    next week is October 9 to 15, which is coming.

  28. Dong says:

    The frame rate is not only thing but the tiny stutter is the worst thing of performance

  29. Dan / DKFlight says:

    I'm excited about the updated physics adding more wind forces and supporting aircraft flying backwards to help with helicopters and tail slides, etc. In terms of aerobatic maneuvers and gyroscopic motion. Will the updates support more of this type of maneuvering? Or any future updates?

    It's my understanding that many of these gyroscopic maneuvers are not yet supported?

    Thanks...

  30. Milan Popovic says:

    Can anybody tell us what is the release date of 11.10? Is that day October 9, or 10, or 11??? Please write us an answer. Thank you!

    • Jennifer Roberts says:

      We can't tell you because we don't know yet. We're still testing internally and we'll release the public beta when we have it ready.

  31. Paulo says:

    Quick question. If i would like to test the Beta and at the same time to continue to have 11.05 release, how can i do that in a MacOS? Is it just copy the 11.05 to a different folder and run the installer from that same new folder?
    Thank you,

    • Jennifer Roberts says:

      On any OS you would make a full copy of your existing X-Plane install, then only update one of them to the beta version.

      • Paulo says:

        Thank you. And regarding the digital license, do I need to do something to have the copy working?

        • Tyler Young says:

          Both installations can use the same product key. 🙂

          • Pablo Bressan says:

            Is it legal to run 2 xplane processes with same serial? Same pc. I understand that's not legal. Am I wrong?

          • Jennifer Roberts says:

            You would need two separate licenses if you want to run X-Plane at the same time on different computers, and network them together for example. It is ok to have multiple versions/installs of X-Plane on one computer and use one key for them all since you would only be running one at a time.

  32. Axel says:

    While we are waiting for the 11.10 update, can you tell us which of the default planes will be equipped with the new G1000? Thanks.

  33. Toufic Zeidan says:

    With the Vulkan implementation will the new ryzen cpu or intel new coffee lake have any advantages over one another? Thank yoi

  34. Milan Popovic says:

    I count down till the new X-Plane 11.10. Is it going to be tomorrow??? 🙂

  35. Axel says:

    Do we need to download a new version of the X-Plane installer to install the upcoming X-Plane 11.10 beta, or can we use the same version of the installer that we used for 11.05?

  36. Milan Popovic says:

    Jennifer could you tell us please what day the X-Plane 11.10 could be download from server? The release day, please 🙂

    THANK YOU Jennifer

    • Karsten Schubert says:

      I would guess that Jennifer can´t give you what Laminar doesn´t know.
      When the product is more or less ready depends on the number and severity of the new and known bugs, not on a date.

Comments are closed.