Sigh. Well, foo.
I was going to write a post about X-Plane 11.50 beta 5 – what’s new in it, the new ways we are debugging GPU crashes, the crash bugs we’ve fixed, etc. A lot of stuff that we thought was pretty good went into beta 5. Cool new technology! Big bug fixes! Lots of winning!
As it turns out, beta 5 is dead. I hit “go” on the release this afternoon, and half an hour ago, I hit “stop.” The auto crash reporter was showing way too many new crashes in memory management that we had not seen before, and this strongly implies a new and serious bug.
Laminar Installer Users: if you were auto-notified to update to beta five and did so, and you are not crashing, you can keep flying! If your beta five is just a smoldering wreckage of crumpled VRAM and GPU parts, you can re-run the installer with “get betas” option checked, and it will take you back to beta 4.
If you were not auto-notified to update to beta five, that’s probably for the best. Please stand by and keep flying beta four; we’ll post a new beta when we’ve gotten to the bottom of this. We have enough captured crash data to investigate.
Steam Users: we did not release the beta five build to Steam and this is probably a good thing; we’ll try again with a new release that isn’t made of plutonium and unicorn hallucinations.
And if you’re going “why didn’t y’all test it before you released it”…we did! None of our machines show these crashes. But we also have probably a dozen PCs total we can run on. Moving to a new driver stack has meant learning about the weird things that happen on your computers and not ours.
Do We Need a Two Tiered Beta System
This came up in our impromptu beta five post-mortem meeting: do we need to bring people into new betas in stages? With code for new drivers, beta five probably won’t be the last beta where we code something we think is helping and discover that it fails catastrophically, but not on our hardware. We need beta victims^H^H^H^H^Htesters to find these bugs, but once we get a dozen crashes, we don’t need anyone else to stub their toe for us to fix our problems.
So we thought about two possible ways to do this:
- A two-tiered system. Early adopters could get an email and hand-update to the new beta before it is put out for auto-update notification.
- Send out the beta update notifications over time, e.g. 10% of users get notified immediately, then another 40% if we don’t see crashes, then the last 50%. (This practice is actually industry standard on mobile apps.)
If you are reading this blog post, this far down, you are probably participating in the beta; I’d be curious what approach you’d find most useful.
#2 seems easier to manage
I was one who went to 5 and t died several flights in a row and kept crashing. I’d going with the 10%, 40% etc. plan to try new betas. Thanks for all your efforts.
Send a notice it is coming..
Either set to auto update and hold on – or – select a time that you feel is reasonable.
Don’t waste Laminar cycles managing beta users. They take on beta as a risk.
Number one…a bit more controlled…
I think you need A two-tiered system.
1st. a beta tester is not a regular user, neither a early adaptor.
A beta tester is a tester that act as user.
A regular user does not like software bugs, he hopes that the software will work as he ‘desperately needs’ the new features.
A tester like the software bugs, he ‘needs’ to find them because thats his job.
But honestly I don’t think 11.5 is at beta stage, to me, as a software developer, looks more at alpha stage.
It’s odd, every other beta has gone rather well. I am out of the beta now, on the 11.4x platform, more stable for me. I have rave reviews for Vulkan’s handling of photoreal scenery in Aerofly FS2, Vulkan can work. My beta problems were with Ortho scenery for Xplane 11, and I have a lot of it, level 17 in California, and 18 surrounding the Phoenix area, I have lived in both places. 11.4x handles my Ortho well, 11.5x no longer does.
My GPU is short on VRAM, only 3GB, but it has never griped while I have run Xplane 11, even with Traffic Global, so the core of XP11 is written quite well, in my humble opinion, for me to enjoy success with the product for the three years that I have, my fav sim by far especially because the aircraft remind me of real flights I have flown and respond the way aircraft do in the turbulent windflow if my conditions are set up just right
10% 40% 50% good for me.. I’m for trying 10% by the way.
I get very smiley each time a new beta turns up. Keep ’em crash– er.. coming!
I believe that is the best standard well shaken down in the industry with two caveats:
1. Randomize (removing recent targets) the 10% so “the load is shared”;
2. Also use a two tier so that folks can sign up as:
a. Earlier Beta Adopters with standard beta warnings; or
b. Volunteer Tester with “you will likely crash and find bugs” warning
I think keep going like you did already. Beta testers should expect crashes. Thank you very much for your great work!
Sitting in the Zibo in VR is just incredible! 🙂 forget sometimes that I’m at home 😉
I would love to be part of an early adopter beta program. I have been running two (sometimes three) installations of x-plane – my 11.41 copy with all sorts of scenery and plugins and a stock version with the beta build. So I think the tiered beta system is good. As long as those participating in it don’t report issues due to plugin x or scenery y.
I like the two tiered system. Choice 1 for me.
Idk how the Steam edition will work with it though
I’d say, do both! Start with voluntary opt-in, just ’cause if one accepts a risk to contribute to progress, at least it’s their choice. It would be interesting to see if you get better quality feedback from folks that agree to be on the bleeding edge. Then do the randomized rollout to everyone else that ticks “beta versions”.
Great to see the prompt disclosure, handled with humor and openness!
Leave it as it is, using betas implies the risk of a malfunction… I wanted to do a quick sortie, having a look at beta 5 but it did not work out… so what, it was a try… one of the last adventures today 😉
Keep on going!!! 🙂
I agree with this.
I agree with this too.
Betas are for the brave, and those (of us) participating should know that already 🙂
Benefits if it works with new specs or goodies, meh if we have to return to an earlier version.
That’s how it should be I guess 🙂
I went also back from 11.50b4 to 11.41 after more and more crashes. But its really depending on the system and installed addons.. a friend of me expirienced not even one crash in whole beta.. and i cant fly an one hour flight without crash… so we have to wait 🙂
I agree with dump the Beta in. It’s at your own risk.
Vulkan has caused me a lot of unknowns and crashes, but loved it and finding out what works and why.
Unfortunately I have seen the reviews on Social Media and YouTube. So I guess it’s risky.
But from a stuck in his ways Prepar3D user ( 15+years with MFS and Prepar3D) I am now a believer. XPlane all the way..
I fly VR only now.
I think 1 is the best!
I want to help testing on Win and Mac!
Out of sheer selfish need, perhaps, I’m happy to keep testing as we are. I realise it’s Beta, I do this for fun, and don’t require to have a rock solid working X-Plane install to support a business or anything like that, and I enjoy testing new tech. So option 1 for me, please!
I agree with 1 Beta too.
Here’s a novel idea….How about building into X-Plane the option to “Revert to Previous build” option? If, for whatever reason, a user is having issues it would be a simpler fix allowing for fewer installs and configurations.
I’m curious about this as well. Seems like a good way to handle it.
I am enjoying the betas as a second install. I had no problems with b5, but I only used it for an hour or so with the LR Baron and then the DA-62. The DA-62 is my only add-on.
I think the 2 tier system sounds like a good idea. And I would not mind being part of it.
You need computer savvy simmers running a second copy, who won’t post “its bust” all over the user groups when a tricky bug comes along.
“Hold my beer”!
I would say that option 2 seems fair to me. If someone is really eager for testing the new beta, he can go and manually check for it (you know what testing a beta means).
Then, for the “big” audience, let’s keep sending the notifications over time.
I agree with this one!
I second that!
My vote is for #2 also.
The current beta thread on the .org forums looks like you could do w/o a tiered system, more people testing -> more noise but also more legit bugs reach you guys sooner. 🙂
BTW, I’m not getting any (new) crashes but heavy flashing artifacts flying over ortho+overlay in VR (just saying so you guys are aware that this would make me a very sad panda, m-kay? :p). Bug report is on the way.
Having a pretty loaded machine with many plugins, I would be happy to help in finding bugs in this virus time where all I do is fly all day 🙂
I guess the beta over time approach is a fair one and more important not a great workload on your side.
So I guess that would be a good way
1st option sounds great. 2nd option would create “feels bad man” emotions when almost everyone’s enjoying a stellar new beta and you happen to be among the last to get the patch.
It would depend on whether the ‘wave’ is for the notification or for the ability to get the beta. It’s the auto-notification that is driving adoption so we could look at _just_ making that a roll-out but anyone gets it if they go and manually chekc the beta,.
+1 like for this approach Ben !
Keep up the good work
Me too! Folks will make conscious decision to get the beta including the risks. Vulkan is good btw
I also like this manual-check approach, not too high & not too fast ;=}
I see. 2nd option without a hard lock for the patch sounds good to me. Thanks for your efforts Ben & the rest.
Ben, I prefer the 1st approach, and count me in as an early adopter.
I hadn’t been able to enjoy very much of Beta 4 currently because of the Windows 10 Kernel but that causes the stutter.
So, whatever you guys decide, just count me in for testing.
Definitely number 2. When I try to downgrade it asks me to overwrite the “old” files, but keeps asking me for each one separately. If I check “Do this for all files in the director” it does nothing.
I think you’ll find it does precisely what it says — only the files in that one (sub) directory. The “problem” is that’s one question for each sub directory under global scenery. An option “do for all directories” would be nice in this case.
Should we test in tiers ? Interesting question.
As with previous beta runs, there is a pattern, beta[n] good, beta[n+1] well uh you know. So it is probably a good idea to test on a small sample first. However there is also a given, a lot of people jumping into the beta run are those craving for performance because they run a lot of plugins and are completely oblivious to the idea to remove them before testing. That is what I also saw in the few logs which have been posted in the last hour on the .org.
In my opinion the 2 tiered system would make the most sense, especially if you can control who is targeted at first. Perhaps you could build a small controlled sample (200-400 volunteers) composed by 60% of people known to run a “clean” system on the different OSs, and 20% running with plugins, 20% with “exotic” systems (low specs, multiscreen, heavy multiplayer, very personal input devices …). So you would be able at once to sort what is related to system issues, and what is related to ecosystem issues ?
Looking at the emotional responses you get from the current beta system I think jrKok has the best suggestion – basically an expanded private beta with people who understand what a beta really is. That would give you far less heartache!
#1 opt in. Send an email and off we go. Look at me MA! I’m a beta tester …….. crash!
BUT, steam users in the opt in beta too? Probably not right?
We could – with a password you can get those early.
My buddy ho did update to b5 and is now rolling back to b4 reports the installer is replacing very laboriously every single Earth Nav folder one at a time.
Not sure if this points to anything?
That’s normal, roll-backs always see the new files as foreign – pick “replace all” to speed it up.
I say keep it the way it is. Beta is industry standard for “this may blow up.”
If you change anything, maybe just make reverting changes quicker by caching the changed files before they are swapped out?
I think the current beta program is fine as it is. If you are running beta only to get the latest and greatest, you’re doing it for the wrong reasons. All throughout 1150 I’ve been dealing with crashes everytime a new beta rolls but then I seem to tweak a few settings and gradually roll back into heavier settings and it will work fine. Today I must have caught that 1.5 hour window and ended up with beta 5. My system crashed a few times and then took off and is currently running fine. The mystery to me is that my texture resolution slider is missing a few options. Before, it would load maximum(no compression) and max now there is only maximum but instead of loading up massive amounts of texture, it only loads about half of what it used to and I get blurry stuff. Still no complaints merely observations. I love seeing all of you guys hard work and I love scrambling to keep up with you. Carry on!
I would leave it like it is. Your beta system has been working fine for all these years like this. If a beta release turns into a catastrophe then you just recall it like you did today. Beta testers should just be using a second copy of X-Plane and not primary. If they update their only copy and it craps out, that’s just too bad!
Real beta testers already do that.
well am one of the victims ,probably u got 6 crash reports only from me lol
,i kept it though and seeing FPS never saw before, did u do something or its placebo effect idk, why dont you share the exciting new features until its out?
i’d vote for option 1
thanks for all the hard work
I think tier one beta testers should be those who aren’t running 3rd party aircraft or a ton of plug-ins. 90% of the Facebook complaints are from those who run the beta expecting to see all their 3rd party aircraft and plug-ins working flawlessly in the beta. If you separate into camps, you can have a better idea of internal on external causes for failure. I’d be in the tier one camp. My 0.02
This is a really good point. I only fly stock, so it can be annoying filtering out peoples’ issues related to plugins and addons.
Your current approach works well. Stick to it.
As mentioned by others – you’ll likely also benefit from more actionable and meaningfull bug reports.
So build your ‘early adopters’ list based on prior, usefull bug reports received.
+1 on this. The better the bug reporting the better the testing. Give the tier one testers a couple days and if all goes well open it up.
Choice 2 (industry standard) looks good. Interestingly, having been royally hacked off by lost device crashes in b4, I loaded b5 in an idle moment onto my test installation and had an immediate crash that seemed to be caused by the previous b4 automatic crash report not having been sent properly. Once I restarted XP11. 50b5 I was able to fly a couple of circuits round my home base with no issues. I have not yet attempted an airline route involving flight longer than 1hr…and this is when I always got the lost device errors in previous betas (like a memory leak issue over time)
Definitely 1, a canary channel, an alpha channel, a beta channel, I really don’t mind how much you’ll make. Option 2 is frustrating, because it’s random, and you’ll get a lot of ‘did you already get the update notification?’ amongst groups of people that usually share their experiences, and their beta testing feedback. Don’t do that (please).
I’m open to early testing. As a VR user , small changes can sometimes make a huge difference. I don’t mind being a Guinea pig
My 2c
checkbox: alpha
checkbox:beta/dropdown steam beta
uncheck alpha revert to last beta, uncheck beta revert to last release
everyones happy?
I think % roll out probably only useful on beta (and you just kinda did that anyway), if automated rollback is to much distraction email would be fine
I personally don’t think over complicating things is the way to go, even with the new approaches you could still have run into the unicron scenario. If having to choose though, I’d say option 2, but you’ll have people asking to be part of the 10%. At least option 2 maybe easier to manage on the back end?
If you go with option 2 you will have loads of people complain because they want to try a new beta but we’re not on the initial release list. Go with option 1 or stay with the current system. We know its a beta and things can go wrong. No problem.
Canary deployment (2nd) is a good option besides being a standard in software update.
Or, as mentioned above, leave it as is, people opt-in to test a beta software they should accept the risk of a regressions happening.
Whatever is easier for you! Number 2 or just keep releasing betas to everyone.
If you are seeing radically different issues e.g on mac or linux or windows compared to the other, then have two tiers. If you are seeing generally common issues, stick with one tier; two tiers will be more to manage for you.
Given the cross platform nature and smaller percentage of users on Mac, linux. Plus the small VR contingent for example I think option 1 is better. You can target recipients who best fill the spectrum. With option 2 there’s no guarantee you’ll hear about a major blocker for one of those use cases until it’s already widely released.
Choice 1 is the way to go. Choice 2, as soon as the word about a new beta spread on some forum everyone goes to download it, even those not notified. And if they are notified but they don’t find the new beta on the installer, they start moaning.
Both have merit.
In the words of Sir Winston Churchill, “Americans will always do the right thing. Usually only after exhausting all other possibilities”.
I’d love to be one of the possibilities…
Would love to be part of it and help testing and finding bugs.
Option 1 looks more appealing.
But we already have kinda tiered system. If you brave enought, you update right after new beta release. If you want to wait for few days, you wait.
My 2¢: I would leave it as is. Why add another administration headache about who to include in early betas (everyone wants to be in early beta anyway). Yes, more noise but as mentioned, you have a more complete overview quicker. If it crashes and burn then it crashes and burns. If people can’t handle that they should stay with 11.4x.
Cheers.
I think option #2 would be easier from the software development process. Meaning a random 10% gets the update and you slowly increase the percentage over the time period you specified. Why do I think #2 is easier, because as I understood #1 would require user’s action to sign up for the early adoption meanwhile #2 you are in complete control. Also, I’d prefer to have the random 10% (or whatever number you like) to eliminate any biased testing. This is just my opinion.
🙂 🙂 🙂 Shit happens, and it’s good you recalled.
I’m one of the victims.
For hours, mostly after completly setting up my Zibo to now contact online-atc being “ready for push” I … entered my email-address and sent the bug-report 🙂 Should have kept the mailaddress in the clipboard, haha.
Meanwhile I had a look into the blog but nothing, so the problem must be on my side as Beta4, I must say, worked amazingly well for me.
At the end I gave up and wanted to prepare for bed (Germany). Ok, one last check on the blog page.
And there it is!
Hey, we are using a Beta and we did it (hopefully) knowing that we might not get “terrain, terrain -pull up” warnings before crashing.
That’s the way a Beta works, haha, beside the word “works” might not be correct here 🙂
I wish you “many happing findings” in the crash-reports.
Funny … as stated in the B5 release, I thought you wanted to have better crash reports, not more ?
Making mistakes also creates knowledge.
As long as you are able to recall a faulty Beta, I have no problem with it.
It would have been nice though having been informed on applikation start that b5 is recalled and b4 is active again.
Anyways, keep up the good work.
Kind regards
Güner
Maybe in opposition to common sense, but I think, to leave everything as it is.
Of course, there are a lot of inappropriate comments and complaints of people who treat the beta, like the new version of XP and run with a thousand plugins, but even those topics discussed in the forum sometimes bring something meaningful to the general discussion 😉 .
Maybe post a notice and a link to a new beta version. Leave up to users to decide if they want to try a new beta initially. Some may want to wait for information on stability but I believe there are enough early adopters that will be willing to test it first rattle out of the box. Just my two cents…
The problem with having a two tiered beta approach is your sample
Size of machines and hardware in relation to the broad spectrum of computers out there you would have to either keep changing the pool of testers if they upgraded their kits on the fly without telling you, potentially having too many with similar specs as a result. I mean, lets face it no one really downgrades their sim computer. A staged release is probably logistically more ideal from a developer perspective, not having to keep a broad spectrum of testers and constantly decide when to change the pool.
On the converse to this, if you released to the first 10% but only 10% of them were able to actually beta test it in time, you could still have a flaming pile of poo to the next 40% and not know
It. It’s schrodingers cat i guess.
With a gradual roll-out, crash reports is the feedback loop. If we open it up to 10% and we don’t hear squawking, it either means:
1. The beta is great! Add more people or:
2. Most of the 10% aren’t paying attention. Add more people.
Either way, you slowly add people until either you reach 100% or the ship hits the fan. No question there’s a risk of low uptake though – we see that with mobile betas, for example….beta users are enthusiastic at first and trail off and we have to add new ones…that’s just software.
I had no issues so far with beta5 installed with no Ortho or LUA. Flew the default C172SP. Three monitors with stock GTX1080 (no over clocking) and i7/7700k (no over clocking) 48GB RAM. I’d certainly be up to test since I’m retired and software testing was my regular day job.
Would it be a viable option to have another checkbox or system, for ‘early insiders’ who wish to opt-in to be the 10% and use the current checkbox for the ‘stable beta’ so to speak?
No matter how much you try to tell people about Betas Ben, they insist on downloading and then complaining because this or that didn’t work “right” – whatever they assume “right” should be. Beta testers not working with a “Plain Jane” copy (or two like I do – an A and a B copy to “leap frog” through. If the current A is working and B is not, use A until the next one comes out and overwrite B at that time. Never had a problem that way and you can keep one with your downloads to try, but be prepared. I you loose interest, you probably shouldn’t be using Betas anyway. I’ve probably stepped on some toes, but griping without submitting “bug reports” gets us nowhere. Thanks Ben and Team.
Somewhat off-topic since I am not answering your question (I have 3 zXxP installations, 2 with the stable release and one of with the latest beta, the latter for helping with bug reporting to LR and others) and will live happily with whatever decision you make. I have a question for you, however: I did install b5 and launched it a few times in the process of helping someone test a plug-in meant for 11.50; in only one of the launch did XP stop responding and apparently crashed (as it says so in the log.txt) and produced a crash report and dump file (but no auto-report-to-LR). I still have those 3 flies (log.txt, .rpt and .dmp); would you like me to file a bug report or do you have enough information already?
Whichever way you choose, I am OK with it… as far as I get to try the beta when it is available. I’ve it installed both on its own without anything else, and with add-ons (other planes, scenery, apps…), so I can see how it behaves in the two situations. Crashes? Yeah, I don’t mind. Seeing the sim evolve is part of the fun. That’s my two cents, other may feel different.
I guess the 1st option is the best, but i would like to be one the beta tester 😀
Maybe you can have your new versions first tested as a private beta, and if ok have it tested as a public beta.
Well, method 2 seems ideal for the rough, but when fine tuned, method one might be better as small things will show up (more probability). Does that make sense ?
I’m reading the dev blog every month trying to understand where you are at, but happy running 11.35 – So, I guess I spend a lot of time integrating all kind of devices and happy to enjoy your marvel (The sim !).
Good luck, – I am awating for a major leap sometime. Let me know when.
I have 3 different versions of X-Plane in separate folders: 11.41, 11.41 with a private fix for a RealSimGear issue, and 11.5 beta. Installing and testing betas is not a problem for me, I’d be happy to be an early adopter. 35 years of IBM software development experience helps make me an objective tester.
Personally it’s a beta, we usually have dual X-Plane set ups for running v11.41 and the beta, the idea is that if it crashes then there is an issue, as long as we can step back to last beta version b4 as it has happened here in b5 then what is the issue, putting forward a test version to some or a few is going to create a lag to everyone else, plus you are not going to get the wide net effect to see any hidden issues as core users usually have standard (and powerful) setups. If it crashes like b5, we know a fix is coming soon anyway… you sign up for this situation when you access the beta?
Ben,
To me, it’s not about which users get access to what, and when. It’s about helping your team quickly get through the issues. I’ve been running test builds this week with Sidney trying to help him with his Device Lost issues and it’s cool to feel like I’m contributing to helping him find bugs that are not easy to reproduce.
Do what you need to do to make your product better quickly, so you can release it to the wider audience, and move on to new feature/product development! If salting in test builds helps you get things done faster, identify the users that can help you move forward, and do it!
Best of luck!
They do do a two tier system. Austin has repeatedly said on good YouTube videos that those who purchased it first from his website get the betas first and then Steam gets the update pushed 2-3 hours later.
#1 makes the most sense imho. At least that group will have agreed to be a victim and knows what its getting into.
with number 2 you get unwitting victims.. er testers (although since they are less likely to read this blog, it may be a purer test)
method #2 makes more sense just before you are comfortable to go back to normal beta release mode (like b4)
Sign me up for guinea pig squadron, even if I crash. You know you will get my crash logs anyway.
How about a tier within the software itself: early betas to the public disallow all plugins + ignore all custom scenery. Later betas allow plugins but no scenery (or scenery but no plugins).
That would discourage “gamers” from jumping into beta too early & allow a more structured set of tests. Plus a configuration (in a *.prf) might re-enable plugins / scenery so more advanced tester can get access.
During the “no plugins beta” stage, you can quickly triage Log files ignoring those which include any plugins.
(I’ll assume there are other “feature selectors” which might be similarly useful — custom shaders, non-LR aircraft, etc.)
I think option #1 makes the most sense. A couple weeks ago I wouldn’t have liked this option, but after reading posts on the .Org I now realize how many people treat a beta like a full release. I’m coming around to insider programs being a necessary evil.
For #2, are you accounting for how fast news of a new beta spreads on the internet? You may get a lot more downloads than the 10%, 20%, ect by everyone who gets a notification when somebody posts about it.
#1 Sounds fine to me Ben
What’s a simmer to do? Locked up, quarantining on broken beta’s and left over pizza. Just not good.
Doug.
Probably you would create a raw “Nightly Build” Opt-In Option.
Meanwhile, you continue the current beta release cycles per current norms
Check Icao: SNZA, i am trying to fly around that area since beta 4 and i just get no scenery load in the area. I have tryied to add ortho in that area and that didn´t work also.
I think that can be a bug.
I also tryied to delete and add new global scenery folder, nothing changed.
Ive tryied to fly around that area with c172, zibo 737 and ff 767.
With flights from SBSJ TO SBC.
I also tryied to delete and add new global scenery folder, nothing changed.
Ive tryied to fly around that area with c172, zibo 737 and ff 767.
With flights from SBSJ TO SBC.
From SBSJ to SBCF
I feel like my little baby getting learn how to step/walk and this makes me happy when you guys publish betas. Keep going on this way we don’t judge you, actually we like your honest and hard work including sense of humour. I believed that LR will overcome all these problem as soon as possible. Cheers.
I think you guys are doing more than enough with the opt in beta option.
If you’re going to go with option number 2 I see the following happening: Rapid reporting of new releases by users in the first 10% + social media networks + impatient users in the other 90% = a flood of “hey how come I didn’t get the update but the update is out now? This is BS!”
If anything, perhaps some annotation on the updater that says some warning if the auto bug tracker gizmo is seeing an abnormally high report of bugs with a new release and a spot to pick the current new update or select the previous update so users can choose to revert back if they want to.
But to reiterate, I think you guys are doing enough as it is. Thank you you all!
Ben, please allow us, Steam users, to use the latest beta too.
As the name “beta” implies, it’s a version to test new features/optimizations and report bugs. Users that opt in to use betas ARE aware (or at least SHOULD be aware) that crashes can occur.
Other games have many options in the beta comboBox so I can’t see a reason that you can’t publish more than 1 beta version.
Just keep the last working beta and the “develop” beta version.
Thanks.
Number 2 seems a lot easier to manage, faster and you can get a much larger sample of people testing, even at 10%. So it makes more sense to me.
But 1 also makes sense, because you would have a sample of people who actually know that they are Beta testers, and that they should report bugs and wait for them to be fixed, instead of whining on social media about XP being broken
So it could be a good idea if Steam users could also opt-in a private Beta, and that would cover number 1 for you, without the need for you to keep sending emails with every update.
Well, opinions are like belly buttons, everybody has one. And, to show off mine, I think that it should actually depend on what is what you and your team needs. I might be mistaken, but I’d say that the bigger the base of bet users, the bigger the chance of catching most of the bugs that might relate to certain specific configurations. Now, I’d say that it is impossible to fix EVERY SINGLE bug out there, but as you start receiving the reports, you will be able to determine your “80-20” curve. Perhaps I would suggest that, in the bug report section, you could add a drop down menu with certain specific areas that might help you to classify the bugs. Like “blurry texture” or “memory crash” or “plug in crash” (I just made up those sections, but I know you get the idea); that way you might be able to focus on those repeated sections, and not dedicate too much time to a bug with a user that has a 80486DX2/66 with 4MB of RAM and a VLB ATi Mach64 that runs XP11 just because he want to know how it feels…
If PR is a concern, we would understand a move to a more controlled opt-in process (which would definitely make some of us sad if we didn’t get in). Otherwise, leave it as is and sleep with a smile knowing you literally hold the dreams of many thousands in the palms of your hands!
That said, making a user provide their email address to receive a link or password (a la steam) to each beta (that could also come with huge font warnings of impending failures and tears) instead of clicking a cute little box in the installer window, *might* deter some of the more emotionally-charged users.
Having a tiered beta release is not needed if you give the option to easily roll back. I did the update the second it came out and I am one of the lucky ones that have not had many problems. 1 crash so far. If you do a tiered release it just delays how long till you guys will find out there is an issue or if its a one off. Couple things people should remember, if you’re doing beta you should have a non beta install so you don’t miss out on flying if there is a big issue. 2 it’s beta and I update it expecting it to have bugs, crash, and possibly not work. Thanks for all the hard work you guys put into the product and for doing it practically for free. I have told many friends who complained about the crashes, hey this is a free update to make the simulator run much faster, no other product gives this level of support and doesn’t charge an arm and a leg for it. ps. if you do a tiered system, count me in as an early adopter!
You should have added option 3, “Leave it as it is”. Most people are assuming they have to choose 1 or 2 . I’d say leave it as it is.
I don’t know if the installer gives you the option to rollback to any given version, but if it doesn’t and you are going to change anything, I’d say making that option available would be the best way to go. But really, I think time spent on squashing the bugs is better than time spent on this infrastructure stuff.
See, nobody is mad at you 😉
Who could still say, despite all your warnings and publications, “Reallyyy, I could break my copy of X-Plane with a beta? No wayyy!!!”.
That being said, if you are willing to improve the way we, users, enter beta phase, how about this idea:
Why not having several types of splash screens :
-“Beta X was released less than 2 hours ago, blablabla special warnings, please hop in with a clean install only, no third party addons etc…”
-“Beta X was released less than 24 hours ago, blablabla special warnings”
-“Beta X was released more than 24 hours ago. Blablabla, as always prefer trying it on a second copy”
This way, you would always get your best 10%, then your best 40% and finally the last 50% of your beta testers.
And if you need to boost the early stages, I would actually like an option in X-Plane saying:
“Yes, I am potentially the front-line kamikaze you are looking for, please notify me as soon as beta is released, call me on my mobile even if I am in the middle of my family dinner, here is my email address:”
This would actually be a mix of options 1 and 2, with everybody being notified at the same time, no complain on social network.
PS:
-I would also like the option “overwrite the old files in all directories” when reverting to previous beta or previous stable version.
-I like the option “remember my email address for the next crash report”.
I’d rather you leave things as they are. I keep three xplane versions: (1) the current release version, (2) the last beta that worked well for me and (3) whatever the latest beta is. I really enjoy the betas and would be disappointed not to have access to them. If folks are getting aggravated that the beta crashes they don’t understand what a beta is!
the percentage proposal is a very good approach. For my part and since I fly with Xplane I practically used all your beta versions, this allows me to better understand the evolution of your program. Jonathan
Keep the current system. We are dealing with humans. They want the goodies IMMEDIATELY and they never consider the risks.
We already have opt-in. It doesn´t help the frustration. All these people moaning about the crashing will sign up to be part of the “exclusive beta test circle”!!
And if you roll out the beta to just 10%, the remaining 90% will cry havoc on the forums for not getting access.
You can´t win 😉
To elaborate: At work, we have a very similar system of crowd beta opt-in. We found that the cost/benefit ratio of a tiered system is low, managment efforts to find (and keep) volunteers that consistently bring the kind of reports we’re looking for, then putting more work into rolling out the stuff in tiers isn’t worth the hassle, it basically just costs time for a number of reasons. Only at very rare occasions, when we really anticipate stuff blowing up in our faces, we put stuff on a PW-protected server and pass the link+PW to a number of people we currently know to be bug-tracking hounds. That doesn’t really avoid any disasters but gives us a feeling that at least we tried. 🙂 The point is, if that happens what’s the harm? It’s beta and we just want to know what’s wrong with it ASAP. It blew up within an hour? Awesome!
I think a combination of the two would be best. A group of 1st level testers who are actually testing and reporting properly to start with. (I would be more than happy to be in this group 😉 If no major problems then switch to option 2 and roll it out to everyone else. That way you will slow up the silly complaints and get the more meaningful reports earlier.
Option 2 would probably give you feedback from a wider range of different HW setups than with option 1. Otherwise keep it as it is.
Sometimes it’s a good idea to not start the x-plane vsak everyday
Agree with the poster who says leave it as is. It’s a beta, after all. If a release is lousy, well, so were all the odd-numbered Star Trek movies–and they were much costlier and slower to iterate. Ya gets yer beta, ya takes yer chances, I say.
Id be happy to be beta tester, have backup, but will only fly the beta, dont have to many addons or plugins so have less issues to deal with. If it crashhes ill go back to finishing airports Thanks for all the hard work
Agree with leave as it is. As many bets’s running as much data you are able to collect especially with the diversity of machines around (Windows, Mac OS, older computers and new computers). Beta is expected to have flaws and flaws could be translate into a eventual crash.
Two-tiered Beta test sounds better than, as today, sending it out to the masses and of your options #2 is probably the best way to go for several reasons :
1. You will get a greater varity of hardware setup that you´ll never get even close to either if you do this “at home” at Laminar or with a solid smaller groupe of “Hardcore Betatesters” that will most certanly be sitting with the same hardware from time to time.
2. You will get a faster and much more accurate feedback that more specific points out real problems instead of getting loads of bugreports concerning bugs that only is related to the users own special setup (both soft- and hardware) and not the maincode itself. Thoose users bugreports will appear sooner or later anyway but for say 80-90% of X-plane users you can release an update that will run soft, smooth and better than ever.
3. You´ll get a “three step rocket release” where most of the issiues connected to the maincode itself will be sorted out before you even reach step three and focus, for your sake, can be set to 1: Maincode and major hardware (Intel or AMD CPU, Nvidia or AMD GPU, OS (Windows, MAC or Linux), 2: Payware and established addon related issiues and 3: Issiues related to simmers use of freeware (and combinations of all types of software) and there special hardware.
4. You will probably get much less nagging from users that just want to be the first with everything and expect everything to work without any issiues from day one. In other words, users that don´t understand what a Beta is there for: testing, testing, testing……
Appriciate your efforts alot!! You´re always perceptive and eigher to solve issiues as fast as possible and your feedback is great. Never try to push issiues aside but taking all problems seriously and do whatever you can to solve them as fast and solid as possible instead of looking for someone else (soft- or hardware developer) to blame.
Keep up the tremendous work you all do at Laminar and count me in if you think I could be of any help in some pre Beta group if you choose that way in the future.
/Best regards
You’re forgetting the cost and personal anguish of build management. If you’ve ever seen the average bug report, it’s hard enough as it is. Throw in any type of release complexity, ben’s going to be sinking his time time into just cutting builds all day. Sounds good on paper, almost never is in practice.
Option 2 is best I think, option 1 just creates anxiety and feels limited to a select chosen few. What I like about Laminar beta programme is anyone can choose to beta test and get the latest iteration.
Option 1 for me
I’d personally be happy with either the status quo, the #1 solution (count me in, if necessary) or the #2 solution, although I get that due/thanks to your openness and transparency you (LR) created a huge expectation in your loyal following that will be hard to contain.
What I’d like to see instead is a big red labelled box when starting up a BETA version: “This is a BETA version, not a final release: bugs, crashes, unstable behaviour are to be expected and we count on you filing the bugs you encounter in order to help us get to the final release as soon as possible.”
And, depending on the stage of the beta program, you may also add: “At this stage, we STRONGLY RECOMMEND that you first run X-Plane without any third party add-on. If nothing is observed, please add one add-on at a time and test – should you get a crash, please file it!”.
Whaddayathink?
I’d happy go with ‘Experimental Release’, ‘Beta Release’, ‘Stable Release’. Or Fast Ring, Slow Ring, Stable Ring. I do feel that the biggest win you would have it making it bleedingly obvous on the sim start up your running “BETA and is not production stable, this is for wide scale user stability and feature testing” (Or words to that effect). I appreciate there is some wording at start up but so many people can’t see it or understand it’s relevence.
With Steam you can have different release branches, perhalps people need to contact you and subscribe for the code to the fast release ring to minimise the brand impact for less experienced users?
Anyway, loving the update. As I fly low level VFR most of the time with Ortho’s and lots of OSM data the extra frames and performance out of my system is welcome! VR is much smoother now, makes me want to upgrade my Oculus!
Hi Ben, I have seen the noise on the .org-forum and people complaining “they cant fly anymore”.
Many people mistake the Beta with a privileged early-access platinum customer something. This is obviously not the case, the Beta is a technical trial and an engineering task.
Therefore my favorite would be number 1. A large group of somehow qualified testers who have applied for the Beta program and clearly understand that it is about testing, filing bugs, systematic reports. Not for recreational use and not for hacking art controls.
I have no idea of the amount of beta testers you have. But I can imagine that if you use a 2-tiered system there are 2 possible problems:
1) Not enough early beta testers to find certain exotic issues. Because not everybody that signed up, actually tries the beta in time (or at all).
2) More work for you; Being a small software shop I can image the extra workload of managing early beta testers would be a hassle.
Personally I do not mind the current situation; release to everybody interested. And if something really bad happens, like with b5, roll back like you did and all is fine.
I believe the 2 tiered system would work better. With users who know the risks and voluntarily help to virtually enlarge your number of testing machines and help ironing out serious and show stopping bugs, before the “real” beta is served to the whole community. Anyway, I’d be glad to help if needed.
The current process is the best way in my opinion, it’s proven to have worked in the case of beta 5.
It’s a BETA. This is what happens and is part of the process. If someone doesn’t understand that, then they shouldn’t be using it. I’m good either way!
I think this beta is special because of the introduction of Vulkan which means a big performance improve for AMD users like me. Therefore I think this beta attracts more people who hope to already use this feature for their normal flights.
For future releases it will get quieter again. So there is no need to change.
Beta testing is an excellent process that helps the RELEASE version to be as stable as possible. The fact that I can run the last stable RELEASE on my machine plus the beta is great. Count me in for any testing of the betas. Nice extension to the experience.. XPlane “community” is a large part of why I enjoy the product.
Keep it coming LR
One question which is not directly related to beta5 :
Is there, in the graphic engine, an algorithm that sort objects to be displayed by distance (like to good old painter algo), so that only close objects (let say the ones whose all vertices are less than a few hundred meters from the user pov) are calculated twice in VR to save GPU time. I mean, at a certain distance, all objects look the same whichever the eye you use… Hope you understand my thought 🙂
No – I have seen proposals for mobile VR to only do stereo for the near field. We do stereo by instancing: we send one scene to the GPU and it projects it once for each eye. The performance for that has been good enough that we haven’t considered doing a far-plane mono render.
Okay, thanks ! 🙂
May I suggest an option to set the virtual IPD (as DCS does) to see the virtual world as we see the real one ? Some cockpits seem to have a scale issue. For example, the Shuttle is so huuuuuge, I feel I got the IPD of an ant !
Hi Ben
my opinion is that you have to leave things as they are
those who try a beta must know what it is facing
thinking about what is most useful for you I would say option 1
keep up you are strong !!!
Well… let me know if you want someone, with software development experience, that will not be angry or upset if it crashes, to check out new betas early. I even have two computers capable of running it comfortably, that I can test on.
Keep in mind though: I try the betas mostly because I am curious about what will come, and like testing out features early. But I will probably not be spending hours upon hours flying in a beta version. I like to do my “fun-flying” in a stable version.
Hey Ben! Don’t be so hard with yourself, the work that you and team have done is outstanding, this is just a small issue and we’re in beta!
Take care, thanks for doing such great job guys. 11.50 is one of the most remarkable products of this year.
Regards,
Guillermo
Hi Ben! I would not vote for either of the two options. Leave the beta system as it is in order to provide – through all the smoke and dust – the broadest sampling of issues with each release. This isn’t a mobile app and most (all?) of the serious beta testers are using two and sometimes three copies. By the way, beta 5 “works” on my rig and over the years I’ve never had ANY stable copy of XP crash – the only crashes are the ones I know I caused. It’s a beta not an “upgrade” as so many posters describe it. My take on this is – and I may be wrong – so many folks have installs that are somewhat borked already by all the plug-in’s and add-on’s they’ve hung on it that they just need that final straw to tip over the bucket (just look at some of the logs they post). So that’s kind of a good thing since you folks need to make XP as bullet proof as possible. You need this feedback. Sending betas out to a small sampling of knowledgeable users who’s systems are probably going to behave like the 10 rigs you work with is not going to give you what you need. I vote for keeping the beta system “as is”. I would consider the check box instead of the auto notification.
Hi Ben,
First of all a big thank you to you and all your team for the great effort!
I would go for option 1. The question is how many and who would you select as early adopters. This would be a sort of private beta testing team.
Kind regards to you all and stay safe!
Thank you for your efforts!
You guys help us to have a better confinement, I’m grateful!
Keep up the great work!
Ben,
I agree with 1 Beta too. However, if you and the team decide on an early adopter program please count me in. BUT PLEASE CONSIDER THE FOLLOWING AS AN OPTION.
Here’s a novel idea….How about building into X-Plane the option to “Revert to Previous build” option? If, for whatever reason, a user is having issues it would be a simpler fix allowing for fewer installs and configurations.
Thanks and keep up the great work.
I dont see an issue with carrying on as you do now unless the volume of bug reports is an issue for Laminar. We can after all go back to the previous beta just as some have done this time if you find a beta release is causing problems and you have to pull it.
In a “normal” beta you can´t. You can only go back to the last stable release (by unchecking the box in the installer). Only if a beta is “pulled back” like beta 5, reverting to the previous beta is possible.
There are many instances where people were able to run beta X – then updated to beta X+1 and had crashes…and then cried and moaned because they couldn´t go back to X.
Think of it this way: There is only space for two versions on Laminars servers. The latest stable and the latest beta.
If you really like a beta and would like to save it – you can. Just copy your installation of X-Plane (the beta one) and don´t overwrite it with a new beta.
Hi Jan,
usually i keep the X-plane.exe from the old version in a safe place and usually its working well copying it back when Beta-X is goin T-U. 😉
Most people don’t know …
Maybe does not work (i haven’t tried…) using a non vulcan version in a vulcan capable install. But then you can use STABLE tickbox of the installer …
Oliver
Both approaches are good, but I personally prefer #1
Looks like I’m late to the party. Anyway, I would choose a method that makes the most sense to you in order to quickly and accurately find and deal with the bugs. Some will complain but we’ll adapt to your system. I’ll continue to test the betas as they become available to me regardless of the approach that you take.
Late to the party too here and I agree with this. Whatever gets the best result for the devs at Laminar. They likely already live on way too much coffee coding and testing it in the wee hours before we even see it. I’m currently not a beta tester but plan to when I have more time at some point.
I would choose a 2 tier solution
Tier I) early access for users who apply to get it with a small form where they have to read the basic rules of beta testing, answer some easy questions like
Beta testing is
a) for getting early access to the hot stuff
b) to be cool
c) to help the developers to find bugs and glitches
When i find a bug
a) i spam facebook and forums to ask who else has this bug and hope the devs will read one of these forums and facebook pages to read my complaints
b) i try to debug and find the root cause
c) i try to reproduce the bug and file a bug report in the X-plane Bug Reporter
……
and then get a code (bound to the user email) that can be entered and saved in the installer to get betas from hour 0
Tier II)
Can get access to stable releases after 24h-96h (depending on the beta release cycle frequency, for 1 week cycles more like 24h, for longer cycles with longer delay) with the installer like now together with Steam users
But more important than changing the beta access:
I recommend that the installer displays the release notes before patching (need to read and scroll down), and topmost the known bugs, the known broken addons and hints to the test focus. I don’t know if something like this would be possible with steam.
I also support the 2 tier approach ,
I always have multiple copies of XP11 to test , and have a group of users I give advice too , I would be keen to help
Regards
Lawrence
I would recommend progressive rollout canary for releasing betas for a couple of reasons;
– it scales well across any size of beta testing user base, providing blast radius protection if a bad build is promoted into the channel.
– it is low maintenance for the team to manage canary builds. You do not have to administer attrition in the pre-beta channel of testers. You can always reliably target 5% of your active testers.
Just keep it like you’ve been doing. Let those who want Betas get them… I love testing them and seeing what’s new even with the crashes…
And to the whiners… IT’S A BETA!!!
Love the work you all are doing!
You cannot really win here no matter what you choose to do:
If the beta crashes, the masses will be sad.
If you keep the beta from the masses, they will be mad.
At the end of the day X-planers are drewling at the side of their mouth for Vulkan and a couple of extra frames – who can blame us? – Laminar gave us a taste and now the community’s addicted.
So, keep it the way it is (with rollback for those suffering from withdrawel) or choose what is best for speedy development.
Hi,
i am on BETA 5 and flew 3 mid lenght segments today. Yesterday it has been only two. 3rd party a/c (MD80), various plugins and LUA scripts (nothing fancy). No problems.
I’d be in for Testing Betas on a second install. (junctions to scenerie and aircraft etc if needed). Let me know.
Oliver aka JetNoise
As answered by few others, I would not change the current “invitation”.
Anyone who wants to play with beta _should_ know that it implies, sometimes, regressions and crashes. It is not always bells and whistles.
The only thing I would change in the updater is the ability to let one to choose to which beta one would go. Not with plenty of entries but few to be able to revert to a previous beta. Of course, it is possible to duplicate the installations but it could ease the rollback process.
Either of the tiered approaches would work but only if Laminar has a way of knowing how many beta installs have occurred. Personally, it might take me a few days before I get around to updating so just because I’ve been invited doesn’t mean I’ve actually installed it. For example: 100 users get invited to try the newest beta. Let’s say only 10 of those users actually install it in the first day. If 5 of those users report a crash, then that’s 50% of those who actually installed it (bad). But if all 100 users installed it, it’s only 5% (not too bad). So, in my opinion, the key metric for the tiered approach would be for Laminar to know how many actual installs have occurred
10-40-100% beta rollouts seem to be industry standard, and it’s what I’d do. (I’m a dev-ish at a large-ish tech company.)
But to be honest, we all opted into the beta. Sometimes betas crash. It’s what they do, and it’s why they’re not final releases. Maybe run a poll, but I’m pretty sure most of us opt-in folks are fine with broken things.
For me will be a pleasure beta test xplane 11 as I do with Skymaxx.
I suggest not to publish beta availability with notification in XP itself, XP notification should be reserved for released versions only. This will save frustration for new users.
Based on what i see in the .org site, many users just update because they had a notification in XP – but don;’t appreciate what they are getting into.
Let those who are really wanting to participate in testing have access to betas by checking in the Laminar website for beta availability.
I am also concerned that if the option of beta tier access is selected the Mac community will not get adequate representation on the testing samples. IN this 11.50 beta the Vulcan and Metal users are seeing different things and are testing different code when it comes to the rendering engine changes itself as part of Vulcan or Metal.
Those interested in the tier option are ok with that as long as they are in tier 1. Not so sure they would like it if they end up in tier 2.
IN addition, the 2 tier approach will slow down getting through all the betas to a release. Because some of the same issues will pop up when all users have access anyway.
ONe way to help this move along would be for the earliest betas not allow access to 3rd party add-ons at first. Even if they are installed. That way you can get to the root of the issue without having to deal with incompatible add-ons causing issues. Then when you have got that addressed, betas can open up for add-ons. This make sense since you already are working with 3rd parry add-on providers separately.
Hawkeye
I have to agree with Dave, as a Mac user I’m not seeing much if any improvement in Metal, I’ve tried all betas up to 4 on a base download updated to 11.41 with no scenery, or plugins added. The only thing added is the Zibo Mod 737, which I understand runs in 11.50. I have submitted things I have found during these updates to the Laminar team. Having said that I’m more than happy to let things run their course and hope that in the fulness of time the Mac community will have a good running XP.
Great work by the Laminar team in these trying times.
If you don’t see an improvement with metal, it’s almost certainly because you are GPU bounds. Apple’s machines rarely can drive their displays at max res, since max res is some crazy high pixel density. When not pixel bound, Metal is a ton faster.
You’re dead right about this. Metal has transformed X Pane for Apple users and 1920×1080 works great. Using Metal I can more or less double my FPS and still moved the sliders to the right.
As a former software engineer (going back to the mid 1970’s – And yes, I do feel old,) I am used to crashes and Ooops moments and a lot of “@)*@(*EWD&)8” language etc. If there’s a bug or an unintended function that comes out on it’s own, I’m usually the one that it finds. I would be glad to try out one of those un-released betas and would be quite happy to render my report.
I have:
1- main flight sim computer called: X-Plane used only for Xplane consisting of:
Asus Crosshair VII X570 motherboard + Ryzen 3800X + 16g Ram +
EVGA RTX2060 super graphics card 8Gig ddr6 ram
2- Left view computer called: Left used only for Xplane consisting of:
Asus M5A97 R2.0 motherboard + AMD FX6300 6core @ 3.5Ghz + 16g Ram +
Asus GT710 1g DDR5 graphics card
3- General purpose computer called: Phil and is also used for right view of Xplane
sim consisting of:
ASUS M5A97R2.0 motherboard + Phenom II 4core @ 3.6Ghz + 16G Ram +
Asus GT710 2g DDR5 graphics card
All above using Windows 10 (latest edition)
If there’s a way to find a bug or other entity, I’ll find it (or they will find me)
P.S. I’ve been using Xplane since version 5…
Thank you for your consideration.
Phil
I’d leave it as it is now, but on the loading screen, add a message/disclaimer, and a good idea would be to create a new branch on steam that allows you revert back to the previous beta. I do love x-plane, it is awesome and the Vulkan API is definitely a massive step in the right direction. Great work!
Keep betas the way they are! It’s a beta! If we wanted it to be easy, we’d stick with the GA releases. If you find something really big, quick like you just did, you can recall it and we can go back one beta.
I appreciate that you’re prompt at being attentive to what’s happening out here in user-land – you have a great product which I’ve spent 2000 hours in front of in the last two years. FWIW, b3 was golden for me. Four caused a lot of sudden halts and I looked and looked for b5 with high hopes.
I’ll be happy to test in whatever way is most helpful to all of us. I still have a great-running version of 11.41, but there’s no value in going back to b4 in my case…
How will this go forward? Will I eventually see a release of b6? No matter… keep up the good work. Thanks a lot,
I have only one problem with this, and it is not related to you being honest about the problem which earns my deepest respect. But going back would be far easier if we hadn’t had to click a hundred times during the process, even with subdirectories checked.
We all want a bug free rock solid x-Plane and this requires testing.
Crashes should be ok for any volunteer opting in for betas.
It is not important whether I get notified or have to check actively myself whether new beta version is available as long as I can get access even if I do not belong to the 10% invited first.
It would be great if you could support up/downgrading to different versions selectable via installer not only for betas but also for general versions. This way people could avoid having to create private backups all the time.
Would it be possible to enforce a second copy install of a beta? Or at least during the first phase of a new beta? That way, users would still have a working version that they could revert to.
Otherwise I think sticking with the current system is good, but maybe some huge splash screen warnings (as has already been suggested) could dampen people’s reactions when things don’t work. Because this is also about PR, right? An unstable beta gives you a bad reputation, but without beta testing you would be getting nowhere. There are reactions on the forums, but there is a large group of loyal defenders jumping in to straighten out what can be expected .
Keep up the good work!
I think, option 1. might be the best. Not all users consider a flaw in beta SW the end of the world as we know it. Unfortunately, it includes for you the task to maybe sort beforehand and separate the less agitated, test-experienced players from the a bit more hysterical “But-Daddy-I-want-it-now!” kind of customers … 😉
Stop the kidding – we are using a smaller number of selected UX customers to try out our newest SW functions and ideas. Made some good experience with this way. (No worry, I’m not competitor.)
I’m on the beta in order to help speed up your development cycle. You can crash my launch screen for all I care.
On that note, I haven’t had a crash yet.
I think that, especially for Steam, you could have 2 beta versions. One could be called something like “experimental”, where it will pull the lastest 11.50bx version, and then different versions for “stable” or something, where it would pull the latest “stable” release version. This would allow users to enjoy the benefits of the latest “stable” beta while at the same time allowing us folk who don’t mind very buggy versions to contribute to bug fixes and error reporting.
Keep up the wonderful work! It’s no surprise why Laminar is my favorite games company!
let’s not forget the meaning of a beta version.
So if people agree to install beta versions they don’t have to complain if things are unstable or even unworkable.
So there are 2 type of beta installers those who really want to help and test things, and the biggest group are early adopters.
So for both groups option #1 will be executed.
#2 people will be frustrated that a friend already have the latest features and they still need to wait.
So choosing to update is an individual call and you guys are not to blame.
Option #2 is only useful if you guys want to avoid being under pressure because to many people can’t use your product any longer. But if you are able to easily do roll backs.
The bigger the test group, the better the reference of the test, and more data can help analyse issues.
Option #2 is a must when rolling out the final product, because here users are awaiting a stable product without issues.
So I say don’t change it.
As a software engineer myself, this is very realistic but glad to see you guys finding a little humor in it.
One thing that our company does especially which things that have the potential to break, major feature or system changes, is find a few users to ‘pilot’ those changes and report issues (Ha, that is punny). Generally, we resolve all of the high priority issues before rolling foward to a more general release. For us typically, the users that pilot changes are the most motivated eager for the change and experienced with our product such that getting useful feedback is fairly responsive.
To relate it to what you guys do, my opinion if I may, would be let users opt-in to pilot higher risk builds, understanding there may be issues. Might be good to provide a build rollback/build selection that could be triggered by the user in the case of large issues. If there is too much feedback or less useful feedback then finding the most eager subset of the user base might be a good idea.
The big question is, did you or the Canadian Superman figure out what blew up b5?
Supersid. We looked through logs and approximately 100% of the GPUs crashing were NVs with drivers new enough that we could activate Aftermath. Since nothing else we did made sense for this catastrophic of an explosion, we tried turning it off and whodathunkit…
Might be useful or not, but apparently there is a “shader cache” bug and some other issues introduced, since the latest Nvidia drivers as of 445.87 where some Vulkan based games are problematic – There’s a hotfix here:
//www.nvidia.com/en-us/geforce/forums/game-ready-drivers/13/373159/announcing-geforce-hotfix-driver-44598-released-42/?commentPage=2
I’ve experienced BSOD’s and blackscreens both in x-plane and other Vulkan based games since using that driver that were not present before.
Necessary but not sufficient condition for the b5 crash?
As I am not a programmer it took a little googling to decipher this, but if I understand it, the tool that allows you to figure out what crashed the GPU, crashed the GPU? Is this something a user can toggle off, or something in your code?
Enabling AfterMath is in programmatic, in our code. We haven’t decided what our long term policy will be, but the short term idea was “use it as widely as possible in b5 to collect the best crash data.” As it turns out, it was the source of crash data. 🙂
I don’t often debug, but when I do…
No, seriously… I find starting with a very basic/generic build allows me to track bad or erroneous (typos) code faster.
I also test anything slated for distribution on the worst possible platforms and then the better.
I run 11b4 on a crappy 3.1 3.4 turbo AMD series 8 CPU, so you’re definitely getting my errors, but I have to say that you’re doing a damned good job keeping such a variety of machine configurations working.
That says a lot.
Mistake away.
#1 -and can I register please. X-plane glider projects group has several plugins live on the org. It would be good to hit the ground running before the public beta goes live
X-Winch, Towplane-X, Therm-X
I have development experience and would be happy to be an early beta tester. To answer the question, I’d suggest the two-tiered system. It’s needs to be opt-in so you have people trained in QA and expecting the beta. I’d you roll it out in stages, you’re going to have a fragmented user base and the same problem you have now on a smaller scale. Your feedback will be fragmented too. You’ll make a mess of it. Open up a “pre-beta” testing group and let people sign up. Give them a means to communicate. They screen betas before it goes to the masses and then all of those opting in to betas on the installer get it if it tests okay. Then it goes to release.
Fragmenting your testing as described in the options above is absurd, and I’m surprised it was suggested in your meeting.
I think the current system is ok. Being able to easily roll back to some previous beta would be nice though.
Hi guys, I like what you are doing with the sim and I like how it is running (besides the occasional bugs and compatibility issues) but besides that, I am running very smooth and can finally land in places like New York and Los Angeles without burning my computer down into pieces. Keep Up the Great Work !!!
Because we all know what UTC is, can these and othwr posts be dated according to UTC? … any other timezone does my head in
As previous X-Plane third-party developer and current beta tester since many many years (X-Plane of course, but not only), if something should change about beta versions availability then I’d agree with option 1.
But I also can’t underestimate how public beta testing is important for developers in order to check important updates, checks done by the greatest possible number of users not only by (relatively not many) selected super users, so limiting in some way the access to the public beta could be a not so good choice.
Just my 2c.
Angelo
I noticed something, reload scenery doesn’t work like in earlier versions, added some new objects and buildings using, I reloaded from the developer menu and nothing showed up, I have to go to different airport and come back to my airport.
As just said before , beta-testers know the risks and are flexible enough to make backups ..
I’m using beta 5 with a full AMD-Win10-System , nearly without problems. Even the FF A320 ultimate
is flying well. After longer flights sometime parts of the scenerie becomes blu , bacause without
a system crash. In the case of bigger problems I switch of Vulkan … flying with lower framerate.
Thanks a lot for Your greate work.
Here’s a thought – when X-Plane generates a message about critical stability, displaying in the UI would be more useful than just writing to the log file.
I’m not sure I agree with that. It depends on the message.
– If the message is technical and engineering needs to see it, an auto-reported crash and its associated data gets to us – the auto reporter doesn’t forget to attach a file or attach an old copy of a file. We don’t have to _check_ whether the data files might be inconsistent.
– A message to a user implies the user should _do_ something. So the question is: what is the action the user should do, and is the user qualified to do it?
Generally by the time we hit a low level condition where X-Plane is going to crash in an expected way, we _do not know_ what the user could have done to avoid the crash. So we err on the side of trying to get good forensics to fix our bugs. The only action we need to encourage the user to do is hit “send” on the crash report.
Is it possible to auto-concent to sending crash bugs at installation (or setting page) stages. For me I can see the files you get sent so I trust the data your submitting and so having it auto submit and just tell me it has done so and to go here to view the details sent would be better for me. You might get more crash reports then.
Thank you as always
Paul
I’m sad I didn’t see this before, I just got a crash on 11.50b5 and I was saying “beta 4 wasn’t this fragile, what happened?”
I don’t mind flying beta and trying out new things and reporting bugs if I find them.
Of course, two-tier beta releasing makes more sense to avoid noise. I’d jump in for the first one.
Only feedback:
– I would love being able to track these kind of things. I found this post by coincidence checking if there was a Beta 6.
– I love how you’re releasing betas so frequently and seeing so many improvements. Way to go!
1. Keep your candid transparency about your work. That helps all of us weigh the risk of trying a pre-final release.
2. I’d say “beta” means beta; close-to-final draft; “swim at your own risk”. No need to complicate things unless, as you mentioned, you don’t have the environment infrastructure to model enough of what’s “in the wild”.
3. Always have an ejector seat option, which is some path back to a usable product while development continues.
I’ve been in product development most of my working life. A software tool like X-Plane has the luxury of being able to use its users as guinea pigs. Any product with the ability to injure does not. Having betas is an accepted practice for that unusual class of products. Yes, the buyer should beware, but your putting realistic warnings out for the “buyer” (beta user) to be well-informed about the decision is key. Keep up the good work.
As for me, I can’t wait for Vulkan’s promise of “smooth as butter” rendering… but I decided I CAN wait. I’ve not joined the beta because I’m more interested in using X-Plane. However, you have my system info. If I can ever be of help in uncovering something, just ask and we’ll test something.
Just keep the betas coming X-Plane is so well delivered meaning u can have a clean stable install fairly side by side…
But I look for betas every day… and for the most part I’d rather you spend your time pushing the envelope of x-plane that worry about mistakes sent out.
We understand 🙂
I own (3) licensed copies of x-plane installed on (3) separate systems and have supported X-plane since day one. What makes X-plane different is the open honesty and the continued communication from all of the X-plane developers. As many have posted “BETA” means “BETA”, I’m sure no one that has posted here thinks it’s a final working version.
We are just guinea pigs making sure that final version does work and, can’t speak for everyone, but I’m happy to be an X-plane guinea pig. So just keep them coming, gives me a reason to have three systems with one dedicated to BETA’s only.
Not a bug report, but just a comment on the 11.50 Beta Vulcan releases. There is opinion that MS 2020 will damage Xplane but on the evidence of this release I don’t think so. I’ve finally bee able to use Xplane with helicopters and VR controllers in this beta and the result is nothing short of phenomenal.
Ok, sure, everything could be improved. A little faster, a little crisper graphics, some more consistency, fewer reloads but wow, what a bucket load of fun. Flying the Vskylabs R66 in VR is incredible fun. MS won’t have VR or helicopters.
Absolutely brilliant job by Laminar on this Vulcan VR release, I can say that I’ve never had so much fun in a flight simulator, just an amazing job.
Thank you Laminar, please keep it up, I will be a fan for many years. Awesome job, and when the hardware (VR resolution, graphics cards, CPU) catches up shortly it will get even more incredible. Outstanding job.
For the Beta would it not be possible to filter bug reports & crashes at the LR end , and as filtered crash reports get less increase the number of reports into the filtered list.
I.e A registration system for beta testers and then LR select the percentage of testers from the applicants,