I don’t usually link to non-X-Plane blogs, but I really liked this pair of posts:
//distractible.org/2009/11/05/top-10-ways-to-annoy-your-doctor/
//distractible.org/2009/11/08/top-10-ways-doctors-can-annoy-patients/
If you live in the US, you’ll definitely appreciate it…the lists are funny and yet have a seed of painful truth in them.
So I decided to try to create my own lists.
I am only tangentially related in tech support – Randy takes on most of the work with some help from Jack. Sometimes very weird reports get escalated to me. (And most of the “let the report sit for a week” comes from me not having time to dig in.)
Anyway, please take these with a grain of salt – they’re meant to be funny and exagerated. Most of our users are very, very helpful in tech support calls, despite the fact that, if you are talking to tech support X-Plane is already hosed. And Randy puts forth some amazing acts of patience in the face of some of the requests he gets. My hope here is only to show that there are two sides to the frustration in a tech support incident, and we’ll all be happier if we can see the whole picture.
Five Things You Can Do To Annoy Tech Support
1. Be As Angry as Possible
Threaten to switch to Microsoft Flight Simulator. Drop the F word a few times. KEEP CAPS LOCK DOWN FOR THE ENTIRE EMAIL. Tech support definitely responds better to users who are angrier – you don’t want to get sub-standard service because you were too nice, right?
2. Omit Information
If you have a second graphics card made in Kazakhstan, over-clocked and running hacked drivers you got off of the pirate bay, don’t tell us. If your computer regularly catches on fire, be sure not to mention that. Did you recompile the Linux Kernel yourself after letting your pet monkey edit the thread scheduler? It’s best we not know.
Extra credit: report a truly bizarre problem, provide no details on your customized configuration, wait a week and tell us how you fixed it by removing a third party program that “enhances” sound or graphics. Priceless!
3. Don’t Include Past Emails In a Thread
Be sure to delete any past information from your email. Change the subject of the email so we can’t tell what the original issue was. If you have more than one email, send replies from different addresses. A perfect reply would be “That didn’t work” sent from an email address that you haven’t used before, without your name included.
4. Email the Last Person You Talked To.
If you just finished up sorting out a shipping problem with the shipping guy, ask him how to create a plugin. If you just got info from the developers about UDP, ask them why your credit card was charged the amount it was charged.
5. Bring Up New Issues In the Middle of Old Ones.
To do this just right, wait until the thread between you and tech support is pretty deep into the meat of a complex issue. Then throw in another paragraph about something else that’s gone wrong. To perfect this technique, try to pick a new problem that the person who you are emailing with isn’t equipped to handle (see point 4) and keep the report vague (see point 2). You can repeat this technique to stretch out a tech support incident indefinitely.
Five Ways Tech Support Can Annoy You
1. Make the User Reinstall the OS
Reinstalling the operating system fixes approximately 0% of user problems, but it takes a really long time, and is almost guaranteed to screw something else up, usually something that wasn’t broken and isn’t related to X-Plane. If a user is a little bit annoyed, this is a great way to pour gasoline on the flames.
This is really a special case of the general strategy “ask the use to do something time consuming, annoying, and unlikely to help.”
2. Forward the User a Huge FAQ, None of Which is Relevant to the Problem
Everyone likes form letters and impersonal service. The FAQ should be badly written, badly formatted, confusing to read, and preferably not accidentally contain the real solution to the problem. If the solution to the problem is in the FAQ, don’t tell the user where in the FAQ to look.
3. Wait a Long Time Between Replies
Tech support incidents are like fine wines – they get better with age. To allow the user’s annoyance to bloom into a finely honed rage, be sure to let each email ‘sit’ for a week before replying. This works especially well if your response is just to ask another question, setting the user up for another week’s delay.
4. Blame Some Other Component
The modern PC is built by approximately 600 different vendors. Blame one of them. The beauty of this strategy is that it is one that can be used by every vendor who provided software or hardware for the PC. Also, because quite often the problem really is with another component, you can claim this with a straight face.
Tip: blame the graphics card maker – ATI and NVidia do not have the resources to pursue every complaint that an over-clocked graphics card running the latest patch to some simulator written by two guys in their bedrooms crashed with the drivers visible somewhere in the callstack. Put the blame on the GPU makers – they don’t have the resources to refute you, no matter how bogus your claim.
5. Forward the User’s Issue Around the Company Until It Gets Lost and Dropped
Everyone in the company has to be in on this strategy for it to work – if one of your idiot coworkers actually solves the user’s problem, well that defeats the purpose. This strategy can be combined with (3) and is sort of a riff on (4) – once the user complains that they got dropped, blame everyone else in the company for the mis-communication.
This is off topic, and I try to keep the off-topic political crap on this blog to a minimum. But I do want to put a plug in for Freecycle.
Freecycle is basically a series of locally based mailing lists for the free exchange of things you would have thrown out. My wife and I use it every time we move – we get packing materials from other people who have just moved, and then give them back to our new local group when we are done. We also freecycle all of “that old junk” that we realize we shouldn’t move with us to the next location.
If you are ever in a situation where you need to throw out items that are potentially useful but just not worth the cost of carrying anymore, please consider Freecycle – it strikes me as a reasonably reliable way to keep items out of landfills. (Especially hard-to-recycle mixed-material items like electronics.)
One note of caution: freecycle is just a group of people exchanging “stuff” in their spare time – typically it could be 2-3 days before someone can pick up your donated item. So…if you have a lot of stuff to give away, start early. I made this mistake in DC and wasn’t able to give away all of the items because people couldn’t pick them up soon enough.
Laminar Research has closed its Washington, DC satellite office and opened a new office outside Boston. Okay – that is completely cheeky – LR employees all work at home, and I have moved from the DC area to the Boston area.
This has put me a little bit behind schedule with, well, everything. I have four or so interesting example plugins almost ready to be published, WED is ready for some kind of developer preview, and I have some X-Plane 940 bug reports to go through. I hope to get the back-log cleared out this week. But first priority: unpacking the computers (and figuring out how to get an internet drop into the new home office).
My last blog post may have expressed some negative views toward software upgrades:
I know I’ll take some slack for this, but: why did you all go and grab an OS update so fast? What have you gained? The down-side is lost time and application compatibility problems…was it worth it?
That blog post was of course not my real opinion, but rather a recreation of the evil Ben. Always upgrade your software to the newest version whenever it comes out!
A while ago, Austin was on FSBreak, and I wrote this post as commentary on his interview. The main point I meant to make was this: from what I have heard from other engineers and seen in my own experience, most software companies prefer to develop new features on top of implementations that are known to have architectural problems. At LR, we fix the implementation architectural problem first, and that has been a net win for us.
Now that’s basically a statement of my opinion on software engineering – in hindsight it probably belongs on my programming blog and not here. Unless you develop X-Plane plugins, you’re not a programmer; I will try to constrain future scenery blog posts to things that non-programming X-Plane users will notice. If you are a plugin developer, you might want to look at the “Hacks of Life” posts tagged with OpenGL.
Anyway, back to the story…the responses to that blog post were all well thought out comments on X-Plane’s quality control process. At the time my immediate reaction was: that’s totally off topic – I’m commenting on architecture and they’re talking about QA. I do think the authors made fair points.
But in hindsight, I think that there’s a deeper issue: one of verifiability. Simply put, my statement that we (LR) rewrite stale implementations is nearly impossible to verify without source code access, something that you can’t get for X-Plane. So from the perspective of anyone outside the company, my original statement is not falsifiable (it cannot be proven false) and thus rather useless as a statement of fact. Even though I claim that we make rapid progress on features by keeping implementations clean, you as a user don’t care how we develop our features – clean architecture, more developers, or the use of time travel and voodoo dolls, it’s a bit moot.
Thus the comments were off topic, but also they were moving away from an unverifiable topic and toward one that users can measure, namely the quality of X-Plane’s betas.
There’s a fair amount of marketing that gets put out in the tech and games industry. It’s a slippery slope from giving a new, real technology a whiz-bang name (e.g. HyperZ is a real technology, and it is good for your frame-rates) to using techno-babble to make the bad seem good (e.g. HyperMemory just means that your video card lacks VRAM and is going to be slow). When new products come out, the feature list is parroted, but it’s not always clear whether the new features turn into real benefits.
So what I’m going to try to do with the scenery blog is: try to keep the blog limited to verifiable, measurable aspects of X-Plane. If we ship X-Plane with “psychoactive rendering*”, I’ll try to explain what the heck that is and why you’d want it, and how you might notice that it’s working.
* X-Plane does not have psycho-active rendering, except possibly when I make a mistake in the shaders and everything turns purple.
If I have not emailed you back, it’s probably because I’ve been very busy trying to interleave X-Plane coding and packing up the house. But it is also possible that my email response bounced because my web server has ended up on a bunch of anti-spam “black-lists”.
Black-listing seems like a good idea at first: we’ll gather up a list of all of the IP addresses from which spam comes from, then publish them. Then your local mail server can use that list to filter spam – you never see it!
In practice it doesn’t work so well…example: //www.mipspace.com/. The IP address for my server (XSquawkBox) is now on this list. Why?
MIPSpace is a list of IP Addresses associated with known commercial marketing companies.
Since my server is used for my own personal email and to run the SDK website, I’m not sure why I am on the list. I have sent them an email to clear things, but in general I hit an anti-spam/black-list bounce somewhat frequently now, and frankly I don’t have time to separately try to clear my name from every guilty-until-proven-innocent blacklist that pops up and screws up my email.
If I seem disproportionately grumpy about this, it could be due to one of two reasons:
-
Not replying to emails is generally bad customer service. (Okay, my in-box is backed up four months…that’s bad.) I don’t like the idea that a customer might perceive us (LR) as being unresponsive because some third party with no skin in the game decides to black-list us.
The blacklist has no incentive to be accurate – it’s not their lost customers if email doesn’t go through.
-
I’m not at all convinced that this is going to cut down unsolicited commercial mail and/or spam.
In the spam case, spammers can send from botnets – they have access to a huge number of ever-changing IPs. Unless we are prepared to blacklist the entire internet, the blacklists are going to pick up more and more false positives while spammers find ways to harvest fresh, untainted IP addresses. The whole IP-reputation strategy assumes that IPs are hard to change. In practice, IPs are very, very easy to change.
Commercial mail is a lost cause too – even if I am being solicited for commercial mail I don’t want, no program or automatic process is ever going to tell the difference between the confirmation of my invoice and a list of discounts from the same company. When it comes to commercial mail, the reputation damage has to be done to the company, not the IP.
(The company does have reputation to risk – if we are known as a company that doesn’t honnor a “do not subscribe” policy, then customers can choose to not buy our products.)
It could also be because I ran out of Viagra and don’t have a diploma from a prestigious non-acredited university.
My family visited DC this weekend and we went out to Udvar-Hazy, the extension of the Smithsonian aerospace museum out near Dulles international airport. My dad took this picture.
That is one of the two radar antennas (and the telescoping arm) used to scan the earth as part of the Shuttle Radar Topography Mission (SRTM). The SRTM is basically the first really good quality most-of-the-earth elevation dataset, and it is the main (but not only) source of elevation data for the X-Plane global scenery.
The telescoping mast shown in the picture (horizontal) extends one of the two radar antennas away from the shuttle when in orbit; had they not been able to retract the antenna they would have had to detach it and leave it in space. Fortunately the mechanism worked properly, so they were able to bring the antenna back for posterity.
With the iPhone and X-Plane 9, we’ve been very busy…with this in mind, I am pleased to announce the latest engineer to join Laminar Research.
“Nubblet” would like you to know that the MacBook Pro is, in fact, “hers”. 🙂
It’s that time of the year again…I’ll be out of the office next week, and more importantly, away from all cell and email contact.
Austin and Randy will be at EAA – stop by and say hi if you’re there. Once Austin gets back, the last few betas should resume – we’re close to an RC on 920 but not quite there.
I’ve been reading Fooled by Randomness (highly recommended – it will either change your life or you won’t finish it because Taleb’s writing style annoys you) – and it has me thinking about the nature of certainty in software development. Consider two approaches to uncertainty and how they are completely at odds with each other.
No Weird Behavior
The “No Weird Behavior” approach goes like this: you never want a harmless behavior inside your code that you don’t understand. The reason is that if you don’t understand the behavior, you don’t really know that it’s harmless!
In fact this isn’t just a theoretical problem – I only truly started to believe in “No Weird Behavior” after fixing several very hard to find, hard to reproduce crash bugs, only to discover (once the analysis was done) that the broken code also produced a very frequent, less harmful behavior. Had I taken the time to investigate the “weird behavior” first (despite it not being of high importance) it would have led me to a ticking time bomb.
No Weird Behavior implies that a bug isn’t fixed until you know why it’s fixed, and that randomly changing code until the behavior goes away is absolutely unacceptable. This shouldn’t be surprising; if a bridge was
swaying would you accept an engineer who fixed it by randomly tightening and loosening screws until it stopped?
Wait And See
I get a lot of email with bug reports, questions, and reports of strange sim behavior. These emails have some unfortunate statistical properties:
- A good number of them are resolved by the user who emailed within a day or two.
- A certain percentage are simply never resolved. (Often I email a question that would point to a clear diagnosis and the user never writes back. I can only speculate that the user answered the question, found the problem, fixed it, and promptly forgot about our emails.)
- A certain percentage are solved by the user, who tells me what the problem was, and it was something I would never, ever be able to help them with. (Things like “I have this program called WickedMP3Player – it turns out if I set its visualizer setting to ‘Rainbow’ then X-Plane stops crashing” when I’ve never even heard of the WickedMP3Player program to begin with.)
- There is a high correlation between bug reports reported by a very small number of users and a resolution from the above categories, or a resolution by the user changing his or her local configuration.
So playing the odds, the right thing to do when presented by a third party with weird behavior is to wait and see who else reports it; the frequency of reports gives us information about the likely resolution.
(And for those of you who think our tech support are being lame when they ask if you’ve updated your drivers, they are playing the odds to the hilt – they ask you about drivers first because updating drivers fixes an absurdly huge number of the tech support calls we get.)
Getting It Wrong
So we have motivation to investigate everything (no weird behavior), motivation to ignore everything (wait and see) and a rule of thumb that the frequency of reports can help us pick which strategy is best. Of course, sometimes the rule of thumb is completely wrong.
One user reported a crash using the current web updater (version 2.04) – I had not seen this crash anywhere and it was inside the operating system UI code, so I assumed it was a local problem, e.g. some kind of extension or add-on that caused the OS problems.
The problem, it turns out, is simply low frequency – as the incorrect code made it into 902b1, I got a few reports from more users and realized that this wasn’t a local problem, it was weird behavior! (The bug will be fixed in the 205 installer and 902b2, both of which will be out in June.)
Consider this: if you make a measurement of a known quantity with a dubious measuring device, you learn more about the measuring device than the object you are measuring. (If you have a ruler and you don’t know the units, you could determine them by measuring yourself.)
In a number of cases, we have seen serious “should-happen-all-the-time” crash bugs that get reported by very few users. (Once we know the actual root cause of the bug, we can deduce logically whether the bug should happen to all users with the configuration or be intermittent.) We can then look back at the actual number of reports to make a judgement call on how much testing is really happening.
For example, we can make some guesses about how quickly a new Macintosh have saturated the X-Plane user base when there is a hardware-specific serious bug in just that machine.
We had this with the iMacs (where the runway lights were broken) and we could watch the machines sell – slowly at first, but then quite quickly. (BTW, I think that 10.5.3 fixes this and anti-aliasing problems.) We can even see who runs with anti-aliasing when there is a setting-specific bug (a lot of you do)!
In the end, I think the right approach to balancing “no weird behavior” and “wait and see” is to remember a truth about uncertainty that is very hard for humans to grasp:
The most likely outcome of an uncertain situation is not guaranteed to happen – in fact a lot of the time it won’t.*
So we can play the rule of thumb and wait and see, but we always have to keep one eye toward the improbable, which is still possible!
* Blatant blog cross-promotion…the point of my
big rant here was essentially the same…it’s easy to expect the most likely outcome, but the unlikely outcome will happen some of the time.
