We have a small handful of web sites (mostly WordPress, but with one custom Node.js+Express app) that we’re looking to hire someone to provide ongoing maintenance for, including bug fixes, new features, dependency updates, etc.
This would be part time (ideally about 10 hours a week) and remote, working whatever hours you prefer. We’d like to start by dividing your time in half between WordPress and Node, but we may shift that balance as time goes on.
This is something we have an ongoing need for, so if you’re a freelancer, this is an easy way to add some stability to your schedule for potentially years.
About the sites you’d be working on
On the WordPress side, we version the themes and all essential plugins in Git, and deploy them via Git push to WP Engine’s servers. We have automated tests in place for a lot of the essential sales functionality, so that you can deploy to a staging environment, test it, then deploy to production once all the tests pass. Our sales site runs on WooCommerce, with a small handful of custom plugins to deal with weird stuff specific to our business.
Our Node app (the Scenery Gateway) is a glorified CRUD frontend for users to upload scenery for the flight simulator. We have a reasonable amount of documentation on the organization of this app and its integration tests—see the README for more info.
Qualifications
A qualified candidate will have:
Outstanding judgement and ability to self-direct. We won’t be looking over your shoulder constantly, so we’ll expect you to be able to prioritize issues, manage your time, and make responsible decisions on the future of these projects.
Experience working remotely.
Experience in both JavaScript and WordPress development. (You don’t need to have written Node.js or Angular.js specifically—experience with React, Ember, Knockout, Vue, <insert JS flavor of the week> are fine.)
Excellent written communication skills in English.
Long-term availability for roughly 10 hours a week.
What to expect from us
We won’t be doing any sort of grueling interviews for this position—we expect to hire based solely on seeing your past experience, plus maybe a Slack chat or two.
We do our best to cultivate long-term relationships with contractors—we want you to love working with us. With that in mind, we will:
Pay you competitively.
Pay you on time, every time.
Respect you as a professional.
How to apply
If this is sounds like a good fit for you, please shoot me an email with the following (my email is my first name at X-Plane.com):
Some code samples/a GitHub link/your Stack Overflow account/etc.—anything that would demonstrate your knowledge of both WordPress & JavaScript stuff.
One or more references we could contact about past freelancing work you’ve done (doesn’t have to be web related)—we’re basically looking to have a third party confirm that you have good judgement, good communication skills, etc.
Fair warning: I expect to get an absolute flood of emails here, so I won’t be able to respond to every one. I’m planning to make a hiring decision two weeks from today (on May 13th), so if you’re interested, please let me know ASAP. 🙂
Since my last Vulkan update, we now have the full sim running natively with Vulkan and Metal! There is still a pretty big list of random things turned off or bypassed to make this happen, but we can fly in the cockpit and use the sim.
Here’s some stuff that is now working:
Using regular and HDR rendering, and SSAO on metal.
Flying with Vulkan on AMD, NVidia, and Intel drivers.
Hardware stereo rendering on OS X – this never existed before Metal because the Mac GL drivers didn’t support it. Hardware stereo rendering is necessary for VR support.
MSAA on Metal – with the restructuring of our code, you’ll be able to change MSAA settings without restarting. I don’t know if this code works on Vulkan right now; it might.
Here’s some stuff that does not work yet:
Plugins are bypassed right now. We have not yet written the plugin-OpenGL-interop layer.
VR only works when using OpenGL as the driver; we need to write some new VR code to pass Metal and Vulkan frames directly to the OVR and Rift APIs.
Screenshots/Movie capture are a work in progress – that’s what I’ve been working on this week.
We still have a number of visual bugs, so screenshots are an important feature so we can run our automated test system. The test system takes hundreds of screenshots of the sim in many configurations and compares them to 11.30 to catch bugs introduced by the new Vulkan and Metal back-ends. Clearly I’ll have to fix my color problems first.
One fun aspect of this port: Metal and Vulkan copy Direct3D’s convention where the viewport Y axis points down and not up. This resulted in a whole series of weird “it’s upside-down again” bugs, most of which have been fixed.
The world is drawn correctly but the atmosphere around it is upside down.
(That airplane might look silly, but at least all of the image is consistently upside down. That’s because the only remaining upside down code is the movie/screenshot capture code itself.)
We should have some performance numbers to post next week; right now our primary focus is fixing bugs, particularly bugs that bring the whole machine down.
Here are a few more pics of things that have gone wrong during development.
X-Plane 11.33b1 is an incremental update that includes updated Gateway airports and translations, crash fixes and diagnostics, minor UI improvements, and bug fixes such as:
XPD-9441 Weight, balance and fuel- Total weight (lbs) does not include weapons.
XPD-9683 Fix dupe banks in FMOD crashing the sim.
XPD-9882 Fixed KOAK nav data.
XPD-9919 Lightning appears in cockpit in VR.
XPD-9991 Crashes on older Intel GPUs.
XPD-9992 Default FMC, Holding Pattern will only be flown once.
XPD-10003 Landing lights has no effect on battery amperage draw.
Just a reminder that the sim will not prompt you to install this beta–you will need to launch the installer manually and opt into betas to get this version.
We are putting more effort into cleaning up frequent crashes. We recently upgraded our back end crash reporting to a fancy paid service that we hope will allow us to gain a lot more insight into what is going wrong. This beta includes additional crash logging for that purpose.
Keeping in the crash-fixing vein, we believe we also fixed two notable crashes in this beta: issues with Intel GPUs and with duplicate FMOD sound banks. We heard from a lot of Intel users about these crashes, so Sidney took a look and was able to find a fix for it. The FMOD crash has been a round for a while and was caused by duplicating an aircraft folder that included FMOD. When an FMOD bank had the same contents on disk at a different file path, FMOD wouldn’t load it. We now handle this case gracefully.
Update: Steam users can get 11.33b1 by selecting the public beta under application properties.
We’re looking to add a part-time technical support team member. This person will share the workload of customer support and have the opportunity to interact with customers to further positive relationships with X-Plane.
Daily Activities
Your day will include:
External communication – using email tools to answer customer requests for help
Internal communication – collaborating with team members to facilitate customer service
Taking opportunities to improve documentation as needed
Reporting and documenting technical issues for resolution
Building positive customer relationships
Some relevant experience for these tasks are:
Excellent written communication in English
Familiarity with both Mac and Windows
Passion for assisting people
Ability to self-manage
Bonus points for experience in:
Using X-Plane
Technical writing
WordPress, JIRA, and / or HelpScout systems
Why work for us?
As a member of our team, you would:
Work on stuff that matters. Real pilots fly safer because of training in X-Plane, and real aerospace organizations (like Boeing, Cessna, and NASA, to name just a few) prototype aircraft in X-Plane before they build them in the real world. Working in customer support is a great opportunity to empower our users to unlock the full potential of X-Plane.
Work remotely. No commute, no cubicles, nothing to impede you from doing great work. (But the rest of the team is just a Slack call away!)
Have the opportunity to follow your interests into other aspects of the business.
How to apply
If this sounds like a good fit for you – we’d love to hear from you! You can submit this form to apply.
Just a quick update on our progress with Vulkan and Metal. We last spoke about this on the live feed a few weeks ago, but we’re a little further along. Here’s the summary:
Plane-Maker and Airfoil Maker run in Vulkan and Metal.
X-Plane runs in Vulkan and Metal up to the main menu (e.g. the app starts) but can’t yet fly or show scenery.
The Vulkan and Metal code runs on Mac, Windows and Linux.
The Vulkan code runs on Nvidia, AMD and Intel drivers.
All shaders are ported.
All of the zoo animals (abstractions around part of the graphics engine) are now complete. We killed off the last 2 or 3 since the live feed.
For the last two weeks, Sidney and I have been working to port all rendering passes to the new code, so we don’t have to go through OpenGL to fly the aircraft. I lost a few days dealing with this:
That turned out to be a single if statement that got reversed deep in the mesh drawing code during one of the porting steps, resulting in some of the runway lights being replaced with random pieces of … heck, I never figured out what the wrong mesh was, just that it came from some other unrelated part of the sim and changed as the camera moved. To make matters worse, the error only appeared on the Mac, so we couldn’t use Windows OpenGL debugging tools to find the issue.
The silver lining is that once we are all Vulkan/Metal, we’ll have at least four separate debugging tools to go after bugs like this.
We don’t have measurements of performance yet; once we can sit in an aircraft in X-Plane running Metal or Vulkan, we can at least get some initial performance numbers.
If any folks from the flight sim community are going to be there as well, I’d love to meet up and talk shop—hit me up on Twitter (@TylerAYoung), or send me an email (my email is my first name at X-Plane.com).
Please note that this is a major update and the minimum system requirements have now changed to: 64 bit versions of Windows 7/10, OSX 10.9, or Linux with GLIBC 2.23. Read More
We’ve posted X-Plane 11.32 release candidate two – it contains very few changes from release candidate one. If you think RC2 changed your framerate from RC1 (for better or worse), it is imaginary.
The big item for RC2 is that X-Plane now uses our own replicating METAR servers. NOAA weather was plagued by 404s when the server posted METARs didn’t meet the date/time scheme X-Plane expected. Our own server should serve the latest weather we have, whatever that is.
Over the last few weeks we have spent a tremendous amount of developer time investigating reports of instability, crashes and performance problems, and the results have been quite unsatisfying. We really haven’t found a series of smoking guns we could fix to improve stability. We have learned some things about X-Plane’s performance and stability though. The rest of this post gets into the weeds; if you tune out (and I won’t fault you if you do) the TL;DR is: please turn on our anonymous analytics, and click “send” if you get the crash report form. The more gathered data we get about crashes, the better shot we have at addressing the issues.
Crash Rates and Plugins
X-Plane’s overall crash rate (all causes is approximately 14% – which is to say, for all of our users using analytics, for every 100 times they launch X-Plane, the sim quits in a way we did not expect or want 14 times. This number has been remarkably stable – it’s not a ton different between 11.26, 11.30, 11.31 or the 11.32 beta.
The sim quitting on purpose because of bad content is not considered a crash. For example, if you load a DSF with a missing .ter file, the sim will refuse to proceed and quit. This failure mode is user hostile in that you can’t fly, but it’s not a crash – the refusal to load is the code working according to design. While I would like to make this code less hostile to users, it’s also worth noting that these cases are ones where the author of the scenery pack would have been able to fix this if they had loaded their own work even just once. That is, they are caused by an add-on that should not have shipped.
(There is an in-between area where an add-on is mis-installed because it has a library dependency that the user hasn’t met. This is a deployment problem that really needs to be solved, but it’s orthogonal to true app crashes.)
We categorize crashes into plugin and non-plugin crashes; starting with 11.32 we actually get a statistical picture of this. A crash is a plugin crash (and you see the “we crashed because of a plugin: XSquawkBox” or whatever) if the sim crashed while executing code on behalf of a plugin or inside the plugin on the main thread.
There are a bunch of cases where plugins do not get correctly tagged – in particular, we regularly see crashes on random worker threads spawned by plugins; since we don’t know whose thread it is, we can’t blame the plugin. For example, the FF A320 crashing inside CEF on a worker thread is registered as an X-Plane crash (and we see it in our auto-reporting view) but it’s not our code and there’s nothing we can do about it.
One thing I’d like to do in future patches is improve diagnostics. The rate of actual blamed plugin crashes appears so far to be quite low, and given that we do see uncaught plugin crashes in our data on a regular basis, I think this is a case where add-on authors can only fix what they can see. If we can attribute all plugin crashes to plugins, then the plugin authors can catch their own bugs. Better diagnostics also helps a user remove a troublesome add-on in the case where that would help.
“Known” Crashes
We have a few cases where X-Plane hits an error condition and deals with it by crashing. This is pretty bad, drives up our crash rate, and is something we need to fix to be less user hostile. For example, if a PNG file is bad (either corrupt contents or the sector on disk that backs it has gone bad) then X-Plane’s response is usually to mysteriously crash. Besides being rude, the crash gives an end user no idea which file is bad, and thus no way to fix it. X-Plane ships with something like 9000+ PNG files and over 2500 DDS files, not counting add-ons, so if we don’t tell you which file is bad, you’re not going to find it by poking around.
FMOD sound bank incompatibilities is another example – if you have two aircraft sharing a byte-wise copy of the same FMOD data (e.g. command-D duplicate the C172) loaded at once – then our FMOD loading code fails and then registers as a crash. The code is working exactly as we designed it, but the design isn’t robust enough. As we’ve learned, in the real world users duplicate aircraft (and their FMOD packs) on disk all the time.
The crashes in this category are things where we’re being user-unfriendly and better code would make these problems go away or leave users with a way to actually fix them. But they’re not case of “weird stuff inside the sim blew up.”
Persistent Stability Problems
In the crash data we do also see a few persistent stability problems. There’s some kind of crash in the ATC system that we’ve seen for a very long time but we don’t know how to reproduce. It’s a case where if enough users do enough random stuff, we hit an edge case in the ATC system that isn’t handled correctly. The solution here is to embed more diagnostics at the crash site until we can understand it from the reports we get from users. Please hit “send” when you crash – don’t worry about filling in the fields – it’s the report itself that we need.
We also see a lot of crashes inside the OpenGL drivers, from all of NVidia, AMD and Intel. Because the IHVs don’t share symbols and source code with us, we really can’t tell what went wrong in these cases.
My hope is that with Vulkan we’ll have better options for in-driver crashes. With Vulkan, they redesigned the error checking model: error checking is a feature you enable (via a configuration option at app startup) that brings a layer of code in on top of the driver to check what the app is doing. With error checking off we get the fastest framerate, and with error checking on we get slow framerate (more error checking means more slow) but some really great diagnostics.
(To put this in perspective: when Sidney ran the Vulkan version of Airfoil-Maker on Linux with the wrong driver installed and no error checking, it rebooted his entire Window server! So no error checking really means: no error checking.)
Since error checking is optional and selectable when the app runs, we could put an option into X-Plane to run in “safe mode” – if a user is hitting persistent stability problems in the driver, that user could turn on validation and possibly capture an error in X-Plane itself that would otherwise just be “the driver crashed.”
Running Out of Memory
I’ve worked with a few users to try to track down the out of memory problems we’ve heard about, and there isn’t an obvious pattern here. Some users report running out of memory in 11.30, but when put back on 11.26, find that they still run out of memory. We get more out-of-GPU-memory complaints on 11.30, but that might be because a few cases in 11.26 that were crashes due to running out of memory now report the problem in an orderly way. In 11.26 they were just mysterious crashes with a “send” form.
In the cases we’ve seen, the user running out of memory was often…actually running out of memory – that is, the surprising thing is not the crash but that X-Plane ever worked at all on those settings. The fundamental problem we face is that we have no visibility into what the OpenGL driver is doing with GPU memory. The OpenGL tries to manage memory no matter how much we ask for, and if it fails, we don’t know what went wrong.
The good news is: we have much better options for Vulkan. With Vulkan, we manage memory, which means we know what’s going on, and we can take steps to avoid out of memory crashes. If we do run out of memory, it should be for much more obvious reasons. We’re still analyzing what we can do about memory with Metal, but the choices should still be better than OpenGL.
The only advice I can offer now if you are seeing persistent memory crashes is: turn your settings down or use less add-ons. If you push X-Plane to the limits of your hardware, it may work for a while and then fail.
Performance
Sidney has looked at a lot of performance data from users who reported low framerate, and in almost every case, the performance has been as-expected. The most common case we see is users with relatively low single-core CPU performance hitting low framerate at high rendering settings while their GPU is bored. To put some numbers on this, if your CPU’s single-core geekbench score is down around 2000, you are almost certainly CPU bound, way at the bottom of what’s okay for X-Plane, and a new GPU won’t help.
As of now, X-Plane cannot use large numbers of cores (e.g. a 32-core machine is useless) and gets only limited performance boosts for framerate with multiple cores under some circumstances. This is something we are working to change in the future, but it’s not going to change quickly. If you are looking to improve performance with hardware, single core speed is still the most important metric.*
In a few rare cases we saw performance that was disproportionately bad compared to what we’d expect from an old CPU. We’re trying to gather more data from these users but the case is rare enough that we haven’t gotten a useful report yet. If we find a smoking gun, we can act on it.
Like memory, Vulkan will help with diagnosing performance. With Vulkan, more of the code is written by us and the Vulkan code we run has very predictable timing. So when we get complaints about performance, we’ll be in a much better position to understand what is slow and why.
What’s Next
I’m actually not sure what the next patch will be, but we do have a bunch of bug fixes to 11.30 waiting to go out once we have stabilization under control. I also have a pile of bugs that I have not yet fixed that are high on my priority list where something in 11.26 stopped working in 11.30. So if you have filed a bug that’s not fixed, we have not forgotten about it – it’s either next to come out or possibly on the short list.
* Yes, we realize that this dependence on single core speed is bad. It’s just going to take time to move to Vulkan and then offload the single thread.
We have received reports in the past about X-Plane 11’s seaplane dynamics when the aircraft is in the water.
If you would like to test sea plane water dynamics with an experimental build, please email Austin directly – he has some fixes for seaplane behavior and he is looking for early testing.
Update: fearless leader^H^H^H^H^H^H^H^H^H^HAustin says he is only looking for testers who have experience flying a real world small float plane with pontoons, so that he can get test feedback from someone who can validate the reality of the physics.
(You will need to install a custom build from him on top of X-Plane 11.31 to do this.)
Update 2: fearless leader^H^H^H^H^H^H^H^H^H^HAustin say he has enough testers. If you’re not getting builds now, you’ll have to wait for a public beta.
This blog post is more or less a “stern talking to” for plugin developers, but before I go there, I want to acknowledge that we (Laminar) screwed up the docs here in a way I didn’t even realize until working on a bug report. A decade ago when the plugin SDK was young, we did have clear docs that the plugin APIs were not at all thread-safe. Sandy and I were also on the plugin dev email list and we’d administer a Scottish-style beating to anyone who even started typing the first few letters of “threading”.
However, this document was lost in the migration from the old XSquawkBox server to developer.x-plane.com. I’m working with Jennifer on new docs now, and I apologize for the thrash any plugin developer gets hit with from not knowing the threading guidelines.
With that in mind: the X-Plane plugin API is not thread-safe. You can only call plugin APIs on the thread that called you. No exceptions!
The plugin SDK was invented for X-Plane 6. At the time, multi-CPU and multi-core hardware was totally unavailable to the flight sim community. Apple hadn’t released the dual-G5 yet, and the Pentium D hadn’t come out yet. There was no point in thinking about multi-core because there weren’t multiple cores.
The plugin guidelines were therefore set up very simply: the API is single threaded; call us back on the thread we call you. The SDK internally has no locks or handling of re-entrancy and has no model to cope with resource sharing or data integrity across threads.
Furthermore, X-Plane’s crash detection system is not meant to categorize other-thread crashes, so you can’t easily tell that your plugin is crashing the sim. You might even crash a different plugin, or some internal part of X-Plane.
One more way we’re not thread safe that might not be obvious: when you read a dataref, you are executing code from another plugin. You can’t do this on a thread if the XPLM isn’t thread-safe, but you also can’t do this if the other plugin isn’t thread safe.
Stopping the Bleeding
In X-Plane 11.30 we made a change to stabilize the sim: we added code to actively detect and ignore plugin calls to the XPLMTerrainProbe APIs from background threads. We did this after seeing automatically reported crashes that turned out to be due to plugins calling the SDK terrain probe API from worker threads. By ignoring the call, we avoid the crash.
My plan is to start doing this for all plugin calls at a patch point in the next few months. The problem here is that there’s basically no such thing as a benign threading bug – the table stakes here are the complete destabilization of the sim.
If you are a plugin author and you are using background threads to call XPLM APIs, please stop doing this now. Please plan to fix this in your plugin as soon as possible. The change will probably not make things any worse – my current idea is to no-op the calls, just like we did with terrain probes. But if your plugin is using these async calls and sometimes succeeding but sometimes crashing, you’re going to stop seeing the crashing and the sometimes lucky “success”.
I’m working with Tyler and Jennifer on a docs update now – hopefully this week all of the docs should be completely consistent. But they’re going to say what I’ve said above: no calls to the XPLM on worker threads.
Will We Ever Be Thread-Safe?
Sandy and I did some work back in the XPLM 2.0 days to start making the SDK partly thread-safe. This work was not completed, but the idea was to at least make a small number of APIs callable from worker threads. For example, by making flight loop callbacks schedulable from worker threads, a plugin could “wake up” SDK code from an async IO callback. That idea still makes some sense and we may get there someday.
For expensive tasks, we’ve already made API changes to address the underlying performance problems. For example, you can’t load an object synchronously from a thread, but you can ask us to load it asynchronously and call you back, and we use one of our threads to offload the work.
Some APIs I expect to never be thread safe. For example, we can’t sanely provide a threaded API to datarefs because we can’t promise that the plugin on the other side of the call is thread-safe. Given that a dataref read function can call other datarefs or other arbitrary plugin code, the opportunity for dead-locks is limitless.
Tyler’s Fever Dream
I should mention something that Tyler has been looking at as a future SDK initiative. This is somewhere between pie-in-the-sky and a fever-dream: plugins running asynchronously in separate processes.
The idea is to have a version of the main SDK APIs that are “fundamentally asynchronous” (e.g. the response comes later and that’s baked into the contract). This would allow plugins to run in another process, with results being communicated via IPC. Out-of-process plugins would have a bunch of advantages:
You could write a plugin in pretty much any language we can write a binding for. Right now, plugins must fit into an unmanaged DLL inside X-Plane; this would allow the full “async API” to be used in .net or even Matlab. The requirement would only be that the plugin “app” environment be able to host unmanaged C code. This is a much less difficult requirement to meet.
Plugins would be isolated by process boundaries; if a plugin crashes, the sim keeps running. Plugins can’t go around trashing each other’s memory.
Use of multiple cores for plugin CPU processing is basically free, because the plugins can execute in parallel.*
Plugins can each have their own CEF instance – or any other library that doesn’t like to be multiply instantiated in a single process.
One of the major hurdles to implementing an API like this was sharing graphics across the process boundary. As it turns out, we have to solve this problem for Vulkan anyway – the same APIs that share surfaces between Metal/Vulkan and OpenGL are IPC-friendly from day 1. So a 2-d plugin OpenGL window in Vulkan doesn’t have to be in X-Plane’s process.
I don’t see async plugins ever replacing 100% of in-process plugins, and this isn’t a plan to kill off the current C API. But I do think out-of-process plugins would be a much better fit for a wide range of plugin tasks.
* Users familiar with game programming will appreciate one of the fundamental dangers here: plugins in other processes might steal oversubscribed cores from the sim itself, causing stutters and unreliable framerate. Tyler and I have talked about that a little bit, and have some ideas for taking plugin background work into account and giving plugins ways to opt in and say “please run this background work for me at a time that won’t hose framerate.”
