I seem to be having the same conversation with lots of third party developers via email, so I’m going to write up some of my recent thinking on Lua – if nothing else, it will save me typing the same thing over and over.
One thing was very clear in the X-Plane 10.20 beta: while authors can’t agree on Gizmo vs. SASL*, the entire authoring community is surprisingly united around Lua as a scripting language – it’s everywhere in the payware add-on market.
But Lua is being used for a lot of different things – and these levels of usage are paralleled in Gizmo and SASL; my opinion on the use of Lua must be qualified by which usage we are talking about.
- The simplest level is “datarefs and math”. At this point, to fully utilize the aircraft SDK, an author needs to be able to create and do very simple math with datarefs – something that is not possible inside an OBJ or panel.** Right now we don’t have an easy way to do this basic dataref math. Writing a C plugin and compiling for 3 platforms is a huge hurdle to jump just to wire up an animation to a custom rotary switch.
- Gizmo and SASL both provide ‘add-on’ SDKs – custom sound APIs, particle systems, physics, gauges – basically replacement SDKs for parts of the sim’s own extension system where authors wanted more than we provide. This stuff isn’t Lua at all – the underlying systems are coded in C++ and thus can only be maintained by the C++ developer who writes the Lua-based plugin. The development cost (in man-hours) to do a particle system in Gizmo or SASL is abotu the same as it would be to build it into the sim.
- Some authors have written fairly huge scripts in Lua – for example, doing complete systems simulations or navigation code in Lua. (At least, that’s what I am told, e.g. the JAR A320 – I haven’t read the Lua scripts myself.) This is “Lua as language for a huge program.”
This is my current thinking on these three tasks:
- Datarefs and math are a great use of Lua – it lowers the bar for authors, it’s exactly what scripting languages in games are meant for, and the underlying code in C++ is finite, limited, and therefore not a maintenance problem. I don’t know what LR’s relation to Lua, Gizmo, SASL, or scripting is yet, but I have been saying for a while (internally in company discussions) that we need something easier than C for this.
- I think that if an authoring SDK is limited in X-Plane, we (LR) need to make it better. In the most useful cases where people are doing things with Gizmo and SASL, we sometimes have on our road map features to add to X-Plane that are similar. (But note that these features aren’t necessarily coming soon – authors get a time to market advantage by using these outside SDKs.) I consider this plugin code to be a possible maintenance problem. For example, you can write graphics code in a plugin, but it may not integrate well with next-generation rendering engine features.
- I don’t see huge plugins or huge scripts as something LR should get involved in. If you want to make a truly huge or complicated add-on, that’s great, but it’s a big undertaking and it just takes a development team. I don’t know if Lua is good for huge development; the people who say no (people like me) have no serious Lua experience, and the people who say yes have no serious C++ experience. So far I’ve only heard from people who have lived on one side of the grass.
Anyway, one thing is clear: having LuaJIT work in a plugin is a necessity for X-Plane now; with 10.20 we’ve sunk the engineering cost to make this happen. I do not yet know how else (or if) we will leverage Lua.
* Don’t even bother to comment on Gizmo vs. SASL – I will delete any attempts to start a Gizmo vs. SASL discussion so fast that your head will spin around 360 degrees!
** No math in OBJs or panels. An OBJ is a 3-d model – it is data that you view, not a simulation to be executed! We do not want to mix visualization with simulation or data with code!
X-Plane 10.20 rc2 is out – just a few crash fixes and the final code to support Lua-based add-ons.
If you make a third party add-on, please: go try your add-on with 10.20 rc2 now!
If we don’t find any new bug reports, 10.20 will go final next weekend.
Edit: the Kingair is, weirdly, missing a small panel of its fuselage in the right rear corner. Tom has already edited the file and I’ll post it shortly. I’m going to let the RC sit for a day or two to see if anything else washes up.
A quick follow-up on yesterday’s post regarding LuaJIT memory failures on Windows:
We will be making a change to how plugins interact with LuaJIT and X-Plane for 10.20 rc2. If you have an add-on that uses a LuaJIT-based plugin (SASL, Gizmo, or FlyLua) you must get new 64-bit binaries for your add-on! If you do not get new binaries, it will only be a matter of time before your 64-bit add-on stops working.
This change is unavoidable – it either has to happen now (in RC, when only beta testers have X-Plane 10.20) or later (when we’ve shipped the product and everyone is happily flying). I think now is better — I would rather not have to do this at all, but the memory problems are not going away.
I am already in direct contact with the plugin developers who use LuaJIT to work with them on the needed changes.
With this in mind, I am hoping to cut RC2 later this week.
A quick status update on X-Plane 10.20:
- The Portuguese language bug in the installer is fixed – thanks to all of the users who helped test the new installer.
- X-Plane 10.20 rc1 has two bugs that we have fixes for: a crash when opening radio controlled planes and a crash on 64-bit Mac under heavy load with Lua-based plugins. We have fixes for both of those; the second has been privately tested for about a week.
- We have one more new bug report that I am investigating: memory failures with Lua-based plugins and Windows. This last bug (if it needs fixing) is serious and will kill our release schedule. If we don’t need to fix it, we’ll get an RC2 out shortly. I hope to have a verdict on the bug by tomorrow.
10.20 release candidate is out now; see the release notes for a list of changes. There are two sets of bugs that we didn’t get to:
- Some users on Windows are having sound problems; I will write more about this shortly in another post; we’ll fix this as soon as we can.
- I have a set of bug reports relating to the airplane exterior lighting; I hope to get those fixed in a 10.21 build (as well as whatever one bug gets reported the day after 10.20 goes final).
Plugin authors: if your plugin has a problem with 10.20, you should have reported it weeks ago. The 2.1.2 SDK is done, 10.20 is a release candidate, so the 64-bit SDK is ready for you and has been for a while now.
We will continue to slip additional airplane improvements and autogen into updates as we get them from our art team.
ppjoy users on Windows have been experiencing a crash on startup; this was a bug in X-Plane 10.10/10.11, induced by particular virtual HID devices that only ppjoy could make. I found the problem and it will be fixed in 10.20.
In the meantime, if you need to use ppjoy and want to work around the problem, set your hat switches to discrete directions, not analog. (X-Plane can’t use an analog hatswitch anyway; most people have this because it is a ppjoy default.)
As a side rant to ppjoy users: I was a bit horrified with the process of installing ppjoy. ppjoy is an unsigned driver so I had to turn off driver signing in Windows. ppjoy is also, as far as I can tell, not hosted anywhere official. So I had to install an unsigned driver off of a file locker onto my Windows machine with the safeties off.
To be clear, I do not think that this is the author’s fault. He is making freeware, and the only thing that would remedy these problems is money. I do not and cannot expect him to give up not only his time (to code) but also pay to solve the distribution problems of official hosting and buying a signing certificate.
Still, the process of taking off all of the safeties to put random third party binary software on my Windows box was unnerving and not something I would ever do as an end-user.
As far as I know, the ppjoy crash and the PS3 controller crash are the only two known regression bugs* with joystick hardware, and they’ll both be fixed in 10.20. (Linux users, needing to edit udev rules to use hardware is not something that we consider to be a bug – see this post.)
When will 10.20 go final? Real soon now. Plugin authors, if you aren’t already running on 10.20 betas, you should have been doing that weeks ago.
* Regression bug means: it used to work in 10.05 and stopped working in 10.10 when we rewrote the joystick code.
It’s been a slow week – I’m sick, Alex was sick, Chris is sick, Chris’s wife is sick, my wife is sick, Chris’s daughter is sick, my son is sick…basically all of New England has bubonic plague. Skype meetings sound like a 19th century sanitarium for TB patients. But we are making progress on 10.20 betas. What’s still left?
- There are a handful of new 10.20 bugs that I still hope to resolve before we go final: sound problems, Intel GPU compatibility, etc.
- The installer needs to be made 64-bit aware.
- There are a handful of authoring bugs that were present in 10.11. I may push these off to a 10.21 bug fix patch, so that we can get 10.20 out the door sooner.
Users: please stop asking your favorite third party developers when they will release a 64-bit version of their add-ons. The devs are really stuck until we finalize 10.20. If they release an add-on before 10.20 goes final and then something comes up during beta, the dev is stuck fire-drilling a quick fix of the add-on.
Thanks to everyone who offered help WRT Intel GPUs. I have been in contact with the Intel driver team and we have a potential work-around for the HD4000 GPUs crashing. We do not yet have a fix or work-around for Gen-4 (GS45 chipset) GPUs crashing.
We also do not have a work-around for black sky with Intel HD GPUs and HDR mode, but honestly if you have an Intel GPU, I recommend keeping HDR off for frame-rate reasons. (I only have the HD3000 though – it’s possible that the GPUs on the new Ivy Bridge chipsets are faster. We’ll know once the shader-compiler issue is fixed.)
EDIT: Let me be more clear: we do not need any more volunteers! Thank you to everyone who helped. If you have this configuration, please just check the release notes in the next beta and test when the bugs are reported fixed.
If you are a Windows 7 X-Plane user with:
- An Ivy Bridge Intel CPU (in other words, one of those really nice shiny new ones)
- In a desktop machine (so that you can remove your GPU) and
You have an appetite for misery- You can spend an hour or two running test builds over the next day or so
Please email me (ben at x-plane dot com). I need to gather diagnostics on problems with the Intel HD4000 motherboard graphics, but I only have the previous-gen chipset.
This series of exchanges happens to me way too often:
User: X-Plane crashes when I change the flap setting.
Me: Which airplane does this happen on.
User: Any airplane.
Me: (Tries one of the default airplanes. Crash does not happen.) I could not reproduce this!
This has happened to me over and over again. Invariably, when told that a bug is reproducible with “any” materials, it turns out that the bug requires something specific, like a particular airplane or scenery pack.
Bugs that occur with any materials are far less common than bugs that happen to particular airplanes or scenery because bugs that happen all of the time are more frequent and easier to spot and fix.
If we ask you which airplane or scenery pack a bug happened on, or ask you for reproduction materials, it means that either:
- We already tried the bug on our own airplane or scenery pack and it didn’t happen or:
- We think it’s very likely that the bug is specific to something in the airplane, based on how the code works or
- We haven’t gotten any other reports of the bug, making it unlikely that it happens for all airplanes.
Now here’s the thing: if you have seen a bug, you saw it with a particular airplane or scenery pack. When you report the bug, just tell us which materials you used!
(One exception: if you’re using a custom airplane, try one of the defaults first – that’ll be the first thing we ask you anyway.)
I’ll end this rant with a philosophical thought: you shouldn’t report a bug as happening with any airplane unless you have tried it with all airplanes.
EDIT: when this article was originally written, the 2.1.2 plugin SDK was not available, which caused a lot of confusion. The new SDK is now posted, and the building instructions are completely updated.
Beta 11 is out and this hopefully has the final set of SDK changes for 64-bit plugins. Besides a bunch of other fixes (see the release notes), here’s the rough state of plugins:
- 32-bit plugins should just work. If you have a plugin that worked in X-Plane 10.11 but is broken in 10.20, please report a bug – even if it’s not your own plugin! I really want to hear about any of these cases.
- 64-bit plugins should just work; if they don’t, it may be due to programmer error, so please only report a 64-bit plugin problem if you wrote the plugin.
- A new SDK (2.1.2) is cut with real frameworks for the Mac; Sandy and I spent a pile of time working on this, but I need to update the wiki instructions. Docs coming in the next 48 hours I hope.
- Name and shame is gone for linking, so the logs should be clean. If your plugin crashes, you still get named and shamed.
