X-Plane’s panel system does not have true “masking” based on a bitmap. You can clip an instrument to its rectangular region, but most masks are made by overlaying another layer or instrument on top of the moving parts. Examples include the circular mask for the moving map and the outside of the artificial horizon dial.
If you are using ATTR_cockpit then what you see in the OBJ is just like what you see in the 2-d panel, and the masking problem is simple: pick a mask that matches your background. In particular, if the back of the panel is black, your mask must be black; if the back of your panel has a gradient, the mask must contain a copy of a slice of the gradient.
But there are two cases where this rule does not work the way you might expect.
Masks for 2-d Spotlights
The 2-d spotlight textures (panel-2, panel-3, etc.) have a strange property: they light the background of the panel (the “burned in layer”) per-pixel but the overlays are lit per vertex. This is a cheat to keep the frame-rate high.
Normally this is not a problem – the moving parts are small and look different enough from the background that the lighting mismatch is not visible. But if you have a mask in the 2-d panel and spot lights, it will not match!
Unfortunately there is not much you can do about this. The only thing you can do is to keep the spotlight color uniform over the entire mask region.
Masks for Cockpit Regions
When you use ATTR_cockpit_region, the lighting model for your 3-d cockpit changes: instead of drawing the panel (as you saw it in 2-d), X-Plane calculates the albedo (day-time) and emissive (“_LIT”) components of the panel separately and then combines them with real 3-d lighting.
The good news is that the spot light problem is no longer a problem. Since the spot lights are 3-d and are applied to these final “panel textures”, a mask that matches the background will blend perfectly.
But in order to mask, you need to know which part of the panel texture you are masking (albedo or emissive!). If you are masking the albedo texture (e.g. a mechanical artificial horizon), create a mask that looks just like the panel background.
But for a glass instrument, the moving parts go into the emissive layer. Your mask must be pure black! Where did I get that from? The emissive layer adds light to the object. Black is an absence of adding light. So pure black ‘erases’ any light-only elements (which include all glass EFIS instruments, etc.).
One nice thing about this strategy: you can build a custom glass instrument (with a black mask) and put it over any background. This means you can reuse your art assets no matter how they are positioned on the panel.