This example draws a 2-d label over the center of gravity of the user’s aircraft. It demonstrates how to implement 2-d OpenGL drawing over the screen (using a drawing callback) that matches locations in the 3-d world without using 3-d drawing callbacks. This technique works with OpenGL, Vulkan, or Metal drivers for X-Plane but requires X-Plane 11.50 or newer for datarefs.

We sometimes call this technique “coach marks” because it involves drawing markings over the 3-d world that match the 3-d world in location but do not interact with the depth buffer.

#include "XPLMDisplay.h"
#include "XPLMGraphics.h"
#include "XPLMDefs.h"
#include "XPLMUtilities.h"
#include "XPLMDataAccess.h"
#include <string.h>

/* This example plugin demonstrates how to use a 2-d drawing callback to draw
 * to the screen in a way that matches the 3-d coordinate system.  Add-ons that
 * need to add 3-d labels, coach marks, or other non-3d graphics that "match"
 * the real world can use this technique to draw on with Metal and Vulkan. */

// Datarefs for the aircraft position.
static XPLMDataRef	s_pos_x = NULL; 
static XPLMDataRef	s_pos_y = NULL; 
static XPLMDataRef	s_pos_z = NULL; 

// Transform matrices - we will use these to figure out where we shuold should have drawn.
static XPLMDataRef	s_matrix_wrl = NULL;
static XPLMDataRef	s_matrix_proj = NULL;
static XPLMDataRef	s_screen_width = NULL;
static XPLMDataRef	s_screen_height = NULL;

// 4x4 matrix transform of an XYZW coordinate - this matches OpenGL matrix conventions.
static void mult_matrix_vec(float dst[4], const float m[16], const float v[4])
{
	dst[0] = v[0] * m[0] + v[1] * m[4] + v[2] * m[8] + v[3] * m[12];
	dst[1] = v[0] * m[1] + v[1] * m[5] + v[2] * m[9] + v[3] * m[13];
	dst[2] = v[0] * m[2] + v[1] * m[6] + v[2] * m[10] + v[3] * m[14];
	dst[3] = v[0] * m[3] + v[1] * m[7] + v[2] * m[11] + v[3] * m[15];
}

// This drawing callback will draw a label to the screen where the 

static int DrawCallback1(XPLMDrawingPhase inPhase, int inIsBefore, void * inRefcon)
{
	// Read the ACF's OpengL coordinates
	float acf_wrl[4] = {	
		XPLMGetDataf(s_pos_x),
		XPLMGetDataf(s_pos_y),
		XPLMGetDataf(s_pos_z),
		1.0f };
		
	float mv[16], proj[16];
	
	// Read the model view and projection matrices from this frame
	XPLMGetDatavf(s_matrix_wrl,mv,0,16);
	XPLMGetDatavf(s_matrix_proj,proj,0,16);
	
	float acf_eye[4], acf_ndc[4];
	
	// Simulate the OpenGL transformation to get screen coordinates.
	mult_matrix_vec(acf_eye, mv, acf_wrl);
	mult_matrix_vec(acf_ndc, proj, acf_eye);
	
	acf_ndc[3] = 1.0f / acf_ndc[3];
	acf_ndc[0] *= acf_ndc[3];
	acf_ndc[1] *= acf_ndc[3];
	acf_ndc[2] *= acf_ndc[3];
	
	float screen_w = XPLMGetDatai(s_screen_width);
	float screen_h = XPLMGetDatai(s_screen_height);
	
	float final_x = screen_w * (acf_ndc[0] * 0.5f + 0.5f);
	float final_y = screen_h * (acf_ndc[1] * 0.5f + 0.5f);

	// Now we have something in screen coordinates, which we can then draw a label on.

	XPLMDrawTranslucentDarkBox(final_x - 5, final_y + 10, final_x + 100, final_y - 10);

	float colWHT[] = { 1.0, 1.0, 1.0 };
	XPLMDrawString(colWHT, final_x, final_y, "TEST STRING 1", NULL, xplmFont_Basic);		
	
	return 1;
}

PLUGIN_API int XPluginStart(char * outName, char * outSig, char * outDesc)
{
	strcpy(outName,"Example label drawing");
	strcpy(outSig,"com.laminar.example_label_drawing");
	strcpy(outDesc,"A plugin that shows how to draw a 3-d-referenced label in 2-d");
	
	XPLMRegisterDrawCallback(DrawCallback1, xplm_Phase_Window, 0, NULL);
	
	s_pos_x = XPLMFindDataRef("sim/flightmodel/position/local_x");
	s_pos_y = XPLMFindDataRef("sim/flightmodel/position/local_y");
	s_pos_z = XPLMFindDataRef("sim/flightmodel/position/local_z");

	// These datarefs are valid to read from a 2-d drawing callback and describe the state
	// of the underlying 3-d drawing environment the 2-d drawing is layered on top of.
	s_matrix_wrl = XPLMFindDataRef("sim/graphics/view/world_matrix");
	s_matrix_proj = XPLMFindDataRef("sim/graphics/view/projection_matrix_3d");

	// This describes the size of the current monitor at the time we draw.
	s_screen_width = XPLMFindDataRef("sim/graphics/view/window_width");
	s_screen_height = XPLMFindDataRef("sim/graphics/view/window_height");
	
	return 1;
}

PLUGIN_API int XPluginEnable()
{
	return 1;
}

PLUGIN_API void XPluginStop()
{
}

PLUGIN_API void XPluginDisable()
{
}

PLUGIN_API void XPluginReceiveMessage(XPLMPluginID inFrom, int inMessage, void * inParam )
{
}

14 comments on “Drawing 2-D That Matches the 3-D world”

  1. Thanks for this code!
    I have used something similar so far, originally likely also by you at Laminar… 😉 In this reduced way labels would “mirror” at the back of the camera, though, i.e. if turning the camera by 180° the label would show up again. Code I’ve seen and used so far used complex code to determine if a sphere is visible. For pure label drawing it would be sufficient to know if the label is in fact “in front” or “behind” a camera. Looking at how values in acf_ndc end up it seems that “acf_ndc[2] < 1.0f" would tell me if the position is in front. That seems to work for me, though I am unable to tell what acf_ndc[2] actually describes, a value otherwise unsed. Does my observation make any sense? Or what would be the fastest way to avoid mirroring?

    Reply

    1. NDC = normalized device coordinates; [0] and [1] are the X and Y axis from -1 to 1, and [2] is the Z-buffer coordinate, mapped from -1 to 1. (OpenGL is a little weird in that the Z sense gets reversed in NDC.

      So naively I would expect acf_ndc[2] > -1.0 to get you “in front of the camera” but I haven’t tested it.

      You’re definitely onto something – acf_ndc[2] is the Z value and anything outside -1..1 is off screen.

      (If you are in Vulkan you might be in reverse Z in which case < 1 might be correc?

      Reply

    2. You are correct the code above needs a fix to exit early if ndc[2] = (n+f)/(n-f). In general though n is much smaller than f (by many orders of magnitude), so the last fraction is very close to -1 anyway.

      Reply

    3. [sorry my above comment was cut I don’t know how, I repeat the original here]

      You are correct the code above needs a fix to exit early if ndc[2] = (n+f)/(n-f). In general though n is much smaller than f (by many orders of magnitude), so the last fraction is very close to -1 anyway.

      Reply

  2. [third try : I understood, it doesn’t want the “less then” sign…maybe you can remove both others]

    You are correct the code above needs a fix to exit early if ndc[2] less than -1 (it is supposed to be OpenGL convention during 2D callbacks, but check it though). Usually such kind of computations are done inside a shader and there is no need to check for the clipping since this is done automatically by the graphics library, but here it is done on the CPU and then only fed into a 2D pipeline, which has no idea about what ndc[2] was.

    Ndc coordinates are those that live after the optical projection, i.e. after projecting the pyramid shaped “physical” view frustum into a “non physical” cube (you may have a look here e.g. //www.songho.ca/opengl/gl_projectionmatrix.html if you are not familiar with).

    In OpenGL convention ndc[2] = -1 exactly means that the point lives on the near frustum plane ( “near clip plane”), and ndc[2] = 1 arises when the point is on the far plane. You don’t have access to the near and far planes directly through datarefs, but if you really need them they can be inverted directly from the projection matrix, and in that case “in front of the camera” would exactly be ndc greater or equal (n+f)/(n-f). In general though n is much smaller than f (by many orders of magnitude), so the last fraction is very close to -1 anyway.

    Reply

  3. I tried this and it draws on top of everything. Is there a way to mask out for example the panels, or maybe draw in a different drawing phase?

    Reply

  4. How about a sample on how to draw in a window with Vulkan ?

    ie : XPLMCreateWindowEx then figure out the position and dimensions, then draw in 2D there, this was working perfectly using opengl.

    I was also using X-Plane to work on terrain rendering by using xplm_Phase_Terrain, it would be nice to have a similar way of doing this with Vulkan.

    On a different subject it is too bad that I have to accept you getting my personnal data in order to ask a question. Not too sure how this fits GDPR here as I am an european citizen.

    PhM

    Reply

    1. You cannot render to X-Plane using the Vulkan API, _even_ if the sim is running in Vulkan.

      GDPR: If you _comment_ on the website, WordPress saves a bunch of info about you for anti-spam and comment moderation. It does this “out of the box”. That’s not negotiable; GDPR considers virtually everything including your IP to be personal data, so a comment feature that didn’t require a GDPR notice would be completely anonymous and untraceable, and therefore would just be a giant spam magnet. We wouldn’t be able to reply because we wouldn’t have time to look for real comments amongst a sea of unstoppable spam.

      Reply

  5. Hello,
    I tried this code, and it can perfectly drawing labels for single monitor setup.
    But if I have 2 or 3 monitors, it can’t draw at other monitors.
    Will you provide the multi monitor drawing sample code?
    Thanks

    Reply

    1. I can confirm this: No labels on 2nd monitor. Also, users report seeing labels in one VR eye only, which might be related (assuming the second eye is technically a second screen). I sense two issues here, which might not be simple to overcome:
      – All projection matrices are (potentially) different on different screens as they are showing the scene from different view points;
      – XLMDrawString (and simlar functions using pixel coordinates) cannot yet draw on the second screen.
      One would assume that for several screens there would be a need for calling the drawing callback once per screen during each cycle, but I can’t find a way of making that happen: It is called once only for the main screen.

      Reply

      1. hmmm – if you set the drawing callback to the _before_ windows phase (the example shows after) do you get multi-monitor working? The second eye is not a second screen, it’s a second pass. But the ‘after’ phase isn’t in the HMD at all, the before is.

        Reply

          1. Also interested in how to overcome this behavior. I have 2 monitors and can’t see labels on the second one.

  6. I am testing this code and it works great when you view the aircraft from outside the cockpit (like chase view). But the label does not appear when inside the 3D cockpit. I tried using different XPLMDrawingPhase enums, but nothing worked. The sample code uses xplm_Phase_Window. I tested this on the default Cessna 172. My goal is to show coaching arrows pointing to different instruments on the panel. Thanks!

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

Please do not report bugs in the blog comments.
Only bugs reported via the X-Plane Bug Reporter are tracked.