XPLM_API int XPLMIsWindowInFront( XPLMWindowID inWindow);
This routine returns true if the window you passed in is the frontmost visible window in its layer (XPLMWindowLayer).
Thus, if you have a window at the front of the floating window layer (xplm_WindowLayerFloatingWindows), this will return true even if there is a modal window (in layer xplm_WindowLayerModal) above you. (Not to worry, though: in such a case, X-Plane will not pass clicks or keyboard input down to your layer until the window above stops “eating” the input.)
Note that legacy windows are always placed in layer xplm_WindowLayerFlightOverlay, while modern-style windows default to xplm_WindowLayerFloatingWindows. This means it’s perfectly consistent to have two different plugin-created windows (one legacy, one modern) both be in the front (of their different layers!) at the same time.