VR: Initial Virtual Reality support - Milestone 1, Scene Inspection

NOTE: While most of the milestone 1 goals are there, a few smaller features and
improvements are still to be done.

Big picture of this milestone: Initial, OpenXR-based virtual reality support
for users and foundation for advanced use cases.
Maniphest Task: https://developer.blender.org/T71347
The tasks contains more information about this milestone.

To be clear: This is not a feature rich VR implementation, it's focused on the
initial scene inspection use case. We intentionally focused on that, further
features like controller support are part of the next milestone.

- How to use?
Instructions on how to use this are here:
https://wiki.blender.org/wiki/User:Severin/GSoC-2019/How_to_Test
These will be updated and moved to a more official place (likely the manual) soon.

Currently Windows Mixed Reality and Oculus devices are usable. Valve/HTC
headsets don't support the OpenXR standard yet and hence, do not work with this
implementation.

---------------

This is the C-side implementation of the features added for initial VR
support as per milestone 1. A "VR Scene Inspection" Add-on will be
committed separately, to expose the VR functionality in the UI. It also
adds some further features for milestone 1, namely a landmarking system
(stored view locations in the VR space)

Main additions/features:
* Support for rendering viewports to an HMD, with good performance.
* Option to sync the VR view perspective with a fully interactive,
  regular 3D View (VR-Mirror).
* Option to disable positional tracking. Keeps the current position (calculated
  based on the VR eye center pose) when enabled while a VR session is running.
* Some regular viewport settings for the VR view
* RNA/Python-API to query and set VR session state information.
* WM-XR: Layer tying Ghost-XR to the Blender specific APIs/data
* wmSurface API: drawable, non-window container (manages Ghost-OpenGL and GPU
  context)
* DNA/RNA for management of VR session settings
* `--debug-xr` and `--debug-xr-time` commandline options
* Utility batch & config file for using the Oculus runtime on Windows.
* Most VR data is runtime only. The exception is user settings which are saved
  to files (`XrSessionSettings`).
* VR support can be disabled through the `WITH_XR_OPENXR` compiler flag.

For architecture and code documentation, see
https://wiki.blender.org/wiki/Source/Interface/XR.

---------------

A few thank you's:
* A huge shoutout to Ray Molenkamp for his help during the project - it would
  have not been that successful without him!
* Sebastian Koenig and Simeon Conzendorf for testing and feedback!
* The reviewers, especially Brecht Van Lommel!
* Dalai Felinto for pushing and managing me to get this done ;)
* The OpenXR working group for providing an open standard. I think we're the
  first bigger application to adopt OpenXR. Congratulations to them and
  ourselves :)

This project started as a Google Summer of Code 2019 project - "Core Support of
Virtual Reality Headsets through OpenXR" (see
https://wiki.blender.org/wiki/User:Severin/GSoC-2019/).
Some further information, including ideas for further improvements can be found
in the final GSoC report:
https://wiki.blender.org/wiki/User:Severin/GSoC-2019/Final_Report

Differential Revisions: D6193, D7098

Reviewed by: Brecht Van Lommel, Jeroen Bakker

12 files changed, 111 insertions, 23 deletions

diff --git a/intern/ghost/CMakeLists.txt b/intern/ghost/CMakeLists.txt
index 07d98475c00..611da8d6a44 100644
--- a/intern/ghost/CMakeLists.txt
+++ b/intern/ghost/CMakeLists.txt
@@ -377,6 +377,9 @@ if(WITH_XR_OPENXR)
   list(APPEND INC_SYS
     ${XR_OPENXR_SDK_INCLUDE_DIR}
   )
+  list(APPEND LIB
+    ${XR_OPENXR_SDK_LIBRARIES}
+  )
 
   set(XR_PLATFORM_DEFINES -DXR_USE_GRAPHICS_API_OPENGL)
 
diff --git a/intern/ghost/GHOST_C-api.h b/intern/ghost/GHOST_C-api.h
index aafe374a93b..aba5b5f733b 100644
--- a/intern/ghost/GHOST_C-api.h
+++ b/intern/ghost/GHOST_C-api.h
@@ -759,6 +759,18 @@ extern GHOST_TSuccess GHOST_ReleaseOpenGLContext(GHOST_ContextHandle contexthand
 /**
  * Get the OpenGL framebuffer handle that serves as a default framebuffer.
  */
+extern unsigned int GHOST_GetContextDefaultOpenGLFramebuffer(GHOST_ContextHandle contexthandle);
+
+/**
+ * Returns whether a context is rendered upside down compared to OpenGL. This only needs to be
+ * called if there's a non-OpenGL context, which is really the exception.
+ * So generally, this does not need to be called.
+ */
+extern int GHOST_isUpsideDownContext(GHOST_ContextHandle contexthandle);
+
+/**
+ * Get the OpenGL framebuffer handle that serves as a default framebuffer.
+ */
 extern unsigned int GHOST_GetDefaultOpenGLFramebuffer(GHOST_WindowHandle windwHandle);
 
 /**
diff --git a/intern/ghost/GHOST_IContext.h b/intern/ghost/GHOST_IContext.h
index a341e18ca0a..33422b8e351 100644
--- a/intern/ghost/GHOST_IContext.h
+++ b/intern/ghost/GHOST_IContext.h
@@ -56,6 +56,15 @@ class GHOST_IContext {
    */
   virtual GHOST_TSuccess releaseDrawingContext() = 0;
 
+  virtual unsigned int getDefaultFramebuffer() = 0;
+
+  virtual GHOST_TSuccess swapBuffers() = 0;
+
+  /**
+   * Returns if the window is rendered upside down compared to OpenGL.
+   */
+  virtual bool isUpsideDown() const = 0;
+
 #ifdef WITH_CXX_GUARDEDALLOC
   MEM_CXX_CLASS_ALLOC_FUNCS("GHOST:GHOST_IContext")
 #endif
diff --git a/intern/ghost/GHOST_Types.h b/intern/ghost/GHOST_Types.h
index adda782b96d..70c4d3ef00c 100644
--- a/intern/ghost/GHOST_Types.h
+++ b/intern/ghost/GHOST_Types.h
@@ -598,6 +598,8 @@ typedef void (*GHOST_TimerProcPtr)(struct GHOST_TimerTaskHandle__ *task, GHOST_T
 
 #ifdef WITH_XR_OPENXR
 
+struct GHOST_XrError;
+struct GHOST_XrDrawViewInfo;
 /**
  * The XR view (i.e. the OpenXR runtime) may require a different graphics library than OpenGL. An
  * offscreen texture of the viewport will then be drawn into using OpenGL, but the final texture
@@ -605,7 +607,7 @@ typedef void (*GHOST_TimerProcPtr)(struct GHOST_TimerTaskHandle__ *task, GHOST_T
  *
  * This enum defines the possible graphics bindings to attempt to enable.
  */
-typedef enum {
+typedef enum GHOST_TXrGraphicsBinding {
   GHOST_kXrGraphicsUnknown = 0,
   GHOST_kXrGraphicsOpenGL,
 #  ifdef WIN32
@@ -614,6 +616,16 @@ typedef enum {
   /* For later */
   //  GHOST_kXrGraphicsVulkan,
 } GHOST_TXrGraphicsBinding;
+
+typedef void (*GHOST_XrErrorHandlerFn)(const struct GHOST_XrError *);
+
+typedef void (*GHOST_XrSessionExitFn)(void *customdata);
+
+typedef void *(*GHOST_XrGraphicsContextBindFn)(enum GHOST_TXrGraphicsBinding graphics_lib);
+typedef void (*GHOST_XrGraphicsContextUnbindFn)(enum GHOST_TXrGraphicsBinding graphics_lib,
+                                                GHOST_ContextHandle graphics_context);
+typedef void (*GHOST_XrDrawViewFn)(const struct GHOST_XrDrawViewInfo *draw_view, void *customdata);
+
 /* An array of GHOST_TXrGraphicsBinding items defining the candidate bindings to use. The first
  * available candidate will be chosen, so order defines priority. */
 typedef const GHOST_TXrGraphicsBinding *GHOST_XrGraphicsBindingCandidates;
@@ -638,13 +650,17 @@ typedef struct {
 
 typedef struct {
   GHOST_XrPose base_pose;
+
+  GHOST_XrSessionExitFn exit_fn;
+  void *exit_customdata;
 } GHOST_XrSessionBeginInfo;
 
-typedef struct {
+typedef struct GHOST_XrDrawViewInfo {
   int ofsx, ofsy;
   int width, height;
 
-  GHOST_XrPose pose;
+  GHOST_XrPose eye_pose;
+  GHOST_XrPose local_pose;
 
   struct {
     float angle_left, angle_right;
@@ -655,19 +671,12 @@ typedef struct {
   char expects_srgb_buffer;
 } GHOST_XrDrawViewInfo;
 
-typedef struct {
+typedef struct GHOST_XrError {
   const char *user_message;
 
   void *customdata;
 } GHOST_XrError;
 
-typedef void (*GHOST_XrErrorHandlerFn)(const GHOST_XrError *);
-
-typedef void *(*GHOST_XrGraphicsContextBindFn)(GHOST_TXrGraphicsBinding graphics_lib);
-typedef void (*GHOST_XrGraphicsContextUnbindFn)(GHOST_TXrGraphicsBinding graphics_lib,
-                                                void *graphics_context);
-typedef void (*GHOST_XrDrawViewFn)(const GHOST_XrDrawViewInfo *draw_view, void *customdata);
-
 #endif
 
 #endif  // __GHOST_TYPES_H__
diff --git a/intern/ghost/intern/GHOST_C-api.cpp b/intern/ghost/intern/GHOST_C-api.cpp
index 60d20474a5a..d43a2637ad3 100644
--- a/intern/ghost/intern/GHOST_C-api.cpp
+++ b/intern/ghost/intern/GHOST_C-api.cpp
@@ -709,6 +709,20 @@ GHOST_TSuccess GHOST_ReleaseOpenGLContext(GHOST_ContextHandle contexthandle)
   return context->releaseDrawingContext();
 }
 
+unsigned int GHOST_GetContextDefaultOpenGLFramebuffer(GHOST_ContextHandle contexthandle)
+{
+  GHOST_IContext *context = (GHOST_IContext *)contexthandle;
+
+  return context->getDefaultFramebuffer();
+}
+
+int GHOST_isUpsideDownContext(GHOST_ContextHandle contexthandle)
+{
+  GHOST_IContext *context = (GHOST_IContext *)contexthandle;
+
+  return context->isUpsideDown();
+}
+
 unsigned int GHOST_GetDefaultOpenGLFramebuffer(GHOST_WindowHandle windowhandle)
 {
   GHOST_IWindow *window = (GHOST_IWindow *)windowhandle;
diff --git a/intern/ghost/intern/GHOST_Context.h b/intern/ghost/intern/GHOST_Context.h
index bbf6d6a510d..0bd6f63d07e 100644
--- a/intern/ghost/intern/GHOST_Context.h
+++ b/intern/ghost/intern/GHOST_Context.h
@@ -120,6 +120,14 @@ class GHOST_Context : public GHOST_IContext {
   }
 
   /**
+   * Returns if the window is rendered upside down compared to OpenGL.
+   */
+  inline bool isUpsideDown() const
+  {
+    return false;
+  }
+
+  /**
    * Gets the OpenGL framebuffer associated with the OpenGL context
    * \return The ID of an OpenGL framebuffer object.
    */
diff --git a/intern/ghost/intern/GHOST_IXrGraphicsBinding.h b/intern/ghost/intern/GHOST_IXrGraphicsBinding.h
index 19fe00cdad5..25281d3d0ba 100644
--- a/intern/ghost/intern/GHOST_IXrGraphicsBinding.h
+++ b/intern/ghost/intern/GHOST_IXrGraphicsBinding.h
@@ -41,6 +41,8 @@ class GHOST_IXrGraphicsBinding {
 #endif
   } oxr_binding;
 
+  virtual ~GHOST_IXrGraphicsBinding() = default;
+
   /**
    * Does __not__ require this object is initialized (can be called prior to
    * #initFromGhostContext). It's actually meant to be called first.
diff --git a/intern/ghost/intern/GHOST_XrContext.cpp b/intern/ghost/intern/GHOST_XrContext.cpp
index 410837e9805..d7b83114c85 100644
--- a/intern/ghost/intern/GHOST_XrContext.cpp
+++ b/intern/ghost/intern/GHOST_XrContext.cpp
@@ -454,10 +454,12 @@ GHOST_TXrGraphicsBinding GHOST_XrContext::determineGraphicsBindingTypeToEnable(
 
 void GHOST_XrContext::startSession(const GHOST_XrSessionBeginInfo *begin_info)
 {
+  m_custom_funcs.session_exit_fn = begin_info->exit_fn;
+  m_custom_funcs.session_exit_customdata = begin_info->exit_customdata;
+
   if (m_session == nullptr) {
     m_session = std::unique_ptr<GHOST_XrSession>(new GHOST_XrSession(this));
   }
-
   m_session->start(begin_info);
 }
 
diff --git a/intern/ghost/intern/GHOST_XrContext.h b/intern/ghost/intern/GHOST_XrContext.h
index b361fb5caf8..5140ad7ea27 100644
--- a/intern/ghost/intern/GHOST_XrContext.h
+++ b/intern/ghost/intern/GHOST_XrContext.h
@@ -33,6 +33,9 @@ struct GHOST_XrCustomFuncs {
   /** Function to release (possibly free) a graphics context. */
   GHOST_XrGraphicsContextUnbindFn gpu_ctx_unbind_fn = nullptr;
 
+  GHOST_XrSessionExitFn session_exit_fn = nullptr;
+  void *session_exit_customdata = nullptr;
+
   /** Custom per-view draw function for Blender side drawing. */
   GHOST_XrDrawViewFn draw_view_fn = nullptr;
 };
diff --git a/intern/ghost/intern/GHOST_XrGraphicsBinding.cpp b/intern/ghost/intern/GHOST_XrGraphicsBinding.cpp
index ddc757b8f8a..f094b0744a2 100644
--- a/intern/ghost/intern/GHOST_XrGraphicsBinding.cpp
+++ b/intern/ghost/intern/GHOST_XrGraphicsBinding.cpp
@@ -116,6 +116,8 @@ class GHOST_XrGraphicsBindingOpenGL : public GHOST_IXrGraphicsBinding {
     oxr_binding.glx.glxDrawable = ctx_glx->m_window;
     oxr_binding.glx.glxContext = ctx_glx->m_context;
     oxr_binding.glx.visualid = visual_info->visualid;
+
+    XFree(visual_info);
 #elif defined(WIN32)
     GHOST_ContextWGL *ctx_wgl = static_cast<GHOST_ContextWGL *>(ghost_ctx);
 
diff --git a/intern/ghost/intern/GHOST_XrSession.cpp b/intern/ghost/intern/GHOST_XrSession.cpp
index 1e2b8c0bc9d..a85bde3cab6 100644
--- a/intern/ghost/intern/GHOST_XrSession.cpp
+++ b/intern/ghost/intern/GHOST_XrSession.cpp
@@ -43,6 +43,7 @@ struct OpenXRSessionData {
   /* Only stereo rendering supported now. */
   const XrViewConfigurationType view_type = XR_VIEW_CONFIGURATION_TYPE_PRIMARY_STEREO;
   XrSpace reference_space;
+  XrSpace view_space;
   std::vector<XrView> views;
   std::vector<std::unique_ptr<GHOST_XrSwapchain>> swapchains;
 };
@@ -81,6 +82,8 @@ GHOST_XrSession::~GHOST_XrSession()
 
   m_oxr->session = XR_NULL_HANDLE;
   m_oxr->session_state = XR_SESSION_STATE_UNKNOWN;
+
+  m_context->getCustomFuncs().session_exit_fn(m_context->getCustomFuncs().session_exit_customdata);
 }
 
 /**
@@ -107,7 +110,7 @@ void GHOST_XrSession::initSystem()
  *
  * \{ */
 
-static void create_reference_space(OpenXRSessionData *oxr, const GHOST_XrPose *base_pose)
+static void create_reference_spaces(OpenXRSessionData *oxr, const GHOST_XrPose *base_pose)
 {
   XrReferenceSpaceCreateInfo create_info = {XR_TYPE_REFERENCE_SPACE_CREATE_INFO};
   create_info.poseInReferenceSpace.orientation.w = 1.0f;
@@ -138,6 +141,10 @@ static void create_reference_space(OpenXRSessionData *oxr, const GHOST_XrPose *b
 
   CHECK_XR(xrCreateReferenceSpace(oxr->session, &create_info, &oxr->reference_space),
            "Failed to create reference space.");
+
+  create_info.referenceSpaceType = XR_REFERENCE_SPACE_TYPE_VIEW;
+  CHECK_XR(xrCreateReferenceSpace(oxr->session, &create_info, &oxr->view_space),
+           "Failed to create view reference space.");
 }
 
 void GHOST_XrSession::start(const GHOST_XrSessionBeginInfo *begin_info)
@@ -184,7 +191,7 @@ void GHOST_XrSession::start(const GHOST_XrSessionBeginInfo *begin_info)
            "detailed error information to the command line.");
 
   prepareDrawing();
-  create_reference_space(m_oxr.get(), &begin_info->base_pose);
+  create_reference_spaces(m_oxr.get(), &begin_info->base_pose);
 }
 
 void GHOST_XrSession::requestEnd()
@@ -343,16 +350,22 @@ void GHOST_XrSession::draw(void *draw_customdata)
   endFrameDrawing(&layers);
 }
 
+static void copy_openxr_pose_to_ghost_pose(const XrPosef &oxr_pose, GHOST_XrPose &r_ghost_pose)
+{
+  /* Set and convert to Blender coodinate space. */
+  r_ghost_pose.position[0] = oxr_pose.position.x;
+  r_ghost_pose.position[1] = oxr_pose.position.y;
+  r_ghost_pose.position[2] = oxr_pose.position.z;
+  r_ghost_pose.orientation_quat[0] = oxr_pose.orientation.w;
+  r_ghost_pose.orientation_quat[1] = oxr_pose.orientation.x;
+  r_ghost_pose.orientation_quat[2] = oxr_pose.orientation.y;
+  r_ghost_pose.orientation_quat[3] = oxr_pose.orientation.z;
+}
+
 static void ghost_xr_draw_view_info_from_view(const XrView &view, GHOST_XrDrawViewInfo &r_info)
 {
   /* Set and convert to Blender coodinate space. */
-  r_info.pose.position[0] = view.pose.position.x;
-  r_info.pose.position[1] = view.pose.position.y;
-  r_info.pose.position[2] = view.pose.position.z;
-  r_info.pose.orientation_quat[0] = view.pose.orientation.w;
-  r_info.pose.orientation_quat[1] = view.pose.orientation.x;
-  r_info.pose.orientation_quat[2] = view.pose.orientation.y;
-  r_info.pose.orientation_quat[3] = view.pose.orientation.z;
+  copy_openxr_pose_to_ghost_pose(view.pose, r_info.eye_pose);
 
   r_info.fov.angle_left = view.fov.angleLeft;
   r_info.fov.angle_right = view.fov.angleRight;
@@ -370,6 +383,7 @@ static bool ghost_xr_draw_view_expects_srgb_buffer(const GHOST_XrContext *contex
 
 void GHOST_XrSession::drawView(GHOST_XrSwapchain &swapchain,
                                XrCompositionLayerProjectionView &r_proj_layer_view,
+                               XrSpaceLocation &view_location,
                                XrView &view,
                                void *draw_customdata)
 {
@@ -386,6 +400,7 @@ void GHOST_XrSession::drawView(GHOST_XrSwapchain &swapchain,
   draw_view_info.ofsy = r_proj_layer_view.subImage.imageRect.offset.y;
   draw_view_info.width = r_proj_layer_view.subImage.imageRect.extent.width;
   draw_view_info.height = r_proj_layer_view.subImage.imageRect.extent.height;
+  copy_openxr_pose_to_ghost_pose(view_location.pose, draw_view_info.local_pose);
   ghost_xr_draw_view_info_from_view(view, draw_view_info);
 
   /* Draw! */
@@ -401,6 +416,7 @@ XrCompositionLayerProjection GHOST_XrSession::drawLayer(
   XrViewLocateInfo viewloc_info = {XR_TYPE_VIEW_LOCATE_INFO};
   XrViewState view_state = {XR_TYPE_VIEW_STATE};
   XrCompositionLayerProjection layer = {XR_TYPE_COMPOSITION_LAYER_PROJECTION};
+  XrSpaceLocation view_location{XR_TYPE_SPACE_LOCATION};
   uint32_t view_count;
 
   viewloc_info.viewConfigurationType = m_oxr->view_type;
@@ -416,11 +432,17 @@ XrCompositionLayerProjection GHOST_XrSession::drawLayer(
            "Failed to query frame view and projection state.");
   assert(m_oxr->swapchains.size() == view_count);
 
+  CHECK_XR(
+      xrLocateSpace(
+          m_oxr->view_space, m_oxr->reference_space, viewloc_info.displayTime, &view_location),
+      "Failed to query frame view space");
+
   r_proj_layer_views.resize(view_count);
 
   for (uint32_t view_idx = 0; view_idx < view_count; view_idx++) {
     drawView(*m_oxr->swapchains[view_idx],
              r_proj_layer_views[view_idx],
+             view_location,
              m_oxr->views[view_idx],
              draw_customdata);
   }
@@ -479,7 +501,8 @@ void GHOST_XrSession::unbindGraphicsContext()
 {
   const GHOST_XrCustomFuncs &custom_funcs = m_context->getCustomFuncs();
   if (custom_funcs.gpu_ctx_unbind_fn) {
-    custom_funcs.gpu_ctx_unbind_fn(m_context->getGraphicsBindingType(), m_gpu_ctx);
+    custom_funcs.gpu_ctx_unbind_fn(m_context->getGraphicsBindingType(),
+                                   (GHOST_ContextHandle)m_gpu_ctx);
   }
   m_gpu_ctx = nullptr;
 }
diff --git a/intern/ghost/intern/GHOST_XrSession.h b/intern/ghost/intern/GHOST_XrSession.h
index 3340385c1b6..ef2da61df3d 100644
--- a/intern/ghost/intern/GHOST_XrSession.h
+++ b/intern/ghost/intern/GHOST_XrSession.h
@@ -25,9 +25,9 @@
 #include <memory>
 
 class GHOST_XrContext;
+class GHOST_XrSwapchain;
 struct OpenXRSessionData;
 struct GHOST_XrDrawInfo;
-struct GHOST_XrSwapchain;
 
 class GHOST_XrSession {
  public:
@@ -74,6 +74,7 @@ class GHOST_XrSession {
       std::vector<XrCompositionLayerProjectionView> &r_proj_layer_views, void *draw_customdata);
   void drawView(GHOST_XrSwapchain &swapchain,
                 XrCompositionLayerProjectionView &r_proj_layer_view,
+                XrSpaceLocation &view_location,
                 XrView &view,
                 void *draw_customdata);
   void beginFrameDrawing();