Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Performance
In order for the underlying mixed reality platform and device to produce the best results, it's important to achieve performing frame rates. The target framerate (ex: 60 FPS or 90 FPS) varies across platforms and devices. However, mixed reality applications meeting framerate have stable holograms and efficient head tracking, hand tracking and more.
Environment tracking
Stable holographic rendering heavily relies on head-pose tracking by the platform & device. Unity renders the scene every frame from the camera pose estimated and supplied by the underlying platform. If this tracking doesn't correctly follow actual head movement, then holograms appear visually inaccurate. This tracking is especially evident and important for AR devices like HoloLens where users can relate virtual holograms to the real world. Performance is significant for reliable head tracking, but there can be other important features, as well. The types of environment elements impacting user experience depend on the targeted platform specifics.
Windows Mixed Reality
The Windows Mixed Reality platform provides some reference material for stabilizing holograms on the platform. There are a handful of key tools though that developers can utilize to improve the hologram visual experience for users.
Depth buffer sharing
Unity developers have the option of sharing the application's depth buffer with the platform. This provides information, where holograms exist for a current frame, that the platform can utilize to stabilize holograms via a hardware-assisted process known as Late-Stage Reprojection.
Late-stage reprojection
At the end of rendering a frame, the Windows Mixed Reality platform takes the color & depth render targets produced by the application and transforms the final screen output to account for any slight head movement since the last head pose prediction. An application's game loop takes time to execute. For example, at 60 FPS, the application is taking ~16.667ms to render a frame. Even though this period may seem like a minuscule amount of time, the user's position and orientation of their head changes resulting in new projection matrices for the camera in rendering. Late-stage reprojection transforms the pixels in the final image to account for this new perspective.
Per-pixel vs stabilization plane LSR
Depending on the device endpoint and OS version running on a Windows Mixed Reality device, the Late-Stage Reprojection algorithm can be performed per-pixel or via a stabilization plane.
Per-pixel depth-based
Per-pixel depth-based reprojection involves utilizing the depth buffer to modify the image output per pixel and thus stabilize holograms at various distances. For example, a sphere 1m away may be in front of a pillar that is 10m away. The pixels representing the sphere have a different transform than the far away pixels representing the pillar if the user has tilted their head slightly. Per-pixel reprojection takes into account this distance difference at every pixel for more accurate reprojection.
Stabilization plane
If it's not possible to create an accurate depth buffer to share with the platform, another form of LSR utilizes a stabilization plane. All holograms in a scene receive some stabilization, but holograms lying in the desired plane receives the maximum hardware stabilization. The point and normal for the plane can be supplied to the platform via the HolographicSettings.SetFocusPointForFrame API provided by Unity.
Depth buffer format
If targeting HoloLens for development, it's highly recommended to utilize the 16-bit depth buffer format compared to 24-bit. This format can save tremendously on performance although depth values have less precision. To compensate for the lower precision and avoid z-fighting, it's recommended to reduce the far clip plane from the 1000m default value set by Unity.
Note
If using 16-bit depth format, stencil buffer required effects won't work because Unity doesn't create a stencil buffer in this setting. Selecting 24-bit depth format conversely will generally create an 8-bit stencil buffer, if applicable on the endpoint graphics platform.
Depth buffer sharing in Unity
In order to utilize depth-based LSR, there are two important steps that developers need to take.
- Under Edit > Project Settings > Player > XR Settings > Virtual Reality SDKs > Enable Depth Buffer Sharing
- If targeting HoloLens, it's recommended to select 16-bit depth format as well.
- When rendering color on screen, render depth as well
Opaque GameObjects in Unity will generally write to depth automatically. However, transparent & text objects will generally not write to depth by default. If utilizing the MRTK Standard Shader or Text Mesh Pro, this can be easily remedied.
Note
To quickly determine which objects in a scene don't write to the depth buffer visually, one can use the Render Depth Buffer utility under the Editor Settings in the MRTK Configuration profile.
Transparent MRTK Standard shader
For transparent materials using the MRTK Standard shader, select the material to view it in the Inspector window. Then click the Fix Now button to convert the material to write to depth (i.e. Z-Write On).
Before
After
Text Mesh Pro
For Text Mesh Pro objects, select the TMP GameObject to view it in the inspector. Under the material component, switch the shader for the assigned material to use the MRTK TextMeshPro shader.
Custom shader
If writing a custom shader, add the ZWrite flag to the top of the Pass block definition to configure the shader to write to the depth buffer.
Shader "Custom/MyShader"
{
SubShader
{
Pass
{
...
ZWrite On
...
}
}
}
Opaque backings
If the above methods don't work for a given scenario (i.e. using Unity UI), it's possible to have another object write to the depth buffer. A common example is using Unity UI Text on a floating panel in a scene. By making the panel opaque or at least writing to depth, then both the text & the panel is stabilized by the platform since their z-values are so close to each other.
WorldAnchors (HoloLens)
Along with ensuring the correct configurations are met to ensure visual stability, it's important to ensure holograms remain stable at their correct physical locations. To inform the platform on important locations in a physical space, developers can use WorldAnchors on GameObjects that need to stay in one place. A WorldAnchor is a component added to a GameObject that takes absolute control over that object's transform.
Devices such as HoloLens are constantly scanning and learning about the environment. Thus, as the HoloLens tracks movement & position in space, its estimates are updated and the Unity coordinate system adjusted. For example, if a GameObject is placed 1m from the camera at start, as the HoloLens tracks the environment, it may realize the physical point where the GameObject is located is actually 1.1m away. This would result in the hologram drifting. Applying a WorldAnchor to a GameObject will enable the anchor to control the object's transform so that the object will remain at the correct physical location (for example, update to 1.1m away instead of 1m at runtime). To persist WorldAnchors across app sessions, developers can employ the WorldAnchorStore to save and load WorldAnchors.
Note
Once a WorldAnchor component has been added to a GameObject, it's not possible to modify that GameObject's transform (i.e. transform.position = x). A developer must remove the WorldAnchor to edit the transform.
WorldAnchor m_anchor;
public void AddAnchor()
{
this.m_anchor = this.gameObject.AddComponent<WorldAnchor>();
}
public void RemoveAnchor()
{
DestroyImmediate(m_anchor);
}
If you want an alternative to manually working with Anchors, check out the Microsoft World Locking Tools.