原生引擎中的擴充眼球追蹤

擴充眼球追蹤是 HoloLens 2 的新功能。它是標準眼球追蹤的超集，只提供結合的眼部注視數據。擴充眼球追蹤也提供個別的眼部注視數據，並可讓應用程式為注視數據設定不同的幀速率，例如 30、60 和 90fps。目前 HoloLens 2 不支援其他功能，例如眼睛幊眼和眼部閲箋。

擴充眼球追蹤 SDK 可讓應用程式存取擴充眼球追蹤的數據和功能。它可以與 WinRT API 或 OpenXR API 搭配使用。

本文涵蓋在原生引擎中使用擴充眼球追蹤 SDK 的方式 (C# 或 C++/WinRT) ，以及 WinRT API。

專案設定

Holographic DirectX 11 App (Universal Windows) 使用 Visual Studio 2019 或更新版本建立或 Holographic DirectX 11 App (Universal Windows) (C++/WinRT) 專案，或開啟現有的全像攝影 Visual Studio 專案。
將擴充眼球追蹤 SDK 匯入專案中。
1. 在 Visual Studio 方案總管中，以滑鼠右鍵按兩下您的專案 -> 管理 NuGet 套件...
2. 請確定右上角的套件來源指向 nuget.org： https://api.nuget.org/v3/index.json
3. 點選「瀏覽器」索引標籤，然後搜尋 Microsoft.MixedReality.EyeTracking。
4. 按兩下 [安裝] 按鈕以安裝最新版的 SDK。
設定注視輸入功能
1. 按兩下方案總管中的 Package.appxmanifest 檔案。
2. 按兩下 [ 功能] 索引 標籤，然後核取 [注視輸入]。

包含頭檔和使用命名空間。

針對 C# 專案：

using Microsoft.MixedReality.EyeTracking;

針對 C++/WinRT 專案：

#include <winrt/Microsoft.MixedReality.EyeTracking.h>
using namespace winrt::Microsoft::MixedReality::EyeTracking;

取用擴充眼球追蹤 SDK API 並實作邏輯。
建置並部署至 HoloLens。

取得注視數據的步驟概觀

透過擴充眼球追蹤 SDK API 取得眼部注視資料需要下列步驟：

從使用者取得眼球追蹤功能的存取權。
監看眼部注視追蹤器連線和中斷連線。
開啟眼部注視追蹤器，然後查詢其功能。
從眼部注視追蹤器重複讀取注視數據。
將注視數據傳輸到其他 SpatialCoordinateSystems。

取得眼球追蹤功能的存取權

若要使用任何眼球相關信息，應用程式必須先要求使用者同意。

var status = await Windows.Perception.People.EyesPose.RequestAccessAsync();
bool useGaze = (status == Windows.UI.Input.GazeInputAccessStatus.Allowed);

auto accessStatus = co_await winrt::Windows::Perception::People::EyesPose::RequestAccessAsync();
bool useGaze = (accessStatus.get() == winrt::Windows::UI::Input::GazeInputAccessStatus::Allowed);

偵測眼部注視追蹤器

眼部注視追蹤器偵測是透過使用類別來 EyeGazeTrackerWatcher 進行。 EyeGazeTrackerAdded 偵測到眼部注視追蹤器或中斷連接時，會分別引發和 EyeGazeTrackerRemoved 事件。

監看員必須明確地使用方法來啟動 StartAsync() ，此方法會在已連線的追蹤器透過事件收到信號 EyeGazeTrackerAdded 時以異步方式完成。

偵測到眼部注視追蹤器時， EyeGazeTracker 會在事件參數中 EyeGazeTrackerAdded 將實例傳遞至應用程式;此外，當追蹤器中斷連線時，對應 EyeGazeTracker 的實例會傳遞至 EyeGazeTrackerRemoved 事件。

EyeGazeTrackerWatcher watcher = new EyeGazeTrackerWatcher();
watcher.EyeGazeTrackerAdded += _watcher_EyeGazeTrackerAdded;
watcher.EyeGazeTrackerRemoved += _watcher_EyeGazeTrackerRemoved;
await watcher.StartAsync();
...

private async void _watcher_EyeGazeTrackerAdded(object sender, EyeGazeTracker e)
{
    // Implementation is in next section
}

private void _watcher_EyeGazeTrackerRemoved(object sender, EyeGazeTracker e)
{
    ...
}

EyeGazeTrackerWatcher watcher;
watcher.EyeGazeTrackerAdded(std::bind(&SampleEyeTrackingNugetClientAppMain::OnEyeGazeTrackerAdded, this, _1, _2));
watcher.EyeGazeTrackerRemoved(std::bind(&SampleEyeTrackingNugetClientAppMain::OnEyeGazeTrackerRemoved, this, _1, _2));
co_await watcher.StartAsync();
...

winrt::Windows::Foundation::IAsyncAction SampleAppMain::OnEyeGazeTrackerAdded(const EyeGazeTrackerWatcher& sender, const EyeGazeTracker& tracker)
{
    // Implementation is in next section
}
void SampleAppMain::OnEyeGazeTrackerRemoved(const EyeGazeTrackerWatcher& sender, const EyeGazeTracker& tracker)
{
    ...
}

開啟眼睛注視追蹤器

接收實 EyeGazeTracker 例時，應用程式必須先呼叫方法來開啟 OpenAsync() 它。然後，它可以視需要查詢追蹤器功能。方法 OpenAsync() 會採用布爾值參數;這表示應用程式是否需要存取不屬於標準眼球追蹤的功能，例如個別眼部注視向量或變更追蹤器的幀速率。

合併注視是所有眼部注視追蹤器所支援的必要功能。其他功能，例如存取個別注視，是選擇性的，而且視追蹤器及其驅動程式而定，可能受到支援或不支援。針對這些選擇性功能， EyeGazeTracker 類別會公開屬性，指出是否支援此功能，例如 AreLeftAndRightGazesSupported 屬性，指出裝置是否支持個別眼部注視資訊。

眼部注視追蹤器所公開的所有空間信息都會與追蹤器本身相關發佈，而追蹤器本身是由 動態節點標識碼所識別。使用 nodeId 取得具有 SpatialCoordinateSystem WinRT API 的，可以將注視數據的座標轉換成另一個座標系統。

private async void _watcher_EyeGazeTrackerAdded(object sender, EyeGazeTracker e)
{
    try
    {
        // Try to open the tracker with access to restricted features
        await e.OpenAsync(true);

        // If it has succeeded, store it for future use
        _tracker = e;

        // Check support for individual eye gaze
        bool supportsIndividualEyeGaze = _tracker.AreLeftAndRightGazesSupported;

        // Get a spatial locator for the tracker, this will be used to transfer the gaze data to other coordinate systems later
        var trackerNodeId = e.TrackerSpaceLocatorNodeId;
        _trackerLocator = Windows.Perception.Spatial.Preview.SpatialGraphInteropPreview.CreateLocatorForNode(trackerNodeId);
    }
    catch (Exception ex)
    {
        // Unable to open the tracker
    }
}

winrt::Windows::Foundation::IAsyncAction SampleEyeTrackingNugetClientAppMain::OnEyeGazeTrackerAdded(const EyeGazeTrackerWatcher&, const EyeGazeTracker& tracker)
{
   auto newTracker = tracker;

   try
   {
        // Try to open the tracker with access to restricted features
        co_await newTracker.OpenAsync(true);

        // If it has succeeded, store it for future use
        m_gazeTracker = newTracker;

        // Check support for individual eye gaze
        const bool supportsIndividualEyeGaze = m_gazeTracker.AreLeftAndRightGazesSupported();

        // Get a spatial locator for the tracker. This will be used to transfer the gaze data to other coordinate systems later
        const auto trackerNodeId = m_gazeTracker.TrackerSpaceLocatorNodeId();
        m_trackerLocator = winrt::Windows::Perception::Spatial::Preview::SpatialGraphInteropPreview::CreateLocatorForNode(trackerNodeId);
   }
   catch (const winrt::hresult_error& e)
   {
       // Unable to open the tracker
   }
}

設定眼部注視追蹤器幀速率

屬性會 EyeGazeTracker.SupportedTargetFrameRates 傳回追蹤器支持的目標幀速率清單。 HoloLens 2 支援 30、60 和 90fps。

EyeGazeTracker.SetTargetFrameRate()使用方法來設定目標幀速率。

// This returns a list of supported frame rate: 30, 60, 90 fps in order
var supportedFrameRates = _tracker.SupportedTargetFrameRates;

// Sets the tracker at the highest supported frame rate (90 fps)
var newFrameRate = supportedFrameRates[supportedFrameRates.Count - 1];
_tracker.SetTargetFrameRate(newFrameRate);
uint newFramesPerSecond = newFrameRate.FramesPerSecond;

// This returns a list of supported frame rate: 30, 60, 90 fps in order
const auto supportedFrameRates = m_gazeTracker.SupportedTargetFrameRates();

// Sets the tracker at the highest supported frame rate (90 fps)
const auto newFrameRate = supportedFrameRates.GetAt(supportedFrameRates.Size() - 1);
m_gazeTracker.SetTargetFrameRate(newFrameRate);
const uint32_t newFramesPerSecond = newFrameRate.FramesPerSecond();

從眼部注視追蹤器讀取注視數據

眼部注視追蹤器會定期在迴圈緩衝區中發佈其狀態。這可讓應用程式在屬於小型時間範圍的時間讀取追蹤器的狀態。例如，它允許擷取追蹤器的最新狀態，或是在某些事件時的狀態，例如來自使用者的手勢。

擷取追蹤器狀態作為 EyeGazeTrackerReading 實例的方法：

與方法會TryGetReadingAtTimestamp()傳回EyeGazeTrackerReading最接近應用程式所傳遞時間TryGetReadingAtSystemRelativeTime()的。追蹤器會控制發佈排程，因此傳回的讀數可能會比要求時間稍早或更新。和 EyeGazeTrackerReading.TimestampEyeGazeTrackerReading.SystemRelativeTime 屬性可讓應用程式知道已發佈狀態的確切時間。
和方法會TryGetReadingAfterTimestamp()傳回第EyeGazeTrackerReading一個，其時間戳絕對優於做為參數傳遞TryGetReadingAfterSystemRelativeTime()的時間。這可讓應用程式循序讀取追蹤器所發佈的所有狀態。請注意，所有這些方法都會查詢現有的緩衝區，並立即傳回。如果沒有可用的狀態，它們會傳回 null (換句話說，它們不會讓應用程式等待狀態發佈) 。

除了時間戳之外，實 EyeGazeTrackerReading 例還有屬性 IsCalibrationValid ，指出眼球追蹤器校正是否有效。

最後，您可以透過一組方法擷取注視資料，例如 TryGetCombinedEyeGazeInTrackerSpace() 或 TryGetLeftEyeGazeInTrackerSpace()。所有這些方法都會傳回布爾值，表示成功。無法取得某些數據可能表示數據不受支援， (EyeGazeTracker 具有偵測此案例的屬性) ，或是追蹤器無法取得數據 (例如，無效的校正或眼部隱藏) 。

例如，如果應用程式想要顯示對應至合併注視的數據指標，它可以使用所準備框架預測的時間戳來查詢追蹤器，如下所示。

var holographicFrame = holographicSpace.CreateNextFrame();
var prediction = holographicFrame.CurrentPrediction;
var predictionTimestamp = prediction.Timestamp;
var reading = _tracker.TryGetReadingAtTimestamp(predictionTimestamp.TargetTime.DateTime);
if (reading != null)
{
    // Vector3 needs the System.Numerics namespace
    if (reading.TryGetCombinedEyeGazeInTrackerSpace(out Vector3 gazeOrigin, out Vector3 gazeDirection))
    {
        // Use gazeOrigin and gazeDirection to display the cursor
    }
}

auto holographicFrame = m_holographicSpace.CreateNextFrame();
auto prediction = holographicFrame.CurrentPrediction();
auto predictionTimestamp = prediction.Timestamp();
const auto reading = m_gazeTracker.TryGetReadingAtTimestamp(predictionTimestamp.TargetTime());
if (reading)
{
    float3 gazeOrigin;
    float3 gazeDirection;
    if (reading.TryGetCombinedEyeGazeInTrackerSpace(gazeOrigin, gazeDirection))
    {
        // Use gazeOrigin and gazeDirection to display the cursor
    }
}

將注視數據轉換成其他 SpatialCoordinateSystem

傳回位置等空間資料的 WinRT API 一律需要 PerceptionTimestamp 和 SpatialCoordinateSystem。例如，若要使用 WinRT API 擷取 HoloLens 2 的合併注視，API SpatialPointerPose.TryGetAtTimestamp () 需要兩個PerceptionTimestamp參數：a SpatialCoordinateSystem 和。當合併注視接著透過 SpatialPointerPose.Eyes.Gaze存取時，其原點和方向會以 SpatialCoordinateSystem 傳入的表示。

擴充的 tye 追蹤 SDK API 不需要採用， SpatialCoordinateSystem 而且注視數據一律會在追蹤器的座標系統中表示。但是，您可以將這些註視數據轉換成另一個座標系統，其中追蹤器的姿勢與另一個座標系統相關。

如上名為「開啟眼睛注視追蹤器」的區段所述，若要取得SpatialLocator眼部注視追蹤器的，請使用 EyeGazeTracker.TrackerSpaceLocatorNodeId 屬性呼叫 Windows.Perception.Spatial.Preview.SpatialGraphInteropPreview.CreateLocatorForNode() 。
透過擷取的 EyeGazeTrackerReading 注視原點和方向與眼部注視追蹤器有關。
SpatialLocator.TryLocateAtTimestamp() 會傳回指定之眼部注視追蹤器 PerceptionTimeStamp 的完整 6DoF 位置，且與指定 SpatialCoordinateSystem的相關，可用來建構 Matrix4x4 轉換矩陣。
使用建構的 Matrix4x4 轉換矩陣，將注視原點和方向傳送至其他 SpatialCoordinateSystem。

下列程式代碼範例示範如何計算位於合併注視方向的 Cube 位置，在注視原點前面兩公尺;

var predictionTimestamp = prediction.Timestamp;
var stationaryCS = stationaryReferenceFrame.CoordinateSystem;
var trackerLocation = _trackerLocator.TryLocateAtTimestamp(predictionTimestamp, stationaryCS);
if (trackerLocation != null)
{
    var trackerToStationaryMatrix = Matrix4x4.CreateFromQuaternion(trackerLocation.Orientation) * Matrix4x4.CreateTranslation(trackerLocation.Position);
    var reading = _tracker.TryGetReadingAtTimestamp(predictionTimestamp.TargetTime.DateTime);
    if (reading != null)
    {
        if (reading.TryGetCombinedEyeGazeInTrackerSpace(out Vector3 gazeOriginInTrackerSpace, out Vector3 gazeDirectionInTrackerSpace))
        {
            var cubePositionInTrackerSpace = gazeOriginInTrackerSpace + 2.0f * gazeDirectionInTrackerSpace;
            var cubePositionInStationaryCS = Vector3.Transform(cubePositionInTrackerSpace, trackerToStationaryMatrix);
        }
    }
}

auto predictionTimestamp = prediction.Timestamp();
auto stationaryCS = m_stationaryReferenceFrame.CoordinateSystem();
auto trackerLocation = m_trackerLocator.TryLocateAtTimestamp(predictionTimestamp, stationaryCS);
if (trackerLocation) 
{
    auto trackerOrientation = trackerLocation.Orientation();
    auto trackerPosition = trackerLocation.Position();
    auto trackerToStationaryMatrix = DirectX::XMMatrixRotationQuaternion(DirectX::XMLoadFloat4(reinterpret_cast<const DirectX::XMFLOAT4*>(&trackerOrientation))) * DirectX::XMMatrixTranslationFromVector(DirectX::XMLoadFloat3(&trackerPosition));

    const auto reading = m_gazeTracker.TryGetReadingAtTimestamp(predictionTimestamp.TargetTime());
    if (reading)
    {
        float3 gazeOriginInTrackerSpace;
        float3 gazeDirectionInTrackerSpace;
        if (reading.TryGetCombinedEyeGazeInTrackerSpace(gazeOriginInTrackerSpace, gazeDirectionInTrackerSpace))
        {
            auto cubePositionInTrackerSpace = gazeOriginInTrackerSpace + 2.0f * gazeDirectionInTrackerSpace;
            float3 cubePositionInStationaryCS;
            DirectX::XMStoreFloat3(&cubePositionInStationaryCS, DirectX::XMVector3TransformCoord(DirectX::XMLoadFloat3(&cubePositionInTrackerSpace), trackerToStationaryMatrix));
        }
    }
}

擴充眼球追蹤 SDK 的 API 參考

namespace Microsoft.MixedReality.EyeTracking
{
    /// <summary>
    /// Allow discovery of Eye Gaze Trackers connected to the system
    /// This is the only class from Extended Eye Tracking SDK that the application will instantiate, 
    /// other classes' instances will be returned by method calls or properties.
    /// </summary>
    public class EyeGazeTrackerWatcher
    {
        /// <summary>
        /// Constructs an instance of the watcher
        /// </summary>
        public EyeGazeTrackerWatcher();

        /// <summary>
        /// Starts trackers enumeration.
        /// </summary>
        /// <returns>Task representing async action; completes when the initial enumeration is completed</returns>
        public System.Threading.Tasks.Task StartAsync();

        /// <summary>
        /// Stop listening to trackers additions and removal
        /// </summary>
        public void Stop();

        /// <summary>
        /// Raised when an Eye Gaze tracker is connected
        /// </summary>
        public event System.EventHandler<EyeGazeTracker> EyeGazeTrackerAdded;

        /// <summary>
        /// Raised when an Eye Gaze tracker is disconnected
        /// </summary>
        public event System.EventHandler<EyeGazeTracker> EyeGazeTrackerRemoved;        
    }

    /// <summary>
    /// Represents an Eye Tracker device
    /// </summary>
    public class EyeGazeTracker
    {
        /// <summary>
        /// True if Restricted mode is supported, which means the driver supports to provide individual 
        /// eye gaze vector and framerate 
        /// </summary>
        public bool IsRestrictedModeSupported;

        /// <summary>
        /// True if Vergence Distance is supported by tracker
        /// </summary>
        public bool IsVergenceDistanceSupported;

        /// <summary>
        /// True if Eye Openness is supported by the driver
        /// </summary>
        public bool IsEyeOpennessSupported;

        /// <summary>
        /// True if individual gazes are supported
        /// </summary>
        public bool AreLeftAndRightGazesSupported;

        /// <summary>
        /// Get the supported target frame rates of the tracker
        /// </summary>
        public System.Collections.Generic.IReadOnlyList<EyeGazeTrackerFrameRate> SupportedTargetFrameRates;

        /// <summary>
        /// NodeId of the tracker, used to retrieve a SpatialLocator or SpatialGraphNode to locate the tracker in the scene
        /// for Perception API, use SpatialGraphInteropPreview.CreateLocatorForNode
        /// for Mixed Reality OpenXR API, use SpatialGraphNode.FromDynamicNodeId
        /// </summary>
        public Guid TrackerSpaceLocatorNodeId;

        /// <summary>
        /// Opens the tracker
        /// </summary>
        /// <param name="restrictedMode">True if restricted mode active</param>
        /// <returns>Task representing async action; completes when the initial enumeration is completed</returns>
        public System.Threading.Tasks.Task OpenAsync(bool restrictedMode);

        /// <summary>
        /// Closes the tracker
        /// </summary>
        public void Close();

        /// <summary>
        /// Changes the target frame rate of the tracker
        /// </summary>
        /// <param name="newFrameRate">Target frame rate</param>
        public void SetTargetFrameRate(EyeGazeTrackerFrameRate newFrameRate);

        /// <summary>
        /// Try to get tracker state at a given timestamp
        /// </summary>
        /// <param name="timestamp">timestamp</param>
        /// <returns>State if available, null otherwise</returns>
        public EyeGazeTrackerReading TryGetReadingAtTimestamp(DateTime timestamp);

        /// <summary>
        /// Try to get tracker state at a system relative time
        /// </summary>
        /// <param name="time">time</param>
        /// <returns>State if available, null otherwise</returns>
        public EyeGazeTrackerReading TryGetReadingAtSystemRelativeTime(TimeSpan time);

        /// <summary>
        /// Try to get first first tracker state after a given timestamp
        /// </summary>
        /// <param name="timestamp">timestamp</param>
        /// <returns>State if available, null otherwise</returns>
        public EyeGazeTrackerReading TryGetReadingAfterTimestamp(DateTime timestamp);

        /// <summary>
        /// Try to get the first tracker state after a system relative time
        /// </summary>
        /// <param name="time">time</param>
        /// <returns>State if available, null otherwise</returns>
        public EyeGazeTrackerReading TryGetReadingAfterSystemRelativeTime(TimeSpan time);
    }

    /// <summary>
    /// Represents a Frame Rate supported by an Eye Tracker
    /// </summary>
    public class EyeGazeTrackerFrameRate
    {
        /// <summary>
        /// Frames per second of the frame rate
        /// </summary>
        public UInt32 FramesPerSecond;
    }

    /// <summary>
    /// Snapshot of Gaze Tracker state
    /// </summary>
    public class EyeGazeTrackerReading
    {
        /// <summary>
        /// Timestamp of state
        /// </summary>
        public DateTime Timestamp;

        /// <summary>
        /// Timestamp of state as system relative time
        /// Its SystemRelativeTime.Ticks could provide the QPC time to locate tracker pose 
        /// </summary>
        public TimeSpan SystemRelativeTime;

        /// <summary>
        /// Indicates user calibration is valid
        /// </summary>
        public bool IsCalibrationValid;

        /// <summary>
        /// Tries to get a vector representing the combined gaze related to the tracker's node
        /// </summary>
        /// <param name="origin">Origin of the gaze vector</param>
        /// <param name="direction">Direction of the gaze vector</param>
        /// <returns></returns>
        public bool TryGetCombinedEyeGazeInTrackerSpace(out System.Numerics.Vector3 origin, out System.Numerics.Vector3 direction);

        /// <summary>
        /// Tries to get a vector representing the left eye gaze related to the tracker's node
        /// </summary>
        /// <param name="origin">Origin of the gaze vector</param>
        /// <param name="direction">Direction of the gaze vector</param>
        /// <returns></returns>
        public bool TryGetLeftEyeGazeInTrackerSpace(out System.Numerics.Vector3 origin, out System.Numerics.Vector3 direction);

        /// <summary>
        /// Tries to get a vector representing the right eye gaze related to the tracker's node position
        /// </summary>
        /// <param name="origin">Origin of the gaze vector</param>
        /// <param name="direction">Direction of the gaze vector</param>
        /// <returns></returns>
        public bool TryGetRightEyeGazeInTrackerSpace(out System.Numerics.Vector3 origin, out System.Numerics.Vector3 direction);

        /// <summary>
        /// Tries to read vergence distance
        /// </summary>
        /// <param name="value">Vergence distance if available</param>
        /// <returns>bool if value is valid</returns>
        public bool TryGetVergenceDistance(out float value);

        /// <summary>
        /// Tries to get left Eye openness information
        /// </summary>
        /// <param name="value">Eye Openness if valid</param>
        /// <returns>bool if value is valid</returns>
        public bool TryGetLeftEyeOpenness(out float value);

        /// <summary>
        /// Tries to get right Eye openness information
        /// </summary>
        /// <param name="value">Eye Openness if valid</param>
        /// <returns>bool if value is valid</returns>
        public bool TryGetRightEyeOpenness(out float value);
    }
}

另請參閱

在 Unity 中使用擴充眼球追蹤

意見反應

此頁面對您有幫助嗎？

Last updated on 2025-02-18