Speech-enable custom controls (.NET)

With the Dragon Medical SpeechKit custom control interface, you can speech-enable controls which aren't natively supported. Your custom control implementation is responsible for the following:

• Emulating a natively-supported text control to enable speech recognition.
• Manipulating the text inside the unsupported control as instructed by Dragon Medical SpeechKit.

Overview

A custom control is an object in your app which implements the ITextControl interface.

Custom controls are speech-enabled by the custom VuiController (an instance of the Nuance.SpeechAnywhere.Custom.VuiController class).

The ITextControl interface specifies how a supported text control appears and behaves; you must implement the necessary functionality in your custom control class. Dragon Medical SpeechKit interacts with custom controls via this interface: it retrieves and alters their text, selection and user input focus state. Your custom control implementation then performs the Dragon Medical SpeechKit requests by manipulating the unsupported text control.

For example, when recognition results arrive for the control, the ReplaceText method is called on the ITextControl interface, instructing your custom control implementation to insert text into the text control.

Model behind the ITextControl interface

The ITextControl interface is the specification of the behavior of a text control as seen by Dragon Medical SpeechKit.

Dragon Medical SpeechKit assumes the following model for any text control.

Text contents

The control contains unstructured plain text, indexed by 0-based character positions. If your control has a different concept of contents (formatted or structured text), you must represent it as described (perform the conversion between plain text and the actual contents of the control in both directions).

Text selection

Exactly one section of text can be selected at any one time. The selection is defined by its start character position and length in characters. The selection is the section of text that will be affected by the next user action and must be restricted to character positions within the boundaries of the text contents.

For dictation, when an utterance is dictated, the results will overwrite the selected text. A zero character selection means that nothing will be overwritten; text will be inserted at the cursor position (the start of the zero length selection). A zero length selection might have a start position equal to the length of the entire text; in this case new text is appended at the end of the current text contents.

Control focus

A maximum of one control can have the focus at any one time. If a control has the focus, the next user action will affect its contents.

For dictation, when an utterance is dictated, the results will be inserted in the focused control. If no control has the focus, dictation results will be inserted to the most recently focused control.

Format strings

Line and paragraph breaks are represented as special character sequences (by default, \r\n for new line and \r\n\r\n for new paragraph). When a user dictates a line or paragraph break, SpeechKit inserts these special characters in the text of your control. Voice commands whose behavior is affected by line and paragraph breaks look for these special characters in the control's text contents.

If line and paragraph breaks are represented by your text control in a different way, you need to do the conversion to and from a format string. For example, if your underlying text control uses HTML, convert the newline format string to/from the <br/> HTML markup, and the new paragraph format string to/from the <p>...</p> HTML markup.

Implementing the ITextControl interface

Dragon Medical SpeechKit expects your custom control to be able to perform these operations. Your class must implement all methods exactly as specified. Poor implementation could lead to erroneous behavior or destroy text in the text control, even if it was not inserted by Dragon Medical SpeechKit.

Manipulating text contents

• Report the total length of the text in the control:

  int TextLength { get; }

• Extract a range of the text contents:

  string GetText(int start, int length)

If length is set to -1, all text from start to the end of the control should be returned.

• Replace a range of the text contents with a string of characters:

  void ReplaceText(int start, int length, string newText)

This operation must leave the selection intact. The same characters in the text which are selected before the operation must remain selected after it. This doesn't mean that the same positions are selected; when insertion affects text contents in lower character positions than the current selection (i.e. to the left of it), the selection start (interpreted in absolute character positions) must be adjusted to the change in the number of characters to the left. If the replaced range intersects the current selection, the selection start after the operation is undefined.

Dragon Medical SpeechKit maintains a history of the text and selection state of the control. Any changes between the current state reported by your control and previous states are interpreted as manual editing performed by the user.

If your implementation needs to manipulate the text (for example, to convert between plain text and Rich Text or HTML), make sure that bi-directional text conversions are symmetric. For example, text inserted via the ReplaceText method must appear unchanged when subsequently queried via the GetText method, unless the user has edited it.

If your implementation fails to fulfill these requirements and the text and/or selection contents returned by your control don't match the state expected by Dragon Medical SpeechKit, it will be assumed that the user has performed manual editing in the control. As a result, Dragon Medical SpeechKit could assume that the user has changed the selection and therefore the cursor isn't moved to the end of newly inserted dictation results, pending cursor navigation/selection commands might be ignored or text might be inserted in an unexpected location.

In addition, text changes by the user might be interpreted as corrections and might affect the learning algorithms that adjust the user's speech profile to the user's way of speaking. Unintended text changes therefore might cause the system to adapt incorrectly, resulting in higher recognition error rates.

Manipulating text selection

• Return the current selection start and length in the control:

  void GetSelection(ref int start, ref int length)

• Set the current selection start and length in the control:

  void SetSelection(int start, int length)

Manipulating focus

• Report if the control has the focus or not:

  bool Focused { get; set; }

A maximum of one control should report being focused. The state reported by your custom control implementation can be tied to the Windows GUI focus state of the underlying text control, as long as it corresponds to the control in which the user's typed input will be entered.

If your UI is designed for the user to edit text in a control even if it doesn't have the Windows GUI focus state (for example, if the GUI focus is on a button on the UI), your custom control implementation also needs to reflect this more advanced focus concept and report the corresponding focused state.

• Set the control to be focused:

  bool Focused { get; set; }

It's assumed that setting the focused state of the control implicitly clears that of the other controls.

• Proactively notify Dragon Medical SpeechKit about having received the focus:

  event Action<object> GotFocus

Firing this event is optional but strongly recommended; if not implemented, user-initiated changes to the focus might not be detected by Dragon Medical SpeechKit in time and some text might be inserted into the wrong control. The parameter of the event invocation must be the custom control that has just received focus.

Format strings

• Report the character sequence representing a line break:

  string NewlineFormatString { get; }

For example, if your underlying text control uses HTML, you could return \n for this property and convert this character from/to the <br/> HTML markup.

• Report the character sequence representing a paragraph break:

  string ParagraphFormatString { get; }

  For example, if your underlying text control uses HTML, return \v for this property and convert this character to the <p>...</p> HTML markup.

Note: All methods and properties specified must be implemented and it's strongly recommended to fire the GotFocus event.

The custom VuiController

The custom VuiController (Nuance.SpeechAnywhere.Custom.VuiController) can interact with your custom control implementations. Its interface resembles that of the regular VuiController, which access the GUI directly (Nuance.SpeechAnywhere.WindowsForms.VuiController and Nuance.SpeechAnywhere.WPF.VuiController).

Regular VuiControllers automatically detect the controls which are speech-enabled by examining the UI of your app. This isn't possible in the custom control use case; the controls as seen by Dragon Medical SpeechKit (your ITextControl implementations) aren't necessarily contained by any UI object. Therefore, you must pass to the custom VuiController all the ITextControl instances you want to be speech-enabled.

• Initializing the custom VuiController.

  Whenever you initialize the custom VuiController, you must pass in the array of ITextControl instances you would like to have speech enabled. This array can contain a mixture of custom control implementations and natively supported controls (both from the WinForms and the WPF frameworks).

• Synchronizing the custom VuiController.

  As with regular VuiControllers, the Synchronize() method is used to notify the custom VuiController about changes in the voice user interface. Changes in the set of speech-enabled controls are communicated by passing the updated array of controls to Synchronize(). You must pass in this array even if your set of speech-enabled controls hasn't changed.

  Note

  The identity of controls in the Dragon Medical SpeechKit document model is tied to the custom control instances you pass to the Open() and Synchronize() methods. If you destroy a custom control instance and replace it with another one, this will be interpreted as an entirely new control, even if it's connected to the same control and tagged with the same ConceptName as the previous instance.

• Enabling/disabling speech recognition for a control.

  It follows from the above that the EnabledControl method is missing, and all controls you pass in to Open() and Synchronize() will be speech-enabled. To disable speech recognition for a custom control, omit it from the array and call Synchronize().

VuiController focus

Custom VuiController instances have an explicit mechanism for controlling which one of them is active. Set their Focused property to true for a given instance to receive speech recognition results. Setting a VuiController to be in focus implicitly removes focus from all other instances, therefore Dragon Medical SpeechKit ensures that only one VuiController can have the focus at any given time. Setting the Focused property of the currently focused VuiController to false will result in no VuiControllers having the focus. In this state, speech recognition isn't possible and the speech bar disappears. Starting speech recognition programmatically will have no effect.