WordPieceTokenizer.CreateAsync Method
Definition
Important
Some information relates to prerelease product that may be substantially modified before it’s released. Microsoft makes no warranties, express or implied, with respect to the information provided here.
Overloads
| CreateAsync(Stream, WordPieceOptions, CancellationToken) |
Create a new instance of the WordPieceTokenizer class asynchronously. |
| CreateAsync(String, WordPieceOptions, CancellationToken) |
Create a new instance of the WordPieceTokenizer class asynchronously. |
CreateAsync(Stream, WordPieceOptions, CancellationToken)
- Source:
- WordPieceTokenizer.cs
- Source:
- WordPieceTokenizer.cs
- Source:
- WordPieceTokenizer.cs
Create a new instance of the WordPieceTokenizer class asynchronously.
public static System.Threading.Tasks.Task<Microsoft.ML.Tokenizers.WordPieceTokenizer> CreateAsync(System.IO.Stream vocabStream, Microsoft.ML.Tokenizers.WordPieceOptions? options = default, System.Threading.CancellationToken cancellationToken = default);static member CreateAsync : System.IO.Stream * Microsoft.ML.Tokenizers.WordPieceOptions * System.Threading.CancellationToken -> System.Threading.Tasks.Task<Microsoft.ML.Tokenizers.WordPieceTokenizer>Public Shared Function CreateAsync (vocabStream As Stream, Optional options As WordPieceOptions = Nothing, Optional cancellationToken As CancellationToken = Nothing) As Task(Of WordPieceTokenizer)Parameters
- vocabStream
- Stream
The path to the WordPiece vocab file.
- options
- WordPieceOptions
The options to use for the WordPiece tokenizer.
- cancellationToken
- CancellationToken
The cancellation token.
Returns
A new instance of the WordPieceTokenizer class.
Remarks
If the PreTokenizer is null, the whitespace pre-tokenizer will be used. When creating the tokenizer, ensure that the vocabulary stream is sourced from a trusted provider.
Applies to
CreateAsync(String, WordPieceOptions, CancellationToken)
- Source:
- WordPieceTokenizer.cs
- Source:
- WordPieceTokenizer.cs
- Source:
- WordPieceTokenizer.cs
Create a new instance of the WordPieceTokenizer class asynchronously.
public static System.Threading.Tasks.Task<Microsoft.ML.Tokenizers.WordPieceTokenizer> CreateAsync(string vocabFilePath, Microsoft.ML.Tokenizers.WordPieceOptions? options = default, System.Threading.CancellationToken cancellationToken = default);static member CreateAsync : string * Microsoft.ML.Tokenizers.WordPieceOptions * System.Threading.CancellationToken -> System.Threading.Tasks.Task<Microsoft.ML.Tokenizers.WordPieceTokenizer>Public Shared Function CreateAsync (vocabFilePath As String, Optional options As WordPieceOptions = Nothing, Optional cancellationToken As CancellationToken = Nothing) As Task(Of WordPieceTokenizer)Parameters
- vocabFilePath
- String
The path to the WordPiece vocab file.
- options
- WordPieceOptions
The options to use for the WordPiece tokenizer.
- cancellationToken
- CancellationToken
The cancellation token.
Returns
A new instance of the WordPieceTokenizer class.
Remarks
If the PreTokenizer is null, the whitespace pre-tokenizer will be used. When creating the tokenizer, ensure that the vocabulary file is sourced from a trusted provider.