BertTokenizer.BuildInputsWithSpecialTokens Method
Definition
Important
Some information relates to prerelease product that may be substantially modified before it’s released. Microsoft makes no warranties, express or implied, with respect to the information provided here.
Overloads
| BuildInputsWithSpecialTokens(IEnumerable<Int32>, IEnumerable<Int32>) |
Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A BERT sequence has the following format:
- single sequence: |
| BuildInputsWithSpecialTokens(IEnumerable<Int32>, Span<Int32>, Int32, IEnumerable<Int32>) |
Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A BERT sequence has the following format:
- single sequence: |
BuildInputsWithSpecialTokens(IEnumerable<Int32>, IEnumerable<Int32>)
- Source:
- BertTokenizer.cs
- Source:
- BertTokenizer.cs
- Source:
- BertTokenizer.cs
Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A BERT sequence has the following format:
- single sequence: [CLS] tokenIds [SEP]
- pair of sequences: [CLS] tokenIds [SEP] additionalTokenIds [SEP]
public System.Collections.Generic.IReadOnlyList<int> BuildInputsWithSpecialTokens(System.Collections.Generic.IEnumerable<int> tokenIds, System.Collections.Generic.IEnumerable<int>? additionalTokenIds = default);member this.BuildInputsWithSpecialTokens : seq<int> * seq<int> -> System.Collections.Generic.IReadOnlyList<int>Public Function BuildInputsWithSpecialTokens (tokenIds As IEnumerable(Of Integer), Optional additionalTokenIds As IEnumerable(Of Integer) = Nothing) As IReadOnlyList(Of Integer)Parameters
- tokenIds
- IEnumerable<Int32>
List of IDs to which the special tokens will be added.
- additionalTokenIds
- IEnumerable<Int32>
Optional second list of IDs for sequence pairs.
Returns
The list of IDs with special tokens added.
Exceptions
When tokenIds is null.
Applies to
BuildInputsWithSpecialTokens(IEnumerable<Int32>, Span<Int32>, Int32, IEnumerable<Int32>)
- Source:
- BertTokenizer.cs
- Source:
- BertTokenizer.cs
- Source:
- BertTokenizer.cs
Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A BERT sequence has the following format:
- single sequence: [CLS] tokenIds [SEP]
- pair of sequences: [CLS] tokenIds [SEP] additionalTokenIds [SEP]
public System.Buffers.OperationStatus BuildInputsWithSpecialTokens(System.Collections.Generic.IEnumerable<int> tokenIds, Span<int> destination, out int valuesWritten, System.Collections.Generic.IEnumerable<int>? additionalTokenIds = default);member this.BuildInputsWithSpecialTokens : seq<int> * Span<int> * int * seq<int> -> System.Buffers.OperationStatusPublic Function BuildInputsWithSpecialTokens (tokenIds As IEnumerable(Of Integer), destination As Span(Of Integer), ByRef valuesWritten As Integer, Optional additionalTokenIds As IEnumerable(Of Integer) = Nothing) As OperationStatusParameters
- tokenIds
- IEnumerable<Int32>
List of IDs to which the special tokens will be added.
- valuesWritten
- Int32
The number of elements written to the destination buffer.
- additionalTokenIds
- IEnumerable<Int32>
Optional second list of IDs for sequence pairs.
Returns
The status of the operation.
Exceptions
When tokenIds is null.