BertTokenizer.CreateTokenTypeIdsFromSequences Method
Definition
Important
Some information relates to prerelease product that may be substantially modified before it’s released. Microsoft makes no warranties, express or implied, with respect to the information provided here.
Overloads
| CreateTokenTypeIdsFromSequences(IEnumerable<Int32>, IEnumerable<Int32>) |
Create a mask from the two sequences passed to be used in a sequence-pair classification task. A BERT sequence pair mask has the following format:
0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1
| first sequence | second sequence |
If |
| CreateTokenTypeIdsFromSequences(IEnumerable<Int32>, Span<Int32>, Int32, IEnumerable<Int32>) |
CreateTokenTypeIdsFromSequences(IEnumerable<Int32>, IEnumerable<Int32>)
- Source:
- BertTokenizer.cs
- Source:
- BertTokenizer.cs
- Source:
- BertTokenizer.cs
Create a mask from the two sequences passed to be used in a sequence-pair classification task. A BERT sequence pair mask has the following format:
0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1
| first sequence | second sequence |
If additionalTokenIds is null, this method only returns the first portion of the type ids (0s).
public System.Collections.Generic.IReadOnlyList<int> CreateTokenTypeIdsFromSequences(System.Collections.Generic.IEnumerable<int> tokenIds, System.Collections.Generic.IEnumerable<int>? additionalTokenIds = default);member this.CreateTokenTypeIdsFromSequences : seq<int> * seq<int> -> System.Collections.Generic.IReadOnlyList<int>Public Function CreateTokenTypeIdsFromSequences (tokenIds As IEnumerable(Of Integer), Optional additionalTokenIds As IEnumerable(Of Integer) = Nothing) As IReadOnlyList(Of Integer)Parameters
- tokenIds
- IEnumerable<Int32>
List of token IDs for the first sequence.
- additionalTokenIds
- IEnumerable<Int32>
Optional list of token IDs for the second sequence.
Returns
List of token type IDs according to the given sequence(s).
Exceptions
When tokenIds is null.
Applies to
CreateTokenTypeIdsFromSequences(IEnumerable<Int32>, Span<Int32>, Int32, IEnumerable<Int32>)
- Source:
- BertTokenizer.cs
- Source:
- BertTokenizer.cs
- Source:
- BertTokenizer.cs
public System.Buffers.OperationStatus CreateTokenTypeIdsFromSequences(System.Collections.Generic.IEnumerable<int> tokenIds, Span<int> destination, out int valuesWritten, System.Collections.Generic.IEnumerable<int>? additionalTokenIds = default);member this.CreateTokenTypeIdsFromSequences : seq<int> * Span<int> * int * seq<int> -> System.Buffers.OperationStatusPublic Function CreateTokenTypeIdsFromSequences (tokenIds As IEnumerable(Of Integer), destination As Span(Of Integer), ByRef valuesWritten As Integer, Optional additionalTokenIds As IEnumerable(Of Integer) = Nothing) As OperationStatusParameters
- tokenIds
- IEnumerable<Int32>
- valuesWritten
- Int32
- additionalTokenIds
- IEnumerable<Int32>