Package opennlp.tools.tokenize

Class WordpieceTokenizer

java.lang.Object

opennlp.tools.tokenize.WordpieceTokenizer

All Implemented Interfaces:

Tokenizer

public class WordpieceTokenizer
extends Object
implements Tokenizer

A Tokenizer implementation which performs tokenization using word pieces.

Adapted under MIT license from https://github.com/robrua/easy-bert.

For reference see:

• https://www.tensorflow.org/text/guide/subwords_tokenizer#applying_wordpiece
• https://cran.r-project.org/web/packages/wordpiece/vignettes/basic_usage.html

Field Summary

Fields

Modifier and Type	Field	Description
static final String	BERT_CLS_TOKEN	BERT classification token: [CLS].
static final String	BERT_SEP_TOKEN	BERT separator token: [SEP].
static final String	BERT_UNK_TOKEN	BERT unknown token: [UNK].
static final String	ROBERTA_CLS_TOKEN	RoBERTa classification token: <s>.
static final String	ROBERTA_SEP_TOKEN	RoBERTa separator token.
static final String	ROBERTA_UNK_TOKEN	RoBERTa unknown token.

Constructor Summary

Constructors

Constructor	Description
WordpieceTokenizer(Set<String> vocabulary)	Initializes a WordpieceTokenizer with a vocabulary and a default maxTokenLength of 50.
WordpieceTokenizer(Set<String> vocabulary, int maxTokenLength)	Initializes a WordpieceTokenizer with a vocabulary and a custom maxTokenLength.
WordpieceTokenizer(Set<String> vocabulary, String classificationToken, String separatorToken, String unknownToken)	Initializes a WordpieceTokenizer with a vocabulary and custom special tokens.

Method Summary

Modifier and Type	Method	Description
int	getMaxTokenLength()
String[]	tokenize(String text)	Splits a string into its atomic parts.
Span[]	tokenizePos(String text)	Finds the boundaries of atomic parts in a string.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

BERT_CLS_TOKEN

public static final String BERT_CLS_TOKEN

BERT classification token: [CLS].

See Also:

• Constant Field Values

BERT_SEP_TOKEN

public static final String BERT_SEP_TOKEN

BERT separator token: [SEP].

See Also:

• Constant Field Values

BERT_UNK_TOKEN

public static final String BERT_UNK_TOKEN

BERT unknown token: [UNK].

See Also:

• Constant Field Values

ROBERTA_CLS_TOKEN

public static final String ROBERTA_CLS_TOKEN

RoBERTa classification token: <s>.

See Also:

• Constant Field Values

ROBERTA_SEP_TOKEN

public static final String ROBERTA_SEP_TOKEN

RoBERTa separator token.

See Also:

• Constant Field Values

ROBERTA_UNK_TOKEN

public static final String ROBERTA_UNK_TOKEN

RoBERTa unknown token.

See Also:

• Constant Field Values

Constructor Details

WordpieceTokenizer

public WordpieceTokenizer(Set<String> vocabulary)

Initializes a WordpieceTokenizer with a vocabulary and a default maxTokenLength of 50.

Parameters:

vocabulary - A set of tokens considered the vocabulary.

WordpieceTokenizer

public WordpieceTokenizer(Set<String> vocabulary,
 int maxTokenLength)

Initializes a WordpieceTokenizer with a vocabulary and a custom maxTokenLength.

Parameters:

vocabulary - A set of tokens considered the vocabulary.

maxTokenLength - A non-negative number that is used as maximum token length.

WordpieceTokenizer

public WordpieceTokenizer(Set<String> vocabulary,
 String classificationToken,
 String separatorToken,
 String unknownToken)

Initializes a WordpieceTokenizer with a vocabulary and custom special tokens. This allows support for models like RoBERTa that use different special tokens instead of the BERT defaults.

Parameters:

vocabulary - The vocabulary.

classificationToken - The CLS token.

separatorToken - The SEP token.

unknownToken - The UNK token.

Method Details

tokenizePos

public Span[] tokenizePos(String text)

Description copied from interface: Tokenizer

Finds the boundaries of atomic parts in a string.

Specified by:

tokenizePos in interface Tokenizer

Parameters:

text - The string to be tokenized.

Returns:

The spans (offsets into s) for each token as the individuals array elements.

tokenize

public String[] tokenize(String text)

Description copied from interface: Tokenizer

Splits a string into its atomic parts.

Specified by:

tokenize in interface Tokenizer

Parameters:

text - The string to be tokenized.

Returns:

The String[] with the individual tokens as the array elements.

getMaxTokenLength

public int getMaxTokenLength()

Returns:

The maximum token length.