Package opennlp.tools.tokenize

Class BPETokenizer

java.lang.Object

opennlp.tools.tokenize.BPETokenizer

All Implemented Interfaces:

opennlp.tools.tokenize.Tokenizer

public class BPETokenizer
extends Object
implements opennlp.tools.tokenize.Tokenizer

A Tokenizer implementation that performs subword tokenization using Byte Pair Encoding (BPE).

BPE iteratively merges the most frequent pair of adjacent symbols, starting from a character-level representation of each word. This allows the tokenizer to handle out-of-vocabulary words by decomposing them into known subword units.

Usage:

 // Train a BPE model from a corpus
 BPETokenizerTrainer trainer = new BPETokenizerTrainer();
 BPEModel model = trainer.train(corpus, 10000, "en");

 // Save the model for later reuse
 model.serialize(Path.of("bpe-en.bin"));

 // Load and tokenize
 BPEModel loaded = new BPEModel(Path.of("bpe-en.bin"));
 BPETokenizer tokenizer = new BPETokenizer(loaded);
 String[] tokens = tokenizer.tokenize("unseen words are split into subwords");
 

The tokenizer first splits text on whitespace, then applies learned merge operations to each word independently. Words are decomposed into characters with an END_OF_WORD marker on the final character, then merges are applied in priority order (as learned during training) until no more merges are applicable. The resulting subword units are returned as tokens.

For reference see:

• Sennrich, R., Haddow, B., & Birch, A. (2016). Neural Machine Translation of Rare Words with Subword Units. https://arxiv.org/abs/1508.07909

See Also:

• BPEModel
• BPETokenizerTrainer
• WordpieceTokenizer

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static final record	BPETokenizer.SymbolPair	Represents a pair of adjacent symbols in BPE.

Field Summary

Fields

Modifier and Type	Field	Description
static final String	END_OF_WORD	Suffix appended to the last symbol of each word during BPE encoding to distinguish word-final characters from word-internal ones.

Constructor Summary

Constructors

Constructor	Description
BPETokenizer(BPEModel model)	Initializes a BPETokenizer from a trained BPEModel.

Method Summary

Modifier and Type	Method	Description
String[]	tokenize(String text)
opennlp.tools.util.Span[]	tokenizePos(String text)

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

END_OF_WORD

public static final String END_OF_WORD

Suffix appended to the last symbol of each word during BPE encoding to distinguish word-final characters from word-internal ones.

Users constructing BPETokenizer.SymbolPair merge rules manually must use this constant to mark word-final symbols (e.g., new SymbolPair("a", "b" + END_OF_WORD)).

See Also:

• Constant Field Values

Constructor Details

BPETokenizer

public BPETokenizer(BPEModel model)

Initializes a BPETokenizer from a trained BPEModel.

Parameters:

model - The trained BPE model. Must not be null.

Throws:

IllegalArgumentException - if model is null.

Method Details

tokenize

public String[] tokenize(String text)

Splits the input text on whitespace, then applies BPE merge operations to each word to produce subword tokens. Words not fully covered by learned merges are decomposed into individual characters.

Specified by:

tokenize in interface opennlp.tools.tokenize.Tokenizer

tokenizePos

public opennlp.tools.util.Span[] tokenizePos(String text)

Returns Span offsets into the original text for each subword token. Each span maps back to the exact character range in the input string.

Specified by:

tokenizePos in interface opennlp.tools.tokenize.Tokenizer