Uses of Package
opennlp.tools.tokenize

Packages that use opennlp.tools.tokenize

Package	Description
opennlp.tools.tokenize

Classes in opennlp.tools.tokenize used by opennlp.tools.tokenize

Class	Description
BPEModel	The BPEModel stores learned BPE merge operations and can be serialized and deserialized for reuse.
BPETokenizer.SymbolPair	Represents a pair of adjacent symbols in BPE.
BPETokenizerFactory	A BaseToolFactory for BPE tokenization that manages the BPE merge rules artifact and its serialization within a BPEModel.
DetokenizationDictionary
DetokenizationDictionary.Operation
SimpleTokenizer	A basic Tokenizer implementation which performs tokenization using character classes.
TokenizerFactory	The factory that provides Tokenizer default implementation and resources.
TokenizerModel	The TokenizerModel is the model used by a learnable Tokenizer.