Package opennlp.tools.tokenize

package opennlp.tools.tokenize

Related Packages

Package	Description
opennlp.tools.tokenize.lang

Class	Description
BPEModel	The BPEModel stores learned BPE merge operations and can be serialized and deserialized for reuse.
BPETokenizer	A Tokenizer implementation that performs subword tokenization using Byte Pair Encoding (BPE).
BPETokenizer.SymbolPair	Represents a pair of adjacent symbols in BPE.
BPETokenizerFactory	A BaseToolFactory for BPE tokenization that manages the BPE merge rules artifact and its serialization within a BPEModel.
BPETokenizerTrainer	Learns BPE merge operations from a training corpus and produces a BPEModel.
DefaultTokenContextGenerator	A default TokenContextGenerator which produces events for maxent decisions for tokenization.
DetokenizationDictionary
DetokenizationDictionary.Operation
DictionaryDetokenizer	A rule based detokenizer.
SimpleTokenizer	A basic Tokenizer implementation which performs tokenization using character classes.
ThreadSafeTokenizerME	Deprecated. As of OPENNLP-1816, TokenizerME is itself thread-safe.
TokenizerCrossValidator	A cross validator for tokenizers.
TokenizerEvaluator	The TokenizerEvaluator measures the performance of the given Tokenizer with the provided reference samples.
TokenizerFactory	The factory that provides Tokenizer default implementation and resources.
TokenizerME	A Tokenizer for converting raw text into separated tokens.
TokenizerModel	The TokenizerModel is the model used by a learnable Tokenizer.
TokenizerStream	The TokenizerStream uses a Tokenizer to tokenize the input string and output samples.
TokenSampleStream	This class is a stream filter which reads in string encoded samples and creates samples out of them.
TokSpanEventStream	This class reads the samples via an Iterator and converts the samples into events which can be used by the maxent library for training.
WhitespaceTokenStream	This stream formats ObjectStream of samples into whitespace separated token strings.