Package opennlp.tools.tokenize

Class TokenizerME

java.lang.Object
opennlp.tools.tokenize.TokenizerME
All Implemented Interfaces:
opennlp.tools.ml.Probabilistic, opennlp.tools.tokenize.Tokenizer
@ThreadSafe public class TokenizerME extends Object implements opennlp.tools.ml.Probabilistic
A Tokenizer for converting raw text into separated tokens. It uses Maximum Entropy to make its decisions. The features are loosely based off of Jeff Reynar's UPenn thesis "Topic Segmentation: Algorithms and Applications.", which is available from his homepage: http://www.cis.upenn.edu/~jcreynar.

This implementation needs a statistical model to tokenize a text which reproduces the tokenization observed in the training data used to create the model. The TokenizerModel class encapsulates that model and provides methods to create it from the binary representation.

A tokenizer instance is thread-safe. One tokenizer can be shared across multiple threads to save memory.

Note: In container environments with classloader isolation (e.g. Jakarta EE), ensure instances do not outlive the application's lifecycle, as underlying components use ThreadLocal state that may pin the classloader.

To train a new model, use train(ObjectStream, TokenizerFactory, TrainingParameters).

Sample usage:

InputStream modelIn;

...

TokenizerModel model = TokenizerModel(modelIn);

Tokenizer tokenizer = new TokenizerME(model);

String tokens[] = tokenizer.tokenize("A sentence to be tokenized.");

See Also:

  • Field Details

  • Constructor Details

  • Method Details

    • probs

      public double[] probs()
      The sequence was determined based on the previous call to tokenizePos(String).
      Specified by:
      probs in interface opennlp.tools.ml.Probabilistic
      Returns:
      an array with the same number of probabilities as tokens were sent to the computational method when tokenizePos(String) was last called; if not applicable, an empty array

    • getTokenProbabilities

      @Deprecated(forRemoval=true, since="2.5.5") public double[] getTokenProbabilities()
      Deprecated, for removal: This API element is subject to removal in a future version.
      Use probs() instead.
      Returns:
      the probabilities associated with the most recent calls to tokenizePos(String); if not applicable, an empty array

    • tokenizePos

      public opennlp.tools.util.Span[] tokenizePos(String d)
      Tokenizes the string.
      Specified by:
      tokenizePos in interface opennlp.tools.tokenize.Tokenizer
      Parameters:
      d - The string to be tokenized.
      Returns:
      A Span array containing individual tokens as elements.

    • clearThreadLocalState

      public void clearThreadLocalState()
      Removes thread-local state to prevent classloader leaks in container environments. Call when the thread is returned to a pool or the tokenizer is no longer needed.

    • train

      public static TokenizerModel train(opennlp.tools.util.ObjectStream<opennlp.tools.tokenize.TokenSample> samples, TokenizerFactory factory, opennlp.tools.util.TrainingParameters mlParams) throws IOException
      Trains a model for the TokenizerME.
      Parameters:
      samples - The samples used for the training.
      factory - A TokenizerFactory to get resources from.
      mlParams - The machine learning train parameters.
      Returns:
      A trained TokenizerModel.
      Throws:
      IOException - Thrown during IO on a temp file created during training, or if reading from the ObjectStream fails.

    • useAlphaNumericOptimization

      public boolean useAlphaNumericOptimization()
      Returns:
      true if the tokenizer uses alphanumeric optimization, false otherwise.

    • tokenize

      public String[] tokenize(String arg0)
      Specified by:
      tokenize in interface opennlp.tools.tokenize.Tokenizer

    • setKeepNewLines

      public void setKeepNewLines(boolean arg0)