Package opennlp.tools.tokenize

Class ThreadSafeTokenizerME

java.lang.Object

opennlp.tools.tokenize.ThreadSafeTokenizerME

All Implemented Interfaces:

AutoCloseable, opennlp.tools.ml.Probabilistic, opennlp.tools.tokenize.Tokenizer

@Deprecated(since="3.0.0")
@ThreadSafe
public class ThreadSafeTokenizerME
extends Object
implements opennlp.tools.tokenize.Tokenizer, opennlp.tools.ml.Probabilistic, AutoCloseable

Deprecated.

As of OPENNLP-1816, TokenizerME is itself thread-safe. Use it directly instead.

A thread-safe version of TokenizerME. Using it is completely transparent. You can use it in a single-threaded context as well, it only incurs a minimal overhead.

Note: This class is deprecated and now delegates to a single shared TokenizerME instance, which is thread-safe as of OPENNLP-1816. Calling close() clears the current thread's thread-local state for compatibility.

See Also:

• Probabilistic
• Tokenizer
• TokenizerME

Constructor Summary

Constructors

Constructor	Description
ThreadSafeTokenizerME(String language)	Deprecated. Initializes a ThreadSafeTokenizerME by downloading a default model for a given language.
ThreadSafeTokenizerME(TokenizerModel model)	Deprecated. Initializes a ThreadSafeTokenizerME with the specified model.
ThreadSafeTokenizerME(TokenizerModel model, Dictionary abbDict)	Deprecated. Instantiates a ThreadSafeTokenizerME with an existing TokenizerModel.

Method Summary

Modifier and Type	Method	Description
void	close()	Deprecated.
double[]	getProbabilities()	Deprecated, for removal: This API element is subject to removal in a future version. Use probs() instead.
double[]	probs()	Deprecated.
String[]	tokenize(String s)	Deprecated.
opennlp.tools.util.Span[]	tokenizePos(String s)	Deprecated.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

ThreadSafeTokenizerME

public ThreadSafeTokenizerME(String language)
                      throws IOException

Deprecated.

Initializes a ThreadSafeTokenizerME by downloading a default model for a given language.

Parameters:

language - An ISO conform language code.

Throws:

IOException - Thrown if the model could not be downloaded or saved.

ThreadSafeTokenizerME

public ThreadSafeTokenizerME(TokenizerModel model)

Deprecated.

Initializes a ThreadSafeTokenizerME with the specified model.

Parameters:

model - A valid TokenizerModel.

ThreadSafeTokenizerME

public ThreadSafeTokenizerME(TokenizerModel model,
 Dictionary abbDict)

Deprecated.

Instantiates a ThreadSafeTokenizerME with an existing TokenizerModel.

Parameters:

model - The TokenizerModel to be used.

abbDict - The Dictionary to be used. It must fit the language of the model.

Method Details

tokenize

public String[] tokenize(String s)

Deprecated.

Specified by:

tokenize in interface opennlp.tools.tokenize.Tokenizer

tokenizePos

public opennlp.tools.util.Span[] tokenizePos(String s)

Deprecated.

Specified by:

tokenizePos in interface opennlp.tools.tokenize.Tokenizer

probs

public double[] probs()

Deprecated.

Specified by:

probs in interface opennlp.tools.ml.Probabilistic

getProbabilities

@Deprecated(forRemoval=true,
            since="2.5.5")
public double[] getProbabilities()

Deprecated, for removal: This API element is subject to removal in a future version.

Use probs() instead.

close

public void close()

Deprecated.

Specified by:

close in interface AutoCloseable