Uses of Class
opennlp.tools.tokenize.TokenSample
Packages that use TokenSample
Package
Description
Experimental package related to converting various corpora to OpenNLP Format.
Experimental package related to the
Arvores Deitadas corpus
format.Experimental package related to the CoNNL-U format.
Experimental package related to the Irish Sentence Bank format.
Experimental package related to the
MASC
corpus format.Contains classes related to finding token or words in a string.
-
Uses of TokenSample in opennlp.tools.cmdline.tokenizer
Methods in opennlp.tools.cmdline.tokenizer with parameters of type TokenSampleModifier and TypeMethodDescriptionvoid
DetokenEvaluationErrorListener.misclassified
(TokenSample reference, TokenSample prediction) void
TokenEvaluationErrorListener.misclassified
(TokenSample reference, TokenSample prediction) -
Uses of TokenSample in opennlp.tools.formats
Methods in opennlp.tools.formats that return types with arguments of type TokenSampleModifier and TypeMethodDescription -
Uses of TokenSample in opennlp.tools.formats.ad
Methods in opennlp.tools.formats.ad that return types with arguments of type TokenSampleModifier and TypeMethodDescription -
Uses of TokenSample in opennlp.tools.formats.conllu
Methods in opennlp.tools.formats.conllu that return TokenSampleMethods in opennlp.tools.formats.conllu that return types with arguments of type TokenSampleModifier and TypeMethodDescription -
Uses of TokenSample in opennlp.tools.formats.convert
Methods in opennlp.tools.formats.convert that return TokenSampleMethods in opennlp.tools.formats.convert that return types with arguments of type TokenSampleModifier and TypeMethodDescription -
Uses of TokenSample in opennlp.tools.formats.irishsentencebank
Methods in opennlp.tools.formats.irishsentencebank that return TokenSampleModifier and TypeMethodDescriptionIrishSentenceBankDocument.IrishSentenceBankSentence.getTokenSample()
Methods in opennlp.tools.formats.irishsentencebank that return types with arguments of type TokenSampleModifier and TypeMethodDescription -
Uses of TokenSample in opennlp.tools.formats.masc
Methods in opennlp.tools.formats.masc that return TokenSampleMethods in opennlp.tools.formats.masc that return types with arguments of type TokenSampleModifier and TypeMethodDescription -
Uses of TokenSample in opennlp.tools.tokenize
Methods in opennlp.tools.tokenize that return TokenSampleModifier and TypeMethodDescriptionstatic TokenSample
Parses a string sample.protected TokenSample
DetokenizerEvaluator.processSample
(TokenSample reference) protected TokenSample
TokenizerEvaluator.processSample
(TokenSample reference) TokenizerStream.read()
TokenSampleStream.read()
Methods in opennlp.tools.tokenize with parameters of type TokenSampleModifier and TypeMethodDescriptionTokSpanEventStream.createEvents
(TokenSample tokenSample) Adds training events to the event stream for each of the specifiedsample
.protected TokenSample
DetokenizerEvaluator.processSample
(TokenSample reference) protected TokenSample
TokenizerEvaluator.processSample
(TokenSample reference) Method parameters in opennlp.tools.tokenize with type arguments of type TokenSampleModifier and TypeMethodDescriptionvoid
TokenizerCrossValidator.evaluate
(ObjectStream<TokenSample> samples, int nFolds) Starts the evaluation.static TokenizerModel
TokenizerME.train
(ObjectStream<TokenSample> samples, TokenizerFactory factory, TrainingParameters mlParams) Trains a model for theTokenizerME
.Constructor parameters in opennlp.tools.tokenize with type arguments of type TokenSampleModifierConstructorDescriptionTokSpanEventStream
(ObjectStream<TokenSample> tokenSamples, boolean skipAlphaNumerics) Initializes a new event stream based on the data stream using aTokenContextGenerator
that relies on aDefaultTokenContextGenerator
.TokSpanEventStream
(ObjectStream<TokenSample> tokenSamples, boolean skipAlphaNumerics, Pattern alphaNumeric, TokenContextGenerator cg) Initializes a new event stream based on the data stream using aTokenContextGenerator
.TokSpanEventStream
(ObjectStream<TokenSample> tokenSamples, boolean skipAlphaNumerics, TokenContextGenerator cg) Initializes a new event stream based on the data stream using aTokenContextGenerator
.WhitespaceTokenStream
(ObjectStream<TokenSample> tokens) Initializes aWhitespaceTokenStream
. -
Uses of TokenSample in opennlp.tools.tokenize.lang.en
Methods in opennlp.tools.tokenize.lang.en that return TokenSample