Package opennlp.tools.tokenize

package opennlp.tools.tokenize

Related Packages

Package	Description
opennlp.tools.tokenize.lattice

Class	Description
Detokenizer	A Detokenizer merges tokens back to their detokenized representation.
Detokenizer.DetokenizationOperation	This enum contains an operation for every token to merge the tokens together to their detokenized form.
SubwordPiece	One subword unit produced by a SubwordTokenizer, including the model representation and source range.
SubwordTokenizer	Splits text into subword units from a fixed model vocabulary, reporting the model id and original-text span for each unit.
TokenContextGenerator	Interface for context generators required for tokenizer implementations.
Tokenizer	The interface for tokenizers, which segment a string into its tokens.
TokenizerEvaluationMonitor	A marker interface for evaluating tokenizers.
TokenSample	A TokenSample is text with token spans.
WhitespaceTokenizer	A basic Tokenizer implementation which performs tokenization using white spaces.
WordpieceEncoder	A SubwordTokenizer implementing the BERT tokenization stages: basic tokenization (control removal, whitespace normalization, CJK isolation, optional lower casing with accent stripping, punctuation isolation) followed by greedy longest-match wordpiece segmentation.
WordpieceTokenizer	A Tokenizer implementation which performs tokenization using word pieces.