Class LatticeTokenizer

java.lang.Object
opennlp.tools.tokenize.lattice.LatticeTokenizer
All Implemented Interfaces:
Tokenizer
public final class LatticeTokenizer extends Object implements Tokenizer
Dictionary-driven segmentation for languages written without spaces: a Viterbi search over the word lattice of a MecabDictionary, minimizing the sum of word costs and connection costs. This is the segmentation approach behind Japanese and Korean morphological analysis. A single decoder serves both because the supplied dictionary provides the language-specific data.

Unknown text is handled through the dictionary's character categories: where the lexicon has no entry, or a category always invokes them, unknown-word candidates are generated per category template, grouping runs of same-category characters when the category requests it. A multi-category run continues while successive assignments overlap. Whitespace cannot join or appear as a morpheme. Every reported span is in original text coordinates.

analyze(String) returns full morphemes with their dictionary features; the Tokenizer view reports just the surfaces and spans.

The tokenizer reads only immutable dictionary state and is safe to share between threads.

Since:
3.0.0

  • Constructor Details

    • LatticeTokenizer

      public LatticeTokenizer(MecabDictionary dictionary)
      Initializes the tokenizer.
      Parameters:
      dictionary - The dictionary to segment with. Must not be null.
      Throws:
      IllegalArgumentException - Thrown if dictionary is null.

  • Method Details

    • analyze

      public List<Morpheme> analyze(String text)
      Segments a text into morphemes with their dictionary features.
      Parameters:
      text - The text to segment. Must not be null.
      Returns:
      The morphemes in text order, spans in original coordinates, whitespace omitted. Never null; empty for empty or all-whitespace input.
      Throws:
      IllegalArgumentException - Thrown if text is null.
      IllegalStateException - Thrown if the dictionary offers no candidate at some position, which a unk.def without a DEFAULT template does.

    • tokenize

      public String[] tokenize(String text)
      Splits a string into its atomic parts.

      Reports the segmented surfaces, whitespace omitted.

      Specified by:
      tokenize in interface Tokenizer
      Parameters:
      text - The string to be tokenized.
      Returns:
      The String[] with the individual tokens as the array elements.
      Throws:
      IllegalArgumentException - Thrown if text is null.
      IllegalStateException - Thrown if the dictionary offers no candidate at some position; see analyze(String).

    • tokenizePos

      public Span[] tokenizePos(String text)
      Finds the boundaries of atomic parts in a string.

      Reports the segmented spans in original text coordinates, whitespace omitted.

      Specified by:
      tokenizePos in interface Tokenizer
      Parameters:
      text - The string to be tokenized.
      Returns:
      The spans (offsets into s) for each token as the individuals array elements.
      Throws:
      IllegalArgumentException - Thrown if text is null.
      IllegalStateException - Thrown if the dictionary offers no candidate at some position; see analyze(String).