Class TokenizerAnnotator

java.lang.Object
opennlp.tools.tokenize.TokenizerAnnotator
All Implemented Interfaces:
opennlp.tools.document.DocumentAnnotator
public final class TokenizerAnnotator extends Object implements opennlp.tools.document.DocumentAnnotator
Adapts a Tokenizer to the document pipeline: provides Layers.TOKENS.

When Layers.SENTENCES is present, each sentence is tokenized separately and the token spans are shifted back to document coordinates; a present-but-empty sentence layer therefore yields a present-but-empty token layer. Only when the sentence layer is absent is the whole text tokenized at once. Either way, every token span refers to the original document text.

Since:
3.0.0

  • Constructor Summary

    Constructors
    Constructor
    Description
    TokenizerAnnotator(opennlp.tools.tokenize.Tokenizer tokenizer)
    Initializes the adapter.

  • Method Summary

    Modifier and Type
    Method
    Description
    opennlp.tools.document.Document
    annotate(opennlp.tools.document.Document document)
    Tokenizes the document and adds the Layers.TOKENS layer, sentence by sentence when a sentence layer is present and over the whole text otherwise.
    Set<opennlp.tools.document.LayerKey<?>>
    provides()
    String
    toString()
    Returns the adapter's simple class name, which names it in pipeline validation messages.

    Methods inherited from class Object

    equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

    Methods inherited from interface opennlp.tools.document.DocumentAnnotator

    requires

  • Constructor Details

    • TokenizerAnnotator

      public TokenizerAnnotator(opennlp.tools.tokenize.Tokenizer tokenizer)
      Initializes the adapter.
      Parameters:
      tokenizer - The tokenizer to delegate to. Must not be null.
      Throws:
      IllegalArgumentException - Thrown if tokenizer is null.

  • Method Details

    • annotate

      public opennlp.tools.document.Document annotate(opennlp.tools.document.Document document)
      Tokenizes the document and adds the Layers.TOKENS layer, sentence by sentence when a sentence layer is present and over the whole text otherwise.
      Specified by:
      annotate in interface opennlp.tools.document.DocumentAnnotator
      Parameters:
      document - The document to annotate. Must not be null.
      Returns:
      A new Document with the Layers.TOKENS layer added. Never null.
      Throws:
      IllegalArgumentException - Thrown if document is null or already carries the Layers.TOKENS layer.

    • provides

      public Set<opennlp.tools.document.LayerKey<?>> provides()
      Specified by:
      provides in interface opennlp.tools.document.DocumentAnnotator

    • toString

      public String toString()
      Returns the adapter's simple class name, which names it in pipeline validation messages.
      Overrides:
      toString in class Object
      Returns:
      the adapter's simple class name, which names it in pipeline validation messages