Class TermVectorAnnotator

java.lang.Object
opennlp.tools.termvector.TermVectorAnnotator
All Implemented Interfaces:
opennlp.tools.document.DocumentAnnotator
public final class TermVectorAnnotator extends Object implements opennlp.tools.document.DocumentAnnotator
Rolls the token layer up into a term vector layer for index consumers: one TermVector per distinct term, carrying the term string, its occurrence count, and (in full mode) the occurrence offsets.

Term identity comes from the annotator's inputs, not from logic of its own. Without a normalizer, the term is the token layer's value as-is, that is, the token's covered text in the original document. With a plain CharSequenceNormalizer, the general path, each token's covered text is normalized on its own to produce the term, so any normalizer works: case folding, NFC, accent folding, a stemmer-backed normalizer. With an OffsetAwareNormalizer, the whole document text is normalized once with its alignment recorded, each token span is mapped forward to the normalized form, and the covered normalized text is the term; this path can see across token boundaries but is limited to alignment-reporting normalizers. On every path, tokens that differ only by a normalization fold (case, an eszett expansion, collapsed whitespace) group together, and the occurrence spans emitted in full mode are the token layer's own spans and therefore always point into the original text. A token whose normalized form is empty, for example one the normalizer deleted entirely, is omitted from the layer; an empty string is no term, and the token layer still accounts for the token.

The layer is document-scoped: each TermVector is a whole-document statistic, so the annotations carry no span of their own and the occurrence offsets live inside the payload. The layer preserves first-occurrence order: the first token of a term fixes its position in the layer.

The annotator holds no per-call state; it is as thread-safe as the normalizer it was built with.

Since:
3.0.0

  • Nested Class Summary

    Nested Classes
    Modifier and Type
    Class
    Description
    static enum 
    TermVectorAnnotator.Mode
    How much each TermVector records.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final opennlp.tools.document.LayerKey<opennlp.tools.termvector.TermVector>
    TERM_VECTORS
    The key of the term vector layer this annotator provides: a document-scoped layer of TermVector values, one per distinct term.

  • Constructor Summary

    Constructors
    Constructor
    Description
    TermVectorAnnotator()
    Initializes a full mode annotator that groups tokens by their covered text as-is.
    TermVectorAnnotator(TermVectorAnnotator.Mode mode)
    Initializes an annotator that groups tokens by their covered text as-is.
    TermVectorAnnotator(opennlp.tools.util.normalizer.CharSequenceNormalizer normalizer)
    Initializes a full mode annotator that groups tokens by their per-token normalized form.
    TermVectorAnnotator(opennlp.tools.util.normalizer.CharSequenceNormalizer normalizer, TermVectorAnnotator.Mode mode)
    Initializes an annotator that groups tokens by their per-token normalized form.
    TermVectorAnnotator(opennlp.tools.util.normalizer.OffsetAwareNormalizer normalizer)
    Initializes a full mode annotator that groups tokens by their normalized form through a whole-document alignment.
    TermVectorAnnotator(opennlp.tools.util.normalizer.OffsetAwareNormalizer normalizer, TermVectorAnnotator.Mode mode)
    Initializes an annotator that groups tokens by their normalized form through a whole-document alignment.

  • Method Summary

    Modifier and Type
    Method
    Description
    opennlp.tools.document.Document
    annotate(opennlp.tools.document.Document document)
    Aggregates the token layer into the TERM_VECTORS layer.
    Set<opennlp.tools.document.LayerKey<?>>
    provides()
    Set<opennlp.tools.document.LayerKey<?>>
    requires()

    Methods inherited from class Object

    equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Field Details

    • TERM_VECTORS

      public static final opennlp.tools.document.LayerKey<opennlp.tools.termvector.TermVector> TERM_VECTORS
      The key of the term vector layer this annotator provides: a document-scoped layer of TermVector values, one per distinct term.

  • Constructor Details

    • TermVectorAnnotator

      public TermVectorAnnotator()
      Initializes a full mode annotator that groups tokens by their covered text as-is.

    • TermVectorAnnotator

      public TermVectorAnnotator(TermVectorAnnotator.Mode mode)
      Initializes an annotator that groups tokens by their covered text as-is.
      Parameters:
      mode - How much each TermVector records. Must not be null.
      Throws:
      IllegalArgumentException - Thrown if mode is null.

    • TermVectorAnnotator

      public TermVectorAnnotator(opennlp.tools.util.normalizer.CharSequenceNormalizer normalizer)
      Initializes a full mode annotator that groups tokens by their per-token normalized form. This is the general path: the normalizer is applied to each token's covered text on its own, so any CharSequenceNormalizer works, including the folds that cannot report an alignment (case folding, NFC, accent folding, stemmer-backed normalizers). The occurrence spans stay the tokens' own spans in the original text.
      Parameters:
      normalizer - The normalizer that defines term identity, applied to each token's covered text. Must not be null.
      Throws:
      IllegalArgumentException - Thrown if normalizer is null.

    • TermVectorAnnotator

      public TermVectorAnnotator(opennlp.tools.util.normalizer.CharSequenceNormalizer normalizer, TermVectorAnnotator.Mode mode)
      Initializes an annotator that groups tokens by their per-token normalized form. This is the general path: the normalizer is applied to each token's covered text on its own, so any CharSequenceNormalizer works, including the folds that cannot report an alignment (case folding, NFC, accent folding, stemmer-backed normalizers). The occurrence spans stay the tokens' own spans in the original text.
      Parameters:
      normalizer - The normalizer that defines term identity, applied to each token's covered text. Must not be null.
      mode - How much each TermVector records. Must not be null.
      Throws:
      IllegalArgumentException - Thrown if normalizer or mode is null.

    • TermVectorAnnotator

      public TermVectorAnnotator(opennlp.tools.util.normalizer.OffsetAwareNormalizer normalizer)
      Initializes a full mode annotator that groups tokens by their normalized form through a whole-document alignment. Prefer the plain-normalizer constructor as the general path; this one only accepts alignment-reporting normalizers but can see across token boundaries, for example a whitespace collapse spanning two tokens.
      Parameters:
      normalizer - The normalizer that defines term identity, applied to the whole document text so token spans can be mapped into the normalized form through its alignment. Must not be null.
      Throws:
      IllegalArgumentException - Thrown if normalizer is null.

    • TermVectorAnnotator

      public TermVectorAnnotator(opennlp.tools.util.normalizer.OffsetAwareNormalizer normalizer, TermVectorAnnotator.Mode mode)
      Initializes an annotator that groups tokens by their normalized form through a whole-document alignment. Prefer the plain-normalizer constructor as the general path; this one only accepts alignment-reporting normalizers but can see across token boundaries, for example a whitespace collapse spanning two tokens.
      Parameters:
      normalizer - The normalizer that defines term identity, applied to the whole document text so token spans can be mapped into the normalized form through its alignment. Must not be null.
      mode - How much each TermVector records. Must not be null.
      Throws:
      IllegalArgumentException - Thrown if normalizer or mode is null.

  • Method Details

    • annotate

      public opennlp.tools.document.Document annotate(opennlp.tools.document.Document document)
      Aggregates the token layer into the TERM_VECTORS layer. A present-but-empty token layer yields a present-but-empty term vector layer.
      Specified by:
      annotate in interface opennlp.tools.document.DocumentAnnotator
      Parameters:
      document - The document to annotate. Must not be null and must contain the Layers.TOKENS layer.
      Returns:
      A new Document with the TERM_VECTORS layer added. Never null.
      Throws:
      IllegalArgumentException - Thrown if document is null, lacks the Layers.TOKENS layer, or already carries the TERM_VECTORS layer.

    • requires

      public Set<opennlp.tools.document.LayerKey<?>> requires()
      Specified by:
      requires in interface opennlp.tools.document.DocumentAnnotator

    • provides

      public Set<opennlp.tools.document.LayerKey<?>> provides()
      Specified by:
      provides in interface opennlp.tools.document.DocumentAnnotator