Class MecabDictionary

java.lang.Object
opennlp.tools.tokenize.lattice.MecabDictionary
public final class MecabDictionary extends Object
An immutable, in-memory dictionary in the MeCab directory format: lexicon entries from the *.csv files, connection costs from matrix.def, character categories from char.def, and unknown-word templates from unk.def, loaded from a user-supplied dictionary directory. No dictionary data is bundled or downloaded by this class.

The same format serves multiple languages: the Japanese IPADIC and UniDic distributions and the Korean mecab-ko-dic all load through this one reader, with the feature columns passed through untouched because their schemas differ.

Each instance uses about 0.75 MB for category tables indexed by the 16-bit code-unit space, so load once and share. Lexicon CSV files under the directory are read in sorted path order so tie-breaking is stable across file systems. Connection costs must cover all matrix cells; missing and duplicate entries are rejected instead of being treated as cost zero. Matrix dimensions and the lexicon entry count are bounded by ResourceLimits.MAX_ENTRIES, and the matrix cell count by ResourceLimits.MAX_MATRIX_CELLS. Lexicon CSV fields may be MeCab-quoted with "" escapes. Word and connection costs must fit in a signed 16-bit integer. An unk.def template must name a category defined by char.def.

Instances are immutable and safe to share between threads.

Since:
3.0.0
See Also:

  • Method Details

    • load

      public static MecabDictionary load(Path directory) throws IOException
      Loads a dictionary directory encoded in UTF-8.
      Parameters:
      directory - The unpacked dictionary directory. Must not be null.
      Returns:
      The loaded dictionary. Never null.
      Throws:
      IOException - Thrown if reading fails or a file is malformed.
      IllegalArgumentException - Thrown if directory is null.

    • load

      public static MecabDictionary load(Path directory, Charset charset) throws IOException
      Loads a dictionary directory.
      Parameters:
      directory - The unpacked dictionary directory holding the *.csv lexicon files, matrix.def, char.def, and unk.def. Must not be null.
      charset - The encoding the distribution uses, for example UTF-8 or EUC-JP. Must not be null.
      Returns:
      The loaded dictionary. Never null.
      Throws:
      IOException - Thrown if reading fails, a required file is missing, a file is malformed, or a lexicon entry's context ids are outside the matrix.def dimensions.
      IllegalArgumentException - Thrown if a parameter is null.