Interface SubwordTokenizer

All Known Implementing Classes:

WordpieceEncoder

public interface SubwordTokenizer

Splits text into subword units from a fixed model vocabulary, reporting the model id and original-text span for each unit.

Segmentation follows model entries, not linguistic token boundaries. Each piece is in the model's normalized form and need not equal the input. Offsets in each SubwordPiece refer to the original input text.

Subword units were introduced to neural models to keep the vocabulary closed while still covering rare and unseen words: byte pair encoding (Sennrich et al., 2016), the WordPiece inventory (Schuster and Nakajima, 2012; Wu et al., 2016) and the unigram language model (Kudo, 2018). This interface abstracts over the segmentation method and describes only what the trained inventories have in common, following the tokenizer-independent encoding contract of SentencePiece (Kudo and Richardson, 2018): every unit is a vocabulary entry with an id and a range over the input.

An implementation may include model control pieces with empty source spans. Their presence and placement are part of that tokenizer's contract, not this interface.

Thread safety is implementation specific.

Since:

3.0.0

See Also:

• SubwordPiece
• Rico Sennrich, Barry Haddow, Alexandra Birch (2016): Neural Machine Translation of Rare Words with Subword Units. ACL 2016, pages 1715-1725
• Taku Kudo (2018): Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates. ACL 2018, pages 66-75
• Taku Kudo, John Richardson (2018): SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. EMNLP 2018 (system demonstrations), pages 66-71

Method Summary

Modifier and Type	Method	Description
List<SubwordPiece>	encode(CharSequence text)	Encodes text into subword pieces.
default int[]	encodeToIds(CharSequence text)	Encodes text into vocabulary ids.
default String[]	encodeToPieces(CharSequence text)	Encodes text into piece strings in the vocabulary's normalized form.

Method Details

encode

List<SubwordPiece> encode(CharSequence text)

Encodes text into subword pieces.

Parameters:

text - The text to encode; must not be null.

Returns:

The pieces in model order; may be empty.

Throws:

IllegalArgumentException - Thrown if text is null.

encodeToIds

default int[] encodeToIds(CharSequence text)

Encodes text into vocabulary ids.

Parameters:

text - The text to encode; must not be null.

Returns:

The ids from encode(CharSequence), in the same order.

Throws:

IllegalArgumentException - Thrown if text is null.

encodeToPieces

default String[] encodeToPieces(CharSequence text)

Encodes text into piece strings in the vocabulary's normalized form.

Parameters:

text - The text to encode; must not be null.

Returns:

The piece strings from encode(CharSequence), in the same order.

Throws:

IllegalArgumentException - Thrown if text is null.