Class WordpieceTokenizer

java.lang.Object

opennlp.tools.tokenize.WordpieceTokenizer

All Implemented Interfaces:

Tokenizer

public class WordpieceTokenizer
extends Object
implements Tokenizer

A Tokenizer implementation which performs tokenization using word pieces.

Adapted under MIT license from https://github.com/robrua/easy-bert.

Note that this tokenizer performs only the wordpiece (subword) stage of BERT tokenization. It does not normalize the input text: no lower casing, no accent stripping, no control character removal. Text that does not match the vocabulary's casing - for uncased models that includes every capitalized word - is mapped to the unknown token. Use WordpieceEncoder for the BERT normalization and wordpiece segmentation stages.

Runs of punctuation are split into individual tokens. A word that cannot be fully represented by vocabulary pieces becomes one unknown token. tokenizePos(String) is not supported.

For reference see:

• https://www.tensorflow.org/text/guide/subwords_tokenizer#applying_wordpiece
• https://cran.r-project.org/web/packages/wordpiece/vignettes/basic_usage.html

See Also:

• WordpieceEncoder

Field Summary

Fields

Modifier and Type	Field	Description
static final String	BERT_CLS_TOKEN	BERT classification token: [CLS].
static final String	BERT_SEP_TOKEN	BERT separator token: [SEP].
static final String	BERT_UNK_TOKEN	BERT unknown token: [UNK].
static final String	ROBERTA_CLS_TOKEN	RoBERTa classification token: <s>.
static final String	ROBERTA_SEP_TOKEN	RoBERTa separator token.
static final String	ROBERTA_UNK_TOKEN	RoBERTa unknown token.

Constructor Summary

Constructors

Constructor	Description
WordpieceTokenizer(Set<String> vocabulary)	Initializes a WordpieceTokenizer with a vocabulary and a default maximum token length of 100 Unicode code points.
WordpieceTokenizer(Set<String> vocabulary, int maxTokenLength)	Initializes a WordpieceTokenizer with a vocabulary and a custom maxTokenLength.
WordpieceTokenizer(Set<String> vocabulary, String classificationToken, String separatorToken, String unknownToken)	Initializes a WordpieceTokenizer with a vocabulary and custom special tokens.
WordpieceTokenizer(Set<String> vocabulary, String classificationToken, String separatorToken, String unknownToken, int maxTokenLength)	Initializes a WordpieceTokenizer with a vocabulary, custom special tokens and a custom maxTokenLength.

Method Summary

Modifier and Type	Method	Description
int	getMaxTokenLength()
String[]	tokenize(String text)	Splits a string into its atomic parts.
Span[]	tokenizePos(String text)	Not supported: wordpiece tokens (subwords, ## continuations and special tokens) have no faithful character spans in the original text.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

BERT_CLS_TOKEN

public static final String BERT_CLS_TOKEN

BERT classification token: [CLS].

See Also:

• Constant Field Values

BERT_SEP_TOKEN

public static final String BERT_SEP_TOKEN

BERT separator token: [SEP].

See Also:

• Constant Field Values

BERT_UNK_TOKEN

public static final String BERT_UNK_TOKEN

BERT unknown token: [UNK].

See Also:

• Constant Field Values

ROBERTA_CLS_TOKEN

public static final String ROBERTA_CLS_TOKEN

RoBERTa classification token: <s>.

See Also:

• Constant Field Values

ROBERTA_SEP_TOKEN

public static final String ROBERTA_SEP_TOKEN

RoBERTa separator token.

See Also:

• Constant Field Values

ROBERTA_UNK_TOKEN

public static final String ROBERTA_UNK_TOKEN

RoBERTa unknown token.

See Also:

• Constant Field Values

Constructor Details

WordpieceTokenizer

public WordpieceTokenizer(Set<String> vocabulary)

Initializes a WordpieceTokenizer with a vocabulary and a default maximum token length of 100 Unicode code points.

Parameters:

vocabulary - A set of tokens considered the vocabulary; must not be null or contain null or empty entries.

Throws:

IllegalArgumentException - Thrown if vocabulary is invalid.

WordpieceTokenizer

public WordpieceTokenizer(Set<String> vocabulary,
 int maxTokenLength)

Initializes a WordpieceTokenizer with a vocabulary and a custom maxTokenLength.

Parameters:

vocabulary - A set of tokens considered the vocabulary; must not be null or contain null or empty entries.

maxTokenLength - The non-negative maximum number of Unicode code points in one token.

Throws:

IllegalArgumentException - Thrown if vocabulary is invalid or maxTokenLength is negative.

WordpieceTokenizer

public WordpieceTokenizer(Set<String> vocabulary,
 String classificationToken,
 String separatorToken,
 String unknownToken)

Initializes a WordpieceTokenizer with a vocabulary and custom special tokens. This allows support for models like RoBERTa that use different special tokens instead of the BERT defaults.

Parameters:

vocabulary - The vocabulary; must not be null or contain null or empty entries.

classificationToken - The CLS token; must not be null or empty.

separatorToken - The SEP token; must not be null or empty.

unknownToken - The UNK token; must not be null or empty.

Throws:

IllegalArgumentException - Thrown if an argument is invalid.

WordpieceTokenizer

public WordpieceTokenizer(Set<String> vocabulary,
 String classificationToken,
 String separatorToken,
 String unknownToken,
 int maxTokenLength)

Initializes a WordpieceTokenizer with a vocabulary, custom special tokens and a custom maxTokenLength.

Parameters:

vocabulary - The vocabulary; must not be null or contain null or empty entries.

classificationToken - The CLS token; must not be null or empty.

separatorToken - The SEP token; must not be null or empty.

unknownToken - The UNK token; must not be null or empty.

maxTokenLength - The non-negative maximum number of Unicode code points in one token.

Throws:

IllegalArgumentException - Thrown if an argument is invalid.

Method Details

tokenizePos

public Span[] tokenizePos(String text)

Not supported: wordpiece tokens (subwords, ## continuations and special tokens) have no faithful character spans in the original text.

Specified by:

tokenizePos in interface Tokenizer

Parameters:

text - The string to be tokenized.

Returns:

The spans (offsets into s) for each token as the individuals array elements.

Throws:

UnsupportedOperationException - Always.

tokenize

public String[] tokenize(String text)

Splits a string into its atomic parts.

Specified by:

tokenize in interface Tokenizer

Parameters:

text - The string to be tokenized.

Returns:

The String[] with the individual tokens as the array elements.

Throws:

IllegalArgumentException - Thrown if text is null.

getMaxTokenLength

public int getMaxTokenLength()

Returns:

The maximum token length.