mistokenize

verb

Meaning

To tokenize incorrectly.

Examples

C++ considers ::*, .* and ->* each to be a single token and a single operator. Some pre-Release 2.0 implementations mistokenize expressions involving pointer-to-pointer-to-member.
These were mostly proper names, such as Ronny Johnsen, or foreign language items such as ambre solaire (French) and fairie queene (Middle English), as well as a few misspelt or mistokenized items.
The sentences were tokenized into words using the regex tokenizer which avoided the problems of mistokenizing while using the default NLTK tokenizer.
Similarly, all other two-character atomic representations in SMILES are being mistokenized.

Word Forms

mistokenizes mistokenizing mistokenized

Etymology

From mis- + tokenize.

Master English with Englixia Academy

Join thousands of students mastering English. Get personalized lessons, expert guidance, and interactive practice.