✂️

tokenization

noun

ˌtoʊ.kə.nɪˈzeɪ.ʃən

Meaning

The process of breaking text into smaller units like words, phrases, or symbols for analysis.

NLP uses tokenization to break text into smaller units.

segmentation, fragmentation, parsing

aggregation, whole text, non-division

a written receipt acknowledging payment or discharge of a debt

a sweet substance made from plants or produced synthetically

A moral or legal entitlement to have or obtain something or to act in a certain way.

a mythical creature that sucks blood from the living; metaphorically, a person who exploits others

The management or organization of a business, institution, or government.

A significant physical or mental transformation.