Tokenization

Tokenization is the process of breaking text into smaller units – so-called tokens – that an AI model can process. A token can be a word, a word part, or even a single character. The number of tokens affects processing speed and cost for AI models. For German, tokenization is often less efficient than for English because words are longer and more complex.

Related Terms

• Large Language Model (LLM)
• Context Window

Sources: a16z AI Glossary · CNET AI Terms Glossary

← Back to AI Glossary