Token

Models and Architectures

The small chunk of text - roughly a word or part of a word - that language models read and write. AI usage is often priced per token.

Language models do not process whole sentences or even whole words. Text is chopped into tokens - common words may be one token while longer or rarer words are split into pieces. As a rule of thumb, 1000 tokens is about 750 English words.Tokens matter practically: AI services bill per token, and every model has a maximum number of tokens it can handle at once, called its context window. Languages that are rare in training data often need more tokens per word, making them slower and costlier.