Before an AI reads a single word of your message, it first has to translate your text into numbers. It does this by breaking language into small chunks called tokens — roughly syllable-sized pieces of words, whole words, spaces, or punctuation marks. Think of it like how a child learns to read: first individual letters, then syllables, then full words. The AI never actually sees the word "unbelievable" — it sees something like "un" + "believ" + "able". Type anything below and hit Tokenize to watch it happen live.
Each colored block is one token. Notice how prefixes like "un–" and suffixes like "–ness" split off as their own pieces. Spaces often attach to the start of the next word — that underscore (_) shows where a space lives.
0
Characters
0
Words
0
Tokens
Hover over a token to see its mathematical ID mapping.
Why this matters
Every AI model has a hard limit on how many tokens it can process at once — called its context window. Run out of space and the AI starts "forgetting" the beginning of your conversation. Tokens also determine cost: most AI services charge per token, not per word. A longer, rambling prompt uses more tokens and gets a slower, pricier response. Understanding tokens helps you write tighter prompts and get better answers.