Tokenization (BPE).

🔧Build it in 5 lines

pip install tiktoken
python3 - <<'PY'
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
for s in ["hello world", "Hello World", " hello", "مرحبا بالعالم", "strawberry", '{"user_id": 12345}']:
    ids = enc.encode(s)
    print(len(ids), [enc.decode([i]) for i in ids])
PY

Run it. Five lines, and each one teaches something:

1️⃣ "hello world" → 2 tokens. Roughly 1 token ≈ 4 chars of English. Your 3800-char lesson ≈ 950 tokens.

2️⃣ "Hello World" → also 2, but DIFFERENT ids than "hello world". Casing changes the tokens. So does the leading space: "hello" and " hello" are two separate tokens. That's why prompts break in weird ways when you strip whitespace.

3️⃣ Arabic "مرحبا بالعالم" → ~10 tokens for 13 characters. Non-English text costs 2–4x more tokens per character than English. Same content, same model, triple the bill — and it eats your context window faster.

4️⃣ "strawberry" → ["str","aw","berry"]. The model never sees the letter r. Asking it to count letters is asking someone to count brushstrokes in a JPEG. This isn't stupidity, it's the input representation.

5️⃣ JSON: {"user_id": 12345} → the number splits into chunks like "123","45". Digits tokenize inconsistently, which is a real part of why arithmetic on long numbers is shaky, and why JSON/XML overhead is ~15–20% of your prompt before any content.

⚠️Where practitioners actually get burned

• Chunking RAG documents by characters (text[i:i+1000]) instead of tokens. Fine for English, blows past your embedding model's limit on Arabic/Chinese/code.
• Sizing context by len(text). Count tokens or you'll get a mid-sentence truncation in production.
• max_tokens is OUTPUT only, and it's tokens not words. max_tokens=100 ≈ 75 English words ≈ 35 Arabic words.
• Assuming token counts transfer between models. Different tokenizer = different count. Re-measure when you switch.
• Whitespace and indentation in code prompts are real tokens. Minifying a payload genuinely saves money.

🧠The mental model — and where it breaks

"Tokens are word pieces" gets you 80% there. The break: tokens are learned by frequency compression on the training corpus, not by linguistics. "unbelievable" may be 1 token while "Hamza" is 3. Rare names, UUIDs, base64 blobs and minified JS explode into near-per-character tokens. Ever wonder why pasting a UUID-heavy log costs so much? Now you know.

🎯Your exercise

Take the largest prompt template in a project you're working on. Count its tokens with tiktoken. Then answer two things:

(a) What percentage is structural overhead — JSON braces, markdown, indentation — versus actual content?

(b) If you translated that prompt to Arabic, what happens to the number?

Bonus: encode "1234567890" and "1,234,567,890". Which is more tokens, and does that change how you'd format numbers you want the model to do math on?