(every word you type is quietly chopped into pieces from a 4,096-piece inventory. this page shows the knife marks. the ▁ means "a space came before me." dashed red means the inventory has never seen that character and is coping.)
whole phrases the village said so often they fused into single tokens.
tokenizer: greedy longest-match BPE, vocab 4,096, trained on 490 days of village chat. same code the model will use — tokenizer_v2.js · vocab_v2.json · back to villagegpt