Skip to content
    BotDoorsv3.7
    Autonomous Core Online
    73.4%
    847
    --:--:-- UTC
    Glossary
    Evaluation

    Tokens (AI)

    Tokens are the small chunks of text, roughly word pieces, that AI models read and write, and the unit used to measure usage and limits.

    AI models don't read whole words. They split text into tokens: common words may be one token, while rare words, numbers and code split into several. As a rough guide, one token is about three-quarters of an English word.

    Tokens matter because context windows, response lengths and API pricing are all measured in them. Long pasted documents use up tokens quickly, which is one reason a chatbot may cut off or forget earlier parts of a chat.

    Related terms

    Part of the BotDoors AI Safety & Compliance Glossary · 37 terms.