Skip to content
AImpact
IT EN
Inference Beginner Also known as: Tokenizzatore

Tokenizer

The component that turns text into tokens before passing it to the model and rebuilds text from output tokens.

ShareLinkedInX

In practice

Different tokenizers produce different counts: the same text may cost more tokens on GPT than on Claude or the other way around. Libraries like tiktoken (OpenAI) let you count tokens locally before calling the API.

Related terms

Token LLM

Seen in the wild

1 entries mentioning it
  1. HuggingFace Transformers 3.0: Rust tokenizers and the Model Hub
    High
← All terms