HuggingFace Transformers 3.0: Rust tokenizers and the Model Hub
In one sentence HuggingFace releases Transformers 3.0 with the Rust-based tokenizers library (up to 100× faster), new NLP pipelines, and tighter Model Hub integration, cementing the de facto standard for using pretrained models in Python.
When a developer wanted to try a model like BERT or GPT-2, there were ten different ways to do it, each with its own code, dependencies, and bugs. HuggingFace unified everything in one Python library, "transformers".
Version 3.0 is a quality leap: tokenizers (the part that splits text into chunks) are rewritten in Rust and become massively faster, ready-made pipelines let you do sentiment analysis or QA in three lines, and the Model Hub becomes the "DockerHub" of AI models.
From here on, if you want to use an AI model from Python, you almost always start with transformers.
Companies
HuggingFace
Tools
Transformers
Tags
Sources