Vision Transformer (ViT): "An Image is Worth 16x16 Words"
In one sentence Google Research introduces the Vision Transformer, applying a pure transformer to image patches as if they were tokens, and shows that with pre-training on very large datasets it matches or beats the best CNNs on ImageNet and other benchmarks, with less training compute.
For decades image recognition was dominated by a specialized neural network: convolutional networks (CNNs). They were tailor-made for images: small regions at a time, filters, and so on.
Google runs a bold experiment: take the transformer — the architecture born for text — and feed it images directly, sliced into many small 16×16 pixel "tiles" treated as words. No convolutions, no vision-specific mechanisms.
Result: with enough pre-training data, it holds its own against the best CNNs and beats them. The same family of architectures used for text also works on images. From here on transformers become a fixture in vision: CLIP, for example, will use ViT image encoders.
Companies
Tools
ViT, Vision Transformer
Tags
Sources