27 entries
EleutherAI releases The Pile, an 825 GB composite text dataset curated from 22 different sources (arXiv, GitHub, PubMed, books, StackExchange…), designed for pre-training large open-source language models.
DeepMind publishes MuZero in Nature: the RL agent learns world dynamics on its own and reaches superhuman performance on Go, chess, shogi, and 57 Atari games without being given the rules.
Google Research presents Big Bird at NeurIPS 2020, a transformer with sparse attention (local + global + random) that scales linearly, reaches SOTA on long-document QA and summarization, and proves Turing-completeness.
DeepMind announces that AlphaFold 2 has won the CASP14 competition with mean GDT >90, on par with experimental methods — widely regarded as solving the 50-year-old protein folding problem.
Microsoft announces a Bing-wide production deployment of Turing-NLR (next-gen NLP) models on Azure GPUs, described as the largest search-quality improvement ever.
DeepMind announces it has acquired MuJoCo, the physics simulator used in most RL and robotics research, and commits to making it free for everyone — a first step toward the full open-source release in 2022.
Google Research publishes mT5, a T5 variant pre-trained on mC4 (multilingual Common Crawl) over 101 languages, which becomes a standard baseline for many cross-lingual NLP tasks.
Google Research introduces the Vision Transformer, applying a pure transformer to image patches as if they were tokens, and shows that with enough pre-training it beats CNNs on ImageNet and other vision benchmarks.
Microsoft announces an exclusive license to integrate and redistribute GPT-3 in its products and cloud services, while OpenAI's public API keeps operating. The first major enterprise deal on foundation models.
Microsoft announces ZeRO Stage 3 in DeepSpeed: by sharding parameters across GPUs in addition to gradients and optimizer states, it enables training of 100B+ parameter models on reasonable-size clusters.
William Falcon and team ship PyTorch Lightning 1.0, a framework that separates research code (model) from engineering (training loop, distributed, checkpointing, logging) and becomes the de facto standard for many open projects.
Posting MLPerf Training 0.7 results, Google reveals TPU v4, a new custom deep-learning accelerator, claiming it built the "world's fastest training supercomputer" with a 4,096-chip pod.
Allen Institute for AI releases Longformer, a transformer that combines local sliding-window attention with global attention on special tokens, scaling linearly up to 4096 tokens and beating RoBERTa on long-document tasks.
HuggingFace releases Transformers 3.0 with the Rust-based tokenizers library (up to 100× faster), new NLP pipelines, and tighter Model Hub integration, cementing the de facto standard for using pretrained models in Python.
Connor Leahy, Sid Black, and Leo Gao found EleutherAI on Discord with the goal of replicating GPT-3 and releasing models, code, and datasets in the open, kicking off projects like GPT-Neo, GPT-J, and The Pile.
Facebook AI publishes wav2vec 2.0, a self-supervised model that learns representations from raw audio and reaches SOTA on LibriSpeech with as little as 10 minutes of labeled data.
OpenAI introduces Image GPT (iGPT), a transformer that treats pixels as tokens and shows that GPT-style sequential generative pretraining works on images too, reaching competitive performance on CIFAR-10.
Two weeks after the paper, OpenAI opens a private beta of the first general API for its language models, available to a few hundred developers building applications directly on top of GPT-3.
OpenAI publishes 'Language Models are Few-Shot Learners' and shows that at 175B parameters a model learns new tasks from a handful of examples in the prompt.
Lewis et al. at Facebook AI publish the RAG paper, combining a dense retriever (DPR) with a seq2seq generator (BART) to answer knowledge-intensive questions without baking all facts into the weights.
At GTC 2020 Jensen Huang announces the A100 GPU built on the Ampere architecture: 54 billion transistors, 40-80 GB HBM2e, TF32, 2:4 structured sparsity, and MIG support.
OpenAI releases Jukebox, a generative model that produces raw songs (audio + vocals + lyrics) conditioned on artist and genre, built on a stack of VQ-VAE and autoregressive transformers.
Facebook AI Research consolidates fairseq as the reference sequence-to-sequence framework: it adds modular support for BART, RoBERTa, mBART, wav2vec and becomes the primary codebase for FAIR's 2020 models.
Clark, Luong, Le, and Manning publish ELECTRA at ICLR 2020: instead of masked language modeling, it trains the model to detect tokens replaced by a small generator, matching BERT with a quarter of the compute.
Microsoft Research unveils Turing-NLG, the largest announced language model to date (17B), made possible by the DeepSpeed/ZeRO optimizer that drastically cuts GPU memory.
Google introduces Meena, a 2.6B-parameter conversational model trained on 341 GB of social dialogue, along with SSA, a new metric for evaluating chatbot quality.
Google Research presents Reformer, a transformer variant using LSH attention and reversible layers to go from O(n²) to O(n log n) and handle sequences up to 64k tokens.