NVIDIA Triton Inference Server 2.0: the move to version 2 and the KFServing protocols
In one sentence NVIDIA releases Triton Inference Server 2.0.0 (NGC container 20.06): new HTTP/REST and gRPC protocols based on the KFServing v2 protocol, with a migration from V1 that requires significant changes.
Once an AI model is ready, the next challenge is running it in production for many users at the same time. NVIDIA provides a dedicated open-source server for this, Triton Inference Server.
Triton receives requests from multiple clients, can group them into batches to make better use of the GPU, and returns the results. It handles models from different frameworks (TensorRT, TensorFlow, ONNX, PyTorch) without rewriting them.
With the 2.0.0 release at the end of June 2020, Triton moves to version 2: its network APIs are realigned to the inference protocol defined by the KFServing community, designed for the Kubernetes ecosystem. Moving from V1 to V2 requires significant changes for users.
Companies
NVIDIA
Tools
Triton Inference Server, TensorRT, ONNX Runtime, PyTorch, TensorFlow
Tags
Sources