Blazingly fast LLM inference engine written in Rust — run Llama, Mistral, Qwen and GGUF models on CPU, CUDA, Metal and WebGPU with an OpenAI-compatible API.
mistral.rs
AI Tools
AI Tools
Blazingly fast LLM inference engine written in Rust — run Llama, Mistral, Qwen and GGUF models on CPU, CUDA, Metal and WebGPU with an OpenAI-compatible API.