llama.cpp

llama.cpp is the de facto standard for local LLM inference: a pure C++ rewrite of the LLaMA architecture with zero dependencies and single-file compilation. It is the underlying engine for almost every local inference tool (Ollama, LM Studio, and others). It pushes inference performance to the limit and runs on CPU, GPU, and NPU alike.

Core capabilities include the GGUF model format (now the de facto standard for open-weight distribution), multiple backends (CUDA, Metal, Vulkan, SYCL), KV cache quantization, Flash Attention, speculative decoding, and server / CLI / embedding invocation modes. Quantization support (Q4_K_M, Q5_K_M, etc.) is the most comprehensive among peers.

The highlight is that low-end hardware can still run 70B models, with excellent cross-platform portability. The downside is a low-level API: users must handle model download, quantization, and parameter tuning themselves, making the onboarding curve notably steeper than wrapper tools like Ollama.

Use cases: edge and embedded inference, self-built inference backends, providing the engine layer for higher-level applications. Those wanting out-of-the-box usage should reach for Ollama instead.