gpt4all
GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.
Local LLMs & Inference77k stars
Local LLMs & Inference
71 open-source LLM inference tools, ranked by GitHub stars. Most popular: gpt4all, ray and gitleaks. Inference engines and local runners serve open models on your own hardware, from laptops to GPU clusters.
Showing 1–60 of 71 tools
ray
Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for…
Local LLMs & Inference44k stars
gitleaks
Find secrets with Gitleaks 🔑
Local LLMs & Inference30k stars
llm-action
本项目旨在分享大模型相关技术原理以及实战经验（大模型工程化、大模型应用落地）
Local LLMs & Inference25k stars
litgpt
20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.
Local LLMs & Inference14k stars
OpenLLM
Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.
Local LLMs & Inference13k stars
openvino
OpenVINO™ is an open source toolkit for optimizing and deploying AI inference
Local LLMs & Inference11k stars
PowerInfer
High-speed Large Language Model Serving for Local Deployment
Local LLMs & Inference9.8k stars
kimi-k3-in-c
A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no…
Local LLMs & Inference9.1k stars
BentoML
The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model…
Local LLMs & Inference8.9k stars
dynamo
A Datacenter Scale Distributed Inference Serving Framework
Local LLMs & Inference8.3k stars
lmdeploy
LMDeploy is a toolkit for compressing, deploying, and serving LLMs.
Local LLMs & Inference8.1k stars
openevolve
Open-source implementation of AlphaEvolve
Local LLMs & Inference7.6k stars
plano
Plano is an AI-native proxy server and data plane for agentic apps. Smart LLM routing, observability, agent…
Local LLMs & Inference7.1k stars
turbo-fieldfare
Gemma 4 26B-A4B inference in ~2 GB of RAM on any M-series MacBook
Local LLMs & Inference6.9k stars
ai-infra-book
《深入理解 AI Infra：量化分析与系统设计》（李博杰 著）开源书稿：从硬件约束和模型架构出发，量化推导 LLM 推理与训练系统设计。含全书正文、PDF、配套计算工具与实验
Local LLMs & Inference6.7k stars
flashinfer
FlashInfer: Kernel Library for LLM Serving
Local LLMs & Inference6.7k stars
kserve
Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework…
Local LLMs & Inference6.1k stars
cactus
Quantization, kernels, runtime and inference engine for mobiles, wearables, smart home and robots.
Local LLMs & Inference6.1k stars
shimmy
⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No…
Local LLMs & Inference6k stars
lemonade
Lemonade helps users discover and run local AI apps by serving optimized LLMs right from their own GPUs and…
Local LLMs & Inference5.9k stars
gpustack
A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU…
Local LLMs & Inference5.8k stars
Awesome-LLM-Inference
📚A curated list of Awesome LLM/VLM Inference Papers with Codes: Flash-Attention, Paged-Attention, WINT8/4…
Local LLMs & Inference5.5k stars
superduper
Superduper: End-to-end framework for building custom AI applications and agents.
Local LLMs & Inference5.3k stars
eko
Eko (Eko Keeps Operating) - Build Production-ready Agentic Workflow with Natural Language - eko.fellou.ai
Local LLMs & Inference5k stars
RuVector
RuVector provides High Performance, Real-Time decisions and agent memory , Self-Learning Ai, Vector GNN DB…
Local LLMs & Inference4.6k stars
optillm
Optimizing inference proxy for LLMs
Local LLMs & Inference4.3k stars
GenerativeAIExamples
Generative AI reference workflows optimized for accelerated infrastructure and microservice architecture.
Local LLMs & Inference4.2k stars
Rapid-MLX
Rapid-MLX is an open-source (Apache 2.0) OpenAI- and Anthropic-compatible LLM inference server for Apple…
Local LLMs & Inference4k stars
AI-Engineer-Headquarters
A collection of scientific methods, processes, algorithms, and systems to build stories & models.
Local LLMs & Inference3.8k stars
ai-engineering-interview-questions
Your Cheat Sheet for AI Engineering Interview – Questions and Answers.
Local LLMs & Inference3.5k stars
distributed-llama
Distributed LLM inference. Connect home devices into a powerful cluster to accelerate LLM inference. More…
Local LLMs & Inference3.1k stars
spiceai
Add a real-time analytics node to your operational database. Spice is a portable, accelerated SQL query…
Local LLMs & Inference3.1k stars
Medusa
Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads
Local LLMs & Inference2.8k stars
MTPLX
The fastest way to run Qwen 3.8 Flash Next, Qwen 3.8 27B and Ternary Bonsai 2 27B on a Mac: 125 tok/s in…
Local LLMs & Inference2.6k stars
EAGLE
Official Implementation of EAGLE-1 (ICML'24), EAGLE-2 (EMNLP'24), and EAGLE-3 (NeurIPS'25).
Local LLMs & Inference2.6k stars
nanocoder
An open coding agent for your terminal, built by a community collective rather than a company. Bring your…
Local LLMs & Inference2.5k stars
neuron-ai
The Agentic Framework of the PHP ecosystem to build production-ready AI driven applications. Connect…
Local LLMs & Inference2.1k stars
aici
AICI: Prompts as (Wasm) Programs
Local LLMs & Inference2.1k stars
HyperQwen
Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127…
Local LLMs & Inference2.1k stars
llama2-webui
Run any Llama 2 locally with gradio UI on GPU or CPU from anywhere (Linux/Windows/Mac). Use `llama2-wrapper`…
Local LLMs & Inference1.9k stars
InferenceX
Open Source AI Accelerator Research Platform Standard / 开源推理研究平台
Local LLMs & Inference1.9k stars
aigrantsindia
The non profit fostering AI in India through credits, grants, resources
Local LLMs & Inference1.8k stars
beta9
Ultrafast serverless GPU inference, sandboxes, and background jobs
Local LLMs & Inference1.8k stars
kvcached
Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond
Local LLMs & Inference1.8k stars
Atomic-Chat
Local AI app and inference engine for agents. Run open-weight LLMs locally — private, 100% offline on your…
Local LLMs & Inference1.8k stars
react-native-executorch
High-performance, privacy-first on-device AI inference library for React Native, powered by PyTorch's…
Local LLMs & Inference1.8k stars
ai-engineering-interview-questions-company-wise
Your Cheat Sheet For AI Engineering Interviews at Top AI Companies - Questions and Answers.
Local LLMs & Inference1.8k stars
xllm
A high-performance inference engine for LLM, VLM, DiT and REC models, optimized for diverse AI accelerators.…
Local LLMs & Inference1.6k stars
splash
A local inference engine for Apple silicon, built around the model.
Local LLMs & Inference1.5k stars
zero-to-sglang
Official SGLang x Datawhale course on LLM inference: understand inference, build a mini-sglang from scratch…
Local LLMs & Inference1.5k stars
scaling-book
Home for "How To Scale Your Model", a short blog-style textbook about scaling LLMs on TPUs
Local LLMs & Inference1.5k stars
awesome-ai-web-search
List of software that allows searching the web with the assistance of AI…
Local LLMs & Inference1.5k stars
BrowserAI
Run local LLMs like llama, deepseek-distill, kokoro and more inside your browser
Local LLMs & Inference1.4k stars
awesome-hacking-lists
A curated collection of top-tier penetration testing tools and productivity utilities across multiple…
Local LLMs & Inference1.4k stars
TensorFold
LLM Inference Engine for Metal and CUDA.
Local LLMs & Inference1.4k stars
jev
jevos: an open-source alternative to Jev and Microsoft-Decision-1. Yes/no decisions, choices and scores with…
Local LLMs & Inference1.4k stars
LeanCopilot
LLMs as Copilots for Theorem Proving in Lean
Local LLMs & Inference1.3k stars
prompt-poet
Streamlines and simplifies prompt design for both developers and non-technical users with a low code approach.
Local LLMs & Inference1.2k stars
tiny-vllm
Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM
Local LLMs & Inference1.2k stars
More categories
AI Agents 73RAG & Search 59Coding Assistants 72Image & Video Generation 71Speech & Audio 68Chat UIs 53LLM Frameworks & MLOps 58