Monday, August 24, 2026
Completes KV-cache- and token-aware request routing in Ray Serve LLM: tokenization on the LLMRouter ingress replica, KV lifecycle events broadcast to every ingress replica, offloaded CPU KV blocks counted toward cache hits. Also experimental Ray Sandbox (gVisor), TorchTPU backend, TPU subslice gang scheduling, Data shuffle v2, bundled Serve LLM on vLLM 0.26.0, and an RCE fix in read_lance / nested pickle handling.
Full rewrite aimed at RL training against the same agent harness used in deployment. ~3,500-line codebase, a proxy so agents keep tools/context/control flow with zero agent-side changes, native Kubernetes Jobs. Coding-agent example: with 6K samples, an end-to-end Qwen3.5-9B workflow improves SWE-bench Verified from 41.8% to 56.4% (+14.6 points). vendor
710 PRs. Overlapped checkpoint staging during CUDA-graph capture (--startup-weight-load-mode overlap); Qwen3-32B on H100 starts 8.6–11.7% faster with prefetch, and 2.38× faster than the plain default: 35.6s vs 84.8s. Cookbooks add Intern-S2-Mobius, several diffusion models, Qwen3.8, Nemotron 3.5 Lightning, DeepSeek-V4-Pro-0813. Compiled-kernel caches live under SGLANG_CACHE_DIR. Remote media downloads bounded to 64 MiB by default. vendor
561 commits. Kimi K3 in one release: model files and kernels, Python and Rust frontends, AttnRes, DeepGEMM, compressed-tensors quantized checkpoints, DSpark AR fusion. Other new models include Qwen3.5 dense/MoE, K-EXAONE-2.0-750B-A37B, VaultGemma, jina-embeddings-v5-text-nano. Environment: PyTorch 2.13.0 / torchvision 0.28.0 / Triton 3.7.1.
Apache-2.0 router redesigned around a native Rust server and switchyard-libsy. Speaks OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages from one TOML config. Routing includes LLM task classification, signal-driven stage routing from coding-agent tool use, session affinity, and context-window eviction/reroute. GitHub notes still call it pre-alpha.
