PyTorch 2.15 brings FP8 and much faster MoE decoding to the Mac.
PyTorch 2.15 makes the Mac a serious PyTorch machine: FP8, much faster MoE decoding on an M3 Ultra, and video decodes that fit in memory
The headline is mixture-of-experts decoding. On an M3 Ultra with torch.compile, Hugging Face measured Qwen3.6-35B-A3B going from 9.4 to 61.3 tokens per second versus PyTorch 2.14, and LFM2.5-8B-A1B from 23.6 to 123.6. The gain comes from doing the per-expert matrix math natively on the GPU instead of looping on the CPU, a dedicated decoding kernel, fewer GPU-to-CPU syncs, and faster expert routing. Dense models get smaller wins from faster single-token linear layers and a vocabulary softmax that is about 8× faster.
FP8 is the second change. PyTorch 2.15 adds 8-bit floating-point tensors on the Mac’s GPU, half the memory of BF16, with scaled matrix multiplies that run there directly. Together with the steady move off Apple’s MPSGraph onto hand-written Metal kernels, the Mac starts to look like a normal PyTorch device rather than a list of unsupported operations.
The image and video fixes may matter most to ComfyUI users. Decoding video with Wan2.2’s tiled VAE used to push driver memory to about 270 GiB; it now peaks around 8 GiB. And the Qwen-Image-2.1 VAE encoder, which silently corrupted images on the Mac in 2.14 (reported by ComfyUI users), now matches the CPU result.
Cautions. Every number is project-reported by the Hugging Face contributors who wrote the code, on one M3 Ultra setup, and the same post’s 2.16 preview lists a lower figure for the same Qwen model under a different test setup, so treat the speedups as directional. PyTorch 2.15 needs macOS 15 and Python 3.11 or newer. What to do: try the release candidate in a separate environment on the models you actually run, and wait for the final release before upgrading anything you depend on.
Sources: Hugging Face blog · PyPI (2.14.1 is latest final)
Databricks says it is expanding its New York City presence as an AI talent hub. Databricks
NVIDIA published an essay on Germany’s ambitions in industrial AI and how it plans to help. NVIDIA blog
Model & runtime releases
| Item | Date | Badge |
|---|---|---|
| PyTorch 2.15 release candidate: MPS FP8, faster decoding, Metal kernels | Oct 20 | Watch |
| TorchTPU open source + vLLM / SGLang TPU backends | Oct 20 | Open |
| llm-d v1.0.0 | Oct 19 | Open |
llama.cpp b11744 --connect · b11755 Metal 9-expert MoE fix | Oct 19–20 | Open |
| mlx-vlm v0.7.9 (Nemotron 3.5 Super VL, FLUX.3, GRPO) | Oct 19 | Open |
| Transformers v5.20.2 (multimodal generate fix) | Oct 19 | Open |
| OpenAI MultiLangBench (Apache 2.0) | Oct 19 | Open |
| Cloudflare Clairic extraction model (Workers AI preview) | Oct 19 | Cloud |
| MiniMax M3.1 Flash: cloud on Ollama; “open-weight” but no weights on HF yet | Oct 19 | Watch |
| Grok 4.7 on Amazon Bedrock in GovCloud (US) | Oct 19 | Cloud |
| TorchNeuron: native PyTorch backend for AWS Trainium | Oct 19 | Cloud |
| Tencent Hy-Memory-30B-A3B GGUF · mlx-community Ming-3.0 tiny speech builds | Oct 19 | On-device |
Security / policy / access
| Item | Date | Badge |
|---|---|---|
| vLLM: nine advisories published (two high), all fixed in 0.32.0 | Oct 20 | Active |
| vLLM 0.32.0 notes still cite GHSA-4hhp for the chat-template fix; the advisory is GHSA-542m | Oct 20 | Watch |
| Lightning AI credential incident: rotate secrets stored in Studios | Oct 15–16 | Active |
llama-server --connect P2P tunnel: off by default, per-run 40-character code | Oct 19 | Watch |
| Cloudflare Clairic: open weights “soon” | Oct 19 | Watch |
| Claude platform cyber verification page unchanged since Oct 15 | Oct 15 | Quiet |
| No new llama.cpp / Ollama / MLX / SGLang advisories | Oct 19–20 | Quiet |
Critical
Running vLLM older than 0.32.0 with image or audio models, or with clients you don’t control? Upgrade now and set allowed origins on purpose. Used Lightning AI Studios? Rotate every secret that lived there, GitHub tokens included.
Watch
PyTorch 2.15 final (release candidate today). Anthropic says agents pushed its monorepo from about 2,500 to over 30,000 merges a day and off git onto an in-house system; CI and source control are the next bottleneck for heavy agent shops. Still waiting: an MLX build of Beam, Reflection’s distillation recipes, Dory runtime support, Nemotron 3.5 Bolt E4B (Oct 23), and MiniMax M3.1 Flash weights.
Opportunity
Hugging Face’s PyTorch 2.15 numbers come from one M3 Ultra setup. Rerunning them on your own Studio or MacBook Pro, against MLX on the same model, is a benchmark nobody has posted yet. Same for MultiLangBench on local models.
Test the PyTorch 2.15 release candidate on your Mac
In a fresh environment: uv venv --python 3.11 pt215 && source pt215/bin/activate, then uv pip install torch==2.15.0 --index-url https://download.pytorch.org/whl/test/cpu and python -c "import torch; print(torch.__version__, torch.backends.mps.is_available())". Keep your 2.14 environment and time the same MoE model in both. Needs macOS 15+.
Find the local servers that answer any website
For each model server on your machine, run curl -si -H "Origin: https://example.com" http://localhost:8000/v1/models | grep -i access-control-allow-origin (swap in each port: vLLM defaults to 8000, llama-server to 9931 on builds since Oct 9, 8080 before that, Ollama to 11434). A * or an echoed origin means any page you visit can call it. Fix it with an origin list and an API key. Corrected Oct 21: llama-server’s default port.
Audit your prompt cache in ten minutes
Log usage.cache_read_input_tokens and usage.cache_creation_input_tokens on the first Claude API call of each agent session. If first-call reads sit near zero, move anything that changes per run, like dates, paths, and dynamic tool lists, after your cache breakpoint, then measure again.
Monday morning through Tuesday morning (Oct 19–20 ET). ASTGL / Fraser takes, sourced.
The Mac stops being PyTorch’s afterthought
For years the honest answer to “can I do this in PyTorch on my Mac?” was “sort of, then rent a GPU.” PyTorch 2.15 is the first release where the Mac story is about speed instead of missing operations. A sixfold jump in MoE decoding is the kind of number that changes which tool you reach for. My take: if you wrote off MPS and moved everything to MLX or llama.cpp, re-test. That goes double for image and video pipelines in ComfyUI, which live in PyTorch anyway. For serving LLMs, MLX and llama.cpp still own the quantization story. And remember these are release-candidate numbers, measured by the people who wrote the kernels.
Sources: Hugging Face blog (Oct 20)
Nine vLLM advisories, one upgrade
Early this morning vLLM published nine security advisories, all fixed in 0.32.0. The two rated high: a disguised EPS image that reaches the host’s Ghostscript, a known code-execution path on affected hosts, and forged shared-memory handles that reach Python’s pickle. The rest are one-request crashes or stalls, from a chat template under 100 bytes that freezes every tenant to an ordinary-looking image that kills DeepSeek-OCR serving. The two IDs that were missing yesterday are now public. My take: if you serve multimodal models or let anyone you don’t trust call your endpoint, 0.32.0 isn’t optional. Read its CORS breaking change first. One wrinkle remains: the release notes still cite an older, unrelated advisory for the chat-template fix; the real one is GHSA-542m.
Sources: vLLM security advisories · GHSA-h3rc · GHSA-542m · vLLM v0.32.0
llama.cpp will put your Studio on the internet, if you ask
Monday evening llama.cpp merged an experimental
--connectflag for llama-server. It opens a peer-to-peer WebRTC tunnel so a browser outside your network can reach the server, with no port forwarding; a 40-character code, new each run, is the key. The pull request also says about 90% of its code was AI-generated. My take: this is genuinely handy for reaching a home Studio from your phone. It is also an exposure decision. Treat the code like a password, set--api-keyas a second lock, and shut it off when you’re done. For anything permanent, I’d still put the server behind a VPN I already trust.Sources: llama.cpp #28277 · b11744
Cloudflare’s 68% is a prompt-cache lesson, not a model lesson
Cloudflare cut its internal AI code reviewer from $2.43 to $0.77 a review. The best detail: the first call of each session hit the prompt cache only 14–26% of the time, because the shared agent instructions sat in the same cached block as the working directory and the date. Splitting that block fixed it. My take: go look at where your cache breakpoint sits today. Anything that changes per run, like timestamps, paths, or a dynamic list of skills, placed before it makes every session pay full price. Cheap fix, real money. Figures are Cloudflare-reported about its own system.
Sources: Cloudflare blog (Oct 19)
OpenAI wants your COBOL, and opened the test
OpenAI says GPT-6 Astra passes 87.4% of a new benchmark that has models build a SQLite-compatible database in 20 languages, up from 26.8% for GPT-5.6 Sol, and that GPT-6.1 Sol migrates COBOL with 86.6% accuracy at low effort. Every score is OpenAI-reported on OpenAI-built tests, and the COBOL tests are hidden. My take: the scores are marketing until someone else reproduces them. The useful part is that MultiLangBench itself is on GitHub under Apache 2.0, so you can point it at your own local models and see where they fall over.
Sources: OpenAI (Oct 19) · MultiLangBench
Google wants PyTorch shops on TPUs without a rewrite
At PyTorch Conference this morning Google open-sourced TorchTPU, its native PyTorch stack for TPUs, with vLLM and SGLang backends, and shipped llm-d 1.0, the Kubernetes serving stack it co-founded with Red Hat and IBM. My take: TorchTPU goes after the old objection that TPUs meant rewriting in JAX. If it holds up, that excuse is gone. For most of us the more useful piece is llm-d: cache-aware routing and KV-cache offload for your own vLLM or SGLang fleet, on any Kubernetes. Speedups in both posts are Google-reported.
Sources: Google Cloud: TorchTPU · Google Cloud: llm-d 1.0 · llm-d v1.0.0