AI Engineering Digest

llama.cpp v0.6.0 turns a month of tips into GA.

Tuesday, October 6, 2026

Issue 32 · sourced briefing

Sources: llama.cpp · vLLM · OpenAI · AWS · mlx-vlm · Google · Apple ML Research · Ollama

Lead Story

llama.cpp v0.6.0 makes the Metal and decision-model work official

Just before 1 PM ET on Monday Oct 5, llama.cpp cut v0.6.0, its first GA since v0.5.0. The Apple GPU speculative-decoding fix, the tensor-API flash-attention kernel, and the /v1/systemone decision-model API that shipped as tip builds last week now sit in a release you can pin.

New in the release: support for Z.ai’s GLM-5.3-Flash (a 320B text-and-vision hybrid MoE), Clef decision models with both text and vision, Nimble, and Qwen4Exp with multi-token-prediction speculative decoding. There is also a new batch API that mixes token and embedding inputs, a reworked web UI that downloads models from the Hugging Face Hub and estimates whether they fit in memory, and ggml 0.26.0 underneath. The project says the new few-row Metal kernels make those matrix multiplies up to about 3× faster on Apple GPUs, and that Qwen4Exp decodes about 1.5× faster on DGX Spark with MTP. Both figures are project-reported.

Why Mac owners should care: last week we said to retest speculative decoding on tip builds but keep v0.5.0 pinned for rollback. With v0.6.0 the fix is in a GA release, so moving the pin is the next step. On the server side, the model list now reports each model’s input and output modalities, embeddings accept image, audio, and video content, and requests that would truncate media partway are rejected.

Before you bump: this release changes the C API. The library header gains the extended batch API, and the saved-session and state-file format versions went up. Retest anything that links libllama or persists sessions. Tip builds kept landing overnight into Tuesday. Those are not GA.

Sources: llama.cpp v0.6.0 release (Oct 5) · ggml-org GLM-5.3-Flash GGUF

By the Numbers

v0.6.0

llama.cpp GA · first since v0.5.0 · Oct 5

~3×

Metal few-row mat-mul on Apple GPUs (project-reported)

vLLM advisories published Oct 6 · all fixed in 0.31.0

753B / ~40B

GLM 5.3 total / active params · now on Bedrock

Quiet lanes

Claude release notes: only the Oct 1 Models API line field since Sep 30 · xAI quiet since Sep 28 · DeepSeek since Sep 10 · Tencent: no new OSS model · Qwen: no new weights since Sep 20 · MLX 0.32.3 / mlx-lm 0.32.0 unchanged · Kolibri mlx-lm PR still open · Ollama 0.40 still RC · SGLang still 0.5.21 · vLLM 0.31.0 / Clef / Kolibri / ThinkingBox not re-led

Commentary

Monday morning after Issue 31 through Tuesday morning (Oct 5–6 ET). ASTGL / Fraser takes, sourced, not recap theater.

GA is the feature

Nothing in llama.cpp v0.6.0 is a surprise if you watch the tip builds. That is the point. Most people running local models on a Mac do not track daily builds, and they should not have to. A GA tag turns last week’s Metal speculative-decoding fix, the tensor-API flash-attention kernel, and the decision-model endpoint into something you can pin, package, and roll back to. If you held on v0.5.0 for stability, this is the release to move to. Do it on purpose, though: the C API and the saved-session format changed, and the speedups in the notes are the project’s own numbers.

Source: llama.cpp v0.6.0

vLLM’s health check was lying

Three vLLM advisories went public early Tuesday, all fixed in 0.31.0. The one to remember: a single 92-byte request to the raw generate endpoint could stall the engine while /health kept returning 200, so nothing restarted it. Another lets a client forge a multimodal cache ID and crash the whole engine with about 70 rejected requests and one follow-up. The third, rated High, lets /tokenize exhaust memory on MOSS-Audio deployments, and --api-key does not cover that route. My take: 0.31.0 is a security upgrade now, not a feature upgrade. Your liveness probe should actually generate a token, and only the routes you use should be reachable.

Sources: GHSA-4xqp · GHSA-p92p · GHSA-823j

Text watermarks are arriving. Do not build a detector business on them.

To meet the EU AI Act, OpenAI now lets API customers anywhere opt in to watermarked text on select models. It stays off by default. ChatGPT and Codex output in the EU gets an invisible watermark over the coming weeks, and the detector goes only to approved researchers. OpenAI’s own numbers explain the caution. At a 1% false-positive target it catches about 80% of 200-token passages and about 95% of 400-token ones. Swapping a quarter of the words for synonyms drops detection to 17%. All of that is OpenAI-reported. Anthropic already watermarks text from Claude Fable 5.1 and Mythos 5.1. If you serve EU users, decide your watermark setting per provider and write it down. Treat a detection result as a weak signal, never as proof.

Sources: OpenAI — EU text provenance (Oct 5) · Claude release notes (Sep 1)

One model family, two lanes

GLM 5.3 reached Amazon Bedrock Monday evening: Z.ai’s 753B-parameter MoE with about 40B active, offered to eligible enterprise customers with prompt caching, cross-Region profiles, and Flex or Priority tiers. Over the same weekend, its smaller sibling GLM-5.3-Flash (320B total, 18B active) got official GGUFs from ggml-org and support in a GA llama.cpp. That is a useful shape for operators: one prompt set, a managed lane for the big model, a local lane for the small one. Two caveats. The coding and cyber benchmark wins are Z.ai-reported, and AWS’s own launch post demos the model running a penetration-testing agent. That is dual-use, so scope it like one.

Sources: AWS What’s New · AWS ML blog · GLM-5.3-Flash GGUF

Decision models reach the MLX vision stack

mlx-vlm 0.7.5 shipped Monday afternoon with a shared, typed decision API, Decider and Laya on top of it, and a decision CLI. It also added conversation compaction to its Responses API server, and 0.7.6 followed within the hour to stream compaction progress and recover oversized requests. Put that next to Ollama’s 0.40 release candidates, which run decision models on MLX, and llama.cpp’s endpoint going GA. In one week, all three Apple Silicon stacks learned the classifier tier. You no longer need a wrapper to run a decision model natively on a Mac. These are days-old versions, so pin exact releases.

Sources: mlx-vlm v0.7.5 · v0.7.6 · Ollama v0.40.0-rc5

Today’s Stories

Security

Three more vLLM denial-of-service holes, all fixed in 0.31

Oct 6: vLLM published three advisories covering versions before 0.31.0. A forged multimodal cache ID can desync the frontend and engine caches and crash the whole engine. One oversized min_tokens value on the raw generate endpoint hangs the engine while health checks stay green. On MOSS-Audio deployments, /tokenize lets a caller grow memory without limit, and that route sits outside API-key protection. Two are rated Moderate and one High. The fix is to upgrade to 0.31.0.

Sources: GHSA-p92p · GHSA-4xqp · GHSA-823j

MLX

mlx-vlm adds a decision API and Responses compaction

Oct 5: mlx-vlm 0.7.5 adds a shared typed decision API with Decider and Laya models plus a decision CLI, conversation compaction in its Responses API, Qwen3-Omni multimodal batching, and fixes for prequantized Qwen4 loading, PhiMoE routing, and Qwen3-VL video timestamps. The 0.7.6 follow-up streams compaction progress and recovers oversized requests. MLX core and mlx-lm have no new releases. Kolibri support for mlx-lm is still an open PR.

Sources: v0.7.5 · v0.7.6

Local · On-device

A 4B Gemma that cleans up speaker labels, as a GGUF

Oct 4: Google researchers published DiarizationLM-Gemma-4-E4B-v1, a fine-tune of Gemma 4 E4B that corrects speaker turns and backchannels in ASR plus diarization transcripts. It ships 4-bit GGUFs of about 5 GB for llama.cpp and Ollama. The card reports lower word-diarization error than the older 8B Llama-based version on all four benchmarks it tests, at half the size. The results are author-reported, and the card says this is not an officially supported Google product.

Source: Model card

Enterprise · OSS models

GLM 5.3 on Bedrock; its Flash sibling runs in llama.cpp

Oct 3–5: Amazon Bedrock now serves Z.ai’s open-weight GLM 5.3 to eligible enterprise customers. It is a 753B MoE with about 40B active, a 1M-token context, always-on reasoning with selectable effort, explicit prompt caching, and US and Global cross-Region profiles, callable through OpenAI-compatible APIs. Separately, ggml-org posted GGUFs of GLM-5.3-Flash (320B, 18B active), which llama.cpp v0.6.0 supports. Benchmark claims are Z.ai-reported.

Sources: AWS WN · GGUF

Policy

OpenAI opens opt-in text watermarks for the API

Oct 5: In response to the EU AI Act, OpenAI lets API customers worldwide opt in to watermarked text on select models, off by default. ChatGPT and Codex output in the EU gets an invisible watermark in the coming weeks, and detector access is limited to approved researchers. OpenAI says detection weakens on short or heavily edited text and plans to open-source the technique. Detection and quality numbers are OpenAI-reported. Engineers serving EU users should record which providers watermark by default.

Source: OpenAI

Enterprise

Nova 2.5 Sonic is GA for voice agents

Oct 5: AWS made Amazon Nova 2.5 Sonic, its speech-to-speech model, generally available on Bedrock. AWS says reasoning, instruction following, and tool-calling accuracy are better and latency is lower, but gives no figures. The model keeps a 256K context, asynchronous tool calls, and mixed voice and text in one session. Strands Bidi Agents is also GA for building voice agents. It runs in four Regions at Nova 2 Sonic pricing.

Source: AWS WN

Security

AWS moves agent pentesting into the CI/CD pipeline

Oct 5: AWS Continuum for Penetration Testing (formerly AWS Security Agent) can now run as a deploy-time step in public preview. It reports findings with severity, affected endpoints, and fixes in the pipeline output, and re-tests automatically once you fix them. AWS says the pasted pipeline snippet takes under five minutes to set up and needs no prior full pentest. Treat it as a preview gate, not a replacement for scoped manual testing.

Source: AWS WN

ML practice

Apple paper: elaborate agent harnesses add little for ML engineering

Oct 1: An Apple Machine Learning Research paper with EPFL authors finds that, on the same time budget and the same frontier model, open-source multi-agent harnesses for autonomous ML engineering do no better than one session of a minimal coding agent with read, write, and bash tools. The authors conclude the backbone model drives the results. These are author findings on current MLE benchmarks, but they are a good reason to benchmark a plain agent before adding orchestration.

Source: Apple ML Research

Other news in AI

Other

Google backs 890 MW of nuclear uprates with Constellation

Oct 6: Google signed a power purchase agreement to finance upgrades at six Constellation nuclear plants. The deal adds 890 MW to the PJM grid before the end of 2032, and Constellation will use Gemini Enterprise in plant operations. It is data-center energy news with no change to any developer surface.

Source: Google

Signal Tables

Model & runtime releases

Model / package	Org	Notes
llama.cpp v0.6.0 Oct 5 · GA	ggml	GA Metal kernels · /v1/systemone · GLM-5.3-Flash · Clef vision · batch API
vLLM v0.31.0 Oct 5	vLLM	GA now also the fix for three Oct 6 advisories
mlx-vlm 0.7.5 / 0.7.6 Oct 5	Blaizzy / mlx-vlm	GA decision API · Responses compaction · Qwen3-Omni batching
Ollama v0.40.0-rc5 Oct 5 evening	Ollama	Watch MLX default on Apple Silicon · still pre-release
GLM 5.3 Bedrock · Oct 5	Z.ai / AWS	Cloud 753B / ~40B active · eligible enterprise customers
GLM-5.3-Flash GGUF Oct 3	ggml-org	Open 320B / 18B active · Q4_K / Q2_K · MTP sidecars
DiarizationLM-Gemma-4-E4B Oct 4	Google researchers	Open 4B · GGUF ~5 GB · ASR speaker-label cleanup
Nova 2.5 Sonic Oct 5	AWS	Voice speech-to-speech GA · 4 Regions
MLX / mlx-lm unchanged	Apple ML	Watch 0.32.3 / 0.32.0 · Metal fixes on main, unreleased

Security / policy / access

Item	Beat	Status
vLLM GHSA-823j High · Oct 6	MOSS-Audio `/tokenize` memory exhaustion · outside API-key routes	Active
vLLM GHSA-p92p / 4xqp Moderate · Oct 6	Forged mm cache ID crash · engine hang with green `/health`	Active
OpenAI text watermark Oct 5	API opt-in, off by default · EU ChatGPT/Codex in coming weeks	Active
OpenAI HIPAA BAA flow Oct 5	Self-serve BAA for eligible API orgs	Active
AWS Continuum in CI/CD Oct 5	Deploy-time pentesting · public preview	Watch
Sonnet 4.5 retirement Claude API · Nov 30	Migrate to Sonnet 5.5 · partner clouds set own schedules	Watch
NIST NCCoE Agentic webinar Oct 28 · comments Nov 9	DevSecOps / agentic AI scoping	Watch

Industry Shorts

Critical · vLLM upgrade · llama.cpp API changes

Upgrade vLLM to 0.31.0: three Oct 6 advisories affect older versions. llama.cpp v0.6.0 changes the C API and bumps saved-session formats, so test before you swap pins. Ollama 0.40 is still a release candidate.

Watch · EU watermark rollout · MLX releases

OpenAI’s EU watermark for ChatGPT and Codex rolls out over the coming weeks, with cloud partners after that. MLX has Metal fixes on main (a residency refresh setting, M1 quantized-matmul tuning) that are not in a release yet. Kolibri’s mlx-lm PR is still open.

Opportunity · HIPAA self-serve · model grouping

Eligible OpenAI API orgs can now accept a BAA in organization settings. Claude’s Models API now returns a line field (Oct 1), so you can group models without parsing IDs. GLM 5.3 on Bedrock supports explicit cache breakpoints of 1,024 tokens or more.

Try This

Move your Studio pin to llama.cpp v0.6.0

Download the macOS Apple Silicon build of v0.6.0 and serve a model you already use. Check that the model list now shows its modalities, then rerun the same prompt set with and without your draft-model setup against v0.5.0. Keep the old binary until your numbers agree.

llama.cpp Metal Upgrade

Make your vLLM liveness probe generate

Replace a /health-only probe with a one-token completion and a short timeout, so a stalled engine actually restarts. At your proxy, block /tokenize and the raw generate routes unless a client needs them. Then upgrade to 0.31.0.

vLLM Security Ops

Clean up meeting transcripts locally

Run the 4-bit DiarizationLM GGUF with the llama-cli command on its model card against one of your own speaker-tagged ASR transcripts. Compare speaker turns before and after. It is small enough for any recent Mac, and nothing leaves the machine.

Gemma 4 GGUF Speech

Issue 32 · Tuesday, October 6, 2026 · Not investment advice · Sources cited inline · Window: after Issue 31 (Mon Oct 5 morning ET) through Tue Oct 6 morning ET · Issue 31 vLLM 0.31.0 / Ollama 0.35.1 / Kolibri / ThinkingBox / DGX Spark not re-led · Issue 30 distillation / SGLang 0.5.21 not re-led · Ollama 0.40 RC not presented as GA · llama.cpp tips after v0.6.0 are not GA · Listen omitted (episode not published)