llama.cpp learns to listen and talk.
llama.cpp’s server learns to listen and talk: live transcription and text-to-speech, next to the model you already serve
Transcription runs over a websocket. You stream raw microphone audio in and get text back as you speak, with partial results and a final transcript for each chunk. It works with models that have a streaming audio encoder; the pull request tested it with Mistral’s Voxtral Mini 4B Realtime, which the llama.cpp team publishes as an Apache 2.0 GGUF. Text-to-speech is a plain HTTP call that returns WAV or raw audio, can stream as it generates, and takes an optional reference recording to match a speaker. The docs cover Qwen3-TTS and Kyutai’s Pocket TTS.
Why it matters: the server many of us already run for chat now covers the ears and the mouth too. On a Studio, one llama.cpp install can transcribe you, answer with a local model, and speak the answer, with no audio leaving the machine. Router mode picks the right model for the websocket from a query parameter, so all three can sit behind one endpoint.
Cautions. Transcription is the only realtime piece; this is not speech-to-speech. The speech endpoint uses llama.cpp’s own request format; an OpenAI-compatible version and named voices are listed as follow-up work. Nobody has published Apple Silicon latency numbers for either endpoint yet. The TTS pull request discloses that most of its code was written by AI from the author’s plan. One more thing if you script against llama-server: its default port has been 9931, not 8080, since builds in early October.
Also overnight in llama.cpp: token log-probabilities now work with speculative decoding, and new builds add NVIDIA’s Nemotron 3.5 embedding models and H2O’s Lightning 4B decision model. Ollama shipped 0.40.7 the same night (card below).
Sources: llama.cpp #30636 (realtime) · #26603 (/tts) · server README · b11781
OpenAI acquired Orchestra Bio, a drug-development coordination platform, and is folding it into its Rosalind life-sciences effort. OpenAI
Google published its annual e-Conomy SEA report, projecting Southeast Asia’s digital economy at $368 billion. Google
Model & runtime releases
| Item | Date | Badge |
|---|---|---|
llama.cpp: /v1/realtime transcription (b11780) + /tts (b11781) | Oct 21 | Open |
| llama.cpp: logprobs with speculative decoding · Nemotron 3.5 Embed · H2O Lightning 4B | Oct 20–21 | Open |
| Ollama v0.40.7 (pplx-decider, Responses fixes) | Oct 21 | Open |
| AnyLanguageModel 1.0 + swift-huggingface 1.0 + swift-xet 1.0 | Oct 20 | On-device |
| OpenAI GPT-Astra / GPT-6.1 Sol / GPT-6 Luna in Google Cloud model garden (private preview) | Oct 21 | Cloud |
| OpenAI 12-hour prompt caching (Responses API) | Oct 20 | Cloud |
| Claude Opus 5.5 and Sonnet 5.5 in Google Antigravity | Oct 20 | Cloud |
| Structured outputs for Claude on Amazon Bedrock | Oct 20 | Cloud |
| Recursive Flint 40B-A2.5B (Apache 2.0 per NVIDIA); weights not public at check | Oct 21 | Watch |
| NVIDIA FP4 “Local” checkpoints: Qwen3.8 Flash Next, GLM-5.3 Flash (NVIDIA license) | Oct 21 | Open |
| mlx-community Hy-Memory-30B-A3B 4-bit / 8-bit | Oct 20 | Open |
| Databricks AI Runtime GA (managed GPU training) | Oct 21 | Cloud |
| Ray 2.58.1 / 2.57.1 (Ray Data fix) | Oct 20 | Open |
Security / policy / access
| Item | Date | Badge |
|---|---|---|
| eMudhra: keys for eight trusted intermediates compromised; update trust stores and images | Oct 19–20 | Active |
| Claude user profiles + organization webhooks (beta) for enforcement events | Oct 20 | Active |
| Claude Inference hooks out of beta (Enterprise) | Oct 20 | Active |
| OpenAI on Google Cloud: zero data retention and Confidential Compute “coming soon” | Oct 21 | Watch |
| NVIDIA OpenShell Reference Architecture (agent isolation across Rubin, Vera, BlueField-4) | Oct 21 | Watch |
| Amazon Bedrock Guardrails: Standard tier for contextual grounding checks | Oct 20 | Cloud |
| Lightning AI platform still offline after Oct 15 credential incident | Oct 21 | Active |
| No new vLLM advisories since the Oct 20 batch; none for llama.cpp / Ollama / MLX / SGLang | Oct 20–21 | Quiet |
Critical
Rebuild agent and model-serving images with current OS and runtime CA bundles after the eMudhra compromise, and restart what needs it. Reselling Claude through a gateway or SaaS? Set up user profiles before one customer’s traffic gets your whole organization restricted.
Watch
Recursive Flint weights on Hugging Face. MiniMax M3.1 Flash, which NVIDIA now calls an open model, still has no weights; Ollama’s cloud tag is free until Oct 22, 7:59 PM ET. OpenAI models in Gemini for business and Antigravity. An OpenAI-compatible llama-server speech endpoint. Still pending: PyTorch 2.15 final, Dory runtime support, Nemotron 3.5 Bolt E4B (Oct 23).
Opportunity
Nobody has published turn latency for a fully local voice loop on Apple Silicon: Voxtral in, a local chat model, Qwen3-TTS out, all on llama-server. A clean M3 Ultra or M4 Max writeup would be the first.
Make your Mac talk with llama-server
On build b11781 or newer, start llama-server on a TTS model (the docs use ggml-org/Qwen3-TTS-12Hz-1.7B-Base-GGUF via -hf), then run curl -X POST http://127.0.0.1:9931/tts -F "input=Hello from my Mac." -F "speaker_ref=@me.wav;type=audio/wav" -o out.wav. The docs strongly recommend a speaker reference clip. Use your own voice. Note the new default port, 9931.
Swap Apple’s model for a local MLX one in a Swift app
Add https://github.com/huggingface/AnyLanguageModel from 1.0.0 with the "MLX" trait, change import FoundationModels to import AnyLanguageModel, and create MLXLanguageModel(modelId: "mlx-community/Qwen3.8-27B-4bit"). Your existing session code stays the same. The post says that model wants 32 GB or more.
Give your stable prompt prefix a 12-hour cache
In a Responses API request to a GPT-5.6-or-later model, add "prompt_cache_breakpoint": {"mode": "explicit", "ttl": "12h"} to the content block that ends your shared instructions, and keep "prompt_cache_options": {"mode": "explicit", "ttl": "30m"} for the rest. Then compare cache-read tokens on the day’s first call before and after.
Tuesday morning through Wednesday morning (Oct 20–21 ET). ASTGL / Fraser takes, sourced.
Your Studio can listen now
With realtime transcription and text-to-speech merged, every piece of a fully local voice assistant now lives in one server most of us already run. You still have to wire the loop yourself: microphone into the websocket, transcript into your chat model, answer into the speech endpoint. My take: this is the weekend project I’d actually do. Before you call it a replacement for a cloud voice API, measure the full turn on your own machine, because nobody has posted Mac numbers. Plan on a small shim, since the speech endpoint isn’t OpenAI-shaped yet. And voice cloning from a reference clip means getting the speaker’s permission first.
Sources: llama.cpp #30636 · #26603
Everyone resells everyone now
This morning Google Cloud put OpenAI’s GPT-Astra, GPT-6.1 Sol, and GPT-6 Luna in its model garden, in private preview, billed on your Google invoice and counted against your cloud commitments. You can also point Codex CLI and Desktop at Google’s Gemini platform for inference. My take: for most teams this is a procurement story, not a capability story. It matters if your company has a big Google commit and a Codex habit. Zero data retention and Confidential Compute for these models are both “coming soon,” so if your compliance case depends on them, wait. “Only hyperscaler with all three leading closed models” is Google’s claim.
Sources: Google Cloud (Oct 21)
Anthropic hands resellers a blast-radius control
Anthropic’s new user profiles let a platform tag each API request with the end customer it is for. If that customer trips Anthropic’s safety classifiers, the restriction or ban lands on their profile instead of your whole organization, and a new webhook tells you when it happens. My take: if you run a gateway or a SaaS product on Claude, this is the most important change in this week’s notes. Without it, one bad customer can take your whole business offline. Register profiles at signup, wire up the webhook, and keep your own internal usage in a separate organization, as the docs advise. Profiles aren’t billed, and Anthropic doesn’t verify the details you send.
Sources: Claude release notes (Oct 20) · User profiles · Organization webhooks
OpenAI’s 12-hour cache is a math problem
OpenAI now lets you keep a cached prompt prefix for at least 12 hours instead of 30 minutes. A 12-hour write costs 2.5× normal input against 1.25× for the default, and cached reads cost the same either way. Our arithmetic from those published rates: if your shared prefix would otherwise go cold and be rewritten twice or more in a 12-hour span, the longer cache pays. My take: this is yesterday’s Cloudflare lesson, part two. Get breakpoint placement right first, then pick the lifetime. One fine-print note: OpenAI says it may keep entries longer than the minimum, and the setting doesn’t change its maximum data-retention policy.
Sources: OpenAI API changelog (Oct 20) · Prompt caching guide
Patch the trust store, not just the browser
Cloudflare says the private keys behind eight widely trusted eMudhra intermediate certificates were compromised, which means someone could forge a certificate for any domain. Browsers can push a fix fast. Your agent containers, Python services, and model downloaders check certificates through their own stack, and Cloudflare notes that updating an SDK alone may not change that. My take: an agent’s outbound HTTPS call carries your API key, so treat this as an AI-stack problem too. Update OS and runtime CA bundles, rebuild images, restart. And never make the new certificate errors go away by turning verification off.
Sources: Cloudflare blog (Oct 20)
NVIDIA’s desk-side pitch takes aim at the Mac Studio
At GTC Berlin this morning NVIDIA said DGX Spark now clusters up to eight units, for about 1 TB of unified memory, and touted Recursive Flint, a 40B mixture-of-experts coding model with 2.5B active that it says runs over 220 tokens per second on an RTX Spark. NVIDIA says Flint’s Apache 2.0 weights are on Hugging Face; at our check Recursive’s org page said “none public yet.” My take: “stack boxes on a desk until a frontier model fits” is the same pitch Apple made with Mac Studios last week. Competition is good for those of us buying local hardware. Wait for the weights and for numbers NVIDIA didn’t produce.
Sources: NVIDIA blog (Oct 21) · RecursiveSI on Hugging Face