Mac / Try this MiniMax M3.1 Flash 3-bit on a 256 GB Studio · Phonon speech in Ollama · d3 local decisions
Try this from Issue 45: run MiniMax M3.1 Flash’s 3-bit MLX build on a 256 GB Studio, make Ollama speak with the Apache-licensed Phonon model, and try local yes-or-no decisions with d3.
pip install -U mlx-lm (card: “Tested with mlx-lm 0.32.2”), then mlx_lm.generate --model mlx-community/MiniMax-M3.1-Flash-Text-3bit --temp 1.0 --top-p 0.95 -p "…" -m 2048 (MiniMax’s recommended sampling). ≈186.5 GB download; peak ≈187 GB in a short test, so keep prompts short at first and close other large apps. Text only (no vision tower). Skip the 4-bit build on 256 GB (≈249 GB peak). To load MiniMax’s original files instead, install mlx-lm from main (#2003), which needed ≈252 GB peak in the PR’s run — more than a 256 GB machine comfortably has. Commercial use needs the license credit + notice.
Ollama ≥0.40.9 (rc3 requires Gradium’s GA checkpoints). ollama run phonon "Hello from Ollama.". phonon / phonon:en / phonon:31m-en are the same 104 MB model — upstream gradium/phonon-31m-en, Apache 2.0. The de, fr, es, pt, multilingual tags map to 40M checkpoints licensed CC BY-NC 4.0 upstream. Apps: OpenAI-compatible /v1/audio/speech on your Ollama host (see Ollama’s docs for request fields).
From the release notes: ollama run d3 "A customer says their card was charged twice." --ask "Is this a billing issue?". Sizes 2B / 4B / 9B / 27B (vLLM Semantic Router’s decision models). Label a few dozen of your own messages first and compare. The cloud counterpart is OpenAI’s Decisions API (Frontier page).
Ollama ≥0.40.8: ollama run flint (23 GB, 256K context; macOS via MLX only for now). llama.cpp ≥b11803 (also needs b11798 for the template): llama-server -hf ggml-org/Flint-GGUF:Q4_K_M --jinja --temp 1.0 --top-p 0.95 --top-k 20, then use http://127.0.0.1:9931/v1 (default port 9931). Recursive: preserve returned reasoning_content in history; output budget ≥32,768 tokens for hard problems; stay within 256K. rec needs M4 Pro / M5 Pro / M5 Max with 48 GB+. Expect the llama.cpp path to be 10–30% slower than it will be after follow-up PRs. Measure tok/s yourself — published figures are vendor numbers.
rg -n "temperature|top_p|top_k|x-api-key|MessageParam\(" . finds sampling params (now 400s), header assumptions (SDK now sends Authorization: Bearer), and Python MessageParam(...) calls (now TypeError). Also grep for workspace_id if you read API keys from the Admin API, and for Text Completions calls. Pin anthropic<2 / @anthropic-ai/sdk<1 until clean. Bedrock users on Claude 4.6-or-earlier models stay on the current SDK.
Needs build b11781+ (Oct 21). Start llama-server on a TTS model — docs use ggml-org/Qwen3-TTS-12Hz-1.7B-Base-GGUF (via -hf). Then curl -X POST http://127.0.0.1:9931/tts -F "input=Hello from my Mac." -F "speaker_ref=@me.wav;type=audio/wav" -o out.wav; JSON form: -H "Content-Type: application/json" -d '{"input": "Hello"}'. Options: lang, seed for reproducible output, response_formatpcm, stream. Docs: a speaker reference is highly recommended (Pocket TTS requires one). For transcription, run a streaming-audio-encoder model (PR test: ggml-org/Voxtral-Mini-4B-Realtime-2602-GGUF) and stream PCM16 to ws://127.0.0.1:9931/v1/realtime with input_audio_buffer.append events. Clone only voices you have permission to use.
SwiftPM: .package(url: "https://github.com/huggingface/AnyLanguageModel", from: "1.0.0", traits: ["MLX"]). Replace import FoundationModels with import AnyLanguageModel; let model = MLXLanguageModel(modelId: "mlx-community/Qwen3.8-27B-4bit") (~16 GB at 4-bit; post says Mac with 32 GB+). Same session code works with the system model and hosted providers. Upgrading from 0.x: see 1.0.0 release notes (breaking JSONValue and @Generable enum changes).
GPT-5.6+ only. Request: "prompt_cache_options": {"mode": "explicit", "ttl": "30m"}; on the content block ending shared instructions add "prompt_cache_breakpoint": {"mode": "explicit", "ttl": "12h"}; leave the current question after the last breakpoint so it isn’t written. 12h writes 2.5× input vs 1.25× for 30m; reads unchanged. Compare cache-read tokens on the day’s first call before/after. Prefix must be ≥1,024 tokens.
Needs macOS 15+ and Python 3.11+. uv venv --python 3.11 pt215 && source pt215/bin/activate; uv pip install torch==2.15.0 --index-url https://download.pytorch.org/whl/test/cpu (the RC index from HF’s post); check python -c "import torch; print(torch.__version__, torch.backends.mps.is_available())". Keep a 2.14.1 environment and time the same model in both, with torch.compile on, at the same prompt and context length; HF’s headline MoE numbers were M3 Ultra + torch.compile. ComfyUI users: the Wan2.2 VAE memory fix and Qwen-Image-2.1 VAE corruption fix are the ones to verify.
curl -si -H "Origin: https://example.com" http://localhost:8000/v1/models | grep -i access-control-allow-origin, repeating for each port (vLLM default 8000, llama-server 9931 since Oct 9 builds, 8080 before — corrected Oct 21, Ollama 11434). * or an echoed https://example.com means a page in your browser can call the server. vLLM ≥0.32.0: set --allowed-origins explicitly and --api-key; llama-server: --api-key (and keep --connect off unless you mean it).
On the first Claude API call of each agent session, log usage.cache_read_input_tokens and usage.cache_creation_input_tokens. Near-zero first-call reads across sessions that share a system prompt mean something per-run sits before your cache breakpoint (date, working directory, repo name, dynamic skill/tool lists). Split the block so the shared part ends on identical bytes with its own breakpoint, then re-measure. Cloudflare’s numbers: first-call hits 14–26% before the fix.
Need llama.cpp ≥ b11701 (Beam). hf download unsloth/Beam-1.0-501B-A23B-GGUF --include "UD-Q2_K_XL/*" --local-dir beam (5 shards, ≈187 GB). Then llama-server -m beam/UD-Q2_K_XL/Beam-1.0-501B-A23B-UD-Q2_K_XL-00001-of-00005.gguf -c 32768 (llama.cpp loads remaining shards; default port 9931). KV at 32K is ≈1.2 GiB per sequence in BF16 per HF, so weights dominate. macOS caps Metal’s working set below physical RAM; if allocation fails, reduce context or try a smaller quant (UD-IQ2_M ≈165 GB). No primary has published Apple Silicon speeds; record yours. Quality at 2-bit is unmeasured on tasks (Unsloth publishes KLD only).
ollama run beam:cloud (Ollama account required; library lists 1M context, tools, thinking; check Ollama pricing). Use the identical prompt and tools as the local run; compare answers, tool-call correctness, and wall-clock time. Reflection’s own API (platform.reflection.ai) is in free public beta per Reflection.
pip install -U mlx mlx-lm (mlx 0.32.4, mlx-lm 0.32.2), then mlx_lm.benchmark --model Qwen/Qwen3.8-27B -p 8192 -g 128 -b 1; repeat with -p 65536. Compare TTFT and decode to Apple’s M3 Ultra / M5 Ultra columns (Apple’s rigs are 512 GB; Qwen 3.8 27B uses up to ≈68 GB at 65K, so a 256 GB Studio is fine).
ollama run muse-spark-open:cloud (Ollama account required; $1.25 / $0.15 cached / $4.25 per 1M tokens per the library page). Or a hosted provider listed by Meta. Note reasoning cannot be turned off; set effort “low” for easy tasks. Run one of your real agent tasks before considering a second Mac Studio.
Steps: (1) du -sh ~/.ollama/models (or your OLLAMA_MODELS path; on an external volume, check that). (2) Update to Ollama 0.40.2 and run each model you use once so the background upgrade finishes. (3) If you won’t downgrade below 0.40, run the backup-removal pipeline from the 0.40.2 release notes (needs jq; it removes only manifests with runner == "ggml" for models that also have a llamacpp manifest). (4) Re-run du. (5) grep -rn "localhost:8080\|127.0.0.1:8080" ~/bin ~/Library/LaunchAgents plus your proxy configs, and start llama-server with --port explicitly; builds from b11521 default to 9931 (Docker keeps 8080).
Steps: (1) Fresh venv: uv pip install "transformers>=5.5.4" pillow pypdfium2 torch. (2) Load lightonai/LightOnOCR-3-0.8B with Qwen3_5ForConditionalGeneration and AutoProcessor as in the model card; try the mps device first and fall back to CPU if an op is unsupported. (3) Render pages with pypdfium2 and run each page twice: empty prompt (transcription) and grounding (boxes, image descriptions, chart tables). (4) Score tables and charts against your current OCR, and draw the 0–1000 normalized boxes on the page to check them. Apache 2.0. Benchmarks in LightOn’s post are LightOn-reported.
Steps: (1) In a fresh virtual environment, pip install "transformers>=5.14" torch torchvision pillow. (2) Load LiquidAI/d1-3B with trust_remote_code=True on mps in bfloat16, using the sample in Liquid’s post. Because this runs model-repo code, pin the revision and run it in a sandbox or throwaway user. (3) Ask three named questions (for example refund / urgent / team) about 50 labeled support tickets. (4) Ask the same questions with clef on Ollama 0.40.x, which runs Clef on MLX on Apple Silicon. (5) Compare accuracy at your threshold and per-question latency. Liquid reports about 30 ms per question on an M5 Pro (Liquid-reported). The d1 license is LFM 1.0, so check it before commercial use.
LiteRT 2.3.0 adds a type-safe Swift wrapper over the C API for iOS and macOS apps, single-kernel Flash-Decode / FlashAttention-2 prefill on Metal, and MTLResidencySet residency for weights and KV cache. Google’s M4 Pro numbers (~38% decode on Qwen3-0.6B; 2.98× 32K prefill) are Google-reported. Rerun them on your own Mac at the context lengths you actually use, and watch memory pressure in Activity Monitor with and without the long-context path.
Steps: (1) On 0.35.1, run two models you use daily with ollama run <model> --verbose on 3–5 fixed prompts, and note the eval rate plus memory from Activity Monitor. (2) Upgrade to v0.40.0. Run the same prompts on a model in the MLX list (qwen3.8, gemma4, qwen3.6, qwen3.5) and on one outside it. (3) Diff outputs at temperature 0. (4) Keep the 0.35.1 installer until the numbers agree. The release notes publish no speed figures, so your numbers are the only ones.
Options: ollama pull embeddinggemma-2 (740M tag 1.3 GB; text+image per library) and POST /api/embed; or llama serve -hf ggml-org/embeddinggemma-2-GGUF and /v1/embeddings; or LiteRT-LM 0.18 litert-lm serve with its OpenAI-compatible /v1/embeddings. Use the card’s prefixes (task: search result | query: … for queries, title: … | text: … for documents; none for images). Store 768-d and 256-d (truncate and re-normalize), then compare recall on 20 known queries. Google reports image embedding at 37.3 ms (26.9 images/s) on a MacBook M5 Pro GPU via LiteRT (Google-reported). Google also released Google AI Edge Foresight for Mac, an experimental, fully local meeting companion built on EmbeddingGemma 2 and Gemma 4.
Steps: (1) download the v0.6.0 macOS arm64 asset (or build tag v0.6.0). Keep your v0.5.0 binary. (2) Serve a model you already benchmark. Call GET /v1/models and confirm the new architecture object lists modalities (#29987). (3) Rerun your spec-decoding prompt set (draft model or MTP) and a serial baseline on both versions. The “up to ~3x faster mat-mul” is project-reported; measure tok/s yourself. (4) If you persist sessions or state with --slot-save-path or the C API, regenerate them: LLAMA_SESSION_VERSION is now 11 and LLAMA_STATE_SEQ_VERSION is 4.
Pull the Q4_K_M GGUF (~5.30 GB) from the model card and run the card’s llama-cli example (--temp 0.0 -n 128) with your ASR output formatted as speaker-tagged text in the card’s prompt template. Compare speaker turns before and after. Remember the card’s caveat: not an officially supported Google product.
Grab a llama.cpp tip at or after b11404 (Oct 5) and the GA v0.5.0 binary. Same model, same fixed prompt set: run serial decode, then your existing draft-model / DFlash configuration, on both builds. Compare decode tok/s and acceptance. The b11404 note says DFlash2 on an M3 Ultra was slower than serial before this change on GPUs without the Metal tensor API; confirm on your own hardware before changing defaults. Tip ≠ GA.
Upgrade to Ollama v0.35.1 (GA). Pull clef-flash; POST to http://localhost:11434/v1/systemone with model, state, optional base64 images, and questions (types such as noul and choice) — example request/response in the release notes. Confirm ollama show reports only decision. On 0.40.0-rc3 (pre-release) the same models run on MLX.
Install Ollama v0.35.0 (pre-release) on a non-prod Mac Studio / MacBook profile. ollama pull nimble, then call /v1/systemone with a choice or score question. Confirm typed choices/probabilities — not chat prose. Keep a GA pin on v0.34.4 for anything customer-facing until a non-pre v0.35.x ships.
Bump mlx-vlm to 0.7.4 on Apple Silicon; smoke DeepSeek V4.1 vision (+ InternVL3.5 if you use it). Pair with MLX core 0.32.3 if you want GDN Metal / M5 Ultra matmul tunings in the same window.
llama.cpp tip b11228 Metal PAD on Apple Silicon (prior · Issue 26)
Bump to tip b11228 on a Mac Studio / MacBook Pro build. Confirm left/circular GGML_OP_PAD paths that previously mismatched Metal vs CPU/CUDA/Vulkan (#29561). Pair with tip b11227 if testing vision/causal_attn paths. GA pin remains v0.5.0 for production.
Ollama 0.40.0-rc0 MLX default on Mac Studio (RC sandbox) (prior · Issue 25)
Install v0.40.0-rc0 on a non-prod Studio profile. Confirm MLX path for a supported architecture (release example: qwen3.8). Keep a pin on v0.34.4 for anything customer-facing until a non-RC v0.40.0 ships. Do not claim universal architecture coverage — notes say additional models enabled during RC.
Upgrade from 0.7.2; re-smoke MiMo multimodal / MXFP4 load and any Qwen-Image or batched-audio paths you use. mlx / mlx-lm still quiet (0.32.2 / 0.31.3).
llama.cpp v0.5.0 Metal MoE / multi-bind on Apple Silicon
Pin v0.5.0 assets from the GA release (macOS Apple Silicon arm64 tarball). Release cites Metal MoE + SSM_CONV fusion (#28948). Optional: llama-server --host with comma-separated binds (#28690). Tip-only: b11156+ for Ling 3.0 VL if testing that architecture. Prefer GA over tip b11158 unless tip features are required.
Install/upgrade GA from v0.34.4 (prefer GA over Issue 23’s -rc0). Compare prompt-processing latency for a Qwen 3.8 MLX model vs v0.34.3. Release cites faster Qwen 3.8 PP + Gemma 4 per-image resolution on Apple Silicon. Optional Hub smoke: mlx-community/Qwen3.8-Flash-Next-4bit-mtp under current mlx-lm — community quant; verify MTP/arch support first.
