To run GLM-5.2 locally with Unsloth, you download one of Unsloth's Dynamic GGUF quantizations and load it with llama.cpp, because the full model is far too large to run any other way. GLM-5.2 is Z.ai's flagship open-weight model, with 744B total parameters, 40B active per token in a mixture-of-experts design, a 1M-token context window, and an MIT license. It was released on June 16, 2026. The catch is hardware. At full BF16 precision the weights are roughly 1.5 TB, so the model does not fit on a laptop, and, to the surprise of many, ollama pull glm-5.2 will not run it on your machine either. The plan in this guide is straightforward. Use Unsloth's Dynamic GGUF quantizations with llama.cpp to run the model on hardware you own, pick the largest quant your memory can hold, and launch a local OpenAI-compatible server you can point any client at.
Does Ollama run GLM-5.2 locally
No. As of mid-2026, Ollama lists GLM-5.2 but the only available tag is glm-5.2:cloud, and according to Thunder Compute, that tag "routes your prompts through Z.ai's hosted infrastructure instead of loading weights on your machine." Running ollama pull glm-5.2 without the cloud suffix fails with a manifest error, because there is no local download tag published for this model.
This matches how Ollama has handled other frontier-scale open models, including Kimi K2. The cloud tag gives you familiar command syntax, but it is an API wrapper rather than local inference. For genuine on-device or self-hosted execution, you need a different toolchain.
That toolchain is Unsloth's Dynamic GGUF quantizations plus llama.cpp. Unsloth publishes GGUF quantizations of GLM-5.2 built specifically for local inference through llama.cpp, and that combination is the realistic path to running the model outside Ollama's cloud tag or Z.ai's own API. The Ollama cloud-only detail above comes from a single source (Thunder Compute), so treat it as accurate as of mid-2026.
How much memory does GLM-5.2 need
GLM-5.2 is a mixture-of-experts model with roughly 744B total parameters and about 40B active per token. Only the active experts do work on any given token, but all 744B parameters have to live in memory at once. That memory is the sum of your GPU VRAM and system RAM, or unified memory on Apple Silicon. It is not VRAM alone.
The table below is Unsloth's inference hardware guidance for the GLM-5.2 Dynamic GGUFs. The "total memory" column is VRAM plus system RAM, or unified memory. Only the 1-bit and 2-bit disk sizes are published exactly. For the higher quants Unsloth gives total-memory ranges rather than fixed disk sizes.
Unsloth's own advice is to make sure your total available memory, including VRAM and system RAM, exceeds the quantized model file size by a comfortable margin. The 2-bit dynamic quant UD-IQ2_M uses 239 GB of disk space, fits directly on a 256 GB unified-memory Mac, and works well on a single 24 GB GPU with 256 GB of RAM using MoE offloading. The 1-bit quant fits in 223 GB, and 8-bit needs 810 GB.
How to read the 76% top-1 accuracy figure
The accuracy numbers in that table are easy to misread. When Unsloth says the Dynamic 1-bit quant reaches around 76.2% top-1 accuracy, that does not mean the model gets a quarter of its answers wrong.
Top-1 accuracy is a greedy-decoding measure. For each token, you check whether the quantized model picks the same argmax token as the full BF16 baseline, then average across a whole corpus. As Unsloth puts it, 76% top-1 "does NOT mean 'The capital of France is' becomes choosing 76% Paris and 24% of Sydney." For a factual answer like that, Paris is still picked 100% of the time. The 76% figure includes filler words and stop words across the entire corpus, where two plausible next tokens can legitimately differ. It does not mean you get gibberish or incorrect outputs 24% of the time.
This holds up because Unsloth Dynamic 2.0 does not quantize every layer to the same bit depth. It "now dynamically adjust[s] the quantization type of every possible layer," keeping sensitive layers at higher precision. Unsloth's KL-divergence benchmarks show that the dynamic 4-bit UD-Q4_K_XL and dynamic 5-bit UD-Q5_K_XL are mostly lossless. There is a larger quality uplift from 4-bit onwards, so for large out-of-distribution tasks, dynamic 4-bit is probably the safest choice.
Pick the right quant for your hardware
The single most important decision is matching a quant to the memory you have. Work out your total memory (GPU VRAM plus system RAM, or unified memory on a Mac), then pick the largest quant that fits under it with margin to spare. There is no way around the floor here. According to Thunder Compute, no single GPU can run GLM-5.2 on its own, because even the most aggressive 1-bit quantization needs about 223 GB, more than any single card offers.
Run GLM-5.2 on a Mac Studio with unified memory
The cleanest single-box path is a Mac Studio with an M3 Ultra or M4 Ultra chip and 256 GB or more of unified memory. The 2-bit dynamic quant fits directly within that shared pool, and because the CPU and GPU share the same memory, there is no PCIe bottleneck between them. The honest tradeoff is cost. A maxed-out Mac Studio runs around $15,000 before you have run a single inference. On this hardware you build llama.cpp with -DGGML_CUDA=OFF, since Metal support is on by default.
Run GLM-5.2 on a consumer multi-GPU rig
A rig with 4x RTX 3090 or 4x RTX 4090 and 192 to 256 GB of system RAM can run the 2-bit dynamic GGUF using CPU and GPU hybrid offloading. Consumer cards in a 4-card setup with system RAM offloading will run the model, but expect single-digit tokens per second. This assumes you already own or are willing to buy four consumer GPUs before confirming the model fits your workflow.
Run GLM-5.2 on one GPU plus a lot of RAM
The most accessible path is a single 24 GB GPU with 256 GB of system RAM, running the 2-bit UD-IQ2_M quant with the MoE expert layers offloaded to the CPU. This works because only the non-MoE layers need to sit on the GPU. Keep in mind that no single GPU alone can hold even the 1-bit quant, so the large pool of system RAM is doing the heavy lifting here. The offloading flags that make this work are covered in the MoE offloading section further down.
Install the toolchain
Before you build anything, install the system prerequisites for your platform. The pure llama.cpp path needs git, cmake, C++ build tools, and, for NVIDIA, a CUDA toolkit. Python (with huggingface_hub) is only needed for the manual weight download step.
On Linux or WSL, you need Ubuntu 20.04 or a similar 64-bit distro, an NVIDIA GPU with drivers installed, and the CUDA toolkit, version 12.4 or newer (12.8 or newer for Blackwell). Install git and the C++ build tools with your package manager.
sudo apt install git build-essential cmake
On macOS 12 Monterey or newer, install Homebrew first, then the build dependencies.
brew install git cmake openssl
On Windows 10 or 11, install git with winget install --id Git.Git -e --source winget, add Visual Studio Build Tools 2022 for the C++ compiler, and an NVIDIA GPU with drivers. If you also want to use Unsloth Studio, install Python 3.11 up to but not including 3.14 and work inside a virtual environment such as uv, venv, or conda.
Build llama.cpp and download the weights
Build llama.cpp from the official repository. Set -DGGML_CUDA=ON for NVIDIA GPUs, or -DGGML_CUDA=OFF for CPU-only inference or Apple Silicon, where Metal support is enabled by default.
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
-DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first \
--target llama-cli llama-server
cp llama.cpp/build/bin/llama-* llama.cpp
Then download only the quant you selected. You can let llama.cpp fetch weights automatically the way ollama run does, but Unsloth notes that process can be very slow, so the manual download below is much faster. Swap the --include pattern for whichever quant you chose from the table above.
pip install -U huggingface_hub hf_transfer
hf download unsloth/GLM-5.2-GGUF \
--local-dir unsloth/GLM-5.2-GGUF \
--include "*UD-IQ2_M*" # 1-bit: *UD-IQ1_S* | 3-bit: *UD-Q3_K_XL* | 4-bit: *UD-Q4_K_XL*
The weights arrive as split files, for example GLM-5.2-UD-IQ2_M-00001-of-00006.gguf. When you launch the model, point the --model flag at part 00001 and llama.cpp finds the remaining split parts automatically.
Run GLM-5.2 with llama-server
Once the build finishes, launch the model with llama-server to expose an OpenAI-compatible endpoint. The temperature and top_p values below match Unsloth's recommended defaults for most tasks.
./llama.cpp/llama-server \
--model unsloth/GLM-5.2-GGUF/UD-IQ2_M/GLM-5.2-UD-IQ2_M-00001-of-00006.gguf \
--alias "unsloth/GLM-5.2" \
--jinja \
--temp 1.0 --top-p 0.95 --min-p 0.01 \
--ctx-size 32768 --port 8001
Include
--jinja. It tells llama.cpp to load Unsloth's fixed chat template, and without it multi-turn chat can break. Unsloth states this requirement explicitly for GLM-4.6, where a non-Unsloth GGUF would handle the first message fine but break on the second. GLM-5.2 uses the same toolchain and template fixes, so the requirement carries forward.
Unsloth's recommended sampler settings are temperature 1.0 and top_p 0.95 for most tasks, and temperature 1.0 with top_p 1.0 for SWE-Bench-Pro-style agentic coding. The maximum context window is 1,048,576 tokens. The --min-p 0.01 flag in the command above is Thunder Compute's own addition and is not part of Unsloth's stated defaults for GLM-5.2, so treat it as optional.
Call the local server from Python
With the server running, you can call it from any OpenAI-SDK-compatible client by pointing the base URL at your instance. No real API key is required.
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8001/v1", api_key="sk-no-key-required")
completion = client.chat.completions.create(
model="unsloth/GLM-5.2",
messages=[{"role": "user", "content": "Outline a plan to refactor a FastAPI service to async I/O."}],
)
print(completion.choices[0].message.content)
The same endpoint works as a custom model target for coding agents like Claude Code or Cursor. Both support pointing at an OpenAI-compatible base URL instead of a hosted provider, so you can drive them from your local GLM-5.2 instance.
Fit more into memory with MoE offloading
MoE offloading is what lets a large quant run on a machine with limited VRAM. The idea is to push the MoE expert layers onto the CPU and system RAM so the non-MoE layers fit on the GPU. You control this with the -ot flag, and you trade VRAM usage against speed as you move more layers off the CPU.
# Least VRAM: all MoE experts on CPU
-ot ".ffn_.*_exps.=CPU"
# Keep more on GPU as VRAM allows
-ot ".ffn_(up|down)_exps.=CPU"
-ot ".ffn_(up)_exps.=CPU"
The first pattern offloads all MoE layers to the CPU and uses the least VRAM, which lets you fit all the non-MoE layers on a single GPU. If you have more GPU memory to spare, the second pattern offloads only the up and down projection layers, and the third offloads only the up projection layers, keeping more of the work on the GPU.
The exact flags and speed numbers here come from Unsloth's GLM-4.6 tutorial, which uses the same model family and offloading technique. For GLM-4.6, the 4-bit quant fit on a 1x 40 GB GPU with MoE layers offloaded to RAM and hit around 5 tokens/s given a bonus 165 GB of RAM, with at least 205 GB of combined memory recommended for 5+ tokens/s. GLM-5.2 is much larger (744B versus 355B), so expect lower throughput than these GLM-4.6 figures. Treat them as a rough ballpark, not a GLM-5.2 measurement.
If your combined memory does not exceed the quant file size, llama.cpp can still run via SSD or HDD offloading. It works, but inference will be noticeably slower. You can tune performance with --threads for the number of CPU threads, --ctx-size for context length, and --n-gpu-layers for how many layers to place on the GPU. Lower --n-gpu-layers if the GPU runs out of memory, and remove it entirely for CPU-only inference.
Control thinking modes and long context
GLM-5.2 has three thinking modes: Non-thinking, Thinking-High, and Thinking-Max. Reasoning is on by default, and you use Max Thinking for the most complicated tasks. The Z.ai blog describes this as flexible thinking effort that balances performance against latency. You can toggle these modes from the command line.
# Disable thinking
--chat-template-kwargs '{"enable_thinking":false}'
# Set reasoning effort
--chat-template-kwargs '{"reasoning_effort":"max"}' # or "high"
# Recent llama.cpp also supports: --reasoning on / --reasoning off
On Windows PowerShell the JSON needs escaped quotes, so use --chat-template-kwargs "{\"enable_thinking\":false}" instead.
Quantize the KV cache for the 1M context window
GLM-5.2 supports a 1M-token context, but a full-precision KV cache eats memory fast, so to use long context you quantize the cache. By default llama.cpp uses f16. Switching to q4_0, which is around 4.5 bits per weight, gives you roughly 3.5x longer context (16 divided by 4.5). Using q4_1 at 5 bits per weight gives about 3.2x with a shifting parameter that improves accuracy slightly, and Unsloth recommends the _1 variants for that reason.
--cache-type-k q4_1 --cache-type-v q4_1
To quantize the V cache as well, you need to build llama.cpp with Flash Attention support via -DGGML_CUDA_FA_ALL_QUANTS=ON and run with --flash-attn enabled. Recent llama.cpp releases added higher-accuracy tricks for KV-cache quantization that make the low-bit caches more usable.
Self-hosting vs the Z.ai API
Self-hosting only makes sense once you compare it against what Z.ai already charges. Below is Z.ai's official API pricing for GLM-5.2, per million tokens.
That $1.40 per million input tokens and $4.40 per million output tokens is the bar a self-hosted setup needs to beat. For lower-volume or exploratory use, the API is simpler and cheaper. Self-hosting wins in a narrower set of situations:
- Hard data-residency requirements.
- Sustained high-volume usage that keeps your hardware busy.
- Fine-tuning on proprietary data.
The breakeven point depends on your actual request volume, since owned hardware and rented GPUs both cost money whether or not tokens are flowing. If you want to test GLM-5.2 at higher precision before committing to a $15,000 Mac Studio or a four-GPU rig, renting GPU instances by the hour lets you try it at data-center grade without the upfront cost. Several providers offer A100 and multi-GPU instances by the hour, so shop around rather than defaulting to any one vendor.
GLM-5 vs GLM-5.1 vs GLM-5.2
Z.ai shipped three releases in the GLM-5 line within about four months. GLM-5 launched as the initial scaling step up from GLM-4.5. GLM-5.1 followed with notable coding gains. GLM-5.2 is the current flagship and the version most third-party providers and community benchmarks reference today.
The headline change in 5.2 is the jump from a roughly 200K context to a 1M-token context, which Z.ai confirms directly, along with improved long-horizon coding. On standard coding benchmarks, GLM-5.2 improves on GLM-5.1 by a wide margin, scoring 81.0 versus 63.5 on Terminal-Bench 2.1 and 62.1 versus 58.4 on SWE-bench Pro. The ~200K context figures for GLM-5 and GLM-5.1 come from Thunder Compute rather than a primary Z.ai source, so treat those as approximate. GLM-5.2 is the default choice unless you have a specific reason to use an earlier release, and it carries forward the architecture and MIT licensing of its predecessors while adding meaningfully more usable context.
Frequently asked questions
Can you run GLM-5.2 with Ollama?
Not locally. Ollama only offers a glm-5.2:cloud tag, which routes requests through Z.ai's hosted infrastructure. For real local or self-hosted inference, use Unsloth's GGUF quantizations with llama.cpp instead.
How much VRAM does GLM-5.2 need?
It is total memory (VRAM plus system RAM), not VRAM alone. Unsloth's GGUF quants range from about 223 to 245 GB at 1-bit and 2-bit, up to about 810 GB at 8-bit. No single GPU can hold the model.
Can GLM-5.2 run on a single GPU?
No. Even 1-bit quantization needs over 220 GB of combined memory, more than any single consumer or workstation GPU offers. A realistic setup needs at least 4 GPUs, or a high-memory unified-memory system like a Mac Studio.
What is the most accessible quant?
The 2-bit dynamic UD-IQ2_M quant. It uses 239 GB of disk space, retains about 82% accuracy, and fits directly on a 256 GB unified-memory Mac or on a 24 GB GPU with 256 GB of RAM using MoE offloading.
Do I need --jinja?
Yes. It loads Unsloth's fixed chat template, and without it multi-turn chat can break. Unsloth states this explicitly for GLM-4.6, and it carries forward to GLM-5.2 on the same toolchain.
Is GLM-5.2 open weight?
Yes. Z.ai released the full weights on Hugging Face under an MIT license on June 16, 2026. The license allows commercial use, fine-tuning, and redistribution with no regional restrictions.
Can I connect GLM-5.2 to Claude Code or Cursor?
Yes. Once llama-server is running, it exposes an OpenAI-compatible endpoint. Point Claude Code or Cursor at that base URL the same way you would point them at any custom model provider.
Wrapping up
GLM-5.2 is runnable on hardware you own, as long as you have the memory for it, using Unsloth Dynamic GGUFs with llama.cpp. Ollama's glm-5.2:cloud tag is an API passthrough, not local inference. Pick the largest quant your total memory can hold, remember to launch with --jinja, and start with the 2-bit UD-IQ2_M quant, which is the most accessible balance of size and accuracy.