Quantization Explained: How to Run 70B AI Models on a EUR 700 Mac Mini

The question we hear most from prospective clients: “How can a model with 70 billion parameters run on a box that fits on my desk?” The answer is quantization, a family of compression techniques that cut a model’s memory footprint by 4 to 8 times while keeping most of its quality. By the end of this post you will know which level to pick, what fits in the machine you already have, and the exact Ollama command to pull it.

AI model quantization

What you need

Any computer with 8 GB of RAM or more. A laptop is enough for a 3B model; 16 GB runs 7B to 14B models; 48 GB or more runs a 70B model.
Ollama installed (curl -fsSL https://ollama.com/install.sh | sh on Linux, brew install ollama on macOS).
Ten minutes and about 5 GB of free disk for the first model.

What quantization does

A standard model stores each parameter as a 16-bit floating-point number (FP16). A 70B model at FP16 needs 140 GB of memory, beyond any consumer device.

Quantization lowers the precision of those numbers: 8 bits per parameter halves the memory, 4 bits quarters it, and 2-bit formats go further still. The model gets smaller, faster and cheaper to run, with a quality loss that is small at 4 bits and noticeable below that.

xychart-beta
    title "70B model: memory by quantization level"
    x-axis ["FP16 (full)", "INT8", "Q6_K", "Q5_K_M", "Q4_K_M", "Q3_K_M", "Q2_K"]
    y-axis "Memory (GB)" 0 --> 150
    bar [140, 70, 56, 48, 40, 35, 25]

At Q4_K_M (4-bit, medium quality) a 70B model drops from 140 GB to about 40 GB, which fits a Mac mini M4 Pro with 48 GB of unified memory or a Mac Studio.

The three formats that matter in 2026

GGUF (what Ollama uses)

GGUF is the format used by llama.cpp and Ollama. It is the standard for consumer hardware because it supports CPU + GPU hybrid inference: the model loads partly into GPU memory and partly into system RAM. A GPU with only 8 GB of VRAM can still hold the compute-heavy layers while the rest sits in RAM. On Apple Silicon the CPU and GPU share one memory pool, which is why Ollama on a Mac is so simple.

GGUF level	Size vs FP16	Quality	Use case
Q2_K	~18%	Rough	Testing only; noticeable degradation
Q3_K_M	~25%	Acceptable	Very memory-constrained devices
Q4_K_M	~28%	Good	Production default; best balance
Q5_K_M	~35%	Very good	When you have spare RAM
Q6_K	~42%	Excellent	Quality-critical applications
Q8_0	~50%	Near-original	When quality matters more than memory

Start with Q4_K_M. If the output is not good enough for your task, step up to Q5_K_M. In our deployments Q4_K_M has been indistinguishable from full precision for the large majority of business tasks: summaries, extraction, classification, drafting.

AWQ (production GPU inference)

AWQ (Activation-Aware Weight Quantization) measures which weights matter most during real inference and protects those from aggressive compression. The result is about 95% quality retention at INT4, versus roughly 92% for GGUF. Major model families ship AWQ checkpoints on Hugging Face, and servers such as vLLM and TensorRT-LLM include optimised AWQ kernels. Best for dedicated GPU servers where throughput matters.

GPTQ (batch processing)

GPTQ runs a small calibration dataset through the model once to choose the quantization parameters. It retains about 90% quality and suits batch jobs where latency is not critical: offline document processing, queued API requests.

How much quality you lose

Method	Quality vs full	Memory saved	Speed	Best for
GGUF Q4_K_M	~92%	~72%	Good (CPU + GPU)	Ollama, Mac, local deployment
AWQ INT4	~95%	~75%	Excellent (GPU)	Production GPU servers
GPTQ INT4	~90%	~75%	Good (GPU)	Batch processing
FP8	~98%	~50%	Best (H100 and newer)	Data-centre NVIDIA hardware
INT8	~97%	~50%	Very good	Balance of quality and size

Figures from the Prem AI and VRLA Tech comparisons. For summarisation, Q&A, classification and code generation the difference between Q4_K_M and full precision is hard to notice. Where it shows: long multi-step reasoning and nuanced creative writing.

Step 1: check what fits in your memory

Your hardware	Memory	Largest model (Q4_K_M)	Example
Jetson Orin Nano	8 GB	7B	Qwen 2.5 7B
Mac mini M4 16 GB	16 GB	14B	DeepSeek R1 14B
Mac mini M4 24 GB	24 GB	27B	Gemma 3 27B
Mac mini M4 Pro 48 GB	48 GB	70B	Llama 3.3 70B
Mac Studio 96 GB	96 GB	109B MoE	Llama 4 Scout
RTX 3090	24 GB VRAM	27B	Gemma 3 27B
RTX 4090	24 GB VRAM	32B	DeepSeek R1 32B

Rule of thumb: model file size plus 1 to 2 GB for the context window, plus whatever your operating system needs. Leave a quarter of the memory free.

Step 2: pull the right tag

Ollama’s default tag for almost every model is Q4_K_M; you never touch the quantization tooling yourself. To choose another level, name it in the tag:

ollama pull llama3.3:70b            # default tag = Q4_K_M, ~43 GB
ollama pull llama3.3:70b-q5_K_M     # ~50 GB, higher quality
ollama pull llama3.3:70b-q8_0       # ~75 GB, near-original
ollama pull qwen2.5:7b              # 4.7 GB, fits a 16 GB laptop

Step 3: verify what you got

ollama show qwen2.5:7b     # prints parameters, context length and "quantization Q4_K_M"
ollama run qwen2.5:7b "Summarise this in one line: quantization trades precision for memory."
ollama ps                  # shows the real memory in use while the model is loaded

If ollama ps shows the model split between CPU and GPU (for example 60%/40% CPU/GPU), the model is too large for your GPU memory and generation will be slow; drop one level or pick a smaller model.

Step 4: decide with a measurement, not a table

Run your own task through Q4_K_M and Q5_K_M and compare the answers side by side. Ask your AI assistant to grade the two outputs against a rubric you write. If you cannot tell them apart, keep Q4_K_M and spend the memory on a longer context.

What we measured

On our NVIDIA GB10 workstation (128 GB unified memory), ollama list and ollama ps on 2026-09-09:

Model	Parameters	Quantization	On disk	In memory (4K context)	Generation
qwen2.5-coder:7b	7.6B	Q4_K_M	4.7 GB	4.7 GB	33.2 tok/s
llama3.1:8b	8B	Q4_K_M	4.9 GB	22 GB at the 131K default context of `ollama run`	37.8 tok/s (2026-09-08)
deepseek-r1:14b	14B	Q4_K_M	9.0 GB	not measured	not measured
qwen3.6:35b	35B	Q4_K_M	23 GB	not measured	not measured
qwen2.5vl:72b	73.4B	Q4_K_M	48 GB	not measured	not measured

The last row is the point of the whole post: a 73-billion-parameter model that would need about 147 GB at FP16 sits in 48 GB on disk at Q4_K_M. Memory in use grows with the context window, and by more than most people expect: the same 4.9 GB Llama 3.1 8B file occupied 22 GB when ollama run opened it with its 131K-token default context. Set num_ctx to what your task needs (4K to 8K for most business work) and the footprint drops to a little over the file size.

The 2026 production stack

Discovery: LM Studio, a GUI for browsing and testing models.
Development and SME deployment: Ollama with GGUF Q4_K_M, the simplest path, works everywhere.
High-throughput production: vLLM with AWQ for API servers that need maximum requests per second.

For our SME clients, step 2 is where most deployments live permanently. Ollama with a Q4_K_M model covers a solo law firm and a 50-person manufacturer alike.

Where this fits

Quantization is what makes the local economics work: a Mac mini or a used mini PC runs models that handle most day-to-day business tasks, electricity is a few euros a month, and your data never leaves the building (GDPR by design). The honest limit: a quantized 7B model is not a frontier model. For difficult reasoning keep a cloud key with a spending cap and route the routine volume locally; the numbers are in Cloud vs local: calculate your break-even.

Next steps

Choose the machine: Edge AI hardware guide 2026.
Choose the model: Best local LLM models Q2 2026.
Go further: Fine-tune a model on your own hardware with LoRA.

Work with us

We size the model and the machine for each client by measuring, not by guessing. If you want to see your own task running on a quantized model on real hardware, book a 15-minute call or see how we work in consulting.