Auto-Tune Local Inference on Your NVIDIA Card: AITune for PyTorch Models and the Three Knobs That Matter for LLMs

By the end of this post you will have measured your own tokens per second, changed the three settings that move that number most, and know whether NVIDIA’s AITune toolkit belongs in your stack. The cheapest place to start is any NVIDIA card with compute capability 5.0 or newer, which reaches back to the GTX 900 series, or a Jetson Orin Nano Super at around EUR 250.

A correction first. An earlier version of this post was a machine rewrite of a press release. Here is what we verified this week. AITune is real: NVIDIA published it on 10 April 2026 under the Apache 2.0 licence at github.com/ai-dynamo/aitune, and MarkTechPost covered the release. It benchmarks several inference backends on your PyTorch model and your GPU, checks that the outputs still match, and keeps the fastest. What it is not is a language-model server. NVIDIA’s own README says that if a dedicated serving framework supports your model, you should use TensorRT-LLM, vLLM or SGLang instead. So: AITune for vision, speech, embedding and diffusion pipelines written in PyTorch; Ollama, vLLM or TensorRT-LLM for LLMs. This post covers both halves.

What you need

• An NVIDIA GPU with compute capability 5.0 or newer and driver 550 or newer, which is Ollama’s stated requirement. A used RTX 3060 12 GB, a laptop RTX 4060 8 GB or a Jetson Orin Nano Super all qualify.
• Ollama installed from ollama.com/download and one model pulled, for example ollama pull qwen2.5-coder:7b (4.7 GB) or ollama pull qwen2.5:3b (1.9 GB) on the Jetson.
• For AITune only: Linux (Ubuntu 22.04 or newer), Python 3.10+, PyTorch 2.8+, and TensorRT 10.3+ if you want that backend. Those are the README’s requirements; it does not mention Jetson, so check your JetPack’s TensorRT version before trying the TensorRT backend there.
• Thirty minutes. No account and no API key.

Step 1: Measure before you touch anything

Ollama’s API returns the token count and the generation time with every answer. That is your benchmark tool.

curl -s localhost:11434/api/generate -d '{
  "model": "qwen2.5-coder:7b",
  "prompt": "Write a Python function that validates a Spanish NIF.",
  "stream": false
}' | python3 -c "import json,sys; d=json.load(sys.stdin); \
print(d['eval_count'], 'tokens in', round(d['eval_duration']/1e9,2), 's =', \
round(d['eval_count']/(d['eval_duration']/1e9),1), 'tok/s; load', round(d['load_duration']/1e9,2), 's')"

On our workstation (NVIDIA GB10, 128 GB unified memory, 9 September 2026) this printed 231 tokens in 6.92 s = 33.4 tok/s; load 10.7 s. The second call showed load 0.2 s: the model was already resident. If ollama ps shows 100% CPU in the Processor column, the GPU was not detected. Check nvidia-smi and the driver version before going further.

Step 2: The three Ollama knobs

Setting	Where	Default	What it does
num_ctx / OLLAMA_CONTEXT_LENGTH	per request or server env	4096 tokens	Size of the context window. Memory grows with it.
OLLAMA_KV_CACHE_TYPE	server env	f16	q8_0 uses about half the cache memory, q4_0 about a quarter.
OLLAMA_FLASH_ATTENTION	server env	auto	Reduces memory use as context grows; 1 forces it on.
num_gpu	per request or Modelfile	all layers	Layers offloaded to the GPU. 0 forces CPU.
keep_alive	per request	5 minutes	How long the model stays loaded. -1 keeps it forever.
OLLAMA_NUM_PARALLEL	server env	1	Requests served at once per model.

Defaults and values are from the Ollama FAQ.

1. Context is the memory knob. We loaded the same qwen2.5-coder:7b twice and read ollama ps: 4.7 GB resident at num_ctx 4096, 6.6 GB at 32768. Set the context you need for the job, not the maximum the model supports. Per request:

curl -s localhost:11434/api/generate -d '{"model":"qwen2.5-coder:7b","prompt":"...","stream":false,"options":{"num_ctx":8192}}'

2. Quantise the cache when memory is tight. On a systemd install, run systemctl edit ollama.service and add under [Service]:

Environment="OLLAMA_KV_CACHE_TYPE=q8_0"
Environment="OLLAMA_FLASH_ATTENTION=1"
Environment="OLLAMA_CONTEXT_LENGTH=8192"

Then systemctl daemon-reload && systemctl restart ollama. On an 8 GB card this is often the difference between a 7B model fitting fully on the GPU and spilling to the CPU.

3. Never accept a CPU/GPU split. We forced "num_gpu": 0 on the same prompt and got 5.2 tok/s from our 20-core ARM CPU against 33 tok/s on the GPU. If ollama ps shows a split like 40%/60% CPU/GPU, pick a smaller quantisation or a smaller model rather than live with the split. Our quantisation guide explains the trade.

Step 3: When several people share the model: vLLM flags

Ollama is the right first server. When a team hits it at once, vLLM’s continuous batching earns its place. Install and run, from the vLLM quickstart:

uv pip install vllm --torch-backend=auto
vllm serve Qwen/Qwen2.5-7B-Instruct \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.85 \
  --enable-prefix-caching

The flags that matter, from the engine arguments reference: --max-model-len caps context so the KV cache is planned for what you use; --gpu-memory-utilization (default 0.92) should come down when the card is shared with anything else; --enable-prefix-caching reuses the computation for repeated system prompts; --kv-cache-dtype can quantise the cache; --max-num-seqs bounds concurrent sequences. We serve Qwen2.5-7B-Instruct this way on port 8200 next to Ollama on 11434, and route between them with LiteLLM.

Step 4: AITune for a PyTorch model

If you run a detector, a transcription model, an embedding model or a diffusion pipeline in your own PyTorch code, this is where AITune fits.

pip install --extra-index-url https://pypi.nvidia.com aitune

import aitune.torch as ait
modules_info = ait.inspect(model, dataset)     # find tunable modules
modules = modules_info.get_modules()
model = ait.wrap(model, modules)
ait.tune(model, dataset)                       # benchmark every backend, keep the fastest
ait.save(model, "tuned.ait")                   # reuse on the next start, no warm-up
# later: ait.load(model, "tuned.ait")

For a first look without touching code, set AUTOWRAPT_BOOTSTRAP=aitune_enable_jit_tuning in the environment and run your script; AITune tunes modules on their first call. The backends it tries are TensorRT, Torch-TensorRT (JIT and AOT), TorchAO, Torch Inductor (JIT and AOT) and ONNX Runtime, per the backend documentation. You supply a small dataset so it can check that the tuned outputs match the originals. We have not benchmarked AITune on a Jetson, so we make no speed claim for it there.

Step 5: TensorRT-LLM when one model serves many users

TensorRT-LLM installs with pip install tensorrt-llm and serves with trtllm-serve. Its README lists Blackwell, H100, H200 and L4 as supported; a Jetson AGX Orin has a dedicated v0.12.0-jetson branch on JetPack 6.1. On an Orin Nano, stay with Ollama. The cost is an engine build per model per GPU, which pays off when the model is fixed and the traffic is high.

What we measured

Machine	Model	Setting	Result	Date
GB10 workstation	qwen2.5-coder:7b	GPU, num_ctx 4096	33.4 tok/s, 4.7 GB resident	2026-09-09
GB10 workstation	qwen2.5-coder:7b	GPU, num_ctx 32768	6.6 GB resident	2026-09-09
GB10 workstation	qwen2.5-coder:7b	num_gpu 0 (CPU only)	5.2 tok/s	2026-09-09
GB10 workstation	llama3.1:8b	GPU, defaults	37.8 tok/s	2026-09-08
Jetson Orin Nano Super (third party)	3B models	GPU	12 to 18 tok/s (Edge AI Vision)	2026-01

One honest note: our GPU is shared with about fifty other services. During a burst of other work, the same 7B model measured between 12 and 25 tok/s. Measure on a quiet machine, and set OLLAMA_MAX_LOADED_MODELS so a second model cannot evict the one you are timing.

Where this fits, and the limits

These settings do not make a 7B model smarter; they make it fit and respond faster on the hardware you already own. For a two-person shop, Step 2 is the whole job. For a team, add Step 3. Use AITune only if you have PyTorch code of your own to run. Cloud remains the right call for the tasks that need a frontier model; the break-even calculation tells you which share that is.

Next steps

• Choose the box: Edge AI hardware guide 2026.
• Fit a bigger model in less memory: Quantization explained.
• Put the tuned server to work: How to deploy AI locally in your business.

Work with us

We tune inference on our own machines every week and we leave clients with the measured numbers, not estimates. If you want your card measured and configured, get in touch or see how our consulting works.