Will it fit on your GPU? Pingala-8 2-bit vs normal 4-bit

How the numbers are calculated

Total = weights + KV cache + overhead allowance. This is the same arithmetic as our guide How much VRAM do you need to run an LLM locally?

• Weights. llama.cpp loads the whole GGUF file, so the weights take about its size. Where we know a real file (Qwen's own GGUF releases, our benchmark files), we use its exact size. Otherwise we estimate parameters x bits per weight / 8, with the bits per weight that llama.cpp measured on whole Llama-3.1-8B files. We checked the estimate against the 34 real files listed below: it came out between 4.9% low and 1.4% high, with the largest gaps for small models with large vocabularies (Qwen3-4B Q2_K).
• KV cache. For every token of context, each layer stores a key and a value: 2 x layers x KV heads x head size x bytes per number. f16 takes 2 bytes per number, q8_0 takes 34 bytes per 32 numbers and q4_0 takes 18 bytes per 32. llama.cpp rounds the context up to a multiple of 256.
• Gemma 3. Five of every six layers use sliding-window attention. By default llama.cpp keeps only the window plus one 512-token batch for those layers, rounded up to a multiple of 256 (1,536 tokens for the 1,024-token window). The --swa-full option turns this off and uses the full size.
• Overhead allowance. 1.5 GB by default, for the compute buffer, the CUDA context and the desktop. It is a rule of thumb, not a measurement, so change it if you know better. A headless Linux machine may need less; a Windows desktop with a browser open may need more.
• Verdict. Fits: the total is within your card. Tight: the weights and cache fit, but less than the allowance is left, so expect to cut the context or close other apps. Does not fit: the weights and cache alone are larger than the card.
• Units. GB here means 10^9 bytes, and card sizes are taken at face value. A card sold as 8 GB has 8 GiB, about 8.6 GB, so the verdict errs on the safe side. Totals are the sum of the rounded parts.

Real use varies. The runtime, its version and settings (batch size, flash attention, parallel slots), the driver and other programs all change the real number. Before you start, run nvidia-smi to see how much VRAM is free; llama.cpp prints the size of the model buffer, the KV cache and the compute buffer as it loads. If a model does not fit, -ngl keeps some layers in system RAM, which runs much slower.

A quantized cache (-ctk q8_0 -ctv q8_0) roughly halves the cache. llama.cpp needs flash attention for a quantized V cache (-fa on). It changes the output slightly, and we have not tested it with Pingala-8.

Sources

Model shapes come from each model's public config.json on Hugging Face, checked against the tensor shapes in its safetensors files (Gemma 3 configs leave some values at library defaults, and Gemma 3 12B's head size is 256, not hidden size / heads). Parameter counts are the text-model tensors in the safetensors headers: a tied output matrix is counted once, as in GGUF, and the Gemma 3 vision tower is left out. Meta's repos are gated, so their configs were read from the ungated unsloth mirrors and the counts checked against Meta's own repo metadata.

GPU memory

From NVIDIA's spec pages, checked on 2026-10-25. Laptop makers can fit other amounts; check yours with nvidia-smi.

• GeForce compare page, GTX 16 Series table: GTX 1650: 4 GB (GDDR5 and GDDR6 versions).
• RTX 3050: 8 GB or 6 GB.
• RTX 3060 / 3060 Ti: RTX 3060 12 GB (also sold with 8 GB), RTX 3060 Ti 8 GB.
• RTX 3090: 24 GB.
• RTX 4060 / 4060 Ti: RTX 4060 8 GB, RTX 4060 Ti 16 GB or 8 GB.
• RTX 4070 family: RTX 4070 12 GB.
• RTX 4080 family: 16 GB.
• RTX 4090: 24 GB.
• RTX 5060 family: RTX 5060 8 GB, RTX 5060 Ti 16 GB or 8 GB.
• RTX 5070 family: RTX 5070 12 GB, RTX 5070 Ti 16 GB.
• RTX 5080: 16 GB.
• RTX 5090: 32 GB.
• RTX 40 Series laptops: 4050 6 GB, 4060 8 GB, 4070 8 GB, 4090 16 GB.
• RTX 50 Series laptops: 5060 8 GB, 5070 Ti 12 GB, 5090 24 GB.
• T4 datasheet: 16 GB GDDR6.

Newest (2026) models

Parameter counts are the safetensors totals that Hugging Face reports for each official repo (they include any vision tower and multi-token-prediction layers in the checkpoint). Qwen3.5, Qwen3.6 and Qwen3.8 dense and A3B models are hybrids: only their full-attention layers keep a per-token KV cache, and each linear-attention layer keeps a fixed f32 state of (conv kernel - 1) x (2 x key heads x key size + value heads x value size) + key size x value heads x value size numbers, as in llama.cpp's src/llama-hparams.cpp. For the other new architectures (Gemma 4, Ministral 3, DeepSeek V4, GLM-5.3, gpt-oss, Kimi-K3, Qwen3.8-Flash-Next) the page shows weights only until their cache formula is checked. Pingala-8 needs our patch on llama.cpp 7ac59a6; GLM-5.3-Flash's architecture (glm5-next) was added to llama.cpp after that commit, and DeepSeek-V4.1-Flash is not in llama.cpp master as of f3269ef.

Bits per weight

From the table for meta-llama/Llama-3.1-8B in llama.cpp's tools/quantize/README.md (commit 3f03b17). These are whole-file averages, so they include the higher-precision tensors that every GGUF file keeps.

Real file sizes

KV cache and llama.cpp defaults

• Bytes per cached number: block_q8_0 is 34 bytes per 32 values and block_q4_0 is 18 bytes per 32 values, in ggml/src/ggml-common.h.
• Sliding-window cache size and padding: src/llama-kv-cache-iswa.cpp; Gemma 3 layer pattern (every sixth layer is global): src/models/gemma3.cpp and src/llama-hparams.cpp. llama.cpp turns off sliding windows for Phi models (src/models/phi3.cpp), and Qwen2.5's config has use_sliding_window: false, so those are counted at full size.
• Context rounded up to 256: src/llama-context.cpp. Default batch of 512 tokens, one sequence, swa_full off: common/common.h.