Knobs you can turn after the model is trained

Five levers — what each one actually changes inside the model, and what it costs.

1. Fine-tuning

Full fine-tune vs LoRA vs QLoRA

Full fine-tune updates every weight. Maximum flexibility — the model can change as much as needed. Maximum cost — for a 70B model, you need enough VRAM to hold the model, the gradients, the optimizer state, and the activations. In practice that's often hundreds of GB across multiple GPUs. Risk: if your fine-tune dataset is small or narrow, you can wreck capabilities the model had before (catastrophic forgetting).

LoRA (Low-Rank Adaptation) is the workhorse of practical fine-tuning. Instead of updating the original weight matrix W (size d × d), you freeze W and learn two small matrices A (d × r) and B (r × d) where r is small (typically 4–64). The effective weight at inference time is W + A·B. Because r is tiny, the trainable parameter count drops by 100×–1000×.

QLoRA combines LoRA with 4-bit quantization of the base model. The base weights are frozen and stored in 4 bits; only the LoRA adapter trains in normal precision. This makes it possible to fine-tune a 70B model on a single 24 GB consumer GPU. The trade is some numerical noise during training; in practice quality remains close to LoRA.

A real LoRA config, line by line

from peft import LoraConfig

config = LoraConfig(
    r              = 16,                              # rank — bigger r, more capacity, more params
    lora_alpha     = 32,                              # scaling factor (effective scale = alpha / r)
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj"],  # which weights to adapt
    lora_dropout   = 0.05,                            # regularization during training
    bias           = "none",                          # don't train bias terms
    task_type      = "CAUSAL_LM",                    # decoder-only LM
)

• r=16 — rank of the adapter. Powers of 2 are conventional. r=4 is tiny (good for very narrow tasks); r=64 starts to compete with full fine-tune in capacity.
• lora_alpha=32 — a scaling knob. The effective adapter contribution is (alpha / r) × A·B. Common pattern: alpha = 2×r.
• target_modules — which weight matrices in each layer get the adapter. Attention projections (q/k/v/o) are the standard set; adding MLP weights (gate_proj, down_proj, up_proj) increases capacity but also parameter count.
• lora_dropout=0.05 — regularization. 0.05–0.1 is typical.
• bias="none" — don't train bias parameters. They're tiny anyway and excluded by convention.

2. Quantization

The basic idea

A weight stored in FP16 is a 16-bit floating-point number — about 65,000 distinct possible values. Most weights in a trained LLM cluster near zero with a long tail of larger values. You don't need 65,000 values to represent that distribution well. Quantization picks a smaller set of representative values and rounds every weight to its nearest one.

The quantization zoo

Several methods exist; they differ in how they pick the buckets and which tensors they touch.

• RTN (round-to-nearest) — simplest. Define a uniform grid; round each weight to the nearest grid point. Fast to apply; quality OK at 8-bit, often poor at 4-bit.
• GPTQ — quantize one column of weights at a time, adjusting the not-yet-quantized columns to compensate for the rounding errors. Much better quality at 4-bit. Takes a few hours to apply.
• AWQ (Activation-aware Weight Quantization) — observes typical activation magnitudes and protects the weights that matter most for them. Often beats GPTQ on perplexity, especially at very low bit-depths.
• GGUF — the format used by llama.cpp. Mixes precision: some layers in 4-bit, others in 5- or 6-bit, with various trade-off "quants" you can pick (Q4_K_M, Q5_K_S, etc.). The de-facto standard for running LLMs on consumer hardware.

Where quantization fails

Not all layers tolerate low-bit representation equally. Some empirical patterns:

• Embedding and output layers are often more sensitive — many recipes leave these in higher precision.
• Attention K/V projections seem to tolerate aggressive quantization well.
• The very first and very last few layers of the stack are often more sensitive than the middle.
• 4-bit is the practical floor for general use. 3-bit and 2-bit work for specific models with dedicated methods, but quality drops are real.

3. Context extension

Why context isn't free

Two costs grow with context length:

• Attention compute is O(n²). Every pair of tokens computes a score. Double the context, four times the cost.
• KV cache memory is O(n). Linear in tokens, but with a large constant. For a 70B-class model, one token of cache is ~2 MB — so 128k tokens is ~256 GB of cache.

RoPE scaling — making old position formulas cover new lengths

A model trained with RoPE on 4k context has rotation angles tuned for positions 0–4096. If you naively feed it 16k tokens, the angles for positions 4097–16383 are out of distribution and quality collapses.

The fix: rescale the rotation frequencies. Several flavors:

• Linear (Position Interpolation) — divide all positions by 4 so 16k tokens "fit" into the 0–4k range the model was trained on. Simple; degrades quality somewhat.
• NTK-aware — scale only the high-frequency dimensions, leaving the low-frequency ones (which encode coarse position) alone. Better quality than linear.
• YaRN — refines NTK with a few additional tricks (length-dependent scaling, attention temperature). The default for many recent long-context models.

Often paired with a short fine-tuning run on long-context data to "heal" the model into the new range.

Sliding window attention

An alternative: instead of every token attending to every previous token, each token only attends to the last k tokens (typical k = 4096). Memory and compute become O(n × k) instead of O(n²). Mistral models use this. Trade: tokens far apart can't directly attend to each other, but the residual stream can carry information across the boundary indirectly through stacked layers.

4. Pruning

Unstructured vs structured

Unstructured pruning identifies the smallest individual weights (close to zero) and zeroes them. Easy to do, can prune 50%+ of weights with little quality loss. Catch: most hardware can't actually skip work for sparse matrices unless the sparsity is highly structured (e.g., 2:4 sparsity). So you save disk space and maybe memory, but rarely compute.

Structured pruning removes whole units — entire attention heads, MLP dimensions, even entire layers. Quality drops more sharply but the resulting model is genuinely smaller and faster on standard hardware. Methods like LLM-Pruner score components by their "importance" and remove the lowest-scoring ones.

Healing fine-tune

After any nontrivial pruning, the model needs recovery training: same data as SFT (or even some pretraining data), short run, learning rate small. This re-routes information through the surviving weights. Skipping it usually leaves visible quality damage.

Pruning vs quantization — when to reach for which

For most "make this model smaller / cheaper" goals, quantization is the first tool to reach for — easier, less destructive, more predictable. Pruning becomes useful when:

• You want to specifically reduce compute, not just memory (structured pruning helps; quantization mostly doesn't).
• You're already at 4-bit quantization and need more reduction.
• You want a smaller model that can be fully fine-tuned downstream (pruning gives you fewer parameters to train).

5. Decoding controls

The core controls

• Temperature — divide all logits by T before softmax. T < 1 sharpens the distribution (more deterministic); T > 1 flattens it (more random). T = 0 picks the top token always (greedy decoding, fully deterministic). T = 1 leaves the distribution as the model emitted.
• Top-k — keep only the top K most-likely tokens; renormalize. K = 1 is greedy; K = 50 keeps the top 50 candidates regardless of their probabilities.
• Top-p (nucleus) — keep the smallest set of tokens whose probabilities sum to at least p. p = 1.0 keeps everything; p = 0.9 keeps the top "90% of probability mass," cutting whatever long tail is left.
• Repetition penalty — add a small penalty to tokens that have appeared recently, to prevent loops like "the the the." Usually 1.05–1.15.

Common combinations

KV cache tuning

Already covered in inference.html, but worth noting here as a "knob": for very long contexts you can configure cache behavior — eviction policies (drop oldest, drop least-attended), cache offloading (move parts to CPU), or attention sinks (always keep the first few tokens). All of these trade a small quality cost for memory savings.

Speculative decoding

Also covered in inference.html. The setting you'd see in an inference engine is something like --speculative-model llama-1b alongside the main model. Output is identical to non-speculative decoding; only latency improves. If your serving infra supports it, almost always worth turning on.

Putting it together — diagnosis preview

Once you have all five levers in your head, the diagnostic move is: given a problem, which lever is most likely to fix it?

For the deeper "what broke and why," see the diagnosis page.