LLM Fine Tuning: Methods, Training Data, and Evaluation

LLM fine tuning is the process of taking a pre-trained foundation model and updating its weights on a smaller, task-specific dataset. Done well, it produces a model that costs less to serve, responds faster, and behaves consistently on your specific use case. Done poorly, it produces a model that looks fine in eval and regresses silently the week after deployment. The difference is usually in how training data is structured and whether evaluation gates exist before anything ships.

Picking a method: SFT, LoRA, or DPO

Before any data work, you need a method. The landscape has consolidated around three main options.

Supervised Fine-Tuning (SFT) trains on prompt-completion pairs. You provide examples of the input and the desired output; the model adjusts weights to fit that distribution. It works well for output format adherence, domain vocabulary, and task specialization. The limitation: SFT alone doesn’t encode preference — it tells the model that the training examples are correct, but doesn’t teach it which of two valid responses is better.

LoRA (Low-Rank Adaptation) is the default approach for adapter-based fine-tuning. Rather than updating all parameters, LoRA injects small trainable matrices into attention layers and keeps the base model frozen. The result: a 7B model fine-tune runs on a single 16–24 GB GPU instead of an A100 cluster. Hugging Face’s TRL + PEFT work ↗ demonstrated this concretely by running RLHF on a 20B-parameter model on a single 24 GB consumer GPU — a setup that would have required multiple A100s previously. QLoRA extends this by quantizing the frozen base weights to 4-bit (NF4 format), cutting memory footprint another 60–70%.

Direct Preference Optimization (DPO) is now the default for alignment-style training, replacing RLHF in most production pipelines. Instead of training a separate reward model and running PPO, DPO trains directly on preference pairs — a chosen response and a rejected response for the same prompt. Per Fireworks AI’s fine-tuning guide ↗, DPO consistently delivers comparable or better alignment quality at significantly lower operational complexity. Removing the reward model also removes one of the most fragile intermediaries in the training pipeline.

Decision logic that holds in practice: if you need output format consistency or domain task specialization, start with SFT + LoRA. If you need behavioral alignment — the model should prefer certain response styles or refuse specific output patterns — add a DPO pass after SFT. Full fine-tuning (updating all weights) is worth considering only when adapter methods hit a quality ceiling, which happens occasionally on tasks requiring deep architectural change but rarely in ordinary production specialization.

Training data: where most fine-tuning projects actually fail

Databricks’ practical guide to LLM fine tuning ↗ states this directly: “a smaller dataset of high-quality examples consistently outperforms a larger dataset with noisy data.” This pattern holds across published benchmarks and, more importantly, across real production projects.

What “high quality” means operationally:

Distribution match. Training data must look like production inputs. If production queries have a specific vocabulary, length distribution, or reasoning pattern, the training set needs to reflect it. The most common failure mode is training on synthetic GPT-4 outputs and serving real user traffic that looks nothing like them — the model overfits the synthetic distribution and falls apart on real inputs.

Schema consistency. Every example should follow the same prompt template. If your dataset mixes instruction formats — some starting with “You are a…” and others with “Given the following…” — the model treats template variation as signal, not your actual task. Pick a format and apply it to every example.

Output quality gates before training. Sample 5–10% of training completions and score them manually before the training run starts. Automated filtering via LLM-as-judge works at scale, but blind trust in it introduces subtle data quality problems that appear as training noise, not obvious errors.

For DPO specifically, the format is triplets: (prompt, chosen completion, rejected completion). Getting good rejected completions is harder than it looks — random samples from the base model aren’t diverse enough to teach meaningful preference. The higher-quality approach is to generate multiple completions per prompt and have domain experts rank them.

Dataset size thresholds that hold for 7B–13B LoRA fine-tunes:

Goal	Minimum examples
Style / tone / output format	200–500
Domain vocabulary adaptation	500–1,500
Task specialization (extraction, classification)	1,000–5,000
Instruction following in a new domain	2,000–10,000

Beyond 10K examples, quality gates matter more than volume. A 5,000-example dataset with strict quality review consistently outperforms 50,000 scraped-and-filtered examples on held-out eval.

Pre-deployment evaluation: what to check before you ship

The comprehensive arXiv survey on fine-tuning methods ↗ covers a seven-stage pipeline from data preparation through deployment, and consistently identifies post-training validation as the weakest link. The typical pattern: teams look at training loss, see it decrease, and ship. That is not evaluation.

Evaluation before deployment should cover three dimensions:

Task eval on a held-out set. Reserve 10–15% of your labeled data and never let it touch training. Score on your actual task metrics — F1 for classification, exact match or ROUGE for generation, schema validation pass rate for structured output. If you don’t have defined task metrics before training starts, you don’t have a fine-tuning project yet.

Behavioral regression on a golden set. A fixed set of prompts with known expected behavior: adversarial inputs, edge cases, known failure patterns from the base model. This set never changes, so pass rate is comparable across every model version. A fine-tuned model that improves task eval scores but drops 5 points on the golden set has learned to pass tests, not to behave better. Score pass/fail, not subjective ratings.

Safety delta against the base model. Fine-tuning narrow datasets can degrade base model safety behaviors in ways that don’t show up in task metrics. Run your safety and refusal evals against both the base model and the fine-tuned adapter and compare. guardml.io ↗ covers guardrail tooling that can be wired into a pre-deployment CI pipeline for automated safety scoring. For the offensive angle — how fine-tuned adapters change the model’s attack surface under adversarial prompts — aisec.blog ↗ tracks the current research on adapter-layer exploits and safety degradation.

Automated evals should run in CI on every training checkpoint. If pass rate on the golden set drops between checkpoints, stop training and investigate before continuing. A checkpoint with low training loss can still fail behavioral requirements — treating loss as a proxy for quality is the single most common evaluation mistake.

On training hyperparameters worth tracking as part of your eval pipeline: log lora_r (rank), lora_alpha, learning rate, and which target modules receive adapters. These are not one-time decisions — they interact with dataset size and model architecture in ways that only become apparent across multiple training runs. Keeping them in your experiment tracker (MLflow, W&B) makes root-cause analysis tractable when a training run produces unexpected results.

---

Sources

• LLM Fine-Tuning: A Comprehensive Review (arXiv 2408.13296) ↗ — Survey covering supervised, instruction-based, and parameter-efficient fine-tuning across the full pipeline from data preparation through deployment validation. Includes a seven-stage pipeline model and coverage of PPO, DPO, and LoRA-based approaches.

• LLM Fine-Tuning: Deep Dive & Best Practices — Fireworks AI ↗ — Production-focused breakdown of SFT, DPO, RLHF, and RLVR with decision frameworks for choosing between alignment methods and data volume thresholds for each approach.

• A Practical Guide to LLM Fine Tuning — Databricks ↗ — Covers data quality requirements, model selection criteria, continuous monitoring strategy, and the argument for quality-over-quantity in training data curation.

• Fine-tuning 20B LLMs with RLHF on a 24GB consumer GPU — Hugging Face ↗ — Practical demonstration of the TRL + PEFT stack enabling RLHF at dramatically reduced GPU requirements via LoRA adapters.