Fine-Tuning LLMs: A Practical Guide

Fine-tuning a large language model means taking a pretrained model and continuing its training on a smaller, task-specific dataset. The goal is to shift the model’s behavior toward your use case without paying the cost of training from scratch. This post covers the practical path: when fine-tuning makes sense, how to prepare data, and how to run training with modern parameter-efficient methods.

When fine-tuning is the right call

Prompting is almost always the first thing to try. It is faster, cheaper, and easier to iterate on. Fine-tuning becomes worth considering when:

• You need the model to follow a specific output format consistently and prompt engineering cannot get it reliable enough.
• You have a domain where the pretrained model lacks knowledge (medical codes, internal jargon, proprietary schemas).
• You need to reduce latency by eliminating long system prompts and few-shot examples.
• You want to distill a larger model’s behavior into a smaller, cheaper one.

If you can solve the problem with a better prompt, do that instead. Fine-tuning is an investment with real costs in data preparation, compute, and ongoing maintenance.

Full fine-tuning vs parameter-efficient methods

Full fine-tuning updates every weight in the model. For a 7B parameter model, that means storing and updating 7 billion floats, which requires multiple high-end GPUs and significant memory.

Parameter-efficient fine-tuning (PEFT) methods freeze most of the model and train only a small number of additional parameters. This dramatically reduces compute and memory requirements while achieving comparable results for most tasks.

LoRA (Low-Rank Adaptation)

LoRA injects small trainable matrices into the model’s attention layers. Instead of updating a weight matrix W directly, LoRA decomposes the update into two smaller matrices: W + BA, where B and A have much lower rank than W.

from peft import LoraConfig, get_peft_model, TaskType

lora_config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=16,                        # rank of the decomposition
    lora_alpha=32,               # scaling factor
    lora_dropout=0.05,           # dropout on LoRA layers
    target_modules=[             # which layers to adapt
        "q_proj",
        "k_proj",
        "v_proj",
        "o_proj",
        "gate_proj",
        "up_proj",
        "down_proj",
    ],
    bias="none",
)

The rank r controls expressiveness. Higher rank means more trainable parameters and more capacity to learn, but also more memory. For most tasks, r=8 to r=32 works well. The lora_alpha parameter controls the scaling of the LoRA update and is typically set to 2x the rank.

QLoRA (Quantized LoRA)

QLoRA takes LoRA further by loading the base model in 4-bit quantized form. The base weights consume roughly a quarter of the memory, while the LoRA adapter weights remain in full precision for training stability.

from transformers import BitsAndBytesConfig
import torch

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",          # normalized float 4-bit
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,     # nested quantization
)

With QLoRA, you can fine-tune a 7B model on a single 24GB GPU (like an RTX 4090 or A10G). A 13B model fits on a single 48GB GPU (A6000 or A40).

Dataset preparation

The quality of your dataset matters more than its size. A few thousand high-quality examples often outperform tens of thousands of noisy ones.

Format: instruction-response pairs

Most fine-tuning uses an instruction-following format. Each example has an instruction (what the model should do) and a response (the ideal output).

# training_data.jsonl - one JSON object per line
{"instruction": "Classify the sentiment of this review as positive, negative, or neutral.", "input": "The battery life is incredible but the screen is too dim outdoors.", "output": "mixed"}
{"instruction": "Extract the company name and ticker from this text.", "input": "Apple Inc. (AAPL) reported quarterly earnings today.", "output": "{\"company\": \"Apple Inc.\", \"ticker\": \"AAPL\"}"}

Applying chat templates

Models expect data formatted according to their chat template. Using the tokenizer’s built-in template ensures consistency.

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")

def format_example(example):
    messages = [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": example["instruction"]},
        {"role": "assistant", "content": example["output"]},
    ]
    return tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=False,
    )

Building the dataset

from datasets import load_dataset

dataset = load_dataset("json", data_files="training_data.jsonl", split="train")

# apply formatting
dataset = dataset.map(
    lambda example: {"text": format_example(example)},
    remove_columns=dataset.column_names,
)

# split into train and validation
split = dataset.train_test_split(test_size=0.1, seed=42)
train_dataset = split["train"]
eval_dataset = split["test"]

print(f"Train: {len(train_dataset)}, Eval: {len(eval_dataset)}")

Data quality checks

Before training, verify your dataset:

# check for duplicates
texts = train_dataset["text"]
unique = set(texts)
print(f"Total: {len(texts)}, Unique: {len(unique)}, Dupes: {len(texts) - len(unique)}")

# check token length distribution
lengths = [len(tokenizer.encode(t)) for t in texts]
print(f"Min: {min(lengths)}, Max: {max(lengths)}, Mean: {sum(lengths)/len(lengths):.0f}")

# check for examples that exceed context window
max_len = 4096
too_long = sum(1 for l in lengths if l > max_len)
print(f"Examples exceeding {max_len} tokens: {too_long}")

Training with Hugging Face and PEFT

Here is a complete training script using QLoRA:

import torch
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    TrainingArguments,
    BitsAndBytesConfig,
)
from peft import LoraConfig, get_peft_model, TaskType, prepare_model_for_kbit_training
from trl import SFTTrainer

# model and tokenizer
model_name = "meta-llama/Llama-3.1-8B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_name)
tokenizer.pad_token = tokenizer.eos_token

# quantization config
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

# load model
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    quantization_config=bnb_config,
    device_map="auto",
    torch_dtype=torch.bfloat16,
)
model = prepare_model_for_kbit_training(model)

# LoRA config
lora_config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    bias="none",
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: 13,631,488 || all params: 8,043,900,928 || trainable%: 0.1695

# training arguments
training_args = TrainingArguments(
    output_dir="./checkpoints",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,
    weight_decay=0.01,
    warmup_ratio=0.03,
    lr_scheduler_type="cosine",
    logging_steps=10,
    save_strategy="epoch",
    evaluation_strategy="epoch",
    bf16=True,
    optim="paged_adamw_8bit",
    report_to="wandb",
    gradient_checkpointing=True,
    max_grad_norm=0.3,
)

# trainer
trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    tokenizer=tokenizer,
    dataset_text_field="text",
    max_seq_length=4096,
    packing=True,  # pack multiple short examples into one sequence
)

trainer.train()
trainer.save_model("./final_model")

Key hyperparameters to tune

Parameter	Typical Range	Notes
Learning rate	1e-5 to 3e-4	Higher for LoRA than full fine-tuning
Epochs	1-5	Watch for overfitting on small datasets
Batch size	4-32 (effective)	Use gradient accumulation to reach target
LoRA rank	8-64	Higher rank = more capacity, more memory
Warmup ratio	0.03-0.1	Short warmup usually sufficient

Evaluating the fine-tuned model

Loss curves

Monitor training and validation loss. If validation loss starts increasing while training loss continues to drop, you are overfitting.

# after training, plot losses
import matplotlib.pyplot as plt

history = trainer.state.log_history
train_loss = [(h["step"], h["loss"]) for h in history if "loss" in h]
eval_loss = [(h["step"], h["eval_loss"]) for h in history if "eval_loss" in h]

plt.figure(figsize=(10, 5))
plt.plot(*zip(*train_loss), label="Train Loss")
plt.plot(*zip(*eval_loss), label="Eval Loss", marker="o")
plt.xlabel("Step")
plt.ylabel("Loss")
plt.legend()
plt.title("Training and Evaluation Loss")
plt.savefig("loss_curves.png")

Task-specific evaluation

Loss alone does not tell you if the model is better at your task. Run task-specific evaluations:

from peft import PeftModel

# load the fine-tuned model
base_model = AutoModelForCausalLM.from_pretrained(
    model_name,
    quantization_config=bnb_config,
    device_map="auto",
)
model = PeftModel.from_pretrained(base_model, "./final_model")

def generate(prompt, max_new_tokens=256):
    messages = [{"role": "user", "content": prompt}]
    input_text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    inputs = tokenizer(input_text, return_tensors="pt").to(model.device)
    
    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            temperature=0.1,
            do_sample=True,
        )
    
    return tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)

# run on test examples
test_cases = [
    {"input": "The food was amazing but service was slow.", "expected": "mixed"},
    {"input": "Terrible experience. Will never return.", "expected": "negative"},
    {"input": "It was okay, nothing special.", "expected": "neutral"},
]

correct = 0
for case in test_cases:
    result = generate(f"Classify the sentiment: {case['input']}")
    match = case["expected"].lower() in result.lower()
    correct += int(match)
    print(f"Input: {case['input']}")
    print(f"Expected: {case['expected']}, Got: {result.strip()}, Match: {match}\n")

print(f"Accuracy: {correct}/{len(test_cases)}")

Comparing against the base model

Always compare your fine-tuned model against the base model on the same test set. If the fine-tuned model is not measurably better, you may need more data, better data, or different hyperparameters.

Merging and deploying

Once you are satisfied with the fine-tuned model, merge the LoRA weights back into the base model for easier deployment:

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

# load in full precision for merging
base_model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="cpu",
)

model = PeftModel.from_pretrained(base_model, "./final_model")
merged_model = model.merge_and_unload()

# save merged model
merged_model.save_pretrained("./merged_model")
tokenizer.save_pretrained("./merged_model")

# optionally push to Hugging Face Hub
merged_model.push_to_hub("your-username/your-model-name")
tokenizer.push_to_hub("your-username/your-model-name")

Common pitfalls

Overfitting on small datasets. If you have fewer than 1,000 examples, use aggressive regularization: higher dropout, lower rank, fewer epochs. Watch validation loss like a hawk.

Wrong chat template. If you format training data differently from how the model was originally trained, the model may learn the wrong patterns. Always use the tokenizer’s built-in template.

Catastrophic forgetting. Fine-tuning too aggressively on a narrow task can degrade the model’s general capabilities. LoRA naturally mitigates this since most weights are frozen, but it can still happen with high learning rates or too many epochs.

Evaluating on training data. Always hold out a test set that was never seen during training. This sounds obvious but is easy to mess up when examples are similar.

Fine-tuning is a tool, not a magic solution. Start with good prompts, move to fine-tuning when you have evidence it will help, and invest heavily in data quality. The model can only learn what you teach it.