Fine-Tuning Pretrained Language Models: A Practical Guide

Pretrained language models have made strong NLP accessible to anyone with a few thousand labelled examples and a GPU. But fine-tuning has pitfalls: unstable runs, overfitting on small data, forgetting, and evaluation mistakes. This lecture collects practical guidance from research and experience into a reliable workflow.

Step 1: choose the right base model#

Consider:

• Architecture: encoder (classification, extraction), encoder–decoder (seq2seq), decoder (generation).
• Language coverage: English-only vs multilingual (XLM-R, mT5) vs language-specific models (e.g. BanglaBERT).
• Domain: general vs biomedical, legal, scientific, social media.
• Size vs latency: DistilBERT/MiniLM for speed; larger models for accuracy.
• Licence: some weights restrict commercial use.
• Tokeniser fertility on your text.

Step 2: prepare data#

• Clean labels; remove duplicates across splits.
• Keep a held-out test set that you touch once.
• Stratify splits; for time-dependent data, split by time.
• Check lengths: truncate or chunk long texts; set max_length to cover most examples.

Step 3: sensible hyperparameters#

Step 4: training loop with Hugging Face#

from datasets import load_dataset
from transformers import (AutoTokenizer, AutoModelForSequenceClassification, TrainingArguments,
                          Trainer, DataCollatorWithPadding, EarlyStoppingCallback)
import numpy as np
from sklearn.metrics import f1_score

ckpt = "xlm-roberta-base"                                     # multilingual encoder
ds = load_dataset("csv", data_files={"train": "train.csv", "validation": "val.csv"})
labels = sorted(set(ds["train"]["label"]))
l2i = {l: i for i, l in enumerate(labels)}
tok = AutoTokenizer.from_pretrained(ckpt)

def prep(batch):
    enc = tok(batch["text"], truncation=True, max_length=256)
    enc["labels"] = [l2i[l] for l in batch["label"]]
    return enc
ds = ds.map(prep, batched=True, remove_columns=ds["train"].column_names)

model = AutoModelForSequenceClassification.from_pretrained(
    ckpt, num_labels=len(labels), id2label=dict(enumerate(labels)), label2id=l2i)

def compute_metrics(p):
    return {"macro_f1": f1_score(p.label_ids, p.predictions.argmax(-1), average="macro")}

args = TrainingArguments("ft-out", learning_rate=2e-5, per_device_train_batch_size=16,
                         gradient_accumulation_steps=2, num_train_epochs=5, warmup_ratio=0.06,
                         weight_decay=0.01, eval_strategy="epoch", save_strategy="epoch",
                         load_best_model_at_end=True, metric_for_best_model="macro_f1",
                         bf16=True, seed=42)
trainer = Trainer(model=model, args=args, train_dataset=ds["train"], eval_dataset=ds["validation"],
                  data_collator=DataCollatorWithPadding(tok), compute_metrics=compute_metrics,
                  callbacks=[EarlyStoppingCallback(early_stopping_patience=2)])
trainer.train()
trainer.save_model("final-model")

Handling small datasets and instability#

Fine-tuning large models on small datasets (a few hundred to a few thousand examples) can be unstable: different random seeds give very different results, and some runs fail entirely (Mosbach et al., 2021; Dodge et al., 2020). Remedies:

• Run several seeds and report mean ± standard deviation; pick the best seed on validation, never on test.
• Use bias-corrected Adam (standard now), warm-up and more training steps — many "failed" runs were simply under-trained.
• Lower the learning rate for larger models.
• Re-initialise the top layer(s) of the encoder, which are most specialised to the pretraining objective.
• Consider SetFit, sentence-embedding classifiers or parameter-efficient fine-tuning (LoRA, adapters), which are often more stable.
• Use intermediate-task transfer: first fine-tune on a related larger dataset (e.g. natural-language inference), then on your task.

Domain- and task-adaptive pretraining#

If your text differs from the pretraining data (clinical notes, humanitarian reports, informal social media), continue pretraining the model with its original objective (MLM) on unlabelled in-domain text before fine-tuning. Gururangan et al. (2020), "Don't Stop Pretraining", showed consistent gains from domain-adaptive (DAPT) and task-adaptive (TAPT) pretraining.

Catastrophic forgetting#

Fine-tuning can erase general capabilities. For single-task deployment this may not matter; for multi-task or continual settings, use lower learning rates, fewer epochs, parameter-efficient methods, or mix in general data.

Evaluate properly#

• Use task-appropriate metrics (macro-F1 for imbalance, entity-level F1 for NER).
• Report variance across seeds.
• Evaluate on slices: languages, dialects, text lengths, sources.
• Compare against simple baselines (TF-IDF + logistic regression) and against zero-shot LLM prompting.
• Check calibration if probabilities drive decisions.
• Error analysis: read the mistakes.