Bag of Words and TF-IDF: Classical Text Representation

Before embeddings and transformers, the workhorse of NLP and search was the vector space model: represent each document by the words it contains. It sounds crude — word order is discarded entirely — yet TF-IDF with a linear classifier remains a surprisingly strong, fast and interpretable baseline, and TF-IDF-style scoring (BM25) still powers many search engines.

Bag of words (BoW)#

Build a vocabulary $V$ of all words in the corpus. Represent each document $d$ as a vector of length $|V|$, where entry $t$ is the count of term $t$ in $d$.

Properties: very high-dimensional (tens of thousands of columns), extremely sparse, and order-free ("dog bites man" = "man bites dog"). Adding n-grams (bigrams like "not good") recovers some local order.

The problem with raw counts#

Common words ("the", "is", "and") dominate counts but carry little meaning. Words appearing in every document do not help distinguish documents. We want to weight terms by how characteristic they are.

TF-IDF#

Term frequency $\text{tf}(t, d)$ measures importance within a document — the raw count, or a dampened version such as $1 + \log(\text{count})$ so that the 50th occurrence matters less than the first.

Inverse document frequency measures rarity across the corpus of $N$ documents, where $\text{df}(t)$ is the number of documents containing $t$:

(Scikit-learn uses a smoothed version $\log\frac{1 + N}{1 + \text{df}(t)} + 1$.) A term in every document gets IDF near its minimum; a term in one document gets a high IDF.

Finally, L2-normalise each document vector so that long documents do not dominate.

Document similarity and retrieval#

With L2-normalised TF-IDF vectors, cosine similarity is just the dot product. To search, embed the query as a TF-IDF vector and rank documents by cosine similarity.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

docs = ["How to register the birth of a newborn child",
        "Vaccination schedule for babies and children",
        "Renewing an expired passport or travel document",
        "Eligibility for monthly cash assistance",
        "Birth certificate replacement if lost"]
vec = TfidfVectorizer(ngram_range=(1, 2), sublinear_tf=True, stop_words="english")
D = vec.fit_transform(docs)
q = vec.transform(["lost birth certificate"])
for i in cosine_similarity(q, D).ravel().argsort()[::-1][:3]:
    print(round(cosine_similarity(q, D[i])[0, 0], 3), docs[i])

Note the limitation: a query "newborn documents" would not match "birth certificate" at all — TF-IDF only matches exact terms. This vocabulary-mismatch problem motivates embeddings and semantic search.

BM25: TF-IDF refined for search#

Okapi BM25, the long-time standard ranking function in search engines, refines TF-IDF with term-frequency saturation and document-length normalisation:

with typical $k_1 \in [1.2, 2]$ and $b = 0.75$. BM25 remains a strong baseline and a component of modern hybrid search (lexical + semantic) used in retrieval-augmented generation.

Text classification with TF-IDF#

TF-IDF + a linear model (logistic regression or linear SVM) is fast, strong and interpretable:

from sklearn.datasets import fetch_20newsgroups
from sklearn.pipeline import make_pipeline
from sklearn.linear_model import LogisticRegression
import numpy as np

train = fetch_20newsgroups(subset="train", categories=["sci.med", "sci.space", "rec.autos"],
                           remove=("headers", "footers", "quotes"))
test = fetch_20newsgroups(subset="test", categories=train.target_names,
                          remove=("headers", "footers", "quotes"))
clf = make_pipeline(TfidfVectorizer(ngram_range=(1, 2), min_df=2, sublinear_tf=True),
                    LogisticRegression(C=10, max_iter=2000))
clf.fit(train.data, train.target)
print("accuracy:", round(clf.score(test.data, test.target), 3))

vocab = clf[0].get_feature_names_out()
for i, c in enumerate(train.target_names):                 # most indicative n-grams per class
    print(c, "->", ", ".join(vocab[np.argsort(clf[1].coef_[i])[-8:]]))

Strengths and limitations#

Strengths: fast, cheap, no GPU; interpretable weights; works well with limited data; strong for topic-like tasks.

Limitations: ignores word order (beyond n-grams) and meaning; no notion that "car" and "automobile" are similar; huge sparse vectors; poor at tasks needing deeper understanding (sarcasm, reasoning, negation scope).