Encoding Categorical Variables: One-Hot, Ordinal, Target and Beyond

District, occupation, language, product type, blood group — tabular data is full of categorical variables. Most algorithms require numeric input, so we must encode categories as numbers. The naive approach — assigning integers 1, 2, 3 — can quietly damage a model. Choosing the right encoding depends on the variable's nature, its cardinality (number of distinct values) and the model.

Nominal vs ordinal#

• Nominal categories have no order: colours, countries, languages.
• Ordinal categories have a natural order: education level, severity (mild < moderate < severe), income bracket.

Ordinal (integer) encoding#

Map categories to integers. Appropriate for ordinal variables in the correct order.

One-hot encoding#

Create one binary column per category:

• The default for nominal variables with low cardinality.
• For linear models with an intercept, drop one column (or rely on regularisation) to avoid perfect collinearity — the "dummy variable trap".
• Handle unseen categories at prediction time (handle_unknown="ignore" produces an all-zero row).
• With high cardinality (thousands of values), one-hot creates huge sparse matrices and rare columns with little data. Group rare categories into "Other" (min_frequency in scikit-learn).

Frequency / count encoding#

Replace each category with how often it appears. Simple, one column, and often useful for trees (frequent vs rare categories behave differently). Different categories with equal counts become indistinguishable.

Target (mean) encoding#

Replace each category with the mean target for that category — e.g. the default rate of each occupation. Compact and powerful for high-cardinality features, but dangerous:

• Leakage: if a row's own label contributes to its encoding, the model sees the answer. Rare categories become near-perfect predictors on training data and fail on test data.
• Noise: a category seen twice has an unreliable mean.

Remedies:

1. Out-of-fold encoding — compute each row's encoding from other folds only (scikit-learn's TargetEncoder does this with cross-fitting).
2. Smoothing — shrink towards the global mean:

where $n_c$ is the category count and $m$ controls the strength of shrinkage — a Bayesian estimate with a prior centred on the global mean.

1. Ordered target statistics (CatBoost) — encode each row using only earlier rows in a random permutation.

Hashing encoding#

Map categories to a fixed number of columns with a hash function. Memory is fixed regardless of cardinality, and new categories need no dictionary — useful for streaming data and huge vocabularies. The cost is collisions (different categories sharing a column) and loss of interpretability.

Learned embeddings#

Neural networks can learn a dense vector for each category via an embedding layer, as they do for words. Similar categories (e.g. products bought together) end up close in embedding space. Entity embeddings are especially effective for high-cardinality features in deep tabular models, and the learned vectors can be reused in other models.

Comparison#

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, TargetEncoder
from sklearn.pipeline import make_pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score

rng = np.random.default_rng(0)
n = 5000
occupations = [f"occ_{i}" for i in range(300)]              # high cardinality
occ_risk = dict(zip(occupations, rng.normal(0, 1, 300)))
df = pd.DataFrame({
    "education": rng.choice(["primary", "secondary", "tertiary"], n),
    "district": rng.choice(["Dhaka", "Chattogram", "Sylhet", "Khulna"], n),
    "occupation": rng.choice(occupations, n),
})
logit = df["occupation"].map(occ_risk) + (df["education"] == "tertiary") * 0.8
y = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)

prep = ColumnTransformer([
    ("edu", OrdinalEncoder(categories=[["primary", "secondary", "tertiary"]]), ["education"]),
    ("dist", OneHotEncoder(handle_unknown="ignore"), ["district"]),
    ("occ", TargetEncoder(target_type="binary", random_state=0), ["occupation"]),  # cross-fitted
])
pipe = make_pipeline(prep, LogisticRegression(max_iter=1000))
print("CV AUC:", cross_val_score(pipe, df, y, cv=5, scoring="roc_auc").mean().round(3))