ML Pipeline • Regularisation

Ridge Regression Workflow

From raw data to coefficient interpretation — a complete 11-step pipeline with cross-validation, feature scaling, diagnostics, and model evaluation.

📐 Overview: This pipeline walks through a rigorous Ridge Regression workflow. Ridge (L2 regularisation) shrinks coefficients to prevent overfitting — but only works correctly when features are scaled and the regularisation strength (α) is properly tuned via cross-validation.

🔗 Key Insight: Every step exists for a reason. Skipping feature scaling or using a single validation set instead of CV can produce misleading results. This guide explains why each step matters.

🎯 Goal: Build a robust Ridge model that generalises well, diagnose its performance honestly, and interpret coefficients in original units.

🔄 Complete Ridge Pipeline Flowchart Infographic

[START] Raw Data (X, y)

▼

2. Split → Train / Test

▼

3. Feature Scaling (StandardScaler)
fit on train, transform both

▼

4. Choose α candidates (logspace)

▼

5. RidgeCV on TRAIN → best α

▼

6. Train final Ridge on full TRAIN + best α

▼

7. Predict on TEST set

▼

8. Compute Metrics
RMSE, MAE, R², Adj R²

10. Diagnostic Plots
residuals, true vs predicted

▼

9. Compare with Baseline
OLS or simple mean

▼

11. Interpret Coefficients
scale back to original units

→

[END]

Step 1 Raw Data (X, y) — The Starting Point

Load your dataset. Ensure features X and target y are clean and separated.

step1_load_data.py

import numpy as np
from sklearn.datasets import make_regression
X, y = make_regression(n_samples=500, n_features=10, noise=15, random_state=42)
print(f"X shape: {X.shape}, y shape: {y.shape}")

🔍 Why this matters: Starting point. Ensure no data leakage. Any preprocessing that uses global statistics must be done after splitting.

Step 2 Split Data → Train / Test

Split before any scaling or preprocessing. The test set simulates unseen data and must remain untouched until final evaluation.

step2_split.py

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)
print(f"Train: {X_train.shape}, Test: {X_test.shape}")

🔍 Why this matters: Essential to estimate generalisation error. Without it, you overfit to training data and get over‑optimistic metrics. The test set is your honest judge.

Step 3 Feature Scaling (StandardScaler) ⚠️ Critical for Ridge

Ridge penalises coefficient magnitudes directly. If features have different scales, larger-scale features are unfairly shrunk. Fit on train only, then transform both.

step3_scale.py

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)   # fit + transform train
X_test_scaled = scaler.transform(X_test)          # transform only test

# Verify: train mean ≈ 0, std ≈ 1
print(f"Train mean: {X_train_scaled.mean(axis=0)[:3]}")
print(f"Train std:  {X_train_scaled.std(axis=0)[:3]}")

🔍 Why this matters: Ridge penalises coefficient magnitudes directly. If features have different scales, larger‑scale features are unfairly shrunk. Scaling ensures the penalty treats all features equally.

Step 4 Choose Alpha Candidate Range (logspace)

A wide log-spaced range (e.g., 10⁻⁴ to 10⁴) captures both weak and strong regularisation without blind guessing. RidgeCV will search this grid efficiently.

step4_alphas.py

alphas = np.logspace(-4, 4, 50)  # 50 values from 1e-4 to 1e4
print(f"Alpha range: {alphas[0]:.4f} → {alphas[-1]:.0f}")
print(f"Number of candidates: {len(alphas)}")

🔍 Why this matters: A wide log‑spaced range (e.g., 1e-4 to 1e4) captures both weak and strong regularisation without blind guessing. Too narrow a range may miss the optimal α.

Step 5 Cross-Validation (RidgeCV) on TRAIN Set → Selects Best α

RidgeCV performs k-fold cross-validation on the training set only. It finds the alpha that minimises average test error across folds — robust against overfitting a single validation split.

step5_ridgecv.py

from sklearn.linear_model import RidgeCV

ridge_cv = RidgeCV(alphas=alphas, cv=5, scoring='neg_mean_squared_error')
ridge_cv.fit(X_train_scaled, y_train)

best_alpha = ridge_cv.alpha_
print(f"Best alpha from CV: {best_alpha:.6f}")

🔍 Why this matters: Finds the alpha that minimises test error on unseen folds. This avoids overfitting the validation set and gives robust alpha selection. 5-fold or 10-fold CV is standard.

Step 6 Train Final Ridge Model on Full TRAIN Set Using Best α

Once alpha is chosen, using all training data (not just CV folds) yields the most stable coefficient estimates. This is your production model.

step6_final_ridge.py

from sklearn.linear_model import Ridge

final_ridge = Ridge(alpha=best_alpha)
final_ridge.fit(X_train_scaled, y_train)

print(f"Intercept: {final_ridge.intercept_:.4f}")
print(f"Coefficients (first 5): {final_ridge.coef_[:5]}")

🔍 Why this matters: Once alpha is chosen, using all training data (not just CV folds) yields the most stable coefficient estimates. Don't retrain on train+test — that would invalidate your evaluation.

Step 7 Predict on TEST Set

This is the moment of truth. Generate predictions on the held-out test set to get an unbiased estimate of real-world performance.

step7_predict.py

y_pred = final_ridge.predict(X_test_scaled)

# Quick peek
comparison = pd.DataFrame({'Actual': y_test[:5], 'Predicted': y_pred[:5]})
print(comparison)

🔍 Why this matters: The final, unbiased estimate of how the model will perform on new, unseen data. Any further tuning based on test performance constitutes data leakage.

Step 8 Compute Performance Metrics

Multiple metrics give a fuller picture. RMSE penalises large errors, MAE is robust to outliers, R² measures variance explained, and Adjusted R² accounts for feature count.

📊 Key Metrics Reference

RMSE

Root Mean Sq Error
Penalises large errors

MAE

Mean Absolute Error
Robust to outliers

R²

Variance explained
0 to 1 (higher better)

Adj R²

R² adjusted for
number of predictors

step8_metrics.py

from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score

rmse = np.sqrt(mean_squared_error(y_test, y_pred))
mae = mean_absolute_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)
n, p = X_test.shape[0], X_test.shape[1]
adj_r2 = 1 - (1 - r2) * (n - 1) / (n - p - 1)

print(f"RMSE:     {rmse:.4f}")
print(f"MAE:      {mae:.4f}")
print(f"R²:       {r2:.4f}")
print(f"Adj R²:   {adj_r2:.4f}")

🔍 Why this matters: Quantifies prediction quality in different ways (RMSE for scale, MAE for robustness, R² for variance explained, etc.). Multiple metrics give a fuller picture — don't rely on just one.

Step 9 Compare with Baseline (OLS or Simple Mean)

A model is only useful if it beats simple benchmarks. Compare Ridge against OLS (no regularisation) and a dummy mean predictor to validate added complexity.

step9_baseline.py

from sklearn.linear_model import LinearRegression
from sklearn.dummy import DummyRegressor

# OLS baseline
ols = LinearRegression()
ols.fit(X_train_scaled, y_train)
ols_pred = ols.predict(X_test_scaled)
ols_rmse = np.sqrt(mean_squared_error(y_test, ols_pred))

# Dummy mean baseline
dummy = DummyRegressor(strategy='mean')
dummy.fit(X_train_scaled, y_train)
dummy_pred = dummy.predict(X_test_scaled)
dummy_rmse = np.sqrt(mean_squared_error(y_test, dummy_pred))

print(f"Ridge RMSE:  {rmse:.4f}")
print(f"OLS RMSE:    {ols_rmse:.4f}")
print(f"Dummy RMSE:  {dummy_rmse:.4f} (mean predictor)")

🔍 Why this matters: A model is only useful if it beats simple benchmarks (e.g., predicting the mean). This step validates that the added complexity of Ridge is actually worthwhile.

Step 10 Diagnostic Plots (Residuals, True vs Predicted)

Metrics can hide problems. Diagnostic plots reveal patterns like heteroscedasticity, non-linearity, or outliers that a single number cannot capture.

📈 Diagnostic Plot Types Visual

Residuals vs Fitted

True vs Predicted

Residual Histogram

Q-Q Plot

step10_diagnostics.py

import matplotlib.pyplot as plt

residuals = y_test - y_pred

fig, axes = plt.subplots(1, 3, figsize=(16, 5))

# 1. Residuals vs Fitted
axes[0].scatter(y_pred, residuals, alpha=0.6, edgecolors='k', linewidth=0.3)
axes[0].axhline(y=0, color='r', linestyle='--')
axes[0].set_xlabel('Fitted Values')
axes[0].set_ylabel('Residuals')
axes[0].set_title('Residuals vs Fitted')

# 2. True vs Predicted
axes[1].scatter(y_test, y_pred, alpha=0.6, edgecolors='k', linewidth=0.3)
axes[1].plot([y_test.min(), y_test.max()], [y_test.min(), y_test.max()], 'r--')
axes[1].set_xlabel('True Values')
axes[1].set_ylabel('Predicted Values')
axes[1].set_title('True vs Predicted')

# 3. Histogram of Residuals
axes[2].hist(residuals, bins=30, edgecolor='k', alpha=0.7)
axes[2].axvline(x=0, color='r', linestyle='--')
axes[2].set_xlabel('Residual')
axes[2].set_title('Residual Distribution')

plt.tight_layout()
plt.show()

🔍 Why this matters: Reveals patterns in residuals (e.g., heteroscedasticity, non‑linearity) that metrics alone miss. A poorly specified model can still have good RMSE — plots catch what numbers hide.

Step 11 Interpret Coefficients (Scale Back to Original Units)

Ridge coefficients from scaled data are in standard deviation units. To interpret them in original units (e.g., "a 1-unit increase in X₁ changes y by β₁"), you must unscale them.

step11_interpret.py

# Coefficients on scaled data
coef_scaled = final_ridge.coef_

# Unscale: divide each coefficient by its feature's std
coef_original = coef_scaled / scaler.scale_

# Create interpretable DataFrame
coef_df = pd.DataFrame({
    'Feature': [f'X{i}' for i in range(len(coef_original))],
    'Coefficient (scaled)': coef_scaled,
    'Coefficient (original units)': coef_original,
    'Abs Importance': np.abs(coef_original)
}).sort_values('Abs Importance', ascending=False)

print(coef_df)

🔍 Why this matters: In business contexts, understanding which features drive predictions is as important as accuracy. Scaling must be reversed to interpret coefficients on original units — "a 1-unit increase in square footage adds $X to price."

Step-by-Step Summary: Why Each Step Matters

Step	What	Why It's Critical
1	Raw Data	Starting point. Ensure no data leakage.
2	Train/Test Split	Essential to estimate generalisation error.
3	Feature Scaling	Ridge penalises coefficient magnitudes. Unequal scales → unfair shrinkage.
4	Alpha Candidates	Wide log‑spaced range captures weak & strong regularisation.
5	Cross‑Validation	Finds alpha that minimises error on unseen folds.
6	Train Final Ridge	Using all training data yields stable coefficient estimates.
7	Predict on Test	Final unbiased estimate of real-world performance.
8	Compute Metrics	Multiple metrics give a fuller picture.
9	Compare Baseline	Validates added complexity is worthwhile.
10	Diagnostic Plots	Reveals patterns metrics miss.
11	Interpret Coefficients	Business value: understand drivers in original units.