[reference] — from the roadmap.sh machine-learning roadmap. The starting point of the algorithm zoo, and often the right ending point too.
Linear regression — the foundation
Fit a straight line (or hyperplane) through the data: predict y = w₁x₁ + w₂x₂ + … + b, choosing weights w that minimize the squared error between predictions and actuals (optimization). Simple, fast, and interpretable — each weight tells you how much that feature moves the prediction.
from sklearn.linear_model import LinearRegression
model = LinearRegression().fit(X_train, y_train)
model.coef_ # the weights — directly interpretable
model.predict(X_test)It’s the baseline you should almost always try first on a regression problem: if a linear model does well, you’re done, and you have an interpretable result. Assumes a roughly linear relationship — its main limitation.
Polynomial regression — curved relationships
Still linear in the weights, but you add powers of features (x², x³) so the model can fit curves. Powerful but the classic overfitting trap: a high enough degree fits the training points perfectly and generalizes terribly (overfitting). Degree is a hyperparameter to tune, not maximize.
Logistic regression — classification, despite the name
Despite “regression” in the name, this is the classification workhorse. It runs a linear combination through a sigmoid to output a probability (0–1), then thresholds it into a class:
from sklearn.linear_model import LogisticRegression
model = LogisticRegression().fit(X_train, y_train)
model.predict_proba(X_test) # calibrated-ish probabilities, not just labelsFast, interpretable, outputs probabilities (useful when you care about confidence, not just the label), and a genuinely strong baseline for binary classification. Extends to multi-class. Trained by minimizing log-loss (log-loss) rather than squared error.
Regularization — the cure for overfitting
The single most important idea attached to linear models: add a penalty on large weights to the loss, discouraging the model from fitting noise. Three flavors:
| Method | Penalty | Effect |
|---|---|---|
| Ridge (L2) | sum of squared weights | shrinks all weights toward zero, none exactly zero — handles correlated features well |
| Lasso (L1) | sum of absolute weights | drives some weights to exactly zero → automatic feature selection |
| Elastic Net | mix of L1 + L2 | best of both; the pragmatic default when unsure |
from sklearn.linear_model import Ridge, Lasso
Ridge(alpha=1.0).fit(X_train, y_train) # alpha = regularization strength, a key hyperparameterRegularization is the concrete mechanism behind the bias-variance tradeoff (overfitting & regularization): more penalty = simpler model = more bias, less variance. alpha is tuned via cross-validation. The same L1/L2 idea reappears in deep learning as weight decay.
When regression is the right tool
For structured/tabular data, a regularized linear or logistic model is often competitive with far fancier methods, trains instantly, and is interpretable — a real advantage when you need to explain a decision (explainable AI). Reach past it to trees/ensembles when relationships are strongly non-linear, or to deep learning for unstructured data (text/images).
Related
- Overfitting & Regularization — what regularization fights
- Trees & Ensembles — the non-linear alternative
- Optimization — how the weights are fit
Software engineering
Infrastructure & security
Computer science
Hardware & engineering