[reference] — from the roadmap.sh machine-learning roadmap. Three more classic supervised algorithms, each with a distinct idea worth knowing and a niche where it shines.

Support Vector Machines (SVM)

An SVM finds the decision boundary that maximizes the margin — the widest possible gap between the two classes. The points sitting on the edge of that gap (the “support vectors”) are the only ones that matter; the rest don’t affect the boundary.

Its superpower is the kernel trick: by implicitly mapping data into a higher-dimensional space, an SVM can find a linear boundary there that corresponds to a complex non-linear boundary in the original space — without ever computing the high-dimensional coordinates.

from sklearn.svm import SVC
SVC(kernel="rbf", C=1.0).fit(X_train, y_train)   # rbf = the common non-linear kernel

Strong on small-to-medium datasets with clear margins and high-dimensional data (text classification historically). The catch: doesn’t scale to very large datasets (training is roughly quadratic), needs feature scaling, and the kernel/C/gamma choices need tuning. Largely superseded by gradient boosting for tabular and by deep learning for text/images, but conceptually important and still useful on the right small problem.

K-Nearest Neighbors (KNN)

The laziest algorithm, and the most intuitive: to classify a new point, find its k nearest points in the training data and take the majority vote (or average, for regression). There’s no training — it just stores the data and does all the work at prediction time.

from sklearn.neighbors import KNeighborsClassifier
KNeighborsClassifier(n_neighbors=5).fit(X_train, y_train)
  • Simple, no assumptions about the data’s shape, naturally handles multi-class.
  • Prediction is slow (must compare to all training points) and it degrades in high dimensions (the “curse of dimensionality” — distances become meaningless when there are many features). Needs scaling, since it’s distance-based.
  • k is the key hyperparameter: small k overfits (noisy), large k oversmooths.

The idea — “similar inputs have similar outputs, measured by distance” — is exactly the intuition behind semantic search over embeddings, which is essentially KNN in a learned vector space.

Naive Bayes

Applies Bayes’ theorem with a “naive” assumption: that all features are independent given the class. That assumption is almost always false, yet the model works surprisingly well anyway — especially for text classification (spam filtering, sentiment):

from sklearn.naive_bayes import MultinomialNB
MultinomialNB().fit(X_train, y_train)   # great for word-count / TF-IDF text features

Extremely fast to train and predict, works with little data, and a strong baseline for text. Its probability estimates are poorly calibrated (the independence assumption), so trust the ranking more than the exact probabilities.

Choosing among them

AlgorithmShines whenAvoid when
SVMsmall/medium data, clear margins, high-dimlarge datasets
KNNsmall data, low dimensions, simple baselinelarge data, many features
Naive Bayestext, high-dim sparse features, tiny datafeatures are strongly dependent and you need calibrated probabilities

All three are worth recognizing, but for most tabular problems gradient boosting will beat them — these earn their place on specific data shapes (SVM on clean small margins, Naive Bayes on text, KNN as a dead-simple baseline).

↑ Back to 03 — Classical ML