[reference] — from the roadmap.sh machine-learning roadmap. The most important family for tabular data — and the one that most often quietly beats deep learning there.

Decision trees

A decision tree learns a sequence of yes/no questions on the features, splitting the data at each node to separate the classes (or reduce variance, for regression):

              age < 30?
             /         \
          yes           no
       income<40k?    owns_home?
        /    \          /    \
     ...     ...      ...    ...

At each node the tree picks the split that best separates the target (by Gini impurity or entropy for classification). The result is highly interpretable — you can read the decision path — and handles non-linear relationships and feature interactions naturally, with no scaling needed.

from sklearn.tree import DecisionTreeClassifier
DecisionTreeClassifier(max_depth=5).fit(X_train, y_train)   # max_depth limits overfitting

The catch: a single unconstrained tree overfits badly — grown deep enough, it memorizes the training data. max_depth and min_samples_leaf constrain it, but the real fix is ensembles.

Random forests — many trees, averaged

Train many decision trees, each on a random subset of the data and a random subset of features, then average their predictions (or majority-vote). This bagging (bootstrap aggregating) dramatically reduces the overfitting of a single tree — individual trees overfit in different ways, and averaging cancels the noise:

from sklearn.ensemble import RandomForestClassifier
RandomForestClassifier(n_estimators=200).fit(X_train, y_train)
model.feature_importances_    # which features mattered — a built-in interpretability win

Robust, strong out-of-the-box with little tuning, resistant to overfitting, and gives feature importances. A superb default for tabular data.

Gradient boosting — the tabular-data champion

Instead of averaging independent trees, boosting builds trees sequentially, each new tree correcting the errors of the ensemble so far. This is usually the most accurate approach on structured/tabular data, full stop:

  • XGBoost, LightGBM, CatBoost — the dominant, highly-optimized implementations. LightGBM is fast on large data; CatBoost handles categorical features gracefully.
from lightgbm import LGBMClassifier
LGBMClassifier(n_estimators=500, learning_rate=0.05).fit(X_train, y_train)

More powerful than random forests but more tuning-sensitive (learning rate, tree depth, number of trees interact — hyperparameter tuning matters here), and it can overfit without care.

Bagging vs boosting — the distinction

Bagging (Random Forest)Boosting (XGBoost/LightGBM)
Trees builtindependently, in parallelsequentially, each fixing the last
Reducesvariance (overfitting)bias (underfitting) and variance
Tuningforgivingsensitive — tune it
Default?great safe defaultbest accuracy, more effort

Why this family matters so much

For structured/tabular data — the most common real-world business case (spreadsheets, transactions, logs) — gradient-boosted trees frequently outperform deep learning while training in a fraction of the time, needing no GPU, no feature scaling, and giving feature importances for free. The reflex to reach for a neural network on tabular data is usually wrong (choosing the approach); start here.

↑ Back to 03 — Classical ML