[reference] — from the roadmap.sh machine-learning roadmap. Making a model’s decisions understandable — increasingly a requirement, not a nicety.
Why interpretability matters
A model can be accurate and still unusable if no one can tell why it decided something. Explainability matters for:
- Trust — stakeholders won’t deploy a black box making consequential calls (loans, medical, hiring).
- Debugging — a model can be right for the wrong reason (the famous case: a classifier that detected “husky vs wolf” by looking at snow in the background). Explanations expose that.
- Fairness & bias — checking a model isn’t keying off a protected attribute (bias & fairness).
- Regulation — laws (GDPR’s “right to explanation,” financial/medical rules) increasingly require explanations for automated decisions.
The interpretability spectrum
Models trade accuracy against transparency:
- Intrinsically interpretable — linear/logistic regression (read the weights), shallow decision trees (read the path). You get explanation for free.
- Black boxes — deep networks, large ensembles. Accurate but opaque; they need post-hoc explanation tools.
A recurring engineering decision: a slightly less accurate but interpretable model is often the better product choice when explanation is required — accuracy isn’t the only axis (model selection).
Global vs local explanations
- Global — what drives the model overall (which features matter most across all predictions). Tree feature importances are a simple global view.
- Local — why this specific prediction came out as it did. Usually what a user or auditor actually asks.
SHAP and LIME — the two standard tools
Both explain any black-box model after training, model-agnostically:
- LIME (Local Interpretable Model-agnostic Explanations) — explains one prediction by fitting a simple interpretable model (a linear model) to the black box’s behavior in the local neighborhood of that input. Fast, intuitive, but the explanation can be unstable (depends on how the neighborhood is sampled).
- SHAP (SHapley Additive exPlanations) — grounded in cooperative game theory (Shapley values): it fairly attributes the prediction among the input features, computing how much each feature pushed the output up or down from a baseline. More theoretically sound and consistent than LIME, does both local and global views, and has fast implementations for trees. The de-facto standard, especially for gradient-boosted models.
import shap
explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X) # per-feature contribution to each prediction
shap.summary_plot(shap_values, X) # global view; force_plot for a single predictionThe honest caveats
- An explanation is an approximation, not the model’s actual reasoning — especially for LIME. Treat explanations as evidence, not ground truth.
- Explaining LLMs is much harder — SHAP/LIME target tabular/feature-based models. For LLMs, “interpretability” means different, less mature techniques (attention analysis, mechanistic interpretability), and in practice grounding/citation (RAG) and evals do more for trust than feature attribution.
Related
- Trees & Ensembles — feature importances, and SHAP’s home turf
- Validation & Tuning — interpretability as a model-selection axis
- Safety & Production — bias/fairness, where explanations are checked
Jump to a subject
Software engineering
Infrastructure & security
Computer science
Hardware & engineering