ML: Classification

Question 1

What is the confusion matrix and why is it important?

Answer 1

A table summarizing TP, TN, FP, FN. It underpins all classification metrics (precision, recall, F1, specificity) and reveals model behavior beyond accuracy.

Question 2

When is accuracy a misleading metric?

Answer 2

In imbalanced datasets (e.g., fraud, disease detection) where accuracy hides minority class failures.

Question 3

What is the difference between precision and recall?

Answer 3

Precision: Of predicted positives, how many are correct? TP/(TP + FP). High precision = model’s pos predictions are highly reliable. Spam filter. Helps avoid false alarms.

Recall: Of actual positives, how many did we catch? TP/(TP + FN). High recall = model catches most positive instances. Cancer/Disease prediction. Helps avoid missed detections.

Question 4

How do you choose between precision and recall?

Answer 4

Depends on business risk:

Precision priority → expensive FP (e.g., sending human reviewers)

Recall priority → expensive FN (e.g., medical diagnosis)

Question 5

Why is F1-score the geometric mean of precision and recall?

Answer 5

It penalizes extreme imbalance by requiring both precision and recall to be high.

Question 6

What does the ROC curve represent?

Answer 6

True Positive Rate vs. False Positive Rate trade-off at various decision thresholds.

Question 7

What does AUC measure?

Answer 7

The probability that the classifier ranks a random positive higher than a random negative. Equivalent to ranking quality.

Question 8

When is PR curve preferred over ROC?

Answer 8

Highly imbalanced datasets → PR curve is more sensitive to performance on the positive class.

Question 9

How are ROC/AUC used in LLM agent evaluation?

Answer 9

Tool selection classification

Retrieval relevance classifier

Routing tasks (which LLM or tool to pick)

Safety classifiers (detect harmful intent)

AUC helps evaluate ranking quality of these decision layers.

Question 10

Why can’t logistic regression be solved with a closed-form solution?

Answer 10

The log-likelihood is concave but not quadratic → derivative doesn’t yield an algebraic solution → requires iterative optimization.

Question 11

How do regularization terms affect logistic regression?

Answer 11

L1 → sparse features

L2 → stable coefficients, mitigates multicollinearity
Regularization improves generalization.

Question 12

How to interpret logistic regression coefficients?

Answer 12

Exponentiating coefficients yields odds ratios; positive weights → increase log-odds of being class 1.

Question 13

Why is logistic regression still important in LLM pipelines?

Answer 13

Lightweight safety and routing classifiers

Calibration layers for confidence

Reward model pre-steps

Linear probing on embeddings

Its interpretability makes it especially valuable in AI agent decision layers.

Question 14

Why is Naive Bayes “naive”?

Answer 14

It assumes conditional independence between features given the class.

Question 15

Why does Naive Bayes often perform surprisingly well?

Answer 15

Even with violated independence assumptions, the ranking of class probabilities often remains correct.

Question 16

When is Naive Bayes especially effective?

Answer 16

Text classification

High-dimensional sparse features

Real-time or low-latency applications

Question 17

What is Laplace smoothing and why is it used?

Answer 17

Adds pseudo-counts to avoid zero probabilities for unseen words/features.

Question 18

What is the intuition behind the SVM margin?

Answer 18

SVM maximizes the smallest distance between the decision boundary and any training point → robustness to noise.

Question 19

Why are kernel methods powerful?

Answer 19

They implicitly map data to high-dimensional spaces without explicitly computing those features (via kernel trick).

Question 20

When are SVMs not a good choice?

Answer 20

Very large datasets → slow training

Very large feature spaces without kernel approximation

When probabilistic outputs are needed (unless Platt scaling)

Question 21

What is the role of C in SVM?

Answer 21

Controls trade-off between maximizing margin and minimizing misclassification.

Large C → focus on correctness

Small C → focus on larger margin

Question 22

What is entropy in a classification tree?

Answer 22

A measure of uncertainty; lower entropy → purer nodes.

H(p)=−∑pi​ log pi​

Question 23

What is Gini impurity?

Answer 23

Equivalent impurity measure, computationally faster.

G=∑pi​(1−pi​)

Question 24

Why do decision trees overfit easily?

Answer 24

They split until leaves become pure, capturing noise. Pruning or limiting depth is needed.

Question 25

How do you prevent overfitting in trees?

Answer 25

Limit depth Min samples per split/leaf Pruning Using ensembles (RF, boosting)

Question 26

Why does randomization help random forests?

Answer 26

Two components reduce variance: Bootstrap sampling Random feature selection per split

Question 27

Why do random forests have low bias and low variance?

Answer 27

Each tree is high variance, low bias → averaging reduces variance, retaining low bias.

Question 28

What hyperparameters matter most?

Answer 28

max_depth n_estimators max_features min_samples_split/leaf

Question 29

How do RFs handle imbalanced data?

Answer 29

Class weights Balanced bootstrapping Threshold moving SMOTE + RF

Question 30

How does boosting differ from bagging?

Answer 30

Bagging: trains independent models in parallel Boosting: trains models sequentially, each correcting predecessor’s errors

Question 31

Why can boosting overfit less than expected?

Answer 31

Boosting focuses on fitting hard examples, but uses shrinkage, depth limits, and regularization, preventing runaway fitting.

Question 32

Why is XGBoost so effective?

Answer 32

Second-order optimization Regularization Built-in handling of missing values Histogram-based splits Parallelization Tree pruning with “loss-guided growth”

Question 33

How do you fix overfitting in boosting?

Answer 33

Increase regularization (λ, α) Reduce tree depth Reduce learning rate Increase min child weight Decrease number of estimators

Question 34

Why do even sophisticated LLM agents rely heavily on classical classifiers?

Answer 34

Agents need deterministic, debuggable, low-latency decision nodes: Safety classifiers Intent detection Routing (choose a model/tool) Detect hallucinations Flagging harmful content Traditional ML models excel here.

Question 35

How is boosting used in LLM / agent systems?

Answer 35

Ranking models (LightGBM/XGBoost) for retrieval re-ranking Safety classification Tool routing based on embedding features Reward modeling features Boosting is widely used because it's interpretable, fast, and structured-data-friendly.

Question 36

How is classification used in RAG systems?

Answer 36

Relevance classification Chunk ranking with boosting models Hallucination detection Query type classification Document quality scoring

Question 37

Why is AUC valuable in retrieval scoring?

Answer 37

Because ranking matters more than raw classification—AUC reflects the correctness of ordering relevant vs irrelevant chunks.

Question 38

How are SVMs/logistic regression used in embedding space?

Answer 38

Linear classifiers on embeddings for: Topic classification Content moderation Multi-label tags Safety filters Weak supervision of agent behavior

Question 39

The agent frequently chooses the wrong tool. What ML approach helps?

Answer 39

Use a structured classifier (logistic regression or boosted trees) on features like: last agent step intent class embedding of user query historical success rate This improves deterministic routing.