ML: Dimensionality Reduction

Question 1

What is the goal of dimensionality reduction?

Answer 1

To reduce the number of features while retaining as much information (variance, structure, or signal) as possible. Benefits include:

Visualization

Noise reduction

Faster computation

Avoiding curse of dimensionality

Question 2

Difference between feature selection and feature extraction?

Answer 2

Selection: choose a subset of original features

Extraction: create new features (linear/nonlinear combinations) from original features

Question 3

What is the “curse of dimensionality”?

Answer 3

As dimensions increase, data becomes sparse, distance metrics lose meaning, and models overfit → DR mitigates this.

Question 4

Why is dimensionality reduction important for LLM embeddings?

Answer 4

Reduce 1536/4096-dim embeddings → faster retrieval and similarity search

Noise reduction → improves clustering & vector search ranking

Visualizing embedding space to debug retrieval or tool routing

Question 5

What is PCA?

Answer 5

PCA is a linear DR method that projects data onto orthogonal directions (principal components) that maximize variance.

Question 6

How is PCA computed?

Answer 6

Center data

Compute covariance matrix

Eigen decomposition of covariance matrix

Select top-k eigenvectors → project data

OR via SVD:

𝑋=𝑈Σ𝑉^𝑇

Question 7

What do eigenvalues in PCA represent?

Answer 7

Variance explained by each principal component. Larger eigenvalue → more variance captured.

Question 8

How do you choose the number of components in PCA?

Answer 8

Cumulative explained variance (e.g., 90–95%)

Scree plot (elbow method)

Downstream task performance

Question 9

What preprocessing is required before PCA?

Answer 9

Centering (subtract mean)

Standardization (divide by std) if features have different scales

Optional whitening for decorrelated components

Question 10

What are limitations of PCA?

Answer 10

Linear assumption → cannot capture nonlinear manifolds

Sensitive to outliers

Components may be hard to interpret

Variance ≠ predictive power

Question 11

When would you use Kernel PCA?

Answer 11

Data lies on a nonlinear manifold

Classical PCA fails to capture structure

Kernel trick maps to higher-dim space before PCA

Question 12

What are t-SNE and UMAP?

Answer 12

t-SNE: nonlinear DR for visualization, preserves local neighborhood structure

UMAP: faster nonlinear DR, preserves local + some global structure, scalable

Question 13

What is a risk of using t-SNE for downstream ML tasks?

Answer 13

t-SNE is stochastic → inconsistent embeddings

Distances are not globally meaningful

Mainly for visualization, not feature extraction for classifiers

Question 14

How are autoencoders used for DR?

Answer 14

Neural networks learn bottleneck representation (compressed latent space)

Nonlinear DR → capture complex manifolds

Can reconstruct original features → minimize information loss

Question 15

What is whitening?

Answer 15

Scaling PCA components so they have unit variance → decorrelated features, used in some preprocessing pipelines.

Question 15

Why is SVD useful for DR?

Answer 15

Decomposes data into orthogonal modes → captures variance

Can compute PCA efficiently via SVD

Handles rectangular matrices (more samples than features or vice versa)

Question 16

When prefer feature selection over extraction?

Answer 16

When interpretability is important

When computation is cheap and features are meaningful individually

Question 17

How do you evaluate PCA or DR performance?

Answer 17

Explained variance ratio

Downstream task accuracy

Reconstruction error (for autoencoders)

Visual inspection for clustering or separation

Question 18

When prefer feature extraction?

Answer 18

High-dimensional data (text, images, embeddings)

Reduce noise, redundancy

Downstream ML benefits from compressed representation

Question 19

How do you decide between linear PCA and nonlinear methods?

Answer 19

Start with PCA for interpretability and speed

Use t-SNE, UMAP, or autoencoders if linear PCA fails to capture important structure

Question 20

How is DR evaluated in retrieval/RAG pipelines?

Answer 20

Embedding compression → check retrieval recall@k

Clustering quality in latent space → silhouette score or kNN accuracy

Speed vs quality trade-off in vector search

Question 21

Your linear regression on PCA components fails. Why?

Answer 21

PCA maximizes variance, not predictive power

Some important low-variance features may be lost

Consider supervised DR (PLS, LDA, or autoencoder with supervised loss)

Question 21

You reduce embeddings from 4096 → 128 dims via PCA. Recall@10 in retrieval drops slightly. What do you do?

Answer 21

Check explained variance → increase components

Try whitening/scaling

Consider nonlinear DR (autoencoder, UMAP)

Evaluate reconstruction or downstream metrics

Question 22

t-SNE shows clusters but nearest neighbors in original space don’t match. Is this a problem?

Answer 22

No, t-SNE preserves local structure and is stochastic → not suitable for quantitative neighbor tasks.

Question 23

You’re building an LLM retrieval pipeline and embeddings are too high-dimensional for FAISS.

Answer 23

Apply PCA/Truncated SVD for linear compression Possibly use product quantization Verify retrieval metrics (recall@k, MRR) after compression

Question 24

Autoencoder latent space produces poor downstream classification accuracy.

Answer 24

Latent dimension too small → information loss Nonlinear features not learned → adjust network depth or activation Consider hybrid supervised + reconstruction loss