Physical study cards

Question 1

Linear regression

Answer 1

Y = x β + e, where Y are observed values, X are the features, b the regression coefficient and e the vector of errors or residuals,
We can solve this for β: β = (X^T X)^-1X ^TY, where (X^T X) must be invertible.
If it is not invertible we can do ridge regression instead or use the pseudo-inverse.

Question 2

What does backpropagation do?

Answer 2

Backprop computes ∇_θL efficiently by applying the chain rule on a computation graph and reusing intermediate results.
Forward pass: compute activations and cache needed intermediates.
Backward pass: propagate upstream gradients δ_v = ∂L/∂v from output back to parameters.
Key rule: if y = f(x), then δ_x = δ_y · (∂y/∂x)

Question 3

Model training code in PyTorch

Answer 3

num_epochs = 1000
for epoch in range(num_epochs):
       predictions = model(X)
       MSE = loss(predictions,y)
       MSE.backward()
       optimizer.step()
       optimizer.zero_grad()

Question 4

Model evaluation code in PyTorch

Answer 4

model.eval()
with torch.no_grad():
       predictions = model(X_test)
       test_MSE=loss(predictions,y_test)

Question 5

Random forest

Answer 5

• ML model consisting of several decision trees
• Randomness between trees can be introduced by:
  1. Bootstrap Aggregation (Bagging)
• randomly sample trees with replacement
  2. Random Feature
• selection: at each node random subset of features is considered

Question 6

How to calculate Entropy (Information Gain) for example for random forests?

Answer 6

H(x) = - ∑₁^c (p_i log₂(p_i), where c is the number of classes
→ the goal is low entropy in child notes

Question 7

How to calculate Gini Impurity?

Answer 7

Gini(X) = 1 - sum(p_i²)
Gini(0,1)=0
Gini(0.5,0.5)=0.5
Gini(0.3,0.7)=0.48
More unequal distribution = more certainty = lower impurity

Question 8

What are Boosted trees?

Answer 8

An ensemble is built by training each new instance to emphasize previously mismodeled samples e.g. AdaBoost

Question 9

What is CrossEntropy?

Answer 9

L_MLM = - ¹/_|B| ∑_B∑_M log(p(s_i|S_M^c))
where B is the batch and M is the set of masked tokens

Question 10

What is a support vector machine?

Answer 10

• model finds a hyperplane that best separates classes
• objective function minimize ¹/₂ ||w||² subject to the constraint y_i(wx_i+b)>=1, where y_i are the labels and y_i in {-1,1}
• for non-linear cases a kernel trick can be applied

Question 11

What is a singular value decomposition?

Answer 11

• a matrix decomposition used for dimensionality reduction in PCA and for data compression
• A = U∑V^T, where U and V are orthogonal matrices
• orthogonal means QQ^T = Q^TQ = 1, a linear transformation that preserves norms of vector
  It is calculated as follows:
• U has eigenvectors of AA^T as columns
• V has eigenvectors of A^TA as columns
• ∑ has singular values = square roots of eigenvalues as diagonal elements in descending order

Question 12

What is UMAP?

Answer 12

“Uniform Manifold and Approximation”
1. Construct high-dimensional graph: calculate distances between data points using a metric, find k-nearest neighbors and create a graph with probability of being connected e^{d(x,y)/ σ_i}
2. Optimization process: find low-dimensional representation of graph, minimize cross-entropy
3. Attractive and repulsive forces: to make distances match those in high-dimension
4. Stochastic gradient descent (SGD)

Question 13

What is PCA?

Answer 13

“Principal component analysis”
- converts a set of observations into a set of linearly uncorrelated variables
1. Standardization: z_i=x_i-μ_i / σ_i
2. Covariance matrix: C = ¹/_(n-1)Z^TZ
3. Eigenvalue decomposition: solve Cv = λv
4. Sort eigenvectors: λ₁, …, λ_k, where λ_k is the largest eigenvalue, v₁, …, v_k are the corresponding eigenvectors
5. Project data onto principal components: T = ZV_k, where T is the transformed data with the rows containing the observations and the columns the principal components
- the total variance is then given by ∑ λ_i

Question 14

Normal distribution

Answer 14

f(x) = (¹/_{σ √2π}) e ^{-1/₂ ((x-μ)/_σ)2}
where σ is the standard deviation
the variance is σ².
For the standard normal distribution: μ=0, σ=1

Question 15

Recurrent layers

Answer 15

Network architecture designed to process sequential data. Has potential for infinite impulse response.
Hidden state
h_t = f_h(W_hxx_t + W_hhh_t−1 + b_h)
Output
y_t = f_y(W_yhh_t + b_y)

Question 16

P-value

Answer 16

probability of obtaining data as extreme as the observed data assuming the null hypothesis is true

Question 17

Pearson correlation

Answer 17

P_X,Y= Cov(X,Y)/σ_Xσ_Y, where σ is the standard deviation

Question 18

Covariance

Answer 18

con(X,Y) = ¹/_n ∑(x_i-E(X))(y_i-E(Y)), where E(X) is the expected value

Question 19

How to prevent overfitting? List 7 methods

Answer 19

• Cross-validation
• Regularization (L1/L2)
• Pruning (e.g. in decision trees)
• Early stopping
• Dropout
• Simpler model with fewer parameters
• Ensemble methods

Question 20

How can you optimize model architecture and execution?

Answer 20

• Mixed precision training: single-precision (32-bit) for weights, biases, losses and half (16-bit) for activations and gradients. This increases speed and decreases memory requirements.
• Gradient checkpointing: only store activations at checkpoint layers (activations are computed forward and then needed for the backward gradient computation), recalculate other during backward pass
• Model simplification:
  a. pruning: remove unimportant parameters
  b. quantization: reduce precision
  c. knowledge distillation: smaller model trained to mimic larger model

Question 21

Transformer architecture

Answer 21

Input sequence -> Tokenizer -> Input embedding layer + positional encoding
Encoder stack:
- multi-head self-attention
- position-wise fully connected feed forward
Decoder stack:
- self-attention for decoder input: K, V
- self-attention for encoder output: Q from decoder
- feed-forward
Output layer:
- linear transformation
- softmax: to create output probabilities

Question 22

Logit function

Answer 22

logit(p) = σ^-1 = ln(p/(1-p)) for p in (0,1)
- maps (0,1) to (-infinity,infinity)
- inverse of the sigmoid function
- logit(0.5) = 0
“logit = logarithm + unit”

Question 23

Attention layers

Answer 23

Attention(Q,K,V) = softmax(QK^T/sqr(d_k))V

• query Q = W_qX+b_q1^T
• keys K = W_kX+b_k1^T
• values V = W_vX+b_v1^T

W_q,W_k,W_v are the parameter matrices

rows are tokens
columns are embeddings of the tokens

Question 24

PyTorch Model

Answer 24

torch.manual_seed(42)
model=nn.Sequential(
nn.Linear(3,8),
nn.ReLU(),
nn.Linear(8,4),
nn.Sigmoid(),
nn.Linear(4,1))
model(X)

Question 25

Confusion matrix

Answer 25

TP-FP FN-TN Precision = TP/(TP+FP) Recall = TP/(TP+FN) Accuracy = (TP + TN)/(TP+FP+TN+FN)

Question 26

L1-Regularization

Answer 26

"Lasso regularization" - loss = ¹/_N ∑(y_i^p-y_i^a)² + λ ∑(||w_j||) - this leads to sparse models - effectively performs feature selection (do this when some features might be irrelevant)

Question 27

Batch Normalization

Answer 27

y = γ [(x - μ_B) / sqrt( sigma _B²+ε)] + β where μ_B is the mini-batch mean (over x_i) and sigma _B the standard deviation γ, β are learnable parameters, ε is added for numerical stability - this adjusts layer outputs to follow the same distribution regardless of noise in inputs - it addresses the issue of internal covariate shift, where layers constantly have to adjust to changing input distributions

Question 28

Cosine similarity

Answer 28

cosine similarity(u,v) = cos(θ_{u, v}) = uv/(||u|| ||v||) = u₁v₁+u₂v₂+u₂v₃/||u|| ||v||

Question 29

ROC curve

Answer 29

- graph of false positive rate vs. true positive rate - random would be the diagonal - TPR = TP/ (TP+FN) -> probability that true positive tests positive - FPR = FP / (FP+TN) -> probability that a true negative will test positive

Question 30

Tanh / Hyperbolic Tangent

Answer 30

tanh(x) = (e^x - e^-x) / (e^x + e^-x) tanh'(x) = 1 - tanh²(x) - maps real numbers to (-1,1) - zero centered: tanh(0)=0

Question 31

Sigmoid function

Answer 31

σ(x) = 1/(1+e^-x) σ'(x) = σ(x)(1 - σ(x)) - maps the real numbers to (0,1) - s shaped with σ(0)=0.5

Question 32

Absolute positional embedding

Answer 32

- PE(pos, 2i) = sin( pos / (10000^{2i / d_model}) ) - PE(pos, 2i+1) = cos( pos / (10000^{2i / d_model}) ) - d_model = dimensionality of embedding - i adjusts the frequency (higher i = lower frequency)

Question 33

Adam optimizer

Answer 33

- first moment vector: m + moving average of gradients - second moment vector: v = moving average of squared gradients **1. Initialization:** m₀=0, v₀=0, t=0 **2. Gradient:** g_t = ∇ f(θ_t-1), where f is the loss function **3.** m_t=β₁ m_t-1+ (1-β₁)g_t **4.** v_t=β₂v_t-1+(1-β₂) g_t² **5. Correct the bias toward 0 from initialization:** m_t(hat)= m_t/(1-β₁^t), v_t(hat)=v_t/(1-β₂^t) **6. Update parameters:** θ_t+1=θ_t - lr (m_t(hat) / sqrt(v_t(hat))+e)

Question 34

L2-Regularization

Answer 34

- "Ridge regression" - loss = ¹/_N ∑(y_i^p-y_i^a)² + λ ∑(w_j²) -> encourages smaller more evenly distributed weights

Question 35

Logarithm rules

Answer 35

Definition: a^log_a(b)=b Rules: log_b(xy) = log_b(x)+log_b(y) log_b(x/y) = log_b(x)-log_b(y) log_b(xⁿ)=nlog_b(x) log(1) = 0 for b towards 0, log(b) goes towards - infinity

Question 36

Mathew's correlation coefficient, MCC

Answer 36

(TP x TN - FP x FN)/sqrt((TP+FP)(TP+FN)(TN+FP)(TN+FN) - similar to F1-score, but even more robust and incorporates all parts of the confusion matrix

Question 37

F1-score

Answer 37

2 * (precision*recall)/(precision+recall) - if there is an imbalance in the class distribution then the precision and recall are the most important metrics

Question 38

What is the kernel trick for SVM?

Answer 38

A trick to apply an SVM to a non-linear space. The data is mapped to a higher dimensional space. - Linear kernel: K(x_i,x_j)=x_ix_j - Radial Basis Function: K(x_i,x_j)=exp(-γ||x_i-x_j||²)

Question 39

T-test

Answer 39

Assumptions: normality, independence t = (mean(x) - μ₀) / (s √n), where mean(x) is the sample mean, μ₀ is the population mean, s is the standard deviation and n is the sample size To calculate the test statistic t is looked up in a table with n-1 degrees of freedom. If t is smaller than alpha the null hypothesis is rejected.

Question 40

Rotary Positional Embedding (RoPE)

Answer 40

**Rotary positional embedding:** Instead of adding a positional embedding to a token embedding, pairs of embedding dimensions are rotated by an angle that depends on token position 1️⃣ Pair up dimensions Take a query or key vector and group dimensions into pairs: (x1​,x2​), (x3​,x4​), … Each pair will be treated as a 2D vector. 2️⃣ Assign a rotation angle per position: For token position p, each pair gets a rotation angle: θp,i​=p⋅ωi​ = fixed frequency (different per dimension pair) Lower dimensions rotate slowly, higher ones rotate faster. 3️⃣ Rotate the vector Each 2D pair is rotated using a standard rotation matrix: (x2i′​x2i+1′​​)=(cosθp,i​sinθp,i​​−sinθp,i​cosθp,i​​)(x2i​x2i+1​​) This is applied to queries and keys, but not values.

Question 41

GeLU function

Answer 41

GELU(x) = x Φ(x), where Φ(x) is the cumulative distribution function of the standard normal distribution - activation function that is an alternative to ReLU - it gives a smooth transition instead of a strong break at 0

Question 42

What’s the backprop rule at a node with multiple children?

Answer 42

If v influences multiple downstream nodes u₁, …, u_k, then gradients add: ∂L/∂v = Σ_i (∂L/∂u_i) (∂u_i/∂v) Interpretation: sum contributions from every outgoing path.

Question 43

What is the formula for binary cross entropy?

Answer 43

BCE=−[y logq + (1−y) log(1−q)], With target y∈{0,1} and predicted q=p^(y=1)