MAP
- Mean Average Precision
- Useful for evaluating ranking algoirthms
ROC-AUC
- Area under the ROC curve
- summarizes a classification model’s ability to distinguish between classes
- an AUC of 0.5 indicates a random guess
- a perfect model would have a AUC of 1.0
Correlation matrix
Shows the correlation coefficient between variables
Residual plots
- A residual is the difference between the target and the predicted value
- A residual plot is a a histogram of residuals
Positive residul
The model is over estimating the target
Negative residual
The model is under estimating the target
What does a residual plot in a bell shape centered on zero indicate?
- The model makes mistakes in a random manner and does not systematically over-predict or under-predict
RMSE
- Used to measure accuracy of regression problems (not classification)
Can a confusion matrix be used to evaluate multi-class classification model?
- The metric is calculated for each class by treating it as a binary classification problem after grouping all the other classes as belonging to the second class.
- Then the binary metric is averaged over all the classes to get either a macro average (treat each class equally) or weighted average (weighted by class frequency) metric.
- In Amazon ML, the macro average F1-measure is used to evaluate the predictive success of a multiclass classifier.
Box plot
- visualize how data is distributed across an axis
- min, max, median, first quartile, third quartile
Box plot positive skew
- median lies closer to the first quartile
- whisker at the upper end is longer
- has positive (right) skew
Box plot normal distribution
- median is in the center
- whiskers are the same length
Box plot negative skew
- median lies closer to the third quartile.
- if the whisker at the lower end is longer, then it has negative (left) skew
ROC
- Receiver Operating Characteristic (ROC) curve
- graph of TP rate against FP rate at various thresholds
- shows the diagnostic ability of a binary classifier system as its discrimination threshold is varied
MAPE
- Mean Absolute Percentage Error
- used to evaluate regression models
What do the whiskers of a box plot mean?
They indicate the minimum and maximum values excluding outliers
k-fold Cross-validation
is a technique for evaluating ML models by training several ML models on subsets of the available input data and evaluating them on the complementary subset of the data. Use cross-validation to detect overfitting, i.e. failing to generalize a pattern.
Transfer learning
- Initialize the network with pre-trained weights in all layers except for the output layer. Initialize the output layer with random weights.
- Then, the whole network is fine-tuned with new data. In this mode, training can be achieved even with a smaller dataset. This is because the network is already trained and therefore can be used in cases without sufficient training data.
What transformation should you use to address positive skew?
Logarithmic transformation
What transformation should you use to address negative skew?
Third-order polynomial transformation
How do you select the optional k paramter?
Choose the elbow in the distortion curve
MICE
- Multiple Imputations by Chained Equations
- advanced statistical method often used in machine learning to handle missing data
- involves creating multiple imputed datasets by using chained equations, where each feature with missing data is modeled based on the other features in the dataset
When is recall most relevant?
when trying to minimize false negatives
When is AUC most relevant
when trying to visualize the tradeoff between false positives and false negatives
