What should be the order of things we address in model improvements?
- Under fitting
- Over fitting
- Distribution shift
- Re-balance data set
Addressing under fitting should be in the fellowing order:
- Make your model bigger
- Reduce regularization
- Error analysis
- Choose a different model architecture (closer to SOTA)
- Tune hyper-parameters
- Add features
Addressing overfitting should be in this order:
- Add more training data
- Add normalization (batch norm, layer norm)
- Add data augmentation
- Increase regularization (drop out, l2, weight decay)
- Error analysis
- Choose a different model
- Tune hyper-parameters
- Early stopping
- Remove features
- Reduce model size
What can you do for Addressing distribution shift? (Validation and test scores not close)
- Analyze test-Val set error (actually look at what is being labeled wrong) and:
- 1 collect more training data to compensate
- 2 synthesize more training data to compensate
- Apply domain adaptation techniques to training and test distributions
What is error analysis
Looking at where the train-Val, and the Val-test are off, look at the specific cases. Split them into different groups and prioritize based on what is easier to deal with
What is domain adaptation?
Techniques to train on “source” distribution and generalize to another “target” using only unlabeled data or limited labeled data.
Should consider using when access to labeled data from test is limited and the access to relatively similar data is plentiful
2 types of domain adaptations:
- Supervised, use when there is limited data from target domain, examples: fine tune pre trained model, add target data to train set
- Unsupervised, you have lots of unlabeled data from target domain, examples: correlation alignment, domain confusion, cycleGAN
