What is the excess risk and how can we decompose it into the sum of two different errors?
What are the three terms that estimation error decomposes into?
What can we decompose the 3rd term into?
(1) - Risk of ERM - Empirical risk of the ERM
(2) - Empirical Risk of the ERM - Empirical Risk of the Best in class function
(3) - Empirical Risk of the best in class function - true risk of the best in class function
Second set of (3) is writing it using the loss function –> empirical average of Z - the expectation of Z
What is Markov’s inequality?
What doe we need to consider in the case of empirical risk of f*F-curly?
What is Chevyshev’s inequality and its proof?
What does the inequality show roughly?
What can we use it to prove?
How lose is the bound? what theory do we use to show this holds?
- Chevyshev is quite loose and covergences slower –> by using the intuition from CLT we get covergence at a rate of ε
-n for large n - Using covergence in distribution, so inquality holds asmptotically rather than strictly
What is Chernoff’s bound? What does it mean?
What is the definition of a sub-Gaussian?
In the expectation is the moment generating function of a centered Gaussian
Given Z is sub-Gaussian, what other random variables are sub-Gaussian?
When you are doing a two sided test, and dealing with Z-µ and µ-Z –> you can simplify this by just doing 2 times the bound due to symmetry
What is Hoeffding’s Lemma?
What is the proof of Hoeffding’s Lemma?
ADD ONCE WE’VE DONE THE PROOF
What is the proposition regarding the distribution of the sum of sub-Gaussian random variables?
What is the proof of this proposition?
- mean isnt important as in the definition of Sub-Gaussian we always center the distribution
What is the Corollary regarding the bounding of a sum of independent sub-Gaussian Random variables?
What Is Hoeffding’s inequality?
What is the one-sided/two-sided bounding of the 0-1 loss classification of the empirical risk?
What about the estimation error?
ADD TO THIS FROM WRITTEN NOTES
What is a Lemma for the probability of the error bounds?
- columns have mean/sum 0? and l2 norm = sqrt(n)
What is the proof of this lemma?
ADD FROM NOTES
-
ST420 - Basic Notation and Concepts18
-
Chapter 1.1: Intro to Statistical Learning - Buzz words and basic concepts11
-
Chapter 1.2: Intro to Statistical Learning - First taste of Statistical Machine Learning15
-
Chapter 1.3: Intro to Statistical Learning - Statistical learning theory - Loss, Risk and ERM25
-
Chapter 1.4: Intro to Statistical Learning - Statistical learning theory - Excess Risk and the Bias-Variance Trade-off24
-
Chapter 2.1: Classification (Part I) - Classification concepts, LDA and QDA14
-
Chapter 2.2: Classification (Part I) - Logistic Regression13
-
Chapter 2.3: Classification (Part I) - Neural Networks20
-
Chapter 3.1: Big Data - Issues arising from Big Data15
-
Chapter 3.2: Big Data - Multiple testing9
-
Chapter 4.1: Regularisation - An overview of regularisation and ridge regression20
-
Chapter 4.2: Regularisation - Lasso and constrained forms14
-
Chapter 4.3: Regularisation - Concentration inequalities17
-
Chapter 4.4: Regularisation- Advanced analysis of Lasso28
-
Chapter 4.5: Regularisation- Extension and related other problems8
-
Chapter 5.1 - Classification (Part II) - Maximal margin classifier and support vector classifier20
-
Chapter 5.2 - Classification (Part II) - Kernel Methods24
-
Chapter 6.1 - Cross validation, bootstrap and tree-based methods - Cross validation and bootstrap18
-
Chapter 6.2 - Cross validation, bootstrap and tree-based methods - Tree-based methods16
-
Chapter 7.1 - Mathematics of statistical machine learning - PAC Bounds11
-
Chapter 7.2 - Mathematics of statistical machine learning - Concentration inequalities8
-
Chapter 7.3 - Mathematics of statistical machine learning - Finite hypotheses classes6
-
Chapter 7.4 - Mathematics of statistical machine learning - From shattering to VC dimension for infinite classes20
-
Chapter 7.4 - Mathematics of statistical machine learning - Bounding Estimation error for infinite classes6
-
Chapter 7.5 - [NON-EXAMINABLE] Mathematics of statistical machine learning - Rademacher complexity0
-
Chapter 7.6 - Mathematics of statistical machine learning - Structural risk minimisation3
-
Chapter 6.3: Cross-validation, boosttrap and tree-based methods - Boosting12
-
Chapter 8.1 - Topics in big data - Bag of little bootstraps7
-
Chapter 8.2 - Topics in big data - Implicit bias of neural networks6
