What is the Lasso optimisation problem?
Can drop the constant term to make the algebra easier
What is the prediction error of the Lasso solution?
What is Theorem 2: the Slow Rate of the prediction error of the Lasso solution?
What is the proof of the Theorem 2?
CHECK: Why does the proof he uses not contain the Y-bar part of the proof?
What can we note about the slow-rate of convergence of Lasso?
What is some useful notation we will use for column and component extraction operations?
How do we denote β* as a sparse vector, though using signal coordinates ?
What can we do once we have set up these sets in an artificial sort of situation?
How does this relate to Lasso, and what do we need to do this?
What is Definition 2: The compatibility condition?
As the infimum of all δ in RP such that δS is not equal to to the zero vector and the L1 norm of the noise coodinates is not too big compared to the L1 of the signal coordinates
numerator:
* Xδ –> average magnitude over all the N coordinates
denominator:
* δ2 –> looking at the signal part of delta
* as |S| is the size of S, the denominator is roughly the average of the S coordinates
- So roughly speaking if your vector is not that concentrated on the noise coordinate, it has at least some significant component in the support set (signal coordinate)
- Then, once you apply the design matrix transformation, the average magnitude will not shrink compared to the average magnitude of the signal set
IF XTX/n has a minimum eigenvalue cmin 0, what do we know about the compatibility condition? How do we verify this?
GO THROUGH THIS
around eigenvector and c_min being the answer
12 Feb 30mins
How does this change in high dimensions?
Theorem 3 (fast rate) of convergence of Lasso?
- The first equation is the prediction error of Lasso
- The second equation is the estimation error of Lasso
What is the proof of Theorem 3 (fast rate) of convergence of Lasso?
How do we know that the compatibility condition is a property of the design matrix?
Why does Lasso not have a closed-form solution, unlike ridge regression?
What is the definition of a subgradient of a convex function?
What do we call the set of subgradients of f at x?
What is Proposition 3 regarding the value of ∂f(x) given f is convex and differentiable at x ∈ int(C)
What is the subgradient of:
f: x I–> |x|
Proposition 4: Let f1, f2: Rd –> R be convex
a) for any α > 0, what is ∂(αf1)(x)
b) ∂(f1 + f2))(x)
c) h(x) = f1(Ax +b) what is ∂h(x)
Proposition 5: What is the (simplified) KKT condition?
- allows us to quantify the maximiser of an objective function, which is convex but not necessarily differentiable
- You can find the minimiser of a differential function such that its gradient is 0
What is the proof of Proposition 5: What is the (simplified) KKT condition?
What is Proposition 6 relating to the subdifferential of the l1-norm ||.||?
What is the proof of Proposition 6
12(2) Feb 20 mins–> go through the proof
What is the theorem that fully characterises Lasso solutions?
What is the proof of Theorem 4, the fully characterised Lasso solution?
EXPAND AND FILL IN THIS PROOF
-
ST420 - Basic Notation and Concepts18
-
Chapter 1.1: Intro to Statistical Learning - Buzz words and basic concepts11
-
Chapter 1.2: Intro to Statistical Learning - First taste of Statistical Machine Learning15
-
Chapter 1.3: Intro to Statistical Learning - Statistical learning theory - Loss, Risk and ERM25
-
Chapter 1.4: Intro to Statistical Learning - Statistical learning theory - Excess Risk and the Bias-Variance Trade-off24
-
Chapter 2.1: Classification (Part I) - Classification concepts, LDA and QDA14
-
Chapter 2.2: Classification (Part I) - Logistic Regression13
-
Chapter 2.3: Classification (Part I) - Neural Networks20
-
Chapter 3.1: Big Data - Issues arising from Big Data15
-
Chapter 3.2: Big Data - Multiple testing9
-
Chapter 4.1: Regularisation - An overview of regularisation and ridge regression20
-
Chapter 4.2: Regularisation - Lasso and constrained forms14
-
Chapter 4.3: Regularisation - Concentration inequalities17
-
Chapter 4.4: Regularisation- Advanced analysis of Lasso28
-
Chapter 4.5: Regularisation- Extension and related other problems8
-
Chapter 5.1 - Classification (Part II) - Maximal margin classifier and support vector classifier20
-
Chapter 5.2 - Classification (Part II) - Kernel Methods24
-
Chapter 6.1 - Cross validation, bootstrap and tree-based methods - Cross validation and bootstrap18
-
Chapter 6.2 - Cross validation, bootstrap and tree-based methods - Tree-based methods16
-
Chapter 7.1 - Mathematics of statistical machine learning - PAC Bounds11
-
Chapter 7.2 - Mathematics of statistical machine learning - Concentration inequalities8
-
Chapter 7.3 - Mathematics of statistical machine learning - Finite hypotheses classes6
-
Chapter 7.4 - Mathematics of statistical machine learning - From shattering to VC dimension for infinite classes20
-
Chapter 7.4 - Mathematics of statistical machine learning - Bounding Estimation error for infinite classes6
-
Chapter 7.5 - [NON-EXAMINABLE] Mathematics of statistical machine learning - Rademacher complexity0
-
Chapter 7.6 - Mathematics of statistical machine learning - Structural risk minimisation3
-
Chapter 6.3: Cross-validation, boosttrap and tree-based methods - Boosting12
-
Chapter 8.1 - Topics in big data - Bag of little bootstraps7
-
Chapter 8.2 - Topics in big data - Implicit bias of neural networks6
