Andrew Ng 斯坦福机器学习公开课听课笔记（11）
Key ideas:
1. Diagnostics for debugging learning algorithms
2. Error analyses and ablative analysis
3 . How to get started on a machine learning problem
- premature (statistical) optimization. 过早统计优化
Debugging Learning Algorithms
Motivating example:
. Anti-spam You carefully choose a small set of 100 words to use as features.
. Bayesian logistic regression, implemented with gradient descent, gets 20% test error, which is unacceptably high.
. What to do next?
. Bayesian logistic regression:
. Common approach: Try improving the algorithm in different ways.
- Try getting more training examples
- Try a smaller set of features
- Try a larger set of features
- Try changing the features: Email header vs. email body features
- Run gradient descent for more iterations
- Try Newton’s method
- Use a different value for lambda
- Try using an SVM
Diagnostic for bias vs. variance
Better approach:
- Run diagnostics to figure out what the problem is.
- Fix whatever the problem is
Suppose you suspect the problem is either”
- Overfiting (high bias, 高偏差-过拟合)
- Too few features to classify spam(high variance，高方差-欠拟合).
Diagnostic:
- Variance: Training error will the much lower than test error.
- Bias: Training error will also be high.
方法：检查训练误差与测试误差的差异，判断是高偏差还是高方差。（图形或数值）
Diagnostics tell you what to try next
Bayesian logistic regression, implemented with gradient descent
Fixes to try:
- Try getting more training examples.
Fixes high variance.
- Try a smaller set of features
Fixes high variance
- Try a larger set of features
Fixes high bias
- Try changing the features: Try Email header vs. email body features
Fixes high bias
- Run gradient descent for more iterations
Fixes optimization algorithm
- Try Newton’s method
Fixes optimization algorithm
- Use a different value for lambda
Fixes optimization objective
- Try using an SVM
Fixes optimization objective
Optimization algorithm diagnostics
. Bias vs. variance is one common diagnostic
. For other problems. it’s usually up to your own ingenuity to construct your own diagnostics to figure out what’s wrong.
. Another example:
- Bayesian logistic regression gets 2% error on spam, and 2% spam error on non-spam. (Unacceptably high error on non-spam.)
- SVM using a linear kernel gets 10% error on spam, and 0.01% error on non-spam. (Acceptable performance)
- But you want to use logistic regression because of computation efficiency. etc
. What to do next?
More diagnostics
. Other common questions:
- Is the algorithm(gradient descent for logistic regression) converging?
- Are you optimizing the right function?
- I.e. what you care about (weights w higher or non-spam than for spam, 准确率的加权值)
- Bayesian logistic regression? Correct value for lambda?
- SVM? Correct value for C?