Description
We introduce basic principles and techniques in the field of machine learning. These are some of the key tools behind the emerging field of data science. These techniques now run behind the scenes to discover patterns and make predictions in various applications in our daily lives. We focus on many of the core machine learning technologies, with motivating applications from a variety of disciplines.
The material for this course—including slides, homework assignments, and the overall website structure—is adapted from Mark Schmidt’s CPSC 340: Machine Learning and Data Mining course at the University of British Columbia, used with his permission. We are grateful for his contribution in providing foundational material that helped shape this course.
Textbooks
There is no required textbook for the class. An introductory book that covers many (but not all) the topics we will discuss is the Artificial Intelligence book of Russell and Norvig (AI:AMA) or the Artificial Intelligence book of Poole and Mackworth. More advanced books include The Elements of Statistical Learning (ESL) by Hastie et al., Murphy’s Machine Learning: A Probabilistic Perspective (ML:APP) which can be accessed through the library, and Bishop’s Pattern Recognition and Machine Learning (PRML). The schedule below also references Introduction to Data Mining (IDM) by Tan, Steinbach, and Kumar.
Grading
- Assignments 30%, Midterm 20%, Final 50%.
- There will be 2% extra credit for submitting teaching evaluation, if more than 80% of students submitted the evaluation.
Schedule
Lecture slides will be posted on the course website before each lecture. The instructor reserves the right to alter this schedule as needed.
| Aug 24, Aug 26 |
Motivation and Syllabus
Exploratory Data Analysis |
What is Machine Learning?
Machine Learning
Rise of the Machines
Talking Machine Episode 1
Mathematics for Machine Learning
Gotta Catch’em all
Why Not to Trust Statistics
Visualization Types
Google Chart Gallery
Other Tools |
Assignment 1 a1.zip
Assignment instructions
Big-O Notes |
| Aug 31, Sep 2 |
Decision Trees
Fundamentals of Learning |
A Visual Introduction to Machine Learning
Decision Trees
Entropy
What is Big O Notation?
AI:AMA 19.2-3, ESL: 9.2, ML:APP 16.2
7 Steps of Machine Learning
IID
Cross-validation
Bias-variance |
Assignment 1 due: Sep 4
Course Notation Guide
Probability Notes
|
| Sep 7, Sep 9 |
Probabilistic Classifiers
Probability Slides |
No Free Lunch
AI:AMA 19.4-5, ESL 7.1-7.4, 7.10, ML:APP 1.4, 6.5
Conditional probability (demo)
Naive Bayes
Probabilities and Battleship
What is the Chance of an Earthquake?
Matters of Degree
AI:AMA 12.6, ESL 4.3, ML:APP 2.2, 3.5, 4.1-4.2 |
Assignment 2 a2.zip
Labor Day - No Class Monday |
| Sep 14, Sep 16 |
Non-Parametric Models
Ensemble Methods
Clustering |
K-nearest neighbours
Decision Theory for Darts
Norms
AI:AMA 19.7, ESL 13.3, ML:APP 1.4
Ensemble Methods
Random Forests
Empirical Study
Kinect
AI:AMA 19.8, ESL: 7.11, 8.2, 15, 16.3, ML:APP 6.2.1, 16.2.5, 16.6
Clustering
K-means clustering (demo)
K-Means++ (demo)
IDM 8.1-8.2, ESL: 14.3 |
Linear Algebra Notes
Linear/Quadratic Gradients |
| Sep 21, Sep 23 |
More Clustering
Outlier Detection
Linear Regression |
DBSCAN (video, demo)
Hierarchical Clustering
Phylogenetic Trees
IDM 8.4
Outlier Detection
Local Outlier Factor
Isolation Forest
Empirical Study
IDM 8.3, ESL 14.3.12, ML:APP 25.5
Linear Regression (demo, 2D data, 2D video)
Least Squares
Essence of Calculus
Partial Derivative
Gradient
ESL 3.1-2, ML:APP 7.1-3, AI:AMA 19.6 |
Assignment 2 due: Sep 25 |
| Sep 28, Sep 30 |
Nonlinear Regression
Gradient Descent |
Why should one learn machine learning from scratch?
Essence of Linear Algebra
Matrix Differentiation
Fluid Simulation (video)
ESL 5.1, 6.3
Gradient Descent
Convex Functions |
Assignment 3 a3.zip |
| Oct 5, Oct 7 |
Robust Regression
Feature Selection
Regularization |
ML:APP 7.4
Genome-Wide Association Studies
AIC
BIC
ESL 3.3, 7.5-7
ESL 3.4., ML:APP 7.5, AI:AMA 19.4 |
Assignment 3 due: Oct 9 |
| Oct 12, Oct 14 |
More Regularization
Linear Classifiers
More Linear Classifiers |
RBF video
RBF and Regularization video
ESL 6.7, ML:APP 13.3-4
Perceptron
ESL 4.5, ML:APP 8.5
Support Vector Machines
ESL 4.4, 12.1-2, ML:APP 8.1-3, 9.5 14.5, AI:AMA 19.6 |
Assignment 4 a4.zip |
| Oct 19, Oct 21 |
Feature Engineering
Kernel Trick |
Gmail Priority Inbox
ESL 12.3, ML:APP 14.1-4 |
Midterm — Oct 21 |
| Oct 26, Oct 28 |
Stochastic Gradient Descent
Boosting
MLE and MAP
Principal Component Analysis |
Stochastic Gradient Descent
Theory and Practice
ML:APP 8.5
AdaBoost (video)
XGBoost (video)
ML:APP 16.4
Maximum Likelihood Estimation
ML:APP 9.3-4
Principal Component Analysis
ESL 14.5, IDM B.1, ML:APP 12.2 |
Assignment 4 due: Oct 30
Max and Argmax Notes |
| Nov 2, Nov 4 |
More PCA
Beyond PCA |
Making Sense of PCA
SVD
Eigenfaces
Non-Negative Matrix Factorization (original - access from UCF)
Recommender Systems
Netflix Prize
ESL 14.6, ML: APP 13.8 |
Assignment 5 a5.zip |
| Nov 9, Nov 11 |
Multi-Dimensional Scaling
Neural Networks
Over-Parameterization |
Nonlinear Dimensionality Reduction
t-SNE demo
UMAP demo
ESL 14.8-9, IDM B.2
Google Video
What is a Neural Network?
Interactive Guide
ML:APP 16.5, ESL 11.1-4, AI:AMA 21.1 |
Assignment 5 due: Nov 13
Veterans Day - No Class Wednesday |
| Nov 16, Nov 18 |
Deep Neural Networks |
Fortune Article
Deep Learning References
Alchemy
ML:APP 28.3, ESL 11.5, AI:AMA 21.2 and 21.4-5 |
Assignment 6 a6.zip |
| Nov 23, Nov 25 |
Convolutions
Convolutional Neural Networks
Autoencoders |
But what is a convolution?
Convolutional Neural Networks
ML:APP 28.4, ESL 11.7, AI:AMA 21.3 |
Assignment 6 due: Dec 4
Thanksgiving - No Class |
| Nov 30, Dec 2 |
Recurrent Neural Networks
LSTMs and Transformers
What do we Learn? |
AI:AMA 21.6-8 |
|
| TBA |
Final Exam |
|
|
