EE 514 (CS 535) Machine Learning

Spring 2023

Syed Babar Ali School of Science and Engineering
Lahore University of Management Sciences



Course Overview

Machine learning (ML) studies the design and development of algorithms that learn from the data and improve their performance through experience. ML refers to a set of methods and that help computers to learn, optimize and adapt on their own. ML has been employed to devise algorithms for diverse applications including object detection or identification in computer vision, sentiment analysis of speaker or writer, detection of disease and planning of therapy in healthcare, product recommendation in e-commerce, learning strategies for playing games, recommending movies to customers, speech recognition systems, fraudulent transaction detection or loan application approval in banking sector, to name a few.

This course provides a thorough introduction to the theoretical foundations and practical applications of ML. We will learn fundamental algorithms in supervised learning and unsupervised learning. We will not only learn how to use ML methods and algorithms but will also try to explain the underlying theory building on mathematical foundations. While reviewing the several problems and algorithms to carry out classification, regression, clustering, dimensionality reduction, we will focus on the core fundamentals which unify all the algorithms. The theory discussed in class will be tested in assignments, quizzes and exams.

Announcements

  • (Jan 17) Welcome to EE 514 (CS 535). Course outline has been posted.

Administrative Details

  • Course Outline (Click to download)

  • Suggested Books:

    • (CB) Pattern Recognition and Machine Learning, Christopher M. Bishop

    • (KM) Machine Learning: a Probabilistic Perspective, Kevin Murphy

    • (TM) Machine Learning, Tom Mitchell

    • (HTF) The Elements of Statistical Learning: Data mining, Inference, and Prediction, by Hastie, Tibshirani, Friedman

    • (DM) Information Theory, Inference, and Learning Algorithms, David Mackay

  • Office Hours and Contact Information

    • Instructor: Zubair Khalid (zubair.khalid@lums.edu.pk), Office hours: Tuesday, Thursday 11 am-12 pm

    • Teaching Assistant: Fatima Naweed (23100042@lums.edu.pk), Office hours: Tuesday, Thursday 2:45 to 3:45 pm

    • Teaching Assistant: Omer Abdul Jalil (23100050@lums.edu.pk), Office hours: Tuesday, Thursday 11 am-12 pm

    • Teaching Assistant: Sharjeel Ahmed Shaikh (23100214@lums.edu.pk), Office hours: Friday, 4pm to 5 pm

    • Teaching Assistant: Abdul Muizz Khan (23100153@lums.edu.pk), Office hours: Monday and Wednesday, 10 am to 11 am

    • Teaching Assistant: Ali Shahryar Khokhar (24100266@lums.edu.pk), Office hours: Monday and Wednesday, 3 pm to 4 pm

Grading Distribution

  • Programming Assignments and Homeworks, 35 %

  • Project, 20 %

  • Quizzes (1 per week), 15 %

  • Final Exam, 30 %

Project Description

Programming Assignments

Homeworks

Homework Solutions
Homework 01 Solutions
Homework 02 Solutions
Homework 03 Solutions

Class Quizzes - Solutions

Schedule

  • Week 01 (Notes 01)

    • Course Introduction

    • Machine Learning Overview

    • Supervised Learning: Formulation, Setup, Train-test split, Generalization

  • Week 02 (Notes 02)

    • k-Nearest Neighbor (kNN) Algorithm, Algorithm Formulation

    • Distance Metrics, Choice of k, Algorithm Convergence, Storage, Time Complexity Analysis, Fast kNN

  • Week 03 (Notes 03)

    • The Curse of Dimensionality and Connection with kNN

    • Dimensionality Reduction: Feature Selection and Extraction, Principal Component Analysis

    • Classifer Performance Evaluation: Confusion Matrix Sensitivity, Specificity, Precision Trade-offs, ROC, AUC, F1-Score and Matthew’s Correlation Coefficient

  • Week 04 (Notes 04)

    • Multi-class Classification, Evaluation, Micro, Macro Averaging

    • Regression: Linear Regression, Polynomial Regression, Overfitting

  • Week 05

    • Regularization (see Week 04 Notes)

  • Week 06

    • Gradient Descent Algorithm (see Week 04 Notes)

  • Week 07 (Notes 05)

    • Bayesian Learning Framework, MAP and ML Hypothesis

    • Logistic Regression: Mathematical Model, Decision Boundaries, Loss/Cost Function, Gradient Descent

    • Multi-class Logistic Regression

    • Linear Regression as ML Estimation

  • Week 08 (Notes 06)

    • Naive Bayes Classifier

    • Naïve Bayes Classifier for Text Classification

    • Bayesian Networks Introduction

  • Week 09 (Notes 07)

    • Perceptron and Perceptron Classifier, Perceptron Learning Algorithm and its Geometric Intuition

    • Perceptron Learning Algorithm Convergence

  • Week 10

    • Hard SVM, Soft SVM, Kernel Trick (Notes 08)

  • Week 11 (Notes 09)

    • Neural Networks Introduction, Model, Forward Pass

    • Neural Networks: Back Propagation

  • Week 12 (Notes 10)

    • Unsupervised Learning, Clustering Overview

    • K-means Clustering

    • Agglomerative Clustering

  • Week 13

    • Decision Trees (Notes 11)

    • Bagging, Random Forest and Boosting

  • Week 14

    • Introduction to Deep Learning and Convolutional Neural Networks (Notes 12