E1 260 Optimization for Machine Learning and Data Science (OptML)

About E1 260 OptML

The main goal of E1 260 course is cover optimization techniques suitable for problems that frequently appear in the areas of data science, machine learning, communications, and signal processing. This course focusses on the computational, algorithmic, and implementation aspects of such optimization techniques.

This is 3:1 credit course.

Prerequisite

Basic linear algebra, probability, and knowledge of Python to conduct simulation exercises.

Lectures

  • Tuesdays and Thursdays 2:00-3:30 pm (Online via MS Teams).

Syllabus

Mathematical background, theory of convex functions, gradient methods, accelerated gradient methods, proximal gradient descent, mirror descent, sub gradient methods, stochastic gradient descent and variants, Project gradient descent and Frank-Wolfe, alternating direction method of multipliers, nonconvex and submodular optimization.

Textbooks

  • A. Beck, First-Order Methods in Optimization, MOS-SIAM Series on Optimization, 2017.

  • S. Bubeck, Convex Optimization: Algorithms and Complexity, Foundations and Trends in Optimization, 2015.

  • F. Bach, “Learning with Submodular Functions: A Convex Optimization Perspective”, Foundations and Trends in Machine Learning, Now Publishers Inc.

  • Other useful resources

  • S. Boyd, N. Parikh, and E. Chu,“ Distributed optimization and statistical learning via the alternating direction method of multipliers”, Foundations and Trends in Machine Learning, Now Publishers Inc.

Course requirements and grading

  • Five assignments (problems and programming): 10% each, i.e., 50% in total

  • Two projects: 20% each, 40% in total

  • Final assessment: 10%

Schedule (pdf)

Lecture numberDate Topic Reading Materials Exercises
1 10-08-2021 Introduction slides
2 02-08-2021 Mathematical background Rate of convergence (Chapter 4.2) lecture notes
3 17-08-2021 Mathematical background lecture notes
4 24-08-2021 Theory of convex function lecture notes
5 26-08-2021 Theory of convex function lecture notes
6 31-08-2021 Theory of convex function lecture notes
7 31-08-2021 Theory of convex function lecture notes Homework 1
8 2-09-2021 Gradient descent lecture notes
9 9-09-2021 Gradient descent lecture notes
10 16-09-2021 Acceleration methods lecture notes
11 21-09-2021 Acceleration methods lecture notes
12 23-09-2021 Projected gradient descent lecture notes
13 25-09-2021 Subgradient method lecture notes
14 28-09-2021 Proximal method lecture notes Homework 2
15 5-10-2021 Frank Wolfe lecture notes
16 7-10-2021 Mirror descent lecture notes
17 12-10-2021 Mirror descent lecture notes Homework 3
18 14-10-2021 Stochastic gradient descent lecture notes
19 21-10-2021 Stochastic gradient descent lecture notes
20 26-10-2021 Stochastic variance reduced gradient lecture notes Homework 4
21 28-10-2021 Duality and KKT conditions lecture notes
22 9-11-2021 ADMM lecture notes
23 16-11-2021 ADMM lecture notes
24 16-11-2021 Submodular optimization lecture notes