This course provides a comprehensive introduction to the fundamental optimization theory and algorithms essential for modern data science. Students will explore how optimization underpins machine learning, statistical modeling, and large-scale data analysis. The course covers unconstrained and constrained optimization techniques, starting with core concepts like convexity, gradients, and Hessians. Key topics include Gradient Descent, Stochastic Gradient Descent (SGD), Newton's Method, and quasi-Newton methods (like L-BFGS). Furthermore, the course will delve into Lagrange multipliers and the Karush-Kuhn-Tucker (KKT) conditions for constrained problems, with practical applications in models like Support Vector Machines (SVMs) and regularized linear regression (Lasso and Ridge). Emphasis will be placed on the practical implementation, computational efficiency, and scalability of these methods in a high-dimensional data context, using popular data science programming environments.

Basic knowledge of calculus, linear algebra, probability theory, and machine learning principles

- Convex Optimization: Algorithms and Complexity, Sebastien Bubeck
- Introductory Lectures on Convex Optimization: A Basic Course, Yurii Nesterov
- Numerical Optimization, Jorge Nocedal and Stephen J. Wright

Week 1: Intro to Optimization
Week 2: Overview of Convex Optimization
Week 3: Linear Programming (LP), Least Squares (LS)
Week 4: Quadratic Programming (QP)
Week 5: Second-Order Cone Programming (SOCP)
Week 6: Semi-Definite Programming (SDP)
Week 7: Strong/Weak Duality
Week 8: Midterm exam
Week 9: Gradient descent
Week 10: Adaptive methods
Week 11: Second order methods
Week 12: Dual methods
Week 13: Distributed optimization
Week 14: Nonconvex optimization
Week 15: Advanced topics
Week 16: Final exam

- This course is a standard lecture-based course.
- Students will need to take two exams; there is no term project.
- Students will be evaluated with letter grading.