Machine Learning for Signal Processing
A.Y. 2020/2021
A.Y. 2020/2021
General Information
Teacher: Danilo Comminiello
Credits: 6 CFU
Scientific sector: ING-IND/31
Course language: English
Offered programs: Laurea Magistrale in Ingegneria Elettronica (LMIE) and Master Degree in Electronics Engineering (MDEE). The course is also commuted for the Laurea Magistrale in Ingegneria delle Comunicazioni (MCOR). However, the course can be also freely chosen and attended by interested students from other degree programs of Sapienza University, upon approval of the study plan.
Calendar: February 25 - May 21, 2021
Class timing: Thursday 8:30-10:00, Friday 8:30-10:45 - Online
Classroom page: Download slides, videolectures, Python notebooks, homeworks and additional material. Access code: h7rggk5.
Online lessons: Connect with your Sapienza account at this Zoom link.
Office hours: by appointment
Course Description
Information conveyed in real-world signals may be often affected by noise, partially corrupted or even unavailable. Thus, extracting desired information from real-world signals can be even very complicated. Machine learning for signal processing (MLSP) is the science that deals with the development of efficient algorithms and models that are able to detect and unveil a possible hidden structure in signals, thus recovering a desired information. This process is autonomously and automatically performed by MLSP algorithms, by simply learning from the available data, which is the basis of any science related to artificial intelligence.
This course aims at presenting the main machine learning paradigms and applying them for the processing of a variety of signals, including audio and speech, images, movies, music, biological, electrical and mechanical, among many others. The course is based on regular classroom lessons, which also include regular exercises in Python on practical problems.
Prerequisites
Most of the prerequisites will be briefly recalled in classes. However, basic knowledge of linear algebra, signal theory and stochastic processes are warmly recommended, as well as basic programming skills.
Exam assessment and grade evaluation
The theoretical skills acquired by the student will be evaluated as well as the ability to apply and implement a specific methodology in a practical problem. Exam grades (in thirties) will be based on 3 homework assignments, respectively counting as the 15%, 35% and 50% of the final score. Upon request, the third homework can be replaced by a small project. From the exam session of October 2021 on, the Homework 3 will be replaced by a small project.
Program
Introduction to MLSP. Introduction to signals, time series and possible representations. Introduction to regression and classification problems. MLSP taxonomy. Typical applicative examples in MLSP. Brief recall of linear algebra, probability and random variables. Practical examples using typical MLSP data and scenarios.
Learning in parametric modeling. Parametric estimation. Bias-variance dilemma. Cost functions and optimization methods for machine learning. Least-square optimal estimation. Linear regression. Stochastic gradient descent algorithms. Introduction to Classification. Bayesian classification. Logistic regression. A tour on classic machine learning classifiers. Regularization and sparsity-aware learning. Nonlinear modeling and estimation. Regression and classification with kernels. Examples.
Deep learning for signal processing. Introduction to neural networks, universal approximation, backpropagation, feed-forward neural networks. Deep learning basics. Convolutional neural networks. Recurrent neural networks for signal processing. Long short-term memory models.
Modern deep learning architectures. Advanced deep neural networks. Examples with application to audio/image/music/biomedical signals.
Textbook and Material
Main textbook:
Sergios Theodoridis, "Machine Learning: A Bayesian and Optimization Perspective", 2nd ed., Elsevier, 2020.
Further references:
Aston Zhang and Zachary C. Lipton and Mu Li and Alexander J. Smola, "Dive Into Deep Learning", 2020 (free pdf available online).
Kevin P. Murphy, "Machine Learning: A Probabilistic Perspective". The MIT Press, 2012.
Sebastian Raschka and Vahid Mirjalili, "Python Machine Learning". Packt Publishing, (3rd. ed.), 2019.
Christopher M. Bishop, "Pattern Recognition and Machine Learning". Springer, 2006 (free pdf available online).
Ian Goodfellow, Yoshua Bengio, Aaron Courville, “Deep Learning”. The MIT Press, 2016 (free pdf available online).
Danilo Comminiello and José C. Príncipe (Eds.), "Adaptive Learning Methods for Nonlinear System Modeling". Elsevier, 2018.
Supplementary material (e.g., course slides, papers) will be provided by the instructor.
Lectures
[L01] Machine Learning for Signal Processing: A Course Introduction. [February 25, 2021]
[L02] A Python Introduction to Machine Learning for Signal Processing. [February 26, 2021]
[L03] Learning Tasks in Machine Learning for Signal Processing. [March 4, 2021]
[L04] Linear Algebra for MLSP. [March 5, 2021]
[L05] Linear Regression. [March 11, 2021]
[L06] Applications of Linear Regression. [March 19, 2021]
[L07] Stochastic Gradient Descent. [March 25, 2021]
[L08] Introduction to Classification: Naive Classifiers. [March 26, 2021]
[L09] Logistic Regression. [April 8, 2021]
[L10] A Tour of Classic Machine Learning Classification Algorithms. [April 15, 2021]
[L11] Regularization and Sparsity-Aware Learning. [April 22/29, 2021]
[L12] Nonlinear Modeling and Learning with Kernels. [April 29/30, 2021]
[L13] From Shallow to Deep Learning. [May 6/7, 2021]
[L14] Deep Learning for Image and Spatial Information Processing. [May 14, 2021]
[L15] Deep Learning for Sequential Information Processing. [May 20, 2021]
Lab Sessions
*Lab notebooks are available on the Classroom page of the course with the code h7rggk5..
[EX01] Introduction to Python. [February 26, 2021]
[EX02] Elements of Linear Algebra and its Application to Signals. [March 5, 2021]
[EX03] Linear Algebra Operations in Signal Processing Problems. [March 12, 2021]
[EX04] Linear Regression. [March 12, 2021]
[EX05] Least-Square Linear and Nonlinear Regression. [March 18, 2021]
[EX06] Noise Cancellation: An MSE Linear Regression Application. [March 19, 2021]
[EX07] Naive Classifiers. [March 26, 2021]
[EX08] Logistic Regression in NumPy. [April 9, 2021]
[EX09] A Tour of Classic Machine Learning Classifiers Using Scikit-learn. [April 16/23, 2021]
[EX10] Playlist Classification on Spotify. [April 23, 2021]
[EX11] Nonlinear Learning with Kernels. [April 30, 2021]
[EX12] Implementation of Multilayer Perceptron from Scratch. [May 7, 2021]
[EX13] Building Neural Networks in TensorFlow. [May 13, 2021]
[EX14] Deep Convolutional Neural Networks. [May 21, 2021]
[EX15] Deep Recurrent Neural Networks. [May 21, 2021]
Exam dates
Students are reminded that exams are booked electronically via the INFOSTUD portal. Booking for extraordinary exam sessions is allowed only to part-time students, students enrolled in supplementary years or non-conventional paths, and graduating students.
Scheduled exam sessions for the year 2020/2021:
Extraordinary Easter Session: April 1, 2021 (upon request only)
Session I: June 22, 2021
Session II: July 16, 2021
Session III: September 10, 2021
Extraordinary Session I: October 19, 2021
Session IV: January 13, 2022
Session V: February 14, 2022
Extraordinary Session II: April 6, 2022
Past academic years
MLSP 2017/2018