Self-supervised learning (SSL) course at CS HSE

This repository contains materials from the lectures.

Usefull links

• Wiki course page: link

• Course page at hse.ru: link

• Lecture recordings on YouTube (in Russian): YouTube

Syllabus

Folders: Lectures, Seminars

• [week1] Introduction to Course, classical pretext tasks for SSL. Recordings: lecture, seminar

• [week2] What to do with a pretrained model? Probing, Linear classifier, Fine-tuning. Recordings: lecture, seminar

• [week3] Contrastive learning for images. Mutual information, SimCLR, MoCo, BYOL, SimSiam, SwAV. Recordings: lecture, seminar

• [week4] Masked Image Modeling. DINO, BEiT, MAE, MaskFeat. Different approaches to improving contrastive learning. Recordings: lecture, seminar

• [week5] Overview of Transformer-based language models. GPT, BERT, XLNet, RoBERTa, ALBERT, MASS, BART, ELECTRA. Recordings: lecture, seminar

• [week6] Model pre-training for the source code domain. code2vec, code2seq, CodeBERT, GraphCodeBERT, CodeT5, Codex. Recordings: lecture

• [week7] Diffusion models for NLP. Theory introduction, Multinomial Diffusion, D3PM, Diffusion-LM, DiffuSeq. Recordings: lecture, seminar

• [week8] Self-supervised learning for audio. Introduction to audio processing. CPC, Wav2Vec 2.0, HUBERT, Multi-format contrastive learning, BYOL-A. Recordings: lecture

• [week9] Self-supervised learning for graphs. Intro to representation learning on graphs. Review of approaches. Recordings: lecture

Homeworks

• [hw1] SSL with pretext tasks. Task description, Kaggle competition
• [hw2] Contrastive learning for images. Task notebook
• [hw3] Pre-training of language models. Task notebook
• [hw4] Pre-training of audio models. Task notebook

Course staff

• Alexander Shabalin

• Ildus Sadrtdinov