| Primary Instructor (IC) | Tirtharaj Dash |
|---|---|
| Teaching Assistants (TAs) | Atharv Sonwane Anmol Agarwal Ameya Laad Akhilesh Adithya |
Course handout can be found here.
Neural Networks has had a long and rich history, and the reincarnated viewpoint has shifted towards "Deep Neural Networks" or "Deep Learning", primarily due to, (a) availability of the large amount of data, (b) extensive use of powerful graphics processors, (c) availability of software libraries to facilitate deep network implementations, and (d) significant involvement of industrial research labs in deep learning research.
This course on "Deep Learning" would focus on the conceptual and mathematical foundation and computational investigations of recent deep models as part of a series of laboratory experiments and projects. For instance, we will focus on newer convolutional neural networks such as VGG Net, ResNet; various sequence models including attention-based models such as transformers; and also we will touch upon graph representation learning using graph neural networks.
At the end of this course, students should be able to (0) pose real-world problems in deep learning, (1) source and prepare datasets, (2) design suitable deep network architecture, (3) prepare input-output representation (and encodings), (4) decide and design a suitable loss function for training a deep network, (5) training and deploying deep models.
Primary textbooks:
- A. Zhang, Z.C. Lipton, M. Li, A.J. Smola, Dive into Deep Learning.
- Ian Goodfellow, Yoshua Bengio, Aaron Courville, Deep Learning, MIT Press.
- Aggarwal, C. C. (2018). Neural networks and deep learning, Springer.
- W.L. Hamilton, Graph Representation Learning Book.
Other good reference books include:
- Graves, A. (2012). Supervised sequence labelling with recurrent neural networks.
- Francois Chollet, Deep Learning with Python, Manning Publishers.
- E. Stevens, L. Antiga, T. Viehmann, Deep Learning with PyTorch, Manning Publishers.
Textbook-3 is available in our library. Other books are available as
e-books as href-ed.
Additioanlly, we may also look at relevant papers from
NeurIPS, ICLR,
ICML, IJCAI,
AAAI. These will be
primarily used during major and minor projects design.
References to the relevent study materials are provided in the lecture slides and in some cases, these are made available separately. Duration of each lecture is 50 min, which may get extended by 10-15 min. I am thankful to my students for tolerating this aspect of my teaching. The recoded videos of live lectures are accessible to the registered students only. The slides and videos may not be in sync in some cases; that is, any slide may span over multiple lecture slots.
| Lecture | In-class Lecture | Extra Materials |
|---|---|---|
| Lecture 0: Introducing the course | slides, video | |
| Tutorial 1: NN Basics | slides, video | |
| Tutorial 2: NN Basics | slides, video1, video2 | A note on Perceptron |
| Tutorial 3: Optimisation Basics | slides | |
| Lecture 1: DL Components and a Bayesian view | slides, video | |
| Lecture 2: Deep Feedforward Nets (MLPs) | slides, video | |
| Lecture 3: Backpropagation in MLP(1) | slides, video | Homework |
| Lecture 4: Backpropagation in MLP(2) | slides, video | Vectorisation, codes |
| Lecture 5: Generalisation(1) | slides, video | What is a dataset? |
| Lecture 6: Generalisation(2) | slides, video | |
| Lecture 7: Dropout, Early Stopping | slides, video | Dropout Paper |
| Lecture 8: Optimisation and Parameter Initialisation | slides, video | |
| Lecture 9: Adaptive Optimisation Methods | slides, video | Adam Paper |
| Lecture 10: Template Matching and CNNs | slides, video | Yann LeCun's CNN Paper |
| Lecture 11: Fully-Connected Layers to Convolutions | slides, video | |
| Lecture 12: CNN Operators (Convolution, Pooling) | slides, video | |
| Lecture 13: Gradient Computation in CNNs | Notes, video | |
| Lecture 14: Modern Convolutional Neural Networks | materials, video | AlexNet, VGG, NiN, GoogLeNet, ResNet |
| Lecture 15: Transfer Learning | video | Notebook |
| Lecture 16: Sequence Learning | materials:8.1-8.3, video | board |
| Lecture 17: Recurrent Neural Nets | video | board |
| Lecture 18: Backprop Through Time (BPTT) | slides, video | |
| Midsem solution discussion | video | |
| Lecture 19: Dealing with Vanishing Gradients in RNN | slides, video | |
| Lecture 20: Seq-to-Seq Learning, Attention Mechanism | slides, video | |
| Lecture 21: Attention and Self-attention | slides, video | Attention is all you need. |
| Lecture 22: Multi-head Attention and Transformer | slides, video | Layer Normalisation |
| Lecture 23: Representation Learning | slides, video | |
| Lecture 24: Representation Learning | slides, video | |
| Lecture 25: Generative Modelling (Intro. to VAE) | slides, video | |
| Lecture 26: Variational AE | slides, video | |
| Lecture 27: Generative Adversarial Network (GAN) | slides, video | |
| Lecture 28: Energy-based Generative Modelling (RBM) | materials | |
| AI Symp 22: Tutorial Talk on GNNs | slides, video |
These slides and videos are not really optimised for general public; so, please email me directly if you find any error or mistake in my lecture slides or videos. This will help me correct these the next time. I sincerely thank the MathCha Editor which I use extensively to draw most of the diagrams.
All our labs will be based on PyTorch. See the official PyTorch Tutorials. We will also have 1 lab on basic Python and PyTorch.
Update: According to students' votes, we are fixing the lab timing to be: Wednesday 6-8 PM. Please make sure to go through the reading materials (given below) before the lab.
To run/edit the colab notebooks, please create a copy in your drive and work on that copy.
| Lab | Reading Material | Recording | Exercise Material |
|---|---|---|---|
| Lab 0: Basics of Python and PyTorch | Python Tutorial Series by Sentdex Python Intro by Justin Johnson Section 2.1-2.3 of d2l.ai |
video | Notebook1 Notebook2 |
| Lab 1: Autograd and MLP | Section 2.4 - 2.5 of d2l.ai Intro to Autograd |
video | Notebook |
| Lab 2: Regularisation | Section 4.5.2 - 4.5.3 of d2l.ai | video | Notebook |
| Lab 3: Hyperparameter Tuning | Section 4.4, 4.6 and 7.5 from d2l.ai Hyperparameter tuning |
video | Notebook 1 Notebook 2 |
| Lab 4: Intro to CNNs | Section 6 from d2l.ai | video | Notebook |
| Lab 5: Intro to CNNs (Continued) | Section 6 from d2l.ai | video | Notebook |
| Lab 6: Intro to RNNs | Section 8 from d2l.ai | part 1 | Intro, Exercises |
| Lab 7: Intro to Transformers and BERT | NeurIPS Paper: Attention is all you need | video | Official Pytorch Notebook |
| Extra 1: AI Symposium 2022 Talk-GNNs | Presentation: slides | video | Official Pytorch Notebook |
| Lab Project | Details | Submission Date | Submission Link |
|---|---|---|---|
| Project 1: Residual MLPs | Assignment 1 | Sep 25, 11:59 PM IST | Google Classroom |
| Project 2: Major Project | Assignment 2 | Dec 7, 11:59 PM IST | Google Classroom |
Non-profitable usage of the materials from this course is absolutely free. However, if you are tech-blogger or a course instructor or a student outside BITS Pilani (Goa Campus) and you are using our course materials for any non-profitable purposes, please credit this course page by providing a proper reference to this page. Any profitable usage of these materials needs permission from the owner (Tirtharaj Dash). Refer the license for more details.