CS7.501 Advanced NLP

Monsoon 2026

IIIT Hyderabad International Institute of Information Technology

Course Overview ↑ Top

Designed for those seeking an advanced understanding, this course explores in-depth topics in Natural Language Processing (NLP). It assumes foundational knowledge and may not suitable for absolute newcomers to the field. While specific prerequisites are not strictly enforced, prior experience is highly encouraged. Tutorials will be held to equip students with necessary mathematical and engineering background.

Previous Editions - Monsoon '25

Introduction to NLP - Spring '26


Instructors

Teaching Assistants

Logistics ↑ Top

All course related annoucements will be made on Moodle.

Teaching Assistant Office Hours

TA Name Office Hours
Manas Mittal Monday: 15:30 - 16:30 (Precog)*
Maitreya Chitale Monday: 21:00 - 22:00 (Online)*
Druhan Shah Wednesday: 11:30 - 12:30 (MT NLP Lab)
Vivek Hruday Thursday: 10:00 - 11:00 (IREL Lab)*
Ishaan Romil Thursday: 19:00 - 20:00 (MT NLP Lab)*
Yajat Rangnekar Friday: 11:30 - 12:30 (MT NLP Lab)*
Monish Singhal Email appointments only*

* Appointments to be requested via email the previous day by 23:59.

Course Schedule ↑ Top

# Date Topic Section Resources
1 30-Jul-26 Course Introduction Logistics, Introduction Backpropagation
Residual Network
Backpropagation
Yes you should understand backprop
Neural Machine Translation by Jointly Learning to Align and Translate
2 3-Aug-26 Recap Transformers Illustrated Transformer
Self Attention, Transformers
Intro to Transformers
Attention in Transformers
Attention is all you need
3 6-Aug-26 Transformers Tokenization WordPiece
SentencePiece
Byte Pair Encoding
Byte Latent Transformer
4 10-Aug-26 Positional Embedding What do position embeddings learn
Rethinking positional encoding in language pretraining
Rotary Position Embedding (RoPE)
Impact of PE on Length Generalization in Transformers
Length generalization with NoPE
5 13-Aug-26 Attention Self Attention from Scratch
MQA
GQA
DeepSeekV2 - MHLA
6 17-Aug-26 Attention - Advances Log Linear Attention
Linear Attention
Gated Attention
Gated Delta Networks
Kimi Linear
7 20-Aug-26 Normalization Layer Norm
Batch Norm
Weight Norm
RMS Norm
Pre Norm - On Layer Normalization in the Transformer Architecture
Normalization in Deep Learning
8 24-Aug-26 Feed Forward Layers Analyzing Feed-Forward Blocks in Transformers through the Lens of Attention Maps
Hands-on: Mixture of Experts with Transformers
9 27-Aug-26 Mixture of Experts Adaptive Mixture of Local Experts
Hierarchical Mixture of Experts
DeepSeek
DeepSpeed MoE
DeepSeek MoE
Mixture of Experts
A Visual Guide to Mixture of Experts
MoE Explained
10 31-Aug-26 Optimizers Gradient Descent
Optimizers
Adam
AdaGrad
RMSProp
Muon
Muon is Scalable for LLM Training
Practical Efficiency of Muon for Pretraining
11 3-Sep-26 Scaling LLMS TBA
12 7-Sep-26 Decoding & Generation Introduction to Decoding Hierarchical Neural Story Generation
Beam Search Strategies
Thorough Examination of Decoding Strategies in LLM Era
13 10-Sep-26 Speculative Decoding Speculative Decoding for Seq2Seq Generation
Speculative Decoding
Speculative Sampling
A Survey on Speculative Decoding
Looking back at speculative decoding
14 17-Sep-26 Advances in Decoding Min-p Sampling
Lookahead Decoding
15 28-Sep-26 Parallel Decoding Accelerating Transformer Inference for Translation via Parallel Decoding
Lookahead Decoding
Medusa
16 5-Oct-26 Evaluation Statistics of Evaluating in NLP TBA
17 8-Oct-26 Evaluation Metrics BLEU
ROUGE
BERTScore
COMET
MEE4
18 12-Oct-26 Benchmarking GLUE: A Multi-Task Benchmark and Analysis Platform for NLU
GEM Benchmark
Gptscore: Evaluate as you desire
M3t: A new benchmark dataset for multi-modal document-level machine translation
G-eval: Nlg evaluation using gpt-4 with better human alignment
A Comprehensive Survey on Sentence-Level Translation Evaluation
19 15-Oct-26 Quantization Quantization Visual Guide to Quantization
Smooth Quant
Optimal Brain Compression (OBQ)
LoRA: Low Rank Adaptation
QLoRA: Efficient Finetuning of Quantized LLMs
GPTQ: Accurate Post-Training Quantization
AWQ: Activation-aware Weight Quantization
Zero Quant
LLM.int8
20 19-Oct-26 Alternate Architectures Current RNN Advances TBA
21 22-Oct-26 State Space Models Mamba
22 26-Oct-26 Generative Modeling TBA
23 29-Oct-26 Diffusion Language Models TBA
24 2-Nov-26 Reasoning in Models RL in NLP Training models to follow instructions with human feedback
Scaling Instruction tuned Model
AlpacaFarm
25 5-Nov-26 Policy Optimizations PPO
DPO
GRPO
26 12-Nov-26 Reasoning Models TBA
27 16-Nov-26 Reasoning Models TBA
28 19-Nov-26 Whats Next

Due Dates ↑ Top

Important dates for the course:

Component Due Date
Team Details Submission 6-Aug
A1 Release 10-Aug
Project Proposals (Interim) 14-Aug
Proposal (Interim) Rejection by TAs 16-Aug
Project Finalization, Mentor Assignment 18-Aug
Proposal (Final) 28-Aug
A1 Due 2-Sep
Proposal Grades 3-Sep
A2 Release 6-Sep
A1 Grades 10-Sep
Project Mid Submission 30-Sept
A2 Due 4-Oct
A3 Release 8-Oct
Project Mid Grades 11-Oct
A2 Grades 14-Oct
Final Project Due 31-Oct
A3 Due 13-Nov
A3 Grades 19-Nov
Final Grade Assignment 7-Dec

Grading and Weightage ↑ Top

Your final grade in this course will be determined by the following components:

Component Weightage
Assignments 6 + 7 + 7 + 10 (30%)
Exams 10 + 10 (20%)
Seminar (Optional) 10 (10%)
Project 50%
Total 100%

Course Projects ↑ Top

The course project is a significant component of this course (50%), providing an opportunity to apply the concepts and techniques learned to a real-world NLP problem. Students will work in teams of 3 or 4 to propose, develop, and present a project. Projects are expected to be at an advanced level. Submission to conferences/workshops is highly encouraged. Read more on research opportunities.

We encourage students to think creatively and explore areas that genuinely interest them.


Component Weightage
Project Outline 10%
Project Mid 20%
Project Final 20%
Total 50%

Project Areas

  • Efficient/Low-Resource Methods
  • Generation/Language Modeling
  • Interpretability/Explainability
  • Multilingual/Cross-lingual
  • Multimodal
  • NLP Applications
  • Evaluation/Benchmarking
  • Retrieval & Extraction
  • Document Understanding
  • Graph + LLMs
  • Reasoning in LLM
  • Diffusion Language Models

Seminar ↑ Top

An optional seminar shall be available for up to 16 students as a substitute for an exam. Approved students will need to thoroughly understand and present a topic to the class and engage in a QnA with the audience. Students will be evaluated on topic understanding, coverage, presentation and ability to answer questions. The seminar will involve combining multiple papers to present the topic (akin to a survey paper), instead of presenting just one paper.

Seminar Topics

Assignments and Project Policies ↑ Top

Late Submission Policy

Assignment and project deadlines are strictly enforced. Late submissions will not be allowed and no extensions will be provided.

Collaboration Policy

Collaboration on assignments is encouraged for discussion of concepts and general approaches, but all submitted answers must be your individual work. For projects, group collaboration is expected, and the contribution of each member should be clearly documented. Any specific collaboration guidelines for individual assignments or projects will be provided with the assignment description.

Consultation and Research Opportunities

Students are highly encouraged to consult with the instructor, teaching assistants and mentors during office hours or by appointment for guidance on assignments, projects, or course topics. If a project developed in this course shows significant potential for a research paper, students may, at their discretion, invite a TA, a mentor or the instructor whose advice substantially benefited the work to be a co-author. This is not a requirement, but can be seen as an opportunity to acknowledge significant contributions and foster academic collaboration.

Other Resources ↑ Top

  • Deep Learning by Ian Goodfellow (Book)
  • Understanding Deep Learning by Simon Prince (Book)
  • Mathematics for Machine Learning by Deisenroth, Faisal, Ong (Book)
  • Build a large language model (from scratch) by Sebastian Raschka (Book + Code)
  • Dive into Deep Learning (Book)
  • PyTorch Tutorials
  • 🤗 Transformers