CS 6501 Natural Language Processing (Spring 2025)

Logistics

Instructor: Yu Meng (yumeng5[at]virginia[dot]edu)
Teaching Assistants: Peng Wang (pw7nc[at]virginia[dot]edu)
Time: Mondays & Wednesdays 2:00pm - 3:15pm
Office Hours: Mondays & Wednesdays after class
Location: Olsson Hall 005

Course Overview

This advanced graduate-level course offers a comprehensive exploration of cutting-edge developments in the field of natural language processing (NLP). With Large Language Models (LLMs) serving as the foundation for state-of-the-art NLP systems, we will cover various topics aiming at gaining a better understanding of LLMs’ design, capabilities, limitations, and future prospects. Key areas include model architecture and design, training methodologies (e.g., pretraining, instruction tuning, RLHF), emergent capabilities (e.g., in-context learning, reasoning), parametric knowledge with retrieval-augmented generation, efficiency (e.g., parameter-efficient training, sparse methods), language agents (e.g., multimodality, computer use), and ethics. This course will be highly research-driven with a substantial focus on reading, presenting and discussing important papers and conducting research projects.

Grading

Paper Presentation (30%) (Signup Sheet): In each lecture starting from the 4th lecture (1/27), a group of 1-2 students will be tasked to present 3 papers selected for the topic they signed up for. The primary objective is to impart knowledge to the rest of the class. The presentation duration is strictly limited to 60 minutes, followed by a 10-minute question-and-answer session with the audience. The presentation will be assessed based on the following criteria (everyone from the same presentation group receives the same score):
- Clarity: Whether the presentation effectively communicates the core concepts and insights contained in the papers.
- Completeness: Whether the presentation adequately covers the essential messages in the 3 assigned papers within the allocated timeframe.
- Teamwork: Whether the presentation is prepared and delivered by all team members.
- Question answering: Whether the presenters effectively answer the questions raised by the audience.
Tips for presentation preparation: It is not necessary to cover every detail of the papers (e.g., if a paper includes lengthy appendices, it is not necessary to discuss them in depth); rather, emphasis should be placed on conveying general ideas and insights: For theoretical papers, you don’t need to go over each proof in detail, but need to explain the major conclusions/insights of the theoretical analysis. For empirical papers, you don’t need to present every piece of experiment results, but need to articulate how the empirical findings support the major claims. A good presentation should highlight
- The major contributions of the paper
- Why these contributions are deemed important (e.g., did they reveal any previously unknown facts or change people’s opinions on a widely acknowledged phenomenon?)
- The most important technical details (e.g., the motivation behind a new training objective/model architecture design and the corresponding implementation)
- The limitations of the work and how they might be addressed in the future
Deadline for slides submission: Send your slides to the instructor and TA via email at least 48 hours before your presentation (e.g., if presenting on Monday, slides should be submitted by Saturday 2:00 pm). You will receive feedback from the instructor to improve your slides, and if necessary, the instructor may schedule a meeting with your team to go over the slides. Late submissions after the deadline will result in a 50% presentation grade deduction.
Participation (20%): For each lecture starting from the 4th lecture (1/27), everyone is required to complete the following two mini-assignments (regardless of whether or not you have signed up to present for that lecture):
- Pre-lecture question: Your task is to read the 3 papers to be introduced in the lecture, and submit a question you have when you read them. The question must not be trivial (e.g., “Does the proposed method work?” / “What is the aim of the paper?”). You are also welcome to raise these questions in the 10-minute question-and-answer session during class. The deadline is one day before the lecture (e.g., For Monday lectures, you need to submit the question by Sunday 11:59 pm).
- Post-lecture feedback: After attending the lecture, you are required to provide feedback to the presenters. You should comment on the clarity, completeness, depth, etc. Your feedback doesn’t have to be long, but should be specific and constructive. The deadline is each Friday (both Monday & Wednesday feedback is due Friday 11:59 pm).
Project (50%): By the end of this course, you are required to complete a research project, present your results, and submit a project report. You are required to work in a team of 1 or 2 students (a larger team size requires prior approval from the instructor). There are two acceptable project types:
- A comprehensive survey report: The survey should carefully examine and summarize existing literature on a topic covered in this course, and provide detailed and insightful discussions on the unresolved issues, challenges, and potential future opportunities within the chosen topic.
- A hands-on project: The project is not constrained to the course topics but must be centered around NLP. The project does not have to involve large language models either. For example, you may choose to train or analyze smaller-scale language models for specific tasks. You are eligible to receive extra credits if the final project reaches a publishable state.
The project grading breakdown (50%) is as follows:
- Project proposal (Guideline): 5% (Deadline: 2/5)
- Mid-term report (Guideline): 10% (Deadline: 3/10)
- Final presentation (Deadline: 4/15; Guideline) and final report (Deadline: 5/6; Guideline): 35%

Schedule (Subject to Changes!)

Date	Topic	Papers	Slides
Introduction to Large Language Models
1/13	Course Overview	-	overview
1/15	Language Model Architecture	Distributed Representations of Words and Phrases and their Compositionality (word2vec) Attention Is All You Need (Transformer)	lm_basics
1/20	No Class (MLK Holiday)	-	-
1/22	Language Model Pretraining & Fine-Tuning	Language Models are Unsupervised Multitask Learners (GPT-2) LLaMA: Open and Efficient Foundation Language Models BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (T5)	pretrain
1/27	In-Context Learning	An Explanation of In-context Learning as Implicit Bayesian Inference Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? Many-Shot In-Context Learning	llm_icl
1/29	Scaling and Emergent Ability	Training Compute-Optimal Large Language Models Emergent Abilities of Large Language Models Are Emergent Abilities of Large Language Models a Mirage?	scaling
2/3	Guest Lecture: Ruiqi Zhong (UC Berkeley)	-	-
Reasoning with Language Models
2/5	Chain-of-Thought Prompting	Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Tree of Thoughts: Deliberate Problem Solving with Large Language Models Least-to-Most Prompting Enables Complex Reasoning in Large Language Models	cot
2/10	Guest Lecture: Weijia Shi (University of Washington)	-	-
2/12	Inference-Time Scaling	Self-Consistency Improves Chain of Thought Reasoning in Language Models Large Language Monkeys: Scaling Inference Compute with Repeated Sampling Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters	inf_scale
Knowledge, Retrieval-Augmented Generation, and Efficiency
2/17	Parametric Knowledge in Language Models	Language Models as Knowledge Bases? Transformer Feed-Forward Layers Are Key-Value Memories Locating and Editing Factual Associations in GPT	knowledge
2/19	Retrieval-Augmented Language Generation (RAG)	Generalization through Memorization: Nearest Neighbor Language Models Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks Dense Passage Retrieval for Open-Domain Question Answering	rag
2/24	Long Context Language Models	Lost in the Middle: How Language Models Use Long Contexts LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning Retrieval meets Long Context Large Language Models	long_context
2/26	Efficiency	Mamba: Linear-Time Sequence Modeling with Selective State Spaces Efficient Streaming Language Models with Attention Sinks Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity	efficiency
Language Model Post-Training
3/3	Instruction Tuning	Finetuned Language Models Are Zero-Shot Learners Cross-Task Generalization via Natural Language Crowdsourcing Instructions Self-Instruct: Aligning Language Models with Self-Generated Instructions	instruct_tune
3/5	Reinforcement Learning from Human Feedback (RLHF)	Training language models to follow instructions with human feedback Direct Preference Optimization: Your Language Model is Secretly a Reward Model SimPO: Simple Preference Optimization with a Reference-Free Reward	rlhf
3/10	No Class (Spring Recess)	-	-
3/12	No Class (Spring Recess)	-	-
Language Agents
3/17	Language Agent Basics	ReAct: Synergizing Reasoning and Acting in Language Models Toolformer: Language Models Can Teach Themselves to Use Tools Reflexion: Language Agents with Verbal Reinforcement Learning	agent
3/19	Language Models for Code	InCoder: A Generative Model for Code Infilling and Synthesis SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering Teaching Large Language Models to Self-Debug	code_lm
3/24	Multimodal Language Models	Flamingo: a Visual Language Model for Few-Shot Learning Visual Instruction Tuning (LLaVA) VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction	multimodal
Ethical Considerations and Evaluations of Language Models
3/26	Security and Jailbreaking	DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models Universal and Transferable Adversarial Attacks on Aligned Language Models Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation	security
3/31	Guest Lecture: Chenglei Si (Stanford)	-	-
4/2	Guest Lecture: Mengzhou Xia (Princeton)	-	-
4/7	Bias and Calibration	RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP LLM Evaluators Recognize and Favor Their Own Generations	bias
4/9	Privacy and Legal Issues	Extracting Training Data from Large Language Models Large Language Models Can Be Strong Differentially Private Learners Quantifying Memorization Across Neural Language Models	privacy
4/14	Guest Lecture: TBD	-	-
4/16	Project Presentations	-	-
4/21	Project Presentations	-	-
4/23	No Class	-	-
4/28	No Class	-	-

Useful Materials

For NLP background: Speech and Language Processing
For deep learning background: Deep Learning

Students with Disabilities or Learning Needs

It is my goal to create a learning experience that is as accessible as possible. If you anticipate any issues related to the format, materials, or requirements of this course, please meet with me outside of class so we can explore potential options. Students with disabilities may also wish to work with the Student Disability Access Center (SDAC) to discuss a range of options to removing barriers in this course, including official accommodations. You may email an SDAC advisor at cmacmasters@virginia.edu to schedule an appointment. For general questions please visit the SDAC website: sdac.studenthealth.virginia.edu. If you have already been approved for accommodations through SDAC, please send me your accommodation letter and meet with me so we can develop an implementation plan together.

Religious Accommodations

It is the University’s long-standing policy and practice to reasonably accommodate students so that they do not experience an adverse academic consequence when sincerely held religious beliefs or observances conflict with academic requirements.

Students who wish to request academic accommodation for a religious observance should submit their request to me by email as far in advance as possible. Students who have questions or concerns about academic accommodations for religious observance or religious beliefs may contact the University’s Office for Equal Opportunity and Civil Rights (EOCR) at UVAEOCR@virginia.edu or 434-924-3200.

Harassment, Discrimination, and Interpersonal Violence

The University of Virginia is dedicated to providing a safe and equitable learning environment for all students. If you or someone you know has been affected by power-based personal violence, more information can be found on the UVA Sexual Violence website that describes reporting options and resources available - www.virginia.edu/sexualviolence.

The same resources and options for individuals who experience sexual misconduct are available for discrimination, harassment, and retaliation. UVA prohibits discrimination and harassment based on age, color, disability, family medical or genetic information, gender identity or expression, marital status, military status, national or ethnic origin, political affiliation, pregnancy (including childbirth and related conditions), race, religion, sex, sexual orientation, or veteran status. UVA policy also prohibits retaliation for reporting such behavior.

If you witness or are aware of someone who has experienced prohibited conduct, you are encouraged to submit a report to Just Report It (justreportit.virginia.edu) or contact EOCR, the office of Equal Opportunity and Civil Rights.

If you would prefer to disclose such conduct to a confidential resource where what you share is not reported to the University, you can turn to Counseling & Psychological Services (“CAPS”) and Women’s Center Counseling Staff and Confidential Advocates (for students of all genders).

As your professor and as a person, know that I care about you and your well-being and stand ready to provide support and resources as I can. As a faculty member, I am a responsible employee, which means that I am required by University policy and by federal law to report certain kinds of conduct that you report to me to the University’s Title IX Coordinator. The Title IX Coordinator’s job is to ensure that the reporting student receives the resources and support that they need, while also determining whether further action is necessary to ensure survivor safety and the safety of the University community.

Student Support Team

You have many resources available to you when you experience academic or personal stresses. In addition to the course staff, the School of Engineering and Applied Science has staff members located in Thornton Hall who you can contact to help manage academic or personal challenges. Please do not wait until the end of the semester to ask for help!

Community and Identity

The Center for Diversity in Engineering (CDE) is a student space dedicated to advocating for underrepresented groups in STEM. It exists to connect students with the academic, financial, health, and community resources they need to thrive both at UVA and in the world. The CDE includes an open study area, event space, and staff members on site. Through this space, we affirm and empower equitable participation toward intercultural fluency and provide the resources necessary for students to be successful during their academic journey and future careers.

Yu Meng (孟瑜)