CS 4501 Natural Language Processing (Fall 2024)

Logistics

Course Overview

In this course, we will explore the foundational concepts and the latest advancements in Natural Language Processing (NLP). The course aims to provide a thorough understanding of both traditional and modern approaches to language modeling, from N-gram language models to large language models (LLMs). We’ll start by covering the fundamentals of language representation, including classic methods such as word embeddings and vector space models. As we progress, we will introduce the Transformer architecture, which underpins today’s most powerful LLMs, and explore how techniques like pretraining and fine-tuning have transformed the field. In the latter part of the course, we will examine advanced topics like reasoning, factual knowledge integration, and Reinforcement Learning from Human Feedback (RLHF), which are essential for building and deploying NLP systems in practical, real-world scenarios.

Grading

Late Day Policy (Only Applied to Assignments, Not Projects!):

Policy on Using LLMs/GenAI:

Collaborative coding with LLMs is allowed, but if you directly copy the answers generated by LLMs (for either conceptual or coding questions), you’ll get a 0 for that entire assignment!

Schedule

WeekDateTopicDeadline
108/28Course Logistics & Overview 
 08/30Course Overview (Continued) 
209/02Intro to Language Modeling & N-gram Language ModelsAssignment 1 out
 09/04N-gram Language Models (Continued) 
 09/06Smoothing & Evaluation of N-gram Language Models 
309/09Intro to Word Senses & SemanticsAssignment 2 out
 09/11Classic Word RepresentationsAssignment 1 due: 09/11 11:59pm
 09/13Vector Space Models 
409/16Word Embeddings 
 09/18Word Embeddings: Word2Vec 
 09/20Word Embeddings (Continued)Project proposal due: 09/20 11:59pm (Guideline)
509/23Intro to Sequence Modeling & Neural Language Models 
 09/25Recurrent Neural Networks (RNNs)Assignment 2 due: 09/25 11:59pm
 09/27RNNs (Continued)Assignment 3 out
609/30Intro to Transformers 
 10/02(No Class) 
 10/04Self-attention 
710/07Transformer Language Model 
 10/09Language Model Pretraining 
 10/11Language Model Pretraining & Fine-tuningAssignment 3 due: 10/11 11:59pm
810/14Fall Reading Days (No Class) 
 10/16Intro to Large Language Models (LLMs)Assignment 4 out
 10/18In-context LearningProject mid-term report due: 10/18 11:59pm (Guideline)
910/21LLM Scaling Laws 
 10/23LLM for Reasoning 
 10/25Question Answering & Parametric Knowledge 
1010/28Retrieval & Non-parametric Knowledge 
 10/30Retrieval-Augmented Generation (RAG) 
 11/01Long-context Issues & Intro to Alignment 
1111/04Instruction TuningAssignment 4 due: 11/04 11:59pm
 11/06Reinforcement Learning from Human Feedback (RLHF) 
 11/08Guest Lecture: Wei Xiong (UIUC) 
1211/11Multimodal LLMsAssignment 5 out
 11/13LLM Agents 
 11/15Guest Lecture: Jiaming Shen (Google DeepMind) 
1311/18Summary & Future of NLP (I) 
 11/20Summary & Future of NLP (II) 
 11/22Guest Lecture: Yizhong Wang (University of Washington) 
1411/25Thanksgiving Recess (No Class) 
 11/27Thanksgiving Recess (No Class) 
 11/29Thanksgiving Recess (No Class)Project presentation slides due: 12/01 11:59pm (Guideline)
1512/02Project PresentationAssignment 5 due: 12/02 11:59pm
 12/04Project Presentation 
 12/06Project PresentationProject report due: 12/13 11:59pm (Guideline)

Useful Materials

Students with Disabilities or Learning Needs

It is my goal to create a learning experience that is as accessible as possible. If you anticipate any issues related to the format, materials, or requirements of this course, please meet with me outside of class so we can explore potential options. Students with disabilities may also wish to work with the Student Disability Access Center (SDAC) to discuss a range of options to removing barriers in this course, including official accommodations. You may email an SDAC advisor at cmacmasters@virginia.edu to schedule an appointment. For general questions please visit the SDAC website: sdac.studenthealth.virginia.edu. If you have already been approved for accommodations through SDAC, please send me your accommodation letter and meet with me so we can develop an implementation plan together.

Religious Accommodations

It is the University’s long-standing policy and practice to reasonably accommodate students so that they do not experience an adverse academic consequence when sincerely held religious beliefs or observances conflict with academic requirements.

Students who wish to request academic accommodation for a religious observance should submit their request to me by email as far in advance as possible. Students who have questions or concerns about academic accommodations for religious observance or religious beliefs may contact the University’s Office for Equal Opportunity and Civil Rights (EOCR) at UVAEOCR@virginia.edu or 434-924-3200.

Harassment, Discrimination, and Interpersonal Violence

The University of Virginia is dedicated to providing a safe and equitable learning environment for all students. If you or someone you know has been affected by power-based personal violence, more information can be found on the UVA Sexual Violence website that describes reporting options and resources available - www.virginia.edu/sexualviolence.

The same resources and options for individuals who experience sexual misconduct are available for discrimination, harassment, and retaliation. UVA prohibits discrimination and harassment based on age, color, disability, family medical or genetic information, gender identity or expression, marital status, military status, national or ethnic origin, political affiliation, pregnancy (including childbirth and related conditions), race, religion, sex, sexual orientation, or veteran status. UVA policy also prohibits retaliation for reporting such behavior.

If you witness or are aware of someone who has experienced prohibited conduct, you are encouraged to submit a report to Just Report It (justreportit.virginia.edu) or contact EOCR, the office of Equal Opportunity and Civil Rights.

If you would prefer to disclose such conduct to a confidential resource where what you share is not reported to the University, you can turn to Counseling & Psychological Services (“CAPS”) and Women’s Center Counseling Staff and Confidential Advocates (for students of all genders).

As your professor and as a person, know that I care about you and your well-being and stand ready to provide support and resources as I can. As a faculty member, I am a responsible employee, which means that I am required by University policy and by federal law to report certain kinds of conduct that you report to me to the University’s Title IX Coordinator. The Title IX Coordinator’s job is to ensure that the reporting student receives the resources and support that they need, while also determining whether further action is necessary to ensure survivor safety and the safety of the University community.

Student Support Team

You have many resources available to you when you experience academic or personal stresses. In addition to the course staff, the School of Engineering and Applied Science has staff members located in Thornton Hall who you can contact to help manage academic or personal challenges. Please do not wait until the end of the semester to ask for help!

Community and Identity

The Center for Diversity in Engineering (CDE) is a student space dedicated to advocating for underrepresented groups in STEM. It exists to connect students with the academic, financial, health, and community resources they need to thrive both at UVA and in the world. The CDE includes an open study area, event space, and staff members on site. Through this space, we affirm and empower equitable participation toward intercultural fluency and provide the resources necessary for students to be successful during their academic journey and future careers.