CSE 8803 DLT: Deep Learning for Text Data
Logistics
- Lecture time: Monday & Wednesday 3:00PM–4:15PM
- Location: CoC Room 52
- Instructor: Chao Zhang
- Teaching Assistant: Lingkai Kong (lkkong@gatech.edu)
- Piazza: Piazza will be the place for course discussions and announcements. If you have questions, please ask it on Piazza because 1) other students may have the same question; 2) you will get help faster compared to sending emails.
- Office Hours:
- Instructor Office Hour: Monday 4:20-5:20 PM
- TA Office Hour: Wednesday 4:20-5:20 PM
Overview
This course will introduce state-of-the-art machine learning techniques for mainstay problems in text data analysis, with particular emphasis on deep learning methods that have recently achieved enormous success. The students will learn about trending problems in this field, key methods for solving these problems, and their advantages and disadvantages. The students are also expected to read and present cutting edge research papers, as well as conduct a research oriented course project. The course will provide useful techniques for students who want to solve practical problems involving text data, and better prepare those who want to do edge-cutting research in text mining, information retrieval, natural language processing, and text-rich interdisciplinary research. The learning objective is that by the end of this course, the students are able to formulate their text analysis problems at hand, choose appropriate statistical models for the problems, and even come up with innovative solutions for solving open research problems in this field.
In the task space, we will cover a range of text data analysis tasks, including:
- Text representation learning
- Text classification (sentiment analysis, document classification)
- Sequence labeling (POS tagging, NER, event extraction)
- Knowledge graph construction and reasoning
- Text generation (text summarization, image captioning)
- Question answering and dialogue systems
In the methodology space, we will introduce state-of-the-art deep learning techniques for these problems, including:
- Deep representation learning (Word2Vec, Transformer, BERT)
- Sequence labeling models (CRF, LSTM-CRF)
- Deep transfer/multi-task learning
- Neural symbolic methods
- Deep generative models (VAE, GAN, and their variants)
Prerequisites for this course: (1) the students should be have basic knowledge in machine learning and taken a relevant course before (e.g., CX4240, CSE6740, CS4641); (2) the students should be comfortable with reading research papers and giving presentations; (3) the students should have solid programming skills—the course project can be programming demanding.
Content
- Background knowledge
- Course overview and logistics
- ML/DL review
- Text representation learning
- Bag-of-words, n-gram, TF-IDF
- Dimension reduction, matrix factorization, topic models
- Word2vec, Glove
- Transformer and BERT
- Deep text classification
- Traditional text classifiers: naive Bayes, logistic regression, SVM
- Deep classifiers: CNN, RNN and their variants
- Attention models
- Learning with less labeled data
- Transfer learning and fine-tuning
- Multitask learning
- Semi-supervised learning
- Weakly supervised learning
- Sequence labeling and structured prediction
- Motivating tasks: POS tagging, NER, event extraction
- Conditional random fields for sequence labeling
- Deep latent variable models for sequence labeling
- Text generation
- Motivating tasks: summarization, image captioning, translation
- Deep language models
- VAE for text generation
- Knowledge representation and reasoning
- Knowledge graph construction
- Knowledger graph reasoning
- Combining logic and neural models
- QA and dialogue systems
- Question answering with knowledge bases
- Dialogue systems
- Project presentations
Schedule
Date | Topic | Presentation | Due |
---|---|---|---|
8/19/19 | Course Overview | Piazza Signup | |
8/21/19 | Machine Learning Review | ||
8/26/19 | Text Representation Learning | Presentation Signup | |
8/28/19 | Word2Vec and Glove | P1 and P2 | |
9/2/19 | No Class (Labor Day) | ||
9/4/19 | ELMo and BERT | P3 and P4 | |
9/9/19 | Deep Text Classification | HW1 Out | |
9/11/19 | Embedding-based Classifiers | P5 and P6 | |
9/16/19 | CNN and RNN | P7 and P8 | |
9/18/19 | Attention Models | P9 and P10 | |
9/23/19 | Learn with limited supervision (Guest Lecture: Judy Hoffman) | ||
9/25/19 | Transfer Learning and Fine-Tuning | P11 and P12 | |
9/30/19 | Semi-Supervised Learning and Active Learning | P13 and P14 | |
10/2/19 | Multitask and Weakly Supervised Learning | P15 and P16 | |
10/7/19 | Sequence Labeling | HW1 Due | |
10/9/19 | CRF and Structured Perceptron | P17 and P18 | |
10/14/19 | No Class (Fall Recess) | ||
10/16/19 | Deep Models for Sequence Labeling | P19 and P20 | Proposal Due |
10/21/19 | Deep Text Generation | HW2 Out | |
10/23/19 | Sequence-to-sequence models | P21 and P22 | |
10/28/19 | VAE for Text Generation | P23 and P24 | |
10/30/19 | Knowledge representation and reasoning | ||
11/4/19 | Knowledge Graph Construction | P25 and P26 | |
11/6/19 | Knowledge Graph Reasoning | P27 and P28 | |
11/11/19 | Neural Symbolic Methods | P29 and P30 | |
11/13/19 | QA and dialogue (No Class, Instructor Travel) | ||
11/18/19 | QA with Knowledge Bases | P31 and P32 | |
11/20/19 | Dialogue Systems | P33 and P34 | HW2 Due |
11/25/19 | Project Presentations | ||
11/27/19 | No Class | ||
12/2/19 | Project Presentations | ||
12/4/19 | Reading Day | Final Report Due |
Grading
- Homework (20%)
- There will be two assignments. Each one is designed for testing your understanding of the taught algorithms.
- All assignments follow the "no-late" policy. Assignments received after the due time will receive zero credit.
- All students are expected to follow the Georgia Tech Academic Honor Code.
- Project (45%)
- You are expected to complete a project on machine learning for text data. Your project needs to be clear about 1) the problem you are attempting to solve; 2) a survey of existing literature for the problem and the technical method you propose to solve the problem; 3) the results and conclusion you attain.
- Each project needs to be completed in a team of 2-3 people. Team members need to clearly claim their contributions in the project report.
- You will need to do the following:
- Project proposal (15%): formulation of the problem, literature survey, technical roadmap, preliminary results, expected outcome
- Presentation (15%): group-wise in-class project presentation
- Final report (15%): a complete and final project report
- The presentation schedule is available here.
- Paper Presentation (30%)
- Each student is expected to read and present 1-2 research paper(s). The presentation will be graded by the instructor and the TA.
- You need to email your slides to the instructor and TA by 9pm EST the night before your presentation.
- The grading of your presentation will be based on quality of slides (10%), presentation clearness (10%), critical and insightful comments (5%), and handling of questions (5%).
- The list of papers and the sign-up sheet are available here.
- Useful tips for presentation:
- Presentation Tips by Jeff Radel
- Oral Presentation Advice by Mark D. Hill
- Class participation (5%)
- Participation in class (including attendance, asking relevant questions in class, volunteering to answer questions on Piazza) will be considered when determining your final grade. It will be especially useful when you are right on the edge of two letter grades.
Resources
- Speech and Language Processing, by Dan Jurafsky and James H. Martin
- Deep Learning for NLP
- Deep Learning, by Ian Goodfellow, Yoshua Bengio, and Aaron Courville
- Dive into Deep Learning, by Aston Zhang, Zack C. Lipton, Mu Li, and Alex Smola
Other resources, such as deep learning toolboxes and datasets, will be provided throughout the course.