CSE 8803 DLT: Deep Learning for Text Data

Table of Contents

Logistics

  • Lecture time: Monday & Wednesday 3:00PM–4:15PM
  • Location: CoC Room 52
  • Instructor: Chao Zhang
  • Teaching Assistant: Lingkai Kong (lkkong@gatech.edu)
  • Piazza: Piazza will be the place for course discussions and announcements. If you have questions, please ask it on Piazza because 1) other students may have the same question; 2) you will get help faster compared to sending emails.
  • Office Hours:
    • Instructor Office Hour: Monday 4:20-5:20 PM
    • TA Office Hour: Wednesday 4:20-5:20 PM

Overview

This course will introduce state-of-the-art machine learning techniques for mainstay problems in text data analysis, with particular emphasis on deep learning methods that have recently achieved enormous success. The students will learn about trending problems in this field, key methods for solving these problems, and their advantages and disadvantages. The students are also expected to read and present cutting edge research papers, as well as conduct a research oriented course project. The course will provide useful techniques for students who want to solve practical problems involving text data, and better prepare those who want to do edge-cutting research in text mining, information retrieval, natural language processing, and text-rich interdisciplinary research. The learning objective is that by the end of this course, the students are able to formulate their text analysis problems at hand, choose appropriate statistical models for the problems, and even come up with innovative solutions for solving open research problems in this field.

In the task space, we will cover a range of text data analysis tasks, including:

  • Text representation learning
  • Text classification (sentiment analysis, document classification)
  • Sequence labeling (POS tagging, NER, event extraction)
  • Knowledge graph construction and reasoning
  • Text generation (text summarization, image captioning)
  • Question answering and dialogue systems

In the methodology space, we will introduce state-of-the-art deep learning techniques for these problems, including:

  • Deep representation learning (Word2Vec, Transformer, BERT)
  • Sequence labeling models (CRF, LSTM-CRF)
  • Deep transfer/multi-task learning
  • Neural symbolic methods
  • Deep generative models (VAE, GAN, and their variants)

Prerequisites for this course: (1) the students should be have basic knowledge in machine learning and taken a relevant course before (e.g., CX4240, CSE6740, CS4641); (2) the students should be comfortable with reading research papers and giving presentations; (3) the students should have solid programming skills—the course project can be programming demanding.

Content

  • Background knowledge
    • Course overview and logistics
    • ML/DL review
  • Text representation learning
    • Bag-of-words, n-gram, TF-IDF
    • Dimension reduction, matrix factorization, topic models
    • Word2vec, Glove
    • Transformer and BERT
  • Deep text classification
    • Traditional text classifiers: naive Bayes, logistic regression, SVM
    • Deep classifiers: CNN, RNN and their variants
    • Attention models
  • Learning with less labeled data
    • Transfer learning and fine-tuning
    • Multitask learning
    • Semi-supervised learning
    • Weakly supervised learning
  • Sequence labeling and structured prediction
    • Motivating tasks: POS tagging, NER, event extraction
    • Conditional random fields for sequence labeling
    • Deep latent variable models for sequence labeling
  • Text generation
    • Motivating tasks: summarization, image captioning, translation
    • Deep language models
    • VAE for text generation
  • Knowledge representation and reasoning
    • Knowledge graph construction
    • Knowledger graph reasoning
    • Combining logic and neural models
  • QA and dialogue systems
    • Question answering with knowledge bases
    • Dialogue systems
  • Project presentations

Schedule

Date Topic Presentation Due
8/19/19 Course Overview   Piazza Signup
8/21/19 Machine Learning Review    
8/26/19 Text Representation Learning   Presentation Signup
8/28/19 Word2Vec and Glove P1 and P2  
9/2/19 No Class (Labor Day)    
9/4/19 ELMo and BERT P3 and P4  
9/9/19 Deep Text Classification   HW1 Out
9/11/19 Embedding-based Classifiers P5 and P6  
9/16/19 CNN and RNN P7 and P8  
9/18/19 Attention Models P9 and P10  
9/23/19 Learn with limited supervision (Guest Lecture: Judy Hoffman)    
9/25/19 Transfer Learning and Fine-Tuning P11 and P12  
9/30/19 Semi-Supervised Learning and Active Learning P13 and P14  
10/2/19 Multitask and Weakly Supervised Learning P15 and P16  
10/7/19 Sequence Labeling   HW1 Due
10/9/19 CRF and Structured Perceptron P17 and P18  
10/14/19 No Class (Fall Recess)    
10/16/19 Deep Models for Sequence Labeling P19 and P20 Proposal Due
10/21/19 Deep Text Generation   HW2 Out
10/23/19 Sequence-to-sequence models P21 and P22  
10/28/19 VAE for Text Generation P23 and P24  
10/30/19 Knowledge representation and reasoning    
11/4/19 Knowledge Graph Construction P25 and P26  
11/6/19 Knowledge Graph Reasoning P27 and P28  
11/11/19 Neural Symbolic Methods P29 and P30  
11/13/19 QA and dialogue    
11/18/19 QA with Knowledge Bases P31 and P32  
11/20/19 Dialogue Systems P33 and P34 HW2 Due
11/25/19 Project Presentations    
11/27/19 Project Presentations    
12/2/19 No Class   Final Report Due
12/4/19 Reading Day    

Grading

  • Homework (20%)
    • There will be two assignments. Each one is designed for testing your understanding of the taught algorithms.
    • All assignments follow the "no-late" policy. Assignments received after the due time will receive zero credit.
    • All students are expected to follow the Georgia Tech Academic Honor Code.
  • Project (45%)
    • You are expected to complete a project on machine learning for text data. Your project needs to be clear about 1) the problem you are attempting to solve; 2) a survey of existing literature for the problem and the technical method you propose to solve the problem; 3) the results and conclusion you attain.
    • Each project needs to be completed in a team of 2-3 people. Team members need to clearly claim their contributions in the project report.
    • You will need to do the following:
      • Project proposal (15%): formulation of the problem, literature survey, technical roadmap, preliminary results, expected outcome
      • Presentation (15%): group-wise in-class project presentation
      • Final report (15%): a complete and final project report
  • Paper Presentation (30%)
    • Each student is expected to read and present 1-2 research paper(s). The presentation will be graded by the instructor and the TA.
    • You need to email your slides to the instructor and TA by 9pm EST the night before your presentation.
    • The grading of your presentation will be based on quality of slides (10%), presentation clearness (10%), critical and insightful comments (5%), and handling of questions (5%).
    • The list of papers and the sign-up sheet are available here.
    • Useful tips for presentation:
  • Class participation (5%)
    • Participation in class (including attendance, asking relevant questions in class, volunteering to answer questions on Piazza) will be considered when determining your final grade. It will be especially useful when you are right on the edge of two letter grades.

Resources

Other resources, such as deep learning toolboxes and datasets, will be provided throughout the course.