CSE 8803 DLT: Deep Learning for Text Data (2020 Fall)

Table of Contents

Logistics

Learning Objective

This course will introduce state-of-the-art machine learning techniques for mainstay problems in text data analysis, with particular emphasis on deep learning methods that have recently achieved enormous success. The students will learn about trending problems in this field, key methods for solving these problems, and their advantages and disadvantages. The students are also expected to read, review, and present research papers, as well as conduct a research oriented course project. The course will provide useful techniques for students who want to solve practical problems involving text data, and better prepare those who want to do edge-cutting research in text mining, natural language processing, artificial intelligence, and text-rich interdisciplinary research. The learning objective is that by the end of this course, the students are able to formulate their text analysis problems at hand, choose appropriate statistical models for the problems, and even come up with innovative solutions for solving open research problems in this field.

Prerequisites for this course: (1) the students should be have basic knowledge in machine learning and taken a relevant course before (e.g., CX4240, CSE6740, CS4641); (2) the students should be comfortable with reading research papers and giving presentations; (3) the students should have solid programming skills—the course project can be programming demanding.

Course Content

In the task space, we will cover a range of text data analysis tasks, including:

  • Text representation learning
  • Text classification (sentiment analysis, document classification)
  • Sequence labeling (POS tagging, NER, event extraction)
  • Text generation (text summarization, image captioning)
  • Knowledge graph construction and reasoning
  • Question answering and dialogue systems

In the methodology space, we will introduce state-of-the-art deep learning techniques for these problems, including:

  • Deep representation learning (Word2Vec, Transformer, BERT)
  • Sequence labeling models (CRF, LSTM-CRF)
  • Deep transfer/multi-task learning
  • Neural symbolic methods
  • Deep generative models (GPT, VAE, and their variants)

Specifically, we have the following modules:

  • Background knowledge
    • Course overview and logistics
    • ML/DL review
  • Text representation learning
    • Bag-of-words, n-gram, TF-IDF
    • Dimension reduction, matrix factorization, topic models
    • Word2vec, Glove
    • Transformer and BERT
  • Deep text classification
    • Traditional text classifiers: naive Bayes, logistic regression, SVM
    • Deep classifiers: CNN, RNN and their variants
    • Attention models
  • Learning with less labeled data
    • Transfer learning and fine-tuning
    • Multitask learning
    • Semi-supervised learning
    • Weakly supervised learning
  • Sequence labeling and structured prediction
    • Motivating tasks: POS tagging, NER, event extraction
    • Conditional random fields for sequence labeling
    • Deep latent variable models for sequence labeling
  • Text generation
    • Motivating tasks: summarization, image captioning, translation
    • Deep language models
    • GPT and VAE for text generation
  • Knowledge representation and reasoning
    • Knowledge graph construction
    • Knowledger graph reasoning
    • Combining logic and neural models
    • Question answering and Dialogue systems with knowledge bases
  • Project presentations

Schedule

This course is hybrid touch point mode. All the lectures and presentations will be delivered online at https://bluejeans.com/5341114422, and we have touch point meetings for in-person discussions. More information about the touch point meetings can be found in the FAQs.

Date Topic Presentation Due
8/17/20 Course Overview   Piazza Signup; Paper Pickup
8/19/20 Machine Learning Review    
  Module 1: Text Representation Learning   Presentation Signup Open Aug 21
8/24/20 Text Representation Learning   Presentation Signup Close
8/26/20 Word2Vec and Glove P1 and P2  
8/31/20 ELMo and BERT P3 and P4  
  Module 2: Text Classification    
9/2/20 Deep Text Classification    
9/7/20 No Class (Labor Day)    
9/9/20 Embedding-based Classifiers P5 and P6 Proposal Due
9/14/20 CNN and RNN P7 and P8 HW1 Out
9/16/20 Attention Models P9 and P10  
9/21/20 Touchpoint Meeting    
  Module 3: Learning with Limited Supervision    
9/23/20 Transfer Learning and Fine-Tuning P11 and P12  
9/28/20 Semi-Supervised and Active Learning P13 and P14  
9/30/20 Multitask and Weakly Supervised Learning P15 and P16  
  Module 4: Sequence Labeling    
10/5/20 Sequence Labeling   HW1 Due
10/7/20 Neural Sequence Labeling P17 and P18 HW2 Out
10/12/20 Low-Resource for Sequence Labeling P19 and P20  
10/14/20 Touchpoint Meeting    
  Module 5: Text Generation    
10/19/20 Deep Text Generation    
10/21/20 Sequence-to-sequence models P21 and P22  
10/26/20 VAE and GPT for Text Generation P23 and P24  
  Module 6: Knowledge Extraction and Exploitation    
10/28/20 Knowledge representation and reasoning    
11/2/20 Knowledge Graph Construction P25 and P26 HW2 Due
11/4/20 Knowledge Graph Reasoning P27 and P28  
11/9/20 QA and Dialogue with Kowledge Bases P29 and P30  
11/11/20 Touchpoint Meeting    
  Projects    
11/16/20 Project Presentations    
11/18/20 Project Presentations    
11/23/20 Project Presentations    
11/24/20     Final Report Due

Disclaimer: The instructor reserves the right to modify the planned schedule and grading policy as needed during the course.

Grading

Homework (20%)

  • There will be two assignments. Each one will test your understanding of the taught methods.
  • Late policy: Assignments are due at 11:59PM of the due date. You will be allowed 2 total late days without penalty for the entire semester. Once those days are used, you will be penalized according to the following policy:
    • Homework is worth full credit before the due time.
    • It is worth 75% credit for the next 24 hours.
    • It is worth 50% credit for the second next 24 hours.
    • It is worth zero credit after that.
  • Follow the Georgia Tech Academic Honor Code.

Paper Review (15%)

  • We have 30 papers to study (list of papers available here), you need to read all of them. Also, you need to write reviews for 15 of them. Each written review will earn you 1 point, if it is reasonable judged by the TAs.
  • How to submit paper reviews? Use this Google form to submit paper review 11:59PM before the presentation day.
  • In addition to the regular 15 points for paper reviewing, there are also 5 bonus points for five top reviewers (which will be selected by the TAs).

Paper Presentation (25%)

  • You need to present 1-2 research paper(s) in the paper list. The paper presentation sign-up sheet is available here (will be open for signup on Aug 21 at 3PM ET).
  • Each presentation is 20 minutes, plus 10 minutes for Q&A and discussion. Each presentation can be done by up to three presenters.
  • You need to post your slides on Canvas by 9pm EST the night before your presentation.
  • The presentation will be graded by the instructor according to the following criteria: quality of slides (5%), presentation clearness (10%), critical and insightful comments and question addressing (10%).
  • If you miss the presentation, you will receive zero credit.
  • Useful tips for presentation:

Project (40%)

  • You need to complete a project on deep learning for text data. Your project needs to be clear about 1) the problem you are attempting to solve; 2) a survey of existing literature for the problem and the technical method you propose to solve the problem; 3) the results and conclusion you attain.
  • Each project needs to be completed in a team of 3-4 people. Team members need to clearly claim their contributions in the project report.
  • You will need to do the following:
    • Project proposal (10%): formulation of the problem, literature survey, technical roadmap, preliminary results, expected outcome
    • Presentation (15%): group-wise project presentation
    • Final report (15%): a complete and final project report
  • The presentation schedule is available here (will be open for signup Oct 28 at 6:15PM)

FAQs

  • What are those "touch point" meetings for?
    • The purpose of touch point meetings is to offer opportunities for students who may need in-person discussions for projects or course content. In each touch point meeting, the instructor or a TA will be present in the classroom physically, to address questions on projects, papers, or other course content for participating students. No new content will be delivered in touch point meetings.
  • Are those touch point meetings required?
    • No, they are completely optional. They are only for students who really need in-person discussions. There will be no penalty even if you attend none of them.
  • If I choose to go to touch point meetings, what protections are available?
    • First, we will follow the social distancing rule and limit the maximum number of students for each touched point meeting. Please register here if you want to attend a touch point meeting: touchpoint meeting student signup form.
    • Second, any student who chooses to attend touch point meeting MUST wear a face mask, according to Georgia Tech's policy. See more information from Georgia Tech at this link and this link.
  • For remote lectures, will there be recordings?
    • Yes, for every remote lecture and paper presentation, we will record on Bluejeans and shared a link after class.

More Resources

Other resources, such as deep learning toolboxes and datasets, will be provided throughout the course.