# CSE8803 DLT: Deep Learning for Text Data (2021 Fall)

## Table of Contents

## Logistics

**Lecture time**: Mon & Wed 3:30PM–4:45PM**Location**: Instructional Center 205**Instructor**: Chao Zhang**Teaching Assistant**: Yuchen Zhuang (yczhuang@gatech.edu); Rongzhi Zhang (rongzhi.zhang@gatech.edu)**Piazza**: https://piazza.com/gatech/fall2021/cse8803dlt**Office Hours**:- Instructor Office Hour: Wed 2-3 PM, https://bluejeans.com/5341114422
- TA Office Hour: Mon 2:30-3:30 PM, https://bluejeans.com/252540071/8394

## Learning Objective

This course will introduce state-of-the-art machine learning techniques for mainstay problems in text data analysis, with particular emphasis on deep learning methods that have recently achieved enormous success. Students will learn about trending problems in this field, key methods for solving these problems, and their advantages and disadvantages. The students are also expected to read, present, and discuss research papers, as well as conduct a research oriented course project.

The learning objective is that by the end of this course, the students are able to formulate their text analysis problems at hand, choose appropriate statistical models for the problems, and even come up with innovative solutions for solving open research problems in this field. The course will be useful for students who want to solve practical problems involving text data, and also for those who want to do edge-cutting research in text mining, natural language processing, artificial intelligence, and text-rich interdisciplinary research.

Prerequisites for this course: the students should be familiar with machine learning and have taken a relevant course before (e.g., CX4240, CSE6740, CS4641); (2) the students should be comfortable with reading research papers and giving presentations; (3) the students should have solid programming skills—the course project can be programming demanding.

## Course Content

In the task space, we will cover a range of text data analysis tasks, including:

- Text representation learning
- Text classification (sentiment analysis, document classification)
- Sequence labeling (POS tagging, NER, event extraction)
- Text generation (text summarization, image captioning)
- Knowledge graph construction and reasoning
- Question answering and dialogue systems

In the methodology space, we will introduce state-of-the-art deep learning techniques for these problems, including:

- Deep representation learning (Word2Vec, Transformer, BERT)
- Sequence labeling models (CRF, LSTM-CRF)
- Deep transfer/multi-task learning
- Neural symbolic methods
- Deep generative models (GPT, VAE, and their variants)

## Schedule

Date | Topic | Presentation | Due |
---|---|---|---|

08/23/2021 | Course Overview | Piazza Signup; Paper Pickup | |

08/25/2021 | Machine Learning Review | Presentation Signup Open Aug 27 | |

08/30/2021 | Module 1: Text Representation Learning |
Presentation Signup Close | |

09/01/2021 | Word2Vec and Glove | P1 and P2 | HW1 Out |

09/06/2021 | No Class (Labor Day) | ||

09/08/2021 | ELMo and BERT | P3 and P4 | |

09/13/2021 | Project Guideline and Examples |
||

09/15/2021 | Module 2: Text Classification |
HW1 Due | |

09/20/2021 | Embedding-based Classifiers | P5 and P6 | |

09/22/2021 | CNN and RNN | P7 and P8 | |

09/27/2021 | Attention Models | P9 and P10 | |

09/29/2021 | Module 3: Learning with Limited Supervision |
HW2 Out | |

10/04/2021 | Transfer Learning and Fine-Tuning | P11 and P12 | |

10/06/2021 | Semi-Supervised and Active Learning | P13 and P14 | |

10/11/2021 | No class (Fall Break) | ||

10/13/2021 | Weakly Supervised Learning | P15 and P16 | HW2 Due |

10/18/2021 | Module 4: Sequence Labeling |
||

10/20/2021 | Neural Sequence Labeling | P17 and P18 | |

10/25/2021 | Low-Resource for Sequence Labeling | P19 and P20 | HW3 Out |

10/27/2021 | Module 5: Text Generation |
||

11/01/2021 | Sequence-to-sequence models | P21 and P22 | |

11/03/2021 | VAE and GPT for Text Generation | P23 and P24 | |

11/08/2021 | Multi-modal Text Generation | P25 and P26 | HW3 Due |

11/10/2021 | Module 6: Knowledge Extraction and Exploitation |
||

11/15/2021 | Knowledge Graph Construction | P27 and P28 | |

11/17/2021 | Knowledge Graph Reasoning | P29 and P30 | |

11/22/2021 | QA and Dialogue with Kowledge Bases | P31 and P32 | |

11/24/2021 | No class | ||

11/29/2021 | Project Presentations | ||

12/01/2021 | Project Presentations | ||

12/06/2021 | Project Presentations | ||

12/10/2021 | Project Report Due |

*Disclaimer: The instructor reserves the right to modify the planned schedule and grading policy as needed during the course.*

## Grading

### Homework (30%)

- There will be three assignments. Each one will test your understanding of the taught methods or the presented papers.
- Late policy: Assignments are due at 11:59PM of the due date. You will be allowed 2 total late days without penalty for the entire semester. Once those days are used, you will be penalized according to the following policy:
- Homework is worth full credit before the due time.
- It is worth 75% credit for the next 24 hours.
- It is worth 50% credit for the second next 24 hours.
- It is worth zero credit after that.

- Follow the Georgia Tech Academic Honor Code.

### Paper Presentation (20%)

- We have 32 papers to study, and you will need to pick 1-2 paper(s) from the list to present. The paper list and presentation sign-up sheet is available here (will be open for signup on Aug 27 at 3PM ET).
- Each presentation is 17 minutes, plus 3 minutes for Q&A. Each presentation can be done by up to three presenters.
- You need to post your slides on Canvas by 9pm EST the night before your presentation.
- The presentation will be graded by the instructor and TAs according to the following criteria: quality of slides, presentation clearness, and question addressing. Your presentation should cover at least the following aspects: 1) What is the problem and background? 2) What are the main challenges of the problem? 3) How does the proposed method work? 4) What are the experimental results and observations?
- If you miss the presentation, unfortunately you will receive zero credit.
- Useful tips for presentation:
- Presentation Tips by Jeff Radel
- Oral Presentation Advice by Mark D. Hill

### Paper Discussion (10%)

- After each paper presentation, we will have 10 minutes for free discussion. The discussion will be generally about the strengths/weakness of the paper, what you like/dislike, the practical applications of the techniques, and comparison/connections with other papers we have studied.
- If you participate in the discussion of one paper and share your thoughts, you will receive 1 discussion point. The total discussion point is 10, which means you need to participate in the discussions for 10 out of the 32 presentations to receive the full score. For each discussion you have participated, please use this form to check-in
*after*you have shared your opinion.

### Project (40%)

You need to complete a project on deep learning for text data. Your project needs to be clear about 1) the problem you are attempting to solve; 2) a survey of existing literature for the problem; 3) the technical method you propose in order to solve the problem; 4) the results and conclusion you attain.

- Each project needs to be completed in a team of 2-4 people. Team members need to clearly claim their contributions in the project report.
- You will need to do the following:
- Presentation (20%): group-wise project presentation
- Final report (20%): a complete and final project report

- The presentation schedule is available here (will be open for signup Nov 22 at 5pm)
- Here are some project guidelines and resources you may find useful.

## More Resources

- Speech and Language Processing, by Dan Jurafsky and James H. Martin
- Deep Learning for NLP
- Deep Learning, by Ian Goodfellow, Yoshua Bengio, and Aaron Courville
- Dive into Deep Learning, by Aston Zhang, Zack C. Lipton, Mu Li, and Alex Smola

Other resources, such as deep learning toolboxes and datasets, will be provided throughout the course.