# CSE 8803 DLT: Deep Learning for Text Data

## Logistics

**Lecture time**: Monday & Wednesday 3:00PM–4:15PM**Location**: CoC Room 52**Instructor**: Chao Zhang**Teaching Assistant**: TBD**Piazza**: TBD- Piazza will be the main place for course discussions and announcements. If you have questions, please
**ask it on Piazza first**because 1) other students may have the same question; 2) you will get help faster compared to sending emails.

- Piazza will be the main place for course discussions and announcements. If you have questions, please
**Office Hours**:- Instructor Office Hour: TBD
- TA Office Hour: TBD

## Overview

This course will introduce state-of-the-art machine learning techniques for mainstay problems in text data analysis, with particular emphasis on deep learning methods that have achieved enormous success recently. Students will learn about trending problems in this field, key methods for solving these problems, and their advantages and disadvantages. The course will provide useful techniques for students who want to solve practical problems involving text data, and better prepare those who want to do edge-cutting research in text mining, information retrieval, natural language processing, and text-rich interdisciplinary research. The learning objective is that by the end of this course, the students are able to formulate their text analysis problems at hand, choose appropriate statistical models for the problems, and even come up with innovative solutions for solving open research problems in this field.

In the task space, we will cover a range of text data analysis tasks, including:

- Text classification (sentiment analysis, document classification)
- Sequence labeling (POS tagging, NER, event extraction)
- Knowledge graph construction and reasoning
- Text generation (text summarization, image captioning)
- A number of other trending topics in this area.

In the methodology space, we will introduce state-of-the-art deep learning techniques for these problems, including:

- Deep representation learning (Word2Vec, Transformer, BERT)
- Sequence labeling models (CRF, LSTM-CRF)
- Deep transfer/multi-task learning
- Neural symbolic methods
- Deep generative models (GAN, VAE, and their variants)

Prerequisites for this course: (1) the students should be have basic knowledge in machine learning and taken a relevant course before (e.g., CX4240, CSE6740, CS4641); (2) the students should have solid programming skills—both the assignments and the course project can be programming demanding.

## Content

- Background knowledge
- Course overview and logistics
- Text data analysis: tasks and applications
- ML/DL review

- Text representation learning
- Bag-of-words, n-gram, TF-IDF
- Dimension reduction, matrix factorization, topic models
- Word2vec, Glove
- Transformer and BERT

- Deep text classification
- Traditional text classifiers: naive Bayes, logistic regression, SVM
- Deep classifiers: CNN, RNN and their variants
- Attention models

- Learning with less labeled data
- Transfer learning and fine-tuning
- Multitask learning
- Semi-supervised learning
- Weakly supervised learning

- Sequence labeling and structured prediction
- Motivating tasks: POS tagging, NER, event extraction
- Conditional random fields for sequence labeling
- Deep latent variable models for sequence labeling

- Text generation
- Motivating tasks: summarization, image captioning, translation
- Deep language models
- VAE and GAN for text generation

- Knowledge representation and reasoning
- Knowledge graph construction
- Knowledger graph reasoning
- Combining logic and neural models

- Other tending topics in text data analysis
- Multimodal analytics of text and other data
- Adversarial learning and model robustness
- Question answering and dialogue systems

- Project presentations

## Schedule

Date | Topic | Assignment | Due | Readings |
---|---|---|---|---|

8/19/19 | Course Overview | Sign up for Piazza | ||

8/21/19 | Text Analysis Basics | |||

8/26/19 | Machine Learning Review | |||

8/28/19 | Bag-of-words, N-Grams, TF-IDF | |||

9/2/19 | No Class (Labor Day) | |||

9/4/19 | Dimension Reduction and Topic Models | Proposal Due | ||

9/9/19 | Word Embedding, Word2Vec, Glove | HW1 Out | ||

9/11/19 | Transformer and BERT | |||

9/16/19 | Traditional Text Classifiers | |||

9/18/19 | Deep Text Classification | |||

9/23/19 | Attention Models | HW1 Due | ||

9/25/19 | Transfer Learning and Fine-Tuning | HW2 Out | ||

9/30/19 | Multitask Learning | |||

10/2/19 | Semi-Supervised Learning | |||

10/7/19 | Weakly Supervised Learning | |||

10/9/19 | Sequence Labeling Tasks | HW2 Due | ||

10/14/19 | No Class (Fall Recess) | |||

10/16/19 | Conditional Random Fields | HW3 Out | Mid Report Due | |

10/21/19 | Deep Models for Sequence Labeling | |||

10/23/19 | Text Generation Tasks | |||

10/28/19 | Deep Language Models | |||

10/30/19 | GAN and VAE for Text Generation | HW3 Out | ||

11/4/19 | Knowledge Graph Construction | |||

11/6/19 | Knowledge Graph Reasoning | |||

11/11/19 | Neural Symbolic Methods | |||

11/13/19 | Other Trending Topics | HW3 Due | ||

11/18/19 | Project Presentations | |||

11/20/19 | Project Presentations | |||

11/25/19 | Project Presentations | |||

11/27/19 | No Class | |||

12/2/19 | No Class | Final Report Due | ||

12/4/19 | Reading Day |

## Grading

- Homeworks (45%)
- There will be three assignments. Each one is designed for testing your understanding of the taught algorithms. It could be either programming or written analysis.
- All assignments follow the "no-late" policy. Assignments received after the due time will receive zero credit.
- All students are expected to follow the Georgia Tech Academic Honor Code.

- Project (50%)
- You are expected to complete a project on machine learning for text data. Your project needs to be clear about 1) the problem you are attempting to solve; 2) a survey of existing literature for the problem and the technical method you propose to solve the problem; 3) the results and conclusion you attain.
- Each project needs to be completed in a team of 2-3 people. Team members need to clearly claim their contributions in the project report.
- You will need to submit the following:
- Project proposal (10%): formulation of the problem and expected outcome
- Mid-term report (10%): literature survey and technical roadmap
- Presentation (15%): group-wise in-class project presentation
- Final report (15%): a complete and final project report

- Class participation (5%)
- Your class participation score will be graded based on attendance and in-class quizzes.
- Participation in class discussions (including asking relevant questions in class, volunteering to answer questions on Piazza) will be considered when determining your final grade. It will be especially useful when you are right on the edge of two letter grades.

## Resources

- Speech and Language Processing, by Dan Jurafsky and James H. Martin
- Deep Learning for NLP
- Deep Learning, by Ian Goodfellow, Yoshua Bengio, and Aaron Courville
- Dive into Deep Learning, by Aston Zhang, Zack C. Lipton, Mu Li, and Alex Smola

Other resources, such as deep learning toolboxes and datasets, will be provided throughout the course.