Shared Tasks

The NLP community has a great tradition of “shared tasks”. Many of these are perfect for a term-project for this class, since they give you a great starting point for a problem definition, training and test data, a standard evaluation metric, and lots of published baselines. Here are some pointers to shared tasks that were featured at CoNLL, SemEval, WMT, and Kaggle.

You are welcome to choose a shared task topic for your term project.

CoNLL Shared Tasks

The Conference on Computational Natural Language Learning (CoNLL) hosts a shared task every year. Here are the past CoNLL shared tasks:

1. Multilingual Parsing from Raw Text to Universal Dependencies
2. Universal Morphological Reinflection
3. Multilingual Shallow Discourse Parsing
4. Shallow Discourse Parsing
5. Grammatical Error Correction English Proceedings
6. Modelling Multilingual Unrestricted Coreference in OntoNotes
7. Modelling Unrestricted Coreference in OntoNotes English
8. Hedge Detection English Proceedings
9. Syntactic and Semantic Dependencies in Multiple Languages
10. Joint Parsing of Syntactic and Semantic Dependencies
11. Dependency Parsing: Multilingual & Domain Adaptation
12. Multi-Lingual Dependency Parsing
13. Semantic Role Labeling English
14. Language-Independent Named Entity Recognition
15. Clause Identification
16. Chunking
17. NP Bracketing

SemEval

The International Workshop on Semantic Evaluation (SemEval) hosts a range of shared tasks every year. Here are links to the SemEval tasks:

SemEval-2017

1. Semantic Textual Similarity
2. Multi­lingual and Cross­-lingual Semantic Word Similarity
3. Community Question Answering
4. Sentiment Analysis in Twitter
5. Fine-Grained Sentiment Analysis on Financial Microblogs and News
6. #HashtagWars. Learning a Sense of Humor
7. Detection and Interpretation of English Puns
8. RumourEval. Determining rumour veracity and support for rumours
9. Abstract Meaning Representation Parsing and Generation
10. Extracting Keyphrases and Relations from Scientific Publications
11. End-User Development using Natural Language
12. Clinical TempEval

SemEval-2016

1. Semantic Textual Similarity. A Unified Framework for Semantic Processing and Evaluation
2. Interpretable Semantic Textual Similarity
3. Community Question Answering
4. Sentiment Analysis in Twitter
5. Aspect-Based Sentiment Analysis
6. Detecting Stance in Tweets
7. Determining Sentiment Intensity of English and Arabic Phrases
8. Meaning Representation Parsing
9. Chinese Semantic Dependency Parsing
10. Detecting Minimal Semantic Units and their Meanings
11. Complex Word Identification
12. Clinical TempEval
13. TExEval-2 – Taxonomy Extraction
14. Semantic Taxonomy Enrichment

SemEval-2015

1. Paraphrase and Semantic Similarity in Twitter
2. Semantic Textual Similarity
3. Answer Selection in Community Question Answering
4. TimeLine. Cross-Document Event Ordering
5. QA TempEval
6. Clinical TempEval
7. Diachronic Text Evaluation
8. SpaceEval
9. CLIPEval Implicit Polarity of Events
10. Sentiment Analysis in Twitter
11. Sentiment Analysis of Figurative Language in Twitter
12. Aspect Based Sentiment Analysis
13. Multilingual All-Words Sense Disambiguation and Entity Linking
14. Analysis of Clinical Text
15. A CPA dictionary-entry-building task
16. Taxonomy Extraction Evaluation
17. Semantic Dependency Parsing

SemEval-2014

1. Evaluation of Compositional Distributional Semantic Models on Full Sentences through Semantic Relatedness and Entailment
2. Grammar Induction for Spoken Dialogue Systems
3. Cross-Level Semantic Similarity
4. Aspect Based Sentiment Analysis
5. L2 Writing Assistant
6. Supervised Semantic Parsing of Spatial Robot Commands
7. Analysis of Clinical Text
8. Broad-Coverage Semantic Dependency Parsing
9. Sentiment Analysis in Twitter
10. Multilingual Semantic Textual Similarity

SemEval-2013

1. TempEval-3 Temporal Annotation
2. Sentiment Analysis in Twitter
3. Spatial Role Labeling
4. Free Paraphrases of Noun Compounds
5. Evaluating Phrasal Semantics
6. Semantic Textual Similarity (becomes *Sem Shared Task)
7. The Joint Student Response Analysis and 8th Recognizing Textual Entailment Challenge
8. Cross-lingual Textual Entailment for Content Synchronization
9. Extraction of Drug-Drug Interactions from BioMedical Texts
10. Cross-lingual Word Sense Disambiguation
11. Evaluating Word Sense Induction & Disambiguation within An End-User Application
12. Multilingual Word Sense Disambiguation
13. Word Sense Induction for Graded and Non-Graded Senses
14. The Coarse-Grained and Fine-Grained Chinese Lexical Sample and All-Words Task

SemEval-2012

1. English Lexical Simplification
2. Measuring Degrees of Relational Similarity
3. Spatial Role Labeling
4. Evaluating Chinese Word Similarity
5. Chinese Semantic Dependency Parsing
6. Semantic Textual Similarity
7. COPA. Choice Of Plausible Alternatives An evaluation of commonsense causal reasoning
8. Cross-lingual Textual Entailment for Content Synchronization

Previous years

Kaggle

Kaggle is a platform for machine learning competitions where people compete to produce the best models for a huge range of different datasets. Companies often offer a reward for their competitions. There’s tons of cool data and competitions that you can base your final project on.

Here are a few relevant competitions:

• Sentiment Analysis on Movie Reviews
• Toxic Comment Classification Challenge
• Convert English text from written expressions into spoken forms
• Quora Question Pairs - Can you identify question pairs that have the same intent?
• The Allen AI Science Challenge - Is your model smarter than an 8th grader?
• DonorsChoose.org Application Screening - Predict whether teachers’ project proposals are accepted
• TensorFlow Speech Recognition Challenge - Can you build an algorithm that understands simple speech commands?
• Spooky Author Identification - Identify horror authors from their writings

You can also check out the Linguistics tag and the Langauges tag for lots of other ideas. Want 130,000 wine reviews with their ratings, or 55,000 song lyrics? Find them on Kaggle.