Highlights

Our proposed model X-Class is able to assign documents to classes (e.g., sports, politics, and science) without any other supervision other than the class names themselves.

Motivation

We propose the task: Text Classification with Extremely Weak Supervision, which is to classify documents to classes, with the plain class name as the only guidance.
Our method X-Class, breaks up this task into three modules

• Class-oriented Document Representation
  • We estimate both the class representation (based on the given class names), and the document representation (guided by the class representations).
• Document-Class Alignment
  • We apply Gaussian Mixture Models to align the document representations into clusters. The GMM is initialized with a prior of every document assigned to its nearest class, and therefore, we know which cluster represents which class.
• Text Classifier Training
  • We further select the confident document-class pairs from the previous step, and train a supervised text classifier (e.g. BERT) above it. This pipeline also illustrates our method.

Please refer to our paper and github for more details. You can also find our presentation and poster for NAACL.

Highlights

We analyzed linguistic properties, model architecture and learning objectives that may contribute to the multilinguality of M-BERT.
Linguistic properties:

• Code switching text (or what we call word-piece overlap) is not the main cause of multilinguality.
• Word ordering is crucial, when words in sentences are randomly permuted, multilinguality is low, however, still significantly better than random.
• (Unigram) word frequency is not enough, as we resampled all words with the same frequency, and found almost random performance. Combining the second and the third property infers that there is language similarity other than ordering of words between two languages, and which unigram frequency does not capture. We hypothesize that it may be similarity of n-gram occurrences.

Architecture:

• Depth of the transformer is the most important.
• Number of attention heads effects the absolute performance on individual languages, but the gap between in-language supervision and cross-language zero-shot learning didn’t change much.
• Total number of parameters, like depth, effects multilinguality.

Learning Objectives:

• Next Sentence Prediction objective, when removed, leads to slight increase in performance.
• Even marking sentences in languages with language-ids, allowing BERT to know exactly which language its learning on, did not hurt performance
• Using word-pieces leads to strong improvements on both source and target language (likely to depend on tasks) and slight improvement cross-lingually comparing to word or character based models.

Motivation

Multilingual BERT (M-BERT) has shown surprising cross lingual abilities — even when it is trained without cross lingual objectives. In this work, we analyze what causes this multilinguality from three factors: linguistic properties of the languages, the architecture of the model, and the learning objectives.

Please refer to our paper and github (tentative, to be moved to CCG) for more details.

Highlights

• We correct the test set of CoNLL03 NER. This higher quality evaluation set can be used in further research. The dataset is avalaible here.
• We design a mistake-aware framework CrossWeigh that fits any NER model that supports weighted training.

Motivation

The label annotation mistakes by human annotators brings up two challenges to NER:

• mistakes in the test set can interfere the evaluation results and even lead to an inaccurate assessment of model performance.
• mistakes in the training set can hurt NER model training.

We address these two problems by:

• manually correcting the mistakes in the test set to form a cleaner benchmark.
• develop framework CrossWeigh for mistake-aware training.

CrossWeigh works with any NER algorithm that accepts weighted training instances. It is composed of two modules. 1) mistake estimation: where potential mistakes are identified in the training data through a cross-checking process and 2) mistake re-weighing: where weights of those mistakes are lowered during training the final NER model.

Please refer to our paper and github for more details.