This repository is the official implementation of A Deep Generative Distance-Based Classifier for Out-of-Domain Detection with Mahalanobis Space (GOLING2020) by Hong Xu, Keqing He, Yuanmeng Yan, Sihong Liu, Zijun Liu, Weiran Xu.

A deep generative distance-based model with Mahalanobis distance to detect OOD samples.

The architecture of the proposed model:

We use anaconda to create python environment:

conda create --name python=3.6

Install all required libraries:

pip install -r requirements.txt

python3 experiment.py --dataset SNIPS --proportion 50 --mode train

   When only training model, there is no need to provide setting parameters.

• LOF

python3 experiment.py --dataset <dataset> --proportion <proportion> --mode test --setting lof --model_dir <model_dir>

• GDA

python3 experiment.py --dataset <dataset> --proportion <proportion> --mode test --setting gda_lsqr_800 --model_dir <model_dir>

• MSP

python3 experiment.py --dataset <dataset> --proportion <proportion> --mode test --setting msp --model_dir <model_dir>

  When only predicting the model(no training), the parameter model_dir is required to represent the folder where the model resides (which contains model.h5 model files).

python3 experiment.py --dataset <dataset> --proportion <proportion> --mode both --setting <setting>

   Setting parameter is required to specify using which algorithm to predict, but model_dir parameter is not required.

python3 experiment.py --dataset <dataset> --proportion <proportion> --mode both --setting <setting> --seen_classes SearchCreativeWork RateBook

python3 experiment.py --dataset SNIPS --proportion 50 --mode test --setting msp_0.5 msp_0.6 msp_0.7 msp_0.8 msp_0.9 --model_dir ./outputs/SNIPS-50-06112350 --seen_classes AddToPlaylist BookRestaurant PlayMusic RateBook

The parameters that must be specified:

• dataset, required, The dataset to use, ATIS or SNIPS or CLINC.
• proportion, required, The proportion of seen classes, range from 0 to 100.
• seen_classes, optional, The random seed to randomly choose seen classes.(e.g.--seen_classes SearchCreativeWork RateBook)
• mode, optional,Specify running mode, onlytrain,onlytest or both
• setting, required, The settings to detect ood samples, e.g.
  • lof：using LOF for predicting.
  • gda_lsqr_800：using GDA for predicting, using lsqr for solver, and the threshold is 800 (Mahalanobis distance).
  • msp: using MSP for predicting.
• model_dir, The directory contains model file (.h5), requried when test only.

The parameters that have default values (In general, it can stay fixed):

• gpu_device, default=1
• output_dir, default="./outputs"
• embedding_file,default="glove.6B.300d.txt"
• embedding_dim, default=300
• max_seq_len, default=None
• max_num_words, default=10000
• max_epoches, default=200
• patience, default=20
• batch_size, default=256

1. Macro f1-score of unknown intents with different proportions (25%, 50% and 75%) of classes are treated as known intents on SNIPS and ATIS datasets.

2. Comparison between our unsupervised OOD detection method and supervised N+1 classification.

@inproceedings{xu2020deep,
  title={A Deep Generative Distance-Based Classifier for Out-of-Domain Detection with Mahalanobis Space},
  author={Xu, Hong and He, Keqing and Yan, Yuanmeng and Liu, Sihong and Liu, Zijun and Xu, Weiran},
  booktitle={Proceedings of the 28th International Conference on Computational Linguistics},
  pages={1452--1460},
  year={2020}
}