This is the official codebase accompanying our EMNLP 2022 paper "SLING: Sino-Linguistic Evaluation of Large Language Models". You can find our paper on arxiv.
See SLING_Data and the readme file in it.
A complete list of all phenomea and paradigms can be found in PhenomenonParadigmList.txt
See SLING_Code and the readme file in it.
python -m virtualenv sling-venv
source sling-venv/bin/activate
pip install torch torchvision # currently, this is the version compatible with CUDA 10.1
pip install transformers
pip install nltk
python SLING_Code/lm_sling.py
python SLING_Code/PanGu_sling.py
python SLING_Code/gpt3_sling.py
python SLING_Code/byt5_sling.py
lstm_sling.py is included in SLING_Code for reference. Details of how to run the CLUECorpusSmall LSTM language model can be found here.
If you use SLING, please cite it as follows:
@inproceedings{sling22,
author={Yixiao Song and Kalpesh Krishna and Rajesh Bhatt and Mohit Iyyer},
booktitle = {Empirical Methods in Natural Language Processing},
Year = "2022",
Title={SLING: Sino Linguistic Evaluation of Large Language Models},
}
If you use the reflexive_auto and reflexive_natural data in the Anaphor folder, please also cite:
@inproceedings{yangcoling2025,
author={Xiulin Yang},
booktitle = {International Conference on Computational Linguistics (COLING2025)},
Year = "2025",
Title={Language Models at the Syntax-Semantics Interface: A Case Study of the Long-Distance Binding of Chinese Reflexive ziji},
}