This directory contains source code accompanying the paper Controlling Conditional Language Models without Catastrophic Forgetting (ICML 2022).

We assume Python 3.7+. To install the dependencies, run the following command:

pip install -r requirements.txt

Then, you can reproduce experiments from the paper by running:

python run.py --config configs/code_pep8.json

substituting code_pep8.json with the appropriate configuration file.

For monitoring the experiment using Weights and Biases, set WANDB_PROJECT_NAME and WANDB_API_KEY.

In configs, we provide configuration files for the experiments described in the paper:

• translation_numbers_en_fr_debug.json corresponds to the terminology-consistent translation task (section 3.2 in the paper),
• summarization.json corresponds to the factually-consistent summarization task (section 3.3 in the paper),
• code_pep8.json corresponds to the PEP8-compliant code generation task (section 3.4 in the paper),
• code_compilability.json corresponds to the compilable code generation task (section 3.4 in the paper).

In contexts, we provide contexts (e.g. source documents for summarization) we used in experiments.

Tomasz Korbak, tomasz.korbak@gmail.com

Hady Elsahar, hady.elsahar@naverlabs.com

Germán Kruszewski, german.kruszewski@naverlabs.com

Marc Dymetman, marc.dymetman@naverlabs.com

@InProceedings{pmlr-v162-korbak22a,
  title = 	 {Controlling Conditional Language Models without Catastrophic Forgetting},
  author =       {Korbak, Tomasz and Elsahar, Hady and Kruszewski, German and Dymetman, Marc},
  booktitle = 	 {Proceedings of the 39th International Conference on Machine Learning},
  pages = 	 {11499--11528},
  year = 	 {2022},
  editor = 	 {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan},
  volume = 	 {162},
  series = 	 {Proceedings of Machine Learning Research},
  month = 	 {17--23 Jul},
  publisher =    {PMLR},
  pdf = 	 {https://proceedings.mlr.press/v162/korbak22a/korbak22a.pdf},
  url = 	 {https://proceedings.mlr.press/v162/korbak22a.html},
}