Sometimes it is desirable to be able to say that a token is in a language different from the main language of the file, and to specify the foreign language. Some corpora have occasional code switching, others have a lot of it. And if the annotators decide to actually annotate the foreign segment following the foreign language rules, the validator needs to know that it should temporarily switch to a different list of auxiliaries and morphological features.
I have now modified the validator so that if it sees Lang=en in the MISC column, it will switch to English for the current token (it affects the auxiliary list, copula list, and feature-value-UPOS combinations). The value is the ISO 639 code as registered for the language in UD (either two-letter ISO 639-1, or three-letter ISO 639-3); it must be lowercased. I have also documented it here, here, and here. So if you need to specify the foreign language for the validator, you can now do so.
Nevertheless, I also found out that various treebanks already try to indicate the language in MISC, and given the lack of standard so far, various approaches are taken:
- French GSD and Naija NSC use the format I use for the validator (but in the case of Naija, maybe it is meant to indicate where the word has been taken from, rather than to say that it is not a Naija word).
- Komi Zyrian uses
Lang=Mixed, Lang=Rus (the values are not UD-registered ISO codes).
- Frisian-Dutch uses
lang=fy, lang=nl (the name of the attribute is not capitalized).
- Turkish-German uses
LangID=TR, LangID=DE, LangID=OTHER (different attribute name; uppercase language codes will not be recognized).
- Hindi-English uses just
hi or en, without saying that these are language codes.
It would be nice if we could harmonize these annotations in future releases. Although not every treebank needs the validator to recognize them (e.g., the massively code-switching treebanks are registered under special user-defined language codes, such as qtd for Turkish-German, so they have their own set of language-specific guidelines).
Sometimes it is desirable to be able to say that a token is in a language different from the main language of the file, and to specify the foreign language. Some corpora have occasional code switching, others have a lot of it. And if the annotators decide to actually annotate the foreign segment following the foreign language rules, the validator needs to know that it should temporarily switch to a different list of auxiliaries and morphological features.
I have now modified the validator so that if it sees
Lang=enin the MISC column, it will switch to English for the current token (it affects the auxiliary list, copula list, and feature-value-UPOS combinations). The value is the ISO 639 code as registered for the language in UD (either two-letter ISO 639-1, or three-letter ISO 639-3); it must be lowercased. I have also documented it here, here, and here. So if you need to specify the foreign language for the validator, you can now do so.Nevertheless, I also found out that various treebanks already try to indicate the language in MISC, and given the lack of standard so far, various approaches are taken:
Lang=Mixed,Lang=Rus(the values are not UD-registered ISO codes).lang=fy,lang=nl(the name of the attribute is not capitalized).LangID=TR,LangID=DE,LangID=OTHER(different attribute name; uppercase language codes will not be recognized).hioren, without saying that these are language codes.It would be nice if we could harmonize these annotations in future releases. Although not every treebank needs the validator to recognize them (e.g., the massively code-switching treebanks are registered under special user-defined language codes, such as
qtdfor Turkish-German, so they have their own set of language-specific guidelines).