UD guidelines currently do not specify how to mark document and paragraph boundaries and for many treebanks such information is not available (original text gone, sentences shuffled etc.) But where it is available, it can be potentially useful for applications, including but not limited to sentence segmentation.
I am going to acquire this sort of annotation from data providers who have it, and make it available in UD release 2.0 in a unified way. This issue is to propose the way it is encoded in the data, and see if there are comments/suggestions (hopefully quickly solvable ones—the data freeze deadline is coming soon).
It turns out that paragraphs are not always necessarily supersets of sentences. In some cases (bulleted list items), a new paragraph may start in the middle of a sentence. (That of course depends on how the two units are defined but I am not looking for any standardized definition on such a short notice. If you have paragraphs, you define what they are.)
As a result, paragraph boundary should be marked at token level. Document boundary will be marked at sentence level. My proposal:
- The first sentence of a new document contains a comment that says
# newdoc. (Nothing more. You can have a separate comment with a document id if you like, but I want to be able to recognize documents even if they don't have ids.) It is not necessary that the first sentence of a CoNLL-U file has the newdoc comment (e.g. if the document is split between dev and test data).
- The first token of a new paragraph contains the attribute
NewPar=Yes in the MISC column. Usually this will also be the first token of a sentence. If it is a multi-word token, the attribute will appear in the line of the multi-word token, not in the line of its first syntactic word.
Specifically seeking feedback from those who indicated they have doc or par info available: @lauma @liljao @jnivre @arademaker @Kira-D @kajad @TomazErjavec @LarsAhrenberg @natko5
and from those who may have to deal with the data :-) @foxik @martinpopel @fginter @spyysalo
(If you already have the info in the data in some form, e.g. inferrable from sentence ids, and prefer me to extract it and convert it to the unified annotation, let me know.)
UD guidelines currently do not specify how to mark document and paragraph boundaries and for many treebanks such information is not available (original text gone, sentences shuffled etc.) But where it is available, it can be potentially useful for applications, including but not limited to sentence segmentation.
I am going to acquire this sort of annotation from data providers who have it, and make it available in UD release 2.0 in a unified way. This issue is to propose the way it is encoded in the data, and see if there are comments/suggestions (hopefully quickly solvable ones—the data freeze deadline is coming soon).
It turns out that paragraphs are not always necessarily supersets of sentences. In some cases (bulleted list items), a new paragraph may start in the middle of a sentence. (That of course depends on how the two units are defined but I am not looking for any standardized definition on such a short notice. If you have paragraphs, you define what they are.)
As a result, paragraph boundary should be marked at token level. Document boundary will be marked at sentence level. My proposal:
# newdoc. (Nothing more. You can have a separate comment with a document id if you like, but I want to be able to recognize documents even if they don't have ids.) It is not necessary that the first sentence of a CoNLL-U file has thenewdoccomment (e.g. if the document is split between dev and test data).NewPar=Yesin the MISC column. Usually this will also be the first token of a sentence. If it is a multi-word token, the attribute will appear in the line of the multi-word token, not in the line of its first syntactic word.Specifically seeking feedback from those who indicated they have doc or par info available: @lauma @liljao @jnivre @arademaker @Kira-D @kajad @TomazErjavec @LarsAhrenberg @natko5
and from those who may have to deal with the data :-) @foxik @martinpopel @fginter @spyysalo
(If you already have the info in the data in some form, e.g. inferrable from sentence ids, and prefer me to extract it and convert it to the unified annotation, let me know.)