CoNLL-U format currently does not specify how to represent all space characters of the original plain text. It only specifies the SpaceAfter=No feature denoting that the current token is not followed by a space in the original text.
However, the ability to represent all space characters would be useful both for:
- UD treebanks, where the exact (or closer) correspondence with the original corpus could be achieved,
- NLP pipelines using CoNLL-U format, so that the original plain text can be reconstructed at any time
Therefore I propose to add the following two features to the MISC column:
SpacesAfter=textSpacesBefore=text (required because there can be spaces before the first token)
The new features have very similar name to existing SpaceAfter, but as they interact with each other, I think it is fine.
The new features can be present in any token (but not word, similarly to SpaceAfter) and have the following semantics:
- if a token has
SpaceAfter=No feature, it cannot have SpacesAfter feature (it is ignored if it is present)
- if a token does not have
SpaceAfter=No and has SpacesAfter=spaces, it was followed by spaces in the original text
- if a token does not have
SpaceAfter=No nor SpacesAfter, it was followed by one space in the original text
- if
SpacesBefore=spaces is present, the token was preceded by spaces in the original text
- the
SpacesBefore can be used only on the first token of every sentence
The content of the SpacesBefore/SpacesAfter features has to be escaped in such a way that it never contains SPACE, TAB, CR, LF, PIPE characters, and it should encode SPACEs efficiently. Many possibilites come to mind:
- JSON-like escaping: the space and pipe characters would have to be
\u0020 and \u007c
- HTML-like escaping using numbered entities like
 
- custom C-like escaping allowing only the following escapes:
\s (space), \t, \r, \n, \p (pipe), \\
Personally I vote for the last possibility -- C-like escaping with \s and \p. It is true that some existing escaping system would help with decoding, but 1) the existing systems seem to handle space quite ineffectively, 2) an existing system does not help with encoding, as it most likely would not escape a space 3) decode the proposed system is a trivial sequence of replaces 4) the proposed system has unique representation of the original text (which is not true for many escaping systems, but I think it is a useful feature in this case).
If a consensus is reached, I will document the new features on the http://universaldependencies.org/format.html site.
CoNLL-U format currently does not specify how to represent all space characters of the original plain text. It only specifies the
SpaceAfter=Nofeature denoting that the current token is not followed by a space in the original text.However, the ability to represent all space characters would be useful both for:
Therefore I propose to add the following two features to the MISC column:
SpacesAfter=textSpacesBefore=text(required because there can be spaces before the first token)The new features have very similar name to existing
SpaceAfter, but as they interact with each other, I think it is fine.The new features can be present in any token (but not word, similarly to
SpaceAfter) and have the following semantics:SpaceAfter=Nofeature, it cannot haveSpacesAfterfeature (it is ignored if it is present)SpaceAfter=Noand hasSpacesAfter=spaces, it was followed byspacesin the original textSpaceAfter=NonorSpacesAfter, it was followed by one space in the original textSpacesBefore=spacesis present, the token was preceded byspacesin the original textSpacesBeforecan be used only on the first token of every sentenceThe content of the
SpacesBefore/SpacesAfterfeatures has to be escaped in such a way that it never contains SPACE, TAB, CR, LF, PIPE characters, and it should encode SPACEs efficiently. Many possibilites come to mind:\u0020and\u007c \s(space),\t,\r,\n,\p(pipe),\\Personally I vote for the last possibility -- C-like escaping with
\sand\p. It is true that some existing escaping system would help with decoding, but 1) the existing systems seem to handle space quite ineffectively, 2) an existing system does not help with encoding, as it most likely would not escape a space 3) decode the proposed system is a trivial sequence of replaces 4) the proposed system has unique representation of the original text (which is not true for many escaping systems, but I think it is a useful feature in this case).If a consensus is reached, I will document the new features on the http://universaldependencies.org/format.html site.