The branch is WIP: existing unit tests succeed, but:

• at least one testsuite case ("bol") fails because the jflex generation output differs
• new unit and/or testsuite test(s) should be added
• a note about this change should be added to the changelog and the manual

I adjusted the dot-newline testsuite case's spec to directly handle surrogate chars; I think that is sufficient testing. I documented unpaired surrogates' exclusion from . matches. So the outlined work is done.

But I wonder if excluding unpaired surrogates from . matches goes far enough? I think we should consider extending this idea to also not match unpaired surrogates in negated character classes ([^...]) and negated Unicode properties (\P{...} - note the capital P); currently in both cases unpaired surrogates will match. @lsf37 what do you think?

Leaving out those unpaired surrogates from negated classes does make sense. My main concern is [^], which should match any single character, and would then also not match those, I guess. In a way, that is correct, because they are not valid characters.

On the other hand, we would then get that [^]* no longer matches any input, which is something we'd want, I think. Similarly, we'd expect the expression [a]|[^a] to match any single character for any a. Maybe we shouldn't expect that, but currently I would.

The basic problem is that there exist inputs that contain sequences that do not resolve to characters, and one of the basic assumptions for the scanner from the dark ages of Java 1.1 is that every input is a sequence of characters.

Would it make sense to make [^] a special case, or is that even worse?

(rebased on master)