ICU-22080 Add plain ASCII as an explicitly detected type by koppor · Pull Request #2127 · unicode-org/icu

koppor · 2022-07-04T20:02:27Z

Checklist

Required: Issue filed: https://unicode-org.atlassian.net/browse/ICU-22080
Required: The PR title must be prefixed with a JIRA Issue number.
Required: The PR description must include the link to the Jira Issue, for example by completing the URL in the first checklist item
Required: Each commit message must be prefixed with a JIRA Issue number.
Issue accepted (done by Technical Committee after discussion)
Tests included, if applicable
API docs and/or User Guide docs changed or added, if applicable

CLAassistant · 2022-07-04T20:02:33Z

All committers have signed the CLA.

jira-pull-request-webhook · 2022-07-04T21:55:07Z

Notice: the branch changed across the force-push!

icu4j/main/classes/core/src/com/ibm/icu/text/CharsetRecog_ASCII.java is different
icu4j/main/tests/charset/src/com/ibm/icu/dev/test/charset/TestCharSetRecognition.java is no longer changed in the branch
icu4j/main/tests/core/src/com/ibm/icu/dev/test/charsetdet/TestCharsetDetector.java is now changed in the branch

View Diff Across Force-Push

~ Your Friendly Jira-GitHub PR Checker Bot

jira-pull-request-webhook · 2022-07-04T23:22:00Z

Notice: the branch changed across the force-push!

icu4j/main/classes/core/src/com/ibm/icu/text/CharsetRecog_ASCII.java is different
icu4j/main/tests/core/src/com/ibm/icu/dev/test/charsetdet/CharsetDetectionTests.xml is now changed in the branch

View Diff Across Force-Push

~ Your Friendly Jira-GitHub PR Checker Bot

jira-pull-request-webhook · 2022-07-06T20:56:43Z

Notice: the branch changed across the force-push!

icu4j/main/classes/core/src/com/ibm/icu/text/CharsetDetector.java is different
icu4j/main/classes/core/src/com/ibm/icu/text/CharsetRecog_ASCII.java is different
icu4j/main/tests/core/src/com/ibm/icu/dev/test/charsetdet/CharsetDetectionTests.xml is different
icu4j/main/tests/core/src/com/ibm/icu/dev/test/charsetdet/TestCharsetDetector.java is different

View Diff Across Force-Push

~ Your Friendly Jira-GitHub PR Checker Bot

richgillam · 2022-07-14T16:03:07Z

I notice the implementation is only in Java. To put a change like this into ICU we'll also need it ported back to C/C++.

markusicu · 2022-09-09T23:44:51Z

I notice the implementation is only in Java. To put a change like this into ICU we'll also need it ported back to C/C++.

Hi @koppor I see that you gave Rich's comment a 👍 but I think @richgillam was really asking whether you would be willing to add the C/C++ port in this PR...

markusicu · 2022-09-10T00:02:10Z

+            return null;
+        } else {
+            // ASCII, because ALL bytes in the stream are <= 127.
+            // However, there could be some unicode (such as Hebrew) which also has this property.


Hm? It could be some unusual charset like UTF-7 or HZ, but those are "prohibited character encodings" in modern HTML and have generally fallen out of favor.

I also don't think that Hebrew is relevant here.

Seems like the confidence for ASCII should be high if all bytes are 00..7F.

markusicu · 2022-09-10T00:23:01Z

FYI @srl295 @aheninger

Also, should we return "US-ASCII" which is more specific than "ASCII"? Or is that too pedantic?

macchiati · 2022-09-10T01:27:54Z

I think ASCII is sufficiently unambiguous.

…

On Fri, Sep 9, 2022 at 5:23 PM Markus Scherer ***@***.***> wrote: FYI @srl295 <https://github.com/srl295> @aheninger <https://github.com/aheninger> Also, should we return "US-ASCII" which is more specific than "ASCII"? Or is that too pedantic? — Reply to this email directly, view it on GitHub <#2127 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACJLEMER6NH6YHFWFK6WXWTV5PIHFANCNFSM52UC2RXQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

srl295 · 2022-09-10T02:28:42Z

FYI @srl295 @aheninger

Also, should we return "US-ASCII" which is more specific than "ASCII"? Or is that too pedantic?

ASCII is the same. US-ASCII is probably slightly more pedantic. The canonical id is ansi xj 1967 something.

So.. I think ASCII is fine probably more recognizable these days.

koppor · 2022-09-24T07:15:50Z

I notice the implementation is only in Java. To put a change like this into ICU we'll also need it ported back to C/C++.
Hi @koppor I see that you gave Rich's comment a 👍 but I think @richgillam was really asking whether you would be willing to add the C/C++ port in this PR...

Oh, OK, I understood the "we" in a wrong way. I would suggest to get the Java code finished and then I'll look around for a C++ expert having the time to work on this.

koppor · 2022-09-24T07:41:36Z

I committed the suggestions using GitHub's features. I did some other tweaks. The tests in "TestCharsetDector" successfully run locally. Not sure why. I think, there will be some other classes failing, so I'll wait for the CI.

After "we" sorted everything out, I'll squash into one commit and add a Co-authored-by for @markusicu

koppor · 2022-11-27T21:02:50Z

Note that I needed to replace ISO-2022-JP in the context of com.ibm.icu.dev.test.charsetdet.TestCharsetDetector#TestBufferOverflow by ASCII as the byte sequence (line 283) does not contain any non-7-bit characters. Think, it is no harm, because the shift state "at the start" is "a bad one".

koppor · 2022-11-27T21:20:00Z

Java code finished. Now, I would try to port it to the C implementation.

koppor · 2026-06-07T13:18:47Z

It was hard to find time and knowledge concerning the CPP port. Fortunately, there is Claude Code now. I have used it to a) do the pot and b) merge the upstream/main.

~~I left the commits separate to ease reviewing. I can do a squash after everything is settled.~~

Update: Will create one commit - I think, for everyone too long ago.

jira-pull-request-webhook · 2026-06-08T08:57:44Z

Hooray! The files in the branch are the same across the force-push. 😃

~ Your Friendly Jira-GitHub PR Checker Bot

srl295 · 2026-06-08T15:16:08Z

+ **********************************************************************
+ *   Copyright (C) 2005-2022, International Business Machines
+ *   Corporation and others.  All Rights Reserved.
+ **********************************************************************
+ */


You can drop the IBM copyright if it's a new file— this was only for source being brought in before 2016.

Yeah, new file. I adapted the copyright accordingly.

Add a CharsetRecog_ASCII recognizer that reports "ASCII" when every byte is 7-bit (<= 127). It reuses CharsetRecog_8859_1 for language detection and confidence (95 when no language matches, otherwise the 8859-1 confidence + 1), and is registered first so pure 7-bit text prefers ASCII over ISO-8859-1. Implemented in both ICU4J (CharsetRecog_ASCII.java) and ICU4C (csrascii.h/.cpp), with the C/C++ files added to sources.txt, the Visual Studio projects and the depstest dependency list. Charset detection tests in both languages are updated to expect ASCII for pure 7-bit input. Co-authored-by: Carl Christian Snethlage <50491877+calixtus@users.noreply.github.com> Co-authored-by: Christoph <siedlerkiller@gmail.com> Co-authored-by: Markus Scherer <markus.icu@gmail.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

jira-pull-request-webhook · 2026-06-08T19:07:37Z

Hooray! The files in the branch are the same across the force-push. 😃

~ Your Friendly Jira-GitHub PR Checker Bot

koppor mentioned this pull request Jul 4, 2022

Fix charset detection with utf16 and others JabRef/jabref#8947

Merged

6 tasks

koppor force-pushed the addASciiTest branch from 8470c4b to d8159ef Compare July 4, 2022 21:55

koppor force-pushed the addASciiTest branch from d8159ef to b1d0fa4 Compare July 4, 2022 23:21

koppor marked this pull request as draft July 5, 2022 06:55

koppor force-pushed the addASciiTest branch from b1d0fa4 to c661193 Compare July 6, 2022 20:56

koppor marked this pull request as ready for review July 6, 2022 22:02

markusicu self-assigned this Jul 14, 2022

markusicu requested a review from richgillam July 14, 2022 16:02

markusicu requested a review from FrankYFTang July 14, 2022 16:03

markusicu reviewed Sep 10, 2022

View reviewed changes

markusicu added the waiting-on-author label Sep 14, 2022

Siedlerchr mentioned this pull request Dec 25, 2022

BOM now missing at beginning of bibliography file -- causes JabRef to not recognize existing library JabRef/jabref#9496

Open

2 tasks

koppor mentioned this pull request Jun 15, 2025

Gradle build updates JabRef/jabref#13319

Merged

2 tasks

koppor mentioned this pull request Jan 6, 2026

Fix empty entries array when exporting group chat to JSON JabRef/jabref#14814

Merged

4 tasks

koppor mentioned this pull request Jan 29, 2026

Manual jlink module override (--jlink-modules) avelino/jbundle#6

Closed

koppor force-pushed the addASciiTest branch from d9080fa to 5195d72 Compare June 8, 2026 08:57

srl295 previously approved these changes Jun 8, 2026

View reviewed changes

koppor dismissed srl295’s stale review via b341d58 June 8, 2026 19:06

koppor force-pushed the addASciiTest branch from b341d58 to c75cace Compare June 8, 2026 19:07

Uh oh!

Conversation

koppor commented Jul 4, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

CLAassistant commented Jul 4, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jira-pull-request-webhook Bot commented Jul 4, 2022

Uh oh!

jira-pull-request-webhook Bot commented Jul 4, 2022

Uh oh!

jira-pull-request-webhook Bot commented Jul 6, 2022

Uh oh!

richgillam commented Jul 14, 2022

Uh oh!

markusicu commented Sep 9, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

markusicu Sep 10, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

markusicu commented Sep 10, 2022

Uh oh!

macchiati commented Sep 10, 2022 via email

Uh oh!

srl295 commented Sep 10, 2022

Uh oh!

koppor commented Sep 24, 2022

Uh oh!

koppor commented Sep 24, 2022

Uh oh!

koppor commented Nov 27, 2022

Uh oh!

koppor commented Nov 27, 2022

Uh oh!

koppor commented Jun 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jira-pull-request-webhook Bot commented Jun 8, 2026

Uh oh!

srl295 Jun 8, 2026

Choose a reason for hiding this comment

Uh oh!

koppor Jun 8, 2026

Choose a reason for hiding this comment

Uh oh!

jira-pull-request-webhook Bot commented Jun 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

koppor commented Jul 4, 2022 •

edited

Loading

CLAassistant commented Jul 4, 2022 •

edited

Loading

koppor commented Jun 7, 2026 •

edited

Loading