Integrate regular and "bidi" German tables by egli · Pull Request #1100 · liblouis/liblouis

egli · 2021-08-20T15:30:15Z

Related to #911

These tests are really weird and will probably never occur in real life text

This table can be used to lowercase all chars before liblouis performs any translation.

This is basically the first step to merge them. I just copied the bidi table ontop of the g0-core table. Thanks to the latinLowercase table all the tests involving caps should still work. I made one small change, namely enable the downgrade emphclass which wasn't in the bidi table.

egli · 2022-01-26T14:49:10Z

So I replaced the g0-core table with the bidi table. Most tests pass with some exceptions:

The tests involving '§'. @BueVest has removed these rules but there are tests for them. Have to check with @mixa72
some tests involving mixed numbers and letters no longer work. I suspect that somehow the letters after the number are not contracted any more. Don't understand that one and have to check with @mixa72
there is a problem with 'ck' in g2. This might be due to some weird pass2 rules (https://github.com/liblouis/liblouis/blob/bidi-merge/tables/de-g0-core.uti#L762)

And the comment says that the line can be removed once #717 has been fixed which it has.

egli · 2022-01-26T16:39:35Z

OK, the problem with ck in g2 is gone. There are still some weird pass2 rules though.

bertfrees · 2022-01-26T19:39:45Z

@egli Already did a round of reviewing. I know it's still a draft, but still, my remarks could be useful.

@mixa72

I checked with @mixa72

@12345678

This essentially merges the g0 bidi core with the g0 core. To achieve that I had to do the following: - expand the detailed accents table. The include (and overwrite) mechanism didn't work as desired. Presumably the base opcode cannot handle overwrites. - Reduce the many context rules that handle single capital letter to just a few. Also I changed the sign that I insert as I had problems with the ck contraction if I use the 45-65 combination. So I use @12345678 instead

egli · 2022-01-28T13:24:00Z

So I managed to get rid of de-g0-bidi-core.uti, i.e. replace it with de-g0-bidi.utb.

There are still some problems remaining:

The translation of stuff like 'kW kVA hPa' now fails. I don't really understand how that worked before
The context+pass2 trick for single capital letters doesn't work for Æ, as we define the lowercase æ only for forward translation, as that we get problems with backwards translating ae otherwise.
There is a problem with words like '8fach' that wasn't there before

FAIL: braille-specs/de-g0-bidi-specs
====================================

Tables have not been indexed yet. Indexing LOUIS_TABLEPATH.
Best match: /home/egli/src/liblouis/tables/de-g0-bidi.utb (18)
Input:    'kW kVA hPa'
Expected: '⠅⠘⠺ ⠅⠘⠧⠁ ⠓⠨⠏⠁' (length 13)
Received: '⠅⣿⠨⠺ ⠅⠘⠧⠁ ⠓⠨⠏⠁' (length 14)
Diff:     Expected '⠘' but received '⣿' in index 1
lou_checkyaml:./braille-specs/de-g0-bidi-specs.yaml:216: Failure
Table: /home/egli/src/liblouis/tables/de-g0-bidi.utb
Display table: unicode-without-blank.dis

Input:    'Æ'
Expected: '⠨⠁⠑' (length 3)
Received: '⠘⠁⠑' (length 3)
Diff:     Expected '⠨' but received '⠘' in index 0

FAIL: braille-specs/de-g1
=========================

Tables have not been indexed yet. Indexing LOUIS_TABLEPATH.
Best match: /home/egli/src/liblouis/tables/de-g1.ctb (19)
Input:    '8fach'
Expected: '⠼⠓⠠⠋⠁⠹' (length 6)
Received: '⠼⠓⠠⠋⠁⠉⠓' (length 7)
Diff:     Expected '⠹' but received '⠉' in index 5
lou_checkyaml:./braille-specs/de-g1.yaml:451: Failure
Table: /home/egli/src/liblouis/tables/de-g1.ctb
Display table: unicode-without-blank.dis

This is probably a workaround around a bug in the base opcode where base doesn't seem to trigger when the lowercase char is preceded with a noback

egli · 2022-02-09T14:39:51Z

I managed to fix the forward translation of Æ with a context rule. I suspect that this is just for a workaround around a bug in the base opcode where the lowercase char is defined with a noback prefix
I also managed to fix the 'kW' problem with some more pass3 trickery
all that's left now are problems with missing contraction after a number to character change

Tables have not been indexed yet. Indexing LOUIS_TABLEPATH.
Best match: /home/egli/src/liblouis/tables/de-g1.ctb (19)
Input:    '8fach'
Expected: '⠼⠓⠠⠋⠁⠹' (length 6)
Received: '⠼⠓⠠⠋⠁⠉⠓' (length 7)
Diff:     Expected '⠹' but received '⠉' in index 5
lou_checkyaml:./braille-specs/de-g1.yaml:451: Failure
Table: /home/egli/src/liblouis/tables/de-g1.ctb
Display table: unicode-without-blank.dis

egli · 2022-02-10T16:49:39Z

Hi @BueVest why does Æ need to be indicated with 46 whereas all the other single capital letters are indicated with 45? Is that test case really correct? See https://github.com/liblouis/liblouis/blob/bidi-merge/tests/braille-specs/de-g0-bidi-specs.yaml#L258

and also move to the pass2 rules to the associated context rules

This is a cheap hack I know. It's for explorative purposes only. What baffles me is that some stuff is contracted but there is no contraction of 'st' and oddly enough in g2 it contracts the 'te' of 'stel' instead of the 'st' and the 'el'

bertfrees · 2022-03-07T10:46:17Z

@@ -1,6 +1,7 @@
-# liblouis: German grade 0 braille
+# liblouis: German grade 0 braille (bidirectional)


"bidirectional" -> "with indication of capitals"

Would "detailed" be OK? Because it is with capitals and with detailed accents

See my other comment.

[ and ] are translated with a leading dot 6. So after a number we need a double dot 6 unless it will be read as a plain )

bertfrees · 2022-03-07T14:46:10Z

+#-index-name: German, uncontracted, detailed
+#-display-name: German uncontracted braille (detailed)


"Detailed" is maybe a bit cryptic for a display name. I'd have called it something like "German uncontracted braille with indication of capitals" (similar to ru-litbrl-detailed.utb). More precise would be "German uncontracted braille with indication of capitals and detailed representation of accented letters", but that may be a bit too long.

But "detailed" might be fine too, dunno.

egli · 2022-03-07T14:48:48Z

@bertfrees any idea why generateDisplayName fails

bertfrees · 2022-03-07T14:54:09Z

@bertfrees any idea why generateDisplayName fails

Just run make in that directory and check generate.log. We should let Github upload the log file if that test fails.

egli · 2022-03-07T14:56:46Z

make
go get golang.org/x/text/language/display
go: downloading golang.org/x/text v0.3.7
go build -buildmode=c-archive displayLanguage.go
displayLanguage.go:5:2: no required module provides package golang.org/x/text/language: go.mod file not found in current directory or any parent directory; see 'go help modules'
displayLanguage.go:6:2: no required module provides package golang.org/x/text/language/display: go.mod file not found in current directory or any parent directory; see 'go help modules'
make: *** [Makefile:19: displayLanguage.h] Error 1

bertfrees · 2022-03-07T15:10:08Z

../../tables/de-g1-detailed.ctb: German, partially contracted, back-translatable != German, partially contracted, detailed
   cat ../../tables/de-g1-detailed.ctb | sed 's/^\(#-index-name: *\).*$/\1German, partially contracted, detailed/g' > ../../tables/de-g1-detailed.ctb.tmp
   mv ../../tables/de-g1-detailed.ctb.tmp ../../tables/de-g1-detailed.ctb
../../tables/de-g1-detailed.ctb: German partially contracted braille (back-translatable) != German partially contracted braille (detailed)
   cat ../../tables/de-g1-detailed.ctb | sed 's/^\(#-display-name: *\).*$/\1German partially contracted braille (detailed)/g' > ../../tables/de-g1-detailed.ctb.tmp
   mv ../../tables/de-g1-detailed.ctb.tmp ../../tables/de-g1-detailed.ctb

egli marked this pull request as draft August 20, 2021 15:30

egli added this to the 3.20 milestone Nov 24, 2021

egli self-assigned this Nov 24, 2021

egli modified the milestones: 3.20, 3.21 Dec 6, 2021

bertfrees added the tables Something that needs to be fixed in table files label Jan 14, 2022

egli linked an issue Jan 20, 2022 that may be closed by this pull request

Better integrate regular and "bidi" German tables #911

Closed

5 tasks

Merge de-g1-bidi-core.cti with de-g1-core.cti

4505ea9

Related to #911

egli force-pushed the bidi-merge branch from d1bb0b2 to 4505ea9 Compare January 26, 2022 14:07

egli added 4 commits January 26, 2022 15:12

Add some number infix tests

992ae3a

These tests are really weird and will probably never occur in real life text

Add lowercase.uti to disable capitalization

5024bb8

This table can be used to lowercase all chars before liblouis performs any translation.

Remove some superfluous numbersigns from some midnums

2329c31

Rename lowercase to latinLowercase and add to Makefile

c1c0067

egli force-pushed the bidi-merge branch from d16195e to c1c0067 Compare January 26, 2022 14:29

Remove a faulty pass2 line that caused problems in g2

b419466

And the comment says that the line can be removed once #717 has been fixed which it has.

bertfrees reviewed Jan 26, 2022

View reviewed changes

Comment thread tables/de-g0-core.uti Outdated

bertfrees reviewed Jan 26, 2022

View reviewed changes

Comment thread tables/de-g0-core.uti Outdated

bertfrees reviewed Jan 26, 2022

View reviewed changes

Comment thread tables/de-g0-core.uti Outdated

bertfrees reviewed Jan 26, 2022

View reviewed changes

Comment thread tests/braille-specs/de-g0-bidi-specs.yaml Outdated

egli force-pushed the bidi-merge branch from b684a20 to 93ceb09 Compare January 27, 2022 12:34

§ does indeed not require a dot 4 in front of it

1bf2a18

I checked with @mixa72

egli force-pushed the bidi-merge branch from 93ceb09 to 1bf2a18 Compare January 27, 2022 12:34

egli added 3 commits January 27, 2022 13:44

Move the mid- and endnum tests to de-g0.yaml

8ebcff7

Remove commented rule

3b852b6

egli added 2 commits February 9, 2022 15:06

Handle Æ with a context rule

0bc7dbb

This is probably a workaround around a bug in the base opcode where base doesn't seem to trigger when the lowercase char is preceded with a noback

Correct midword capitalization marks

d396cd5

egli added 4 commits February 11, 2022 10:10

Drop the special handling of Æ

281ceb8

Use virtual dots to handle Einzelgroßbuchstaben und Großbuchstabenfolgen

30ce121

and also move to the pass2 rules to the associated context rules

ch and st should be contracted according to the Systematik

f54a4a4

egli mentioned this pull request Feb 18, 2022

Only disable contraction in numeric mode if nocontractsign is actually defined #1159

Merged

Merge branch 'issue/615' into bidi-merge

678352a

bertfrees reviewed Mar 7, 2022

View reviewed changes

This was referenced Mar 7, 2022

Refactor the German test files #1175

Open

Better integrate regular and "bidi" German tables #911

Closed

egli marked this pull request as ready for review March 7, 2022 12:02

egli added 6 commits March 7, 2022 13:21

Merge branch 'master' into bidi-merge

bd8c350

Fix the handling of [ and ]

b3611f9

[ and ] are translated with a leading dot 6. So after a number we need a double dot 6 unless it will be read as a plain )

Remove a hardcoded number indicator

3e2e022

Lowercasing İ removes the accent, so mark this test as xfail

deead1f

Rename the bidi tables to detailed

a37adbe

Tranquilize the formatting check

9061dbf

bertfrees reviewed Mar 7, 2022

View reviewed changes

Improve meta data

1f54042

egli merged commit 79365f1 into master Mar 7, 2022

egli deleted the bidi-merge branch May 25, 2023 13:03

		@@ -1,6 +1,7 @@
		# liblouis: German grade 0 braille
		# liblouis: German grade 0 braille (bidirectional)

		#-index-name: German, uncontracted, detailed
		#-display-name: German uncontracted braille (detailed)

Uh oh!

Conversation

egli commented Aug 20, 2021

Uh oh!

egli commented Jan 26, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

egli commented Jan 26, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bertfrees commented Jan 26, 2022

Uh oh!

egli commented Jan 28, 2022

Uh oh!

egli commented Feb 9, 2022

Uh oh!

egli commented Feb 10, 2022

Uh oh!

bertfrees Mar 7, 2022

Choose a reason for hiding this comment

Uh oh!

egli Mar 7, 2022

Choose a reason for hiding this comment

Uh oh!

bertfrees Mar 7, 2022

Choose a reason for hiding this comment

Uh oh!

bertfrees Mar 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bertfrees Mar 7, 2022

Choose a reason for hiding this comment

Uh oh!

egli commented Mar 7, 2022

Uh oh!

bertfrees commented Mar 7, 2022

Uh oh!

egli commented Mar 7, 2022

Uh oh!

bertfrees commented Mar 7, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

egli commented Jan 26, 2022 •

edited

Loading

bertfrees Mar 7, 2022 •

edited

Loading