@kermitt2 I've checked the two file .wf and they look different:

(base) ➜  wordforms git:(improvement/fix-lexicon) ✗ head *.wf
==> english.wf <==
'd      have    Vafs--
'd      would   Vmfp--
'll     will    Vmfp--
'm      be      Vafps1
's      be      Vafps3
're     be      Vafpp-
's      have    Vafps3
's      have    S-
've     have    Vafpp-
Aachen  =       Np---

==> german.wf <==
1\abbestelle\1\35\ab-be-stel-le\N
2\abbestellst\0\35\ab-be-stellst\N
3\abbestellen\1\35\ab-be-stel-len\N
4\abbestellte\0\35\ab-be-stell-te\N
5\abbestelltet\0\35\ab-be-stell-tet\N
6\abbestellet\0\35\ab-be-stel-let\N
7\abbestellend\0\35\ab-be-stel-lend\N
8\aale\0\3\aa-le\N
9\aalst\0\3\aalst\N
10\aalt\0\3\aalt\N

and the loading part seems to be also depending on the language,

public final void addDictionary(String path, String lang) {
        File file = new File(path);
        if (!file.exists()) {
            throw new GrobidResourceException("Cannot add entries to dictionary (language '" + lang +
                "'), because file '" + file.getAbsolutePath() + "' does not exists.");
        }
        if (!file.canRead()) {
            throw new GrobidResourceException("Cannot add entries to dictionary (language '" + lang +
                "'), because cannot read file '" + file.getAbsolutePath() + "'.");
        }
        InputStream ist = null;
        InputStreamReader isr = null;
        BufferedReader dis = null;
        try {
            ist = new FileInputStream(file);
            isr = new InputStreamReader(ist, StandardCharsets.UTF_8);
            dis = new BufferedReader(isr);

            String l = null;
            while ((l = dis.readLine()) != null) {
                if (l.length() == 0) continue;
                // the first token, separated by a tabulation, gives the word form
                if (lang.equals(Language.EN)) {
                    // multext format
                    StringTokenizer st = new StringTokenizer(l, "\t");
                    if (st.hasMoreTokens()) {
                        String word = st.nextToken();
                        if (!dictionary_en.contains(word))
                            dictionary_en.add(word);
                    }
                } else if (lang.equals(Language.DE)) {
                    // celex format
                    StringTokenizer st = new StringTokenizer(l, "\\");
                    if (st.hasMoreTokens()) {
                        st.nextToken(); // id
                        String word = st.nextToken();
                        word = word.replace("\"a", "ä");
                        word = word.replace("\"u", "ü");
                        word = word.replace("\"o", "ö");
                        word = word.replace("$", "ß");
                        if (!dictionary_de.contains(word))
                            dictionary_de.add(word);
                    }
                }
            }
        } catch (IOException e) {
            throw new GrobidException("An exception occurred while running Grobid.", e);
        } finally {
            IOUtils.closeQuietly(ist, isr, dis);
        }
    }

Looks like we should load the german.wf with the DE language 🤔

Am I missing something?

German lexicon loaded with EN language? #1361

Description

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions