Create sensible default settings for "Enable save actions" and "Cleanup" dialogs by motokito · Pull Request #2051 · JabRef/jabref

motokito · 2016-09-24T11:21:52Z

Sensible default settings for "Enable save actions" field of database properties dialog and "Run field formatter" field of Cleanup eintries dialog are now identical 🐨
Please take a look for merge. THX 🐰

Change the default setting - by click on "Reset" button:

Change the default setting - by click on "Recommend for BibTeX" button:

Change the default setting - by click on "Recommend for BibLaTeX" button:

For further information have a look at:
JabRef#128
😄

Change in CHANGELOG.md described
Screenshots added (for bigger UI changes)
Manually tested changed features in running JabRef
Internal QS

koppor · 2016-09-24T21:36:33Z

 - Fixed [#1958](https://github.com/JabRef/jabref/issues/1958): Verbatim fields are no longer checked for HTML encoded characters by integrity checks
 - Fixed [#1937](https://github.com/JabRef/jabref/issues/1937): If no help page for the current chosen language exists, the english help page will be shown
 - Fixed [#1996](https://github.com/JabRef/jabref/issues/1996): Unnecessary other fields tab in entry editor removed (BibTeX mode)
+- Fixed [#128](https://github.com/koppor/jabref/issues/128): Sensible default settings for "Enable save actions" and "Cleanup"


Use koppor/#128 as normal numbers reference to the JabRef /jabref repo.

I have done it 🐨

koppor · 2016-09-29T02:12:25Z

Sorry, but not all settings from the original dialog have been included:

I miss the "convert to real superscripts". I think, this is "Ordinals to superscript"
Please go through line by line and state which new converter you chose. Example:
- Run HTML converter on title -> title: HTML to LaTeX
I would also love to have a toggle button for "Unicode conversion", which adds/removes the respective formatters being equivalent to Run Unicode converter on title, author(s), and abstract.
In case you are in context, could you check, why "Reformat ISSN" is there? Isn't that just another formatter?

Refs https://github.com/JabRef/help.jabref.org/issues/8

tobiasdiez · 2016-09-29T08:58:37Z

I don't like all these "title" formatters. I would propose to have a minimal default preset which does not change to much. For example, "protect terms" is really hard to revert by hand. Thus I would only include the formatters which do a minimal job which probably everybody wants (i.e normalized months, dates and pages) but not more.

motokito · 2016-09-29T12:59:55Z

@koppor oliver what do you think about tobiasdiez idea ??? 😏

motokito · 2016-09-29T13:08:48Z

I have a question about the function of each checkbox. Which change will be setted, if i select one. Maybe like "Convert 1st, 2nd, ... to real superscript, what will be change if i select it and i approve with "OK".

Also i have not understand, what do you meant in this task:

Please go through line by line and state which new converter you chose.

Thy for explaining 😄

koppor · 2016-09-29T22:10:20Z

To real superscripts is ordinals to latex. You could also have tried it out by creating an inprocredings entry containing a bootitle such as "2nd conference on something". Or just testing it with available bibtex files and checking the deltas (e.g., using git)

koppor · 2016-09-29T22:12:37Z

@tobiasdiez We should have buttons for presets. Minimal, Recommended (as in old JabRef versions) and Maximal (if we can come up w something useful) I think, I really want the ordinals thing included. Don't like the protect terms. Just want to put automatically braces around the complete title. But we don't have a formatter for that...

motokito · 2016-09-30T11:48:16Z

@koppor Event without the checkbox "convert to real superscripts" you can add the entryfield manually to the fieldformatter with the converter you chose eg.

Run Field Formatter -> choose entryfield -> booktitle: -> choose converter for superscripts -> for example HTML to Latex -> add

with this everyone can add his own converter for his own use thats why the checkbox "convert 1st, 2nd, ... to real superscripts" is not needed.

koppor · 2016-09-30T16:56:26Z

I know how to use the dialog. I want sensible defaults - see JabRef#128

boceckts

Still a changelog entry due to merge I guess, other than that LGTM.

boceckts · 2016-10-07T11:56:07Z

 - Fixed [#2104](https://github.com/JabRef/jabref/issues/#2104): Crash after saving BibTeX source with parsing error
 - Fixed [#2109](https://github.com/JabRef/jabref/issues/#2109): <kbd>Ctrl-s</kbd> doesn't trigger parsing error message
 - Fixed RTFChars would only use "?" for characters with unicode over the value of 127, now it uses the base character (é -> e instead of ?)
+- Fixed [#1996](https://github.com/JabRef/jabref/issues/1996): Unnecessary other fields tab in entry editor removed (BibTeX mode)


Remove this line.

motokito · 2016-10-10T15:00:03Z

The convert checkbox of old cleanup entries dialog is defined now in field formatter like following:

Run HTML converter on title -> title: HTML to LateX
Run Unicode converter on title, author(s), and abstract -> title: , author: or abstract: Unicode to LateX or LateX to Unicode
Run filter on title keeping the case of selected words -> title: , journal: or journal title: Protect terms
Remove unneccessary $, {, and } and move adjacent numbers into equations -> all_text_fields: LaTeX cleanup
Add brackets and replace separators with their non-breaking version for units -> all_text_fields: Units to LaTeX
Convert 1st, 2nd, ... to real superscripts -> all_text_fields: Ordinals to Latex superscript
Format content of month field to #mon# -> month: Normalize month
Ensure that page ranges are of the form num1--num2 -> pages: Normalize page numbers
Format date field in the form yyyy-mm or yyyy-mm-dd -> date: Normalize date

All are done, please take a look for merge 🎅

koppor · 2016-10-10T21:57:23Z

Can the unicode converter (line 2) also be run on all_text_fields?

Run filter on title keeping the case --> This is "protect terms". Please enable it on title, journal, and journal title.

koppor · 2016-10-10T21:59:45Z

    public static final String FIELD_SEPARATOR = "/";

+    public static final String ABSTRACT_ALL_FIELD = "all";
+    public static final String ABSTRACT_ALL_TEXT_FIELDS_FIELD = "all_text_fields";


Can this be named all-text-fields. I think dash ("-") is more intuitive than underscore ("_").

koppor · 2016-10-10T22:00:47Z

+    private List<FieldChange> cleanupAllTextFields(BibEntry entry) {
+        List<FieldChange> fieldChanges = new ArrayList<>();
+        Set<String> fields = entry.getFieldNames();
+        fields.removeAll(Arrays.asList(FieldName.DOI, FieldName.FILE, FieldName.URL, FieldName.URI));


Please introduce a class constant ALL_TEXT_FIELDS containing these fields. Please also add ISBN and ISSN, MONTH, DATE, YEAR,

oscargus · 2016-10-12T05:37:35Z

Some comments:

Most bst:s doesn't change the case of Journal titles, so no need to protect them (please provide any counter examples if I am wrong).
Adding braces around a complete field is in most cases the wrong way to do this as it will basically disable the bst handling of the field. (Note to @koppor )
Using an all_text_fields meta field would be quite nice. I'd suggest HTML to LaTeX on those and maybe Unicode to LaTeX (although this is indeed really BibTeX/Biblatex dependent, therefore my hesitation, although we may have "recommended for BibTeX" and "recommended for Biblatex" versions...).

tobiasdiez · 2016-10-12T07:31:13Z

I'm really no big fan of the proposed default values:

ProtectTermsFormatter: not needed in BibLaTex (and depends on the bibtex style)
UnitsToLatexFormatter: I use the si-units package and thus this conversion is counterproductive.
LatexCleanupFormatter: Is not good enough to be default in my opinion, but if the majority wants it then it would be fine with me
HtmlToLatexFormatter: for BibLaTeX you want Html -> Unicode instead
OrdinalsToSuperscriptFormatter: as a user of the recommended nth-package, this conversion is again suboptimal

In my opinion all these formatters depend on too much assumptions (BibTeX vs BibLaTeX and used packages) to be default. I find it better to have a minimal default preset and let the user decide which additional formatters he wants. Adding something feels better than removing something.
But any way...its just default values.

motokito · 2016-10-12T13:59:10Z

@koppor: all-text-fields will be function with unicode converter
i have done all. Please take a look for merge. THX 🍪

motokito · 2016-10-12T19:22:57Z

@koppor: i have implemented recommend button for BibLatex /BibTex and refactoring the default setting
please take a look for merge BIG THX 😄

boceckts · 2016-10-13T09:21:27Z

+        String mode;
+
+        if (isBibTex) {
+            mode = "BibTex";


Must be "BibTeX"

boceckts · 2016-10-13T09:21:44Z

+        if (isBibTex) {
+            mode = "BibTex";
+        } else {
+            mode = "BibLaTex";


Must be "BibLaTeX"

boceckts · 2016-10-13T09:22:45Z

+            mode = "BibLaTex";
+        }
+
+        recommendButton = new JButton(Localization.lang("Recommend for %0", mode));


Should be "Recommended"

boceckts · 2016-10-13T09:23:42Z

+        defaultFormatters.add(new FieldFormatterCleanup(FieldName.ABSTRACT_ALL_TEXT_FIELDS_FIELD, new OrdinalsToSuperscriptFormatter()));
        DEFAULT_SAVE_ACTIONS = new FieldFormatterCleanups(false, defaultFormatters);
+
+        List<FieldFormatterCleanup> recommendBibTexFormatters = new ArrayList<>(defaultFormatters);


Also rename variable to recommendedBibTeXFormatters.

boceckts · 2016-10-13T09:24:10Z

+        recommendBibTexFormatters.add(new FieldFormatterCleanup(FieldName.EDITOR, new UnicodeToLatexFormatter()));
+        RECOMMEND_BIBTEX_ACTIONS = new FieldFormatterCleanups(false, recommendBibTexFormatters);
+
+        List<FieldFormatterCleanup> recommendBibLatexFormatters = new ArrayList<>(defaultFormatters);


Also rename variable to recommendedBibLaTeXFormatters

…tabase properties

* rename all_text_fields to all-text-fields * remove ISBN and ISSN, MONTH, DATE, YEAR form all-text-fields

…tton incl. defaut setting

* Rename BibTex and BibLaTex to BibTeX and BibLaTeX * Rename Formatter list of BibTeX and BibLaTeX

"Recommended for %0" button is now disabled, if save actions are not enabled

* Testcase ensureNoDuplicates in JabRef_vi.properties

tobiasdiez · 2016-10-17T11:30:29Z

        List<String> fieldNames = InternalBibtexFields.getAllPublicFieldNames();
-        fieldNames.add(BibEntry.KEY_FIELD);
-        fieldNames.add("all");
+        fieldNames.addAll(Arrays.asList(BibEntry.KEY_FIELD, FieldName.ABSTRACT_ALL_FIELD, FieldName.ABSTRACT_ALL_TEXT_FIELDS_FIELD));


and extract it to a new method InternalBibtexFields.getAllPublicAndInteralFieldNames().
Btw, I would suggest to rename the ABSTRACT prefix to INTERNAL (when reading "FieldName.abstract_Something" I always think of the "abstract" field)

tobiasdiez · 2016-10-17T11:32:09Z

+        defaultFormatters.add(new FieldFormatterCleanup(FieldName.DATE, new NormalizeDateFormatter()));
        defaultFormatters.add(new FieldFormatterCleanup(FieldName.MONTH, new NormalizeMonthFormatter()));
-        defaultFormatters.add(new FieldFormatterCleanup(FieldName.BOOKTITLE, new OrdinalsToSuperscriptFormatter()));
+        defaultFormatters.add(new FieldFormatterCleanup(FieldName.ABSTRACT_ALL_TEXT_FIELDS_FIELD, new OrdinalsToSuperscriptFormatter()));


I would completely remove the OrdinalsToSuperscript from the defaults.

tobiasdiez · 2016-10-17T11:33:37Z

        DEFAULT_SAVE_ACTIONS = new FieldFormatterCleanups(false, defaultFormatters);
+
+        List<FieldFormatterCleanup> recommendedBibTeXFormatters = new ArrayList<>(defaultFormatters);
+        recommendedBibTeXFormatters.add(new FieldFormatterCleanup(FieldName.TITLE, new HtmlToLatexFormatter()));


Is there a reason to not run the HTML/Unicode -> Latex converter on all [text] fields? Similar question for the Latex/HTML -> Unicode for biblatex.

@koppor what do you think about that ??? 😄

tobiasdiez · 2016-10-17T11:35:21Z

+        defaultFormatters.add(new FieldFormatterCleanup(FieldName.ABSTRACT_ALL_TEXT_FIELDS_FIELD, new OrdinalsToSuperscriptFormatter()));
        DEFAULT_SAVE_ACTIONS = new FieldFormatterCleanups(false, defaultFormatters);
+
+        List<FieldFormatterCleanup> recommendedBibTeXFormatters = new ArrayList<>(defaultFormatters);


Make it more clear that default formatters are also added:
recommendedbibTexFormatters = new ArrayList();
recommendendBibTexFormatters.addAll(defaulttFormatters);

(same for biblatex)

tobiasdiez · 2016-10-17T11:37:05Z

+    private List<FieldChange> cleanupAllTextFields(BibEntry entry) {
+        List<FieldChange> fieldChanges = new ArrayList<>();
+        Set<String> fields = entry.getFieldNames();
+        fields.removeAll(Arrays.asList(FieldName.DOI, FieldName.FILE, FieldName.URL, FieldName.URI, FieldName.ISBN, FieldName.ISSN, FieldName.MONTH, FieldName.DATE, FieldName.YEAR));


return these fields from a new method FieldNames.getNotTextFieldNames()

tobiasdiez · 2016-10-17T11:40:11Z

    }

-    private static void insertCleanupPreset(Map<String, Object> storage, CleanupPreset preset) {
+    private static void insertCleanupPreset(Map<String, Object> storage) {


rename method to insertDefaultCleanupPreset

…ultSetting

koppor · 2016-10-25T18:24:58Z

It is part of the old dialog, isn't it? It doesnt have to be part for the minimum cleanups, but for the recommendations, it must!

koppor · 2016-10-25T18:29:33Z

Its my favourite. So keep it, please.

* create test for fieldformattercleanup * refactoring for default

tobiasdiez · 2016-10-26T02:34:39Z

        recommendedBibTeXFormatters.add(new FieldFormatterCleanup(FieldName.JOURNAL, new UnicodeToLatexFormatter()));
        recommendedBibTeXFormatters.add(new FieldFormatterCleanup(FieldName.AUTHOR, new UnicodeToLatexFormatter()));
        recommendedBibTeXFormatters.add(new FieldFormatterCleanup(FieldName.EDITOR, new UnicodeToLatexFormatter()));
+        defaultFormatters.add(new FieldFormatterCleanup(FieldName.INTERNAL_ALL_TEXT_FIELDS_FIELD, new OrdinalsToSuperscriptFormatter()));


recommendedBibTeXFormatters instead of defaultFormatters (same below)

…ultSetting # Conflicts: # CHANGELOG.md

motokito · 2016-10-27T05:19:41Z

@koppor all thing are doing. Pls take a look for merge 🎅

tobiasdiez · 2016-10-27T05:23:20Z

+        recommendedBibTeXFormatters.add(new FieldFormatterCleanup(FieldName.JOURNAL, new UnicodeToLatexFormatter()));
+        recommendedBibTeXFormatters.add(new FieldFormatterCleanup(FieldName.AUTHOR, new UnicodeToLatexFormatter()));
+        recommendedBibTeXFormatters.add(new FieldFormatterCleanup(FieldName.EDITOR, new UnicodeToLatexFormatter()));
+        defaultFormatters.add(new FieldFormatterCleanup(FieldName.INTERNAL_ALL_TEXT_FIELDS_FIELD, new OrdinalsToSuperscriptFormatter()));


its still "defaultFormatters" here instead of "recommendedBibTeXFormatters"....

tobiasdiez · 2016-10-27T05:23:51Z

        resetButton = new JButton(Localization.lang("Reset"));
        resetButton.addActionListener(e -> ((CleanupActionsListModel) actionsList.getModel()).reset(defaultFormatters));

+        boolean isBibTeX = !JabRefGUI.getMainFrame().getCurrentBasePanel().getDatabaseContext().isBiblatexMode();


work with "isBibLatex" instead of negating it.

* rename the recommended list * change "isBibTex" to "isBibLatex"

koppor · 2016-10-27T09:55:05Z

Good enough for me. Follow ups:

remove empty lines before closing { and if-statements
unifiy mode string generation with panel

Reason for second one: I assume that

 if (isBibLaTeX) {
            mode = "BibLaTeX";

also appears somewhere else in the code.

* Fixing: Sensible default settings for "Enable save actions" and "Cleanup" * Sensible default settings for "Enable save actions" in database properties and "Cleanup entries" in menu are now identical * Insert allTextField fieldName and refactoring the reset entries of database properties * CHANGELOG Fixing * Feedback * rename all_text_fields to all-text-fields * remove ISBN and ISSN, MONTH, DATE, YEAR form all-text-fields * Create Recommend Button for BibTex and BibLatex and refactor reset button incl. defaut setting * Refactoring: * Rename BibTex and BibLaTex to BibTeX and BibLaTeX * Rename Formatter list of BibTeX and BibLaTeX * Refactoring_15102016_2343: "Recommended for %0" button is now disabled, if save actions are not enabled * Fix LocalizationConsistencyTest FAIL: * Testcase ensureNoDuplicates in JabRef_vi.properties * Refactoring: * create test for fieldformattercleanup * refactoring for default * add OrdinalsToSuperscriptFormatter to recommand button * REFACTORING_27102016_0744: * rename the recommended list * change "isBibTex" to "isBibLatex" * Follow to refactor_27102016_1340

koppor reviewed Sep 24, 2016

View reviewed changes

boceckts added the stupro label Sep 25, 2016

Siedlerchr reviewed Sep 28, 2016

View reviewed changes

Comment thread CHANGELOG.md Outdated

Siedlerchr approved these changes Sep 28, 2016

View reviewed changes

koppor reviewed Sep 29, 2016

View reviewed changes

Comment thread src/main/java/net/sf/jabref/logic/exporter/FieldFormatterCleanups.java Outdated

boceckts suggested changes Oct 7, 2016

View reviewed changes

boceckts approved these changes Oct 9, 2016

View reviewed changes

boceckts added the status: ready-for-review Pull Requests that are ready to be reviewed by the maintainers label Oct 9, 2016

boceckts changed the title ~~[WIP] Create sensible default settings for "Enable save actions" and "Cleanup" dialogs~~ Create sensible default settings for "Enable save actions" and "Cleanup" dialogs Oct 9, 2016

koppor removed the status: ready-for-review Pull Requests that are ready to be reviewed by the maintainers label Oct 10, 2016

koppor reviewed Oct 10, 2016

View reviewed changes

boceckts added status: ready-for-review Pull Requests that are ready to be reviewed by the maintainers and removed status: ready-for-review Pull Requests that are ready to be reviewed by the maintainers labels Oct 13, 2016

boceckts reviewed Oct 13, 2016

View reviewed changes

boceckts added the status: ready-for-review Pull Requests that are ready to be reviewed by the maintainers label Oct 13, 2016

motokito added 7 commits October 17, 2016 06:57

Insert allTextField fieldName and refactoring the reset entries of da…

0a3d823

…tabase properties

CHANGELOG Fixing

e209e5a

Feedback

d597ad9

* rename all_text_fields to all-text-fields * remove ISBN and ISSN, MONTH, DATE, YEAR form all-text-fields

Create Recommend Button for BibTex and BibLatex and refactor reset bu…

897fb9e

…tton incl. defaut setting

Refactoring:

704eca7

* Rename BibTex and BibLaTex to BibTeX and BibLaTeX * Rename Formatter list of BibTeX and BibLaTeX

Refactoring_15102016_2343:

e4afabf

"Recommended for %0" button is now disabled, if save actions are not enabled

Fix LocalizationConsistencyTest FAIL:

75497c1

* Testcase ensureNoDuplicates in JabRef_vi.properties

tobiasdiez requested changes Oct 17, 2016

View reviewed changes

koppor added status: waiting-for-feedback The submitter or other users need to provide more information about the issue and removed status: ready-for-review Pull Requests that are ready to be reviewed by the maintainers labels Oct 24, 2016

Merge remote-tracking branch 'upstream/master' into cleanupDialogDefa…

c3dee55

…ultSetting

motokito added 2 commits October 25, 2016 21:11

Refactoring:

d18bbce

* create test for fieldformattercleanup * refactoring for default

add OrdinalsToSuperscriptFormatter to recommand button

5677942

tobiasdiez reviewed Oct 26, 2016

View reviewed changes

Merge remote-tracking branch 'upstream/master' into cleanupDialogDefa…

b15e7fa

…ultSetting # Conflicts: # CHANGELOG.md

tobiasdiez reviewed Oct 27, 2016

View reviewed changes

REFACTORING_27102016_0744:

b593ced

* rename the recommended list * change "isBibTex" to "isBibLatex"

koppor merged commit e5cfb05 into JabRef:master Oct 27, 2016

motokito mentioned this pull request Oct 27, 2016

Refactoring for PR #2051 #2202

Merged

koppor removed the status: waiting-for-feedback The submitter or other users need to provide more information about the issue label Jan 30, 2017

This was referenced Nov 8, 2019

Sensible default settings for "Enable save actions" and "Cleanup" JabRef/jabref-koppor#128

Closed

Add switch to change from biblatex to bibtex and vice versa (fi… #5565

Merged

Ka0o0 mentioned this pull request Nov 9, 2019

Adopt recommended file save actions upon database mode change #5579

Open

ThiloteE mentioned this pull request May 12, 2022

Enhance "Recommended" settings for field formatters (which are e.g. used to cleanup entries) #8790

Open

Uh oh!

Conversation

motokito commented Sep 24, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

koppor commented Sep 29, 2016

Uh oh!

Uh oh!

tobiasdiez commented Sep 29, 2016

Uh oh!

motokito commented Sep 29, 2016

Uh oh!

motokito commented Sep 29, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

koppor commented Sep 29, 2016 via email

Uh oh!

koppor commented Sep 29, 2016 via email

Uh oh!

motokito commented Sep 30, 2016

Uh oh!

koppor commented Sep 30, 2016

Uh oh!

boceckts left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

motokito commented Oct 10, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

koppor commented Oct 10, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

motokito Oct 12, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

koppor Oct 10, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

oscargus commented Oct 12, 2016

Uh oh!

tobiasdiez commented Oct 12, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

motokito commented Oct 12, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

motokito commented Oct 12, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

motokito commented Sep 24, 2016 •

edited

Loading

motokito commented Sep 29, 2016 •

edited

Loading

motokito commented Oct 10, 2016 •

edited

Loading

motokito Oct 12, 2016 •

edited

Loading

koppor Oct 10, 2016 •

edited

Loading

tobiasdiez commented Oct 12, 2016 •

edited

Loading

motokito commented Oct 12, 2016 •

edited

Loading