Add an importer for Citavi backup files by lishangyu9 · Pull Request #8848 · JabRef/jabref

lishangyu9 · 2022-05-25T05:48:48Z

Fixes #8322 .
This importer for Citavi support ‘ctv5bak’ and ‘ctv6bak’ files.

Change in CHANGELOG.md described in a way that is understandable for the average user (if applicable)
Tests created for changes (if applicable)
Manually tested changed features in running JabRef (always required)
Screenshots added in PR description (for UI changes)
Checked developer's documentation: Is the information available and up to date? If not, I outlined it in this pull request.
Checked documentation: Is the information available and up to date? If not, I created an issue at https://github.com/JabRef/user-documentation/issues or, even better, I submitted a pull request to the documentation repository.

What we have added

The Citavi file format support
Add the definition of Citavi file in the “StandardFileType.java”.
The XSD file defined the Citavi XML file
Add the file “scitavi.xsd” for generating the java class “CitaviExchangeData.java”.
The importer for Citavi file “ctv5bak” and “ctv6bak”
- Add the class “src/main/java/org/jabref/logic/importer/fileformat/CitaviXmlImporter.java”.
  Because the “ctv6bak” and “ctv5bak” are compressed files. We add the function “getReaderFromZip” to unzip the file before parsing.
- The XML file inside the “ctv6bak” is UTF-8 BOM format. This format cannot be recognized by the default process of XML. Therefore We create a function “checkForUtf8BOMAndDiscardIfAny” to delete the byte at the beginning of the XML file.
- The process of parsing XML data is similar to “EndnoteImporter”. However, the date structures of Endnote and Citavi are different. The primary different methods are “getAuthorName”, “getEditorName”, “getKeywords” and “getPublisher”. These data are located in different places with reference data. For example, we need to go through the author’s matching list and match the references ID. If the id is matched we got the author’s ID list. Finally, we can get the author’s name.
The test of Citavi importer
The test class of Citavi importer is “CitaviXmlImporterTestFiles.java”.
There are 4 test cases:
“CitaviXmlImporterTest1.ctv6bak”, “CitaviXmlImporterTest1.bib”
“CitaviXmlImporterTest2.ctv6bak”, “CitaviXmlImporterTest2.bib”
“CitaviXmlImporterTest3.ctv6bak”, “CitaviXmlImporterTest3.bib”
“CitaviXmlImporterTest4.ctv6bak”, “CitaviXmlImporterTest4.bib”

How to use the Citavi Importer

Click the “File” → ”Import” → ”Import into new library”.
Select the file. Click “Open”.
The file is imported

Add generated citavi xsd schema

… utf-8 BOM bytes in the XML file

…authors

Fix some resouces leaks Use AuthorListParser

Siedlerchr · 2022-05-25T19:32:39Z

Thank you very much for your work! Impressive 👍 I took a more in-depth look at the author handling code, that's really ugly but I couldn't find a general better approach. However, you do not have to iterate over the whole String, the ids are all UUIDs that have a fixed length of 36 characters. So you can improve your loops.

Second thing: I implemented an example of using JabRef's Author and AuthorLIst class for adding new authors. You just need to adjust your test bib files (Lastname, Firstname), can be done in JabRef's GUI: (Quality-> Cleanup -> Normalize in persons) and maybe the same for other persons fields.

Store referenceId and identified persons also

Siedlerchr · 2022-05-27T20:46:29Z

@lishangyu9 I came up with a more efficient solution for the authors and editors.
In essential, the oneToN is similar to a relational database schema where the first part as you already identified is the referenced and the other ones are the person ids.

Instead of iterating through the whole author + person list for every reference, I create a mapping between referenceId and Persons. In addition, I introduced a second Map for storing Persons we already know (if the same author with the same uuid comes up multiple times, so we don't have to through all Person objects everytime.

Antoher thing we can use for our advantage: We know that all UUIDs have a length of 36 chars and we know that the first only the first UUID is the referenceID. So we can safely assume that the rest of the string are the Person ids. So we can use split to get an array of all person ids if we just split at the semicolon, after we removed the refId from it.

This can be adapted for other oneToN relationships as well

lishangyu9 · 2022-05-28T04:26:07Z

Thank you @Siedlerchr . We will check and fix it today.

lishangyu9 · 2022-05-28T08:34:35Z

Hi @Siedlerchr, we fixed other's OneToN relationships, "Keywords" and "Publishers". We followed the method you have made and created the "buildKeywordList" and "buildPublisherList" functions for mapping the data and reference Id. We appreciate your help.

Siedlerchr

Just some minor changes, but overall looks very good to me!

Siedlerchr · 2022-05-29T09:05:49Z

@claell You were using citavi previously, it would be nice if you could testt this PR.
(best is using github cli tools, gh pr checkout 8848 and then ./gradlew run should assemble & run jabref

…porter.java Co-authored-by: Christoph <siedlerkiller@gmail.com>

claell · 2022-05-31T14:28:40Z

@Siedlerchr when I find the time, I'll try that. I don't have GitHub's cli tools installed. Is there another easy way or should I just use them? Also, I likely cannot do too in depth testing due to constrained time.

calixtus · 2022-05-31T14:44:00Z

gh cli is supposed to be the easy way... you can of course also follow the "command line instructions" on the bottom of this page

calixtus · 2022-05-31T14:44:33Z

koppor

In general, it is OK. Some nitpicks, but we can fix them for ourselves later.

The thing I see here, that there was strong test-driven development somehow:

buildPersonList, buildKeywordList, buildPublisherList should all have been package private and tested by CitaviXmlImporter.

However, I think, this project is over. Think about using JUnit and the play / debug button also for methods themselves the next java project.

calixtus · 2022-06-01T06:27:23Z

I just mentioned you are mixing the old io and the new nio file apis. Is there a chance to switch completely to the new nio API? Provides thread safety i.a....

Siedlerchr · 2022-06-01T07:21:54Z

I just mentioned you are mixing the old io and the new nio file apis. Is there a chance to switch completely to the new nio API? Provides thread safety i.a....

@calixtus Ehm they are meant to be interoperable, for example Files.newInputStream returns an InputStream. And Inputstream is just the base class in java.io. There is not all in nio.

* upstream/main: Add an importer for Citavi backup files (#8848) Reviewdoc: Comment on PRs (#8878) Squashed 'buildres/csl/csl-styles/' changes from 649aac4..e740261 Use JDK 15 text blocks to improve injected languages readability (#8874) Fix fetcher tests (#8877) Fix #8390 by allowing multiple group deletion for Remove groups > Kee… (#8875) Add restart warning on SSL configuration change (#8871) Update to lucene 9.2 (#8868) Fix for removing several groups deletes only one of them (#8801) Disable Write XMP Button in General tab of Entry-Editor when action is in progress (#8728) Bump jsoup from 1.14.3 to 1.15.1 (#8864) Bump unirest-java from 3.13.8 to 3.13.10 (#8869) Bump unoloader from 7.3.2 to 7.3.3 (#8863) Bump pascalgn/automerge-action from 0.15.2 to 0.15.3 (#8860) Bump classgraph from 4.8.146 to 4.8.147 (#8861) Bump mockito-core from 4.5.1 to 4.6.0 (#8862) Lucence dir checkers should only delete lucence dirs (#8854) Update README.md (#8858) Update adr.md Update adr.md

Siedlerchr · 2022-06-01T20:13:47Z

Thanks again for your valuable contribution and your quick follow up! 👍
We just noticed by chance that the pages field contained unicode minus signs instead of the normal ascii ones. So I created a follow up PR which calls the pages formatter

Siedlerchr and others added 30 commits April 14, 2022 18:21

First draft for citavi

23fd2d2

Add generated citavi xsd schema

add 'CitaviXmlImporter.java' first draft, based on endnotexmlimporter.

4ce0f3a

Add '.ctv6bak' and '.ctv5bak' file type in the file type list.

0ddefdf

Modify the citavi importer, draft of importDatabase()

f339dab

Modify the citavi importer, draft of isRecognizedFormat()

6adf1b6

fix spell error for citavi

be26116

update getReaderFromZip()

1d79364

XSD file for Citavi updated and isRecognizedFormat method updated

9a1e6ff

update xsd

9f14063

update xsd

ffd23b0

update CitaviXmlImporter.java

177930b

update CitaviXmlImporter.java. Add a function to check and delete the…

48ebb0b

… utf-8 BOM bytes in the XML file

update xsd file and parseData function in citaviimporter

51d95ee

Add CitaviXmlImporterTestFiles.java

1dee43d

Merge remote-tracking branch 'origin/citavixml' into citavixml

ec8b195

fix codestyle

b72c43c

Merge remote-tracking branch 'origin/citavixml' into citavixml

df6cf9c

CitaviXmlImporterTestFiles.java update the test functions

c75f0f6

Add test example

6f2ea32

XSD file updated. Importing functions in CitaviXmlImporter updated.

bd09c1b

Merge remote-tracking branch 'origin/citavixml' into citavixml

d13c4f7

Add clean up for temp file

f8e4179

Add test files

b97e3cb

Importing authors' name functionality fixed

c4cb2ac

Some other fields added

d995c40

fix the author and editor format

97e2bdc

Small bug fixed

571ed8e

fixed the null pointer exception of publishers ,keywords,editors and …

8b1e979

…authors

add the test examples.

3e52a56

update CHANGELOG.md

c65628f

Remove unused import format preferences

c17a6ad

Fix some resouces leaks Use AuthorListParser

Siedlerchr added 2 commits May 25, 2022 21:34

checkstyle

fd8ee90

Store knownPersons in a HashMap, for efficient lookup

447e7ad

Store referenceId and identified persons also

Siedlerchr added 2 commits May 27, 2022 22:46

checkstyle

a4df2cc

remove blank lines

3b1823f

update the logic to parse Keywords and Publishers.

901555e

Siedlerchr added the status: ready-for-review Pull Requests that are ready to be reviewed by the maintainers label May 28, 2022

Siedlerchr approved these changes May 29, 2022

View reviewed changes

lishangyu9 and others added 5 commits May 30, 2022 09:52

Update src/main/java/org/jabref/logic/importer/fileformat/CitaviXmlIm…

8953487

…porter.java Co-authored-by: Christoph <siedlerkiller@gmail.com>

Update src/main/java/org/jabref/logic/importer/fileformat/CitaviXmlIm…

9188f9f

…porter.java Co-authored-by: Christoph <siedlerkiller@gmail.com>

update CHANGELOG.md

6aad659

Merge remote-tracking branch 'origin/citavixml' into citavixml

c13816e

delete the optional warp of getPages()

53e6f31

koppor approved these changes May 31, 2022

View reviewed changes

Comment thread src/main/java/org/jabref/logic/importer/fileformat/CitaviXmlImporter.java Outdated

Comment thread src/main/java/org/jabref/logic/importer/fileformat/CitaviXmlImporter.java

lishangyu9 added 2 commits June 1, 2022 11:17

delete the JavaDoc lines

17a254e

add an empty line between varable declaration and method.

41f7bfb

Siedlerchr merged commit 943489e into JabRef:main Jun 1, 2022

Siedlerchr mentioned this pull request Jun 1, 2022

Normalize pages in citavi to avoid unicode minus signs #8881

Merged

6 tasks

Uh oh!

Conversation

lishangyu9 commented May 25, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What we have added

The Citavi file format support

The XSD file defined the Citavi XML file

The importer for Citavi file “ctv5bak” and “ctv6bak”

The test of Citavi importer

How to use the Citavi Importer

Uh oh!

Siedlerchr commented May 25, 2022

Uh oh!

Siedlerchr commented May 27, 2022

Uh oh!

lishangyu9 commented May 28, 2022

Uh oh!

lishangyu9 commented May 28, 2022

Uh oh!

Siedlerchr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Siedlerchr commented May 29, 2022

Uh oh!

claell commented May 31, 2022

Uh oh!

calixtus commented May 31, 2022

Uh oh!

calixtus commented May 31, 2022

Uh oh!

koppor left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

calixtus commented Jun 1, 2022

Uh oh!

Siedlerchr commented Jun 1, 2022

Uh oh!

Siedlerchr commented Jun 1, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

lishangyu9 commented May 25, 2022 •

edited

Loading