Implementation of shared database support by obraliar · Pull Request #1451 · JabRef/jabref

obraliar · 2016-05-26T03:25:22Z

Implementation of shared database support for MySQL, PostgreSQL and Oracle database systems.

Made changes:

Old sqlpackage removed. (This request aims to replace the old system completely)
Automatic change push to the server via EventBus. (Support for all kinds of EntryEvents, see AddEventSystem #1028)
Full synchronization support for MySQL, PostgreSQL and Oracle
(manual and semi-automated pull)
Conflict avoidance by version control and Optimistic Offline Lock
Efficient pull using version numbers
Synchronization of metadata with conflicts avoiding mechanisms
Metadata appliance
Groups synchronization
Mode synchronization (mode switch on the fly possible)
Adjusted tab and window titling
New localization entries
UI dialog for opening new or existing shared database
UI dialogs for resolving conflicts between BibEntries
Key bindings
Connection loss handling
Switch to offline working mode
Driver availibilty check (e.g. for the dropdown list)
Tests for all classes localized in shared package
Synchronization tests

Screenshots:

Change in CHANGELOG.md described
Tests created for changes
Screenshots added (for bigger UI changes)

obraliar · 2016-05-26T03:25:59Z

Issue: #970

Siedlerchr · 2016-05-26T09:43:05Z

    private final List<Object> openDatabaseOnlyActions = new LinkedList<>();
    private final List<Object> severalDatabasesOnlyActions = new LinkedList<>();
    private final List<Object> openAndSavedDatabasesOnlyActions = new LinkedList<>();
+    private final List<Object> remoteDatabasesOnlyActions = new LinkedList<>();


Could you please investigate if its possible to use a more concrete typ here and above then object?
Using Object in a generic type is making the whole concept of generics ad absurdum.

Those lists are used to add actions. Unfortunately there are differnt types of them, such as AbstractAction, GeneralActions which extends MnemonicAwareAction or just the interface Action.
I think this could be a good stuff for a new PR.

Siedlerchr · 2016-05-26T09:57:26Z

Looks good overall, just some minor remarks 👍 Please also add some new Database tests.

simonharrer · 2016-05-26T10:08:26Z

Why do you want to include Oracle? What is the reasoning here? Is this a DB that is likely to be used in this context? I am not really sure as Oracle is expensive and hard to maintain.

I am against changing the events and adding a location to them. Why must this be differentiated in the first place?

All the metadata must be stored in the DB as well, including biblatex or bibtex mode (this is not something the user should change). Also do not forget the save actions, as they have an influence on the values in the DB as well.

What happens if a change locally cannot be stored in the DB as there are conflicts?

koppor · 2016-05-26T12:30:20Z

Me really wants Oracle. Some bigger institutes use Oracle and they don't want to run other databases. At least, this is, what I understood.

I will contact them again and ask how important Oracle is.

Currently, it seems that the hard part of the Oracle implementation is already done. The synchronization via events seems to be easy - see implementation hints at #970.

simonharrer · 2016-05-26T12:52:04Z

Ok, then please remove the lib file and use a file from maven repositories for the ojdbc.jar

obraliar · 2016-05-28T16:59:23Z

@simonharrer

Why must this be differentiated in the first place?

DBSynchronizer class listens to EntryEvents like EntryAddedEvent and pushes the new entry to the server. Such an event is sent from the method BibDatabase.insertEntry. The problem is that DBSynchronizer (and other classes in remote package) itself also use local methods such as BibDatabase.insertEntry or BibEntry.setField to update the local database. In this case the sent event should not trigger the SQL inserting/updating/deleting methods again (see https://git.io/vrHOw). But as I described in the code comments, if we remove the EventLocation it would not harm. It only causes a little redundancy.

All the metadata must be stored in the DB as well, including biblatex or bibtex mode...

Okay, this is going to be implemented.

Also do not forget the save actions, as they have an influence on the values in the DB as well.

Unfortunately I didn' t understand that completely. Now if you work on remote database you've still the possibility to save the file locally. Otherwise the changes are pushed by EntryEvents. Are there some other data which are involved within the saving process?

What happens if a change locally cannot be stored in the DB as there are conflicts?

In this mode the newest version is held by remote database. Changes are going to be pushed automatically and every time you push an entry your local database is also going to be updated (this process is going to be accelerated via hashing etc). So there is no scenario a conflict can occur except connection problems. But in this case the editor should restrict any editings (TODO). Or two people are working at the same time at one field. In this case the last push will win.

simonharrer · 2016-05-29T07:46:43Z

The save actions control stuff like that a specific field must be lower or upper case letters. Think about it: person A always uses lower case letters for the title, and person B uses always upper case letters for the title - what effect will this have on the database sync when basically the same data is used? They will create a lot of traffic if they work concurrently as the save of one will overwrite all the title fields of the other, and it will create conflicts. You can only overcome this if both know the correct save actions, and that is the reason why the save actions are stored in the metadata.

Regarding resolving conflicts: sounds ok to be practical. It feels wrong, however, from my background with using version control systems.

tobiasdiez · 2016-06-04T11:37:43Z

I think it would be very profitable to unite the code for file-based exports with the code for sql-exports. For example, I really like the immediately sync changes to the sql database feature, which also would be nice for bib-files (similar to intellij's autosave). On the other hand some features which are currently missing for sql exports (like save actions or how external changes are handled) are already present for bib files.
I did some refactorings in #1472 to make this unification easier.

obraliar · 2016-06-04T13:32:56Z

Currently I'm working on save actions. As I see the remote synchronization differs a lot from the local saving of metadata and defaults (think about the relational structure). Unifiying the code would dissolve the remote package and a lot of things would have to be moved/passed through to BibtexParser/BibtexDatabaseWriter which would blow up those classes. Furthermore it would cost a lot of time which is not present.

Currently I'm going to implement this in remote package. If there is some time capacity left after the first release, we could make some considerations to implement this and other customer wishes.

obraliar · 2016-06-24T20:44:24Z

@simonharrer @koppor It seems that ojdbc.jar is not longer available in maven repositories.
(see http://central.maven.org/maven2/ojdbc/ojdbc/14/)

compile group: 'ojdbc', name: 'ojdbc', version: '14' in build.gradle causes:

Execution failed for task ':tasks'.
> Could not resolve all dependencies for configuration ':runtime'.
   > Could not find ojdbc.jar (ojdbc:ojdbc:14).
     Searched in the following locations:
         https://jcenter.bintray.com/ojdbc/ojdbc/14/ojdbc-14.jar

koppor · 2016-07-17T12:55:20Z

Since groups are a very important feature, @obraliar will work on groups in this PR, too. Other than groups, the feature is ready to be investigated.

oscargus · 2016-07-17T13:52:09Z

 Converts_HTML_code_to_Unicode.=
+Connection=
+Host=
+Port=


If possible it would be nice to reuse the existing translations of Port and Database. (One can consider reusing "Server hostname" and possibly "Username" as well, to not add more strings.)

obraliar · 2016-07-17T14:43:55Z

@Goldfinger2007: Your BibEntry contains some characters, which invalidates the SQL query. This is going to be fixed. Thanks for the test.

simonharrer · 2016-08-03T07:02:37Z

+    private String type;
+    private String driverPath;
+    private String urlPattern;
+    private int defaultPort;


make these parameters final

koppor · 2016-08-04T19:13:34Z

Conflict resolution with Optimistic Offline Lock 👍

After resolving the minor issues from above, I vote for merging to prevent the continuous merging conflicts.

Things to be solved after merging (Starting from Thursday, 2016-08-18): - [ ] Database properties dialog: - [ ] Do not show "Save sort order" - database sorts arbitrarily 😇 - [ ] Do not show "Database protection" - makes no sense in a shared database setting - [ ] Session restore: JabRef should open all `[shared]` tabs after startup - [ ] Store used connections in the "open remote database dialog"

Follow up at #1703

Support for MySQL, PostgreSQL and Oracle database systems. Full entry synchronisation. Full meta data synchronization. Group synchrnoization. Semi-automatic/event based synchronization. Version control system. Conflicts resolving machanisms. User interfaces for different situations. Extensive synchronization and class tests.

tobiasdiez · 2016-09-09T23:07:57Z

@obraliar Could you please answer http://discourse.jabref.org/t/new-sql-structure/197/4 ? Thanks!

obraliar force-pushed the SharedDatabaseSupport branch 2 times, most recently from e79c5ff to add5fb3 Compare May 26, 2016 03:38

Siedlerchr reviewed May 26, 2016
View reviewed changes

tobiasdiez added the stupro-ready-for-internal-review label May 26, 2016

obraliar force-pushed the SharedDatabaseSupport branch from add5fb3 to e7dcebc Compare May 28, 2016 16:59

Braunch added the stupro label May 30, 2016

obraliar force-pushed the SharedDatabaseSupport branch 7 times, most recently from d47d740 to ee64d6a Compare June 3, 2016 20:08

tobiasdiez mentioned this pull request Jun 4, 2016

Refactor writing of bib files #1472

Merged

3 tasks

koppor mentioned this pull request Jun 6, 2016

Fix #1476 NPE when importing from SQL DB because of missing DatabaseMode #1477

Merged

obraliar force-pushed the SharedDatabaseSupport branch 3 times, most recently from e092e5c to bc26166 Compare June 23, 2016 18:43

oscargus reviewed Jul 17, 2016
View reviewed changes

koppor changed the title ~~Implementation of shared database support (base system)~~ [WIP] Implementation of shared database support (base system) Jul 18, 2016

obraliar force-pushed the SharedDatabaseSupport branch 8 times, most recently from b468ef8 to 5cca450 Compare July 19, 2016 12:46

koppor mentioned this pull request Jul 29, 2016

Update to Java 8 regarding the library/API usage #1154

Closed

4 tasks

simonharrer reviewed Aug 3, 2016
View reviewed changes

This was referenced Aug 9, 2016

Improve shared database feature #1703

Closed

Create blog entry for shared database support JabRef/blog.jabref.org#1

Open

Enable testing of Oracle on TravisCI JabRef/jabref-koppor#151

Closed

Uh oh!

Conversation

obraliar commented May 26, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

obraliar commented May 26, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Siedlerchr May 26, 2016

Choose a reason for hiding this comment

Uh oh!

obraliar May 28, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Siedlerchr commented May 26, 2016

Uh oh!

simonharrer commented May 26, 2016

Uh oh!

koppor commented May 26, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

simonharrer commented May 26, 2016

Uh oh!

obraliar commented May 28, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

simonharrer commented May 29, 2016

Uh oh!

tobiasdiez commented Jun 4, 2016

Uh oh!

obraliar commented Jun 4, 2016

Uh oh!

obraliar commented Jun 24, 2016

Uh oh!

koppor commented Jul 17, 2016

Uh oh!

oscargus Jul 17, 2016

Choose a reason for hiding this comment

Uh oh!

obraliar commented Jul 17, 2016

Uh oh!

simonharrer Aug 3, 2016

Choose a reason for hiding this comment

Uh oh!

obraliar Aug 4, 2016

Choose a reason for hiding this comment

Uh oh!

koppor commented Aug 4, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tobiasdiez commented Sep 9, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

obraliar commented May 26, 2016 •

edited

Loading

obraliar commented May 26, 2016 •

edited

Loading

obraliar May 28, 2016 •

edited

Loading

koppor commented May 26, 2016 •

edited

Loading

obraliar commented May 28, 2016 •

edited

Loading

koppor commented Aug 4, 2016 •

edited

Loading