Fix issue #11189 - Implement a caching solution with local storage for citation relations #11845

alexandre-cremieux · 2024-09-28T20:38:04Z

High level description

This contributions aims to provide a local storage solution for citations relations using MVStore. Please see #11189 for more information.

Fixes #11189

Implementation details

No new dependency added.

Caching strategy

The caching logic of the citations/references is now fully handled by h2.mvstore.MVStore. The store is responsible of the in memory caching and the offline storage logic.

This aims to avoid JabRef to access the Internet if the user already searched for citations/references related to one of its bib entry.

The cache/local store is common to all the libraries managed by the JabRef instance.

Search caching high level logic

Check if the fetch is possible: if the cache does not contain anything or if the local storage TTL ran out
If the fetch is possible: fetch and insert into cache (in memory and locally)
Read from cache (if the in-memory cache is empty then load from local storage and update in-memory cache)

Serialization

MVStoreDAO uses the canonical representation to serialize a BibEntry and the BibtexParser to deserialize it.

Configuration

the search service is a singleton injected in each new Citations Relations tab using the JabRef IoC provider
the store path is configured to target the citations folder under JabRef's app directory

Setting translation

The the local storage TTL setting message has been traduced in:

DE
FR
EN
PL
IT
BR

Refactoring

This contributions simplifies the citations/references fetching and caching logic by introducing two layers:

service
repository

This should help to make this feature more extendable without modifying orchestration logic following open/close principle.

Screen shots

Mandatory checks

I own the copyright of the code submitted and I license it under the MIT license
Change in CHANGELOG.md described in a way that is understandable for the average user (if applicable)
Tests created for changes (if applicable)
Manually tested changed features in running JabRef (always required)
Screenshots added in PR description (for UI changes)
Checked developer's documentation: Is the information available and up to date? If not, I outlined it in this pull request.
Checked documentation: Is the information available and up to date? If not, I created an issue at https://github.com/JabRef/user-documentation/issues or, even better, I submitted a pull request to the documentation repository.

github-actions

Your code currently does not meet JabRef's code guidelines.
We use Checkstyle to identify issues.
The tool reviewdog already placed comments on GitHub to indicate the places. See the tab "Files" in you PR.
Please carefully follow the setup guide for the codestyle.
Afterwards, please run checkstyle locally and fix the issues.

You can check review dog's comments at the tab "Files changed" of your pull request.

github-actions

Your code currently does not meet JabRef's code guidelines.
We use Checkstyle to identify issues.
The tool reviewdog already placed comments on GitHub to indicate the places. See the tab "Files" in you PR.
Please carefully follow the setup guide for the codestyle.
Afterwards, please run checkstyle locally and fix the issues.

You can check review dog's comments at the tab "Files changed" of your pull request.

github-actions

Your code currently does not meet JabRef's code guidelines.
We use OpenRewrite to ensure "modern" Java coding practices.
The issues found can be automatically fixed.
Please execute the gradle task rewriteRun, check the results, commit, and push.

You can check the detailed error output by navigating to your pull request, selecting the tab "Checks", section "Tests" (on the left), subsection "OpenRewrite".

github-actions

Your code currently does not meet JabRef's code guidelines.
We use OpenRewrite to ensure "modern" Java coding practices.
The issues found can be automatically fixed.
Please execute the gradle task rewriteRun, check the results, commit, and push.

You can check the detailed error output by navigating to your pull request, selecting the tab "Checks", section "Tests" (on the left), subsection "OpenRewrite".

alexandre-cremieux

Add code explanations to the PR

src/main/java/org/jabref/gui/entryeditor/citationrelationtab/BibEntryRelationsCache.java

src/main/java/org/jabref/gui/entryeditor/citationrelationtab/BibEntryRelationsRepository.java

src/main/java/org/jabref/gui/entryeditor/citationrelationtab/CitationRelationsTab.java

src/main/java/org/jabref/logic/citation/repository/LRUBibEntryRelationsCache.java

src/main/java/org/jabref/logic/citation/repository/LRUBibEntryRelationsRepository.java

src/main/java/org/jabref/logic/importer/fetcher/CitationFetcher.java

src/main/java/org/jabref/logic/citation/service/SearchCitationsRelationsService.java

Siedlerchr · 2024-09-29T14:19:32Z

Please no force push if not needed. All commits will be squashed when merged

* Move repository, cache, and fetcher to logic package * Move citations model to model/citations/semanticscholar package

* Introduce service layer * Rename LRU cache implementation * Add tests helpers for repository

* Move logic from repository to service * Refactor repositories * Update tab configuration

alexandre-cremieux · 2024-09-29T15:02:44Z

Please no force push if not needed. All commits will be squashed when merged

Sorry, just re-based main branch locally.

…lation-tab-logic # Conflicts: # src/main/java/org/jabref/gui/entryeditor/citationrelationtab/CitationRelationsTab.java

koppor

In general looks good. Some minor comments.

Sorry for delay. Please go ahead with everything.

src/main/java/org/jabref/logic/citation/service/SearchCitationsRelationsService.java

src/main/java/org/jabref/gui/entryeditor/citationrelationtab/CitationRelationsTab.java

src/test/java/org/jabref/logic/citation/service/SearchCitationsRelationsServiceTest.java

koppor · 2024-10-10T20:31:12Z

Small other comments - IntelliJ proposed to extract a method

private static SearchCitationsRelationsService getSearchCitationsRelationsService(BibEntry cited, List<BibEntry> citationsToReturn, BibEntryRelationsRepository citationsToReturn1) {

in the tests - maybe you can also include that.

koppor · 2024-10-10T20:31:37Z

@alexandre-cremieux Please pull before you continue working on it - I merged main for you (and resolved conflicts).

src/main/java/org/jabref/logic/citation/repository/LRUBibEntryRelationsRepository.java

src/main/java/org/jabref/logic/citation/service/SearchCitationsRelationsService.java

alexandre-cremieux · 2024-10-11T21:24:42Z

@alexandre-cremieux Please pull before you continue working on it - I merged main for you (and resolved conflicts).

Thanks for the review and the merge. I will resume the work on this branch and apply the changes.

koppor · 2024-11-08T09:19:45Z

@alexandre-cremieux Sorry for the merge conflicts - can you handle them? I was always happy with IntelliJ's "resolve merge conflicts" dialog. Hope, it works in this case, too.

alexandre-cremieux · 2024-11-08T21:03:04Z

@alexandre-cremieux Sorry for the merge conflicts - can you handle them? I was always happy with IntelliJ's "resolve merge conflicts" dialog. Hope, it works in this case, too.

Hello @koppor . Seems that we have new conflicts to resolve to be able to merge main. But I will do that when the feature will be fully developed. Was quite busy last month, I resumed the work this week. PR comments were addressed.

alexandre-cremieux · 2025-04-28T19:09:45Z

Hello @calixtus

I remove the LRU caching layer and updated the MVStore related code. The store is now opened when the service is created and closed only when the application is shutdown.

I tested locally, of course, and it is working. There should not be much change to do anymore and the code should be close to what is expected.

Could you please review ?

jabgui/src/main/java/org/jabref/gui/entryeditor/citationrelationtab/CitationRelationsTab.java

trag-bot · 2025-04-28T19:17:27Z

jabgui/src/main/java/org/jabref/gui/entryeditor/citationrelationtab/CitationRelationsTab.java

+    /**
+     * TODO: Make the method return a callable and let the calling method create the background task.
+     */
+    private BackgroundTask<List<BibEntry>> createBackgroundTask(


The method should return a Callable instead of directly creating a BackgroundTask, as indicated by the TODO comment, to improve flexibility and separation of concerns.

...main/java/org/jabref/logic/citation/repository/BibEntryCitationsAndReferencesRepository.java

...java/org/jabref/logic/citation/repository/BibEntryCitationsAndReferencesRepositoryShell.java

jablib/src/main/java/org/jabref/logic/citation/repository/BibEntryRelationRepository.java

trag-bot · 2025-04-28T19:28:47Z

jablib/src/main/java/org/jabref/logic/importer/ImporterPreferences.java

@@ -159,4 +164,16 @@ public ObjectProperty<PlainCitationParserChoice> defaultPlainCitationParserPrope
    public void setDefaultPlainCitationParser(PlainCitationParserChoice defaultPlainCitationParser) {
        this.defaultPlainCitationParser.set(defaultPlainCitationParser);
    }
+
+    public int getCitationsRelationsStoreTTL() {


The method should return an Optional instead of a primitive int to avoid returning null and to align with modern Java practices.

trag-bot · 2025-04-28T19:29:29Z

...g/jabref/logic/importer/fetcher/citation/semanticscholar/SemanticScholarCitationFetcher.java

@@ -14,14 +14,14 @@

 import com.google.gson.Gson;

-public class SemanticScholarFetcher implements CitationFetcher, CustomizableKeyFetcher {
+public class SemanticScholarCitationFetcher implements CitationFetcher, CustomizableKeyFetcher {


The class name was changed from SemanticScholarFetcher to SemanticScholarCitationFetcher, which is a reformatting change without adding new statements. This violates the guideline against reformatting code without adding new statements.

jablib/src/main/java/org/jabref/logic/util/BackgroundTask.java

jablib/src/main/java/org/jabref/logic/util/Directories.java

jablib/src/main/java/org/jabref/model/entry/BibEntry.java

Obsolete/addressed

subhramit

I just skimmed through the code, just some formatting suggestions.
Others would be doing a thorough review.

subhramit · 2025-04-29T11:08:24Z

jabgui/src/main/java/org/jabref/gui/JabRefGUI.java

+        JabRefGUI.citationsAndRelationsSearchService = new SearchCitationsRelationsService(
+                preferences.getImporterPreferences()
+        );


Nitpick: one line enough for this

subhramit · 2025-04-29T11:09:38Z

jabgui/src/main/java/org/jabref/gui/entryeditor/citationrelationtab/CitationRelationsTab.java

-            importCitingButton,
-            citingProgress,
-            true));
+        refreshCitingButton.setOnMouseClicked(event -> {


Suggested change

refreshCitingButton.setOnMouseClicked(event -> {

refreshCitingButton.setOnMouseClicked(_ -> {

subhramit · 2025-04-29T11:10:00Z

jabgui/src/main/java/org/jabref/gui/entryeditor/citationrelationtab/CitationRelationsTab.java

+        refreshCitingButton.setOnMouseClicked(event -> {
+            searchForRelations(entry, citingListView, abortCitingButton,
+                    refreshCitingButton, CitationFetcher.SearchType.CITES, importCitingButton, citingProgress);
+        });

        refreshCitedByButton.setOnMouseClicked(event -> searchForRelations(entry, citedByListView, abortCitedButton,


Suggested change

refreshCitedByButton.setOnMouseClicked(event -> searchForRelations(entry, citedByListView, abortCitedButton,

refreshCitedByButton.setOnMouseClicked(_ -> searchForRelations(entry, citedByListView, abortCitedButton,

subhramit · 2025-04-29T11:11:34Z

jabgui/src/main/java/org/jabref/gui/preferences/websearch/WebSearchTab.java

@@ -82,6 +84,25 @@ public void initialize() {
        defaultPlainCitationParser.itemsProperty().bind(viewModel.plainCitationParsers());
        defaultPlainCitationParser.valueProperty().bindBidirectional(viewModel.defaultPlainCitationParserProperty());

+        viewModel.citationsRelationsStoreTTLProperty()
+                 .addListener((observable, oldValue, newValue) -> {


Suggested change

.addListener((observable, oldValue, newValue) -> {

.addListener((_, _, newValue) -> {

subhramit · 2025-04-29T11:11:58Z

jabgui/src/main/java/org/jabref/gui/preferences/websearch/WebSearchTab.java

+                 });
+        citationsRelationStoreTTL
+                .textProperty()
+                .addListener((observable, oldValue, newValue) -> {


Suggested change

.addListener((observable, oldValue, newValue) -> {

.addListener((_, _, newValue) -> {

subhramit · 2025-04-29T11:16:21Z

jablib/src/main/java/org/jabref/logic/citation/SearchCitationsRelationsService.java

+            || !this.relationsRepository.containsReferences(referencer);
+        if (isFetchingAllowed) {
+            try {
+                var references = this.citationFetcher.searchCiting(referencer);


We usually prefer usage of strict types, but since I see 89 occurrences of it in this PR, I think it is better to let it stay and give energy to other important comments left by the senior developers (if any).

I anyway plan to refactor these throughout the codebase in future.

alexandre-cremieux · 2025-04-29T19:19:55Z

I just skimmed through the code, just some formatting suggestions. Others would be doing a thorough review.

Thank you @subhramit . Will take your review in account. However, as you mentioned it, please excuse me if I wait for others to review before making the changes. That way I will solve everything in one pass.

koppor

The serialization and deserialization of BibEntries won't work in all cases. The chosen "canonical representation" is not full BibTeX and there is risk, it cannot be parsed to BibEntry back.

I think, a similar method as org.jabref.gui.ClipBoardManager#serializeEntries and org.jabref.logic.importer.fileformat.BibtexParser#singleFromString.

koppor · 2025-05-02T13:06:31Z

...ib/src/main/java/org/jabref/logic/citation/repository/MVStoreBibEntryRelationRepository.java

+        this.store.close();
+    }
+
+    static class BibEntrySerializer extends BasicDataType<BibEntry> {


I think, this class is wrong - one should use code similar as org.jabref.gui.ClipBoardManager#serializeEntries

Reason: The result is close to BibTeX, but not a valid BibTeX representation in all cases - see documentation of the used org.jabref.model.entry.CanonicalBibEntry#getCanonicalRepresentation

…00-refactor-citation-relation-tab-logic

jabref-machine · 2025-06-05T02:34:17Z

Your pull request needs to link an issue correctly.

To ease organizational workflows, please link this pull-request to the issue with syntax as described in https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue:

Linking a pull request to an issue using a keyword

You can link a pull request to an issue by using a supported keyword in the pull request's description or in a commit message.

Examples

✅ Fixes #xyz links pull-request to issue. Merging the PR will close the issue.
✅ Fixes https://github.com/JabRef/jabref/issues/xyz links pull-request to issue. Merging the PR will close the issue.
✅ Fixes https://github.com/Koppor/jabref/issues/xyz links pull-request to issue. Merging the PR will close the issue.
❌ Fixes [#xyz](https://github.com/JabRef/jabref/issues/xyz) links pull-request to issue. Merging the PR will NOT close the issue.

jabref-machine · 2025-06-05T02:37:18Z

JUnit tests of jablib are failing. You can see which checks are failing by locating the box "Some checks were not successful" on the pull request page. To see the test output, locate "Tests / Unit tests (pull_request)" and click on it.

You can then run these tests in IntelliJ to reproduce the failing tests locally. We offer a quick test running howto in the section Final build system checks in our setup guide.

trag-bot · 2025-06-05T02:38:47Z

jabgui/src/main/java/org/jabref/gui/JabRefGUI.java

@@ -407,6 +415,8 @@ public void stop() {
        stopBackgroundTasks();
        LOGGER.trace("Shutting down thread pools");
        shutdownThreadPools();
+        LOGGER.trace("Closing citations and relations search service");
+        citationsAndRelationsSearchService.close();


The method 'close()' should not be used for normal control flow. Consider using a more descriptive method name that indicates the specific action being performed.

trag-bot · 2025-06-05T02:38:48Z

jabgui/src/main/java/org/jabref/gui/JabRefGUI.java

@@ -66,6 +67,8 @@ public class JabRefGUI extends Application {

    // AI Service handles chat messages etc. Therefore, it is tightly coupled to the GUI.
    private static AiService aiService;
+    // CitationsAndRelationsSearchService is here configured for a local machine and so to the GUI.


The comment is trivial and does not add new information beyond what the code already conveys. Comments should provide additional context or reasoning.

koppor · 2025-06-05T02:39:14Z

The the local storage TTL setting message has been traduced in:

I think, this could be overwritten by crowdin - see https://docs.jabref.org/contributing/how-to-translate-the-ui

I removed the translations for now - maybe, we can bring them back after a merge.

trag-bot · 2025-06-05T02:41:27Z

jabgui/src/main/java/org/jabref/gui/preferences/websearch/WebSearchTab.java

+                        citationsRelationStoreTTL.setText(newValue.replaceAll("\\D", ""));
+                        return;
+                    }
+                    viewModel.citationsRelationsStoreTTLProperty().set(Integer.parseInt(newValue));


Directly parsing the newValue without validation can lead to NumberFormatException. Ensure the input is validated before parsing to prevent runtime errors.

alexandre-cremieux marked this pull request as draft September 28, 2024 20:38

github-actions bot reviewed Sep 28, 2024

View reviewed changes

alexandre-cremieux force-pushed the fix-issue-11189-part-00-refactor-citation-relation-tab-logic branch from 9fe8522 to cbe9e96 Compare September 28, 2024 21:15

github-actions bot reviewed Sep 28, 2024

View reviewed changes

alexandre-cremieux force-pushed the fix-issue-11189-part-00-refactor-citation-relation-tab-logic branch from cbe9e96 to 8231340 Compare September 28, 2024 21:51

github-actions bot reviewed Sep 28, 2024

View reviewed changes

alexandre-cremieux force-pushed the fix-issue-11189-part-00-refactor-citation-relation-tab-logic branch from 8231340 to 33967c2 Compare September 28, 2024 22:20

github-actions bot reviewed Sep 28, 2024

View reviewed changes

alexandre-cremieux force-pushed the fix-issue-11189-part-00-refactor-citation-relation-tab-logic branch from 33967c2 to 592d4d7 Compare September 28, 2024 22:47

koppor changed the title ~~Fix issue 11189 part 00 refactor citation relation tab logic~~ Fix issue #11189 part 00 refactor citation relation tab logic Sep 29, 2024

alexandre-cremieux force-pushed the fix-issue-11189-part-00-refactor-citation-relation-tab-logic branch 2 times, most recently from d94f4d3 to 3155242 Compare September 29, 2024 13:24

alexandre-cremieux commented Sep 29, 2024

View reviewed changes

alexandre-cremieux mentioned this pull request Sep 29, 2024

Implement caching for citation relations #11189

Open

alexandre-cremieux added 3 commits September 29, 2024 17:00

Refactor Citations Relations Tab (JabRef#11189)

3785cb1

* Move repository, cache, and fetcher to logic package * Move citations model to model/citations/semanticscholar package

Refactor Citations Relations Tab (JabRef#11189)

8048da8

* Introduce service layer * Rename LRU cache implementation * Add tests helpers for repository

Refactor Citations Relations Service Layer (JabRef#11189)

18db75e

* Move logic from repository to service * Refactor repositories * Update tab configuration

alexandre-cremieux force-pushed the fix-issue-11189-part-00-refactor-citation-relation-tab-logic branch from 3155242 to 18db75e Compare September 29, 2024 15:01

Merge branch 'main' into fix-issue-11189-part-00-refactor-citation-re…

0e9de3c

…lation-tab-logic # Conflicts: # src/main/java/org/jabref/gui/entryeditor/citationrelationtab/CitationRelationsTab.java

koppor requested changes Oct 10, 2024

View reviewed changes

koppor marked this pull request as ready for review October 10, 2024 20:29

InAnYan requested changes Oct 11, 2024

View reviewed changes

Address PR comments (JabRef#11901)

9a31735

alexandre-cremieux marked this pull request as ready for review April 28, 2025 19:09