Parse MeSH terms in PubMed MEDLINE records

[ryan-carpenter]

Is your suggestion for improvement related to a problem? Please describe.

MEDLINE records are indexed with headings and subheadings (MeSH terms), having a one-to-many relationship between headings and subheadings. PubMed displays the MeSH terms individually (in pairs), like this.

Notice that the heading "Kidney Diseases" repeats for each associated subheading, with trailing asterisks denoting "Major topics". This is not how the MeSH terms appear in PubMed exports, and therefore, not how JabRef imports them.

This is how the terms come in PubMed text files.

MH  - *Kidney Diseases/diagnosis/epidemiology/physiopathology/therapy

JabRef imports this unchanged as one keyword:

*Kidney Diseases/diagnosis/epidemiology/physiopathology/therapy

This is how the terms appear in PubMed xml.

<MeshHeading>
    <DescriptorName UI="D007674" MajorTopicYN="Y">Kidney Diseases</DescriptorName>
    <QualifierName UI="Q000175" MajorTopicYN="N">diagnosis</QualifierName>
    <QualifierName UI="Q000453" MajorTopicYN="N">epidemiology</QualifierName>
    <QualifierName UI="Q000503" MajorTopicYN="N">physiopathology</QualifierName>
    <QualifierName UI="Q000628" MajorTopicYN="N">therapy</QualifierName>
</MeshHeading>

Again, JabRef imports this as one keyword, this time separating the subheadings with a comma:

Kidney Diseases, diagnosis, epidemiology, physiopathology, therapy

Describe the solution you'd like

I would like JabRef to import MeSH terms as individual keywords using the same format as PubMed where each heading has a maximum of one subheading and the major topic is displayed as an asterisk at the end of the heading or subheading string. Keywords generated from plain text or xml files from PubMed should have the same format in JabRef.

The keywords should look like this:
Kidney Diseases*/diagnosis Kidney Diseases*/epidemiology Kidney Diseases*/physiopathology Kidney Diseases*/therapy

The bibtex source should look like this (assuming the user-define keyword separator is a semicolon):

Kidney Diseases*/diagnosis; Kidney Diseases*/epidemiology; Kidney Diseases*/physiopathology; Kidney Diseases*/therapy

Parsing MeSH terms this way lets the keywords fit better in the GUI and makes it easier to search and filter by keyword.

Additional context
Ideally, the MEDLINE importer (and other importers) would check if the user-defined keyword separator is included in the input, and warn or choose a substitution in case of conflict. List items are appear one per line in PubMed text files, so the keyword separator should not be found in any lines that begin with MH - .

jabref/src/main/java/org/jabref/model/entry/KeywordList.java

Line 66 in 3a8cba4

* Parses the keyword list and uses {@link Keyword#DEFAULT_HIERARCHICAL_DELIMITER} as hierarchical delimiter.

Regex for moving asterisks to the end.

(?<slash>/{0,1})\*(?<subhead>.+?(?=/|$))(?<=^MH  - .*)

Replace with

"${slash}${subhead}*

Discussion on JabRef Discourse

[ryan-carpenter]

Edits:

• Removed * from the xml-import example. The MEDLINE importer currently does not mark the major topic.
• Updated suggested import format to Heading/subheading instead of Heading /subheading

Keywords could have the format Heading / subheading, Heading /subheading, or Heading/subheading. I suggest the last one, because this is the format of PubMed text files and provides the greatest compatibility with existing libraries. This is the format JabRef already uses, as long as the heading has only one subheading, so it makes sense to keep this pattern.

[ungerts]

Implemented a prototype to identify potential challenges or problems related to this issue.

I encountered two main problems:

• Inconsistent Keyword Separators in JabRef: JabRef uses different characters as keyword separators depending on the context. For instance, MedlineImporter.class uses the semicolon (;), defined as a constant within the class. Meanwhile, the Preferences configuration uses a comma (,) as the default separator. This inconsistency can lead to confusion or unexpected behavior in standard configurations.

• Comma in MeSH Terms: MeSH terms may contain commas, which conflict with the default keyword separator (,). Currently, there is no mechanism to escape or handle such cases, resulting in the incorrect splitting of MeSH terms.

In my opinion, addressing these inconsistencies is essential to developing a sustainable and robust solution.

[koppor]

"Models, Molecular" seems like a hierarchical keyword? Is this always the case that comma is used for hierarchy?

[ungerts]

Although it might look like a hierarchy, it's not actually one. I experimented a bit with the MeSH database, and treating the comma in Models, Molecular as a hierarchical delimiter doesn’t work. In that interpretation, you’d end up with two hierarchies named Model, which doesn’t make sense.

Here’s the actual hierarchy:

Analytical, Diagnostic and Therapeutic Techniques and Equipment Category
        Investigative Techniques
            Models, Theoretica
                Models, Molecular
                    Molecular Docking Simulation
                    Molecular Dynamics Simulation
                    Pharmacophore

Looking at the hierarchy for Diabetes Mellitus reveals even more complex scenarios.

[ungerts]

It seems there will also be some UI-related issues—the keyword box is too small to handle a large set of keywords.

[koppor]

It seems there will also be some UI-related issues—the keyword box is too small to handle a large set of keywords.

This refs #6856

[koppor]

• Inconsistent Keyword Separators in JabRef: JabRef uses different characters as keyword separators depending on the context. For instance, MedlineImporter.class uses the semicolon (;), defined as a constant within the class. Meanwhile, the Preferences configuration uses a comma (,) as the default separator. This inconsistency can lead to confusion or unexpected behavior in standard configurations.

I deduct:

1. Have the keyword separator config be used in MedlineImporter
2. Have escaping for the keyword separator be implemented, especially
   • org.jabref.gui.fieldeditors.KeywordsEditor
   • org.jabref.model.entry.KeywordList --> filed as Implement escaping for keyword separators #12810

[ryan-carpenter]

"Models, Molecular" seems like a hierarchical keyword? Is this always the case that comma is used for hierarchy?

No, unfortunately, commas do not reflect the hierarchy even though it often appears this way.

Here is an example where the comma looks like part of the hierarchy and where it would be possible to abbreviate the keyword by removing the redundant portion Aged.

Now consider an example where it is clear that commas do not in fact represent the hierarchy. Notice that Models, Animal has the peer Models, Theoretical but neither of these is a child of Models. Similarly, Disease Models, Animal is not a child of Disease Models, and in fact there is no subject in MEDLINE called "Disease Models". Note: there are three hierarchies shown because Disease Models, Animal exists in each of them.

[ryan-carpenter]

It seems there will also be some UI-related issues—the keyword box is too small to handle a large set of keywords.

This is a problem in the current situation as well. MEDLINE terms that have multiple subheadings are too long for the entry editor. Since keywords do not wrap, they get cut off and cannot be seen beyond the right boundary of the field.

[ryan-carpenter]

I deduct:

1. Have the keyword separator config be used in `MedlineImporter`

2. Have escaping for the keyword separator be implemented, especially

Escaping is the only real solution, because the keyword separator comes from the source and not the user. Different sources use different separators, so the user preference alone cannot be relied upon. Substitution is another possibility, but this too would be unreliable since the substitution would have to use a character that does not occur in the source.