[REVIEW]: Chemiscope: interactive structure-property explorer for materials and molecules

[whedon]

Submitting author: @Luthaf (Guillaume Fraux)
Repository: https://github.com/cosmo-epfl/chemiscope
Version: 0.2.0
Editor: @cMadan
Reviewer: @kblin, @mkhorton
Archive: 10.5281/zenodo.3909400

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/fd6f554ee0fbc5ae622a0b3853ff6306"><img src="https://joss.theoj.org/papers/fd6f554ee0fbc5ae622a0b3853ff6306/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/fd6f554ee0fbc5ae622a0b3853ff6306/status.svg)](https://joss.theoj.org/papers/fd6f554ee0fbc5ae622a0b3853ff6306)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@kblin & @mkhorton, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

1. Make sure you're logged in to your GitHub account
2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @cMadan know.

✨ Please try and complete your review in the next two weeks ✨

Review checklist for @kblin

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@Luthaf) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

Review checklist for @mkhorton

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@Luthaf) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[whedon]

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @kblin, @mkhorton it looks like you're currently assigned to review this paper 🎉.

⭐ Important ⭐

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

2. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf

[whedon]

Reference check summary:

OK DOIs

- 10.1073/pnas.1108486108 is OK
- 10.1103/physrevb.87.184115 is OK
- 10.1088/1367-2630/15/9/095003 is OK
- 10.1186/s13321-014-0041-7 is OK
- 10.1093/bioinformatics/btx760 is OK
- 10.1093/bioinformatics/btt270 is OK
- 10.1038/s41467-018-06972-x is OK
- 10.5281/zenodo.3541831 is OK
- 10.1039/c6cp00415f is OK
- 10.1186/s13321-017-0192-4 is OK
- 10.1039/c7sc04665k is OK
- 10.1016/j.commatsci.2019.02.040 is OK
- 10.1063/1.5090481 is OK
- 10.1103/physrevlett.98.146401 is OK
- 10.1021/ci300415d is OK
- 10.1038/sdata.2014.22 is OK
- 10.1103/physrevlett.104.136403 is OK
- 10.1162/089976698300017467 is OK
- 10.1002/anie.200603675 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[whedon]

👉 Check article proof 📄 👈

[mkhorton]

Hi @Luthaf, I'm trying the interactive version on chemiscope.org and the Jmol viewer doesn't seem to be showing anything, looks like it should be an easy fix:

Safari Version 13.0.4 (15608.4.9.1.3), macOS 10.15.2 (19C57), no extensions

[Luthaf]

Thanks for the report @mkhorton, it should be fixed now. If not, feel free to open more issues on chemiscope repo!

[arfon]

Dear authors and reviewers

We wanted to notify you that in light of the current COVID-19 pandemic, JOSS has decided to suspend submission of new manuscripts and to handle existing manuscripts (such as this one) on a "best efforts basis". We understand that you may need to attend to more pressing issues than completing a review or updating a repository in response to a review. If this is the case, a quick note indicating that you need to put a "pause" on your involvement with a review would be appreciated but is not required.

Thanks in advance for your understanding.

Arfon Smith, Editor in Chief, on behalf of the JOSS editorial team.

[arfon]

👋 @kblin & @mkhorton - just a friendly check-in to see how things are going with your reviews?

[Luthaf]

@whedon generate pdf

[whedon]

👉 Check article proof 📄 👈

[mkhorton]

Hi @arfon, I can't seem to check-off items on the review?

[arfon]

@whedon re-invite @mkhorton as reviewer

[whedon]

I'm sorry human, I don't understand that. You can see what commands I support by typing:

@whedon commands

[arfon]

@whedon re-invite @mkhorton as reviewer

[whedon]

The reviewer already has a pending invite.

@mkhorton please accept the invite by clicking this link: https://github.com/openjournals/joss-reviews/invitations

[arfon]

@mkhorton it looks like you need to accept the invitation to the repo? ☝️

[mkhorton]

@arfon ah, thanks -- I thought I'd accepted for a previous review. I found the link but it says invite has expired.

[arfon]

@whedon re-invite @mkhorton as reviewer

[whedon]

OK, the reviewer has been re-invited.

@mkhorton please accept the invite by clicking this link: https://github.com/openjournals/joss-reviews/invitations

[arfon]

@arfon ah, thanks -- I thought I'd accepted for a previous review. I found the link but it says invite has expired.

OK, could you try again now? Sorry about the disruption, it seems like GitHub expires repository invitations these days after a certain period.

[mkhorton]

Great, got it now. Thanks :)

[mkhorton]

Hi @Luthaf, first my sincere apologies in the delay in making significant progress on this review. The code looks excellent, very well structured, and fills a clear need (indeed, we work on similar problems ourselves, and it's been a common source of frustration that all these nice global similarity measures exist, but are pretty inaccessible to the average user).

I have two queries:

1. Are there any automated tests? I couldn't find any.

2. Could you improve the paragraph for people looking to contribute and/or where to find support, just a minor point.

[Luthaf]

Are there any automated tests? I couldn't find any.

No, unfortunately, I could not find a nice way to do automated tests for web user interfaces. If you know of a good framework for this, I am interested. I'd like to add such tests at some point.

In the mean time, I have been using the example files (which you can get locally with npm run download-example-input) to test the behaviour of the code on varied examples.

Could you improve the paragraph for people looking to contribute and/or where to find support, just a minor point.

Sure!

[mkhorton]

If you know of a good framework for this, I am interested. I'd like to add such tests at some point.

Yes, there are many test frameworks available. It is a bit of a pain to write tests for web apps, but it's worth it. Even minimal tests that just check your server loads and displays a graph and molecule can be valuable if, for example, something like a dependency being updated breaks your app.

Some frameworks include Jest for testing components, browser testing like Selenium or visual diffing like Percy (commercial).

Automated testing is part of the requirements for JOSS paper acceptance.

[Luthaf]

Yes, there are many test frameworks available. It is a bit of a pain to write tests for web apps, but it's worth it. Even minimal tests that just check your server loads and displays a graph and molecule can be valuable if, for example, something like a dependency being updated breaks your app.

Some frameworks include Jest for testing components, browser testing like Selenium or visual diffing like Percy (commercial).

Percy sounds interesting, but I would rather use something free and open source, even if the setup is a bit more complex. However I disagree that minimal tests are useful: when updating dependencies I will be manually checking that basics still work. IMO, more interesting tests would be workflow/user story tests (when I click on this element, this other element changes, and this modal opens); do you know a framework for that?

Automated testing is part of the requirements for JOSS paper acceptance.

From JOSS documentation

Tests

Authors are strongly encouraged to include an automated test suite covering the core functionality of their software.

Good: An automated test suite hooked up to an external service such as Travis-CI or similar
OK: Documented manual steps that can be followed to objectively check the expected functionality of the software (e.g., a sample input file to assert behavior)
Bad (not acceptable): No way for you the reviewer to objectively assess whether the software works

I hoped that chemiscope would fell in the OK category here, even if it is not in the Good one.

[mkhorton]

I hoped that chemiscope would fell in the OK category here, even if it is not in the Good one.

Yes, I agree, on the OK criteria I can check it off.

In my role as a reviewer though, I do want to echo the JOSS guidelines and strongly encourage tests. Tests in JavaScript are definitely possible, and there are open source solutions (Jest is fully open source). From my own past experience, writing web apps is a lot more brittle than other kinds of apps precisely because they tend to have very deep dependency trees and lots of small packages.

when updating dependencies I will be manually checking that basics still work.

The intent here is for other contributors to the project too, and help them, help them know when they break things and how to add new features that don't conflict with requirements for the project. Tests help communicate your intent for what you want the code to do.

Your call though -- I agree that this meets the minimal requirements, so I won't hold up the review over it :)

[Luthaf]

We've made some tiny phrasing changes & improved the use of descriptor vs representation: lab-cosmo/chemiscope@8c11a56, I hope it is still possible to do it now!

[danielskatz]

As AEiC on duty this week, I'll read this in the next few hours and either proceed with acceptance or provide feedback.

[danielskatz]

Please see lab-cosmo/chemiscope#55

Also, is "Additional" in the first paragraph needed?
And the caption of figure 2 seems to end abruptly.

[Luthaf]

Thanks for the feedback, I hope I addressed your comments in lab-cosmo/chemiscope@a982842

[danielskatz]

@whedon accept

[whedon]

Attempting dry run of processing paper acceptance...

[whedon]

Reference check summary:

OK DOIs

- 10.1073/pnas.1108486108 is OK
- 10.1103/physrevb.87.184115 is OK
- 10.1088/1367-2630/15/9/095003 is OK
- 10.1186/s13321-014-0041-7 is OK
- 10.1093/bioinformatics/btx760 is OK
- 10.1093/bioinformatics/btt270 is OK
- 10.1038/s41467-018-06972-x is OK
- 10.5281/zenodo.3541831 is OK
- 10.1039/c6cp00415f is OK
- 10.1186/s13321-017-0192-4 is OK
- 10.1039/c7sc04665k is OK
- 10.1016/j.commatsci.2019.02.040 is OK
- 10.1063/1.5090481 is OK
- 10.1103/physrevlett.98.146401 is OK
- 10.1021/ci300415d is OK
- 10.1038/sdata.2014.22 is OK
- 10.1103/physrevlett.104.136403 is OK
- 10.1162/089976698300017467 is OK
- 10.1002/anie.200603675 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[whedon]

👋 @openjournals/joss-eics, this paper is ready to be accepted and published.

Check final proof 👉 openjournals/joss-papers#1528

If the paper PDF and Crossref deposit XML look good in openjournals/joss-papers#1528, then you can now move forward with accepting the submission by compiling again with the flag deposit=true e.g.

@whedon accept deposit=true

[danielskatz]

@Luthaf - please change the metadata (title and authors) in the zenodo archive to match the paper

[Luthaf]

@danielskatz should be good now, let me know if something is missing

[danielskatz]

It seems like you added a subtitle rather than changing the title?

[Luthaf]

Sorry, it is better now?

[danielskatz]

looks good - thanks

[danielskatz]

@whedon accept deposit=true

[whedon]

Doing it live! Attempting automated processing of paper acceptance...

[whedon]

🐦🐦🐦 👉 Tweet for this paper 👈 🐦🐦🐦

[whedon]

🚨🚨🚨 THIS IS NOT A DRILL, YOU HAVE JUST ACCEPTED A PAPER INTO JOSS! 🚨🚨🚨

Here's what you must now do:

0. Check final PDF and Crossref metadata that was deposited 👉 Creating pull request for 10.21105.joss.02117 joss-papers#1530
1. Wait a couple of minutes to verify that the paper DOI resolves https://doi.org/10.21105/joss.02117
2. If everything looks good, then close this review issue.
3. Party like you just published a paper! 🎉🌈🦄💃👻🤘

Any issues? Notify your editorial technical team...

[danielskatz]

Thanks to @kblin & @mkhorton for reviewing!
And @cMadan for editing!

Congratulations to @Luthaf (Guillaume Fraux) and co-authors!

[danielskatz]

@arfon - this doesn't seem to have had the normal final message added with the badge link etc...

[Luthaf]

Thanks everyone involved here, the discussions have been very interesting!

[whedon]

🎉🎉🎉 Congratulations on your paper acceptance! 🎉🎉🎉

If you would like to include a link to your paper from your README use the following code snippets:

Markdown:
[![DOI](https://joss.theoj.org/papers/10.21105/joss.02117/status.svg)](https://doi.org/10.21105/joss.02117)

HTML:
<a style="border-width:0" href="https://doi.org/10.21105/joss.02117">
  <img src="https://joss.theoj.org/papers/10.21105/joss.02117/status.svg" alt="DOI badge" >
</a>

reStructuredText:
.. image:: https://joss.theoj.org/papers/10.21105/joss.02117/status.svg
   :target: https://doi.org/10.21105/joss.02117

This is how it will look in your documentation:

We need your help!

Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

• Volunteering to review for us sometime in the future. You can add your name to the reviewer list here: https://joss.theoj.org/reviewer-signup.html
• Making a small donation to support our running costs here: https://numfocus.org/donate-to-joss

[arfon]

@arfon - this doesn't seem to have had the normal final message added with the badge link etc...

Seems like there's some inconsistency with GitHub webhooks being sent out today.