[REVIEW]: Geometric nested sampling

[whedon]

Submitting author: @SuperKam91 (Kamran Javid)
Repository: https://github.com/SuperKam91/gns
Version: v1.01
Editor: @mbobra
Reviewer: @ziotom78, @zhampel
Archive: 10.5281/zenodo.3569956

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/34c9c9b4b15a502199f0b4c68ec7e48b"><img src="https://joss.theoj.org/papers/34c9c9b4b15a502199f0b4c68ec7e48b/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/34c9c9b4b15a502199f0b4c68ec7e48b/status.svg)](https://joss.theoj.org/papers/34c9c9b4b15a502199f0b4c68ec7e48b)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@ziotom78 & @zhampel, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

1. Make sure you're logged in to your GitHub account
2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @mbobra know.

✨ Please try and complete your review in the next two weeks ✨

Review checklist for @ziotom78

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@SuperKam91) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

Review checklist for @zhampel

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@SuperKam91) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[whedon]

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @ziotom78, @zhampel it looks like you're currently assigned to review this paper 🎉.

⭐ Important ⭐

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

2. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

👉 Check article proof 📄 👈

[mbobra]

👋@ziotom78 @zhampel Thank you again for volunteering to review. The review instructions are listed above. Please let me know if you have any questions!

[ziotom78]

I am still completing the review of the code, but I started reading the paper as well. One thing I note is Fig. 2, which has two "X" marks that are explained neither in the caption nor in the text. The figure is obviously Fig. 3 in the arXiv paper 1905.09110; however, in the arXiv paper Fig. 3 makes sense because it is better explained and placed alongside Fig. 2. I believe that you should include the same figure and the same level of detail in this paper as well.

If the paper length is an issue, you could scale down Fig. 2 in the JOSS paper a bit: the information content is very small (a sphere, three points, and three labels), but the space it takes is really huge.

[SuperKam91]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

👉 Check article proof 📄 👈

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

👉 Check article proof 📄 👈

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

👉 Check article proof 📄 👈

[zhampel]

@SuperKam91 As I begin my review, can you please include comprehensive guidelines for contributing or reporting issues to your codebase? Here's an example.

[SuperKam91]

I am still completing the review of the code, but I started reading the paper as well. One thing I note is Fig. 2, which has two "X" marks that are explained neither in the caption nor in the text. The figure is obviously Fig. 3 in the arXiv paper 1905.09110; however, in the arXiv paper Fig. 3 makes sense because it is better explained and placed alongside Fig. 2. I believe that you should include the same figure and the same level of detail in this paper as well.

If the paper length is an issue, you could scale down Fig. 2 in the JOSS paper a bit: the information content is very small (a sphere, three points, and three labels), but the space it takes is really huge.

Hi @ziotom78 thank you for your comments on the paper. While I gave some information on the coordinate labels in the caption of Figure 2 (n.b. captions should be visible in the .pdfs generated in this thread, but probably not in the .md file itself), I have added a couple of sentences in the main text elaborating more on them, at end of the main body of text. With regards to the figure sizes, I have tried to reduce them using different syntaxes to the one suggested in the JOSS paper.md example. These worked in the .md files, but not in the .pdf generated here, and so I have had to revert back to the original sizing. It would be very useful if the JOSS template provided an example of how to scale images which is compatible with the pdf generation used here.

[SuperKam91]

Hi @zhampel , I have added a section explaining how to contribute to the project/ report issues at the bottom of the README of the repo.

[ziotom78]

As issue #3 is now closed, I think that the paper and the package are ok, but let's wait for @zhampel's approval too. The functionality provided by the package looks good, nice work!

However, there are a few suggestions I would like to give to @SuperKam91 :

• The API documentation is ok, but it is surely not «good» according to JOSS' guidelines:

  • It is just a list of functions provided by the package, with no example for any of them. Compare this with the documentation of the Python's standard library (e.g., the datetime module): short code snippet explain how each function is meant to be used.

  • As the list of functions provided by the package is very long, some text introducing each module (e.g., gns.keeton_calculations) would surely have helped interested readers. Again, Python's documentation is a good example to be used as reference.

• The README is overly long, and it provides long textual descriptions of how to use the library, when short Python snippets would have provided the same information in a much clearer form.

• The review process was quite long because there were trivial errors in the source code that should have been spotted by the author.

In the future, please read carefully Author and Reviewer Guides before submitting a paper to JOSS, as this will ensure a much faster and smoother review process.

[mbobra]

👋 @zhampel Let me know if you have any questions or need any help completing your review!

[zhampel]

@SuperKam91
I agree with @ziotom78 as well. Your submission's functionality is as advertised, but the documentation is lacking in detail. For example, while not explicitly required, function input types and descriptors provide users context for implementation. This is also reflected in the difficulty in scanning the example code in the README, which I would not characterize as a usage guide.

With respect to testing, it doesn't appear that the scripts in the test directory strictly qualify as unit tests, specifically the lack of assertion type tests. This would also be facilitated or at least clarified by care detailing of input argument types and type-checks as appropriate. This is also evident in abundant commented code ending with #debug or similar. For example on line 120 of nested_run.py:

#elif setupDict['sampler'] == 'MH' or setupDict['sampler'] == 'MH geom': #temporary whilst testing geometric ns

If such lines are intended for debugging or testing, then the preference would be to add in some testing capability or flags to ensure such tests are performed with requisite feedback.

Furthermore, I think the paper and docs still have some errors. Examples:

• on page 3: "For example, parameters which... the trial distribution..."
  The sentence is not complete as 'For example' and 'the trial distribution' do not relate properly.
  You can simply state 'periodic properties' instead of 'circular or periodic'
• In the 4th paragraph of the online docs: "Note for sampling spherical sampling, two physical parameters..."
• You also use quotes frequently when they are not necessary.

I recommend that the written components of the submission be re-reviewed and corrected, as well as addressing the lack of clear comments as well as copious commented code within the main codebase. I also highly recommend use of a style guide for Python code such as pep8.

[SuperKam91]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

👉 Check article proof 📄 👈

[arfon]

@SuperKam91 - this PR will fix your paper compilation: SuperKam91/gns#6

[SuperKam91]

@SuperKam91 - this PR will fix your paper compilation: SuperKam91/gns#6

So sorry about that @arfon, don't know how that occurred!

[SuperKam91]

@whedon generate pdf

[whedon]

👉 Check article proof 📄 👈

[whedon]

Reference check summary:

OK DOIs

- 10.1080/00031305.1995.10476177 is OK
- 10.1093/mnras/sty3115 is OK
- 10.17863/CAM.38865 is OK
- 10.1093/mnras/stz826 is OK
- 10.5281/zenodo.3627542 is OK
- 10.17863/CAM.40616 is OK
- 10.1093/mnras/stz2341 is OK
- 10.1111/j.1365-2966.2009.15247.x is OK
- 10.1111/j.1365-2966.2009.14548.x is OK
- 10.1214/06-BA127 is OK
- 10.1093/mnras/stv1911 is OK
- 10.1093/mnrasl/slv047 is OK
- 10.1111/j.1365-2966.2007.12353.x is OK
- 10.1093/mnras/sty074 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[Kevin-Mattheus-Moerman]

@SuperKam91 thanks for making those changes. Looks like we are all set now.

[Kevin-Mattheus-Moerman]

@whedon accept

[whedon]

Attempting dry run of processing paper acceptance...

[whedon]

Reference check summary:

OK DOIs

- 10.1080/00031305.1995.10476177 is OK
- 10.1093/mnras/sty3115 is OK
- 10.17863/CAM.38865 is OK
- 10.1093/mnras/stz826 is OK
- 10.5281/zenodo.3627542 is OK
- 10.17863/CAM.40616 is OK
- 10.1093/mnras/stz2341 is OK
- 10.1111/j.1365-2966.2009.15247.x is OK
- 10.1111/j.1365-2966.2009.14548.x is OK
- 10.1214/06-BA127 is OK
- 10.1093/mnras/stv1911 is OK
- 10.1093/mnrasl/slv047 is OK
- 10.1111/j.1365-2966.2007.12353.x is OK
- 10.1093/mnras/sty074 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[whedon]

Check final proof 👉 openjournals/joss-papers#1261

If the paper PDF and Crossref deposit XML look good in openjournals/joss-papers#1261, then you can now move forward with accepting the submission by compiling again with the flag deposit=true e.g.

@whedon accept deposit=true

[Kevin-Mattheus-Moerman]

@whedon accept deposit=true

[whedon]

Doing it live! Attempting automated processing of paper acceptance...

[whedon]

🐦🐦🐦 👉 Tweet for this paper 👈 🐦🐦🐦

[whedon]

🚨🚨🚨 THIS IS NOT A DRILL, YOU HAVE JUST ACCEPTED A PAPER INTO JOSS! 🚨🚨🚨

Here's what you must now do:

0. Check final PDF and Crossref metadata that was deposited 👉 Creating pull request for 10.21105.joss.01809 joss-papers#1262
1. Wait a couple of minutes to verify that the paper DOI resolves https://doi.org/10.21105/joss.01809
2. If everything looks good, then close this review issue.
3. Party like you just published a paper! 🎉🌈🦄💃👻🤘

Any issues? notify your editorial technical team...

[whedon]

🎉🎉🎉 Congratulations on your paper acceptance! 🎉🎉🎉

If you would like to include a link to your paper from your README use the following code snippets:

Markdown:
[![DOI](https://joss.theoj.org/papers/10.21105/joss.01809/status.svg)](https://doi.org/10.21105/joss.01809)

HTML:
<a style="border-width:0" href="https://doi.org/10.21105/joss.01809">
  <img src="https://joss.theoj.org/papers/10.21105/joss.01809/status.svg" alt="DOI badge" >
</a>

reStructuredText:
.. image:: https://joss.theoj.org/papers/10.21105/joss.01809/status.svg
   :target: https://doi.org/10.21105/joss.01809

This is how it will look in your documentation:

We need your help!

Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

• Volunteering to review for us sometime in the future. You can add your name to the reviewer list here: https://joss.theoj.org/reviewer-signup.html
• Making a small donation to support our running costs here: https://numfocus.org/donate-to-joss

[SuperKam91]

Many thanks all for your help, I have made a small donation to JOSS in the hope that you all can keep up the good work in the future.

I have noticed a small typo in the references of the paper. Is it possible to update the publication now I have ratified it?

If not absolutely no worries, I will just include it in the arXiv version instead.

[SuperKam91]

@whedon check references

[whedon]

Reference check summary:

OK DOIs

- 10.1080/00031305.1995.10476177 is OK
- 10.1093/mnras/sty3115 is OK
- 10.17863/CAM.38865 is OK
- 10.1093/mnras/stz826 is OK
- 10.5281/zenodo.3627542 is OK
- 10.17863/CAM.40616 is OK
- 10.1093/mnras/stz2341 is OK
- 10.1111/j.1365-2966.2009.15247.x is OK
- 10.1111/j.1365-2966.2009.14548.x is OK
- 10.1214/06-BA127 is OK
- 10.1093/mnras/stv1911 is OK
- 10.1093/mnrasl/slv047 is OK
- 10.1111/j.1365-2966.2007.12353.x is OK
- 10.1093/mnras/sty074 is OK

MISSING DOIs

- None

INVALID DOIs

- https://doi.org/10.1093/mnras/stz2341 is INVALID because of 'https://doi.org/' prefix

[SuperKam91]

@whedon check references

[whedon]

Reference check summary:

OK DOIs

- 10.1080/00031305.1995.10476177 is OK
- 10.1093/mnras/sty3115 is OK
- 10.17863/CAM.38865 is OK
- 10.1093/mnras/stz826 is OK
- 10.5281/zenodo.3627542 is OK
- 10.17863/CAM.40616 is OK
- 10.1093/mnras/stz2341 is OK
- 10.1111/j.1365-2966.2009.15247.x is OK
- 10.1111/j.1365-2966.2009.14548.x is OK
- 10.1214/06-BA127 is OK
- 10.1093/mnras/stv1911 is OK
- 10.1093/mnrasl/slv047 is OK
- 10.1111/j.1365-2966.2007.12353.x is OK
- 10.1093/mnras/sty074 is OK

MISSING DOIs

- None

INVALID DOIs

- https://doi.org/10.1093/mnras/stz2341 is INVALID because of 'https://doi.org/' prefix

[arfon]

Many thanks all for your help, I have made a small donation to JOSS in the hope that you all can keep up the good work in the future.

❤️ Many thanks!

I have noticed a small typo in the references of the paper. Is it possible to update the publication now I have ratified it?

Yes, please go ahead and make the fix in the master branch of your repository and mention me here (@arfon) when you've done this.

[SuperKam91]

@whedon check references

[whedon]

Reference check summary:

OK DOIs

- 10.1080/00031305.1995.10476177 is OK
- 10.1093/mnras/sty3115 is OK
- 10.17863/CAM.38865 is OK
- 10.1093/mnras/stz826 is OK
- 10.5281/zenodo.3627542 is OK
- 10.17863/CAM.40616 is OK
- 10.1093/mnras/stz2341 is OK
- 10.1111/j.1365-2966.2009.15247.x is OK
- 10.1111/j.1365-2966.2009.14548.x is OK
- 10.1214/06-BA127 is OK
- 10.1093/mnras/stv1911 is OK
- 10.1093/mnrasl/slv047 is OK
- 10.1111/j.1365-2966.2007.12353.x is OK
- 10.1093/mnras/sty074 is OK

MISSING DOIs

- None

INVALID DOIs

- https://doi.org/10.1093/mnras/stz2341 is INVALID because of 'https://doi.org/' prefix

[SuperKam91]

Many thanks all for your help, I have made a small donation to JOSS in the hope that you all can keep up the good work in the future.

Many thanks!

I have noticed a small typo in the references of the paper. Is it possible to update the publication now I have ratified it?

Yes, please go ahead and make the fix in the master branch of your repository and mention me here (@arfon) when you've done this.

Hi @arfon that's great, thank you so much. I have made the changes to master branch so we should be good to go. However, when "checking references" above it seems to have cached an old commit (perhaps I didn't wait long enough but I don't think this is usually an issue) which doesn't include the fix, so let me know if you have any problems with this.

[arfon]

OK, this should be fixed now. The PDF might take a few hours to show up as fixed online due to caching.

Also, I think you're right, there's some kind of caching issue with Whedon too that I need to look into.

[kthyng]

@arfon I've been having troubles with this too, like in #1656. I'll check it soon and it's probably now worked out but it gets confusing!