[REVIEW]: pyCeterisParibus: explaining Machine Learning models with Ceteris Paribus Profiles in Python

[whedon]

Submitting author: @kmichael08 (Michał Kuźba)
Repository: https://github.com/ModelOriented/pyCeterisParibus
Version: v0.5.2
Editor: @katyhuff
Reviewer: @janfreyberg, @JustinShenk
Archive: 10.5281/zenodo.2667756

Status

Status badge code:

HTML: <a href="http://joss.theoj.org/papers/aad9a21c61c01adebe11bc5bc1ceca92"><img src="http://joss.theoj.org/papers/aad9a21c61c01adebe11bc5bc1ceca92/status.svg"></a>
Markdown: [![status](http://joss.theoj.org/papers/aad9a21c61c01adebe11bc5bc1ceca92/status.svg)](http://joss.theoj.org/papers/aad9a21c61c01adebe11bc5bc1ceca92)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@janfreyberg & @JustinShenk, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

1. Make sure you're logged in to your GitHub account
2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.theoj.org/about#reviewer_guidelines. Any questions/concerns please let @katyhuff know.

✨ Please try and complete your review in the next two weeks ✨

Review checklist for @janfreyberg

Conflict of interest

• As the reviewer I confirm that I have read the JOSS conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Version: v0.5.2
• Authorship: Has the submitting author (@kmichael08) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Authors: Does the paper.md file include a list of authors with their affiliations?
• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

Review checklist for @JustinShenk

Conflict of interest

• As the reviewer I confirm that I have read the JOSS conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Version: v0.5.2
• Authorship: Has the submitting author (@kmichael08) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Authors: Does the paper.md file include a list of authors with their affiliations?
• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

[whedon]

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @janfreyberg, it looks like you're currently assigned as the reviewer for this paper 🎉.

⭐ Important ⭐

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

2. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

For a list of things I can do to help you, just type:

@whedon commands

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

👉 Check article proof 📄 👈

[JustinShenk]

Overview: pyCeterisParibus is a library for explaining machine learning models with ceteris paribus profiles. These are useful for adding to visual story telling and supporting model interpretability. The idea is great, the implementation is clean, and I may use it in some of my projects. Some minor improvements are suggested below.

Installation: Installed without issues via pip and local copy of the source code.

Functionality: I was able to run the example on my Mac, but was not able to load the plot, due to an issue with how file paths are handled on Macs. I have opened a pull request at ModelOriented/pyCeterisParibus#24 fixing this issue on my machine. After this is accepted or otherwise addressed I will consider it completed.
I opened an issue (ModelOriented/pyCeterisParibus#23) regarding the scrollbars obscuring the data. This could be fixed by adding additional padding to the bottom of the frame.

Performance: No measure of performance is given, but the model loaded fast on the Titanic dataset.

Documentation: The explanation for how the model works could be improved. For example, in the paper the author's write "For this purpose, methods for sampling and selecting neighbouring observations are implemented along with the Gower's distance [@gower] function. A more detailed description might be found in the package documentation." I was not able to find description of Gower's distance in the linked to readthedocs. Adding details of how the model works would be helpful for people who are not familiar with Gower's distance or how it applies to machine learning models.

Software Paper: The software paper has a few minor typos or questionable stylistic choices for an academic paper:

• "He died on [the] Titanic"
• "1. [rather first or 1st] class"
• BUT. Not really a typo, but stylistically questionable to have a period after "BUT". Would be more fitting of an academic journal without the word.
• Local [I]nterpretable [M]odel-agnostic [E]xplanations (LIME)
• Capitalization of "machine learning"
• "(e.g.[,] a bank)"
• "black-boxes" [no space needed]
• "leads to the next industrial revolution" - unverified claim

Example Usage: The notebooks and example scripts without problems.

References: Every reference mentioned in the paper is documented as BibTex entries.

[kmichael08]

@JustinShenk Great thanks for all these valuable remarks! I merged your pull request. Also, I applied your comments referred to the paper and put the Gower's distance description in the documentation. I'll solve the scrollbar problem (ModelOriented/pyCeterisParibus#23) as fast as possible.

[kmichael08]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

👉 Check article proof 📄 👈

[JustinShenk]

@kmichael08 Thanks for the quick response and edits.

@katyhuff My review (#1389 (comment)) is now complete.

[janfreyberg]

Installation

Installing the package in a fresh docker alpine image leads to dependencies being installed that I don't think need to be, for example sphinx, m2r, codecov, etc.

There are a few ways around this so I haven't made a PR but I think you can do the following:

Split requirements into actual requirements (what's needed to run the package), documentation requirements, and test requirements (e.g. pytest). You can add these as additional requirements using the extras_require key in setup.py, or simply install them from txt files wherever you need them.

Additionally, as far as I can tell tensorflow is never imported and so should be removed from the requirements.

I would even go so far as to say XGBoost and sklearn should not be in the requirements, even though you use it in the paper and documentation, becuase it's not essential to the functioning of the package. Instead, you could make a note that people should install them to run the examples.

Otherwise, installation works great.

Functionality / Performance

This all worked great for me.

Documentation

I think the docs can be improved:

• the index page should be clearer; I would remove the automated sphinx content and add more of a "landing" page that contains an introduction
• I would consider adding the jupyter notebooks to the docs using a tool like sphinx-jupyter

But that's just a recommendation.

Paper

The paper is very good. Only point: the R package CeterisParibus is not included in the references.

[katyhuff]

Thanks for the speedy reviews, @janfreyberg @JustinShenk . And, thanks for responding quickly to the suggestions @kmichael08 .

@kmichael08 , there are a few items in @janfreyberg's review that will need to be handled before we should move forward with acceptance:

• Package installation instructions should be cleaned up to remove tensorflow if it's not used anywhere.
• Though I'm not sure the best citation for it, I do agree with @janfreyberg that the paper would benefit from a citation to the CeterisParibus R package.

The rest of the comments from @janfreyberg would certainly clean things up, but aren't explicitly need for our JOSS requirements, so I'll just recommend that you consider the recommendation from @janfreyberg : "Split requirements into actual requirements (what's needed to run the package), documentation requirements, and test requirements (e.g. pytest). You can add these as additional requirements using the extras_require key in setup.py, or simply install them from txt files wherever you need them.... I would even go so far as to say XGBoost and sklearn should not be in the requirements, even though you use it in the paper and documentation, becuase it's not essential to the functioning of the package. Instead, you could make a note that people should install them to run the examples."

I have looked over the package and have found it installs pretty easily. Once you've seen this message @kmichael08 and implemented the two above changes, please ping me and we'll move on with next steps.

[kmichael08]

@whedon generate pdf