Skip to content

[PRE REVIEW]: calzone: A Python package for measuring calibration of probabilistic models for classification #7598

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
editorialbot opened this issue Dec 17, 2024 · 62 comments
Closed

[PRE REVIEW]: calzone: A Python package for measuring calibration of probabilistic models for classification #7598

editorialbot opened this issue Dec 17, 2024 · 62 comments
Assignees
@fabian-s @MrShoenel @pescap @patRyserWelch8
Labels
pre-review Python TeX Track: 5 (DSAIS) Data Science, Artificial Intelligence, and Machine Learning

Comments

@editorialbot
Copy link
Collaborator

editorialbot commented Dec 17, 2024 •
edited
Loading

Submitting author: @qiancao (Qian Cao)
Repository: https://github.com/DIDSR/calzone/
Branch with paper.md (empty if default branch): main
Version: 0.1.0
Editor: @fabian-s
Reviewers: @MrShoenel, @pescap, @patRyserWelch8
Managing EiC: Chris Vernon

Status

status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/fc9232bcdbc721957ec8564fdddb8671"><img src="https://joss.theoj.org/papers/fc9232bcdbc721957ec8564fdddb8671/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/fc9232bcdbc721957ec8564fdddb8671/status.svg)](https://joss.theoj.org/papers/fc9232bcdbc721957ec8564fdddb8671)

Author instructions

Thanks for submitting your paper to JOSS @qiancao. Currently, there isn't a JOSS editor assigned to your paper.

@qiancao if you have any suggestions for potential reviewers then please mention them here in this thread (without tagging them with an @). You can search the list of people that have already agreed to review and may be suitable for this submission.

Editor instructions

The JOSS submission bot @editorialbot is here to help you find and assign reviewers and start the main review. To find out what @editorialbot can do for you type:

@editorialbot commands
@editorialbot editorialbot added pre-review Track: 5 (DSAIS) Data Science, Artificial Intelligence, and Machine Learning labels Dec 17, 2024
@editorialbot
Copy link
Collaborator Author

Hello human, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf

@editorialbot
Copy link
Collaborator Author

Software report:

github.com/AlDanial/cloc v 1.90  T=0.14 s (812.8 files/s, 386204.5 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
CSV                              3              0              0          20003
TeX                             28           1561           2628           8983
HTML                            23            857             69           7308
SVG                              3              0              0           2689
Python                          10            354            638           1246
CSS                              6            223             57           1015
JavaScript                      12            138            228            912
Jupyter Notebook                14              0           3530            504
Markdown                         2             64              0            261
YAML                             3             12             20             66
make                             2             22             13             50
DOS Batch                        2             17              2             47
reStructuredText                 4             32             63             39
TOML                             1              5              0             38
-------------------------------------------------------------------------------
SUM:                           113           3285           7248          43161
-------------------------------------------------------------------------------

Commit count by author:

    90	jasonfan1997
    30	Qian Cao
    12	qiancao
     4	Kwok Lung(Jason) Fan
     1	Jason Fan

@editorialbot
Copy link
Collaborator Author

Checking the BibTeX entries failed with the following error:

Failed to parse BibTeX entry: cite-key missing

@editorialbot
Copy link
Collaborator Author

Paper file info:

📄 Wordcount for paper.md is 2737

✅ The paper includes a Statement of need section

@editorialbot
Copy link
Collaborator Author

License info:

✅ License found: MIT License (Valid open source OSI approved license)

@editorialbot
Copy link
Collaborator Author

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

@editorialbot
Copy link
Collaborator Author

Five most similar historical JOSS papers:

dsBinVal: Conducting distributed ROC analysis using DataSHIELD
Submitting author: @schalkdaniel
Handling editor: @csoneson (Active)
Reviewers: @patRyserWelch8, @brunomontezano, @AnthonyOfSeattle
Similarity score: 0.6381

metrica: an R package to evaluate prediction performance of regression and classification point-forecast models
Submitting author: @adriancorrendo
Handling editor: @osorensen (Active)
Reviewers: @wiljnich, @simonpcouch
Similarity score: 0.6351

modeLLtest: An R Package for Unbiased Model Comparison using Cross Validation
Submitting author: @ShanaScogin
Handling editor: @usethedata (Retired)
Reviewers: @andrewheiss, @usethedata
Similarity score: 0.6166

hopit: An R Package for Analysis of Reporting Behavior Using Generalized Ordered Probit Models
Submitting author: @MaciejDanko
Handling editor: @alexhanna (Retired)
Reviewers: @soodoku, @mkearney
Similarity score: 0.6097

Metrics As Scores: A Tool- and Analysis Suite and Interactive Application for Exploring Context-Dependent Distributions
Submitting author: @mrshoenel
Handling editor: @mikldk (Retired)
Reviewers: @mdhaber, @kostiantyn-kucher
Similarity score: 0.6088

⚠️ Note to editors: If these papers look like they might be a good match, click through to the review issue for that paper and invite one or more of the authors before considering asking the reviewers of these papers to review again for JOSS.

@jasonfan1997
Copy link

@editorialbot check references

@editorialbot
Copy link
Collaborator Author 

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

✅ OK DOIs

- 10.1002/sim.5941 is OK
- 10.1609/aaai.v29i1.9602 is OK
- 10.1175/WAF993.1 is OK
- 10.1002/sim.8281 is OK
- 10.1016/j.acra.2008.04.022 is OK
- 10.1080/03610928008827941 is OK
- 10.1002/qj.456 is OK
- 10.1002/9781118445112.stat08078 is OK
- 10.1177/0962280216661371 is OK
- 10.1016/0895-4356(92)90192-P is OK
- 10.1093/biomet/45.3-4.562 is OK
- 10.1093/biostatistics/kxq045 is OK
- 10.1080/03610928008827941 is OK
- 10.1093/jamia/ocz228 is OK

🟡 SKIP DOIs

- No DOI given, and none found for title: Metrics of Calibration for Probabilistic Predictio...
- No DOI given, and none found for title: Applied logistic regression
- No DOI given, and none found for title: MAPIE: an open-source library for distribution-fre...
- No DOI given, and none found for title: Uncertainty Toolbox: an Open-Source Library for As...
- No DOI given, and none found for title: Scikit-learn: Machine learning in Python
- No DOI given, and none found for title: On Calibration of Modern Neural Networks
- No DOI given, and none found for title: Measuring Calibration in Deep Learning.
- No DOI given, and none found for title: Posterior Re-calibration for Imbalanced Datasets

❌ MISSING DOIs

- 10.2307/2276774 may be a valid DOI for title: Probable inference, the law of succession, and sta...
- 10.2307/2529131 may be a valid DOI for title: Maximum Likelihood Estimation for the Beta-Binomia...
- 10.2307/2346866 may be a valid DOI for title: Reliability of Subjective Probability Forecasts of...
- 10.21236/ada454828 may be a valid DOI for title: Strictly proper scoring rules, prediction, and est...
- 10.1201/9781439891148-8 may be a valid DOI for title: Generalized Linear Models
- 10.32614/cran.package.resourceselection may be a valid DOI for title: ResourceSelection: Resource Selection (Probability...
- 10.1109/cvprw50498.2020.00171 may be a valid DOI for title: Multivariate Confidence Calibration for Object Det...
- 10.1002/sim.4780050506 may be a valid DOI for title: Probabilistic prediction in patient management and...
- 10.1016/j.jbi.2017.10.008 may be a valid DOI for title: Beyond discrimination: a comparison of calibration...

❌ INVALID DOIs

- None

@jasonfan1997
Copy link

@editorialbot check references

@editorialbot
Copy link
Collaborator Author 

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

✅ OK DOIs

- 10.1002/sim.5941 is OK
- 10.1609/aaai.v29i1.9602 is OK
- 10.2307/2276774 is OK
- 10.1175/WAF993.1 is OK
- 10.1002/sim.8281 is OK
- 10.2307/2529131 is OK
- 10.2307/2346866 is OK
- 10.1016/j.acra.2008.04.022 is OK
- 10.1080/03610928008827941 is OK
- 10.1002/qj.456 is OK
- 10.1198/016214506000001437 is OK
- 10.1201/9781439891148-8 is OK
- 10.1002/9781118445112.stat08078 is OK
- 10.1177/0962280216661371 is OK
- 10.1016/0895-4356(92)90192-P is OK
- 10.32614/cran.package.resourceselection is OK
- 10.1109/cvprw50498.2020.00171 is OK
- 10.1093/biomet/45.3-4.562 is OK
- 10.1093/biostatistics/kxq045 is OK
- 10.1080/03610928008827941 is OK
- 10.1093/jamia/ocz228 is OK
- 10.1002/sim.4780050506 is OK
- 10.1016/j.jbi.2017.10.008 is OK

🟡 SKIP DOIs

- No DOI given, and none found for title: Metrics of Calibration for Probabilistic Predictio...
- No DOI given, and none found for title: Applied logistic regression
- No DOI given, and none found for title: MAPIE: an open-source library for distribution-fre...
- No DOI given, and none found for title: Uncertainty Toolbox: an Open-Source Library for As...
- No DOI given, and none found for title: Scikit-learn: Machine learning in Python
- No DOI given, and none found for title: On Calibration of Modern Neural Networks
- No DOI given, and none found for title: Measuring Calibration in Deep Learning.
- No DOI given, and none found for title: Posterior Re-calibration for Imbalanced Datasets

❌ MISSING DOIs

- None

❌ INVALID DOIs

- None

@jasonfan1997
Copy link

@editorialbot generate pdf

@editorialbot
Copy link
Collaborator Author

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

@editorialbot
Copy link
Collaborator Author

Five most similar historical JOSS papers:

dsBinVal: Conducting distributed ROC analysis using DataSHIELD
Submitting author: @schalkdaniel
Handling editor: @csoneson (Active)
Reviewers: @patRyserWelch8, @brunomontezano, @AnthonyOfSeattle
Similarity score: 0.6389

metrica: an R package to evaluate prediction performance of regression and classification point-forecast models
Submitting author: @adriancorrendo
Handling editor: @osorensen (Active)
Reviewers: @wiljnich, @simonpcouch
Similarity score: 0.6364

modeLLtest: An R Package for Unbiased Model Comparison using Cross Validation
Submitting author: @ShanaScogin
Handling editor: @usethedata (Retired)
Reviewers: @andrewheiss, @usethedata
Similarity score: 0.6170

hopit: An R Package for Analysis of Reporting Behavior Using Generalized Ordered Probit Models
Submitting author: @MaciejDanko
Handling editor: @alexhanna (Retired)
Reviewers: @soodoku, @mkearney
Similarity score: 0.6101

Metrics As Scores: A Tool- and Analysis Suite and Interactive Application for Exploring Context-Dependent Distributions
Submitting author: @mrshoenel
Handling editor: @mikldk (Retired)
Reviewers: @mdhaber, @kostiantyn-kucher
Similarity score: 0.6100

⚠️ Note to editors: If these papers look like they might be a good match, click through to the review issue for that paper and invite one or more of the authors before considering asking the reviewers of these papers to review again for JOSS.

@crvernon
Copy link

crvernon commented Jan 3, 2025

@editorialbot query scope

@jasonfan1997 I am going to run this through a review with our larger editorial board to ensure it meets our submission requirements. I'll get back to you ASAP.

In the meantime, the paper length should be reduced to around 1000 words.

Thanks!

@editorialbot
Copy link
Collaborator Author

Submission flagged for editorial review.

@editorialbot editorialbot added the query-scope Submissions of uncertain scope for JOSS label Jan 3, 2025
@jasonfan1997
Copy link

@crvernon Hi, we reduced the paper length and pushed the new version to the repo. Thank you.

@crvernon
Copy link

@qiancao and @jasonfan1997 - Your submission has passed scope query. I am going to go ahead and get a topic editor assigned to this one. Thanks for your patience.

@crvernon crvernon removed the query-scope Submissions of uncertain scope for JOSS label Jan 17, 2025
@crvernon
Copy link

@editorialbot invite @fabian-s as editor

👋 @fabian-s can you take this one on as editor?

@editorialbot
Copy link
Collaborator Author

Invitation to edit this submission sent!

@fabian-s
Copy link

@editorialbot add @fabian-s as editor

@editorialbot
Copy link
Collaborator Author

Assigned! @fabian-s is now the editor

@fabian-s
Copy link

fabian-s commented Jan 17, 2025 •
edited
Loading

hi @adriancorrendo @schalkdaniel @giuseppec

would any of you be willing to review this submission for JOSS? We carry out our checklist-driven reviews here in GitHub issues and follow these guidelines: https://joss.readthedocs.io/en/latest/review_criteria.html

@qiancao
Copy link

qiancao commented Feb 13, 2025

Thank you everyone! Looking forward to your feedback.

@simonpcouch
Copy link

Hey yall—I'm unable to make this review happen at the moment but appreciate you reaching out.

@fabian-s
Copy link

👋 @andrewheiss, @usethedata

would any of you be willing to review this submission for JOSS? We carry out our checklist-driven reviews here in GitHub issues and follow these guidelines: https://joss.readthedocs.io/en/latest/review_criteria.html

Even if not, it would be kind of you reply to this message so I can start looking for other potential reviewers.

@usethedata
Copy link

I do not have bandwidth right now to handle a review. I expect to have time to return to these kinds of things in May.

@fabian-s
Copy link

fabian-s commented Mar 3, 2025

👋 @arunmano121, @andrewheiss, @esadr

would any of you be willing to review this submission for JOSS? We carry out our checklist-driven reviews here in GitHub issues and follow these guidelines: https://joss.readthedocs.io/en/latest/review_criteria.html

Even if not, it would be kind of you reply to this message so I can start looking for other potential reviewers.

@andrewheiss
Copy link

Oh, apologies for the delay here! I also don't have time/bandwidth to handle this review :(

@brunomontezano
Copy link

hi @ShanaScogin @MrShoenel @brunomontezano

would any of you be willing to review this submission for JOSS? We carry out our checklist-driven reviews here in GitHub issues and follow these guidelines: https://joss.readthedocs.io/en/latest/review_criteria.html.

Even if not, please reply to this message so I can start looking for other potential reviewers.

I'm currently unavailable due to other academic commitments. Sorry about the delay for response.

@fabian-s
Copy link

@crvernon having a ridiculously hard time recruiting reviewers for this, as you can see -- please advise..

@fabian-s
Copy link

@crvernon @openjournals/joss-eics ping

@kthyng
Copy link

kthyng commented Mar 26, 2025

@fabian-s Follow up by email to people you've pinged on github sometimes helps if they aren't getting github notifications.

Perhaps @qiancao could look through the reviewer database again and suggest more people, and also may have suggestions from their extended (not-conflicted) network.

@qiancao
Copy link

qiancao commented Mar 26, 2025

@fabian-s Follow up by email to people you've pinged on github sometimes helps if they aren't getting github notifications.

Perhaps @qiancao could look through the reviewer database again and suggest more people, and also may have suggestions from their extended (not-conflicted) network.

Thank you. We will take a look at the reviewer database.

@fabian-s
Copy link

👋 @arunmano121 @esadr @pescap

would any of you be willing to review this submission for JOSS? We carry out our checklist-driven reviews here in GitHub issues and follow these guidelines: https://joss.readthedocs.io/en/latest/review_criteria.html

Even if not, it would be kind of you reply to this message so I can start looking for other potential reviewers.

@pescap
Copy link

pescap commented Mar 31, 2025 •
edited
Loading

@fabian-s no problem, I gladly accept!

@fabian-s
Copy link

ty so much @pescap!

@openjournals openjournals deleted a comment from editorialbot Mar 31, 2025
@fabian-s
Copy link

@editorialbot add @pescap as reviewer

@editorialbot
Copy link
Collaborator Author

@pescap added to the reviewers list!

@fabian-s
Copy link

@editorialbot remove @üpescap as reviewer

@editorialbot
Copy link
Collaborator Author

@üpescap removed from the reviewers list!

@esadr
Copy link

esadr commented Apr 1, 2025

Thanks for the invitation, @fabian-s . Unfortunately, I don't have the bandwidth to take on this review at the moment. Wishing you all the best with the process!

@fabian-s
Copy link

fabian-s commented Apr 7, 2025

👋 @patRyserWelch8 @brunomontezano @AnthonyOfSeattle

would any of you be willing to review this submission for JOSS? We carry out our checklist-driven reviews here in GitHub issues and follow these guidelines: https://joss.readthedocs.io/en/latest/review_criteria.html

Even if not, it would be kind of you reply to this message so I can start looking for other potential reviewers.

@patRyserWelch8
Copy link

patRyserWelch8 commented Apr 7, 2025 via email

@fabian-s
Copy link

fabian-s commented Apr 8, 2025

@editorialbot add @patRyserWelch8 as reviewer

@editorialbot
Copy link
Collaborator Author

@patRyserWelch8 added to the reviewers list!

@fabian-s
Copy link

fabian-s commented Apr 8, 2025

@editorialbot start review

@editorialbot
Copy link
Collaborator Author

OK, I've started the review over in #8026.

@fabian-s
Copy link

fabian-s commented Apr 8, 2025

@patRyserWelch8 sorry to have bothered you, I got confused with another submission, we didn't actually need another reviewer here -- apologies!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@fabian-s fabian-s

@MrShoenel MrShoenel

@pescap pescap

@patRyserWelch8 patRyserWelch8

Labels
pre-review Python TeX Track: 5 (DSAIS) Data Science, Artificial Intelligence, and Machine Learning
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests