[PRE REVIEW]: TorchMetrics - Measuring Reproducibility in PyTorch

[whedon]

Submitting author: @justusschock (Justus Schock)
Repository: https://github.com/PyTorchLightning/metrics
Version: v0.6.2
Editor: @taless474
Reviewers: @inpefess, @richrobe, @reneraab
Managing EiC: Kyle Niemeyer

⚠️ JOSS reduced service mode ⚠️

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/561d9bb59b400158bc8204e2639dca43"><img src="https://joss.theoj.org/papers/561d9bb59b400158bc8204e2639dca43/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/561d9bb59b400158bc8204e2639dca43/status.svg)](https://joss.theoj.org/papers/561d9bb59b400158bc8204e2639dca43)

Author instructions

Thanks for submitting your paper to JOSS @justusschock. Currently, there isn't an JOSS editor assigned to your paper.

The author's suggestion for the handling editor is @taless474.

@justusschock if you have any suggestions for potential reviewers then please mention them here in this thread (without tagging them with an @). In addition, this list of people have already agreed to review for JOSS and may be suitable for this submission (please start at the bottom of the list).

Editor instructions

The JOSS submission bot @whedon is here to help you find and assign reviewers and start the main review. To find out what @whedon can do for you type:

@whedon commands

[whedon]

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks.

⚠️ JOSS reduced service mode ⚠️

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf

[whedon]

Wordcount for paper.md is 1166

[whedon]

Software report (experimental):

github.com/AlDanial/cloc v 1.88  T=0.64 s (500.3 files/s, 74158.1 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                         262           6863          13957          21241
YAML                            20            143            124            872
Markdown                         9            345              0            853
reStructuredText                12            707            812            774
JSON                             1              0              0             95
TeX                              1              5              0             80
JavaScript                       1              3             11             50
TOML                             2              9              1             37
make                             2             10              9             31
DOS Batch                        1              8              1             26
Jinja Template                   1              0              0             16
SVG                              3              0              0             11
Dockerfile                       1              6             12              8
HTML                             1              2              0              8
Bourne Shell                     1              0              0              2
-------------------------------------------------------------------------------
SUM:                           318           8101          14927          24104
-------------------------------------------------------------------------------


Statistical information for the repository 'a3756dcbd4bc7bcd9b45fef7' was
gathered on 2021/12/18.
The following historical commit information, by author, was found:

Author                     Commits    Insertions      Deletions    % of changes
Abe Botros                       1           179             55            0.24
Abhik Banerjee                   1             1              1            0.00
Abhinav Gupta                    1            19              7            0.03
Adrian Wälchli                   3             9             15            0.02
Aki Nitta                        2           175             61            0.24
Akihiro Nitta                    3            64             92            0.16
Alan Du                          1           199              0            0.20
Alexander Senov                  1            51             44            0.10
Ananya Harsh Jha                 4          1803           7573            9.45
Anselm Coogan                    1           210             50            0.26
Arnaud Gelas                     2           157            183            0.34
Artur Jaroszewicz                1            46              2            0.05
Arvind Muralie                   1           760              0            0.77
Ashutosh Kumar                   1           161             61            0.22
Bernardo Lourenço                1             2              2            0.00
Bhadresh Savani                  3           433             29            0.47
Björn Barz                       1            57              6            0.06
CSautier                         1             8              3            0.01
Cookie_thief                     2            51              6            0.06
Daniel Stancl                    8          4674            549            5.26
Davide Fiocco                    1            27             27            0.05
Dipam Vasani                     2           189              0            0.19
Dusan Drevicky                   4           133              9            0.14
Edward Williams                  1           534             11            0.55
Ethan Harris                     1           475              0            0.48
Frankie Robertson                1             5              6            0.01
Gagan Bhatia                     2           601              2            0.61
GiannisVagionakis                1           289              0            0.29
HT Liu                           1            23              1            0.02
Haswanth Aekula                  1           309              0            0.31
Ihar                             1            60             39            0.10
J. Sebastian Paez                2            60             39            0.10
Jan-Henrik Lambrecht             1           241              1            0.24
Jeff Yang                        5           569             77            0.65
Jirka                           30           619            725            1.35
Jirka Borovec                  116         12834           9472           22.47
Johannes Pitz                    1            35             21            0.06
Juan Pablo Carzolio              1             3              3            0.01
Justus Schock                    8          5195            826            6.07
Krzysztof Woś                    1             1              1            0.00
Luca Di Liello                  10          4233            707            4.98
Maheshwari Natarajan             4          1180             91            1.28
Marijan Smetko                   1           159              3            0.16
Maxim Grechkin                  12           743            141            0.89
Mr. Leu                          1            84             54            0.14
Nicki Skafte                    86         14219           4187           18.54
Nicki Skafte Detlefs            11          1560            592            2.17
Nishant Prabhu                   1             1              1            0.00
Olof Harrysson                   1            22              2            0.02
Paul Grundmann                   1            69             17            0.09
Philip E Blair                   1            33              1            0.03
Pranjal  Datta                   2           409             22            0.43
Pranjal Gulati                   3          1175             43            1.23
Prudhvi Rampey                   1            62             13            0.08
Raj Singh                        1           206              1            0.21
Roger Shieh                      2            16             18            0.03
Rohit Gupta                      3           472            286            0.76
Santiago Castro                  3             3              4            0.01
Shiv Dhar                        1             1              1            0.00
Sordie                           1            37              7            0.04
Stephen Malina                   1           313              1            0.32
Tadej Svetina                    6          3466            661            4.16
Teddy Koker                      7           465            182            0.65
Tobias Kupek                     6           891             38            0.94
Tony                             1             8              8            0.02
Travis Addair                    1            34             13            0.05
Tri Dao                          1            16             11            0.03
Vatsalya Chaubey                 1           101              6            0.11
William Falcon                   7           762             42            0.81
XFarooqi                         1            68             84            0.15
Xavier Holt                      1             1              1            0.00
Xavier Sumba                     1           255              0            0.26
Xingdong Zuo                     1             1              2            0.00
Yasser Souri                     1             3              9            0.01
Yigit Ozen                       1             1              1            0.00
Younghun Roh                     2            75             26            0.10
ananthsub                        2            43              2            0.05
arvindmuralie77                  1           447             28            0.48
bibinwils                        1             1              1            0.00
deepsource-autofix[b            12            83             92            0.18
deng-cy                          1           455            151            0.61
discort                          1            16              1            0.02
edenlightning                    2            13             17            0.03
elias-ramzi                      1             8              7            0.02
hlin09                           1            30              1            0.03
hugoperrin                       1           270              0            0.27
j-dsouza                         1           127              2            0.13
jirka                            3             7             13            0.02
karthikrangasai                  5          2642            407            3.07
kingyiusuen                      1           150            103            0.25
manipopopo                       2           208             48            0.26
prajakta0111                     1            15              1            0.02
pwwang                           1             3              3            0.01
quancs                           7          2691             13            2.72
simran2905                       1             2              0            0.00
thomas chaton                    6           416             76            0.50
thomasgaudelet                   1            51             34            0.09
twsl                             2           513            243            0.76
victorjoos                       1            30             20            0.05
yuntai                           1            71             34            0.11

Below are the number of rows from each author that have survived and are still
intact in the current revision:

Author                     Rows      Stability          Age       % in comments
Abhinav Gupta                 5           26.3          0.0               20.00
Akihiro Nitta               183          285.9          5.3                0.00
Alexander Senov              49           96.1          4.0                4.08
Ananya Harsh Jha            712           39.5          0.0                3.79
Anselm Coogan               187           89.0          6.1                8.02
Arnaud Gelas                 30           19.1         11.0                0.00
Artur Jaroszewicz            34           73.9          4.7               11.76
Arvind Muralie             1123          147.8          7.7                4.81
Ashutosh Kumar              161          100.0          0.0                0.62
Bhadresh Savani             263           60.7          6.8                4.18
Björn Barz                   57          100.0          0.6               10.53
Daniel Stancl              4333           92.7          1.6               11.03
Davide Fiocco                27          100.0          8.6                0.00
Dipam Vasani                181           95.8          9.2                0.00
Edward Williams             509           95.3          4.5               11.39
Ethan Harris                421           88.6          8.8               10.93
Gagan Bhatia                314           52.2          4.5               21.34
GiannisVagionakis           254           87.9          5.7               17.32
Haswanth Aekula             263           85.1          5.7               16.35
Ihar                         52           86.7          7.5                7.69
J. Sebastian Paez            39           65.0          8.6                0.00
Jan-Henrik Lambrecht        235           97.5          1.0               11.49
Jeff Yang                    17            3.0          0.0                0.00
Jirka Borovec              6661           51.9          6.3               20.10
Johannes Pitz                25           71.4          7.0                0.00
Juan Pablo Carzolio           2           66.7          0.0                0.00
Justus Schock               545           10.5          9.8                3.30
Luca Di Liello             3179           75.1          6.3               12.46
Maheshwari Natarajan       1052           89.2          1.1               16.25
Maxim Grechkin              617           83.0          8.0                7.94
Mr. Leu                      73           86.9          0.4                2.74
Nicki Skafte               9008           63.4          5.0               11.41
Nishant Prabhu                1          100.0          3.8                0.00
Olof Harrysson               22          100.0          0.1                4.55
Paul Grundmann               50           72.5          4.7               16.00
Philip E Blair               26           78.8          9.9               15.38
Pranjal  Datta              318           77.8          8.4               16.04
Pranjal Gulati             1067           90.8          4.4                9.47
Raj Singh                   205           99.5          4.4               19.02
Santiago Castro               2           66.7          3.9                0.00
Stephen Malina              307           98.1          2.8                6.19
Tadej Svetina              2041           58.9          7.6                7.74
Teddy Koker                 220           47.3          3.4                5.91
Tobias Kupek                622           69.8          1.6               14.31
Tony                          8          100.0          4.4                0.00
Travis Addair                19           55.9          0.0                0.00
Vatsalya Chaubey             87           86.1          4.5                1.15
William Falcon              131           17.2          0.0               99.24
XFarooqi                     64           94.1          2.9               18.75
Xavier Holt                   1          100.0          2.2                0.00
Xingdong Zuo                  1          100.0          0.8                0.00
Yasser Souri                  3          100.0          3.7                0.00
ananthsub                     3            7.0          1.4                0.00
bibinwils                     1          100.0          1.9                0.00
deepsource-autofix[b         72           86.7          5.9                0.00
deng-cy                     186           40.9         11.3               26.34
discort                      12           75.0          4.8                0.00
edenlightning                 7           53.8          9.3               14.29
hlin09                       25           83.3          7.7                0.00
hugoperrin                  229           84.8          5.1               15.28
karthikrangasai            2153           81.5          3.2               12.45
kingyiusuen                  71           47.3          4.0                8.45
manipopopo                  135           64.9         10.0               20.74
prajakta0111                  4           26.7          9.6                0.00
quancs                     2521           93.7          3.5               15.43
simran2905                    2          100.0          5.9                0.00
thomas chaton               324           77.9          6.1                8.33
thomasgaudelet               33           64.7          9.1                0.00
twsl                        505           98.4          0.4                7.72
victorjoos                   26           86.7          8.6                0.00
yuntai                       10           14.1         10.5                0.00

[whedon]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[whedon]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- None

MISSING DOIs

- 10.1163/1574-9347_bnp_e612900 may be a valid DOI for title: Keras

INVALID DOIs

- None

[kyleniemeyer]

Hello @justusschock, thanks for your interest in JOSS. Unfortunately we do not have any editors available in this area right now, so your submission will be in our waitlist until someone becomes available soon.

[Borda]

Hello, I guess that one of these authors could be good candidate for review:

• 20 Popular Machine Learning Metrics. Part 1: Classification & Regression Evaluation Metrics
• 11 Important Model Evaluation Metrics for Machine Learning Everyone should know
• Performance Metrics in Machine Learning [Complete Guide]
• Metrics To Evaluate Machine Learning Algorithms in Python

[justusschock]

Other possible reviewers could include (obviously not an exhaustive list):

• fepegar
• sergioburdisso
• vimalthilak
• richrobe
• inpefess
• smith42

[arfon]

@whedon invite @galessiorob as editor

👋 @galessiorob – would you be able to edit this submission for us?

[whedon]

@galessiorob has been invited to edit this submission.

[kthyng]

@galessiorob Just a ping in the new year! Are you interested in editing this submission?

[taless474]

@kthyng I would like to edit this submission if possible!

[danielskatz]

great - thanks!

[danielskatz]

@whedon assign @taless474 as editor

[whedon]

OK, the editor is @taless474

[taless474]

@EduPH are you interested in reviewing this?

[taless474]

@inpefess are you interested in reviewing this?

[inpefess]

@taless474

are you interested in reviewing this?

yes

[taless474]

@whedon assign @inpefess as reviewer

[whedon]

OK, @inpefess is now a reviewer

[taless474]

@shervinmin are you interested in reviewing this?

[EduPH]

@taless474 sorry to respond late. I'm very busy these days and I think it is far from my expertise. Thank you for considering me.

[taless474]

@Borda I like your suggestions. I asked @shervinmin and he hasn't responded yet. Do you like to mention anyone here?

[shervinmin]

Hi, Thanks for asking me, but I think this is out of my scope of work/research. Perhaps you can ask people who work on similar areas, Bests

…

On Tue, Jan 18, 2022 at 1:15 PM Bita Hasheminezhad ***@***.***> wrote: @Borda <https://github.com/Borda> I like your suggestions. I asked @shervinmin <https://github.com/shervinmin> and he hasn't responded yet. Do you like to mention anyone here? — Reply to this email directly, view it on GitHub <#4008 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXF7TU3EH7MXLBDG2HM2ULUWXJ6NANCNFSM5KJ6MLYQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[justusschock]

@taless474 Sorry for asking again, but is there something we can do to speed up the process?

[taless474]

I apologize; we need to find someone
@richrobe, @Smith42, @sergioburdisso is any of you interested in reviewing this work?

[richrobe]

Hi @taless474!

I would be happy to review this work for JOSS, but I'm not that familiar with pytorch. However, a colleague of mine, @reneraab, is very familiar with pytorch, but has not yet submitted or reviewed for JOSS, so he's not that familiar with the reviewing process. Would it be possible to do the review together?

[Borda]

but I'm not that familiar with pytorch

I would say that pyTorch is rather a tool than the core that paper aims at... 🐰

[taless474]

@whedon add @richrobe as reviewer

[whedon]

OK, @richrobe is now a reviewer

[taless474]

@whedon add @reneraab as reviewer

[whedon]

OK, @reneraab is now a reviewer

[taless474]

Hi @taless474!

I would be happy to review this work for JOSS, but I'm not that familiar with pytorch. However, a colleague of mine, @reneraab, is very familiar with pytorch, but has not yet submitted or reviewed for JOSS, so he's not that familiar with the reviewing process. Would it be possible to do the review together?

@richrobe, thank you for your response. I believe we can have 3 reviewers for this work, so I added @reneraab. @reneraab, thank you for offering to review. Please let us know if you have any questions.

[taless474]

@whedon start review

[whedon]

OK, I've started the review over in #4101.