[PRE REVIEW]: Litrepl: Literate Paper Processor Promoting Transparency More Than Reproducibility

[editorialbot]

Submitting author: @sergei-mironov (Sergei Mironov)
Repository: https://github.com/sergei-mironov/litrepl
Branch with paper.md (empty if default branch): paper
Version: v3.12.0
Editor: @matthewfeickert
Reviewers: Pending
Managing EiC: Daniel S. Katz

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/a180a5cc8c73c740e4ed3de22218c09f"><img src="https://joss.theoj.org/papers/a180a5cc8c73c740e4ed3de22218c09f/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/a180a5cc8c73c740e4ed3de22218c09f/status.svg)](https://joss.theoj.org/papers/a180a5cc8c73c740e4ed3de22218c09f)

Author instructions

Thanks for submitting your paper to JOSS @sergei-mironov. Currently, there isn't a JOSS editor assigned to your paper.

@sergei-mironov if you have any suggestions for potential reviewers then please mention them here in this thread (without tagging them with an @). You can search the list of people that have already agreed to review and may be suitable for this submission.

Editor instructions

The JOSS submission bot @editorialbot is here to help you find and assign reviewers and start the main review. To find out what @editorialbot can do for you type:

@editorialbot commands

[editorialbot]

Hello human, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

✅ OK DOIs

- 10.1109/52.311070 is OK
- 10.1109/MCSE.2021.3059263 is OK
- 10.1038/s41597-022-01720-9 is OK
- 10.1017/S0956796810000195 is OK

🟡 SKIP DOIs

- No DOI given, and none found for title: Literate Programming
- No DOI given, and none found for title: Jupyter Notebooks - a publishing format for reprod...
- No DOI given, and none found for title: Recursive Functions of Symbolic Expressions and th...
- No DOI given, and none found for title: A Programming Language
- No DOI given, and none found for title: APL Demonstration 1975
- No DOI given, and none found for title: Terminal graphics protocol
- No DOI given, and none found for title: Lark
- No DOI given, and none found for title: POSIX Standard: Base Specifications, Issue 8
- No DOI given, and none found for title: Project Jupyter: Computational Narratives as the E...

❌ MISSING DOIs

- 10.1109/mcse.2007.53 may be a valid DOI for title: IPython: A System for Interactive Scientific Compu...
- Entry without DOI or title found

❌ INVALID DOIs

- None

[editorialbot]

Software report:

github.com/AlDanial/cloc v 1.98  T=0.05 s (1122.2 files/s, 224293.3 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                          12            191            228           1729
Bourne Shell                     8            220             34           1500
Markdown                        11            427             76           1357
SVG                              1              0              0            837
TeX                              6            161             50            819
C/C++ Header                     2             85            306            729
vim script                       3             60             48            629
Nix                              3             67             28            427
make                             1             16              0             85
INI                              1              5              0             45
Text                             3              6              0             28
-------------------------------------------------------------------------------
SUM:                            51           1238            770           8185
-------------------------------------------------------------------------------

Commit count by author:

   585	Sergei Mironov
   115	Sergey Mironov

[editorialbot]

Paper file info:

📄 Wordcount for paper.md is 1870

✅ The paper includes a Statement of need section

[editorialbot]

License info:

✅ License found: BSD 3-Clause "New" or "Revised" License (Valid open source OSI approved license)

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[editorialbot]

Five most similar historical JOSS papers:

Rclean: A Tool for Writing Cleaner, More Transparent Code
Submitting author: @MKLau
Handling editor: @danielskatz (Retired)
Reviewers: @danielskatz
Similarity score: 0.6292

Caliban: Docker-based job manager for reproducible workflows
Submitting author: @sritchie
Handling editor: @diehlpk (Active)
Reviewers: @lukasheinrich, @arokem
Similarity score: 0.6019

Ipyannotator: the infinitely hackable annotation framework
Submitting author: @itepifanio
Handling editor: @danielskatz (Active)
Reviewers: @csadorf, @matthewfeickert
Similarity score: 0.5965

Reducing the efforts to create reproducible analysis code with FieldTrip
Submitting author: @matsvanes
Handling editor: @crsl4 (Active)
Reviewers: @gflofst, @ashahide
Similarity score: 0.5962

funsies: A minimalist, distributed and dynamic workflow engine
Submitting author: @clavigne
Handling editor: @diehlpk (Active)
Reviewers: @gflofst, @vijaysm
Similarity score: 0.5957

⚠️ Note to editors: If these papers look like they might be a good match, click through to the review issue for that paper and invite one or more of the authors before considering asking the reviewers of these papers to review again for JOSS.

[danielskatz]

👋 @sergei-mironov - thanks for your submission. As it's not clear to me that this is research software as defined by JOSS, the editors will now discuss this. You should hear back in a week or two.

If this does pass this review, I will then ask you to reduce the length of the paper to closer to 1000 words, the JOSS maximum. There also appear to a be couple of reference/DOI issues that we will also discuss.

[danielskatz]

@editorialbot query scope

[editorialbot]

Submission flagged for editorial review.

[sergei-mironov]

👋 @sergei-mironov - thanks for your submission. As it's not clear to me that this is research software as defined by JOSS, the editors will now discuss this. You should hear back in a week or two.

If this does pass this review, I will then ask you to reduce the length of the paper to closer to 1000 words, the JOSS maximum. There also appear to a be couple of reference/DOI issues that we will also discuss.

@danielskatz Thank you for the feedback. I have added the suggested DOI and updated the summary to indicate that the outcomes of the described project serve as a research instrument.

[sergei-mironov]

Hi @danielskatz Do you have any updates regarding this subject?

[danielskatz]

@sergei-mironov - this has just passed our scope review. I'll look for an editor next.

[danielskatz]

👋 @sneakers-the-rat - Would you be able to edit this submission?

[danielskatz]

@editorialbot invite @sneakers-the-rat as editor

[editorialbot]

Invitation to edit this submission sent!

[sergei-mironov]

@danielskatz I am glad to know this. Meanwhile, I will reduce the length of the text to fit your criteria.

[danielskatz]

@sergei-mironov - any news on the shortened paper? I will continue looking for an editor as well, and we have some new ones coming on board soon...

[danielskatz]

👋 @fabian-s - Would you be able to edit this submission?

[danielskatz]

@editorialbot invite @fabian-s as editor

[editorialbot]

Invitation to edit this submission sent!

[fabian-s]

@danielskatz I'd rather not, sorry.

I did not vote on scope of this submission, but I would definitely not have supported taking this on and would rather spend my time on submissions I'm more interested in. It also doesn't seem fair to the submitter to have an editor that doesn't see the point of the paper from the outset, since we aim for a collaborative, constructive review process at JOSS.

To me, this looks like an extremely feature-limited early prototype version of things like Quarto/JupyterNotebook/Rmarkdown/Sweave/.... and I don't see why anybody would choose to use this instead of these much more mature frameworks for literate programming. (no offense, @sergei-mironov)

[sergei-mironov]

@sergei-mironov - any news on the shortened paper? I will continue looking for an editor as well, and we have some new ones coming on board soon...

@danielskatz I have shortened the paper text. For the last version, if I remove the yaml header and a few technical comments, my word counter would show 990. Please let me know if you want it to be even shorter.

[sergei-mironov]

@danielskatz The original text compared my solution with Jupyter as it is probably the most popular and mature approach. I also have added the brief comparison with other projects including the ones you requested, in form of a table. Here is the direct link pointing to the text. I might need some advises on how to format the references within this table. The ideas is to have table-local footnotes, if the publishing system allows it. Is it possible to express it in the JOSS Markdown dialect? Please let me know if you want me to make further updates.

[danielskatz]

@editorialbot generate pdf

@sergei-mironov - note that you can do this too... editorialbot commands need to be the first entry in a new comment.

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[editorialbot]

Five most similar historical JOSS papers:

Rclean: A Tool for Writing Cleaner, More Transparent Code
Submitting author: @MKLau
Handling editor: @danielskatz (Retired)
Reviewers: @danielskatz
Similarity score: 0.6335

PhysioLabXR: A Python Platform for Real-Time, Multi-modal, Brain–Computer Interfaces and Extended Reality Experiments
Submitting author: @ApocalyVec
Handling editor: @mstimberg (Active)
Reviewers: @lucask07, @nastaran62
Similarity score: 0.6129

funsies: A minimalist, distributed and dynamic workflow engine
Submitting author: @clavigne
Handling editor: @diehlpk (Active)
Reviewers: @gflofst, @vijaysm
Similarity score: 0.6093

Ipyannotator: the infinitely hackable annotation framework
Submitting author: @itepifanio
Handling editor: @danielskatz (Active)
Reviewers: @csadorf, @matthewfeickert
Similarity score: 0.6074

GPP, the Generic Preprocessor
Submitting author: @Logological
Handling editor: @gkthiruvathukal (Active)
Reviewers: @Smattr, @drj11
Similarity score: 0.6069

⚠️ Note to editors: If these papers look like they might be a good match, click through to the review issue for that paper and invite one or more of the authors before considering asking the reviewers of these papers to review again for JOSS.

[danielskatz]

@xuanxu & @arfon - Can we handle footnotes/references in a table? Do you a suggestion for what's best in this case? And should we add something in our example paper to explain this case?

@danielskatz The original text compared my solution with Jupyter as it is probably the most popular and mature approach. I also have added the brief comparison with other projects including the ones you requested, in form of a table. Here is the direct link pointing to the text. I might need some advises on how to format the references within this table. The ideas is to have table-local footnotes, if the publishing system allows it. Is it possible to express it in the JOSS Markdown dialect? Please let me know if you want me to make further updates.

[danielskatz]

@sergei-mironov - I think the content you added is fine, though we can work on the formatting.

[danielskatz]

I'm going to mark this as waitlisted now as I wait for an editor. We'll have some new editors starting fairly soon, which should help

[xuanxu]

Can we handle footnotes/references in a table? Do you a suggestion for what's best in this case? And should we add something in our example paper to explain this case?

Probably @tarleb can answer here

[tarleb]

Footnotes in tables become normal, document-wide footnotes.

Maybe definition lists might be an option here?

[sergei-mironov]

@tarleb I have committed the version in which both the table and the local footnotes are encoded as inline Latex. Pandoc seems to process this input well, so please take a look. Is it acceptable for you?

[danielskatz]

@editorialbot generate pdf

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[danielskatz]

This formatting looks good

[editorialbot]

Five most similar historical JOSS papers:

Rclean: A Tool for Writing Cleaner, More Transparent Code
Submitting author: @MKLau
Handling editor: @danielskatz (Retired)
Reviewers: @danielskatz
Similarity score: 0.6570

CMinx: A CMake Documentation Generator
Submitting author: @ryanmrichard
Handling editor: @danielskatz (Active)
Reviewers: @robertodr, @peanutfun
Similarity score: 0.6297

Ipyannotator: the infinitely hackable annotation framework
Submitting author: @itepifanio
Handling editor: @danielskatz (Active)
Reviewers: @csadorf, @matthewfeickert
Similarity score: 0.6288

Inscriptis - A Python-based HTML to text conversion library optimized for knowledge extraction from the Web
Submitting author: @AlbertWeichselbraun
Handling editor: @sbenthall (Active)
Reviewers: @reality, @rlskoeser
Similarity score: 0.6288

PhysioLabXR: A Python Platform for Real-Time, Multi-modal, Brain–Computer Interfaces and Extended Reality Experiments
Submitting author: @ApocalyVec
Handling editor: @mstimberg (Active)
Reviewers: @lucask07, @nastaran62
Similarity score: 0.6283

⚠️ Note to editors: If these papers look like they might be a good match, click through to the review issue for that paper and invite one or more of the authors before considering asking the reviewers of these papers to review again for JOSS.

[danielskatz]

👋 @matthewfeickert - Would you be willing to edit this submission?

[danielskatz]

@editorialbot invite @matthewfeickert as editor

[editorialbot]

Invitation to edit this submission sent!

[matthewfeickert]

@danielskatz Sorry for late response. In transit now but yes, I can be there editor.

[danielskatz]

Thanks @matthewfeickert - I'll assign you

[danielskatz]

@editorialbot assign @matthewfeickert as editor

[editorialbot]

Assigned! @matthewfeickert is now the editor

[matthewfeickert]

@sergei-mironov Thanks for your patience. Though before I spend reviewer time on this I'm going to request that you reread the Submitting a paper to JOSS documentation and also suggest that you read the Review criteria docs to better understand how your submission would be evaluated.

At the moment, the documentation is rather weak as there is only a README and a few other Markdown documents. In general we're looking for much more robust documentation.

While I appreciate that Litrepl is meant to be a CLI tool/text editor plugin only, and not be used as a Python library, we still expect full API documentation with examples. From simply running litrepl --help it can be seen that is not provided.

The tests consist of a single Shell script, but it isn't clear what coverage this provides, and as you're using external Python dependencies outside of the CPython standard library there should be automated tests for all Python versions Litrepl supports. Use with modern Python versions raises SyntaxWarnings that should have been caught by automated testing.

I am also not very convinced by the statement of need and examples as to how Litrepl is research software. The table in the paper (which needs to have a Table number as a reference and a caption) seems to also not be making a strong case for Litrepl, but that's a decision that should be left to reviewers. Though please make the arguments very clear for them.

At the moment the submission does not feel like it is in a complete state to have reviewers actively engage with it. It should be clear to the reviewers exactly what they are evaluating through the documentation, sufficient test set, and clear statement of need.

If you would like to continue with the review at this stage, please address all the above issues within two weeks (by 2025-04-09).

If that timeline isn't sufficient or viable for your current availability, we can negotiate on that, but if you feel like you would like more time to fully address everything, my suggestion there, given the scope of work that would need to be done, is that you instead withdraw your submission for now and resubmit in the future whenever you have had the time to carefully go through and prepare the project. That's totally fine, not uncommon, and future (re)submissions would be welcome.

[sergei-mironov]

Hi @matthewfeickert , thank you for paying attention to this submission. I appreciate your points; let me first comment on the scope issue, and then I will answer the technical ones.

I am also not very convinced by the statement of need and examples as to how Litrepl is research software.

I admit that the Litrepl scope might be considered a corner case by some people. I tend to call it research software because publishing research results is a valuable part of the process, and I believe that Litrepl facilitates the delivery of these results from the computation core to output documents in the most direct and observable manner. I would like to argue politely that according to the modern definition of 'reproducibility', neither Litrepl nor other tools (not even Jupyter, despite the claims of its authors) make papers reproducible by themselves; they need to be part of a reproducible research environment to achieve this. Litrepl, however, seems to me the lightest option by far for those who are ready to cope with its limitations. I would like to try passing the review process to publish these arguments.

The documentation is rather weak as there is only a README and a few other Markdown documents

Sure, I admit that the Readme file is the document that currently contains the most information. I am ready to provide documents in other formats too. I think that for a CLI application, a manpage might be a good addition. Please let me know which format you find appropriate.

we still expect full API documentation with examples

I did not originally plan to provide Python documentation because I did not consider this project to be a Python library (so, strictly speaking, there is no public API). However, the latest changes introduced a few distinct modules which might be interesting to a wider audience, namely the interpreter wrapper classes and the text document grammar combinators. I am ready to provide the documentation; just let me know which format you find best. Would you consider very detailed code comments as good documentation, or should it be a Read the Docs page?

The tests consist of a single Shell script, but it isn't clear what coverage this provides

The single shell script you mentioned contains approximately 30 integration test scenarios, and some of them are reused with different interpreters. I test many aspects of parsing and evaluation, including integration with the Vim editor I currently maintain. Since the project relies on POSIX APIs, I think that wrapping POSIX calls in standard Python tests would be unnecessary.

I agree that measuring the coverage would be a good thing to do, so I am ready to integrate the codecov tools. Thank you for mentioning the SyntaxWarning issue; I will fix it as soon as possible, along with adding an option to test more Python versions.

If you would like to continue with the review at this stage, please address all the above issues within two weeks (by 2025-04-09).

I'll be busy for the next two weeks, but after that, before the middle of April, I hope to have plenty of free time to work on this project, including improving the tests and adding the documentation. Would it be okay if we extended the deadline by a week? I will do my best to resolve the issues before April 16, 2025. What do you think?

[sergei-mironov]

Let me post an update on this issue. At the time of the writing I have improved the Litrepl tool in the following ways:

• Integrated the code coverage metrics, it turned out to be approximately 88%. The badge is published and the detailed report is available in the corresponding Readme file section.
• Improved the test system so it can test Litrepl using differetn Python interpreters. Fixed a number of compatibility issues.
• Fixed the SyntaxWarning when running tests using newer Pythons
• Write readthedocs-style documentation
• Generate a manpage

There are some requests that remains undone. I plan to address them within a couple of weeks.

• Write API documentation

[matthewfeickert]

This is a form notifcation to all JOSS work that I'm involved with editing at the moment:

Sorry for being slow to respond to comments here and follow up. I've been sick and then had work travel and so am behind on all JOSS work. Please ping me if you do not have a more full response to ongoing work within 24 hours.

If you have questions or concerns, please feel free to ask for additional information.

[danielskatz]

@matthewfeickert - How is this coming? Is it ready to get reviews and start the review?

[matthewfeickert]

I'm sorry for my extended lack of response — I've been ill and mostly offline for anything that hasn't been work critical. I'm returning today (2025-05-08) but realistically my earliest reply will be end of day in the US.