[REVIEW]: MAHOS: Measurement Automation Handling and Orchestration System

[editorialbot]

Submitting author: @ktahar (Kosuke Tahara)
Repository: https://github.com/ToyotaCRDL/mahos
Branch with paper.md (empty if default branch):
Version: v0.2.3
Editor: @arfon
Reviewers: @sidihamady, @aquilesC
Archive: 10.5281/zenodo.10074373

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/e8c1073c9ef24b593aa83ab92eb7ed89"><img src="https://joss.theoj.org/papers/e8c1073c9ef24b593aa83ab92eb7ed89/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/e8c1073c9ef24b593aa83ab92eb7ed89/status.svg)](https://joss.theoj.org/papers/e8c1073c9ef24b593aa83ab92eb7ed89)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@FWuellhorst & @sidihamady & @aquilesC, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review.
First of all you need to run this command in a separate comment to create the checklist:

@editorialbot generate my checklist

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @arfon know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Checklists

📝 Checklist for @aquilesC

📝 Checklist for @sidihamady

[editorialbot]

Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1126/science.276.5321.2012 is OK
- 10.1103/PhysRevLett.92.076401 is OK
- 10.1088/0953-8984/18/21/S08 is OK
- 10.21105/joss.05304 is OK
- 10.1016/j.softx.2017.02.001 is OK
- 10.5281/zenodo.3732545 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[editorialbot]

Software report:

github.com/AlDanial/cloc v 1.88  T=1.00 s (746.7 files/s, 215099.3 lines/s)
--------------------------------------------------------------------------------
Language                      files          blank        comment           code
--------------------------------------------------------------------------------
HTML                            166          28339            502          98305
Python                          230          10327           4587          45341
Qt                               20              0              0           9659
SVG                             125              6             18           7685
reStructuredText                170           2507           2912           1341
TOML                             12            153              8           1173
CSS                               6            185             42            842
JavaScript                        8            124            183            778
TeX                               1              7              0            112
YAML                              3              7             18             76
C++                               1             11              2             71
Markdown                          1              9              0             62
make                              2             13              3             38
Bourne Again Shell                1              2              0             18
JSON                              1              0              0             12
INI                               1              0              1              7
--------------------------------------------------------------------------------
SUM:                            748          41690           8276         165520
--------------------------------------------------------------------------------


gitinspector failed to run statistical information for the repository

[editorialbot]

Wordcount for paper.md is 525

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[arfon]

@FWuellhorst, @sidihamady, @aquilesC – This is the review thread for the paper. All of our communications will happen here from now on.

Please read the "Reviewer instructions & questions" in the first comment above. Please create your checklist typing:

@editorialbot generate my checklist

As you go over the submission, please check any items that you feel have been satisfied. There are also links to the JOSS reviewer guidelines.

The JOSS review is different from most other journals. Our goal is to work with the authors to help them meet our criteria instead of merely passing judgment on the submission. As such, the reviewers are encouraged to submit issues and pull requests on the software repository. When doing so, please mention https://github.com/openjournals/joss-reviews/issues/5938 so that a link is created to this thread (and I can keep an eye on what is happening). Please also feel free to comment and ask questions on this thread. In my experience, it is better to post comments/questions/suggestions as you come across them instead of waiting until you've reviewed the entire package.

We aim for the review process to be completed within about 4-6 weeks but please make a start well ahead of this as JOSS reviews are by their nature iterative and any early feedback you may be able to provide to the author will be very helpful in meeting this schedule.

Also, noting that @FWuellhorst said they would not be able to contribute a review until mid-November.

[aquilesC]

Review checklist for @aquilesC

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the https://github.com/ToyotaCRDL/mahos?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@ktahar) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
• Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
• Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
• Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[sidihamady]

Review checklist for @sidihamady

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the https://github.com/ToyotaCRDL/mahos?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@ktahar) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
• Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
• Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
• Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[aquilesC]

First, I want to congratulate @ktahar for a very good job. I myself have been working on these kind of approaches to creating distributed software control, but I think Mahos is an elegant solution, with clean code and well documented.

In going through the checklist, there are some items pertaining to the paper itself (and hence I don't know if creating an issue in the software repo is the best approach) that I think can be improved:

• Abstract: It is not clear what the software does. There is a generic description of how instruments can be controlled, but no reference to what MAHOS actually does.
• Statement of need: There is a description of the field and a good selection of other libraries on which MAHOS builds, but it is not clear why MAHOS is needed. What are the other libraris missing that this library addresses?
• And a reviewed statement of need can also be reflected in the documentation (or the other way around. I believe the documentation makes a clearer case than the paper in that regard).

[ktahar]

Hi @aquilesC. Thank you so much for your review. I'm glad you liked MAHOS.

I tried to improve the paper according to your comments, by stressing reason why mahos is created / difference from existing libraries.
The key point is MAHOS provides a framework for distributed system but the existing libraries provide rather monolithic ones.
Please let me know if this point looks still unclear in the paper.

The overview section in the documentation is updated accordingly:
https://toyotacrdl.github.io/mahos/overview.html#what-is-this

[ktahar]

@editorialbot generate pdf

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[sidihamady]

Dear @ktahar,

First of all, congratulations on this very interesting work. A modular instrumentation framework with the three components, low-level drivers, acquisition and analysis and GUI, is an interesting approach. It is already provided by proprietary systems such as NI tools, to name only the most widespread. Implementing it with Python would combine the free and open source aspect and the power of the data analysis tools of Python packages.

I have some preliminary questions before reviewing the code:

• You are the only author of the manuscript and the only committer on github. I would like to know if others helped in the development process?
• It seems to me in section "Summary" there is confusion between distributed (i.e. network-distributed) and modular. In a laboratory (at least in the field of applied physics), the modular aspect is more important and easier to handle than the network-distributed aspect. Could you clarify this point? May be the word "Modular" could be added to the title.
• In the JOSS submission rules ("Substantial scholarly effort" section), it is specified (i) "Age of software (is this a well-established software project) / length of commit history."; (ii) "Whether the software has already been cited in academic papers.". Concerning the first point, the code has been hosted on github for 3 months. Could you tell us if the code management was done previously using another version control system (public or private)? And if the software was used and then cited in published works?

Regards

[aquilesC]

Thanks for the update, @ktahar . I think the manuscript is moving in the proper direction, but there are some things that can still be clarified.

In the abstract, the sentence: It would become very12 difficult to write debuggable, testable, and maintainable programs on a monolithic framework is quite a strong opinion. Many frameworks are based on the monolithic approach and deliver testable, debuggable and extensible programs. I think it may be valuable to focus on the strengths of MAHOS and who exactly will benefit.

In the statement of need, the sentence: " and talk26 to each other" can be further expanded. How is this achieved? What decisions were made? For example, why selecting ZeroMQ over other message brokers.

I believe one of the strengths of this approach is that the data generated by a node is accessible by any other program implementing the same ZeroMQ framework. This allows to write a control software in Python and visualization software in any other language. It even allows to write measurements in a Jupyter notebook and keep them documented. However, I can't gather this behavior from the manuscript, and is hidden below the layer of the 'distributed' sentence.

[ktahar]

Dear @sidihamady and @aquilesC ,

Thank you so much for your comments and suggestions.
I fixed the paper and introduction of doc clarify the difference between "modular" and "distributed".

• "Modular" is a very important aspect that makes the software maintainable, and MAHOS also tries to be like this.
  But, as @aquilesC pointed out, this point is also realized in the existing libraries too.
• "Distributed" approach makes MAHOS unique. I added explanations of why this could be benefitial: flexible multi-host configuration and data / service accessibility. The latter contains what @aquilesC has suggested.
• Added short discussion on possible downside of "distributed" approach: overhead of data transfer. Moving to intra-process configuration can reduce this. And the ability to do so is one of the reasons to select ZeroMQ.

I hope these changes could remove the ambiguities and answer your comments properly.

To @sidihamady :

You are the only author of the manuscript and the only committer on github. I would like to know if others helped in the development process?

For now, I am only significant contributor to this code base.
(Some implementations of confocal microscope / ODMR experiments are based on that of the predecessor project developed at Tokyo Institute of Technology. But the most part of that was written by me. The professors at Tokyo Tech approved to publish MAHOS in current form as single-author project.)

Could you tell us if the code management was done previously using another version control system (public or private)?

Yes, privately. As commented in first commit ToyotaCRDL/mahos@b24a208 ,
I re-initialized the repository just before making this public, to avoid exposing internal information of the company.
We had 789 commits since 2022-01-19 in older repo.

And if the software was used and then cited in published works?

It has been used for this work, but has not been cited there (because this paper had not been published yet).

To @aquilesC :

This allows to write a control software in Python and visualization software in any other language.

This point is going to be interesting and I actually considered (multiple times) to enable this. In current version, it is a little hard to use the language other than Python because the data is serialized using Python standard pickle.
But, it will not be too hard to move on to other serialization library with multi-language support (such as Protocol Buffers) in future version.
I have added a note in the documentation here:
https://toyotacrdl.github.io/mahos/arch_node.html#data-transport-and-serialization

[ktahar]

@editorialbot generate pdf

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[sidihamady]

Dear @ktahar,

Thank you for these clarifications.

(i) You could perhaps add a header to the source files specifying the type of copyright (BSD) and referring to the LICENSE file.

(ii) The documentation is well done and installation is simple (at least under my systems: Debian and AlmaLinux). It might be necessary to add an install-mahos.sh installation script but users can also do that.
But there are 2 issues (that I added to github) : (i) user should update PATH for virtualenv ; (ii) install python C dev package. After these issues were resolved, testing was successful.

Regards,
Sidi

[ktahar]

Dear @sidihamady ,

Thank you for good suggestions.
I added license notes at the header of source files, and fixed the installation document.

Thanks for the suggestion regarding the installation script too.
But I would prefer not adding it at the moment due to following reasons:

• Assuming the script would contain package installations (sudo apt install... or pip install...), it will be good for some users (who understands what's done by that), but it might be a little unsafe for the others. It can break their computer environment.
• Using virtualenv is my preference and recommendation but this is not a must. Different tools (anaconda etc.) can be used too. The scripted installation will enforce only one way.
• Maintaining the script working for all the possible environments would be rather difficult. (For me, it's going to be quite hard on Windows.)

Best regards,
Kosuke

[sidihamady]

Dear @ktahar,

The end user can create an installation script himself and adapt it to his platform (Linux I mean); No problem :)

The code is particularly well written and even if the documentation in the code files is not exhaustive, its clarity makes it easy to understand.

Regards,
Sidi

[aquilesC]

Dear @ktahar and @arfon , I am done with the review of the paper, I think is ready for publication. It is an interesting piece of software and I am curious to see how it is going to develop further. I have always been a fan of ZMQ for building flexible acquisition software, and MAHOS is a very good example of what can be achieved.

[arfon]

Thanks @aquilesC and @sidihamady for your reviews. @FWuellhorst – as we have two complete reviews, I'm going to remove you as a reviewer at this point.

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[ktahar]

@aquilesC and @sidihamady : Thank you again for the reviews. Your comments and suggestions were really helpful.

@arfon : Thanks for quickly managing this.
I've fixed paper a little, made a release on github https://github.com/ToyotaCRDL/mahos/releases/tag/v0.2.3 ,
and archived it on zenodo https://zenodo.org/records/10074373 .

DOI: 10.5281/zenodo.10074373

[ktahar]

@aquilesC - After discussion above, I've added a (rather experimental) capability to enable custom serializer for foreign language interfacing. Please check out examples/custom_serializer if you're interested. The note here is updated too.

[aquilesC]

@ktahar , that was very quick! The example is simple but it shows the power of interfacing across languages. I'm sure this pattern is very powerful in the long run (for instance to leverage µManager, or WebUI technologies.) Looking forward to seeing how this evolves!

[arfon]

@editorialbot set v0.2.3 as version

[editorialbot]

Done! version is now v0.2.3

[arfon]

@editorialbot set 10.5281/zenodo.10074373 as archive

[editorialbot]

Done! archive is now 10.5281/zenodo.10074373

[arfon]

@editorialbot recommend-accept

[editorialbot]

Attempting dry run of processing paper acceptance...

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1126/science.276.5321.2012 is OK
- 10.1103/PhysRevLett.92.076401 is OK
- 10.1088/0953-8984/18/21/S08 is OK
- 10.21105/joss.05304 is OK
- 10.1016/j.softx.2017.02.001 is OK
- 10.5281/zenodo.3732545 is OK
- 10.5281/zenodo.8396120 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[editorialbot]

👋 @openjournals/pe-eics, this paper is ready to be accepted and published.

Check final proof 👉📄 Download article

If the paper PDF and the deposit XML files look good in openjournals/joss-papers#4773, then you can now move forward with accepting the submission by compiling again with the command @editorialbot accept

[arfon]

@editorialbot accept

[editorialbot]

Doing it live! Attempting automated processing of paper acceptance...

[editorialbot]

Ensure proper citation by uploading a plain text CITATION.cff file to the default branch of your repository.

If using GitHub, a Cite this repository menu will appear in the About section, containing both APA and BibTeX formats. When exported to Zotero using a browser plugin, Zotero will automatically create an entry using the information contained in the .cff file.

You can copy the contents for your CITATION.cff file here:

CITATION.cff

cff-version: "1.2.0"
authors:
- family-names: Tahara
  given-names: Kosuke
  orcid: "https://orcid.org/0000-0002-9474-8970"
doi: 10.5281/zenodo.10074373
message: If you use this software, please cite our article in the
  Journal of Open Source Software.
preferred-citation:
  authors:
  - family-names: Tahara
    given-names: Kosuke
    orcid: "https://orcid.org/0000-0002-9474-8970"
  date-published: 2023-11-11
  doi: 10.21105/joss.05938
  issn: 2475-9066
  issue: 91
  journal: Journal of Open Source Software
  publisher:
    name: Open Journals
  start: 5938
  title: "MAHOS: Measurement Automation Handling and Orchestration
    System"
  type: article
  url: "https://joss.theoj.org/papers/10.21105/joss.05938"
  volume: 8
title: "MAHOS: Measurement Automation Handling and Orchestration System"

If the repository is not hosted on GitHub, a .cff file can still be uploaded to set your preferred citation. Users will be able to manually copy and paste the citation.

Find more information on .cff files here and here.

[editorialbot]

🐘🐘🐘 👉 Toot for this paper 👈 🐘🐘🐘

[editorialbot]

🚨🚨🚨 THIS IS NOT A DRILL, YOU HAVE JUST ACCEPTED A PAPER INTO JOSS! 🚨🚨🚨

Here's what you must now do:

0. Check final PDF and Crossref metadata that was deposited 👉 Creating pull request for 10.21105.joss.05938 joss-papers#4774
1. Wait five minutes, then verify that the paper DOI resolves https://doi.org/10.21105/joss.05938
2. If everything looks good, then close this review issue.
3. Party like you just published a paper! 🎉🌈🦄💃👻🤘

Any issues? Notify your editorial technical team...

[arfon]

@sidihamady, @aquilesC – many thanks for your reviews here! JOSS relies upon the volunteer effort of people like you and we simply wouldn't be able to do this without you ✨

@ktahar – your paper is now accepted and published in JOSS ⚡🚀💥

[editorialbot]

🎉🎉🎉 Congratulations on your paper acceptance! 🎉🎉🎉

If you would like to include a link to your paper from your README use the following code snippets:

Markdown:
[![DOI](https://joss.theoj.org/papers/10.21105/joss.05938/status.svg)](https://doi.org/10.21105/joss.05938)

HTML:
<a style="border-width:0" href="https://doi.org/10.21105/joss.05938">
  <img src="https://joss.theoj.org/papers/10.21105/joss.05938/status.svg" alt="DOI badge" >
</a>

reStructuredText:
.. image:: https://joss.theoj.org/papers/10.21105/joss.05938/status.svg
   :target: https://doi.org/10.21105/joss.05938

This is how it will look in your documentation:

We need your help!

The Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

• Volunteering to review for us sometime in the future. You can add your name to the reviewer list here: https://reviewers.joss.theoj.org/join
• Making a small donation to support our running costs here: https://numfocus.org/donate-to-joss