JOSS review: Software paper

[rabdill]

Another issue documenting progress on openjournals/joss-reviews#5658, this one about the reviewer checklist items regarding the manuscript.

Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?

The only shortcoming of the presented summary is that it doesn't indicate what an administrator would need to do to implement CRSocket in their experimental protocol. Phrases like "event-driven client/server web app infrastructure" are technically descriptive, but I believe it excludes a "non-specialist audience" who may need this tool but are not web developers. As it's currently described, it's not obvious to me that this is not "plug and play" software that someone could use without advanced technical skills. That's fine, but I think it would be helpful if this were more clearly explained.

State of the field: Do the authors describe how this software compares to other commonly-used packages?

The paper has a "Statement of need" section that describes the project's origins and issues it aims to resolve ("potential sources of distraction," "make communication problems visible for the tester," etc.). However, there is no discussion of its relation to other work—how are people accomplishing this task now? Why is this new architecture an improvement, and in what ways? It's helpful to know that the code includes a "basic authentication scheme," for example, but is the implication that existing options do not?

References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

This is a very minor note, but in places where multiple references are provided for a single statement, the formatting of these citations is slightly off: Each reference is given its own entry, like this:

many types of data collection [@Frank2016], [@Semmelmann2016], [@Kanerva2019], but the

The effect in the compiled version is that there are three separate parentheticals separated by commas:

many types of data collection (Frank et al., 2016), (Semmelmann et al., 2016), (Kanerva et al., 2019), but the

If the references were formatted as specified, this shouldn't happen. I believe this is the correct setup:

[@Frank2016; @Semmelmann2016; @Kanerva2019]

[henrikdvn]

Regarding summary: I will try to make this more available to non-programmers and clarify that usage requires web programming skills.

Regarding the current state of the field: Some general aspects are fairly well described in the sources. but I guess most people don't look at them so I will provide some more details in the text. When it comes to developmental psychology, the main problem is that research experiments where infants engage actively with digital devices are quite rare. Preparing and engaging infants with limited linguistic capabilities is hard enough with familiar objects and interactions, but even harder with "virtual environments" presented on a tablet. Consequently, there is not much literature on how this is currently accomplished. However, I will have a second look.

Thanks for info on citation syntax. Will fix it :)

[rabdill]

This sounds promising. If there are not many examples of this already being done, that may be another way to address this also—my impression, as someone in another field, was that this framework resolved issues that were unaddressed in competing products, but if this is the software that actually opens up new possibilities for working with a broader range of subjects, that's very exciting too.

[henrikdvn]

I have now updated the paper and hopefully addressed most of these issues. Will appreciate all types of feedback.

[henrikdvn]

Hope all issues have been resolved

[lukaszjablonski]

I think, as already mentioned by @rabdill, the paper is still missing "State of the field". Current version of paper.md is explaining that others are employing tablets for the ECITT task:

Current approaches to digital tasks are either based on self-administration or on handing over pre-configured devices to the participants.

As far as I understand from the paper, the difference of CRSocket to other available tools is that experimenter have better control since subject device can be controlled from experimenter device:

Controlling tasks from a separate device can help meet some of the challenges mentioned above by enabling a bigger toolbox related to motivation, preparation and individual task adaption, based on researchers' ability to engage with and understand the participants.

This exact same idea and control is available in the setup described by Holmboe et al. (2021) to which the paper is referring already:

The primary novelty of the ECITT is that it can be "minimally modified to suit different ages, whilst remaining structurally equivalent" [@Holmboe2021].

and the Holmboe et al. (2021) claim that subject device can be controlled by experimenter device too:

Stimuli were presented on an Apple iPad tablet (the ‘responder’), with a screen size of 9.7 inches and a resolution of 2048 × 1536 pixels. The experimenter controlled stimulus presentation on the iPad via a smartphone (the ‘controller’) using a wi-fi network.

I would expect that as a "State of the field", for which JOSS ask question: "Do the authors describe how this software compares to other commonly-used packages?", the paper should provide a comparison or summary of what experimenter will gain by using CRSocket in comparison to tool used by Holmboe et al. (2021) since as far as I understand both implement the ECITT.

Is Holmboe et al. (2021) not available for public/free use? Also, are there other tools that implement the ECITT?

Such discussion will help reader and person potentially interested in using CRSockets to understand its benefits and will make the paper stronger.

[henrikdvn]

Just to clarify: The software referenced in Holmboe et al. (2021) is the original ECITT software. The original code, which is an undocumented prototype with low reusability potential, was included with the paper as supplementary material. To my knowledge, there is currently no other tool implementing the ECITT test suite and no other software designed for digital tasks that implement controller-responder functionality. This is actually quite challenging. I have to turn down usage requests because I don't have time to support or maintain that software properly. I hope, however, that the CRSocket component will make it easier to implement alternative tools for the ECITT and similar tasks.

[lukaszjablonski]

@henrikdvn, that is exactly what makes your software much more important here. I would highlight that in the paper and that was actually my only point.