[joss] Run automated tests with GH Actions?

[kinow]

Hi,

Related to openjournals/joss-reviews#5135, the item

Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?

Looks like GH Actions is only building the PDF at the moment? Any plans to add tests to the frontend and to the backend, and run them automatically on GH Actions?

[kinow]

It's not only useful to show that the commits & code are being tested, but also to show users & devs how to run the tests in an automated way.

I followed the README instructions to run the tests, for instance, and it displayed the browser windows for me to close, and then to proceed. I tried npm run test -- --browsers ChromeHeadless (I have limited... Angular? knowledge, sorry).

[kinow]

Another advantage of running in an automated way: I ran the backend tests, and one is failing for me:

INFO  [alembic.runtime.migration] Running upgrade  -> 9912972d4f95, empty message
--------------------------------------------------------------------------------------- Captured stderr call ----------------------------------------------------------------------------------------
ERROR [src] Exception on /labelling/create [POST]
Traceback (most recent call last):
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context
    self.dialect.do_execute(
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
    cursor.execute(statement, parameters)
sqlite3.IntegrityError: UNIQUE constraint failed: labelling.u_id, labelling.a_id, labelling.lt_id

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/flask/app.py", line 2077, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/flask/app.py", line 1525, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/flask_cors/extension.py", line 165, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/flask/app.py", line 1523, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/flask/app.py", line 1509, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "/home/kinow/Development/typescript/workspace/lama/backend/src/app_util.py", line 169, in decorated_function
    return f(*args, **kwargs)
  File "/home/kinow/Development/typescript/workspace/lama/backend/src/app_util.py", line 257, in decorated_function
    return f(*args, **kwargs)
  File "/home/kinow/Development/typescript/workspace/lama/backend/src/app_util.py", line 289, in decorated_function
    return f(*args, **kwargs)
  File "/home/kinow/Development/typescript/workspace/lama/backend/src/routes/labelling_routes.py", line 100, in post_labelling
    db.session.commit()
  File "<string>", line 2, in commit
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1451, in commit
    self._transaction.commit(_to_root=self.future)
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 829, in commit
    self._prepare_impl()
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl
    self.session.flush()
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 3383, in flush
    self._flush(objects)
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 3522, in _flush
    with util.safe_reraise():
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
    compat.raise_(
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
    raise exception
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 3483, in _flush
    flush_context.execute()
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute
    rec.execute(self)
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute
    util.preloaded.orm_persistence.save_obj(
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj
    _emit_insert_statements(
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements
    c = connection._execute_20(
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20
    return meth(self, args_10style, kwargs_10style, execution_options)
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 332, in _execute_on_connection
    return connection._execute_clauseelement(
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement
    ret = self._execute_context(
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context
    self._handle_dbapi_exception(
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2043, in _handle_dbapi_exception
    util.raise_(
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
    raise exception
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context
    self.dialect.do_execute(
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (sqlite3.IntegrityError) UNIQUE constraint failed: labelling.u_id, labelling.a_id, labelling.lt_id
[SQL: INSERT INTO labelling (u_id, a_id, lt_id, l_id, p_id, remark, time) VALUES (?, ?, ?, ?, ?, ?, ?)]
[parameters: (2, 1, 2, 1, 1, 'This is an emotion', '00:00:00.000000')]
(Background on this error at: https://sqlalche.me/e/14/gkpj)
----------------------------------------------------------------------------------------- Captured log call -----------------------------------------------------------------------------------------
ERROR    src:app.py:1455 Exception on /labelling/create [POST]
Traceback (most recent call last):
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context
    self.dialect.do_execute(
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
    cursor.execute(statement, parameters)
sqlite3.IntegrityError: UNIQUE constraint failed: labelling.u_id, labelling.a_id, labelling.lt_id

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/flask/app.py", line 2077, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/flask/app.py", line 1525, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/flask_cors/extension.py", line 165, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/flask/app.py", line 1523, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/flask/app.py", line 1509, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "/home/kinow/Development/typescript/workspace/lama/backend/src/app_util.py", line 169, in decorated_function
    return f(*args, **kwargs)
  File "/home/kinow/Development/typescript/workspace/lama/backend/src/app_util.py", line 257, in decorated_function
    return f(*args, **kwargs)
  File "/home/kinow/Development/typescript/workspace/lama/backend/src/app_util.py", line 289, in decorated_function
    return f(*args, **kwargs)
  File "/home/kinow/Development/typescript/workspace/lama/backend/src/routes/labelling_routes.py", line 100, in post_labelling
    db.session.commit()
  File "<string>", line 2, in commit
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1451, in commit
    self._transaction.commit(_to_root=self.future)
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 829, in commit
    self._prepare_impl()
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl
    self.session.flush()
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 3383, in flush
    self._flush(objects)
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 3522, in _flush
    with util.safe_reraise():
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
    compat.raise_(
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
    raise exception
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 3483, in _flush
    flush_context.execute()
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute
    rec.execute(self)
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute
    util.preloaded.orm_persistence.save_obj(
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj
    _emit_insert_statements(
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements
    c = connection._execute_20(
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20
    return meth(self, args_10style, kwargs_10style, execution_options)
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 332, in _execute_on_connection
    return connection._execute_clauseelement(
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement
    ret = self._execute_context(
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context
    self._handle_dbapi_exception(
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2043, in _handle_dbapi_exception
    util.raise_(
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
    raise exception
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context
    self.dialect.do_execute(
  File "/home/kinow/Development/typescript/workspace/lama/backend/venv/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (sqlite3.IntegrityError) UNIQUE constraint failed: labelling.u_id, labelling.a_id, labelling.lt_id
[SQL: INSERT INTO labelling (u_id, a_id, lt_id, l_id, p_id, remark, time) VALUES (?, ?, ?, ?, ?, ?, ?)]
[parameters: (2, 1, 2, 1, 1, 'This is an emotion', '00:00:00.000000')]
(Background on this error at: https://sqlalche.me/e/14/gkpj)

Now I have no way of knowing if there's something wrong on my code, if maybe it's trying to access a port bound by the running Docker, or if it's just because I am using the latest code 🙂

[JarlJansen123]

Hi Bruno, thank you for the detailed feedback.

Looks like GH Actions is only building the PDF at the moment? Any plans to add tests to the frontend and to the backend, and run them automatically on GH Actions?

GH actions is indeed only building the PDF at the moment. As demonstrated with the issues you ran into, it might indeed be useful to add this to the repository. I will see whether I can implement it. I do think the review process can continue without the automatic testing through GH actions?

I followed the README instructions to run the tests, for instance, and it displayed the browser windows for me to close, and then to proceed. I tried npm run test -- --browsers ChromeHeadless (I have limited... Angular? knowledge, sorry).

Yes sorry about this. We placed all environment variables into a .env file, Docker dynamically creates a TypeScript file with some variables that the front-end needs from this .env file. Since tests are not running through Docker this lead to some issues. I have created a file with a dummy variable, this is sufficient for the front-end tests to run (the tests do not depend on this variable for their correctness so there shouldn't be any issues). If you now follow the instructions in the readme the frontend tests will run. After they have completed a browser window will open showing how many have passed and failed (pass/fail rate should also display in your command prompt).

I ran the backend tests, and one is failing for me:

Indeed a bit sloppy on our part, sorry for this. I have resolved the issue.

[kinow]

Hi @JarlJansen123 , @muctadir ,

I'm going through the review checklist for the JOSS paper again today. Confirmed two items had been fixed.

Indeed a bit sloppy on our part, sorry for this. I have resolved the issue.

I had a look at the latest GH Actions on this repo, and also at the files under .github, but I cannot find any new workflows or automated tests added & executed.

Could you provide some more information on how this has been addressed, please?

Thank you!

[kinow]

Actually, I believe this is the last point pending in my review now 👍 fixed once we have automated tests that can be executed by readers/users, and that are preferably being run automatically in a CICD pipeline with the code. Cheers.

[JarlJansen123]

Hi, so in the readme.md there is a detailed description on how to run the tests manually, so automated tests can be executed by readers/users as far as I know. They are currently not automatically executed by a pipeline, but as far as I understand this is not necessary for JOSS submission.

Please let me know if anything is unclear or missing.

[kinow]

I will try to execute the tests from the readme.

but as far as I understand this is not necessary for JOSS submission

Correct. You must provide automated tests but they do not need to run in gh actions.

Note, however, that there is a distinction between a Good and an OK approach to pass the review.

https://joss.readthedocs.io/en/latest/review_criteria.html#tests

Good: An automated test suite hooked up to continuous integration (GitHub Actions, Circle CI, or similar)
OK: Documented manual steps that can be followed to objectively check the expected functionality of the software (e.g., a sample input file to assert behavior)

I thought since you have gh actions already, hooking up ng and pytest there would be pretty simple.

But if you prefer to keep tests executed manually I think that's Ok too.

If the tests pass on my environment I will check that final item off the review list, and end my review with a note about the tests being manually executed and not with every commit or release (which is Ok to pass the review but I believe it should be made clear to users/readers in the review issue).

Cheers

Bruno