Wokwi Featured Functions (RDT-568)

[hfudev]

Wokwi support more functionalities other than serial I/O. like pushing buttons, set/get the state of sensors (described here)

Once the API is settled, implement this in pytest-embedded-wokwi

[hayschan]

Hello Hanxi,

Thank you for your response in the previous discussion regarding the CI/CD docker.

Over the past few days, I've been delving into the Unity unit testing framework and pytest-embedded as part of our ESP-IDF software development. I'm currently working on a project that involves testing a matrix keyboard, and I've successfully completed the development of the test_app. To streamline the testing process and eliminate the need for manual intervention (i.e., physically pressing the keyboard), I am considering the implementation of Wokwi.

However, I've encountered a couple of areas where additional resources or documentation would be immensely helpful:

1. API Documentation: The API section for pytest-embedded on Wokwi appears to be incomplete or unpopulated at this time.
2. Practical Examples: The current repository lacks examples that demonstrate the integration of Wokwi with pytest-embedded.

Furthermore, the existing examples, such as confirming a "Hello World" output, don't fully represent complex, real-world scenarios. It would be highly advantageous to have examples that cover more intricate use cases, like I2C interaction or WiFi connectivity testing, timers.

I believe addressing these areas would significantly enhance the utility and applicability of pytest-embedded for more complex development environments.

Looking forward to further updates or guidance on these matters.

Best regards.

[hfudev]

Hi @hayschan! Thank you for your input.

API Documentation

pytest-embedded-wokwi is a wrapper of the wokwi-cli. For now both wokwi-cli and pytest-embedded-wokwi are in alpha. We're working together with @urish to expose more APIs for testing.

Practical Examples

Yes as you said, we're lack of real world examples here in this repo. We will improve it in a forseeable future. For now if you're working together with ESP-IDF projects, you may take a look at the example projects in ESP-IDF repo: wifi getting started example

please note that the pytest.mark.esp32 marker is not applicable with vanilla pytest-embedded. We wrote some customized conftest.py in the esp-idf repo to support the syntactic sugar. You may use

import pytest

pytest.mark.parametrize('target', [
    'esp32',
], indirect=True)

instead.

[urish]

Hello! I just created a demo repo, based on espressif's gh-esp-test-template, which shows a complete setup using pytest embedded together with Wokwi in GitHub CI:

https://github.com/wokwi/wokwi-esp-test-template

It supports (almost) all espressif's ESP32 chips (including the not-yet-released ESP32-P4), and could serve for building more elaborate examples in the future.

Example workflow run (with 7 targets): https://github.com/wokwi/wokwi-esp-test-template/actions/runs/7528389475

[hayschan]

based on espressif's gh-esp-test-template, which shows a complete setup using pytest embedded together with Wokwi in GitHub CI:

To make sure everything is working at least locally, I have tried to run the Wokwi CLI on my computer.

I have two files that is important:

1. test.cpp for writing the TEST_CASE
2. pytest_test.py for using the pytest-embedded API

I encountered some problem.

Original problem

In my test.cpp, I have serveral test cases and app_main.

The app_main() uses unity_run_menu().

extern "C" void app_main(void)
{
    unity_run_menu();
}

And the pytest-embedded python file uses run_all_single_board_cases().

import pytest
from pytest_embedded import Dut

def test_aip1629(dut: Dut)-> None:
    dut.run_all_single_board_cases()

Then, this will happen:

1. I built the app by idf.py build, and then run pytest and wokwi using pytest -s --embedded-services idf,wokwi --tb short.
2. The app is uploaded to Wokwi CLI with success
3. Received the following error very quickly

   2024-01-16 21:21:05 Wokwi CLI v0.8.0
   2024-01-16 21:21:06 Connected to Wokwi Simulation API 1.0.0-20240114
   2024-01-16 21:12:18 Press ENTER to see the list of tests.
   FAILED
   
   =================================================== FAILURES =================================================== 
   _________________________________________________ test_aip1629 _________________________________________________ 
   d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded\dut.py:75: in wrapper
       index = func(self, pattern, *args, **kwargs)  # noqa
   d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded\dut.py:120: in expect
       return self.pexpect_proc.expect(pattern, **kwargs)
   d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pexpect\spawnbase.py:354: in expect
       return self.expect_list(compiled_pattern_list,
   d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pexpect\spawnbase.py:383: in expect_list
       return exp.expect_loop(timeout)
   d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pexpect\expect.py:181: in expect_loop
       return self.timeout(e)
   d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pexpect\expect.py:144: in timeout
       raise exc
   E   pexpect.exceptions.TIMEOUT: <pytest_embedded.log.PexpectProcess object at 0x00000181F61CF350>
   E   searcher: searcher_re:
   E       0: re.compile(b"Here's the test menu, pick your combo:(.+)Enter test for running.")
   E   <pytest_embedded.log.PexpectProcess object at 0x00000181F61CF350>
   E   searcher: searcher_re:
   E       0: re.compile(b"Here's the test menu, pick your combo:(.+)Enter test for running.")
   
   The above exception was the direct cause of the following exception:
   pytest_aip1629.py:38: in test_aip1629
       dut.run_all_single_board_cases()
   d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded_idf\unity_tester.py:428: in run_all_single_board_cases
       for case in self.test_menu:
   d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded_idf\unity_tester.py:243: in test_menu
       self._test_menu = self._parse_test_menu()
   d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded_idf\unity_tester.py:125: in _parse_test_menu
       res = self.confirm_write(trigger, expect_pattern=pattern)
   d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded_idf\unity_tester.py:102: in confirm_write
       raise err
   d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded_idf\unity_tester.py:94: in confirm_write
       res = self.expect(expect_pattern, timeout=timeout)
   d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded\dut.py:82: in wrapper
       raise e.__class__(debug_str) from e
   E   pexpect.exceptions.TIMEOUT: Not found "Here's the test menu, pick your combo:(.+)Enter test for running."    
   E   Bytes in current buffer (color code eliminated): .
   =========================================== short test summary info ============================================ 
   FAILED pytest_aip1629.py::test_aip1629 - pexpect.exceptions.TIMEOUT: Not found "Here's the test menu, pick your combo:(.+)Enter test for running."

My way to solve this

By simply changing the unity functions used in app_main from unity_run_menu() to unity_run_all_tests(), all test cases could be run successfully.

extern "C" void app_main(void)
{
-    unity_run_menu();
+    unity_run_all_tests();
}

New error occured

Strangely, even after running all tests, it will still fail with TIMEOUT with the following error code:

2024-01-16 21:21:32 ./main/aip1629_test.cpp:331:Test Display During Heating:PASS
2024-01-16 21:21:32 Running Test Invalid Inputs...
2024-01-16 21:21:32 ./main/aip1629_test.cpp:340:Test Invalid Inputs:PASS
2024-01-16 21:21:32 I (10950) main_task: Returned from app_main()
FAILED

=================================================== FAILURES =================================================== 
_________________________________________________ test_aip1629 _________________________________________________ 
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded\dut.py:75: in wrapper
    index = func(self, pattern, *args, **kwargs)  # noqa
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded\dut.py:141: in expect_exact
    return self.pexpect_proc.expect_exact(pattern, **kwargs)
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pexpect\spawnbase.py:432: in expect_exact
    return exp.expect_loop(timeout)
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pexpect\expect.py:181: in expect_loop
    return self.timeout(e)
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pexpect\expect.py:144: in timeout
    raise exc
E   pexpect.exceptions.TIMEOUT: <pytest_embedded.log.PexpectProcess object at 0x0000019A33A84B50>
E   searcher: searcher_string:
E       0: b'Press ENTER to see the list of tests'
E   <pytest_embedded.log.PexpectProcess object at 0x0000019A33A84B50>
E   searcher: searcher_string:
E       0: b'Press ENTER to see the list of tests'

The above exception was the direct cause of the following exception:
pytest_aip1629.py:38: in test_aip1629
    dut.run_all_single_board_cases()
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded_idf\unity_tester.py:428: in run_all_single_board_cases
    for case in self.test_menu:
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded_idf\unity_tester.py:243: in test_menu
    self._test_menu = self._parse_test_menu()
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded_idf\unity_tester.py:124: in _parse_test_menu
    self.expect_exact(ready_line)
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded\dut.py:82: in wrapper
    raise e.__class__(debug_str) from e
E   pexpect.exceptions.TIMEOUT: Not found "Press ENTER to see the list of tests"
E   Bytes in current buffer (color code eliminated): _task: Returned from app_main()
=========================================== short test summary info ============================================ 
FAILED pytest_aip1629.py::test_aip1629 - pexpect.exceptions.TIMEOUT: Not found "Press ENTER to see the list of tests"
============================================== 1 failed in 30.24s ==============================================

[hayschan]

Another combination but error persists

I have also tried changing the python test file. So my combination of Python and C++ looks is as followed.

from pytest_embedded_idf.dut import IdfDut

def test_app(dut: IdfDut):
    dut.expect_unity_test_output()

extern "C" void app_main(void)
{
    unity_run_all_tests();
}

An error about regex failing to find test cases results will show.

2024-01-16 21:28:15 ./main/aip1629_test.cpp:340:Test Invalid Inputs:PASS
2024-01-16 21:28:15 I (10950) main_task: Returned from app_main()
2024-01-16 21:28:36 Timeout: simulation did not finish in 30000ms
FAILED

=================================================== FAILURES =================================================== 
___________________________________________________ test_app ___________________________________________________ 
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded\dut.py:75: in wrapper
    index = func(self, pattern, *args, **kwargs)  # noqa
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded\dut.py:120: in expect
    return self.pexpect_proc.expect(pattern, **kwargs)
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pexpect\spawnbase.py:354: in expect
    return self.expect_list(compiled_pattern_list,
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pexpect\spawnbase.py:383: in expect_list
    return exp.expect_loop(timeout)
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pexpect\expect.py:181: in expect_loop
    return self.timeout(e)
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pexpect\expect.py:144: in timeout
    raise exc
E   pexpect.exceptions.TIMEOUT: <pytest_embedded.log.PexpectProcess object at 0x000002AC105CF350>
E   searcher: searcher_re:
E       0: re.compile(b'^[-]+\\s*(\\d+) Tests (\\d+) Failures (\\d+) Ignored\\s*(?P<result>OK|FAIL)')
E   <pytest_embedded.log.PexpectProcess object at 0x000002AC105CF350>
E   searcher: searcher_re:
E       0: re.compile(b'^[-]+\\s*(\\d+) Tests (\\d+) Failures (\\d+) Ignored\\s*(?P<result>OK|FAIL)')

The above exception was the direct cause of the following exception:
pytest_aip1629.py:44: in test_app
    dut.expect_unity_test_output()
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded\dut.py:168: in expect_unity_test_output
    self.expect(UNITY_SUMMARY_LINE_REGEX, timeout=timeout)
d:\.espressif\python_env\idf5.1_py3.11_env\Lib\site-packages\pytest_embedded\dut.py:82: in wrapper
    raise e.__class__(debug_str) from e
E   pexpect.exceptions.TIMEOUT: Not found "re.compile(b'^[-]+\\s*(\\d+) Tests (\\d+) Failures (\\d+) Ignored\\s*(?P<result>OK|FAIL)', re.MULTILINE)"
E   Bytes in current buffer (color code eliminated): Wokwi CLI v0.8.0 (2a4bae54982c) Connected to Wokwi Simulation API 1.0.0-20240114-g15d4d362 Starting simulation... ESP-ROM:esp32s3-20210327 Build:Mar 27 2021 rst:0x1... (total 8457 bytes)
=========================================== short test summary info ============================================ 
FAILED pytest_aip1629.py::test_app - pexpect.exceptions.TIMEOUT: Not found "re.compile(b'^[-]+\\s*(\\d+) Tests (\\d+) Failures (\\d+) Ignored\\s*...
========================================= 1 failed in 60.26s (0:01:00) =========================================

[hfudev]

@hayschan For your c code, it looks good to me. Usually a simple unity_run_menu works

ref to our test app

seems like wokwi-cli does not support "write" yet. I tried calling the wokwi-cli locally without pytest-embedded, and the results are about the same

Running console component tests


Press ENTER to see the list of tests.


1

2

(The 1, 2, and empty lines are what I typed.)

Please note that wokwi-cli is still under active development, and we appreciate your patience as we work to improve it.

[hayschan]

Thanks for the confirmation, @hfudev

I have subscribed to Wokwi's club to support @urish 's work. Really excited about wokwi-cli. Once it is completed, it will make my life way easier. 😆

For now, I will be using the GitHub action self hosted runner until the completion of wokwi-cli.

[urish]

Thanks for confirming! What's the quickest way for me to reproduce this locally?

[hayschan]

Thanks for confirming! What's the quickest way for me to reproduce this locally?

I have created an example repo for you to clone. @urish

Please go to the test_apps path for the component test project.

I have already tried on my computer. The project can be built and has the symptoms I mentioned.