Implement PortAudio loopback test in Python

[mgeier]

See https://subversion.assembla.com/svn/portaudio/portaudio/trunk/qa/loopback/README.txt

With a bit of luck this can be used with the JACK dummy backend as an automated test on http://travis-ci.org ...

[mgeier]

Git link: https://app.assembla.com/spaces/portaudio/git/source/master/qa/loopback

[tgarc]

It seems like porting this would substantially increase the size of sounddevice's codebase. It also seems like, while they are useful tests, porting them wouldn't really serve the purpose of testing the functionality of sounddevice. Thoughts?

[mgeier]

Yes, to port the whole thing 1:1 is probably too much.
But some subset might still be interesting.
It doesn't have to be the exact functionality from PortAudio either, but I guess that could serve as an inspiration for some kinds of similar tests.

My goal with that would be twosome:

• for users to be able to quickly check if the advertised latency values are accurate on their system. I guess this very much depends on the host API and the hardware in use, therefore users should be able to run this without much hassle on their machines whenever they are curious. They could probably also run the original test programs from PortAudio, but those might be hard to get on their platform.

• fully automated tests that can run on a CI server without user interaction (and without any hardware devices present). This would of course only need to test the functionality of the sounddevice module, it doesn't need to test PortAudio iself. It doesn't need to test all the host APIs, just a single one with a virtual loopback cable (JACK dummy driver, ALSA with snd-aloop, ...?) would be enough.

It definitely doesn't have to cover the whole scope of the PortAudio tests. Especially the whole auto-detection mechanism of loopback cables isn't necessary since we can just specify which device and which channels we want to test.

It might even make sense to encapsulate the signal generation and detection algorithms into a separate project, since this really isn't the job of the sounddevice module. And it could theoretically be re-used by other Python libraries.

Or probably there are already libraries available that do all this stuff?

Would you be interested in working on that?

[tgarc]

It might me interesting to implement separately..I'd have to spend time exploring that code, I've only just skimmed over it so far. Certainly if you make any progress you should make a branch or another repository so that I could play with it as well.

I think some functionality for latency testing would be really interesting! Probably wouldn't be that difficult to write either. I think it would be useful for users too.

In the short term though, I think the priority would be to add a testing module for sounddevice that tests some basic functionality and loopback tests that work across platforms.

[mgeier]

I've actually already created a branch named loopback, but I didn't get further than adding a file with a few notes I made when trying to analyze the PortAudio loopback code.

I find the code quite unstructured and confusing and I don't even really understand what happens when running the test program. There is so much output that I'm having a hard time finding any relevant information.

TBH, I'm not really motivated to implement those tests, but I think it would be really great to have.
I've recently started a companion project (https://github.com/mgeier/python-rtmixer, it's in very early development!) where the main focus is on low latency. For that I would definitely also need a means of reliably measuring the actual latency.

[tgarc]

Very interesting project! I would think having a non GIL dependent callback would greatly expand the possible use cases for sounddevice. I will definitely take a look at this. Thanks for sharing.

Are you able to test in Linux? I want to try and write a simple test program but I will initially write it just for ALSA to prove things out.

[mgeier]

Thanks, I'd love to hear your feedback on this!

Yes, I'm mainly using Linux. I'm looking forward to your tests!

[mgeier]

I've just seen that there is some recent activity on the PyAudio project: http://people.csail.mit.edu/hubert/pyaudio/

Among other things, there are now loopback unit tests: https://people.csail.mit.edu/hubert/git/?p=pyaudio.git;a=blob;f=tests/pyaudio_tests.py;hb=HEAD

[tgarc]

I created a simple loopback test for ALSA and commited to a branch on my fork:

https://github.com/tgarc/python-sounddevice/tree/loopback_tests

It requires that you have the snd-aloop module loaded as well as the python-soundfile, and pytest libraries installed. There is one line in the included utils.py file that makes it incompatible across python2 and 3:

https://github.com/tgarc/python-sounddevice/blob/loopback_tests/tests/utils.py#L201

Just use line 201 for python 2 and line 202 for python 3. I still can't figure out how to throw exceptions with a custom traceback in a backwards compatible way without resorting to using future or some other external library.

[mgeier]

Cool! I copied the .asoundrc to my home directory (am I supposed to do that?), loaded the snd-aloop module and ran pytest. And it passes!

It's a lot of code though, it will take me a few days to digest that ...

I don't know a solution for the exception+traceback problem. I think it can only be done with eval() or other evil stuff. IIRC six does such things.
What about dropping the traceback for Python 2?
Then it should be easily doable by catching the AttributeError caused by .with_traceback() in Python 2, right?

[tgarc]

Correct, just copy asoundrc to your home directory. There is a variable defined near the top called TEST_LENGTHS - this gives a list of test times in seconds. So if you want to run 20 1min tests you would change it to TEST_LENGTHS = [60]*20.

Yeah, I know it's a lot of code in utils and worse, it's poorly documented. I will go back and some documentation hopefully tonight. The code isn't required of course, it just makes writing tests a whole lot easier and I had already written that code for other purposes previously.

[tgarc]

sorry this has been on the back burner for me recently but it is something I intend to complete. The loopback_tests example I gave earlier is a start, but is just too complicated. Hope to get this implemented soon.

[mgeier]

No worries. I'd very much like to have such tests, but it is not at all urgent. After all, the original issue is more than a year old ...

I think it would help if you could try to better isolate the part that generates the test signal and the part that analyzes the recorded signal(s).
Probably there could be different kinds of test signals to test different things. We could also try to provide the signals and analysis methods that are used by PortAudio and PyAudio, respectively.
All this could then be re-used by any Python-based testing application, and therefore doesn't have to be part of the sounddevice module.
It would be definitely interesting to test competing Python libraries with the same test framework.
It could probably even be used for Python-based testing of non-Python applications?

BTW, does your test signal only work on virtual loopback devices or also with an analog loopback cable (i.e. in the presence of noise and potential level differences)?

The integration of those tests into the sounddevice module is another story, and I have the feeling that your current implementation is indeed too complicated in that regard.
Also, (and I don't know if your current implementation already allowes that,) the same tests should be available for playrec(), Stream.write() + Stream.read(), as well as probably some simple handwritten callback setup.
And even more than for the sounddevice module, I'm looking forward to using those tests for https://github.com/mgeier/python-rtmixer/!