liblsl on ESP32-S3

[ValentinPintat]

Hi,
We have a project where we want to implement LSL on an ESP32-S3 to create outlets for our sensors.
We first try to put micro python on the ESP but we cannot access liblsl with upip.
So we thought about putting the C librairies directly build inside our ESP.

Do you have any advice? What would be the closest example?

Thank you very much
Valentin

[cboulay]

I'm fairly certain that you'll have to self-compile liblsl, whether you write your app in C or in Python, but I don't know if it's even possible on your platform. You'll need a C-compiler and at least unix-like networking libraries. I took a quick look at ESP-IDF (https://www.espressif.com/en/products/sdks/esp-idf) and it looks like it should have everything you need.

The script at the root of the repo should give you a good starting point: https://github.com/sccn/liblsl/blob/master/standalone_compilation_linux.sh

[ValentinPintat]

Hi,
Thank you very much for your answer.
We'll try with that starting point then !

[tstenner]

Originally I thought it'd be difficult, because the target platform needs a working environment for Asio, but apparently that's a thing nowadays.

[ValentinPintat]

I don't get it, what'is the link beetween ASIO and liblsl ?

[cboulay]

ASIO is a liblsl dependency. You'll find it in the thirdparty folder.

[dhairyashah1]

I'm working on the project mentioned by @ValentinPintat
I included the src, include, thirdparty mentioned here.
boost;asio port is there already for esp idf

I was wondering if any other boost file will be required for creating a basic lsl outlet from lslboost

[ValentinPintat]

Is there an easy way to make liblsl available on micropython ?

[cboulay]

"easy" is relative I guess.

1. Build liblsl using that compilation script
2. Install pylsl from source. python -m pip install git+https://github.com/labstreaminglayer/liblsl-Python.git
3. Set an environment variable to tell pylsl where to find liblsl.so built in step 1. Either PYLSL_LIB to the .so file or LD_LIBRARY_PATH to the folder containing the .so file. Alternatively, you can copy the library into a path that's already on the search path, such as in the python package itself.

Step 1 should be possible, but I wouldn't know and I have no way to test.

[ValentinPintat]

Thank you very much for your replies !

Unfortuneltly we don't have those skills in our lab, we're more on the hardware side.
But there is still something we want to try !
In the past a student in our Lab achived to implement an LSL outlet on an ESP32 but now all the librairies are dead so we have to redo everything.
In his report he said there are 4 stages to start streaming data from an outlet with LSL. (Discovery, Subscription, Data transfer, and Closing).
Today we want to try to work on the first phase which is the Discovery and go step by step.

From what I understand, on the discovery phase when the computer is running Labrecorder, an UDP stream is send from Labrecorder to all the network using the host gateway 239.255.172.215 and the port 16571.

So if we re catching that UDP stream and we're sending back with our ESP32 the proper XML info we should see the stream appear on Labrecorder ?
Do you know if those informations are still right for the labrecorder that we have today using liblsl V1.16 ?

[ValentinPintat]

Hello !
Here are some news on the project:
We manage to get 2 LSL:ShortInfo comming from Labrecorder with our ESP32 on the port 16571.

We can see that Labrecorder is indicating us 2 ports 16572 and 16573.

We know that we have to answer by an XML message but it's not clear for us on what port we have to send the answer ?
If we answer properly on the right port will it show as a stream on Labrecorder or will it show as a stream only when we will establish the TCP connection for the Subscritption part ? If we have the right answer will we have an acknolegement from Labrecorder ?

Thank you very much

[cboulay]

I'm very confused. Are you trying to control LabRecorder? Or are you trying to read streams from other devices / applications? Or are you trying to stream data out in a way that LabRecorder will recognize it?

[ValentinPintat]

Sorry about the confusion !
I dont't want to control Labrecorder.
I want to do something similar as AudioCapture.exe but embeded in an ESP32.
If i'm correct AudioCapture.exe is only creating an Outlet and we can see that outlet available on Labrecorder.

Unfortenely it's very difficult for me to put all the liblsl library on my ESP so instead I want to try to implement an outlet.
I managed to create UDP streams, TCP connections and embed XML on my ESP32.

So my question is what are the steps to create an outlet by using TCP, UDP, IGMP and XML. What are the exchange beetween Labrecorder and an outlet ? I want to use Lab recorder to see if my Outlet is visible on the Network, to connect to it and record my data in xdf format to see if it's well transmited.

[cboulay]

The part of the source you should be looking at is here:
https://github.com/sccn/liblsl/blob/master/src/stream_outlet_impl.cpp#L14

There's a UDP time server (necessary for clock sync), a UDP multicast responder (necessary to respond to queries to find streams), and finally the data tcp server.

here's how to respond to a query.

here's how to respond to a clock sync.

This method shows you how to initiate the protocol handshake for the TCP server.

IMO the path to getting liblsl compiled on the ESP32 might be shorter than the path to implementing an outlet from scratch, especially if you want it to buffer data, to be robust to network drops, to transmit to multiple inlets, etc.

In any case, I think your development test bench should include:

1. a simple outlet (e.g., from pylsl/examples)
2. a simple script that runs the resolver, makes an inlet from the retrieved StreamInfo, then pulls samples continuously
3. Wireshark
4. A lot of time.

Then, when you've characterized all the packets the simple outlet provides, you can swap it out for your custom implementation on ESP32 until you match the packets 1:1.

[tstenner]

I once wrote a wireshark dissector that splits some LSL packets.

So my question is what are the steps to create an outlet by using TCP, UDP, IGMP and XML. What are the exchange beetween Labrecorder and an outlet ? I want to use Lab recorder to see if my Outlet is visible on the Network, to connect to it and record my data in xdf format to see if it's well transmited.

I would use the liblsl-Python examples (HandleMetadata first, then ReceiveData and ReceiveAndPlot) for that. Anyways, the necessary steps for stream discovery are

• listen for (broadcast / multicast) discovery packets on port 16571.
• match the XPath query against your stream metadata
• if it matches, send a reply packet

[ValentinPintat]

Whaou ! Thank you very much for that !

• listen for (broadcast / multicast) discovery packets on port 16571.
• match the XPath query against your stream metadata
• if it matches, send a reply packet

We manged to try those three steps but we still have some questions.
Do we have to send the reply on the port 16571 ?
If the reply is correct :

• will we see our "Stream" appearing in green on Labrecorder ?
• will we have an answer from Labrecorder maybe on another port (16572 ?)
• At what step our stream will apprear green on Labrecorder ?

By green I mean that :

[tstenner]

Do we have to send the reply on the port 16571?

The discovery (shortinfo) request has the query in the first line, followed by \r\n and the response port and a query id (source). UDP isn't connection oriented, so the source port for the discovery packet generally isn't the port the reply has to go to.

[thiago-roque07]

Hi!

I'm also working on a project to embed liblsl into an ESP32.
@ValentinPintat and @dhairyashah1, would you mind sharing how you did it?
I'm planning to use IDF standard C instead of micropython, so technically should be easier.
For now I'm getting error with PTHREADS.

`-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - not found
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - no
CMake Error at /usr/share/cmake-3.27/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
Could NOT find Threads (missing: Threads_FOUND)
Call Stack (most recent call first):
/usr/share/cmake-3.27/Modules/FindPackageHandleStandardArgs.cmake:600 (_FPHSA_FAILURE_MESSAGE)
/usr/share/cmake-3.27/Modules/FindThreads.cmake:226 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
main/lib/liblsl/CMakeLists.txt:167 (find_package)

-- Configuring incomplete, errors occurred!
FAILED: build.ninja `

I'm using the Espressif IDF extension to VS Code.

[thiago-roque07]

After a bit of investigation:
It seems that Espressif IDF has its own PTHREAD library as part of the IDF framework, and for some reason the "find_package(Threads REQUIRED)" inside liblsl is not finding the pthread library from IDF.

[lucaskdc]

After a bit of investigation: It seems that Espressif IDF has its own PTHREAD library as part of the IDF framework, and for some reason the "find_package(Threads REQUIRED)" inside liblsl is not finding the pthread library from IDF.

Note that the FindThreads (a builtin cmake module) tries to find Threads from the system libraries. I guess that configuring the FindThreads through some cmake variables (CMAKE_THREAD_LIBS_INIT and CMAKE_USE_PTHREADS_INIT) could allow it to find the pthreads from the esp32's lib.

Check this issue regarding Threads being not found on linux: alicevision/geogram#2 (comment)

[ValentinPintat]

Hello, we managed to embed lsl on the ESP32-S3.
I have to see with the director of my Lab if I can share the code.
But I think the student who work on that project kind of "hacked" the system instead of using the regular library.
So Basically you have to take care of the exchanges during the TCP and UDP handshake, data transmission, etc....

[thiago-roque07]

@lucaskdc
Thank you for the info. I'll try that later.

@ValentinPintat
Please, do check if you can share that. I'd love to see that, and it would be of great value for the community!

[drLaba]

IMO the path to getting liblsl compiled on the ESP32 might be shorter than the path to implementing an outlet from scratch, especially if you want it to buffer data, to be robust to network drops, to transmit to multiple inlets, etc.

This seems indeed achievable, thank you @cboulay. I was able to compile liblsl for ESP32 (some comments about that below).

However, I’ve begun to question whether this is the right goal to pursue in the first place.
While the ESP32 may have the hardware capabilities, isn’t it pushing the intended use of LSL too far?
It increasingly appears to me that the library is primarily designed for desktop and mobile platforms, particularly Windows, isn’t it?

@tstenner mentioned that the typical pipeline is one of "device → vendor SDK → liblsl → network → liblsl → receiver".
However, he has also maintained the build script for "embedded" (standalone_compilation_linux.sh). Was your intention to use embedded version as a desktop alternative or as the data-acquiring device itself? BTW, building and testing liblsl on Raspberry Pi 4B running Bullseye was straightforward, thank you.

I would greatly appreciate it if you could share any information you have about manufacturers using LSL directly on their devices.
JFYI I'm not a manufacturer myself, just an independent neurologist doing research. I was about to start coding the data transfer part when I remembered the Lab Streaming Layer mentioned in OpenBCI's documentation. I must admit, it is tempting to try to use it instead of developing (and testing...) my own solution.

I'm primarily concerned about reliability (testing) and performance. Yet, there seems to even be Catch2 support on ESP32. From my casual review of the codebase, it seems that liblsl uses exception handling as a C++ alternative to C return codes. However, exception throwing is expected to come at a considerable performance penalty on ESP32. That said, during performance-critical tasks (successfully streaming data to receivers) no exception throwing should take place, should it. The ESP32 does offer a POSIX Threads-compatible API mentioned by @thiago-roque07. Yet, it is implemented on FreeRTOS, therefore liblsl's implicit thread handling may somewhat limit the FreeRTOS development approach.
None of the above seems to be a deal-breaker, but I have a gut feeling that using liblsl on ESP32 is a stretch. I don't really need the automatic host discovery or multi-host streaming as well, I would actually feel more confident with explicit networking.

I would greatly appreciate your opinion.

For the sample compilation steps, I've:

• used ESP-IDF framework
• used hello_world as a starting point
• created a simple component
  1. put liblsl repo inside the default /components/ directory
  2. used a very minimal CMakeLists file, based on the standalone_compilation_linux.sh script

     idf_component_register(
     	SRCS 
     		"src/api_config.cpp"
     		"src/ all the other .cpp files simply listed here
     		      but SRC_DIRS with globs would probably work as well"
     		"thirdparty/pugixml/pugixml.cpp"
     		"thirdparty/loguru/loguru.cpp"
     	INCLUDE_DIRS "include" "lslboost" "thirdparty/pugixml" "thirdparty/loguru"
     	PRIV_REQUIRES pthread lwip sock_utils espressif__asio
     )

• moved Espressif's ASIO implementation (directory) from managed (/managed_components) to custom (/components) components, since the compiler complained about some type casting in a printf() (if I remember correctly) and that allowed me an easy and quick one-line fix
• added liblsl (as per component's directory name) as a private dependency of the main component's (app's entrypoint) CMakeLists file
• "brute-force" renamed using err_t to using err_lslt in all the source files, as err_t was conflicting with lwip declared one
  • I'd look for a more suitable approach after establishing a functional setup
• //EDIT I forgot I've also enabled C++ exception handling in project configuration (via idf.py menuconfig), because the build system output was very verbose about it

The resultant binary was not exactly small (over 800KB), especially given that no Wi-Fi nor data acquisition-specific libraries were added yet. I used the default partition table.

```txt
binary size 0xc7f90 bytes. Smallest app partition is 0x100000 bytes. 0x38070 bytes (22%) free.
```

It compiled though. Appending lslver.c to the Hello World app worked just fine.

```txt
Hello world!
This is esp32 chip with 2 CPU core(s), WiFi/BTBLE, silicon revision v3.1, 2MB external flash
Minimum free heap size: 293432 bytes

LSL version: 116
Unknown (not set by build system)
0.108420
```

• I did confirm that setting the LSL_LIBRARY_INFO_STR in the component's (library's) CMakeLists file worked, but I didn't take the time to set it to an actual value. It was v1.16.2-34-g7e61a2ef, in case anyone is curious.

I haven't been able to successfully run the SendData examples (either the C or C++ version) yet. However, I haven't investigated it thoroughly, as I've started to question whether I should pursue this at all—hence this post.

```txt
assert failed: pthread_self pthread.c:583 (false && "Failed to find current thread ID!")
```